Unneeded artifacts in Spicy container

[awelzel]

Somewhere in the realm of zeek/zeek#2650, the spicy:latest image contains extra zkg state that seems unneeded to ship.

Without optimizing a lot, it seems from the 3.6G payload in the image, we can delete ~2GB of build/test artifacts without functional impact.

This is likely not concretely related to Spicy, but the zkg/spicy-plugin ecosystem. The container is built here, so opening the ticket here for now.

A few observations:

1. I thought build directories of zkg are ephemeral and deleted afterwards? They seem to linger on in zkg/clones
2. spicy-plugin is installing {{src}} and {{CMakeFiles}} into /opt/zeek/lib/plugins/ directories that looks like just cmake artifacts
3. Why isn't /root/.zkg/testing ephemeral
4. Could we strip .htlo files by default?

This is probably not container specific either, but likely to happen in "production environments" using zkg with spicy-plugin and spicy analyzers, too.

$ docker pull zeekurity/spicy:latest            
latest: Pulling from zeekurity/spicy                                                   
Digest: sha256:6d7ee8e1985dc9fe620352a836a15bae0c42e9a0817e6605a061e9267fb4491c                                                                                               
Status: Image is up to date for zeekurity/spicy:latest                                 
docker.io/zeekurity/spicy:latest                                                       
$ docker run --rm -it zeek/spicy:latest /bin/bash                                                                                      
root@8a51a6f8da71:~# cd /                                                              
root@8a51a6f8da71:/# du -d 3 -h | sort -h
...
225M    ./usr/lib/locale
230M    ./opt/spicy/lib
297M    ./opt/spicy
543M    ./usr/lib
660M    ./usr
778M    ./root/.zkg/clones   # See below for `build` directories
791M    ./opt/zeek/lib     # See below for CMakeFiles from spicy-plugin
821M    ./opt/zeek
943M    ./root/.zkg/testing   # This seems not needed
1.3G    ./opt
1.7G    ./root
1.7G    ./root/.zkg
3.6G    .

The build directories within the zkg clones should not be needed (not sure why they are actually still around, that might be a bug in zkg).

root@8a51a6f8da71:/# du -h -d 0 /root/.zkg/clones/package/spicy-*/build
31M     /root/.zkg/clones/package/spicy-dhcp/build
27M     /root/.zkg/clones/package/spicy-dns/build
34M     /root/.zkg/clones/package/spicy-http/build
86M     /root/.zkg/clones/package/spicy-ldap/build
47M     /root/.zkg/clones/package/spicy-pe/build
471M    /root/.zkg/clones/package/spicy-plugin/build
28M     /root/.zkg/clones/package/spicy-png/build
15M     /root/.zkg/clones/package/spicy-tftp/build
34M     /root/.zkg/clones/package/spicy-zip/build

The installed spicy-plugin contains CMakeFiles and src:

root@8a51a6f8da71:/# du -h -d 1 /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/
40K     /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/tests
28K     /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/cmake
20K     /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/spicy
26M     /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/dist
67M     /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/bin
100K    /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/include
163M    /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/src   # ?
98M     /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/lib
119M    /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/CMakeFiles   # ?
471M    /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/

So, it seems to remove all these artifacts without harm from what I can tell and shrink the unpacked filesystem size from 3.6G to 1.7G.

root@8a51a6f8da71:/# rm -rf /root/.zkg/clones/package/spicy-*/build
root@8a51a6f8da71:/# rm -rf /root/.zkg/testing/
root@8a51a6f8da71:/# rm -rf /opt/zeek/lib/zeek/plugins/packages/spicy-plugin/{CMakeFiles,src}
root@8a51a6f8da71:/# du -h -d 3 . | sort -h
...
225M    ./usr/lib/locale
230M    ./opt/spicy/lib
297M    ./opt/spicy
510M    ./opt/zeek/lib  # stripping htlo files found here will reduce this down to 220M, and /opt/zeek to 251M
540M    ./opt/zeek
543M    ./usr/lib
660M    ./usr
964M    ./opt
1.7G    .

[bbannier]

I am not sure we want to spend a lot of effort optimizing this image as the Zeek container comes with Spicy now. As far as I recall we also do not use it in any of our CI anymore.

[bbannier]

As for the unneeded files from zkg, I'd rather see us implement a solution for zeek/package-manager#140 that adding one-off fixes just here.