lets pick an archive format

[max-mapper]

we wanna support an archive format for storing + exchanging compressed files. in our specific use case the files will be container filesystems (e.g. a debootstrapped ubuntu.iso), as well as a set of research data (csvs etc) that might be used in an analysis pipeline

two features we want:

• format supports random access on compressed archives
• try and use something that exists and is on peoples computers already (e.g. they can double click the thing to extract it)

some candidates:

• zip
• something like tar or xz with block compression and an external file index for files -> byte offsets

research:

• http://asdf-standard.readthedocs.org/en/latest/
• https://wiki.freedesktop.org/www/Software/systemd/Particle/
• https://github.com/stevvooe/continuity
• bundle: filesystem metadata format opencontainers/runtime-spec#11
• https://github.com/vasi/pixz
• What does this format need to do? ipfs-inactive/archive-format#7
• http://libguestfs.org/virt-sparsify.1.html

[wohali]

Another thing worth looking at: https://github.com/fidlej/idzip

[onbjerg]

pifs? 🌚

[max-mapper]

@mrocklin reminded me to look into dar http://dar.linux.free.fr/doc/man/dar.html, seems pretty good so far

[max-mapper]

some initial cons of dar:

• libdar is 2.7mb of c++ code
• development is still on sourceforge
• seems quite complex, other independent implementations would likely be very difficult (contrast this to tar which is quite easy to implement)

[ekg]

Another con is that it doesn't seem widely used.
On Nov 29, 2015 10:59 PM, "maxogden" [email protected] wrote:

some initial cons of dar:

• libdar is 2.7mb of c++ code
• development is still on sourceforge
• seems quite complex, other independent implementations would likely
  be very difficult (contrast this to tar which is quite easy to implement)

—
Reply to this email directly or view it on GitHub
#27 (comment)
.