updates: Enable air-gapped updating via a USB drive

[ethanjli]

Devices without an internet connection need a way to get software updates via a USB drive. This would mean updating the local pallet (in a way that the updated pallet is still customizable) and all of its dependencies, which would include:

• Delivering updates to a pallet and all of its required repos and pallets, or delivering a new staged pallet bundle to drop in to the stage store. If we deliver updates as staged pallet bundles, we would also want to be able to copy everything from the bundle into the cache of repos+pallets and into the local pallet, to enable customization of the pallet. Otherwise, we would want to provide the required repos and pallets in a way that is easy to merge into the caches of repos and pallets.
• Delivering all file downloads needed for the updated pallet. If we deliver them through the staged pallet bundle, we would want to provide a way to copy all of the downloaded files from the staged pallet bundle back into the download cache. Otherwise, we would want to provide these file downloads in a way that is easy to merge into the download cache.
• Delivering all container images needed for the updated pallet. Either we need to start including container images in staged pallet bundles or we need to include container images in some format which can be loaded into the Docker daemon.

If we deliver the archive of everything needed as a custom OCI artifact or as an OCI container image (I prefer the latter, for compatibility with skopeo and crane), then we could use cosign to authenticate our archives (e.g. as in this example).

If it's reasonably simple to rehydrate things from a pallet bundle into our caches, e.g. by providing container images and file downloads from an optional cache subdirectory in the bundle and providing some additional metadata in a caches section of the bundle manifest (or maybe having the manifest's caches section record where each exported download file should be copied to in the downloads cache), that may be a really elegant approach - because then the bundle could also be dropped directly into the stage store for use even if the cache files aren't loaded into the Forklift cache. Such a bundle would be fully self-contained for deployment anywhere (assuming a compatible CPU architecture, Forklift version, Docker, etc.). And then on systems where user customization isn't important (e.g. a fleet of uniform devices) and we don't need to run plt subcommands, we would just drop bundles into Forklift's stage store with a stage import-bundle subcommand without having a local pallet or caches.

Ideally, the subcommands for air-gapped updates would look like:

• plt switch --from-bundle {tarball path} if we want to rehydrate things from a bundle into the local pallet and the cache
• stage import-bundle {tarball path} (with an optional flag to set it as the next staged bundle) if we just want to copy the bundle into the stage store

Ideally, the subcommands for making update bundles would look like:

• plt export-bundle {output path} with an optional --include-caches flag which is enabled by default
• stage export-bundle {bundle name or index} {output path} (or stage plt --bun={bundle name or index} export-bundle {output path}) with an optional --include-caches flag which is enabled by default and which, if enabled for a bundle without a complete caches directory, may require downloading some additional files

We should probably establish a convention that the bundle should be named {pallet path}@{version or pseudoversion with a "dirty" indicator}+{cpu architecture}.forklift-bundle.tar.gz

[ethanjli]

For delivery of container images, here are some leads:

• https://github.com/mesosphere/mindthegap
• https://github.com/containers/skopeo?tab=readme-ov-file#syncing-registries and https://github.com/containers/image/blob/main/docs/containers-registries.conf.5.md#remapping-and-mirroring-registries
• https://github.com/containers/skopeo?tab=readme-ov-file#copying-images (e.g. as in https://github.com/PlanktoScope/PlanktoScope/blob/e3dbfc7ed25da6c0cd5e3159cf6880f3e3abda38/.github/workflows/build-os.yml#L161 and https://github.com/PlanktoScope/PlanktoScope/blob/e3dbfc7ed25da6c0cd5e3159cf6880f3e3abda38/software/distro/setup/base-os/forklift/precache-image.sh#L14) and https://pkg.go.dev/github.com/docker/docker/client#Client.ImageLoad (e.g. as in https://github.com/PlanktoScope/PlanktoScope/blob/e3dbfc7ed25da6c0cd5e3159cf6880f3e3abda38/software/distro/setup/base-os/forklift/transfer-precached-image.sh#L7)

For authentication of update payloads, here are some leads:

• https://github.com/sigstore/cosign
• https://www.youtube.com/watch?v=fJCHObiMY6M

[ethanjli]

We should also add a GitHub Actions workflow to automatically build update bundles for each CPU architecture, so that they can be attached to GitHub Releases. Then we can tell Forklift to stage import-bundle a .forklift-bundle.tar.gz file (which is limited to 2 GB on FAT32 filesystens) or a .forklift-bundle directory (whose individual tar archives would be limited to 2 GB on FAT32 filesystems).

Note that PlanktoScope OS should add some mechanism to make it easy for users to mount USB drives - either automatically or with some easy web interface (e.g. Cockpit or some preexisting project).

[ethanjli]

If we choose to include container image archives in staged pallet bundles, perhaps a compact on-disk layout for them could be to hard-link them into Forklift's container image cache (see #245). We'd probably want to gzip them (see #245 (comment)). We'd need to make sure that the hard link is resolved so that the archives get copied into the staged pallet bundles when they're compressed into a .tar.gz file, instead of just being saved as links.

If we can somehow include container images via OCI image layering (but in such a way that the container images are stored in subdirectories), that could introduce some storage efficiencies for shared base layers. idk if that's even possible though.

[ethanjli]

If we publish staged pallet bundles as OCI artifacts (either OCI container images or custom OCI artifacts), it might be worth showing how to add them to the ArtifactHub OCI artifact index. See https://universal-blue.discourse.group/t/listing-your-custom-image-on-artifacthub/6446 for details.