From 89993b3a1f4674a7b2a0ce64bd1d6c783799183b Mon Sep 17 00:00:00 2001 From: Ben Zhang Date: Mon, 16 Sep 2024 22:44:50 -0700 Subject: [PATCH] Add documentation about caching the provisioner container (#3176) ## Description In https://github.com/WATonomous/infra-config/pull/3175, we attempted to use `COPY --chmod` to allow caching for different systems (with different `umask`s). However, it didn't work. It seems that Docker calculates whether a file has changed without accounting for the `--chmod` (it's also mentioned [here](https://github.com/docker/buildx/issues/1311)). This PR reverts that change and documents this quirk. ## Checklist - [x] I have read and understood the [WATcloud Guidelines](https://cloud.watonomous.ca/docs/community-docs/watcloud/guidelines) - [x] I have performed a self-review of my code - [ ] POST-MERGE: investigate additional improvements: https://github.com/WATonomous/infra-config/issues/3178 --- .../watcloud/development-manual.mdx | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/pages/docs/community-docs/watcloud/development-manual.mdx b/pages/docs/community-docs/watcloud/development-manual.mdx index 095817e..6f5e35d 100644 --- a/pages/docs/community-docs/watcloud/development-manual.mdx +++ b/pages/docs/community-docs/watcloud/development-manual.mdx @@ -149,6 +149,35 @@ index 0641ded10f..5290b02478 100644 - ./outputs:/infra-config/outputs:rw ``` +### Caching + +We use GitHub Container Registry to cache the provisioner Docker image[^caching-details]. +The cache is used to speed up both [CI](https://github.com/WATonomous/infra-config/pull/3170) +and [local development](https://github.com/WATonomous/infra-config/pull/3175). + +The cache is automatically used. Without any changes, the following command should complete quickly[^build-time-with-cache] +and show that every stage is loaded from cache: + +```bash copy +docker compose build provisioner +``` + +On some setups, `COPY` and `ADD` commands may not be cached. +For example, when there is a custom `umask` set when git checks out files, the permissions of the files may not match the cache. +This is due to how git and Docker handle file permissions differently[^git-docker-permission-difference]. +To fix this, we can run the following to reset the non-executable bits of the files to the default: + +```bash copy +git ls-files | xargs -I '{}' chmod u+rw,go+r-w {} +``` + +[^caching-details]: [Here](https://github.com/WATonomous/infra-config/blob/121af9af1dbe78e187670163545fa6537a26757f/.github/workflows/push-images.yml#L62) is where we push the cache, and [here](https://github.com/WATonomous/infra-config/blob/121af9af1dbe78e187670163545fa6537a26757f/docker-compose.yml#L5-L6) is where we use it. The cache lives [here](https://github.com/WATonomous/infra-config/pkgs/container/infra-config). + +[^build-time-with-cache]: At the time of writing (2024-09-16), the build time with cache is about 30 seconds on a single core (Docker immediately recognizes that every layer can be cached, and downloads the image from the cache). The build time without cache is about 3 minutes and 40 seconds on 8 cores. + +[^git-docker-permission-difference]: Git [only preserves the executable bit](https://stackoverflow.com/a/3211396/4527337) of files, and uses the umask ([defaults to `022`](https://man7.org/linux/man-pages/man2/umask.2.html) on most systems) to determine the permissions of the files it creates. Docker, on the other hand, [uses all permission bits](https://docs.docker.com/engine/reference/builder/#copy) when using `COPY` or `ADD`. + + ### Port forwarding Sometimes, we may need to access services running in the container from the host machine.