Docker Buildkit and GitHub docker registry

Building a docker image in insecure mode (--privileged when running)

Requires a docker buildkit instance with --allow-insecure-entitlement security.insecure set

docker buildx create \
  --driver docker-container \
  --use \
  --buildkitd-flags "--allow-insecure-entitlement security.insecure" \
  --driver-opt network=host \
  --name "insecure"

Due to the use of the docker-container driver the buildkit instance does not have access to the local docker images (docker image ls) so needs to pull them from a registry. (Using the docker driver results in the error failed to find driver "docker").

Therefore the --driver-opt network=host is added, so the spawned docker buildkit instance can communicate with a local running docker registry.

docker run -d -p 127.0.0.1:5000:5000 --name registry --rm registry:2

--cache-from and docker buildkit

For the --cache-from statements to work, the images need to be built with an additional flag:

--build-arg BUILDKIT_INLINE_CACHE=1

Also, important to note is that the GitHub docker registry does not seem to work (just yet) with the --cache-from statements and docker buildkit: https://github.com/containerd/containerd/issues/3291#issuecomment-645974051. As a workaround the image can be pulled from the GitHub registry into the local docker daemon and pushed into a local docker registry before building the actual image.

Complete example with insecure buildkit instance and using --cache-from

# Get current commit (for a unique sha)
GIT_SHA=$(git rev-parse HEAD)

# Setup local docker registry
docker run -d -p 127.0.0.1:5000:5000 --name registry --rm registry:2 1>&2 || true

# Create buildkit instance with allows insecure run commands
docker buildx create \
  --driver docker \
  --use \
  --buildkitd-flags "--allow-insecure-entitlement security.insecure" \
  --driver-opt network=host \
  --name "insecure"

# Login to the GitHub registry
docker login -u mvgijssel -p "password" docker.pkg.github.com

# Pull the docker image from GitHub
docker pull "docker.pkg.github.com/mvgijssel/setup/some-image:latest"

# Tag the image with the local registry prefixed
docker tag "docker.pkg.github.com/mvgijssel/setup/some-image:latest" "localhost:5000/some-image:latest"

# Push the image into the local registry
docker push "localhost:5000/some-image:latest"

# Build the image in buildkit with inline caching set
docker buildx build \
  --progress plain \
  --load \
  --build-arg BUILDKIT_INLINE_CACHE=1 \
  --cache-from "localhost:5000/some-image:latest"
  --tag "localhost:5000/some-image:$GIT_SHA" \
  .

# Push the image into the local registry so it an be used by other Dockerfiles 
# Depending on this image on the same machine / session
docker push "localhost:5000/some-image:$GIT_SHA"

# Tag the image with the GitHub registry
docker tag "localhost:5000/some-image:$GIT_SHA" "docker.pkg.github.com/mvgijssel/setup/some-image:$GIT_SHA"

# Push the image to GitHub registry
docker push "docker.pkg.github.com/mvgijssel/setup/some-image:$GIT_SHA"