Use docker to create a buck2 image

[adam-singer]

Description

Combine a nix layer ontop of a ubuntu image for
easy to use toolchain for building buck2.

Fixes #1274

Type of change

• New feature (non-breaking change which adds functionality)

---

This change is 

[allada]

Reviewed 2 of 3 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and all files reviewed, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 5 discussions need to be resolved

---

tools/toolchain-buck2/Dockerfile line 15 at r2 (raw file):

# limitations under the License.

FROM ubuntu:22.04@sha256:f9d633ff6640178c2d0525017174a688e2c1aef28f0a0130b26bd5554491f0da AS dependencies

nit: Please add a date on when this hash was grabbed.

---

tools/toolchain-buck2/Dockerfile line 37 at r2 (raw file):

# hadolint ignore=DL3003,DL3059
RUN git clone https://github.com/TraceMachina/buck2

nit: Shouldn't we grab mainline?

---

tools/toolchain-buck2/Dockerfile line 41 at r2 (raw file):

RUN echo 'if [ -d "/buck2" ]; then\n  nix develop "/buck2" --impure --command bash\nfi' >> /etc/profile
# hadolint ignore=DL3059
RUN bash -c 'source /etc/profile && cd buck2 && nix develop'

nit: Maybe only git clone the repo to grab the .nix and/or .flake files then remove the files so we have a smaller image?

---

tools/toolchain-buck2/toolchain-buck2.sh line 2 at r2 (raw file):

#!/usr/bin/env bash

nit: can we have the copyright and a description of what this file is for?

---

tools/toolchain-buck2/toolchain-buck2.sh line 13 at r2 (raw file):

SRC_ROOT=$(git rev-parse --show-toplevel)
FLAKE_NIX_FILE="${SRC_ROOT}/flake.nix"
echo "WARNING: This script will modify and revert the flake.nix"

nit: Maybe instead make sure the git repo is clean?

[adam-singer]

Reviewable status: 1 of 1 LGTMs obtained, and all files reviewed, and pending CI: Installation / macos-13, and 5 discussions need to be resolved

---

tools/toolchain-buck2/Dockerfile line 37 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: Shouldn't we grab mainline?

yes we could, originally I had some custom chains in this repo. I don't think I need them any longer.

---

tools/toolchain-buck2/Dockerfile line 41 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: Maybe only git clone the repo to grab the .nix and/or .flake files then remove the files so we have a smaller image?

Going to remove this logic, it isn't doing what I had hoped. I had hoped that by doing this it would enter me into a nix shell when nativelink starts. The reality is it all gets ignored once we smash together the nativelink image with this image. Will sync with @aaronmondal on other options. Ideally this is where entrypoint script or other things discussed can make our lives easier.

---

tools/toolchain-buck2/toolchain-buck2.sh line 13 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: Maybe instead make sure the git repo is clean?

I don't have a strong opinion on that atm, rather leave as is due to making multiple changes at the same time to quickly test.

[adam-singer]

Reviewable status: 1 of 1 LGTMs obtained, and all files reviewed, and pending CI: Installation / macos-13

---

tools/toolchain-buck2/Dockerfile line 15 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: Please add a date on when this hash was grabbed.

Done.

---

tools/toolchain-buck2/Dockerfile line 37 at r2 (raw file):

Previously, adam-singer (Adam Singer) wrote…

yes we could, originally I had some custom chains in this repo. I don't think I need them any longer.

Done

---

tools/toolchain-buck2/Dockerfile line 41 at r2 (raw file):

Previously, adam-singer (Adam Singer) wrote…

Going to remove this logic, it isn't doing what I had hoped. I had hoped that by doing this it would enter me into a nix shell when nativelink starts. The reality is it all gets ignored once we smash together the nativelink image with this image. Will sync with @aaronmondal on other options. Ideally this is where entrypoint script or other things discussed can make our lives easier.

Done

---

tools/toolchain-buck2/toolchain-buck2.sh line 2 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: can we have the copyright and a description of what this file is for?

Done.

[aaronmondal]

Reviewed 3 of 3 files at r3, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and all files reviewed, and 8 discussions need to be resolved

---

tools/toolchain-buck2/Dockerfile line 30 at r3 (raw file):

RUN curl -L https://go.dev/dl/go1.23.0.linux-amd64.tar.gz -o go1.23.0.linux-amd64.tar.gz
RUN rm -rf /usr/local/go && tar -C /usr/local -xzf go1.23.0.linux-amd64.tar.gz

You can consolidate these with a pipe to reduce a layer.

---

tools/toolchain-buck2/Dockerfile line 32 at r3 (raw file):

RUN rm -rf /usr/local/go && tar -C /usr/local -xzf go1.23.0.linux-amd64.tar.gz

RUN curl -L https://nixos.org/nix/install -o install-nix.sh && \

The new nix installer provides an SBOM and automatically sets up flake support: https://github.com/DeterminateSystems/nix-installer

---

tools/toolchain-buck2/Dockerfile line 41 at r3 (raw file):

RUN git clone https://github.com/facebook/buck2
# hadolint ignore=DL3059
RUN bash -c 'source /etc/profile && cd buck2 && nix develop && cd .. && rm -rf buck2'

To get the tooling from buck2 into the image you can use this instead:

# Last pinned 2024-08-24
nix develop github:facebook/buck2/d76c189ed6092d7b53506b9411241680923d593b

---

tools/toolchain-buck2/toolchain-buck2.sh line 33 at r3 (raw file):

function ecr_login() {
    aws ecr get-login-password --profile ${ECR_PROFILE} --region ${ECR_REGION} | docker login --username ${ECR_USER} --password-stdin ${ECR}

nit: linebreaks

---

tools/toolchain-buck2/toolchain-buck2.sh line 39 at r3 (raw file):

docker buildx build --no-cache=${BUILDX_NO_CACHE} \
    --platform linux/amd64 \
    -t localhost:5001/toolchain-buck2:latest \

You can use something like this to make this more generic:

IMAGE_TAG=$(nix eval .#nativelink-worker-toolchain-buck2.imageTag --raw)
IMAGE_NAME=$(nix eval .#nativelink-worker-toolchain-buck2.imageName --raw)

docker.... \
  -t localhost:5001/${IMAGE_NAME}:${IMAGE_TAG}

edit: Ah we're creating multiple images here. Could we make this structure a bit clearer (e.g. via comments)? It's a bit confusing what the difference between toolchain-buck2, nativelink-toolchain-buck2 and nativelink-worker-toolchain-buck2 is.

---

tools/toolchain-buck2/toolchain-buck2.sh line 73 at r3 (raw file):

set +o pipefail
SHA256_HASH=$(
    nix run .#nativelink-worker-toolchain-buck2.copyTo docker://localhost:5001/nativelink-toolchain-buck2:latest -- --dest-tls-verify=false 2>&1 |

nit: linebreaks

---

tools/toolchain-buck2/toolchain-buck2.sh line 83 at r3 (raw file):

# Patch flake.nix with sha256 value.
sed -i -E "s|sha256 = \"\"; # DO NOT COMMIT BUCK2 SHA256 VALUE|sha256 = \"${SHA256_HASH}\"; # DO NOT COMMIT BUCK2 SHA256 VALUE|" "${FLAKE_NIX_FILE}"

no nit: probably can't linebreak here 😞

---

tools/toolchain-buck2/toolchain-buck2.sh line 98 at r3 (raw file):

fi

# Wrap it with nativelink to turn it into a worker.

The create-worker no longer wraps an image with nativelink. Instead, it now just adds a temp directory, a user, coreutils and bash. NativeLink is now mounted into the image at runtime.

---

tools/toolchain-buck2/toolchain-buck2.sh line 105 at r3 (raw file):

# Pull in to local docker and tag.
docker pull localhost:5001/nativelink-toolchain-buck2:latest

Instead of docker, use skopeo copy since that's what copyTo uses under the hood as well (and it's already in the flake). This should let you collapse this to just an ecr_login and a skopeo copy invocation.

[adam-singer]

Reviewable status: 1 of 1 LGTMs obtained, and all files reviewed, and 8 discussions need to be resolved

---

tools/toolchain-buck2/Dockerfile line 30 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

You can consolidate these with a pipe to reduce a layer.

sg, if you have any other ideas of installing go, I'm open to that. Installing via nix might be what I want. Didn't want to mess around with nix too much when we got to this state, whats an easy/quick way to install this version of go without much drama?

---

tools/toolchain-buck2/Dockerfile line 32 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

The new nix installer provides an SBOM and automatically sets up flake support: https://github.com/DeterminateSystems/nix-installer

k, will need to test that out, see what it entails

---

tools/toolchain-buck2/Dockerfile line 41 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

To get the tooling from buck2 into the image you can use this instead:

# Last pinned 2024-08-24
nix develop github:facebook/buck2/d76c189ed6092d7b53506b9411241680923d593b

nice, that will sync all the nix deps

---

tools/toolchain-buck2/toolchain-buck2.sh line 83 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

no nit: probably can't linebreak here 😞

Yes, this is a regex, we probably could but its not worth it.

Done.

---

tools/toolchain-buck2/toolchain-buck2.sh line 105 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Instead of docker, use skopeo copy since that's what copyTo uses under the hood as well (and it's already in the flake). This should let you collapse this to just an ecr_login and a skopeo copy invocation.

Can you provide an example? Are you saying just run .copyTo ?

[adam-singer]

Reviewable status: 1 of 1 LGTMs obtained, and 2 of 4 files reviewed, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable, and 6 discussions need to be resolved

---

tools/toolchain-buck2/Dockerfile line 41 at r3 (raw file):

Previously, adam-singer (Adam Singer) wrote…

nice, that will sync all the nix deps

This isn't perfect atm, I'm wondering of ways we can propagate information out of the container into a metadata layer or manifest labels such that external system could consume them if they needed. I think this is "ok" for now.

Done.

---

tools/toolchain-buck2/toolchain-buck2.sh line 33 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

nit: linebreaks

Done.

---

tools/toolchain-buck2/toolchain-buck2.sh line 73 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

nit: linebreaks

Done.

---

tools/toolchain-buck2/toolchain-buck2.sh line 98 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

The create-worker no longer wraps an image with nativelink. Instead, it now just adds a temp directory, a user, coreutils and bash. NativeLink is now mounted into the image at runtime.

Done.

---

tools/toolchain-buck2/toolchain-buck2.sh line 105 at r3 (raw file):

Previously, adam-singer (Adam Singer) wrote…

Can you provide an example? Are you saying just run .copyTo ?

Done.

[adam-singer]

Reviewable status: 1 of 1 LGTMs obtained, and 2 of 4 files reviewed, and 6 discussions need to be resolved

---

tools/toolchain-buck2/toolchain-buck2.sh line 39 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

You can use something like this to make this more generic:

IMAGE_TAG=$(nix eval .#nativelink-worker-toolchain-buck2.imageTag --raw)
IMAGE_NAME=$(nix eval .#nativelink-worker-toolchain-buck2.imageName --raw)

docker.... \
  -t localhost:5001/${IMAGE_NAME}:${IMAGE_TAG}

edit: Ah we're creating multiple images here. Could we make this structure a bit clearer (e.g. via comments)? It's a bit confusing what the difference between toolchain-buck2, nativelink-toolchain-buck2 and nativelink-worker-toolchain-buck2 is.

will follow up on this one because we don't need create worker going forward for this kind of setup

Done.

[aaronmondal]

Reviewed 2 of 2 files at r4, all commit messages.
Reviewable status: complete! 2 of 1 LGTMs obtained, and all files reviewed

---

tools/toolchain-buck2/Dockerfile line 30 at r3 (raw file):

Previously, adam-singer (Adam Singer) wrote…

sg, if you have any other ideas of installing go, I'm open to that. Installing via nix might be what I want. Didn't want to mess around with nix too much when we got to this state, whats an easy/quick way to install this version of go without much drama?

Yeah via nix is probably the way, i.e something like nix profile install nixpkgs#go, but we can figure out the details of that later.