-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Local Remote Execution for Rust #1510
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 1 of 2 LGTMs obtained, and 37 of 38 files reviewed, and pending CI: Bazel Dev / macos-13, Bazel Dev / macos-14, Bazel Dev / ubuntu-24.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Coverage, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / lre-cc / ubuntu-24.04, Local / lre-rs / macos-14, Local / lre-rs / ubuntu-24.04, NativeLink.com Cloud / Remote Cache / macos-14, NativeLink.com Cloud / Remote Cache / ubuntu-24.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / lre-cc / large-ubuntu-22.04, Remote / lre-rs / large-ubuntu-22.04, Web Platform Deployment / macos-14, Web Platform Deployment / ubuntu-24.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 6 discussions need to be resolved (waiting on @allada)
web/platform/src/content/docs/docs/contribute/bazel.mdx
line 29 at r2 (raw file):
```sh bazel run nativelink -- \ $(pwd)/nativelink-config/examples/basic_cas.json
this should be /basic_cas.json5
web/platform/src/content/docs/docs/contribute/bazel.mdx
line 58 at r2 (raw file):
```sh bazel run -c opt --extra_toolchains=@rust_toolchains//:all nativelink -- \ $(pwd)/nativelink-config/examples/basic_cas.json
dito
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 1 of 2 LGTMs obtained, and 36 of 38 files reviewed, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / macos-13, Bazel Dev / macos-14, Bazel Dev / ubuntu-24.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Coverage, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / lre-cc / ubuntu-24.04, Local / lre-rs / macos-14, Local / lre-rs / ubuntu-24.04, NativeLink.com Cloud / Remote Cache / macos-14, NativeLink.com Cloud / Remote Cache / ubuntu-24.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / lre-cc / large-ubuntu-22.04, Remote / lre-rs / large-ubuntu-22.04, Web Platform Deployment / macos-14, Web Platform Deployment / ubuntu-24.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 4 discussions need to be resolved (waiting on @allada)
local-remote-execution/overlays/lre-rs.nix
line 152 at r1 (raw file):
Previously, allada (Nathan (Blaise) Bruer) wrote…
nit: Why duplicate this as it is listed one line below on all of these?
This comment is there so that the generated file shows how to get this tag: https://github.com/TraceMachina/nativelink/pull/1510/files#diff-5a46d2c5053d31d4bf9ad698fbfb1187e54d6131385a01c8c502ae8ab838dff3
The setup automatically adds these flake outputs when the lre flake module is enabled, so we can be sure that these paths are correct.
The idea is that e.g. cloud users only need to look at the platform definitions on the Bazel side to figure out how to get these hashes without going into the nix generator code.
local-remote-execution/rust/BUILD.bazel
line 77 at r1 (raw file):
Previously, allada (Nathan (Blaise) Bruer) wrote…
nit: Lets document on why these defaults.
Done. These are rules_rust
's default, but I've already experimented with adjustments to these and we'll likely want to change them in the future.
local-remote-execution/rust/BUILD.bazel
line 109 at r1 (raw file):
Previously, allada (Nathan (Blaise) Bruer) wrote…
nit: ditto...
Also, IIRC
fastbuild
is opt=1
Done. Left a comment that this is the default. This seems to be a deliberate choice by rules_rust. It seems better to investigate further before we deviate from this.
I already noticed though that since we have much more scaling we can probably start doing something like codegen-units=1
for opt builds in the future as the Bazel builds with mold seem fast enough to handle it. For now we can still override the behavior via the extra_rustc_flags
setting in a .bazelrc
.
local-remote-execution/rust/BUILD.bazel
line 123 at r1 (raw file):
Previously, allada (Nathan (Blaise) Bruer) wrote…
Don't we want to include these libs as part of the toolchain and/or put the full paths?
Ah this is a tricky one. The actual search paths to these aren't actually specified, but set by the linker which sets them via (capital) -L
. In a wrapper.
You can inspect this behavior like this:
bazel build local-remote-execution/examples:lre-rs --platforms=@local-remote-execution//rust/platforms:x86_64-unknown-linux-gnu
INFO: Invocation ID: 9a4841c6-358b-4ec4-b1d8-3429f5216c2b
WARNING: Build option --platforms has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //local-remote-execution/examples:lre-rs (0 packages loaded, 1822 targets configured).
INFO: From Compiling Rust bin lre-rs (1 files):
LC_ALL="C" PATH="bazel-out/k8-fastbuild/bin/external/local-remote-execution~/rust/rust-stable-x86_64-linux_impl/lib/rustlib/x86_64-unknown-linux-gnu/bin:/nix/store/kl18sr2a45vyyqv39clrjclmadacy9mc-rust-default-1.82.0/lib/rustlib/x86_64-unknown-linux-gnu/bin:/nix/store/kl18sr2a45vyyqv39clrjclmadacy9mc-rust-default-1.82.0/lib/rustlib/x86_64-unknown-
linux-gnu/bin:/bin:/usr/bin:/usr/local/bin" VSLANG="1033" "/nix/store/lw8g7lpw6rwjmz6wxp2grzaixlvn02a7-customClang/bin/customClang" "-m64" "/tmp/rustcByxKBc/symbols.o" "bazel-out/k8-fastbuild/bin/local-remote-execution/examples/lre_rs.lre_rs.c59005c95ffd25cf-cgu.0.rcgu.o" "bazel-out/k8-fastbuild/bin/local-remote-execution/examples/lre_rs.936mcif4h
kbf22dehgh7mxezr.rcgu.o" "-Wl,--as-needed" "-Wl,-Bstatic" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-22be60875a4ac8d7.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpani
c_unwind-4b832a03827ff95e.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-a25e7f31e2204719.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-1f1858edbb50ccb5.rli
b" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-1319b75889377807.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-c9972d847170eb93.rlib" "/nix/store/8xz2aamlsz01
36jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-0868a75d0b80f801.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-f4254a923640cbea.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8k
d5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-f21de9b2e2c92770.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-cf868f78468d45dd.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rus
t-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-cce9a55233783113.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-6ef1f692f91f321e.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unkno
wn-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-d2b4a3bc57672e2c.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-9d8e4e6f4cb45592.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_
64-unknown-linux-gnu/lib/liblibc-858b7338441a56b6.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-d182631d04e1eed2.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librust
c_std_workspace_core-7874c355ab5ed077.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-46b25da62cc69443.rlib" "/nix/store/8xz2aamlsz0136jzlv15v9cyjdy8kd5l-rust-std-1.82.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins
-c71612932829263c.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "bazel-out/k8-fastbuild/bin/external/local-remote-execution~/rust/rust-stable-x86_64-linux_impl/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "bazel-out/k8-fastbuild/bin/external/local-remote-execution~/r
ust/rust-stable-x86_64-linux_impl/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "bazel-out/k8-fastbuild/bin/local-remote-execution/examples/lre-rs" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "-fuse-ld=/nix/store/9ms692fd1qf6435c1kzxvsjrg3fg58ys-llvm-binutils-wrapper-19.1.3/bin/ld.mold" "-B/nix/store/lw8g7lpw6rwjmz6wxp2grzai
xlvn02a7-customClang/bin" "-Wl,-no-as-needed" "-Wl,-z,relro,-z,now" "-Wl,--push-state,-as-needed" "-lstdc++" "-Wl,--pop-state" "-Wl,--push-state,-as-needed" "-lm" "-Wl,--pop-state"
INFO: Found 1 target...
Target //local-remote-execution/examples:lre-rs up-to-date:
bazel-bin/local-remote-execution/examples/lre-rs
INFO: Elapsed time: 0.769s, Critical Path: 0.41s
INFO: 35 processes: 34 internal, 1 linux-sandbox.
INFO: Build completed successfully, 35 total actions
Note the path to mold that it expands to. This mold the linker wrapper script from the lre-cc toolchain config:
"-fuse-ld=/nix/store/9ms692fd1qf6435c1kzxvsjrg3fg58ys-llvm-binutils-wrapper-19.1.3/bin/ld.mold", |
(You can do cat /nix/store/9ms692fd1qf6435c1kzxvsjrg3fg58ys-llvm-binutils-wrapper-19.1.3/bin/ld.mold
to see its contents)
In this wrapper script the -L
variables are set to the nix store paths.
The final executable is linked correctly to the nix store variants:
ldd bazel-bin/local-remote-execution/examples/lre-rs
linux-vdso.so.1 (0x00007fb190dc7000)
libgcc_s.so.1 => /nix/store/nhdjqagplar2fdrqnrxlab8vg9i36b3d-gcc-13.3.0-libgcc/lib/libgcc_s.so.1 (0x00007fb190d43000)
libc.so.6 => /nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 (0x00007fb190b4a000)
/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/ld-linux-x86-64.so.2 => /nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib64/ld-linux-x86-64.so.2 (0x00007fb190dc9000)
Ideally we'd indeed want to have the explicit paths to all the rust tools directly in this file without any module extension at all. But unfortunately rules_rust doesn't support raw strings to executables yet in the same way that rules_cc does. If it did we could use string_flag
s to make these values customizable and e.g. change a single tool without invalidating all outputs of the toolchain.
So this variant is not ideal, but it's still correct as the libraries are bundled via lre-cc.
This will be "fixed" when we can fix this todo: https://github.com/TraceMachina/nativelink/pull/1510/files#diff-39883609ed94bf2ed003211776a71f854571a21d3e9262d2146843549922d506
web/platform/src/content/docs/docs/contribute/bazel.mdx
line 29 at r2 (raw file):
Previously, SchahinRohani (Schahin) wrote…
this should be
/basic_cas.json5
Done. Nice catch!
web/platform/src/content/docs/docs/contribute/bazel.mdx
line 58 at r2 (raw file):
Previously, SchahinRohani (Schahin) wrote…
dito
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1 of 1 files at r3.
Reviewable status: 1 of 2 LGTMs obtained, and 37 of 38 files reviewed, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / macos-13, Bazel Dev / macos-14, Bazel Dev / ubuntu-24.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Coverage, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / lre-cc / ubuntu-24.04, Local / lre-rs / macos-14, Local / lre-rs / ubuntu-24.04, NativeLink.com Cloud / Remote Cache / macos-14, NativeLink.com Cloud / Remote Cache / ubuntu-24.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / lre-cc / large-ubuntu-22.04, Remote / lre-rs / large-ubuntu-22.04, Web Platform Deployment / macos-14, Web Platform Deployment / ubuntu-24.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 2 discussions need to be resolved (waiting on @allada)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The biggest gap that I see here is the absence of a README that explains clearly how to use this addition. I understand it will be used automatically, but we need to spell it out for our users and there needs to be a documentation page that ales it easy to exploit.
Reviewable status: 1 of 2 LGTMs obtained, and 37 of 38 files reviewed, and pending CI: docker-compose-compiles-nativelink (22.04), and 2 discussions need to be resolved (waiting on @allada)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The biggest gap that I see here is the absence of a README that explains clearly how to use this addition. I understand it will be used automatically, but we need to spell it out for our users and there needs to be a documentation page that ales it easy to exploit.
I agree, but I'd prefer to do this in a separate change so that I can create uniform docs between lre-cc
and lre-rs
without expanding the (already pretty massive) scope of this PR further. Definitely will need docs and example setups though.
Reviewable status: 1 of 2 LGTMs obtained, and 37 of 43 files reviewed, and pending CI: Analyze (python), Bazel Dev / macos-13, Bazel Dev / macos-14, Bazel Dev / ubuntu-24.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Coverage, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / lre-cc / ubuntu-24.04, Local / lre-rs / macos-14, Local / lre-rs / ubuntu-24.04, NativeLink.com Cloud / Remote Cache / macos-14, NativeLink.com Cloud / Remote Cache / ubuntu-24.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / lre-cc / large-ubuntu-22.04, Remote / lre-rs / large-ubuntu-22.04, Web Platform Deployment / macos-14, Web Platform Deployment / ubuntu-24.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 2 discussions need to be resolved (waiting on @allada)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 30 of 38 files at r1, 1 of 1 files at r2, 1 of 1 files at r3, 5 of 5 files at r4, all commit messages.
Reviewable status: 2 of 2 LGTMs obtained, and all files reviewed, and 3 discussions need to be resolved
.github/workflows/native-bazel.yaml
line 76 at r4 (raw file):
if [ "$RUNNER_OS" == "Linux" ] || [ "$RUNNER_OS" == "macOS" ]; then bazel test //... \ --extra_toolchains=@rust_toolchains//:all \
nit: ditto.
local-remote-execution/examples/BUILD.bazel
line 8 at r4 (raw file):
srcs = ["hello.cpp"], copts = ["--verbose"], linkopts = ["-Wl,--verbose"],
can we remove verbose? It makes the stderr/stdout really noisy for me.
.github/workflows/main.yml
line 162 at r4 (raw file):
bazel clean && \ bazel test //... \ --extra_toolchains=@rust_toolchains//:all \
These should probably not use this, since this workflow is testing docker compose and we'd rather test the default toolchains before anything else.
This change implements ~40 Rust toolchain configurations that reuse Nix's executables and make them available via Bazel. The new setup is fully reproducible, supports musl and glibc and works from all nix-supported hosts to all sensible crosscompilation targets seamlessly through remote and local execution. Until we have the actual infrastructure set up to fully test all configurations we have these 12 which are practically usable immediately: - MacOS native, aarch64 and x86_64, stable and nightly - Linux native, aarch64 and x64_64, stable and nightly, glibc and musl The channel is configurable via a new `--@local-remote-execution//rust:channel` flag while the libc version is configurable via the target platform. For instance: ```bash bazel build \ nativelink \ --@local-remote-execution//rust:channel=nightly \ --platforms=@local-remote-execution//rust/platforms:x86_64-unknown-linux-gnu bazel build \ nativelink \ --platforms=@local-remote-execution//rust/platforms:x86_64-unknown-linux-musl ```
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 34 of 38 files at r1, 1 of 1 files at r2, 1 of 1 files at r3, 5 of 5 files at r4, 3 of 3 files at r5, all commit messages.
Dismissed @allada from 3 discussions.
Reviewable status: complete! 2 of 2 LGTMs obtained, and all files reviewed
.github/workflows/main.yml
line 162 at r4 (raw file):
Previously, allada (Nathan (Blaise) Bruer) wrote…
These should probably not use this, since this workflow is testing docker compose and we'd rather test the default toolchains before anything else.
Done. This is actually correct as using rules_rust
's default toolchains requires explicitly registering these. Previously we did this via the .bazelrc
but since we have multiple toolchain configurations now it's no longer possible to do this as it would override the lre-rs
toolchain configuration. I've fixed the example in the docker-compose readme and added an explainer.
local-remote-execution/examples/BUILD.bazel
line 8 at r4 (raw file):
Previously, allada (Nathan (Blaise) Bruer) wrote…
can we remove verbose? It makes the stderr/stdout really noisy for me.
Done.
This change implements ~40 Rust toolchain configurations that reuse Nix's executables and make them available via Bazel.
The new setup is fully reproducible, supports musl and glibc and works from all nix-supported hosts to all sensible crosscompilation targets seamlessly through remote and local execution.
Until we have the actual infrastructure set up to fully test all configurations we have these 12 which are practically usable immediately:
The channel is configurable via a new
--@local-remote-execution//lre-rs:channel
flag while the libc version is configurable via the target platform. For instance:This change is