chromium: Prepare for Rust.

[rakuco]

This is kind of a meta-bug to make everyone aware of https://groups.google.com/a/chromium.org/g/chromium-packagers/c/Gl30hsQ85g8/m/XTc46sRgAQAJ -- tl;dr at some point in the not-too-distant future building Chromium will also require a Rust toolchain.

I know basically nothing about Rust and Rust toolchains and I can imagine integrating one into our build will not be fun. And supporting dunfell will probably make things extra challenging :-)

[rakuco]

cc @MaxIhlenfeldt @nrpt-m @rwmacleod @kraj

[MaxIhlenfeldt]

Good catch, thanks! Looks like we're going to have some fun with build toolchain stuff in the near future 😅

I've used Rust before, but I don't think I've ever done cross compilation with it before, nor have I built the compiler itself.

[otavio]

Dunfell will be challenging, mainly due to the difference in how it fetches crates. In Dunfell's case, meta-rust could be an option; however, we will need to cut it off eventually.

[rwmacleod]

We'll have to see what version of Rust is required to decide if we keep dunfell going.

For kirkstone, we can also consider using:
https://git.yoctoproject.org/meta-lts-mixins/log/?h=kirkstone/rust-1.68

In theory that layer can be used to provide the same ( or closer to the same ) version of Rust used on master. We could make a mickledore branch as well. Anyway, it seems like we can only wait for now.

[rakuco]

I think once we have M113 there should be enough code for one to experiment passing enable_rust=true to GN (it's actually true by default in more recent milestones), seeing what's required and what parts of the toolchain stuff mentioned in the chromium-packagers email needs to be replicated without Chromium's bundled toolchain.

[rwmacleod]

I think once we have M113 there should be enough code for one to experiment passing enable_rust=true to GN (it's actually true by default in more recent milestones), seeing what's required and what parts of the toolchain stuff mentioned in the chromium-packagers email needs to be replicated without Chromium's bundled toolchain.

Unfortunately there's a new zero-day high severity CVE that is fixed in 114 so unless Rust is required for 114, we should do that update and then work on Rust support.

https://hothardware.com/news/google-emergency-0-day-patch-billions-chrome-installs

[MaxIhlenfeldt]

Looks like m115 is the first version that looks for a Rust toolchain by default:

| ERROR at //build/config/rust.gni:126:24: Script returned non-zero exit code.
|       rustc_revision = exec_script("//tools/rust/update_rust.py",
|                        ^----------
| Current dir: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-wayland-qemuarm64-mickledore/tmp-glibc/work/cortexa57-oe-linux/chromium-ozone-wayland/115.0.5790.98-r0/chromium-115.0.5790.98/out/Release/
| Command: python3 /home/gitlab-runner/yocto-chromium/builds/ci-chromium-wayland-qemuarm64-mickledore/tmp-glibc/work/cortexa57-oe-linux/chromium-ozone-wayland/115.0.5790.98-r0/chromium-115.0.5790.98/tools/rust/update_rust.py --print-package-version
| Returned 1 and printed out:
| 
| The expected Rust version is 2a8221dbdfd180a2d56d4b0089f4f3952d8c2bcd-1-llvmorg-17-init-10134-g3da83fba-2 (or fallback 2a8221dbdfd180a2d56d4b0089f4f3952d8c2bcd-1-llvmorg-17-init-10134-g3da83fba-1 but the actual version is None
| Did you run "gclient sync"?
| 
| See //BUILD.gn:17:1: whence it was imported.
| import("//build/config/rust.gni")
| ^-------------------------------
| WARNING: exit code 1 from a shell command.
NOTE: recipe chromium-ozone-wayland-115.0.5790.98-r0: task do_configure: Failed

Looking at //build/config/rust.gni, it seems we would need to set these two GN args to use a custom toolchain:

  # Chromium provides a Rust toolchain in //third_party/rust-toolchain when
  # checkout_rust is True (which is being rolled out by default over time).
  #
  # To use a custom toolchain instead, specify an absolute path to the root of
  # a Rust sysroot, which will have a 'bin' directory and others. Commonly
  # <home dir>/.rustup/toolchains/nightly-<something>-<something>
  rust_sysroot_absolute = ""

  # If you're using a Rust toolchain as specified by rust_sysroot_absolute,
  # set this to the output of `rustc -V`. Changing this string will cause all
  # Rust targets to be rebuilt, which allows you to update your toolchain and
  # not break incremental builds.
  rustc_version = ""

In the mean time, I'll try doing a build with enable_rust = false.

[rakuco]

Thanks for picking this up.

I've worked with upstream to at least get their Publish tarball and Build from Tarball bots to work with Rust and have the latter build the toolchain rather than use bundled binaries (so that it behaves more like Chromium's LLVM toolchain from a packager's perspective).

Rust became official on Linux in M116, and the bots are packaging and testing Rust with M117 on, so YMMV with M117. The bot's steps can at least provide some guidance on what needs to be done -- if we can bypass some of that by using OE's Rust stuff, even better.

[MaxIhlenfeldt]

The upstream bot changes sound great, thanks!

m116 uses a revision of Rust that according to Rust's GitHub should be included in Rust 1.71.0. I don't know what Chromium's exact requirements are, and if using e.g. https://git.yoctoproject.org/meta-lts-mixins/log/?h=kirkstone/rust-1.68 as suggested by @rwmacleod might work.

I'd be happy to do some test builds with m116 after the m115 update is done, but I won't be able to test dunfell unless we can find a fix for #726. I can both try using OE's Rust and building Chromium's custom toolchain. The latter might require using m117, though, as for m116 the tarballs don't include the toolchain sources - unless we can get the new m116 tarballs to also include the sources? Not sure whether that's possible, I'll ask on the Chromium Slack.

[rakuco]

unless we can get the new m116 tarballs to also include the sources? Not sure whether that's possible, I'll ask on the Chromium Slack.

That's not difficult, see https://source.chromium.org/chromium/chromium/tools/build/+/main:recipes/recipes/publish_tarball.py;l=359;drc=b6ab7fb2e7075263b4e7f9421708ee53b59dc5d0

I added the M117 check there because back then the bot was failing to create M115 tarballs and there were no M116 releases being made at the time.

[MaxIhlenfeldt]

Ah, great, could you please change that check to use m116 then? Or should I send a CL?

[rakuco]

Sorry if this sounds dumb but I really have zero knowledge of how Rust works: doesn't it make sense to try to use OE's toolchain first and, if that doesn't work, experiment with downloading rustc/building Chromium's Rust stuff in the recipe before sending a CL that might ultimately not be of use?

[MaxIhlenfeldt]

I think it does make sense to try OE's toolchain first. My motivation is that in the case this should turn out to not work, already having m116 tarballs available that include Chromium's toolchain sources would speed up the testing a bit. Also, other distros might appreciate having such tarballs, so even if in the end such a change might not be of use to us, it might be to others?

[rakuco]

I guess that's fair. I'll try to send a CL later today.

[rakuco]

I guess that's fair. I'll try to send a CL later today.

So I've tried to do that, but M116 lacks --sync-for-gnrt in tools/rust/build_rust.py. Another option would be trying to package some M117 dev release and build some minimal target that requires Rust (I've asked for one here). If it's small enough and doesn't require a ton of Chromium to be built, it means one wouldn't need to worry about weird clang issues or a lack of patches here and there and focus on getting Rust to work.

[MaxIhlenfeldt]

Another option would be trying to package some M117 dev release and build some minimal target that requires Rust

That sounds like a reasonable alternative, thanks for looking into it!

[medemi68]

I'm currently experiencing issues building chromium-x11 in the kirkstone branch with this issue.

[rakuco]

Could you elaborate? We haven't done anything related to Rust support yet so my guess is that you're having a separate issue.

[ensc]

I am seeing the rust toolchain problems already with recent master

| ERROR at //build/config/rust.gni:143:22: Script returned non-zero exit code.
|     rustc_revision = exec_script("//tools/rust/update_rust.py",
|                      ^----------
| Current dir: /builds/prj.2285/tmp-build/tmp/work/armv8a-oe-linux/chromium-ozone-wayland/114.0.5735.198-r0/chromium-114.0.5735.198/out/Release/
| Command: python3 /builds/prj.2285/tmp-build/tmp/work/armv8a-oe-linux/chromium-ozone-wayland/114.0.5735.198-r0/chromium-114.0.5735.198/tools/rust/update_rust.py --print-package-version
| Returned 1 and printed out:
| 
| The expected Rust version is 17c11672167827b0dd92c88ef69f24346d1286dd-1-llvmorg-17-init-8029-g27f27d15-3 (or fallback 17c11672167827b0dd92c88ef69f24346d1286dd-1-llvmorg-17-init-8029-g27f27d15-1 but the actual version is None
| Did you run "gclient sync"?
| 
| See //BUILD.gn:17:1: whence it was imported.
| import("//build/config/rust.gni")
| ^-------------------------------

[MaxIhlenfeldt]

I first saw that error in 115, so the fix is currently part of the 115 update PR: https://github.com/OSSystems/meta-browser/pull/741/files#diff-436bebc7f6c04ff6040b2273ced509fa01ef9453a4fe30c569445ee21e37e364R305

But if it happens with 114 already, we can land that change separately, as the update PR will likely still take some time to be merged.

[medemi68]

I saw that exact error in 114.

[MaxIhlenfeldt]

Re-posting this announcement from chromium-dev@:

Hello,

As of the M121 milestone, Rust is now part of the Chromium build
https://bugs.chromium.org/p/chromium/issues/detail?id=1431991 in a way
that can not be disabled, and the C++ code paths that were replaced by Rust
will be removed in the near future.

This means the following flags are no longer supported in Chromium:
- checkout_rust: False in gclient
- enable_rust = false in GN

The GN flag will continue to exist into the future for other downstream
projects that make use of our //build directory but which do not need to
use Rust, however Chromium will no longer build with it disabled.

Cheers,
Dana for the Chrome Rust team.

i.e. we will have to find a solution for this bug before the next update (121 is scheduled to be released on Jan 23). @kraj

[MaxIhlenfeldt]

I've managed to get a 120 build with enable_rust=true to work. I'm using meta-rust-bin to provide the compiler, because there's no meta-rust for dunfell in OE and https://github.com/meta-rust/meta-rust doesn't support dunfell as well. Also, Chromium needs a nightly Rust compiler, which none of the layers support out of the box, but meta-rust-bin at least has an easy-to-use script that generates a recipe for a given nightly compiler version.

I needed to tweak some things, but nothing too big; the two main changes are:

1. We need a different clang_base_path for the host and the target. I have a simple downstream patch, and have filed https://crbug.com/1513425 to see if it can be upstreamed.
2. Because the Rust compiler uses a different LLVM version than clang, we can't do ThinLTO with artifacts produced by the Rust compiler (else we get an error like Unknown attribute kind (86) (Producer: 'LLVM17.0.3-rust-1.75.0-nightly' Reader: 'LLVM 15.0.7')). This should be fixed by setting toolchain_supports_rust_thin_lto=false, but for some reason I had to disable ThinLTO completely via use_thin_lto=false. I'll check with upstream to see if this is expected or not.

Before I polish everything and send a PR, does anyone have concerns regarding this approach?

[ensc]

Also, Chromium needs a nightly Rust compiler

for master (rust 1.74.1 atm), it might be worth to test whether

export RUSTC_BOOTSTRAP = "1"

is enough to make this work.

[rwmacleod]

With dunfell scheduled to be EOL in April 2024, I think it shouldn't guide our approach to supporting chromium with rust and we should just declare dunfell to be unsupported for chromium >= 121. There were some problems, that I don't recall the specifics of, with meta-rust-bin, so it might be best to avoid that layer.

Is the version of rust in kirkstone new enough for chromium or do we need a rust mixin layer with 1.7[45]?

[otavio]

I strongly support the decision to declare Dunfell as unsupported. However, I would like to point out that using a nightly Rust compiler can significantly increase build time, unless you are using meta-rust-bin.

[MaxIhlenfeldt]

I forgot about RUSTC_BOOTSTRAP, thanks for the suggestion! afaict, Chromium only uses two nightly compiler flags, and for both of them the compiler error when using stable rustc disappeared with RUSTC_BOOTSTRAP=1 in a quick local test. I'll try a build with OE's meta-rust instead of meta-rust-bin with RUSTC_BOOTSTRAP=1.

Is the version of rust in kirkstone new enough for chromium or do we need a rust mixin layer with 1.7[45]?

I haven't actually tried, but I don't think so. One of the nightly options Chromium uses was introduced in 1.69, so kirkstone's Rust 1.68 mixin will probably not work.

[MaxIhlenfeldt]

Alright, I just did a successful build (qemuarm64, Nanbield) using OE's rust recipe. Will polish my patches a bit and upload a prototype PR hopefully later today, then we can start testing different releases and targets.

[MaxIhlenfeldt]

Up now at #782, sorry for the delay.