Add download-only flag to bootstrap for early exit

[rornic]

Resolves #83455.

Adds the --download-only to bootstrap.py so it can be run as follows:

./x.py --download-only

which allows the downloads to happen independently of other commands.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Mark-Simulacrum (or someone else) soon.

Please see the contribution instructions for more information.

[Mark-Simulacrum]

I am not sure we should have a dedicated flag for this; we already have a number of downloads (e.g., submodule updates after #97513, IIRC LLVM toolchains today) that are separate from bootstrap.py, and some of those are even dynamic.

I am concerned that this flag will end up needing to get threaded around all over the place and bail outs added. CI today has a special cased make prepare which basically accomplishes the same as this flag (a little more, I believe, but not by much). It does also download the Cargo dependencies, which seems like an important omission from the goals of a network-less build.

For distros and others building offline, I believe my current understanding and expectation is that most folks are bootstrapping from a locally installed toolchain (i.e., config.toml rustc=, cargo= configured) and the rust-src tarball. That combination essentially ensures that we build reasonably offline, and doesn't need special flags passed to do so (for the most part, anyway).

[Mark-Simulacrum]

cc @jyn514

[jyn514]

Oh, I didn't realize make prepare already existed. I agree that's better than adding a new flag - we may need to update it to ensure everything is downloaded though, I think currently it only downloads the main toolchain and dependencies? i.e. not rustfmt, llvm, etc.

[Mark-Simulacrum]

That seems plausible (it just runs a normal x.py build with a dummy step, IIRC); expanding it to download other things could make sense. I'm not sure how much duplication or awkward plumbing that will require (e.g., I believe we avoid downloading LLVM until we need it, and not all users of make prepare will).

[jyn514]

@Mark-Simulacrum it should be fairly simple to add a function that downloads everything I think (just calling functions it know it will do downloads if necessary, like llvm_link_static(), etc). Then we can call that function from rustbuild when we notice make prepare being run:

rust/src/bootstrap/lib.rs

Line 678 in 00ce472

if path == Path::new("nonexistent/path/to/trigger/cargo/metadata") {

[Mark-Simulacrum]

I'm not a fan of having to maintain a list of "all downloads", particularly since I'm not sure that's a well-defined list. For example, should it include the git repositories cloned by cargotest? Should it include random URLs visited by linkcheck?

Obviously there's a boundary we can draw here, but I think before we invest I'd like to have someone who is actively interested in using this say more about their use case -- if the existing logic is enough for them or the rust-src tarball + custom-sourced rustc/cargo tarballs are enough, then I'm inclined to do nothing further.

[jyn514]

@smijolovic can you try out make prepare and let us know whether that does what you need? And if not, exactly what would you need to predownloaded in addition?

[jyn514]

Another thing we can do is to add an x.py prepare subcommand that does the same thing as make prepare, but without requiring you to run configure.

I don't think we're going to land this PR as-is, though.