Policy on large prebuilt libraries on crates.io?

[madsmtm]

Greetings! @silvanshade and I are discussing prebuilding Clang/LLVM libraries in madsmtm/objc2#345, and I have a few questions about that for the crates.io team.

Motivation

There exists several tools in the ecosystem that uses libclang, usually to do some sort of C/C++ to Rust transformation. Prominent examples include bindgen, autocxx and my own tool for generating icrate.

This is nice because libclang can be installed fairly easily on most operating systems, and the API is quite stable, even between major versions. It is, sadly, also limited in the kind of semantic analysis you can do, since there are many C/C++ quirks and attributes that are not exposed in its API.

So instead of using libclang, another option would be to use Clang/LLVM libraries directly. This unfortunately forces downstream users of your crate to somehow get/build a compatible version of Clang/LLVM libraries outside of Cargo, which is difficult and annoying for users, and has issues with reproducibility. This is the approach taken by c2rust and rustc itself (although the bootstrap script mitigates it by providing prebuilt libraries (example url)).

Proposed solution

To make it easier to use Clang/LLVM libraries directly from Rust, I propose developing a crate that helps ensure that the desired Clang/LLVM libraries is available (potentially integrated with the existing llvm-sys crate). This crate will bundle the required sources in the crate itself (a quick test says this will be around ~30MB).

These sadly take a long time to compile, so to mitigate that issue, the crate will supply prebuilt libraries for common target platforms (of course, user configurable, I'm aware of the serde_derive suddenly shipping a binary debacle a while ago).

These will also be uploaded to crates.io, and the correct libraries will be chosen by Cargo in a similar manner as is done for windows-sys. The size of the LLVM libraries are ~150MB, Clang libraries ~85MB and maybe also Clang binaries ~30MB (all after being compressed by Cargo). We would need one crate per supported target, so probably a few GB in total. @silvanshade may be able to answer if these can be mader smaller.

The current upload size limit is 10MiB, which the proposed crates would clearly hit.

Questions

What is your policy on prebuilt libraries on crates.io?

What is your policy on increasing this size limit for a crate?

What about for something like this, which is important for bindgen (an official Rust project), and has the potential to vastly improving Rust's user story on interacting with C/C++/Objective-C?

I can see that storing several GBs of data in total is troublesome, but is there another number you would be more comfortable with? (If it helps, we would be limiting ourselves (barring critical bugs) to releasing only major versions of Clang/LLVM, to reduce the strain on crates.io's servers).

If this is not acceptable, which alternative would the crates.io team recommend that we do instead? (Either from the list below, or other alternatives).

Alternatives

1. We could host these ourselves on GitHub (already done in here).
   • Requires downloading these in a build script, which is bad for compile times, offline users and sandboxing.
2. Every tool that needs Clang/LLVM libraries could provide their own prebuilt dylib (which could reduce the required binary size because of dead code elimination).
   • Possible, depends on what your
3. Only allow building from source.
   • Painfully slow, though workable.
4. Include Clang/LLVM libraries as a rustup component.
   • Difficult to ensure that the correct version is installed. Updating Rust now might break crates.
5. Include bindgen as a rustup component.
   • Requires a stable bindgen v1.0.

Prior art

Previously discussed in rust-lang/rust-bindgen#918.

The openssl crate has an option to build OpenSLL from sources, which are distributed via. openssl-src. The download size for that is ~8MB.

[Turbo87]

What is your policy on prebuilt libraries on crates.io?

we don't support prebuilt libraries on crates.io. any change to this policy will have to go through the RFC process.

What is your policy on increasing this size limit for a crate?

we usually decide this on a case-by-case basis in the team meeting and grant exceptions for cases where there are no viable alternatives.

What about for something like this

since this is primarily about pre-built libraries and less about the size increase this will likely need a wider discussion in an RFC.

If this is not acceptable, which alternative would the crates.io team recommend that we do instead?

I personally don't know enough about clang to decide what the best alternative would be.

[the8472]

rustc already contains a libllvm, is that not enough?

Anyway this sounds like either

• a fully general argument for distributing binary artifacts via crates.io, which is a huge can of worms and needs an RFC
• pleading a special case for libllvm (that it is important for rust compilation), in which case it might be reasonable as rustup component or including whatever additional features are needed in the already shipped one

[silvanshade]

Thanks for starting a discussion about this @madsmtm.

The size of the LLVM libraries are ~150MB, Clang libraries ~85MB and maybe also Clang binaries ~30MB (all after being compressed by Cargo). We would need one crate per supported target, so probably a few GB in total. @silvanshade may be able to answer if these can be made smaller.

I don't really know of a way to reduce the size further beyond what I've already done in llvmup/toolchains, which really only amounts to selecting the minimum viable set of components and compiling for MinSizeRel. It might be possible to reduce the size further by reorganizing the actual library archives somehow but that would start to become more difficult to maintain, if it even made a difference.

Alternatives

1. We could host these ourselves on GitHub (already done in here).

   • Requires downloading these in a build script, which is bad for compile times, offline users and sandboxing.

I actually think the install process might not be so bad in this sense.

The way I had intended this to work is the component tarballs are streamed through a download, checksum, unpack pipeline asynchronously and simultaneously. In my limited testing it shouldn't be take more than a couple minutes for a full toolchain install (llvm, clang, mlir, etc.) on a reasonably fast connection and workstation.

Of course, that's still not "fast", but I think the more difficult issue is the fact that downloading artifacts in build.rs potentially breaks some functionality (e.g., --offline) and expectations, given how cargo currently works.

Aside from security concerns, there's also no way to provide feedback to the user that the download is happening, since even cargo:warning=MESSAGE doesn't output until after the script finishes.

2. Every tool that needs Clang/LLVM libraries could provide their own prebuilt dylib (which could reduce the required binary size because of dead code elimination).

This could be interesting. I suspect the current size limit would still be hit in most cases but it could certainly reduce the maximum size by quite a lot, at the cost of additional complexity in producing the builds.

3. Only allow building from source.

   • Painfully slow, though workable.

One thing to note is that, although initial compile times would be quite long, subsequent builds can be very fast by using ccache. In my experience, subsequent builds can take only seconds or a couple minutes, depending on whether the CMake build directory still exists, and depending on whether or not LTO is used. But always it's still significantly faster than a fresh, uncached compile.

That might make building from source more acceptable, but still leaves the size limit issue, since the LLVM source tree is 122MB.

4. Include Clang/LLVM libraries as a rustup component.

   • Difficult to ensure that the correct version is installed. Updating Rust now might break crates.

If this could be made to work somehow it would almost seem like the ideal solution in the long run.

The potential breakage from updating issue is probably unavoidable (without a huge maintenance overhead) since the LLVM C++ APIs don't have a stable interface. That would probably mean that a range of rustup components must be available to be installed with the rest of the Rust toolchain components. But I'm not sure if this even feasible with the current rustup design.

[silvanshade]

rustc already contains a libllvm, is that not enough?

Generally, no.

The libLLVM artifact is configurable so I'm not even sure what all is included in the rustc case but what originally prompted this effort was the need to have access to the Clang C++ libraries.

Sometimes these are described as libTooling, although in our case we actually potentially need access to the full set libraries, which requires a proper LLVM component distribution, as described in the toolchains repo linked above.

There is reason to advocate for more general access to these libraries in any case, since the ability to interface with MLIR, ClangIR, and other LLVM ecosystem components would be useful.