From d14beb6dbc45c0bbbcbdcd69b16080fc42e3291c Mon Sep 17 00:00:00 2001 From: Austin Seipp Date: Tue, 25 Jun 2024 00:03:06 -0500 Subject: [PATCH] docs: add buck2 docs Signed-off-by: Austin Seipp --- docs/buck2.md | 384 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 384 insertions(+) create mode 100644 docs/buck2.md diff --git a/docs/buck2.md b/docs/buck2.md new file mode 100644 index 0000000000..63fc3df8e5 --- /dev/null +++ b/docs/buck2.md @@ -0,0 +1,384 @@ +# Buck2 builds + +> [!NOTE] +> This document is primarily of interest to developers. See also [Contributing] +> for more information on how to contribute in general. + +There is experimental support for building `jj` with [Buck2], a hermetic and +reproducible build system, an alternative to Cargo. + +> [!IMPORTANT] +> Buck2 support is a work in progress, and is not yet complete. It is not +> recommended for primary development use. + +## Why Buck2 + +Jujutusu currently uses Cargo as its build system, much like the vast majority +of Rust projects. Currently, Cargo is in no way a major strain on the project, +but as we want to tackle more complex projects in the future some limitations +are worth thinking about. + +### Multi-language support + +Most projects eventually evolve to include more than a single programming +language and due to many constraints this is often fairly natural, but +conceptually difficult to model. + +We have two cases where this is particularly relevant for different reasons, where +we want to integrate other languages into our Rust-based project: + +#### Case 1: C dependencies + +For instance, today, we rely on several large C libraries which are effectively +abstracted out by Rust libraries using complex `build.rs` scripts. This largely +works out due to the (very high) quality of the Rust ecosystem and maintiners of +these particular packages, but it does mean we are effectively at the mercy of +an unbounded amount of C code that can be included in the blink of an eye. + +Furthermore, tracking and auditing C code carefully is a requirement for more +advanced security and integration approaches like fuzzing. Being able to +represent your C dependencies and manage them is crucial. + +#### Case 2: JavaScript UX integration + +But tomorrow, we would like to include HTTP components or Tauri projects in this +repository, which would necessitate a way of incorporating JavaScript and more +complex and refined build processes. This means we need to stitch together the +build process of `npm` with `cargo` in a way that is easy for every developer to +test. However, these tools have no way to relate their underlying dependency +graphs in any way, and so the build process becomes overly crude approximations +of `make` that are difficult to maintain. + +As a concrete example, the [diffedit3] program is a mixed Tauri/localhost-webapp +that functions as a 3-way diff editor. We would like to include it in the +Jujutusu repository, but it is difficult to do so in a way that reliably ensures +we can build the components together. Our options are both suboptimal: + +- Commit prebuilt `diffedit3.js` output from a JS compiler + - Conceptually simple. + - But bloats the repository size, slowing development. + - Easy to get out of sync. You have to do it manually and add CI to check it. + - Impossible to audit; could be malicious! It's much better to represent the + build process as a *pure* pipeline. + - If you have to do this, **make it fully automated**. +- Write wrapper tools to build the JavaScript as part of the `Cargo` process + - Easy to forget or misuse. + - Difficult to maintain an integration with `Cargo` anyway! + - Often doesn't generalize well to other languages or tools + - No matter what, a source of user complaints and occasionally painful + debugging sessions (out of date dependencies, missed inputs, etc.) + +[diffedit3]: https://github.com/ilyagr/diffedit3 + +### Hermetic builds + +> [!INFO] +> Buck2 builds of Jujutsu are not yet hermetically sound + +Buck2 is *hermetic*, meaning that the build graph (ideally!) encodes the +relationships between all input files in the entire system. Hermetic builds are +a powerful concept that can be used to ensure that the build process is fully +reproducible and deterministic up to a very large size. + +Hermetic builds also allow for the process to be cached and served back to users +as well, meaning that the time taken for a build can often be reduced to nearly +zero even from a cold local cache. + +### Programmable design + +Buck2 is configured with Starlark, a deterministic and reproducible language +that is a subset of Python (with types!). This means that the build process can +be treated as a programming problem and designed in an open-ended way instead of +designed around one concept about how a build should work. + +### Early movers advantage + +Most of the previous advantages are seen as only being most useful to those with +lots of developers. The process is described as long and arduous and painful to +port everything to a hermetic build. + +However, by waiting too late to make the transition to hermetic tools that +represent the build as a pure graph, the pain of fixing this is made infinitely +more acute by the process of time and accumulated debt. + +While most of our current daily problems are easily tractable and we are +iterating easily with the tools we have, by exploring Buck2 early, we can avoid +and eliminate obstacles up-front that would make the process far more difficult +later. Even if we eventually choose some other build tool or none at all, +representing the idea as a pure build graph can help guide overall design +decisions. + +## Current support & feature parity + +Some notes about build compatibility are included below. + +### Overall status + +Legend: + +- ✅: Supported +- ⚠️: Partial support/WIP +- ❌: Not supported +- ❓: Investigation needed + +| Feature | Status | +|------------------------|--------| +| `rust-analyzer` | ⚠️ | +| CI setup (GHA) | ❌ | +| RBE/GHA Action Cache | ❌ | +| Hermetic toolchain | ⚠️ | + +| Unique features | Status | +|------------------------|--------| +| Auto `gen-protos` | ✅ | +| [OpenSSL-on-Win32](https://github.com/martinvonz/jj/pull/3554) | ❓ | + +### Known buck2 bugs + +NIH. + +### vs Cargo + +Legend: + +- ✅: Supported +- ⚠️: Partial support/WIP +- ❌: Not supported + +| Feature | Cargo | Buck2 | +|------------------------|-------|-------| +| `rust-analyzer` | ✅ | ⚠️ | +| Fully working build | ✅ | ✅ | +| Debug/Release configs | ✅ | ✅ | +| Full test suite | ✅ |️ ❌ | +| Release-able binaries | ✅ |️ ❌ | + +### Platform support + +Legend: + +- ✅: Supported +- ⚠️: Partial support/WIP +- ❌: Not supported + +| OS | Architecture | Status | +|---------|--------------|--------| +| Linux | x86_64 | ✅ | +| Linux | aarch64 | ✅ | +| macOS | x86_64 | ❌ | +| macOS | aarch64 | ⚠️ | +| Windows | x86_64 | ✅ | +| Windows | aarch64 | ❌ | + +## Step 1: Installing Dotslash + +We use [Dotslash] to manage Buck2 build tooling in a way that's consistent +across all developers and amenable to version control. This helps ensure every +developer gets consistent results (which is a big part of the selling point of +hermetic build tools.) + +You can install Dotslash binaries by following the instructions at: + +- + +Or, if you have Rust installed, you can install Dotslash by running: + +```sh +cargo install dotslash +``` + +Or, if you have Nix, you can install Dotslash with Nix by running: + +```sh +nix profile install 'nixpkgs#dotslash' +``` + +## Step 2: Building `jj` with Buck2 + +Assuming `dotslash` exists somewhere in your `$PATH`, you can now build `jj` +with the included `buck2` dotslash file that exists under `./tools/bin`: + +```sh +./tools/bin/buck2 run cli -- version +``` + +Dotslash will transparently run the correct version of `buck2`, and `buck2` will +build everything needed to run `jj`. + +--- + +## Buck2 crash course + +The following is an extremely minimal crash course in Buck2 concepts and how to +use it. + +### Targets + +Buck2 is used to build **targets**, that exists in **packages**, which are part +of a **cell**. The most explicit syntax for referring to a target is the +following: + +```text +cell//path/to/package:target-name +``` + +You normally use a so-called "fully qualified" name like the above as an +argument to `buck2 build`. + +A cell is a short name that maps to a directory in the code repository. A +package is a subdirectory underneath the cell that contains the build rules for +the targets. A target is a buildable unit of code, like a binary or a library, +named in the `BUILD` file inside that package. + +A fully-qualified reference to a target works anywhere in the source code tree, +so you can build or test any component no matter what directory you're in. + +So, given a cell named `foobar//` located underneath `code/foobar`, and a +package `bar/baz` in that cell, leads to a file + +```text +code/foobar/bar/baz/BUILD +``` + +Which contains the targets that can be built. + +There are several shorthands for a target: + +- NIH. + +### Graphs: Target & Action + +NIH. + +### Package files + +NIH. + +### Mode files + +In order to support concepts like debug and release builds, we use the concept +of "mode files" in Buck2. These are files that contain a list of command line +options to apply to a build to achieve the desired effect. + +For example, to build in debug mode, you can simply include the contents of the +file `mode//debug` (using cell syntax) onto the command line. This can +conveniently be done with "at-file" syntax when invoking `buck2`: + +```sh +buck2 build cli @mode//debug +buck2 build cli @mode//release +``` + +Where `@path/to/file` is the at-file syntax for including the contents of a file +on the command line. This syntax supports `cell//` references to Buck cells, as +well. + +In short, `buck2 build @mode//file` will apply the contents of `file` to your +invocation. We keep a convenient set of these files maintained under the +`mode//` cell, located under [`./buck/mode`](../buck/mode). + +#### At-file syntax + +The `buck2` CLI supports a convenient modern feature called "at-file" syntax, +where the invocation `buck2 @path/to/file` is effectively equivalent to the +bash-ism `buck2 $(cat path/to/file)`, where each line of the file is a single +command line entry, in a consistent and portable way that doesn't have any limit +to the size of the underlying file. + +For example, assuming the file `foo/bar` contained the contents + +```text +--foo=1 +--bar=false +``` + +Then `buck2 --test @foo/bar` and `buck2 --test --foo=1 --bar=false` are +equivalent. + +### Buck Extension Language (BXL) + +NIH. + +## Examples + +Some examples are included below. + +### Run the Jujutsu CLI + +The following shorthand is equivalent to the full target `root//cli:cli`: + +```sh +buck2 run //cli +``` + +This works anywhere in the source tree. It can be shortened to `buck2 run cli` +if you're in the root of the repository. + +### Run BoringSSL `bssl speed` tests + +```sh +buck2 run third-party//bssl @mode//release -- speed +``` + +### Build all third-party Rust dependencies + +```sh +buck2 build third-party//rust +``` + +### Build all `http_archive` dependencies + +Useful for downloading all dependencies, then testing clean build times afterwards. + +```sh +buck2 build $(buck2 uquery "kind('http_archive', deps('//...'))" | grep third-party//) +``` + +## Development notes + +Notes for `jj` developers using Buck2. + +### Build mode reference + +You can pass these to any `build` or `run` invocation. + +- `@mode//debug` +- `@mode//release` + +### Cargo dependency management + +Although Buck2 downloads and runs `rustc` on its own to build crate +dependencies, our `Cargo.toml` build files act as the source of truth for +dependency information in both Cargo and Buck2. + +Updating the dependency graph for Cargo-based projects typically comes in one of +two forms: + +- Updating a dependency version in the top-level workspace `Cargo.toml` file +- Adding a newly required dependency to `[dependencies]` in the `Cargo.toml` + file for a crate + +After doing either of these actions, you can synchronize the Buck2 dependencies +with the Cargo dependencies with the following command: + +```bash +buck2 -v0 run third-party//rust:sync.py +``` + +This must be run from the root of the repository. Eyeball the output and make +sure it looks fine before committing the changes. + +This step will re-synchronize all `third-party//rust` crates with the versions +in the workspace Cargo file, and then also update the `BUILD` files in the +source code with any newly added build dependencies that were added or removed +(not just updated). + +### `rust-analyzer` support + +Coming soon. + + + +[Contributing]: https://martinvonz.github.io/jj/latest/contributing/ +[Buck2]: https://buck2.build/ +[Dotslash]: https://dotslash-cli.com/