Skip to content

Commit

Permalink
docs: add buck2 docs
Browse files Browse the repository at this point in the history 
Signed-off-by: Austin Seipp <[email protected]>
  • Loading branch information
@thoughtpolice
thoughtpolice committed Jun 26, 2024
1 parent dc27ffe commit cdc7216
Showing 1 changed file with 384 additions and 0 deletions.
384 changes: 384 additions & 0 deletions docs/buck2.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,384 @@
# Buck2 builds

> [!NOTE]
> This document is primarily of interest to developers. See also [Contributing]
> for more information on how to contribute in general.
There is experimental support for building `jj` with [Buck2], a hermetic and
reproducible build system, an alternative to Cargo.

> [!IMPORTANT]
> Buck2 support is a work in progress, and is not yet complete. It is not
> recommended for primary development use.
## Why Buck2

Jujutusu currently uses Cargo as its build system, much like the vast majority
of Rust projects. Currently, Cargo is in no way a major strain on the project,
but as we want to tackle more complex projects in the future some limitations
are worth thinking about.

### Multi-language support

Most projects eventually evolve to include more than a single programming
language and due to many constraints this is often fairly natural, but
conceptually difficult to model.

We have two cases where this is particularly relevant for different reasons, where
we want to integrate other languages into our Rust-based project:

#### Case 1: C dependencies

For instance, today, we rely on several large C libraries which are effectively
abstracted out by Rust libraries using complex `build.rs` scripts. This largely
works out due to the (very high) quality of the Rust ecosystem and maintiners of
these particular packages, but it does mean we are effectively at the mercy of
an unbounded amount of C code that can be included in the blink of an eye.

Furthermore, tracking and auditing C code carefully is a requirement for more
advanced security and integration approaches like fuzzing. Being able to
represent your C dependencies and manage them is crucial.

#### Case 2: JavaScript UX integration

But tomorrow, we would like to include HTTP components or Tauri projects in this
repository, which would necessitate a way of incorporating JavaScript and more
complex and refined build processes. This means we need to stitch together the
build process of `npm` with `cargo` in a way that is easy for every developer to
test. However, these tools have no way to relate their underlying dependency
graphs in any way, and so the build process becomes overly crude approximations
of `make` that are difficult to maintain.

As a concrete example, the [diffedit3] program is a mixed Tauri/localhost-webapp
that functions as a 3-way diff editor. We would like to include it in the
Jujutusu repository, but it is difficult to do so in a way that reliably ensures
we can build the components together. Our options are both suboptimal:

- Commit prebuilt `diffedit3.js` output from a JS compiler
- Conceptually simple.
- But bloats the repository size, slowing development.
- Easy to get out of sync. You have to do it manually and add CI to check it.
- Impossible to audit; could be malicious! It's much better to represent the
build process as a *pure* pipeline.
- If you have to do this, **make it fully automated**.
- Write wrapper tools to build the JavaScript as part of the `Cargo` process
- Easy to forget or misuse.
- Difficult to maintain an integration with `Cargo` anyway!
- Often doesn't generalize well to other languages or tools
- No matter what, a source of user complaints and occasionally painful
debugging sessions (out of date dependencies, missed inputs, etc.)

[diffedit3]: https://github.com/ilyagr/diffedit3

### Hermetic builds

> [!INFO]
> Buck2 builds of Jujutsu are not yet hermetically sound
Buck2 is *hermetic*, meaning that the build graph (ideally!) encodes the
relationships between all input files in the entire system. Hermetic builds are
a powerful concept that can be used to ensure that the build process is fully
reproducible and deterministic up to a very large size.

Hermetic builds also allow for the process to be cached and served back to users
as well, meaning that the time taken for a build can often be reduced to nearly
zero even from a cold local cache.

### Programmable design

Buck2 is configured with Starlark, a deterministic and reproducible language
that is a subset of Python (with types!). This means that the build process can
be treated as a programming problem and designed in an open-ended way instead of
designed around one concept about how a build should work.

### Early movers advantage

Most of the previous advantages are seen as only being most useful to those with
lots of developers. The process is described as long and arduous and painful to
port everything to a hermetic build.

However, by waiting too late to make the transition to hermetic tools that
represent the build as a pure graph, the pain of fixing this is made infinitely
more acute by the process of time and accumulated debt.

While most of our current daily problems are easily tractable and we are
iterating easily with the tools we have, by exploring Buck2 early, we can avoid
and eliminate obstacles up-front that would make the process far more difficult
later. Even if we eventually choose some other build tool or none at all,
representing the idea as a pure build graph can help guide overall design
decisions.

## Current support & feature parity

Some notes about build compatibility are included below.

### Overall status

Legend:

- ✅: Supported
- ⚠️: Partial support/WIP
- ❌: Not supported
- ❓: Investigation needed

| Feature | Status |
|------------------------|--------|
| `rust-analyzer` | ⚠️ |
| CI setup (GHA) | ❌ |
| RBE/GHA Action Cache | ❌ |
| Hermetic toolchain | ⚠️ |

| Unique features | Status |
|------------------------|--------|
| Auto `gen-protos` | ✅ |
| [OpenSSL-on-Win32](https://github.com/martinvonz/jj/pull/3554) | ❓ |

### Known buck2 bugs

NIH.

### vs Cargo

Legend:

- ✅: Supported
- ⚠️: Partial support/WIP
- ❌: Not supported

| Feature | Cargo | Buck2 |
|------------------------|-------|-------|
| `rust-analyzer` | ✅ | ⚠️ |
| Fully working build | ✅ | ✅ |
| Debug/Release configs | ✅ | ✅ |
| Full test suite | ✅ |️ ❌ |
| Release-able binaries | ✅ |️ ❌ |

### Platform support

Legend:

- ✅: Supported
- ⚠️: Partial support/WIP
- ❌: Not supported

| OS | Architecture | Status |
|---------|--------------|--------|
| Linux | x86_64 | ✅ |
| Linux | aarch64 | ✅ |
| macOS | x86_64 | ❌ |
| macOS | aarch64 | ⚠️ |
| Windows | x86_64 | ✅ |
| Windows | aarch64 | ❌ |

## Step 1: Installing Dotslash

We use [Dotslash] to manage Buck2 build tooling in a way that's consistent
across all developers and amenable to version control. This helps ensure every
developer gets consistent results (which is a big part of the selling point of
hermetic build tools.)

You can install Dotslash binaries by following the instructions at:

- <https://dotslash-cli.com/docs/installation/>

Or, if you have Rust installed, you can install Dotslash by running:

```sh
cargo install dotslash
```

Or, if you have Nix, you can install Dotslash with Nix by running:

```sh
nix profile install 'nixpkgs#dotslash'
```

## Step 2: Building `jj` with Buck2

Assuming `dotslash` exists somewhere in your `$PATH`, you can now build `jj`
with the included `buck2` dotslash file that exists under `./tools/bin`:

```sh
./tools/bin/buck2 run cli -- version
```

Dotslash will transparently run the correct version of `buck2`, and `buck2` will
build everything needed to run `jj`.

---

## Buck2 crash course

The following is an extremely minimal crash course in Buck2 concepts and how to
use it.

### Targets

Buck2 is used to build **targets**, that exists in **packages**, which are part
of a **cell**. The most explicit syntax for referring to a target is the
following:

```text
cell//path/to/package:target-name
```

You normally use a so-called "fully qualified" name like the above as an
argument to `buck2 build`.

A cell is a short name that maps to a directory in the code repository. A
package is a subdirectory underneath the cell that contains the build rules for
the targets. A target is a buildable unit of code, like a binary or a library,
named in the `BUILD` file inside that package.

A fully-qualified reference to a target works anywhere in the source code tree,
so you can build or test any component no matter what directory you're in.

So, given a cell named `foobar//` located underneath `code/foobar`, and a
package `bar/baz` in that cell, leads to a file

```text
code/foobar/bar/baz/BUILD
```

Which contains the targets that can be built.

There are several shorthands for a target:

- NIH.

### Graphs: Target & Action

NIH.

### Package files

NIH.

### Mode files

In order to support concepts like debug and release builds, we use the concept
of "mode files" in Buck2. These are files that contain a list of command line
options to apply to a build to achieve the desired effect.

For example, to build in debug mode, you can simply include the contents of the
file `mode//debug` (using cell syntax) onto the command line. This can
conveniently be done with "at-file" syntax when invoking `buck2`:

```sh
buck2 build cli @mode//debug
buck2 build cli @mode//release
```

Where `@path/to/file` is the at-file syntax for including the contents of a file
on the command line. This syntax supports `cell//` references to Buck cells, as
well.

In short, `buck2 build @mode//file` will apply the contents of `file` to your
invocation. We keep a convenient set of these files maintained under the
`mode//` cell, located under [`./buck/mode`](../buck/mode).

#### At-file syntax

The `buck2` CLI supports a convenient modern feature called "at-file" syntax,
where the invocation `buck2 @path/to/file` is effectively equivalent to the
bash-ism `buck2 $(cat path/to/file)`, where each line of the file is a single
command line entry, in a consistent and portable way that doesn't have any limit
to the size of the underlying file.

For example, assuming the file `foo/bar` contained the contents

```text
--foo=1
--bar=false
```

Then `buck2 --test @foo/bar` and `buck2 --test --foo=1 --bar=false` are
equivalent.

### Buck Extension Language (BXL)

NIH.

## Examples

Some examples are included below.

### Run the Jujutsu CLI

The following shorthand is equivalent to the full target `root//cli:cli`:

```sh
buck2 run //cli
```

This works anywhere in the source tree. It can be shortened to `buck2 run cli`
if you're in the root of the repository.

### Run BoringSSL `bssl speed` tests

```sh
buck2 run third-party//bssl @mode//release -- speed
```

### Build all third-party Rust dependencies

```sh
buck2 build third-party//rust
```

### Build all `http_archive` dependencies

Useful for downloading all dependencies, then testing clean build times afterwards.

```sh
buck2 build $(buck2 uquery "kind('http_archive', deps('//...'))" | grep third-party//)
```

## Development notes

Notes for `jj` developers using Buck2.

### Build mode reference

You can pass these to any `build` or `run` invocation.

- `@mode//debug`
- `@mode//release`

### Cargo dependency management

Although Buck2 downloads and runs `rustc` on its own to build crate
dependencies, our `Cargo.toml` build files act as the source of truth for
dependency information in both Cargo and Buck2.

Updating the dependency graph for Cargo-based projects typically comes in one of
two forms:

- Updating a dependency version in the top-level workspace `Cargo.toml` file
- Adding a newly required dependency to `[dependencies]` in the `Cargo.toml`
file for a crate

After doing either of these actions, you can synchronize the Buck2 dependencies
with the Cargo dependencies with the following command:

```bash
buck2 -v0 run third-party//rust:sync.py
```

This must be run from the root of the repository. Eyeball the output and make
sure it looks fine before committing the changes.

This step will re-synchronize all `third-party//rust` crates with the versions
in the workspace Cargo file, and then also update the `BUILD` files in the
source code with any newly added build dependencies that were added or removed
(not just updated).

### `rust-analyzer` support

Coming soon.

<!-- References -->

[Contributing]: https://martinvonz.github.io/jj/latest/contributing/
[Buck2]: https://buck2.build/
[Dotslash]: https://dotslash-cli.com/

0 comments on commit cdc7216

Please sign in to comment.