Skip to content

Commit

Permalink
Make Native Link installable via nix (TraceMachina#442)
Browse files Browse the repository at this point in the history 
This commit makes Native Link directly buildable/runnable via nix:

```bash
nix run github:TraceMachina/native-link /path/to/config.json
```

To reduce closure size for upcoming container images the environment
that builds the derivation is a fairly customized LLVM-based toolchain.
This toolchain should make it easier to migrate to more sophisticated
approaches in the future.

Test infrastructure is not yet enabled as it'll be a larger effort to
make our testsuite compatible with strictly controlled environments like
the nix build sandbox.

Importantly, the updated README now makes extensive use of emojis ✨
  • Loading branch information
@aaronmondal
aaronmondal authored Dec 5, 2023
1 parent 5123ffc commit b8f3ef1
Show file tree
Hide file tree
Showing 5 changed files with 229 additions and 118 deletions.
13 changes: 13 additions & 0 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,19 @@ various Cloud CLIs for you:
@rules_rust//:rustfmt
```

## Updating Rust dependencies

After modifying the corresponding `Cargo.toml` file in either the top level or
one of the crate subdirectories run the following command to rebuild the crate
index:

```
CARGO_BAZEL_REPIN=1 bazel sync --only=crate_index
```

This updates `Cargo.lock` and `Cargo.Bazel.lock` with the new dependency
information.

## Conduct

Native Link Code of Conduct is available in the
Expand Down
180 changes: 90 additions & 90 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,132 +2,132 @@

[![CI](https://github.com/tracemachina/native-link/workflows/CI/badge.svg)](https://github.com/tracemachina/native-link/actions/workflows/main.yml)

An extremely fast and efficient bazel cache service (CAS) written in rust.
Native link is an extremely (blazingly?) fast and efficient build cache and
remote executor for systems that communicate using the [Remote execution
protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) such as [Bazel](https://bazel.build), [Buck2](https://buck2.build), [Goma](https://chromium.googlesource.com/infra/goma/client/) and
[Reclient](https://github.com/bazelbuild/reclient).

The goals of this project are:
1. Stability - Things should work out of the box as expected
2. Efficiency - Don't waste time on inefficiencies & low resource usage
3. Tested - Components should have plenty of tests & each bug should be regression tested
4. Customers First - Design choices should be optimized for what customers want
Supports Unix based operating systems and Windows.

## Overview
## ❄️ Installing with Nix

Native Link is a project that implements the Bazel's [Remote Execution protocol](https://github.com/bazelbuild/remote-apis) (both CAS/Cache and remote execution portion).
**Installation requirements:**

When properly configured this project will provide extremely fast and efficient build cache for any systems that communicate using the [RBE protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) and/or extremely fast, efficient and low foot-print remote execution capability.
* Nix with [flakes](https://nixos.wiki/wiki/Flakes) enabled

Unix based operating systems and Windows are fully supported.
This build does not require cloning the repository, but you need to provide a
config file, for instance the one at [native-link-config/examples/basic_cas.json](./native-link-config/examples/basic_cas.json).

## TL;DR
The following command builds and runs Native Link in release (optimized) mode:

If you have not updated Rust or Cargo recently, run:

`rustup update`

To compile and run the server:
```sh
# Install dependencies needed to compile Native Link with bazel on
# worker machine (which is this machine).
apt install -y gcc g++ lld python3
nix run github:TraceMachina/native-link ./basic_cas.json
```

# Install cargo (if needed).
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
For use in production pin the executable to a specific revision:

# --release causes link-time-optmization to be enabled, which can take a while
# to compile, but will result in a much faster binary.
cargo run --release --bin cas -- ./native-link-config/examples/basic_cas.json
```
In a separate terminal session, run the following command to connect the running server launched above to Bazel or another RBE client:
```sh
bazel test //... \
--remote_instance_name=main \
--remote_cache=grpc://127.0.0.1:50051 \
--remote_executor=grpc://127.0.0.1:50051 \
--remote_default_exec_properties=cpu_count=1
nix run github:TraceMachina/native-link/<revision> ./basic_cas.json
```
This will cause bazel to run the commands through an all-in-one `CAS`, `scheduler` and `worker`. See [here](https://github.com/tracemachina/native-link/tree/master/native-link-config) for configuration documentation and [here](https://github.com/tracemachina/native-link/tree/main/deployment-examples/terraform) for an example of multi-node cloud deployment example.

## Example Deployments
We currently have a few example deployments in [deployment-examples directory](https://github.com/tracemachina/native-link/tree/master/deployment-examples).
## 🌱 Building with Bazel

### Terraform
The [terraform deployment](https://github.com/tracemachina/native-link/tree/master/deployment-examples/terraform) is the currently preferred method as it leverages a lot of cloud resources to make everything much more robust.
**Build requirements:**

The terraform deployment is very easy to setup and configure. This deployment will show off remote execution capabilities and cache capabilities.
* Bazel 6.4.0+
* A recent C++ toolchain with LLD as linker

## Status
> [!TIP]
> This build supports Nix/direnv which provides Bazel but no C++ toolchain
> (yet).
This project can be considered ~stable~ and is currently used in production systems. Future API changes will be kept to a minimum.
The following commands place an executable in `./bazel-bin/cas/cas` and start
the service:

## Build Requirements
We support building with Bazel or Cargo. Cargo **might** produce faster binaries because LTO (Link Time Optimization) is enabled for release versions, where Bazel currently does not support LTO for rust.
```sh
# Unoptimized development build on Unix
bazel run cas -- ./native-link-config/examples/basic_cas.json

### Bazel requirements
* Bazel 6.4.0+
* gcc
* g++
* lld
* python3
# Optimized release build on Unix
bazel run -c opt cas -- ./native-link-config/examples/basic_cas.json

#### Bazel building for deployment
```sh
# On Unix
bazel build cas
# Unoptimized development build on Windows
bazel run --config=windows cas -- ./native-link-config/examples/basic_cas.json

# On Windows
bazel build --config=windows cas
# Optimized release build on Windows
bazel run --config=windows -c opt cas -- ./native-link-config/examples/basic_cas.json
```
#### Bazel building for release
```sh
# On Unix
bazel build -c opt cas

# On Windows
bazel build --config=windows -c opt cas
```
> **Note**
> Failing to use the `-c opt` flag will result in a very slow binary (~10x slower).
## 🦀 Building with Cargo

These will place an executable in `./bazel-bin/cas/cas` that will start the service.
**Build requirements:**

### Cargo requirements
* Cargo 1.73.0+
#### Cargo building for deployment
```sh
cargo build
* A recent C++ toolchain with LLD as linker

> [!TIP]
> This build supports Nix/direnv which provides Cargo but no C++
> toolchain/stdenv (yet).
```bash
# Unoptimized development build
cargo run --bin cas -- ./native-link-config/examples/basic_cas.json

# Optimized release build
cargo run --release --bin cas -- ./native-link-config/examples/basic_cas.json
```
#### Cargo building for release

## 🧪 Evaluating Native Link

Once you've built Native Link and have an instance running with the
`basic_cas.json` configuration, launch a separate terminal session and run the
following command to connect the running server launched above to Bazel or
another RBE client:

```sh
cargo build --release
bazel test //... \
--remote_instance_name=main \
--remote_cache=grpc://127.0.0.1:50051 \
--remote_executor=grpc://127.0.0.1:50051 \
--remote_default_exec_properties=cpu_count=1
```
> **Note**
> Failing to use the `-c opt` flag will result in a very slow binary (~10x slower).
> This is also significantly slower than building without `--release` because link-time-optimization
> is enabled by default with the flag.

### Configure
This causes bazel to run the commands through an all-in-one `CAS`, `scheduler`
and `worker`.

Configuration is done via a JSON file that is passed in as the first parameter to the `cas` program. See [here](https://github.com/tracemachina/native-link/tree/master/native-link-config) for more details and examples.
## ⚙️ Configuration

## How to update internal or external rust deps
The `cas` executable reads a JSON file as it's only parameter. See [native-link-config](./native-link-config)
for more details and examples.

In order to update external dependencies `Cargo.toml` is not the source of truth, instead7 these are tracked in `tools/cargo_shared.bzl`. It is done this way so both Bazel and Cargo can use the same dependencies that can be derived from the same source location.
## 🚀 Example Deployments

All external dependencies are tracked in a generated `@crate_index` workspace and locked in `Cargo.Bazel.lock`. Some updates to `BUILD` files will require regenerating the `Cargo.toml` files. This is done with the `build_cargo_manifest.py`.
You can find a few example deployments in the [deployment-examples directory](./deployment-examples).

To regenerate the `@crate_index`:
```bash
# This will pin the new dependencies and generate new lock files.
CARGO_BAZEL_REPIN=1 bazel sync --only=crate_index
# This will update the Cargo.toml files with the new dependencies
# weather they are local or external.
python3 ./tools/build_cargo_manifest.py
```
See the [terraform deployments](./deployment-examples/terraform) for an example
deployments that show off remote execution and cache capabilities.

## 🏺 History

## History
This project was first created due to frustration with similar projects not
working or being extremely inefficient. Rust was chosen as the language to write
it in because at the time Rust was going through a revolution in the new-ish
feature `async-await`. This made making multi-threading extremely simple when
paired with a runtime like [tokio](https://github.com/tokio-rs/tokio) while
still giving all the lifetime and other protections that Rust gives. This pretty
much guarantees that we will never have crashes due to race conditions. This
kind of project seemed perfect, since there is so much asynchronous activity
happening and running them on different threads is most preferable. Other
languages like `Go` are good candidates, but other similar projects rely heavily
on channels and mutex locks which are cumbersome and have to be carefully
designed by the developer. Rust doesn't have these issues, since the compiler
will always tell you when the code you are writing might introduce undefined
behavior. The last major reason is because Rust is extremely fast, +/- a few
percent of C++ and has no garbage collection (like C++, but unlike `Java`, `Go`,
or `Typescript`).

This project was first created due to frustration with similar projects not working or being extremely inefficient. Rust was chosen as the language to write it in because at the time rust was going through a revolution in the new-ish feature `async-await`. This made making multi-threading extremely simple when paired with a runtime (like [tokio](https://github.com/tokio-rs/tokio)) while still giving all the lifetime and other protections that Rust gives. This pretty much guarantees that we will never have crashes due to race conditions. This kind of project seemed perfect, since there is so much asynchronous activity happening and running them on different threads is most preferable. Other languages like `Go` are good candidates, but other similar projects rely heavily on channels and mutex locks which are cumbersome and have to be carefully designed by the developer. Rust doesn't have these issues, since the compiler will always tell you when the code you are writing might introduce undefined behavior. The last major reason is because Rust is extremely fast, +/- a few percent of C++ and has no garbage collection (like C++, but unlike `Java`, `Go`, or `Typescript`).
## 📜 License

# License
Copyright 2020-2023 Trace Machina, Inc.

Software is licensed under the Apache 2.0 License. Copyright 2020-2023 Trace Machina, Inc.
Licensed under the Apache 2.0 License, SPDX identifier `Apache-2.0`.
59 changes: 33 additions & 26 deletions flake.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit b8f3ef1

Please sign in to comment.