Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: use cross to build binaries #507

Open
wants to merge 45 commits into
base: v1.0
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
16a4a7e
linux are completed
lifeizhou-ap Dec 20, 2024
742a1e1
Merge branch 'v1.0' into lifei/aarch64-linux
lifeizhou-ap Dec 20, 2024
0d3528e
use cross to build
lifeizhou-ap Dec 20, 2024
b2dbcbe
added toolchain properites
lifeizhou-ap Dec 20, 2024
af4bec3
added debug message to find pkg-config path
lifeizhou-ap Dec 20, 2024
86cbb51
set pkg env variable
lifeizhou-ap Dec 20, 2024
31f8405
temperarily change the trigger of the job. disabled release
lifeizhou-ap Dec 20, 2024
1c96c6b
added backtrace
lifeizhou-ap Dec 20, 2024
d85aaff
set backtrace to full
lifeizhou-ap Dec 20, 2024
54ab26f
set sysroot env
lifeizhou-ap Dec 20, 2024
e0b4c27
set more env var
lifeizhou-ap Dec 20, 2024
63f5895
more env and with specific toolkit
lifeizhou-ap Dec 20, 2024
cf33c51
revert toolchain
lifeizhou-ap Dec 20, 2024
6914123
run on macos
lifeizhou-ap Dec 20, 2024
13bdc49
revert change
lifeizhou-ap Dec 20, 2024
1f33d37
remove env var
lifeizhou-ap Dec 20, 2024
809d65b
not install pkg-config
lifeizhou-ap Dec 20, 2024
ab1c3f7
add release step
lifeizhou-ap Dec 20, 2024
a42963b
fixed wrong variable
lifeizhou-ap Dec 20, 2024
80317f8
remove unnecessary env
lifeizhou-ap Dec 20, 2024
7abd3d3
turn on x86_64 linux
lifeizhou-ap Dec 20, 2024
a71ffe7
try x86_64 linux
lifeizhou-ap Dec 20, 2024
4902cea
fixed workflow trigger
lifeizhou-ap Dec 20, 2024
d8c804b
turn on release x86_64 linux
lifeizhou-ap Dec 20, 2024
cddd0ca
start macos x86_64
lifeizhou-ap Dec 20, 2024
366aec2
fixed workflow script
lifeizhou-ap Dec 20, 2024
ad78352
run aarch64 macos
lifeizhou-ap Dec 20, 2024
b5bbd30
added target of toolchain
lifeizhou-ap Dec 20, 2024
cea042f
fixed variable
lifeizhou-ap Dec 20, 2024
fde066c
reverted change
lifeizhou-ap Dec 20, 2024
457d55e
turn on full backtrace
lifeizhou-ap Dec 20, 2024
319fab6
added CROSS_NO_WARNINGS=0
lifeizhou-ap Dec 20, 2024
075ec23
try x86_64 macos
lifeizhou-ap Dec 20, 2024
77f414d
turn on release for macos
lifeizhou-ap Dec 20, 2024
aaefbe4
include all arch since they are all working now
lifeizhou-ap Dec 20, 2024
05ccb59
clean up and add the notes for testing release on local machine
lifeizhou-ap Dec 20, 2024
b676a03
Merge branch 'v1.0' into lifei/aarch64-linux
lifeizhou-ap Dec 20, 2024
13d28c7
reverted tag pattern change
lifeizhou-ap Dec 20, 2024
b29ec37
changed toolchain in build
lifeizhou-ap Dec 20, 2024
66a3a72
final testing
lifeizhou-ap Dec 20, 2024
04f5212
revert the toolchain change
lifeizhou-ap Dec 20, 2024
efb6e1a
update the doc
lifeizhou-ap Dec 20, 2024
c258cc3
reverted the trigger tag pattern
lifeizhou-ap Dec 20, 2024
d4d0f82
update the doc
lifeizhou-ap Dec 20, 2024
24cc7cb
used correct build name
lifeizhou-ap Dec 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 16 additions & 46 deletions .github/workflows/cli-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,73 +17,43 @@ jobs:
strategy:
fail-fast: false
matrix:
include:
# Linux x86_64 build on Ubuntu
- os: ubuntu-latest
architecture: x86_64
suffix: unknown-linux-gnu

# TODO: Linux aarch64 (ARM64) build on Ubuntu via cross-compilation - probably use `cross-rs`

# macOS x86_64 build
- os: macos-latest
architecture: x86_64
suffix: apple-darwin

# macOS ARM64 build
- os: macos-latest
architecture: aarch64
suffix: apple-darwin
os: [ ubuntu-latest, macos-latest ]
architecture: [ aarch64, x86_64 ]
include:
- os: ubuntu-latest
target-suffix: unknown-linux-gnu
- os: macos-latest
target-suffix: apple-darwin

steps:
# Step 1: Checkout the code
- name: Checkout code
uses: actions/checkout@v4

# Step 2: Set up Rust toolchain
# This will install the specified target as per matrix
- name: Set up Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The toolchain actions-rust-lang/setup-rust-toolchain@v1 does not with cross. Not sure why.

We can use hermit in the future to set up the rust so that we have consistent environment locally and the build environment

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that is fine

uses: actions-rs/toolchain@v1
with:
target: ${{ matrix.architecture }}-${{ matrix.suffix }}

# Step 3: Install dependencies for macOS ARM64 builds
- name: Install dependencies for macOS ARM64
if: matrix.os == 'macos-latest' && matrix.architecture == 'aarch64'
run: |
brew install llvm
rustup target add aarch64-apple-darwin
echo "export PATH=$(brew --prefix llvm)/bin:\$PATH" >> $GITHUB_ENV

# Step 4: Install Libs for Ubuntu (x86_64 and aarch64)
- name: Install Libs for Ubuntu
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt update -y
sudo apt install -y libdbus-1-dev gnome-keyring libxcb1-dev
gnome-keyring-daemon --components=secrets --daemonize --unlock <<< 'foobar'

toolchain: stable

# Step 5: Build the project
- name: Build
- name: build
run: |
export TARGET=${{ matrix.architecture }}-${{ matrix.suffix }}
cargo build --release --target $TARGET
export TARGET=${{ matrix.architecture }}-${{ matrix.target-suffix }}
rustup target add "${TARGET}"
cargo install cross --git https://github.com/cross-rs/cross
CROSS_NO_WARNINGS=0 cross build --release --target ${TARGET}
cd target/${TARGET}/release
tar -cjf goose-${TARGET}.tar.bz2 goose goosed
echo "ARTIFACT=target/${TARGET}/release/goose-${TARGET}.tar.bz2" >> $GITHUB_ENV

# Step 6: Upload artifacts
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: goose-${{ matrix.architecture }}-${{ matrix.suffix }}
name: goose-${{ matrix.architecture }}-${{ matrix.target-suffix }}
path: ${{ env.ARTIFACT }}

release:
name: Release
runs-on: ubuntu-latest
needs: [build]
needs: [ build ]
permissions:
contents: write
steps:
Expand Down
19 changes: 19 additions & 0 deletions Cross.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[target.aarch64-unknown-linux-gnu]
xargo = false
pre-build = [
"dpkg --add-architecture $CROSS_DEB_ARCH && apt-get update --fix-missing && apt-get install --assume-yes libxcb1-dev:$CROSS_DEB_ARCH libdbus-1-dev:$CROSS_DEB_ARCH",
]

# If you run the build on your local machine,
# This is a workaround for the missing pkg-config path on aarch64
# You also need to add pkg-config:$CROSS_DEB_ARCH to the apt-get install command above
#[target.aarch64-unknown-linux-gnu.env]
#passthrough = ["PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig"]

# If you run the build on your local machine,
# You need to add pkg-config:$CROSS_DEB_ARCH to the apt-get install command below
[target.x86_64-unknown-linux-gnu]
xargo = false
pre-build = [
"dpkg --add-architecture $CROSS_DEB_ARCH && apt-get update --fix-missing && apt-get install --assume-yes libxcb1-dev:$CROSS_DEB_ARCH libdbus-1-dev:$CROSS_DEB_ARCH",
]
4 changes: 2 additions & 2 deletions crates/goose/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ sha2 = "0.10"
base64 = "0.21"
url = "2.5"
urlencoding = "2.1.3"
axum = { version = "0.7" }
axum = "0.7"
tower-http = { version = "0.5", features = ["cors"] }
webbrowser = "0.8"
dotenv = "0.15"
Expand Down Expand Up @@ -83,4 +83,4 @@ path = "examples/databricks_oauth.rs"

[[bench]]
name = "tokenization_benchmark"
harness = false
harness = false
110 changes: 110 additions & 0 deletions docs/run_cross_local.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Instructions for running cross to test release builds locally

## Prerequisites
Before start, check the comments in `Cross.toml` to turn on some commented configs for the target you want to build.

## Targets
### aarch64-unknown-linux-gnu

#### Build release
```sh
CROSS_BUILD_OPTS="--platform linux/amd64 --no-cache" CROSS_CONTAINER_OPTS="--platform linux/amd64" cross build --release --target aarch64-unknown-linux-gnu
```

#### Inspect container created by cross for debugging
```sh
docker run --platform linux/amd64 -it <image-id> /bin/bash
```

#### Testing the binary

1. Download docker image for testing environment
```sh
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker pull arm64v8/ubuntu
```
2. Run the container
pwd is the directory contains the binary built in the previous step on your host machine
```sh
docker run -v $(pwd):/app -it arm64v8/ubuntu /bin/bash
```

3. Install dependencies in the container and set up api testing environment
```sh
apt update
apt install libxcb1 libxcb1-dev libdbus-1-3 nvi
mkdir -p ~/.config/goose
# create goose config file
# set api key env variable
```

### x86_64-unknown-linux-gnu

#### Build release
```sh
CROSS_BUILD_OPTS="--platform linux/amd64 --no-cache" CROSS_CONTAINER_OPTS="--platform linux/amd64" cross build --release --target x86_64-unknown-linux-gnu
```
#### inspect container created by cross for debugging
```sh
docker run --platform linux/amd64 -it <image-id> /bin/bash
```

#### Testing the binary

1. Download docker image for testing environment
```sh
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker pull --platform linux/amd64 debian:latest
```

2. Run the container
pwd is the directory contains the binary built in the previous step on your host machine
```sh
docker run --platform linux/amd64 -it -v "$(pwd)":/app debian:latest /bin/bash
```

3. Install dependencies in the container and set up api testing environment
```sh
apt update
apt install libxcb1 libxcb1-dev libdbus-1-3 nvi
mkdir -p ~/.config/goose
# create goose config file
# set api key env variable
```

### aarch64-apple-darwin

#### build release
```sh
cross build --release --target aarch64-apple-darwin
```
There is no docker image available for aarch64-apple-darwin. It will fall back to your host machine for building the binary if your host machine matches.

#### testing the build
If the binary is signed with a certificate, run
```sh
xattr -d com.apple.quarantine goose
````

### x86_64-apple-darwin

#### build release
```sh
cross build --release --target x86_64-apple-darwin
```

There is no docker image available for x86_64-apple-darwin. It will fall back to your host machine for building the binary if your host machine matches.

#### testing the build
1. If the binary is signed with a certificate, run
```sh
xattr -d com.apple.quarantine goose
````
2. If you are on Apple Silicon (ARM), you can use Rosetta to test the binary
```sh
softwareupdate --install-rosetta # make sure Rosetta 2 is installed
```

```sh
arch -x86_64 ./goose help
```
Loading