Skip to content

Commit

Permalink
CI: Add incremental mutation testing.
Browse files Browse the repository at this point in the history
Note that this only runs on pull requests, because it needs to know a
base revision for the diff.

Also improve `cargo install` usage. We don't need to cache the builds
because `Swatinem/rust-cache` caches the binaries.
  • Loading branch information
kpreid committed Jan 14, 2024
1 parent 4be26e5 commit ad8948a
Show file tree
Hide file tree
Showing 3 changed files with 60 additions and 34 deletions.
73 changes: 52 additions & 21 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -138,17 +138,12 @@ jobs:
- run: df -h .

- name: Install wasm-pack
run: cargo install [email protected]
# Use workspace target dir for cargo install's build, so that the build will be cached.
env:
CARGO_TARGET_DIR: target/
- name: Install tools
# Note: Swatinem/rust-cache will cache these installed binaries, so we don't have to worry
# about caching the builds of them.
run: |
cargo install [email protected] [email protected] [email protected]
- name: Install cargo-about
run: cargo install [email protected]
# Use workspace target dir for cargo install's build, so that the build will be cached.
env:
CARGO_TARGET_DIR: target/
- run: df -h .

Expand Down Expand Up @@ -265,17 +260,10 @@ jobs:
- name: Compile xtask
run: cargo build --package xtask

- name: Install wasm-pack
run: cargo install [email protected]
# Use workspace target dir for cargo install's build, so that the build will be cached.
env:
CARGO_TARGET_DIR: target/

- name: Install cargo-about
run: cargo install [email protected]
# Use workspace target dir for cargo install's build, so that the build will be cached.
env:
CARGO_TARGET_DIR: target/
- name: Install tools
# These should already be installed but we might have a cache miss, so don't fail in that case.
run: |
cargo install [email protected] [email protected]
- run: df -h .

Expand All @@ -294,7 +282,50 @@ jobs:
name: cargo-timings features
path: |
target/cargo-timings/cargo-timing-*.html
# Run mutation testing against the changed code
# Based on example from https://mutants.rs/pr-diff.html
incremental-mutants:
runs-on: ubuntu-latest
# Running on push would require more work to determine the base revision
if: github.event_name == 'pull_request'
steps:
- uses: actions/checkout@v4
with:
# Needed to ensure `main` branch is needed
fetch-depth: 0

# Load cache before doing any Rust builds
# Note: we aren't depending on the build job so this cache might be stale, but the
# shorter time-to-useful-results seems worth it.
- uses: Swatinem/[email protected]
with:
# Reuse the cache from the normal `build` job instead of creating an independent one,
# to reduce redundant work and save disk space — but don't *write* to that cache, so
# we don't bloat it.
prefix-key: "v1-rust-ubuntu-stable-locked"
shared-key: "build"
save-if: false

# Install cargo-mutants
# This should usually be already installed, but is present in case of cache miss
- run: cargo install --version 24.1.0 cargo-mutants

- name: Relative diff
run: |
git branch -av
git diff origin/${{ github.base_ref }}.. | tee git.diff
- name: Mutants
# See mutants.yml for notes on the exclusions
run: cargo mutants --in-diff git.diff --package all-is-cubes --exclude-re '::fmt ->' --exclude tests.rs -- --all-features

- uses: actions/upload-artifact@v3
if: always()
with:
name: mutants-incremental.out
path: mutants.out

miri:
runs-on: ubuntu-latest

Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/mutants.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# Note that incremental mutation testing is done in ci.yml

name: Mutation Testing

permissions: {}
Expand All @@ -24,9 +26,8 @@ jobs:

- name: Run mutation testing for all-is-cubes
timeout-minutes: 60
# TODO: un-exclude src/vui eventually
# TODO: maybe un-exclude fmt()
run: cargo mutants --package all-is-cubes --exclude-re '::fmt ->' --exclude tests.rs --exclude src/vui* -- --all-features
run: cargo mutants --package all-is-cubes --exclude-re '::fmt ->' --exclude tests.rs -- --all-features
# TODO: Other crates' reasons for non-inclusion:
# all-is-cubes-content: insufficient test coverage
# all-is-cubes-desktop: insufficient test coverage, unclear if even feasible
Expand Down
16 changes: 5 additions & 11 deletions .github/workflows/unstable-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,17 +95,11 @@ jobs:
cargo xtask update "${{ matrix.depversions }}"
cargo tree --all-features
- name: Install wasm-pack
run: cargo install [email protected]
# Use workspace target dir for cargo install's build, so that the build will be cached.
env:
CARGO_TARGET_DIR: target/

- name: Install cargo-about
run: cargo install [email protected]
# Use workspace target dir for cargo install's build, so that the build will be cached.
env:
CARGO_TARGET_DIR: target/
- name: Install tools
# Note: Swatinem/rust-cache will cache these installed binaries, so we don't have to worry
# about caching the builds of them.
run: |
cargo install [email protected] [email protected]
- name: Compile basic tests
# compile is broken out so we have visibility into compile vs. run times
Expand Down

0 comments on commit ad8948a

Please sign in to comment.