Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cache garbage collection #12634

Merged
merged 31 commits into from
Nov 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
094632b
Add -Zgc flag.
ehuss Sep 7, 2023
9588fb6
Add a du utility function.
ehuss Sep 7, 2023
34fdf5f
Add a module with sqlite utilities.
ehuss Sep 7, 2023
da3ca05
Add a global cache garbage collector.
ehuss Sep 7, 2023
8f327a7
Add benchmarks for the global cache tracker.
ehuss Sep 7, 2023
809d4e9
Add unstable documentation for -Zgc
ehuss Sep 7, 2023
2a922d7
Add tracking issue number
ehuss Sep 7, 2023
14073f9
Don't repeat values in `--gc` help output
ehuss Oct 18, 2023
0fe72eb
Fix typos
ehuss Oct 18, 2023
afe71ea
Fix rustfmt formatting
ehuss Oct 18, 2023
bf0a653
Use value_parser for human size and duration CLI arguments.
ehuss Oct 18, 2023
6110d9d
Simplify with split_at
ehuss Oct 18, 2023
14d78e4
Fix to correctly use config.extra_verbose()
ehuss Oct 18, 2023
5e27630
Use average days in a month instead of fixed 30.
ehuss Oct 19, 2023
5d35867
Add a trace message when running offline.
ehuss Oct 19, 2023
803cb81
Add tracking issue
ehuss Oct 19, 2023
79da175
Always assert the package cache lock is held.
ehuss Oct 19, 2023
7c55a4e
Add headers to unstable gc docs.
ehuss Oct 19, 2023
4ad8adc
Fix unstable gc.auto config not showing the table name.
ehuss Oct 19, 2023
e7cebdc
Tweak unstable docs wording to avoid some confusion.
ehuss Oct 19, 2023
a86c4aa
Some doc clarifications and fixes.
ehuss Oct 19, 2023
4d9b165
Move types to be near their impls.
ehuss Oct 19, 2023
73b87a2
Use constants for auto gc config defaults.
ehuss Oct 20, 2023
73bf9fc
Consolidate the git checkout du logic.
ehuss Oct 20, 2023
89b870e
Use newtype wrapper for parent table IDs.
ehuss Oct 20, 2023
ead7904
Add `gc` tracing target.
ehuss Oct 20, 2023
9a41a61
Remove shared-target gc kind.
ehuss Oct 20, 2023
56c8d90
Use walkdir's built-in sorting option.
ehuss Oct 20, 2023
7dd15fe
Move gc.auto.frequency default to a constant.
ehuss Oct 20, 2023
1e1a42c
Use unreachable instead of panic.
ehuss Oct 21, 2023
0cd970b
Move gc options under a `cargo clean gc` subcommand.
ehuss Nov 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 85 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ pretty_assertions = "1.4.0"
proptest = "1.3.1"
pulldown-cmark = { version = "0.9.3", default-features = false }
rand = "0.8.5"
regex = "1.9.3"
rusqlite = { version = "0.29.0", features = ["bundled"] }
rustfix = "0.6.1"
same-file = "1.0.6"
security-framework = "2.9.2"
Expand Down Expand Up @@ -162,6 +164,8 @@ pasetors.workspace = true
pathdiff.workspace = true
pulldown-cmark.workspace = true
rand.workspace = true
regex.workspace = true
rusqlite.workspace = true
rustfix.workspace = true
semver.workspace = true
serde = { workspace = true, features = ["derive"] }
Expand Down
39 changes: 36 additions & 3 deletions benches/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,23 @@ cd benches/benchsuite
cargo bench
```

The tests involve downloading the index and benchmarking against some
However, running all benchmarks would take many minutes, so in most cases it
is recommended to just run the benchmarks relevant to whatever section of code
you are working on.

## Benchmarks

There are several different kinds of benchmarks in the `benchsuite/benches` directory:

* `global_cache_tracker` — Benchmarks saving data to the global cache tracker
database using samples of real-world data.
* `resolve` — Benchmarks the resolver against simulations of real-world workspaces.
* `workspace_initialization` — Benchmarks initialization of a workspace
against simulations of real-world workspaces.

### Resolve benchmarks

The resolve benchmarks involve downloading the index and benchmarking against some
real-world and artificial workspaces located in the [`workspaces`](workspaces)
directory.

Expand All @@ -21,15 +37,32 @@ faster. You can (and probably should) specify individual benchmarks to run to
narrow it down to a more reasonable set, for example:

```sh
cargo bench -- resolve_ws/rust
cargo bench -p benchsuite --bench resolve -- resolve_ws/rust
```

This will only download what's necessary for the rust-lang/rust workspace
(which is about 330MB) and run the benchmarks against it (which should take
about a minute). To get a list of all the benchmarks, run:

```sh
cargo bench -- --list
cargo bench -p benchsuite --bench resolve -- --list
```

### Global cache tracker

The `global_cache_tracker` benchmark tests saving data to the global cache
tracker database using samples of real-world data. This benchmark should run
relatively quickly.

The real-world data is based on a capture of my personal development
environment which has accumulated a large cache. So it is somewhat arbitrary,
but hopefully representative of a challenging environment. Capturing of the
data is done with the `capture-last-use` binary, which you can run if you need
to rebuild the database. Just try to run on a system with a relatively full
cache in your cargo home directory.

```sh
cargo bench -p benchsuite --bench global_cache_tracker
```

## Viewing reports
Expand Down
6 changes: 6 additions & 0 deletions benches/benchsuite/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ publish = false

[dependencies]
cargo.workspace = true
cargo-util.workspace = true
criterion.workspace = true
flate2.workspace = true
rand.workspace = true
tar.workspace = true
url.workspace = true

Expand All @@ -26,3 +28,7 @@ harness = false
[[bench]]
name = "workspace_initialization"
harness = false

[[bench]]
name = "global_cache_tracker"
harness = false
Loading
Loading