Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks using iai-callgrind #447

Merged
merged 5 commits into from
Jan 16, 2025

Conversation

tgross35
Copy link
Contributor

@tgross35 tgross35 commented Jan 16, 2025

Running walltime benchmarks in CI is notoriously unstable, Introduce benchmarks that instead use instruction count and other more reproducible metrics, using iai-callgrind 1, which we are able to run in CI with a high degree of reproducibility.

Inputs to this benchmark are a logspace sweep, which gives an approximation for real-world use, but may fail to indicate outlier cases.

@tgross35 tgross35 mentioned this pull request Jan 16, 2025
Benchmarks need a way to limit how many iterations get run. Introuce a
way to inject this information here.
@tgross35 tgross35 force-pushed the icount-benchmarks branch 2 times, most recently from 56e19b4 to 26aaf84 Compare January 16, 2025 09:06
@tgross35 tgross35 changed the title WIP: iai-callgrind Add benchmarks using iai-callgrind Jan 16, 2025
Running walltime benchmarks in CI is notoriously unstable, Introduce
benchmarks that instead use instruction count and other more
reproducible metrics, using `iai-callgrind` [1], which we are able to
run in CI with a high degree of reproducibility.

Inputs to this benchmark are a logspace sweep, which gives an
approximation for real-world use, but may fail to indicate outlier
cases.

[1]: https://github.com/iai-callgrind/iai-callgrind
Add support in `ci-util.py` for finding the most recent baseline and
downloading it, which new tests can then be compared against.

Arbitrarily select nightly-2025-01-16 as the rustc version to pin to in
benchmarks.
The icount benchmarks are what we will be relying on in CI more than the
existing benchmarks. There isn't much reason to keep these around, but
there isn't much point in dropping them either. So, just reduce the
runtime.
This failed a couple of times recently in CI, once on i686 and once on
aarch64-apple:

    thread 'main' panicked at crates/libm-test/benches/random.rs:76:65:
    called `Result::unwrap()` on an `Err` value: ynf

    Caused by:
        0:
               input:    (681, 509.90924) (0x000002a9, 0x43fef462)
               expected: -3.2161271e38          0xff71f45b
               actual:   -inf                   0xff800000
        1: mismatched infinities

    thread 'main' panicked at crates/libm-test/benches/random.rs:76:65:
    called `Result::unwrap()` on an `Err` value: ynf

    Caused by:
        0:
               input:    (132, 50.46604) (0x00000084, 0x4249dd3a)
               expected: -3.3364996e38          0xff7b02a5
               actual:   -inf                   0xff800000
        1: mismatched infinities

Add a new override to account for this.
@tgross35 tgross35 enabled auto-merge January 16, 2025 09:53
@tgross35 tgross35 merged commit a298d92 into rust-lang:master Jan 16, 2025
35 checks passed
@tgross35 tgross35 deleted the icount-benchmarks branch January 16, 2025 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant