Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make benchmark setup consistent #16733

Merged
merged 10 commits into from
Dec 10, 2024
Merged

Conversation

BD103
Copy link
Member

@BD103 BD103 commented Dec 9, 2024

Objective

Solution

Tip

I recommend reviewing this PR commit-by-commit, instead of all at once!

In 5d26f56 I reorganized how benches were registered. Now this is one [[bench]] per Bevy crate. In each crate benchmark folder, there is a main.rs that calls criterion_main!. I also disabled automatic benchmark discovery, which isn't necessarily required, but may clear up confusion with our custom setup. I also fixed a few errors that were causing the benchmarks to fail to compile.

In afc8d33 I ran rustfmt on all of the benchmarks.

In d6cdf96 I fixed all of the Clippy warnings.

In ee94d48 I fixed some of the benchmarks' usage of black_box(). I ended up opening rust-lang/rust#133942 due to this, which should help prevent this in the future.

In cbe1688 I renamed all of the ECS benchmark groups to be called benches, to be consistent with the other crate benchmarks.

In e701c21 and 8815bb7 I re-ordered some imports and module definitions, and uplifted fragmentation/mod.rs to fragementation.rs.

Finally, in b0065e0 I organized Cargo.toml and bumped Criterion to v0.5.

Testing

  • cd benches && cargo clippy --benches
  • cd benches && cargo fmt --all

BD103 added 4 commits December 9, 2024 14:16
See rust-lang/rust#133942, there's not much point in using `black_box()` on a unit type, especially since `map.insert()` has side-effects and will be executed.
@BD103 BD103 added C-Code-Quality A section of code that is hard to understand or change C-Benchmarks Stress tests and benchmarks used to measure how fast things are A-Cross-Cutting Impacts the entire engine D-Straightforward Simple bug fixes and API improvements, docs, test and examples S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Dec 9, 2024
BD103 added 5 commits December 9, 2024 14:39
Now the distinguishing factor will be their module path, removing the need to rename the group itself if it switches modules.
Modules first, imports second
There's nothing else within the module folder, so it doesn't need to be a folder!
Copy link
Contributor

@bushrat011899 bushrat011899 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Definitely a cleaner structure.

harness = false

[[bench]]
name = "reflect_function"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was previously running function-reflection-specific benches like:

cargo bench --bench reflect_function --all-features

Does this change mean I won't be able to run specific sets of benches anymore?

Copy link
Member Author

@BD103 BD103 Dec 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but only temporarily! Just like unit tests, you can filter what subset of benchmarks you want to run:

# Will run `typed/function`, `typed/closure`, `typed/closure_mut`, and more.
cargo bench --bench reflect -- typed

My plan with #16647 is to rename all benchmarks to include their module paths, so you'll be able to run:

# This is equivalent to `cargo bench --bench reflect_function`, using the new system.
cargo bench --bench reflect -- bevy_reflect::function

This PR was a precursor, since I didn't want to have too many changes at once.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to second this, because when I still was putting the stress test , it was very annoying to test just the bench while not testing everything else. Can you still run specific benchmarks (or groups of them) with this reorganization?

@BD103 BD103 added S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it S-Needs-Review Needs reviewer attention (from anyone!) to move forward and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it labels Dec 9, 2024
Copy link
Contributor

@Trashtalk217 Trashtalk217 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@alice-i-cecile alice-i-cecile added S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Dec 10, 2024
@alice-i-cecile
Copy link
Member

@BD103 once this is merge conflict free I'll press the button. Thanks for this: the benchmarks folder has been in a state of disarray for way too long.

@BD103

This comment was marked as resolved.

@BD103 BD103 force-pushed the bench-consistency branch from be39a7d to df26ba2 Compare December 10, 2024 20:24
@BD103
Copy link
Member Author

BD103 commented Dec 10, 2024

@alice-i-cecile should be good to go now!

@alice-i-cecile alice-i-cecile added this pull request to the merge queue Dec 10, 2024
Merged via the queue into bevyengine:main with commit c4a24d5 Dec 10, 2024
31 of 32 checks passed
@BD103 BD103 deleted the bench-consistency branch December 10, 2024 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Cross-Cutting Impacts the entire engine C-Benchmarks Stress tests and benchmarks used to measure how fast things are C-Code-Quality A section of code that is hard to understand or change D-Straightforward Simple bug fixes and API improvements, docs, test and examples S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants