chore(starknet_gateway): create bench file for benchmark test #2404

ArniStarkware · 2024-12-02T12:06:46Z

We can later discuss the details of the test cases we want to benchmark exactly.

Running the benchmarking action:
cargo bench --bench gateway_bench

Prints:

declares                time:   [0.0000 ps 0.0000 ps 0.0000 ps]
                        change: [-62.446% -29.235% +34.904%] (p = 0.33 > 0.05)
                        No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
  4 (4.00%) high mild
  9 (9.00%) high severe

deploy_accounts         time:   [0.0000 ps 0.0000 ps 0.0000 ps]
                        change: [-54.552% -11.417% +70.199%] (p = 0.73 > 0.05)
                        No change in performance detected.
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe

invokes                 time:   [0.0000 ps 0.0000 ps 0.0000 ps]
                        change: [-46.424% -0.1889% +83.908%] (p = 1.00 > 0.05)
                        No change in performance detected.
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe

all_transaction_types   time:   [0.0000 ps 0.0000 ps 0.0000 ps]
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe

reviewable-StarkWare · 2024-12-02T12:06:52Z

This change is

ArniStarkware · 2024-12-02T12:07:01Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2024-12-02T12:07:19Z

Artifacts upload workflows:

ArniStarkware · 2024-12-02T12:09:54Z

The insparation is blockifier_bench.

codecov · 2024-12-02T12:28:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.49%. Comparing base (e3165c4) to head (e73d179).
Report is 647 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2404       +/-   ##
===========================================
+ Coverage   40.10%   77.49%   +37.38%     
===========================================
  Files          26      387      +361     
  Lines        1895    40863    +38968     
  Branches     1895    40863    +38968     
===========================================
+ Hits          760    31667    +30907     
- Misses       1100     6941     +5841     
- Partials       35     2255     +2220

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-12-24T10:10:40Z

Benchmark movements:
tree_computation_flow performance improved 😺
tree_computation_flow time: [34.467 ms 34.506 ms 34.547 ms]
change: [-4.1866% -2.5624% -1.1267%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild

alonh5

Reviewable status: 0 of 3 files reviewed, 1 unresolved discussion (waiting on @ArniStarkware)

crates/starknet_gateway/bench/gateway_bench.rs line 14 at r4 (raw file):

use criterion::{criterion_group, criterion_main, Criterion};

pub fn declare_benchmark(c: &mut Criterion) {

Can you explain these? What are they running?

alonh5

Reviewed 1 of 3 files at r2, 1 of 1 files at r4, all commit messages.
Reviewable status: 2 of 3 files reviewed, 2 unresolved discussions (waiting on @ArniStarkware)

crates/starknet_gateway/Cargo.toml line 42 at r4 (raw file):

assert_matches.workspace = true
cairo-lang-sierra-to-casm.workspace = true
criterion = { workspace = true, features = ["html_reports"] }

What is this feature for?

Code quote:

"html_reports"

alonh5

Reviewed 1 of 3 files at r2.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @ArniStarkware)

ArniStarkware

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @alonh5)

crates/starknet_gateway/bench/gateway_bench.rs line 14 at r4 (raw file):

Previously, alonh5 (Alon Haramati) wrote…

Can you explain these? What are they running?

These = declare_benchmark etc?
These are the tests that are going to be run.

Should I have just one function for now? (just invoke_benchmark - which is being filled in the next PR: #2911)?

I wrote a comment on this PR demonstrating how the printed report of the test looks like.

ArniStarkware

Reviewable status: 2 of 3 files reviewed, 2 unresolved discussions (waiting on @alonh5)

crates/starknet_gateway/Cargo.toml line 42 at r4 (raw file):

Previously, alonh5 (Alon Haramati) wrote…

What is this feature for?

Removed. Done.

alonh5

Reviewed 2 of 2 files at r5, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @ArniStarkware)

crates/starknet_gateway/bench/gateway_bench.rs line 14 at r4 (raw file):

Previously, ArniStarkware (Arnon Hod) wrote…

These = declare_benchmark etc?
These are the tests that are going to be run.

Should I have just one function for now? (just invoke_benchmark - which is being filled in the next PR: #2911)?

I wrote a comment on this PR demonstrating how the printed report of the test looks like.

Yes, let's just have one. We can add more later easily.

ArniStarkware

Reviewable status: 1 of 3 files reviewed, 1 unresolved discussion (waiting on @alonh5)

crates/starknet_gateway/bench/gateway_bench.rs line 14 at r4 (raw file):

Previously, alonh5 (Alon Haramati) wrote…

Yes, let's just have one. We can add more later easily.

Done.

alonh5

Reviewed 2 of 2 files at r6, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ArniStarkware)

ArniStarkware requested a review from alonh5 December 2, 2024 12:08

ArniStarkware marked this pull request as ready for review December 2, 2024 12:08

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch 4 times, most recently from ae752f9 to 456c83f Compare December 24, 2024 08:51

ArniStarkware mentioned this pull request Dec 24, 2024

feat(starknet_gateway): create transaction generator towards benchmarking #2911

Merged

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch from 456c83f to b6d6af6 Compare December 24, 2024 10:02

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch from b6d6af6 to cdc7718 Compare December 25, 2024 08:26

ArniStarkware changed the base branch from main to arni/gateway/bench/mock_dependencies December 25, 2024 08:26

ArniStarkware mentioned this pull request Dec 25, 2024

test(starknet_gateway): move mock dependancies to test utils #2940

Closed

ArniStarkware force-pushed the arni/gateway/bench/mock_dependencies branch from 98bed5b to 63a13ec Compare December 25, 2024 10:00

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch 2 times, most recently from 17f602a to 165237b Compare December 25, 2024 12:01

ArniStarkware changed the base branch from arni/gateway/bench/mock_dependencies to main December 25, 2024 12:01

ArniStarkware mentioned this pull request Dec 25, 2024

refactor(starknet_gateway): gateway business logic towards bench test #2948

Merged

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch 2 times, most recently from 2a139c3 to 7cdfc96 Compare December 26, 2024 11:49

ArniStarkware changed the base branch from main to arni/gateway/bench/business_logic December 26, 2024 11:49

This was referenced Dec 26, 2024

chore(starknet_gateway): use the get nonce method of the stateful validator trait #2982

Merged

wip: add tracing to the benched function #2983

Closed

alonh5 requested changes Dec 26, 2024

View reviewed changes

alonh5 reviewed Dec 26, 2024

View reviewed changes

ArniStarkware requested a review from alonh5 December 26, 2024 14:22

ArniStarkware commented Dec 26, 2024

View reviewed changes

ArniStarkware force-pushed the arni/gateway/bench/business_logic branch from 416c277 to f191cec Compare December 26, 2024 14:43

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch from 7cdfc96 to 0d28f81 Compare December 26, 2024 14:43

ArniStarkware commented Dec 26, 2024

View reviewed changes

ArniStarkware force-pushed the arni/gateway/bench/business_logic branch from f191cec to 2a1b6aa Compare December 26, 2024 15:11

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch from 0d28f81 to d46eac5 Compare December 26, 2024 15:11

ArniStarkware force-pushed the arni/gateway/bench/business_logic branch from 2a1b6aa to 9f138f5 Compare December 26, 2024 15:14

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch 2 times, most recently from 952b87c to 12457a7 Compare December 26, 2024 15:46

ArniStarkware force-pushed the arni/gateway/bench/business_logic branch from 9f138f5 to 4dc42d6 Compare December 29, 2024 07:44

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch from 12457a7 to 351dacc Compare December 29, 2024 07:44

ArniStarkware force-pushed the arni/gateway/bench/business_logic branch from 4dc42d6 to a3070b8 Compare December 29, 2024 11:40

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch from 351dacc to 6686f4c Compare December 29, 2024 11:40

alonh5 requested changes Dec 29, 2024

View reviewed changes

ArniStarkware added 2 commits December 29, 2024 15:37

refactor(starknet_gateway): gateway business logic towards bench test

e87e6f4

chore(starknet_gateway): create bench file for benchmark test

08feed4

ArniStarkware force-pushed the arni/gateway/bench/business_logic branch from a3070b8 to e87e6f4 Compare December 29, 2024 13:40

ArniStarkware force-pushed the arni/gateway/bench/create_bench_file branch from 6686f4c to 08feed4 Compare December 29, 2024 13:41

ArniStarkware requested a review from alonh5 December 29, 2024 13:41

ArniStarkware commented Dec 29, 2024

View reviewed changes

alonh5 approved these changes Dec 29, 2024

View reviewed changes

ArniStarkware changed the base branch from arni/gateway/bench/business_logic to main December 29, 2024 14:12

ArniStarkware added this pull request to the merge queue Dec 29, 2024

Merged via the queue into main with commit 3bf13f0 Dec 29, 2024
38 checks passed

github-actions bot locked and limited conversation to collaborators Dec 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(starknet_gateway): create bench file for benchmark test #2404

chore(starknet_gateway): create bench file for benchmark test #2404

ArniStarkware commented Dec 2, 2024 •

edited

Loading

reviewable-StarkWare commented Dec 2, 2024

ArniStarkware commented Dec 2, 2024 •

edited

Loading

github-actions bot commented Dec 2, 2024 •

edited

Loading

ArniStarkware commented Dec 2, 2024

codecov bot commented Dec 2, 2024

github-actions bot commented Dec 24, 2024

alonh5 left a comment

alonh5 left a comment

alonh5 left a comment

ArniStarkware left a comment

ArniStarkware left a comment

alonh5 left a comment

ArniStarkware left a comment

alonh5 left a comment

chore(starknet_gateway): create bench file for benchmark test #2404

chore(starknet_gateway): create bench file for benchmark test #2404

Conversation

ArniStarkware commented Dec 2, 2024 • edited Loading

reviewable-StarkWare commented Dec 2, 2024

ArniStarkware commented Dec 2, 2024 • edited Loading

github-actions bot commented Dec 2, 2024 • edited Loading

ArniStarkware commented Dec 2, 2024

codecov bot commented Dec 2, 2024

Codecov Report

github-actions bot commented Dec 24, 2024

alonh5 left a comment

Choose a reason for hiding this comment

alonh5 left a comment

Choose a reason for hiding this comment

alonh5 left a comment

Choose a reason for hiding this comment

ArniStarkware left a comment

Choose a reason for hiding this comment

ArniStarkware left a comment

Choose a reason for hiding this comment

alonh5 left a comment

Choose a reason for hiding this comment

ArniStarkware left a comment

Choose a reason for hiding this comment

alonh5 left a comment

Choose a reason for hiding this comment

ArniStarkware commented Dec 2, 2024 •

edited

Loading

ArniStarkware commented Dec 2, 2024 •

edited

Loading

github-actions bot commented Dec 2, 2024 •

edited

Loading