[short][move e2e bench] Add gas metrics and more existing transaction types #15302

igor-aptos · 2024-11-18T19:35:00Z

Description

This allows for easier understanding of gas calibration. Looking at gas/s metrics (and execution and io breakdown) makes it easy to calibrate new constants.

How Has This Been Tested?

used to calibrate vector::move_range native

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-11-18T19:35:04Z

⏱️ 2h 39m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / single-node-performance	35m	🟥 🟥 🟩
rust-move-tests	12m	🟩
rust-move-tests	12m	🟩
rust-move-tests	12m	🟩
rust-move-tests	12m	🟩
rust-move-tests	11m	🟥
check-dynamic-deps	10m	🟩 🟩 🟩 🟩 🟩
rust-cargo-deny	9m	🟩 🟩 🟩 🟩 🟩
execution-performance / test-target-determinator	9m	🟩 🟩
test-target-determinator	9m	🟩 🟩
check	7m	🟩 🟩
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
fetch-last-released-docker-image-tag	3m	🟩 🟩
general-lints	2m	🟩 🟩 🟩 🟩 🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

igor-aptos · 2024-11-18T19:35:20Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

georgemitenkov · 2024-11-18T19:49:32Z

aptos-move/e2e-tests/src/executor.rs

@@ -144,6 +144,13 @@ pub enum GasMeterType {
    UnmeteredGasMeter,
 }

+#[derive(Clone)]
+pub struct TimeAndGas {
+    pub elapsed_micros: u128,


nit: elapsed_micro_secs?

rust uses term micros, i.e. Duration::from_micros

georgemitenkov · 2024-11-18T19:52:38Z

aptos-move/e2e-tests/src/executor.rs

@@ -144,6 +144,13 @@ pub enum GasMeterType {
    UnmeteredGasMeter,
 }

+#[derive(Clone)]
+pub struct TimeAndGas {


nit: calling it Measurement can be more consitent with what you call it in e2e-benchmarks

georgemitenkov · 2024-11-18T19:55:34Z

aptos-move/e2e-benchmark/src/main.rs

        );

        json_lines.push(json!({
            "grep": "grep_json_aptos_move_vm_perf",
            "transaction_type": entry_point_name,
-            "wall_time_us": elapsed_micros,
+            "wall_time_us": measurement.elapsed_micros,
+            "gps": (measurement.execution_gas + measurement.io_gas) as u128 / measurement.elapsed_micros,


gas/us for better readability? Plus it is not seconds?

parsing with / in logs can be more complicated - this json is for humio graphs

gas unit is 1/10^6 gas internal units, and same as micros, so this is gas unit / s

I see, can you maybe put a comment for that here? gps is ok 👍

… types

graphite-app · 2024-11-19T18:54:50Z

aptos-move/e2e-benchmark/src/main.rs

+InitializeVectorPicture { length: 128 }	6	0.973	1.066	75.0
+VectorPicture { length: 128 }	6	0.955	1.092	22.0
+VectorPictureRead { length: 128 }	6	0.952	1.047	21.0


The timing data for InitializeVectorPicture, VectorPicture, and VectorPictureRead operations appears to be using measurements from when length was 40, but the code now tests with length of 128. These calibration values should be regenerated with the new length to maintain benchmark accuracy. Without updated measurements, performance regressions or improvements may go undetected since the baseline values don't reflect the current workload.

Spotted by Graphite Reviewer

Is this helpful? React 👍 or 👎 to let us know.

github-actions · 2024-11-19T21:14:20Z

✅ Forge suite `compat` success on `be3ccc58ef518259d63fab7f3e3613f2a6214268` ==> `830e5e8e317dd10b5277238a79df4bf3e361c9e8`

Compatibility test results for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 830e5e8e317dd10b5277238a79df4bf3e361c9e8 (PR)
1. Check liveness of validators at old version: be3ccc58ef518259d63fab7f3e3613f2a6214268
compatibility::simple-validator-upgrade::liveness-check : committed: 12387.38 txn/s, latency: 2373.79 ms, (p50: 1700 ms, p70: 1900, p90: 2800 ms, p99: 27000 ms), latency samples: 496260
2. Upgrading first Validator to new version: 830e5e8e317dd10b5277238a79df4bf3e361c9e8
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 5188.30 txn/s, latency: 5629.99 ms, (p50: 6100 ms, p70: 6500, p90: 7100 ms, p99: 7300 ms), latency samples: 97740
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 5123.89 txn/s, latency: 6295.16 ms, (p50: 6700 ms, p70: 7100, p90: 7500 ms, p99: 8300 ms), latency samples: 171840
3. Upgrading rest of first batch to new version: 830e5e8e317dd10b5277238a79df4bf3e361c9e8
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6421.85 txn/s, latency: 4450.92 ms, (p50: 4800 ms, p70: 5100, p90: 5700 ms, p99: 6000 ms), latency samples: 123320
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6544.36 txn/s, latency: 4944.65 ms, (p50: 5300 ms, p70: 5600, p90: 6600 ms, p99: 7100 ms), latency samples: 221920
4. upgrading second batch to new version: 830e5e8e317dd10b5277238a79df4bf3e361c9e8
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11100.38 txn/s, latency: 2452.62 ms, (p50: 2400 ms, p70: 2500, p90: 4200 ms, p99: 4400 ms), latency samples: 189920
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11203.24 txn/s, latency: 2779.77 ms, (p50: 2500 ms, p70: 2600, p90: 5500 ms, p99: 7300 ms), latency samples: 361800
5. check swarm health
Compatibility test for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 830e5e8e317dd10b5277238a79df4bf3e361c9e8 passed
Test Ok

github-actions · 2024-11-19T21:14:59Z

✅ Forge suite `realistic_env_max_load` success on `830e5e8e317dd10b5277238a79df4bf3e361c9e8`

two traffics test: inner traffic : committed: 13379.60 txn/s, submitted: 13380.23 txn/s, expired: 0.63 txn/s, latency: 2949.16 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 12700 ms), latency samples: 5087220
two traffics test : committed: 99.92 txn/s, latency: 1536.91 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 11500 ms), latency samples: 1840
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.461, avg: 1.779", "ConsensusProposalToOrdered: max: 0.333, avg: 0.301", "ConsensusOrderedToCommit: max: 0.378, avg: 0.369", "ConsensusProposalToCommit: max: 0.698, avg: 0.669"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.04s no progress at version 5466246 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.68s no progress at version 5464893 (avg 7.96s) [limit 15].
Test Ok

github-actions · 2024-11-19T21:15:48Z

✅ Forge suite `framework_upgrade` success on `be3ccc58ef518259d63fab7f3e3613f2a6214268` ==> `830e5e8e317dd10b5277238a79df4bf3e361c9e8`

Compatibility test results for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 830e5e8e317dd10b5277238a79df4bf3e361c9e8 (PR)
Upgrade the nodes to version: 830e5e8e317dd10b5277238a79df4bf3e361c9e8
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1345.53 txn/s, submitted: 1348.72 txn/s, failed submission: 3.19 txn/s, expired: 3.19 txn/s, latency: 2304.49 ms, (p50: 2100 ms, p70: 2400, p90: 3600 ms, p99: 5700 ms), latency samples: 118100
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1337.32 txn/s, submitted: 1340.17 txn/s, failed submission: 2.85 txn/s, expired: 2.85 txn/s, latency: 2210.05 ms, (p50: 1800 ms, p70: 2300, p90: 3900 ms, p99: 5700 ms), latency samples: 122000
5. check swarm health
Compatibility test for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 830e5e8e317dd10b5277238a79df4bf3e361c9e8 passed
Upgrade the remaining nodes to version: 830e5e8e317dd10b5277238a79df4bf3e361c9e8
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1425.48 txn/s, submitted: 1429.51 txn/s, failed submission: 4.03 txn/s, expired: 4.03 txn/s, latency: 2208.26 ms, (p50: 2100 ms, p70: 2400, p90: 3300 ms, p99: 4800 ms), latency samples: 127360
Test Ok

igor-aptos mentioned this pull request Nov 18, 2024

[move-stdlib] Use vector::move_range inside vector, and evaluate performance / calibrate gas #14862

Merged

8 tasks

igor-aptos marked this pull request as ready for review November 18, 2024 19:36

igor-aptos requested review from georgemitenkov, msmouse, vgao1996 and ziaptos November 18, 2024 19:37

vgao1996 approved these changes Nov 18, 2024

View reviewed changes

georgemitenkov approved these changes Nov 18, 2024

View reviewed changes

[short][move e2e bench] Add gas metrics and more existing transaction…

89444f9

… types

igor-aptos force-pushed the igor/add_gas_to_move_e2e branch from 9289a13 to 538ad1f Compare November 19, 2024 18:24

igor-aptos requested review from lightmark and grao1991 as code owners November 19, 2024 18:24

igor-aptos force-pushed the igor/add_gas_to_move_e2e branch from 538ad1f to 54cc4b0 Compare November 19, 2024 18:53

graphite-app bot reviewed Nov 19, 2024

View reviewed changes

igor-aptos enabled auto-merge (squash) November 19, 2024 19:10

This comment has been minimized.

Sign in to view

addressing comments

830e5e8

igor-aptos force-pushed the igor/add_gas_to_move_e2e branch from 54cc4b0 to 830e5e8 Compare November 19, 2024 20:42

This comment has been minimized.

Sign in to view

igor-aptos merged commit 6acc3d5 into main Nov 19, 2024
48 checks passed

igor-aptos deleted the igor/add_gas_to_move_e2e branch November 19, 2024 22:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[short][move e2e bench] Add gas metrics and more existing transaction types #15302

[short][move e2e bench] Add gas metrics and more existing transaction types #15302

igor-aptos commented Nov 18, 2024 •

edited

Loading

trunk-io bot commented Nov 18, 2024 •

edited

Loading

igor-aptos commented Nov 18, 2024 •

edited

Loading

georgemitenkov Nov 18, 2024

igor-aptos Nov 18, 2024

georgemitenkov Nov 18, 2024

georgemitenkov Nov 18, 2024

igor-aptos Nov 18, 2024

georgemitenkov Nov 18, 2024

graphite-app bot Nov 19, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

[short][move e2e bench] Add gas metrics and more existing transaction types #15302

[short][move e2e bench] Add gas metrics and more existing transaction types #15302

Conversation

igor-aptos commented Nov 18, 2024 • edited Loading

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Nov 18, 2024 • edited Loading

igor-aptos commented Nov 18, 2024 • edited Loading

georgemitenkov Nov 18, 2024

Choose a reason for hiding this comment

igor-aptos Nov 18, 2024

Choose a reason for hiding this comment

georgemitenkov Nov 18, 2024

Choose a reason for hiding this comment

georgemitenkov Nov 18, 2024

Choose a reason for hiding this comment

igor-aptos Nov 18, 2024

Choose a reason for hiding this comment

georgemitenkov Nov 18, 2024

Choose a reason for hiding this comment

graphite-app bot Nov 19, 2024

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Nov 19, 2024

✅ Forge suite compat success on be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 830e5e8e317dd10b5277238a79df4bf3e361c9e8

github-actions bot commented Nov 19, 2024

✅ Forge suite realistic_env_max_load success on 830e5e8e317dd10b5277238a79df4bf3e361c9e8

github-actions bot commented Nov 19, 2024

✅ Forge suite framework_upgrade success on be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 830e5e8e317dd10b5277238a79df4bf3e361c9e8

igor-aptos commented Nov 18, 2024 •

edited

Loading

trunk-io bot commented Nov 18, 2024 •

edited

Loading

igor-aptos commented Nov 18, 2024 •

edited

Loading

✅ Forge suite `compat` success on `be3ccc58ef518259d63fab7f3e3613f2a6214268` ==> `830e5e8e317dd10b5277238a79df4bf3e361c9e8`

✅ Forge suite `realistic_env_max_load` success on `830e5e8e317dd10b5277238a79df4bf3e361c9e8`

✅ Forge suite `framework_upgrade` success on `be3ccc58ef518259d63fab7f3e3613f2a6214268` ==> `830e5e8e317dd10b5277238a79df4bf3e361c9e8`