Integrated Event Profiling

[Hirevo]

This PR integrates a profiler within the interpreter (only the bytecode interpreter for now), which records events along with their timings and allows to generate callgraphs, flamegraphs and other visualisations of the interpreter's runtime.

It makes use of the measureme crate, which is an event recorder created by the Rust compiler team and used for the -Z self-profile compiler flag in the nightly Rust compiler.

The profiling features are gated behind a new Cargo feature flag called profiler for the bytecode interpreter, to avoid slowing down normal builds.

A profiler-instrumented binary can be built like so:

cargo build -p som-interpreter-bc --release --features profiler

After running the instrumented interpreter, a file called som-trace.mm_profdata is created in the current directory, containing all of the recorded events and their timings.

A collection of separate tools (also from the Rust compiler team) can then be used to visualise this data in various ways:

• summarize: produces a summary as an ASCII table for the terminal.

Example

• crox: transforms the data into a chrome_profiler.json file, compatible with the Chromium performance tools.

Example

• flamegraph: transforms the data into a flamegraph, similar to the one from perf tools.
  The output file is named rustc.svg.

Example

These examples are all generated from the exact same som-trace.mm_profdata file, obtained from running the SOM Test Suite.

Unlike measureme, these tools are not released on crates.io, so the main way to install them is to install them directly from the git repository, like so:

# For summarize:
cargo install --git 'https://github.com/rust-lang/measureme' --tag '11.0.1' summarize
# For crox:
cargo install --git 'https://github.com/rust-lang/measureme' --tag '11.0.1' crox
# For flamegraph:
cargo install --git 'https://github.com/rust-lang/measureme' --tag '11.0.1' flamegraph

[som-rs-benchmarker]

Here are the benchmark results for feat/profiler (commit: 681bc46):

AST interpreter

+-----------------+---------------------------------------+---------------------------+
| Benchmark       | master (base)                         | feat/profiler (head)      |
+-----------------+---------------------------------------+---------------------------+
| Bounce          | 210.40 ms ± 9.46 (195.96..225.68)     | 1.08x ± 0.06 (0.99..1.13) |
| BubbleSort      | 265.21 ms ± 8.20 (256.16..279.12)     | 0.96x ± 0.08 (0.81..1.04) |
| DeltaBlue       | 160.43 ms ± 9.48 (152.47..183.33)     | 1.01x ± 0.07 (0.91..1.04) |
| Dispatch        | 188.78 ms ± 8.34 (181.54..209.37)     | 1.01x ± 0.05 (0.97..1.05) |
| Fannkuch        | 125.48 ms ± 6.77 (119.25..141.37)     | 1.00x ± 0.09 (0.85..1.06) |
| Fibonacci       | 381.68 ms ± 14.11 (355.36..405.86)    | 1.02x ± 0.05 (0.95..1.09) |
| FieldLoop       | 337.86 ms ± 6.54 (328.92..350.15)     | 1.02x ± 0.04 (0.96..1.07) |
| GraphSearch     | 82.57 ms ± 2.37 (78.47..86.78)        | 0.99x ± 0.05 (0.87..1.02) |
| IntegerLoop     | 329.19 ms ± 8.57 (317.84..344.69)     | 0.97x ± 0.05 (0.90..1.02) |
| JsonSmall       | 209.51 ms ± 16.93 (193.99..243.91)    | 1.03x ± 0.10 (0.95..1.08) |
| List            | 231.90 ms ± 5.61 (224.04..242.44)     | 0.99x ± 0.04 (0.92..1.04) |
| Loop            | 435.17 ms ± 16.88 (414.31..463.97)    | 1.03x ± 0.05 (0.99..1.07) |
| Mandelbrot      | 255.85 ms ± 5.12 (249.60..265.68)     | 0.98x ± 0.03 (0.96..1.01) |
| NBody           | 215.31 ms ± 11.23 (202.67..237.06)    | 0.96x ± 0.11 (0.77..1.06) |
| PageRank        | 300.23 ms ± 16.85 (286.04..345.57)    | 0.95x ± 0.08 (0.86..1.03) |
| Permute         | 304.59 ms ± 17.06 (287.34..336.64)    | 1.00x ± 0.08 (0.91..1.06) |
| Queens          | 253.43 ms ± 13.59 (228.32..272.76)    | 1.05x ± 0.09 (0.91..1.12) |
| QuickSort       | 82.16 ms ± 7.86 (73.60..96.36)        | 1.08x ± 0.11 (1.03..1.13) |
| Recurse         | 279.66 ms ± 21.85 (263.79..324.09)    | 0.98x ± 0.10 (0.85..1.04) |
| Richards        | 4045.11 ms ± 87.93 (3894.33..4161.91) | 1.00x ± 0.04 (0.95..1.03) |
| Sieve           | 426.96 ms ± 8.94 (414.27..437.44)     | 1.04x ± 0.04 (0.96..1.08) |
| Storage         | 83.72 ms ± 2.68 (79.18..88.07)        | 0.94x ± 0.10 (0.79..1.03) |
| Sum             | 170.28 ms ± 17.98 (155.05..216.13)    | 1.06x ± 0.11 (1.03..1.09) |
| Towers          | 312.78 ms ± 9.91 (295.39..332.90)     | 0.99x ± 0.06 (0.89..1.03) |
| TreeSort        | 160.20 ms ± 7.78 (152.69..176.07)     | 1.00x ± 0.06 (0.91..1.04) |
| WhileLoop       | 367.55 ms ± 14.71 (357.22..396.66)    | 0.97x ± 0.07 (0.89..1.05) |
|                 |                                       |                           |
| Average Speedup |              (baseline)               | 1.00x ± 0.01 (0.94..1.08) |
+-----------------+---------------------------------------+---------------------------+

The raw ReBench data files are available for download here: baseline and head

Bytecode interpreter

+-----------------+---------------------------------------+---------------------------+
| Benchmark       | master (base)                         | feat/profiler (head)      |
+-----------------+---------------------------------------+---------------------------+
| Bounce          | 86.39 ms ± 2.63 (82.54..91.42)        | 0.96x ± 0.04 (0.92..0.99) |
| BubbleSort      | 126.03 ms ± 2.37 (121.83..129.62)     | 0.97x ± 0.13 (0.73..1.09) |
| DeltaBlue       | 68.03 ms ± 2.39 (65.77..74.01)        | 0.82x ± 0.16 (0.61..1.01) |
| Dispatch        | 93.31 ms ± 8.62 (86.62..111.56)       | 1.00x ± 0.17 (0.74..1.11) |
| Fannkuch        | 57.35 ms ± 1.86 (55.08..60.66)        | 1.03x ± 0.04 (0.98..1.05) |
| Fibonacci       | 153.62 ms ± 4.29 (146.33..161.70)     | 0.97x ± 0.04 (0.92..1.00) |
| FieldLoop       | 207.05 ms ± 6.17 (197.67..220.02)     | 0.99x ± 0.04 (0.93..1.03) |
| GraphSearch     | 43.52 ms ± 5.94 (37.75..56.15)        | 1.10x ± 0.17 (0.96..1.17) |
| IntegerLoop     | 163.98 ms ± 5.01 (157.25..171.66)     | 0.98x ± 0.05 (0.90..1.02) |
| JsonSmall       | 104.09 ms ± 9.61 (96.99..129.18)      | 1.04x ± 0.10 (0.99..1.07) |
| List            | 119.97 ms ± 3.18 (116.64..125.69)     | 0.96x ± 0.05 (0.88..1.00) |
| Loop            | 206.90 ms ± 8.68 (199.18..224.32)     | 0.97x ± 0.10 (0.77..1.04) |
| Mandelbrot      | 130.99 ms ± 6.51 (126.17..148.60)     | 0.95x ± 0.11 (0.77..1.03) |
| NBody           | 97.10 ms ± 8.37 (90.37..116.18)       | 1.02x ± 0.10 (0.97..1.08) |
| PageRank        | 141.69 ms ± 7.39 (133.95..158.84)     | 1.01x ± 0.07 (0.95..1.06) |
| Permute         | 134.53 ms ± 5.97 (129.61..144.79)     | 1.00x ± 0.09 (0.82..1.05) |
| Queens          | 104.66 ms ± 8.47 (93.88..121.27)      | 1.00x ± 0.09 (0.96..1.05) |
| QuickSort       | 36.89 ms ± 5.23 (32.31..49.12)        | 1.02x ± 0.25 (0.68..1.15) |
| Recurse         | 128.37 ms ± 5.77 (123.30..143.42)     | 0.99x ± 0.10 (0.84..1.07) |
| Richards        | 1709.58 ms ± 49.68 (1652.49..1805.86) | 1.01x ± 0.04 (0.95..1.05) |
| Sieve           | 192.20 ms ± 7.17 (183.51..207.84)     | 0.96x ± 0.10 (0.77..1.03) |
| Storage         | 39.74 ms ± 1.34 (37.95..41.51)        | 0.97x ± 0.14 (0.71..1.10) |
| Sum             | 78.85 ms ± 0.78 (77.78..80.17)        | 0.94x ± 0.09 (0.74..1.01) |
| Towers          | 142.12 ms ± 5.98 (135.66..155.60)     | 1.00x ± 0.05 (0.95..1.04) |
| TreeSort        | 60.25 ms ± 5.99 (54.43..71.12)        | 1.06x ± 0.11 (1.00..1.10) |
| WhileLoop       | 202.34 ms ± 9.68 (194.47..227.38)     | 0.97x ± 0.08 (0.86..1.04) |
|                 |                                       |                           |
| Average Speedup |              (baseline)               | 0.99x ± 0.02 (0.82..1.10) |
+-----------------+---------------------------------------+---------------------------+

The raw ReBench data files are available for download here: baseline and head

The benchmarks were run using ReBench v1.2.0
The statistical analysis was done using rebench-tabler v0.1.0

The source code of this benchmark runner is available as a GitHub Gist for more details about the setup

[som-rs-benchmarker]

Here are the benchmark results for feat/profiler (commit: a2e442f):

AST interpreter

+-----------------+---------------------------------------+---------------------------+
| Benchmark       | master (base)                         | feat/profiler (head)      |
+-----------------+---------------------------------------+---------------------------+
| Bounce          | 199.96 ms ± 8.55 (187.13..212.33)     | 1.03x ± 0.08 (0.94..1.10) |
| BubbleSort      | 273.13 ms ± 16.94 (251.67..306.42)    | 1.00x ± 0.08 (0.94..1.06) |
| DeltaBlue       | 167.87 ms ± 8.38 (161.21..189.59)     | 1.04x ± 0.09 (0.91..1.11) |
| Dispatch        | 187.03 ms ± 6.46 (179.90..200.12)     | 0.97x ± 0.04 (0.93..1.00) |
| Fannkuch        | 129.08 ms ± 14.10 (119.72..168.35)    | 0.97x ± 0.13 (0.79..1.02) |
| Fibonacci       | 365.88 ms ± 10.33 (344.83..382.48)    | 0.96x ± 0.07 (0.86..1.04) |
| FieldLoop       | 329.98 ms ± 7.23 (318.64..345.19)     | 0.98x ± 0.05 (0.89..1.03) |
| GraphSearch     | 84.67 ms ± 6.04 (78.44..97.52)        | 0.99x ± 0.12 (0.80..1.07) |
| IntegerLoop     | 356.01 ms ± 37.39 (317.45..421.82)    | 1.04x ± 0.12 (0.99..1.10) |
| JsonSmall       | 211.94 ms ± 22.89 (194.73..275.38)    | 0.99x ± 0.11 (0.93..1.06) |
| List            | 233.13 ms ± 9.58 (222.94..256.93)     | 1.01x ± 0.06 (0.95..1.05) |
| Loop            | 420.68 ms ± 16.61 (399.04..453.52)    | 0.93x ± 0.09 (0.75..1.04) |
| Mandelbrot      | 261.24 ms ± 7.15 (252.25..273.99)     | 1.00x ± 0.04 (0.96..1.05) |
| NBody           | 217.95 ms ± 9.22 (205.12..232.53)     | 0.99x ± 0.11 (0.77..1.06) |
| PageRank        | 318.78 ms ± 22.78 (289.29..347.31)    | 1.05x ± 0.09 (0.96..1.10) |
| Permute         | 307.40 ms ± 14.29 (291.27..332.66)    | 1.02x ± 0.08 (0.87..1.07) |
| Queens          | 250.90 ms ± 16.44 (233.27..287.93)    | 1.02x ± 0.09 (0.89..1.07) |
| QuickSort       | 75.59 ms ± 2.16 (72.74..79.76)        | 1.01x ± 0.04 (0.98..1.05) |
| Recurse         | 280.63 ms ± 19.74 (264.41..321.53)    | 0.98x ± 0.09 (0.89..1.05) |
| Richards        | 4045.80 ms ± 57.23 (3967.96..4122.25) | 1.00x ± 0.02 (0.98..1.03) |
| Sieve           | 427.48 ms ± 17.63 (406.58..468.85)    | 0.99x ± 0.05 (0.95..1.04) |
| Storage         | 83.45 ms ± 7.08 (78.17..103.10)       | 0.98x ± 0.09 (0.95..1.01) |
| Sum             | 163.91 ms ± 4.23 (157.11..172.84)     | 0.96x ± 0.06 (0.87..1.02) |
| Towers          | 323.74 ms ± 25.78 (300.00..388.75)    | 0.97x ± 0.09 (0.88..1.03) |
| TreeSort        | 154.41 ms ± 4.21 (149.53..165.11)     | 0.98x ± 0.04 (0.95..1.01) |
| WhileLoop       | 375.37 ms ± 18.53 (352.26..409.01)    | 1.00x ± 0.06 (0.95..1.04) |
|                 |                                       |                           |
| Average Speedup |              (baseline)               | 0.99x ± 0.02 (0.93..1.05) |
+-----------------+---------------------------------------+---------------------------+

The raw ReBench data files are available for download here: baseline and head

Bytecode interpreter

+-----------------+---------------------------------------+---------------------------+
| Benchmark       | master (base)                         | feat/profiler (head)      |
+-----------------+---------------------------------------+---------------------------+
| Bounce          | 90.98 ms ± 3.43 (86.55..95.69)        | 0.95x ± 0.09 (0.81..1.02) |
| BubbleSort      | 130.42 ms ± 11.07 (119.74..149.92)    | 1.06x ± 0.09 (1.02..1.10) |
| DeltaBlue       | 72.87 ms ± 5.51 (67.98..86.08)        | 1.02x ± 0.10 (0.90..1.09) |
| Dispatch        | 98.04 ms ± 9.72 (88.29..114.30)       | 1.09x ± 0.11 (1.03..1.13) |
| Fannkuch        | 57.19 ms ± 1.92 (54.69..60.42)        | 1.02x ± 0.04 (0.99..1.05) |
| Fibonacci       | 151.84 ms ± 2.84 (148.60..156.66)     | 0.92x ± 0.08 (0.83..1.02) |
| FieldLoop       | 206.47 ms ± 5.99 (198.34..216.48)     | 0.93x ± 0.05 (0.87..0.98) |
| GraphSearch     | 38.10 ms ± 1.04 (36.30..39.77)        | 0.95x ± 0.03 (0.91..0.98) |
| IntegerLoop     | 166.14 ms ± 12.01 (149.91..185.00)    | 0.99x ± 0.10 (0.85..1.05) |
| JsonSmall       | 101.44 ms ± 7.91 (95.49..118.56)      | 1.00x ± 0.11 (0.82..1.10) |
| List            | 119.61 ms ± 4.34 (115.66..130.10)     | 1.01x ± 0.05 (0.92..1.07) |
| Loop            | 217.92 ms ± 10.40 (207.04..237.68)    | 1.05x ± 0.06 (0.99..1.10) |
| Mandelbrot      | 131.62 ms ± 4.70 (127.69..141.95)     | 0.99x ± 0.06 (0.90..1.05) |
| NBody           | 93.17 ms ± 1.95 (90.40..96.39)        | 0.98x ± 0.03 (0.93..1.01) |
| PageRank        | 140.41 ms ± 13.23 (132.19..176.74)    | 0.92x ± 0.12 (0.82..1.01) |
| Permute         | 139.24 ms ± 4.81 (134.90..151.53)     | 1.06x ± 0.06 (0.99..1.11) |
| Queens          | 101.17 ms ± 5.88 (96.17..116.47)      | 1.00x ± 0.08 (0.90..1.07) |
| QuickSort       | 34.03 ms ± 0.98 (32.84..35.72)        | 0.95x ± 0.11 (0.75..1.06) |
| Recurse         | 134.68 ms ± 6.93 (126.86..145.08)     | 1.01x ± 0.08 (0.88..1.06) |
| Richards        | 1688.94 ms ± 41.42 (1620.62..1760.54) | 0.99x ± 0.04 (0.93..1.03) |
| Sieve           | 211.73 ms ± 15.12 (192.44..238.33)    | 1.06x ± 0.11 (0.92..1.14) |
| Storage         | 39.76 ms ± 4.75 (35.26..51.54)        | 0.94x ± 0.21 (0.64..1.08) |
| Sum             | 91.50 ms ± 14.33 (80.62..127.32)      | 1.03x ± 0.22 (0.80..1.17) |
| Towers          | 144.40 ms ± 7.35 (135.59..160.36)     | 1.02x ± 0.07 (0.95..1.08) |
| TreeSort        | 53.69 ms ± 1.85 (51.02..56.99)        | 0.92x ± 0.10 (0.78..1.03) |
| WhileLoop       | 209.32 ms ± 7.88 (201.53..228.56)     | 1.03x ± 0.05 (0.95..1.06) |
|                 |                                       |                           |
| Average Speedup |              (baseline)               | 1.00x ± 0.02 (0.92..1.09) |
+-----------------+---------------------------------------+---------------------------+

The raw ReBench data files are available for download here: baseline and head

The benchmarks were run using ReBench v1.2.0
The statistical analysis was done using rebench-tabler v0.1.0

The source code of this benchmark runner is available as a GitHub Gist for more details about the setup

[som-rs-benchmarker]

Here are the benchmark results for feat/profiler (commit: 604a8e1):

AST interpreter

+-----------------+----------------------------------------+---------------------------+
| Benchmark       | master (base)                          | feat/profiler (head)      |
+-----------------+----------------------------------------+---------------------------+
| Bounce          | 206.84 ms ± 17.85 (186.35..235.56)     | 0.98x ± 0.10 (0.89..1.09) |
| BubbleSort      | 302.19 ms ± 32.44 (268.48..350.46)     | 1.08x ± 0.13 (0.92..1.13) |
| DeltaBlue       | 190.69 ms ± 18.51 (172.07..225.42)     | 1.10x ± 0.13 (0.97..1.22) |
| Dispatch        | 193.36 ms ± 5.73 (186.41..202.97)      | 0.96x ± 0.09 (0.79..1.05) |
| Fannkuch        | 143.37 ms ± 22.07 (120.31..178.34)     | 0.93x ± 0.19 (0.77..1.07) |
| Fibonacci       | 434.64 ms ± 31.39 (407.51..515.00)     | 1.10x ± 0.09 (1.01..1.17) |
| FieldLoop       | 343.26 ms ± 15.32 (325.08..372.27)     | 0.93x ± 0.10 (0.78..1.08) |
| GraphSearch     | 86.51 ms ± 2.10 (82.45..89.23)         | 0.93x ± 0.10 (0.74..1.02) |
| IntegerLoop     | 356.92 ms ± 21.87 (326.93..391.08)     | 0.98x ± 0.07 (0.89..1.05) |
| JsonSmall       | 216.35 ms ± 10.26 (203.80..240.05)     | 0.93x ± 0.07 (0.83..1.00) |
| List            | 253.60 ms ± 12.87 (238.56..280.79)     | 0.97x ± 0.11 (0.78..1.09) |
| Loop            | 442.50 ms ± 17.10 (419.01..478.50)     | 0.99x ± 0.06 (0.93..1.05) |
| Mandelbrot      | 289.77 ms ± 17.80 (262.70..321.93)     | 1.04x ± 0.09 (0.95..1.12) |
| NBody           | 246.61 ms ± 26.95 (218.51..299.38)     | 1.10x ± 0.13 (1.04..1.23) |
| PageRank        | 327.92 ms ± 15.27 (291.92..345.53)     | 1.05x ± 0.07 (0.97..1.13) |
| Permute         | 329.59 ms ± 21.07 (307.71..377.54)     | 0.95x ± 0.10 (0.78..1.01) |
| Queens          | 275.02 ms ± 18.06 (255.01..308.64)     | 1.06x ± 0.09 (0.99..1.15) |
| QuickSort       | 89.72 ms ± 11.57 (75.59..109.59)       | 1.05x ± 0.24 (0.76..1.22) |
| Recurse         | 279.71 ms ± 13.76 (257.17..300.81)     | 0.96x ± 0.06 (0.90..1.01) |
| Richards        | 4585.98 ms ± 269.61 (4288.50..5209.93) | 1.01x ± 0.10 (0.91..1.13) |
| Sieve           | 472.01 ms ± 33.74 (441.86..546.17)     | 1.07x ± 0.10 (0.97..1.17) |
| Storage         | 99.11 ms ± 8.91 (90.95..119.92)        | 1.01x ± 0.14 (0.84..1.13) |
| Sum             | 176.62 ms ± 13.38 (167.48..213.27)     | 1.00x ± 0.10 (0.92..1.09) |
| Towers          | 332.20 ms ± 22.22 (305.78..380.62)     | 0.88x ± 0.12 (0.73..1.03) |
| TreeSort        | 171.98 ms ± 12.93 (155.91..203.74)     | 1.02x ± 0.10 (0.92..1.14) |
| WhileLoop       | 402.25 ms ± 33.95 (357.43..466.77)     | 1.01x ± 0.09 (0.94..1.06) |
|                 |                                        |                           |
| Average Speedup |               (baseline)               | 1.00x ± 0.02 (0.88..1.10) |
+-----------------+----------------------------------------+---------------------------+

The raw ReBench data files are available for download here: baseline and head

Bytecode interpreter

+-----------------+----------------------------------------+---------------------------+
| Benchmark       | master (base)                          | feat/profiler (head)      |
+-----------------+----------------------------------------+---------------------------+
| Bounce          | 95.51 ms ± 11.08 (83.16..117.14)       | 1.15x ± 0.16 (1.03..1.30) |
| BubbleSort      | 120.99 ms ± 12.19 (108.61..147.57)     | 1.01x ± 0.14 (0.82..1.10) |
| DeltaBlue       | 65.25 ms ± 5.08 (56.98..72.31)         | 0.99x ± 0.09 (0.89..1.06) |
| Dispatch        | 90.80 ms ± 8.87 (81.48..112.45)        | 1.00x ± 0.14 (0.84..1.12) |
| Fannkuch        | 55.73 ms ± 6.72 (49.83..73.13)         | 1.07x ± 0.15 (0.92..1.15) |
| Fibonacci       | 160.88 ms ± 18.97 (144.72..207.95)     | 1.04x ± 0.16 (0.91..1.19) |
| FieldLoop       | 203.69 ms ± 16.16 (181.73..231.67)     | 1.14x ± 0.11 (1.01..1.19) |
| GraphSearch     | 46.22 ms ± 7.39 (36.94..57.41)         | 1.25x ± 0.28 (0.88..1.41) |
| IntegerLoop     | 158.42 ms ± 10.77 (142.91..173.95)     | 0.97x ± 0.10 (0.88..1.10) |
| JsonSmall       | 101.44 ms ± 12.07 (88.92..123.56)      | 1.05x ± 0.14 (0.92..1.18) |
| List            | 125.91 ms ± 10.52 (112.61..138.78)     | 1.06x ± 0.14 (0.95..1.24) |
| Loop            | 221.87 ms ± 31.65 (192.17..281.67)     | 1.09x ± 0.18 (0.93..1.18) |
| Mandelbrot      | 146.79 ms ± 21.56 (121.45..197.34)     | 1.18x ± 0.19 (1.04..1.27) |
| NBody           | 105.64 ms ± 12.18 (85.32..122.02)      | 1.07x ± 0.19 (0.83..1.23) |
| PageRank        | 151.08 ms ± 23.41 (127.48..205.39)     | 1.04x ± 0.19 (0.90..1.16) |
| Permute         | 134.76 ms ± 11.74 (116.67..153.48)     | 1.05x ± 0.15 (0.85..1.19) |
| Queens          | 123.56 ms ± 34.55 (102.13..218.89)     | 1.31x ± 0.39 (1.07..1.44) |
| QuickSort       | 33.89 ms ± 3.56 (31.46..43.05)         | 1.13x ± 0.14 (0.98..1.20) |
| Recurse         | 130.49 ms ± 7.81 (119.04..147.39)      | 1.03x ± 0.09 (0.93..1.11) |
| Richards        | 1731.95 ms ± 118.22 (1536.46..1931.49) | 1.05x ± 0.08 (1.01..1.12) |
| Sieve           | 206.01 ms ± 22.00 (178.85..242.34)     | 1.07x ± 0.13 (0.96..1.15) |
| Storage         | 35.11 ms ± 2.09 (32.37..38.68)         | 0.91x ± 0.11 (0.80..1.07) |
| Sum             | 78.90 ms ± 4.58 (74.85..90.63)         | 0.97x ± 0.08 (0.90..1.08) |
| Towers          | 140.70 ms ± 14.47 (121.81..163.97)     | 1.13x ± 0.14 (1.02..1.21) |
| TreeSort        | 56.75 ms ± 6.93 (51.17..74.33)         | 0.98x ± 0.24 (0.74..1.21) |
| WhileLoop       | 180.50 ms ± 8.16 (174.12..196.95)      | 0.95x ± 0.11 (0.77..1.06) |
|                 |                                        |                           |
| Average Speedup |               (baseline)               | 1.07x ± 0.03 (0.91..1.31) |
+-----------------+----------------------------------------+---------------------------+

The raw ReBench data files are available for download here: baseline and head

The benchmarks were run using ReBench v1.2.0
The statistical analysis was done using rebench-tabler v0.1.0

The source code of this benchmark runner is available as a GitHub Gist for more details about the setup