Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmarking batch span processor #6

Open
TommyCpp opened this issue May 16, 2021 · 4 comments
Open

Benchmarking batch span processor #6

TommyCpp opened this issue May 16, 2021 · 4 comments

Comments

@TommyCpp
Copy link
Owner

TommyCpp commented May 16, 2021 •
edited
Loading

Baseline

It's the current implementation, using Mutex

Benchmark

Batch span processor benchmark

BatchSpanProcessor/with 1 concurrent task                        
                        time:   [736.37 us 746.43 us 758.77 us]
                        change: [-12.015% -5.8075% +0.2269%] (p = 0.08 > 0.05)
                        No change in performance detected.
Found 2 outliers among 50 measurements (4.00%)
  1 (2.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.9227 s (6375                                                                                                           BatchSpanProcessor/with 2 concurrent task                        
                        time:   [912.78 us 926.24 us 940.15 us]
                        change: [-26.831% -22.599% -18.845%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 5.1091 s (3825                                                                                                           BatchSpanProcessor/with 4 concurrent task                        
                        time:   [1.3338 ms 1.3462 ms 1.3591 ms]
                        change: [-23.494% -15.897% -8.8477%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 50 measurements (6.00%)
  2 (4.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 6.5640 s (2550                                                                                                           BatchSpanProcessor/with 8 concurrent task                        
                        time:   [2.7932 ms 2.8755 ms 2.9622 ms]
                        change: [-5.9656% -0.5466% +5.1050%] (p = 0.86 > 0.05)
                        No change in performance detected.
Found 4 outliers among 50 measurements (8.00%)
  3 (6.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 16 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 6.0s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 5.9593 s (1275                                                                                                          BatchSpanProcessor/with 16 concurrent task                        
                        time:   [4.5020 ms 4.5346 ms 4.5782 ms]
                        change: [-36.961% -29.674% -21.758%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 50 measurements (8.00%)
  1 (2.00%) high mild
  3 (6.00%) high severe
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 5.4220 s (600                                                                                                           BatchSpanProcessor/with 32 concurrent task                        
                        time:   [9.0829 ms 9.1563 ms 9.2414 ms]
                        change: [-18.375% -15.488% -12.672%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) high severe

Trace benchmark

Gnuplot not found, using plotters backend
start-end-span/always-sample                                                                             
                        time:   [3.3237 us 3.3370 us 3.3501 us]
                        change: [+517.41% +522.69% +527.56%] (p = 0.00 < 0.05)
                        Performance has regressed.
start-end-span/never-sample                                                                             
                        time:   [668.72 ns 670.24 ns 672.01 ns]
                        change: [+313.80% +316.24% +318.30%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 14 outliers among 100 measurements (14.00%)
  5 (5.00%) high mild
  9 (9.00%) high severe

Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0298 s (682k iter                                                                                                          start-end-span-4-attrs/always-sample                        
                        time:   [7.1765 us 7.2468 us 7.3096 us]
                        change: [+447.12% +454.10% +461.43%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
  2 (2.00%) low severe
  8 (8.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0016 s (11M iterat                                                                                                          start-end-span-4-attrs/never-sample                        
                        time:   [213.42 ns 214.26 ns 215.30 ns]
                        change: [+18.254% +19.204% +20.178%] (p = 0.00 < 0.05)
                        Performance has regressed.

Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0069 s (1.4M iter                                                                                                          start-end-span-8-attrs/always-sample                        
                        time:   [2.9509 us 2.9793 us 3.0101 us]
                        change: [+56.510% +58.479% +60.830%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0001 s (19M iterat                                                                                                          start-end-span-8-attrs/never-sample                        
                        time:   [232.11 ns 232.98 ns 233.95 ns]
                        change: [+5.3952% +6.3716% +7.2540%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild

Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0126 s (1.                                                                                                          start-end-span-all-attr-types/always-sample                        
                        time:   [2.4506 us 2.4971 us 2.5485 us]
                        change: [+55.970% +58.608% +61.277%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  2 (2.00%) high mild
  6 (6.00%) high severe
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0003 s (21M                                                                                                          start-end-span-all-attr-types/never-sample                        
                        time:   [220.32 ns 220.54 ns 220.77 ns]
                        change: [-0.5896% -0.3117% -0.0442%] (p = 0.03 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0129 s                                                                                                           start-end-span-all-attr-types-2x/always-sample                        
                        time:   [3.8000 us 3.8261 us 3.8486 us]
                        change: [+17.408% +19.605% +21.895%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) low mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0012 s (                                                                                                          start-end-span-all-attr-types-2x/never-sample                        
                        time:   [273.85 ns 274.30 ns 274.79 ns]
                        change: [-73.278% -73.042% -72.679%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
@TommyCpp
Copy link
Owner Author

TommyCpp commented May 16, 2021 •
edited
Loading

Async-channels

Using async-channels create's mpmc channels.

Benchmark

Batch span processor benchmark

BatchSpanProcessor/with 1 concurrent task                        
                        time:   [874.92 us 898.70 us 925.73 us]
                        change: [+11.302% +18.913% +31.195%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 50 measurements (6.00%)
  2 (4.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.4168 s (6375                                                                                                           BatchSpanProcessor/with 2 concurrent task                        
                        time:   [750.50 us 767.57 us 781.88 us]
                        change: [-25.132% -21.966% -18.419%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 6.2837 s (5100                                                                                                           BatchSpanProcessor/with 4 concurrent task                        
                        time:   [926.02 us 1.0427 ms 1.1733 ms]
                        change: [-32.265% -27.603% -21.226%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
  2 (4.00%) high mild
  6 (12.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 5.4831 s (3825                                                                                                           BatchSpanProcessor/with 8 concurrent task                        
                        time:   [1.4492 ms 1.4608 ms 1.4696 ms]
                        change: [-51.020% -48.392% -45.599%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.3568 s (2550                                                                                                          BatchSpanProcessor/with 16 concurrent task                        
                        time:   [2.4467 ms 2.4725 ms 2.5012 ms]
                        change: [-48.672% -47.217% -45.926%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) low mild
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 5.8s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 5.7869 s (1275                                                                                                          BatchSpanProcessor/with 32 concurrent task                        
                        time:   [4.5102 ms 4.5577 ms 4.6029 ms]
                        change: [-51.147% -50.440% -49.718%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high mild

Trace benchmark

start-end-span/always-sample                                                                             
                        time:   [470.40 ns 474.71 ns 480.32 ns]
                        change: [-86.105% -85.948% -85.778%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 17 outliers among 100 measurements (17.00%)
  6 (6.00%) low severe
  7 (7.00%) low mild
  4 (4.00%) high severe
start-end-span/never-sample                                                                            
                        time:   [168.75 ns 169.04 ns 169.36 ns]
                        change: [-74.908% -74.832% -74.752%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low severe
  8 (8.00%) low mild

Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0019 s (4.4M iter                                                                                                          start-end-span-4-attrs/always-sample                        
                        time:   [1.1569 us 1.1708 us 1.1842 us]
                        change: [-84.375% -84.118% -83.883%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0005 s (26M iterat                                                                                                          start-end-span-4-attrs/never-sample                        
                        time:   [193.12 ns 193.64 ns 194.18 ns]
                        change: [-14.138% -13.368% -12.622%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 18 outliers among 100 measurements (18.00%)
  3 (3.00%) low severe
  10 (10.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe

Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0014 s (2.9M iter                                                                                                          start-end-span-8-attrs/always-sample                        
                        time:   [1.7045 us 1.7198 us 1.7343 us]
                        change: [-44.939% -44.101% -43.218%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low severe
  5 (5.00%) low mild
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0001 s (21M iterat                                                                                                          start-end-span-8-attrs/never-sample                        
                        time:   [231.13 ns 231.88 ns 232.99 ns]
                        change: [-4.0637% -3.0699% -1.8872%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  7 (7.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high mild
  4 (4.00%) high severe

Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0061 s (3.                                                                                                          start-end-span-all-attr-types/always-sample                        
                        time:   [1.5574 us 1.5925 us 1.6251 us]
                        change: [-39.649% -38.382% -37.079%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0007 s (23M                                                                                                          start-end-span-all-attr-types/never-sample                        
                        time:   [213.96 ns 214.78 ns 215.60 ns]
                        change: [-3.8833% -3.5341% -3.1886%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 22 outliers among 100 measurements (22.00%)
  6 (6.00%) low severe
  3 (3.00%) low mild
  7 (7.00%) high mild
  6 (6.00%) high severe

Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0003 s                                                                                                           start-end-span-all-attr-types-2x/always-sample                        
                        time:   [2.7983 us 2.8120 us 2.8255 us]
                        change: [-26.867% -25.453% -23.972%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) low severe
  4 (4.00%) low mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0011 s (                                                                                                          start-end-span-all-attr-types-2x/never-sample                        
                        time:   [273.13 ns 273.86 ns 274.60 ns]
                        change: [-2.9915% -1.7610% -0.8519%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) low mild
  2 (2.00%) high mild

@TommyCpp
Copy link
Owner Author

TommyCpp commented May 16, 2021 •
edited
Loading

Relay message via thread

This implementation uses an unbounded crossbeam channel to send a message to a dedicated thread, where the thread then relays the message to the exporter in async.

Benchmark

Batch span processor benchmark

BatchSpanProcessor/with 1 concurrent task                        
                        time:   [563.76 us 589.51 us 618.06 us]
                        change: [-30.874% -26.045% -20.092%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 50 measurements (12.00%)
  2 (4.00%) low mild
  2 (4.00%) high mild
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.6841 s (7650                                                                                                           BatchSpanProcessor/with 2 concurrent task                        
                        time:   [717.01 us 719.22 us 722.06 us]
                        change: [-26.555% -23.393% -20.053%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
  1 (2.00%) low mild
  2 (4.00%) high mild
  5 (10.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 5.9721 s (6375                                                                                                           BatchSpanProcessor/with 4 concurrent task                        
                        time:   [960.43 us 965.29 us 970.95 us]
                        change: [-30.879% -28.904% -26.468%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  1 (2.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 6.2065 s (3825                                                                                                           BatchSpanProcessor/with 8 concurrent task                        
                        time:   [1.6212 ms 1.6277 ms 1.6350 ms]
                        change: [-45.144% -42.513% -39.689%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 50 measurements (10.00%)
  1 (2.00%) low mild
  2 (4.00%) high mild
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.5828 s (2550                                                                                                          BatchSpanProcessor/with 16 concurrent task                        
                        time:   [2.5997 ms 2.6474 ms 2.7094 ms]
                        change: [-42.897% -41.019% -39.159%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 6.6s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 6.5905 s (1275                                                                                                          BatchSpanProcessor/with 32 concurrent task                        
                        time:   [5.5081 ms 5.5455 ms 5.5788 ms]
                        change: [-41.664% -40.795% -39.963%] (p = 0.00 < 0.05)
                        Performance has improved.

Trace benchmark

start-end-span/always-sample                                                                            
                        time:   [481.14 ns 485.30 ns 489.27 ns]
                        change: [-86.535% -86.248% -85.973%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low severe
  9 (9.00%) low mild
start-end-span/never-sample                                                                            
                        time:   [176.38 ns 176.95 ns 177.47 ns]
                        change: [-73.528% -73.442% -73.358%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild

Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0025 s (2.7M iter                                                                                                          start-end-span-4-attrs/always-sample                        
                        time:   [1.5561 us 1.5714 us 1.5870 us]
                        change: [-78.917% -78.591% -78.258%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0008 s (23M iterat                                                                                                          start-end-span-4-attrs/never-sample                        
                        time:   [199.82 ns 200.00 ns 200.21 ns]
                        change: [-10.304% -9.5578% -8.8123%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) low mild
  1 (1.00%) high mild

Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0051 s (2.0M iter                                                                                                          start-end-span-8-attrs/always-sample                        
                        time:   [2.3176 us 2.3313 us 2.3459 us]
                        change: [-23.575% -22.399% -21.305%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high severe
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0005 s (21M iterat                                                                                                          start-end-span-8-attrs/never-sample                        
                        time:   [219.61 ns 220.01 ns 220.42 ns]
                        change: [-8.8478% -8.0600% -7.2765%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0018 s (2.                                                                                                          start-end-span-all-attr-types/always-sample                        
                        time:   [1.8282 us 1.8375 us 1.8463 us]
                        change: [-29.704% -28.325% -26.824%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) low severe
  6 (6.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0001 s (21M                                                                                                          start-end-span-all-attr-types/never-sample                        
                        time:   [223.30 ns 223.66 ns 224.02 ns]
                        change: [+0.9638% +1.1981% +1.4397%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0172 s                                                                                                           start-end-span-all-attr-types-2x/always-sample                        
                        time:   [3.1789 us 3.1948 us 3.2104 us]
                        change: [-14.686% -13.092% -11.468%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0009 s (                                                                                                          start-end-span-all-attr-types-2x/never-sample                        
                        time:   [257.17 ns 257.75 ns 258.42 ns]
                        change: [-7.4291% -6.1687% -5.1378%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe

@TommyCpp
Copy link
Owner Author

TommyCpp commented May 16, 2021 •
edited
Loading

Runtime channel(Tokio)

This implementation uses tokio's mpsc channel.

Benchmark

Batch span processor benchmark

BatchSpanProcessor/with 1 concurrent task                        
                        time:   [763.66 us 777.63 us 794.86 us]
                        change: [-5.5082% -0.8943% +3.4321%] (p = 0.72 > 0.05)
                        No change in performance detected.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.1869 s (7650                                                                                                           BatchSpanProcessor/with 2 concurrent task                        
                        time:   [683.31 us 689.33 us 695.83 us]
                        change: [-30.661% -28.072% -25.529%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
  1 (2.00%) low severe
  3 (6.00%) low mild
  2 (4.00%) high mild
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 6.0023 s (6375                                                                                                           BatchSpanProcessor/with 4 concurrent task                        
                        time:   [872.21 us 877.92 us 884.16 us]
                        change: [-35.918% -32.276% -27.417%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 50 measurements (12.00%)
  1 (2.00%) low mild
  2 (4.00%) high mild
  3 (6.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 5.6743 s (3825                                                                                                           BatchSpanProcessor/with 8 concurrent task                        
                        time:   [1.4650 ms 1.4771 ms 1.4894 ms]
                        change: [-50.676% -48.612% -46.793%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 50 measurements (6.00%)
  3 (6.00%) high mild
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.7458 s (2550                                                                                                          BatchSpanProcessor/with 16 concurrent task                        
                        time:   [2.6165 ms 2.6337 ms 2.6510 ms]
                        change: [-45.358% -43.865% -42.602%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high mild
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 6.8s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 6.7980 s (1275                                                                                                          BatchSpanProcessor/with 32 concurrent task                        
                        time:   [5.1859 ms 5.2923 ms 5.4093 ms]
                        change: [-43.492% -42.330% -41.255%] (p = 0.00 < 0.05)
                        Performance has improved.

Trace benchmark

start-end-span/always-sample                                                                            
                        time:   [439.83 ns 444.14 ns 448.29 ns]
                        change: [-86.724% -86.631% -86.544%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) low severe
  3 (3.00%) low mild
start-end-span/never-sample                                                                            
                        time:   [174.11 ns 174.33 ns 174.54 ns]
                        change: [-74.100% -74.022% -73.949%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low severe
  4 (4.00%) low mild
  1 (1.00%) high mild

Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0005 s (4.1M iter                                                                                                          start-end-span-4-attrs/always-sample                        
                        time:   [1.1029 us 1.1181 us 1.1347 us]
                        change: [-84.832% -84.602% -84.372%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0001 s (27M iterat                                                                                                          start-end-span-4-attrs/never-sample                        
                        time:   [187.30 ns 187.55 ns 187.77 ns]
                        change: [-16.726% -15.987% -15.251%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
  5 (5.00%) low severe
  10 (10.00%) low mild

Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0036 s (2.7M iter                                                                                                          start-end-span-8-attrs/always-sample                        
                        time:   [1.7884 us 1.8112 us 1.8354 us]
                        change: [-42.353% -41.313% -40.264%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
  8 (8.00%) high mild
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0006 s (22M iterat                                                                                                          start-end-span-8-attrs/never-sample                        
                        time:   [221.78 ns 222.11 ns 222.41 ns]
                        change: [-8.7823% -7.9412% -7.1224%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low severe
  4 (4.00%) low mild

Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0038 s (2.                                                                                                          start-end-span-all-attr-types/always-sample                        
                        time:   [1.6476 us 1.6681 us 1.6882 us]
                        change: [-36.034% -34.687% -33.448%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0004 s (25M                                                                                                          start-end-span-all-attr-types/never-sample                        
                        time:   [212.80 ns 213.34 ns 213.90 ns]
                        change: [-6.0818% -5.3116% -4.4650%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0070 s                                                                                                           start-end-span-all-attr-types-2x/always-sample                        
                        time:   [2.9687 us 2.9791 us 2.9889 us]
                        change: [-22.268% -20.748% -19.142%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  6 (6.00%) low severe
  3 (3.00%) low mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0008 s (                                                                                                          start-end-span-all-attr-types-2x/never-sample                        
                        time:   [265.47 ns 266.35 ns 267.49 ns]
                        change: [-5.3977% -4.1198% -3.1305%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  2 (2.00%) low severe
  8 (8.00%) low mild

@TommyCpp
Copy link
Owner Author

TommyCpp commented May 17, 2021 •
edited
Loading

Benchmark Baseline(Current implementation) mpmc channel new thread Tokio channel
BSP/1 task 746.43us 898.70us(+18%) 589.51us(-26%) 777.63us(-0.8%)
BSP/2 tasks 926.24us 767.57us(-21%) 719us(-23%) 689.33us(-28.072%)
BSP/4 tasks 1.3ms 1.04ms(-27%) 965us(-28%) 877.92us(-32.27%)
BSP/8 tasks 2.8ms 1.46ms(-48%) 1.62ms(-42%) 1.47ms(-48%)
BSP/16 tasks 4.5ms 2.47ms(-47%) 2.64ms(-41%) 2.6337ms(-43%)
BSP/32 tasks 9.1ms 4.5ms(-50%) 5.54ms(-40%) 5.2923ms(-42%)
start-end-span/always-sample 3.37us 474.71ns(-85%) 485.30ns(-86%) 444.14ns(-86%)
start-end-span/never-sample 670.24ns 169.04ns(-74%) 176.95ns(-73%) 174.33ns(-74%)
start-end-span-4-attrs/always-sample 7.24us 1.1708us(-84%) 1.57us(-78%) 1.1181us(-84%)
start-end-span-4-attrs/never-sample 214.26ns 193.64ns(-13%) 200.00ns(-9%) 187.55ns(-15.987%)
start-end-span-8-attrs/always-sample 2.97us 1.71us(-44%) 2.33us(-22%) 1.81us(-41.313%)
start-end-span-8-attrs/never-sample 232.98ns 231.88ns(-3.06%) 220.01ns(-8.06%) 222.11ns(-7.9%)
start-end-span-all-attr-types/always-sample 2.49us 1.5us(-38%) 1.8375us(-28.325%) 1.6681us(-34%)
start-end-span-all-attr-types/never-sample 220.54ns 214ns(-3%) 223.66ns(+1.19%) 213.34ns(-5.311%)
start-end-span-all-attr-types-2x/always-sample 3.82us 2.81us(-25%) 3.1948us(-13.092%) 2.9791us(-20%)
start-end-span-all-attr-types-2x/never-sample 274.30ns 273.86ns(-1.76%) 257.74ns(-6.1%) 266.35ns(-4.1198%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@TommyCpp