Skip to content
Benjamin Zaitlen edited this page Dec 13, 2019 · 8 revisions

Dask + cuDF + UCX MErge Benchmarking

2 GPUs

With UCX (NVLink & ib Enabled; dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 1,2
==========================
Total time  | 1.51 s
Total time  | 1.71 s
Total time  | 1.16 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(01,02)     | 8.98 GB/s 11.46 GB/s 19.07 GB/s (19.20 GB)
(02,01)     | 9.30 GB/s 11.52 GB/s 18.87 GB/s (19.20 GB)

With UCX (Only NVLink Enabled; dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 1,2
==========================
Total time  | 1.23 s
Total time  | 1.22 s
Total time  | 1.14 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(01,02)     | 20.42 GB/s 21.08 GB/s 22.13 GB/s (19.20 GB)
(02,01)     | 20.04 GB/s 22.05 GB/s 22.51 GB/s (19.20 GB)

With TCP (dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | tcp
Device(s)   | 1,2
==========================
Total time  | 52.86 s
Total time  | 64.20 s
Total time  | 54.20 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(01,02)     | 89.30 MB/s 118.65 MB/s 141.55 MB/s (21.60 GB)
(02,01)     | 81.78 MB/s 96.84 MB/s 141.15 MB/s (21.60 GB)

4 GPUs

With UCX (NVLink & ib Enabled; dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 0,1,2,3
==========================
Total time  | 2.97 s
Total time  | 2.89 s
Total time  | 2.77 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(00,01)     | 1.39 GB/s 1.51 GB/s 2.94 GB/s (9.60 GB)
(00,02)     | 1.34 GB/s 1.55 GB/s 2.67 GB/s (9.60 GB)
(00,03)     | 1.44 GB/s 1.84 GB/s 3.42 GB/s (9.60 GB)
(01,00)     | 1.38 GB/s 1.50 GB/s 2.70 GB/s (9.60 GB)
(01,02)     | 1.51 GB/s 1.80 GB/s 3.51 GB/s (12.00 GB)
(01,03)     | 1.59 GB/s 1.71 GB/s 1.99 GB/s (7.20 GB)
(02,00)     | 1.18 GB/s 1.50 GB/s 1.71 GB/s (7.20 GB)
(02,01)     | 1.32 GB/s 1.46 GB/s 3.05 GB/s (9.60 GB)
(02,03)     | 1.37 GB/s 1.63 GB/s 3.72 GB/s (12.00 GB)
(03,00)     | 1.39 GB/s 1.81 GB/s 3.91 GB/s (12.00 GB)
(03,01)     | 1.42 GB/s 1.56 GB/s 3.11 GB/s (9.60 GB)
(03,02)     | 1.33 GB/s 1.60 GB/s 2.99 GB/s (7.20 GB)

With UCX (Only NVLink Enabled; dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 0,1,2,3
==========================
Total time  | 1.81 s
Total time  | 1.68 s
Total time  | 1.67 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(00,01)     | 3.63 GB/s 3.88 GB/s 4.61 GB/s (12.00 GB)
(00,02)     | 3.51 GB/s 3.91 GB/s 4.46 GB/s (7.20 GB)
(00,03)     | 3.91 GB/s 4.13 GB/s 4.72 GB/s (9.60 GB)
(01,00)     | 3.64 GB/s 4.14 GB/s 8.24 GB/s (14.40 GB)
(01,02)     | 3.84 GB/s 4.18 GB/s 4.44 GB/s (7.20 GB)
(01,03)     | 3.85 GB/s 3.88 GB/s 4.46 GB/s (7.20 GB)
(02,00)     | 2.53 GB/s 3.80 GB/s 6.82 GB/s (7.20 GB)
(02,01)     | 3.88 GB/s 4.21 GB/s 4.85 GB/s (9.60 GB)
(02,03)     | 3.63 GB/s 4.24 GB/s 5.05 GB/s (12.00 GB)
(03,00)     | 4.02 GB/s 4.38 GB/s 4.66 GB/s (7.20 GB)
(03,01)     | 3.55 GB/s 3.86 GB/s 4.23 GB/s (7.20 GB)
(03,02)     | 3.77 GB/s 3.98 GB/s 10.58 GB/s (14.40 GB)

With TCP (dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | tcp
Device(s)   | 0,1,2,3
==========================
Total time  | 65.14 s
Total time  | 88.68 s
Total time  | 90.60 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(00,01)     | 48.76 MB/s 56.70 MB/s 57.42 MB/s (9.60 GB)
(00,02)     | 46.50 MB/s 56.02 MB/s 77.26 MB/s (9.30 GB)
(00,03)     | 61.93 MB/s 65.69 MB/s 76.17 MB/s (6.90 GB)
(01,00)     | 39.59 MB/s 46.15 MB/s 53.39 MB/s (7.90 GB)
(01,02)     | 51.95 MB/s 56.68 MB/s 81.24 MB/s (8.60 GB)
(01,03)     | 45.64 MB/s 61.88 MB/s 93.96 MB/s (12.00 GB)
(02,00)     | 36.81 MB/s 43.89 MB/s 61.16 MB/s (9.60 GB)
(02,01)     | 49.12 MB/s 60.38 MB/s 105.98 MB/s (11.00 GB)
(02,03)     | 49.67 MB/s 55.76 MB/s 74.83 MB/s (10.30 GB)
(03,00)     | 51.42 MB/s 64.27 MB/s 82.52 MB/s (11.70 GB)
(03,01)     | 41.32 MB/s 53.54 MB/s 75.44 MB/s (11.00 GB)
(03,02)     | 46.47 MB/s 58.26 MB/s 98.87 MB/s (10.00 GB)

8 GPUs

With UCX (NVLink & ib Enabled; dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 0,1,2,3,4,5,6,7
==========================
Total time  | 12.24 s
Total time  | 11.33 s
Total time  | 9.17 s
Total time  | 8.92 s
Total time  | 5.95 s
Total time  | 5.20 s
Total time  | 5.24 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(00,01)     | 170.87 MB/s 268.29 MB/s 369.26 MB/s (8.75 GB)
(00,02)     | 159.53 MB/s 263.56 MB/s 341.53 MB/s (9.10 GB)
(00,03)     | 171.01 MB/s 265.22 MB/s 403.75 MB/s (11.85 GB)
(00,04)     | 181.48 MB/s 254.46 MB/s 357.78 MB/s (11.50 GB)
(00,05)     | 171.15 MB/s 269.89 MB/s 392.40 MB/s (11.50 GB)
(00,06)     | 167.22 MB/s 289.16 MB/s 373.49 MB/s (13.55 GB)
(00,07)     | 167.07 MB/s 291.28 MB/s 402.55 MB/s (10.80 GB)
(01,00)     | 152.43 MB/s 242.44 MB/s 458.86 MB/s (9.10 GB)
(01,02)     | 163.07 MB/s 240.60 MB/s 355.73 MB/s (9.10 GB)
(01,03)     | 153.51 MB/s 263.49 MB/s 476.21 MB/s (17.35 GB)
(01,04)     | 202.00 MB/s 292.19 MB/s 365.60 MB/s (10.15 GB)
(01,05)     | 157.53 MB/s 241.64 MB/s 364.72 MB/s (11.85 GB)
(01,06)     | 162.38 MB/s 239.02 MB/s 342.63 MB/s (11.50 GB)
(01,07)     | 176.97 MB/s 322.70 MB/s 801.58 MB/s (14.25 GB)
(02,00)     | 127.99 MB/s 267.93 MB/s 459.03 MB/s (13.20 GB)
(02,01)     | 136.31 MB/s 244.81 MB/s 348.59 MB/s (9.10 GB)
(02,03)     | 138.94 MB/s 245.86 MB/s 498.27 MB/s (7.70 GB)
(02,04)     | 191.54 MB/s 248.28 MB/s 623.21 MB/s (11.15 GB)
(02,05)     | 138.30 MB/s 241.75 MB/s 550.28 MB/s (10.80 GB)
(02,06)     | 140.61 MB/s 246.98 MB/s 449.69 MB/s (13.90 GB)
(02,07)     | 147.66 MB/s 279.35 MB/s 340.71 MB/s (11.15 GB)
(03,00)     | 147.57 MB/s 269.69 MB/s 402.41 MB/s (9.80 GB)
(03,01)     | 177.81 MB/s 279.29 MB/s 886.40 MB/s (17.00 GB)
(03,02)     | 159.23 MB/s 244.44 MB/s 328.62 MB/s (11.85 GB)
(03,04)     | 152.27 MB/s 299.56 MB/s 487.58 MB/s (11.85 GB)
(03,05)     | 152.83 MB/s 287.98 MB/s 435.10 MB/s (14.25 GB)
(03,06)     | 151.87 MB/s 298.05 MB/s 397.76 MB/s (9.45 GB)
(03,07)     | 123.38 MB/s 240.92 MB/s 381.74 MB/s (9.45 GB)
(04,00)     | 197.14 MB/s 280.78 MB/s 430.51 MB/s (12.55 GB)
(04,01)     | 179.70 MB/s 292.00 MB/s 569.28 MB/s (9.80 GB)
(04,02)     | 174.79 MB/s 322.72 MB/s 540.76 MB/s (11.85 GB)
(04,03)     | 198.26 MB/s 322.06 MB/s 661.30 MB/s (16.30 GB)
(04,05)     | 161.50 MB/s 289.76 MB/s 557.92 MB/s (11.85 GB)
(04,06)     | 182.68 MB/s 236.98 MB/s 347.70 MB/s (13.90 GB)
(04,07)     | 149.81 MB/s 254.47 MB/s 389.29 MB/s (9.45 GB)
(05,00)     | 165.85 MB/s 263.07 MB/s 541.80 MB/s (9.45 GB)
(05,01)     | 149.32 MB/s 309.41 MB/s 657.16 MB/s (14.60 GB)
(05,02)     | 125.34 MB/s 297.96 MB/s 566.81 MB/s (8.75 GB)
(05,03)     | 201.06 MB/s 304.08 MB/s 722.50 MB/s (11.15 GB)
(05,04)     | 135.99 MB/s 282.90 MB/s 553.04 MB/s (9.80 GB)
(05,06)     | 147.04 MB/s 271.71 MB/s 545.92 MB/s (11.50 GB)
(05,07)     | 164.19 MB/s 314.66 MB/s 616.88 MB/s (14.60 GB)
(06,00)     | 184.96 MB/s 293.26 MB/s 543.58 MB/s (9.80 GB)
(06,01)     | 172.11 MB/s 268.53 MB/s 528.09 MB/s (11.85 GB)
(06,02)     | 174.05 MB/s 210.60 MB/s 362.06 MB/s (9.45 GB)
(06,03)     | 186.72 MB/s 333.26 MB/s 651.96 MB/s (9.45 GB)
(06,04)     | 196.32 MB/s 274.06 MB/s 508.44 MB/s (14.95 GB)
(06,05)     | 184.75 MB/s 277.72 MB/s 459.42 MB/s (17.00 GB)
(06,07)     | 160.21 MB/s 299.32 MB/s 446.58 MB/s (12.20 GB)
(07,00)     | 150.14 MB/s 264.18 MB/s 519.00 MB/s (13.90 GB)
(07,01)     | 128.83 MB/s 292.49 MB/s 526.69 MB/s (9.45 GB)
(07,02)     | 137.73 MB/s 300.03 MB/s 773.75 MB/s (10.80 GB)
(07,03)     | 136.52 MB/s 227.20 MB/s 363.01 MB/s (9.10 GB)
(07,04)     | 155.90 MB/s 255.37 MB/s 488.56 MB/s (11.15 GB)
(07,05)     | 123.74 MB/s 254.76 MB/s 365.85 MB/s (8.75 GB)
(07,06)     | 137.92 MB/s 268.29 MB/s 479.20 MB/s (13.55 GB)

With TCP (dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | tcp
Device(s)   | 0,1,2,3,4,5,6,7
==========================
Total time  | 86.69 s
Total time  | 101.35 s
Total time  | 108.21 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(00,01)     | 19.59 MB/s 26.60 MB/s 35.07 MB/s (3.95 GB)
(00,02)     | 24.35 MB/s 34.26 MB/s 53.47 MB/s (8.75 GB)
(00,03)     | 26.40 MB/s 31.16 MB/s 43.46 MB/s (6.35 GB)
(00,04)     | 24.12 MB/s 29.29 MB/s 35.10 MB/s (3.60 GB)
(00,05)     | 19.22 MB/s 22.44 MB/s 46.70 MB/s (3.60 GB)
(00,06)     | 26.06 MB/s 29.12 MB/s 34.80 MB/s (3.95 GB)
(00,07)     | 17.63 MB/s 25.02 MB/s 41.12 MB/s (8.75 GB)
(01,00)     | 26.31 MB/s 31.75 MB/s 41.44 MB/s (3.95 GB)
(01,02)     | 22.20 MB/s 28.00 MB/s 35.52 MB/s (3.95 GB)
(01,03)     | 20.61 MB/s 28.88 MB/s 38.56 MB/s (4.30 GB)
(01,04)     | 23.38 MB/s 30.86 MB/s 35.09 MB/s (3.95 GB)
(01,05)     | 19.21 MB/s 21.89 MB/s 34.56 MB/s (6.70 GB)
(01,06)     | 22.67 MB/s 24.49 MB/s 28.53 MB/s (6.35 GB)
(01,07)     | 20.78 MB/s 26.22 MB/s 36.89 MB/s (6.70 GB)
(02,00)     | 23.70 MB/s 28.09 MB/s 57.90 MB/s (4.65 GB)
(02,01)     | 32.89 MB/s 60.63 MB/s 79.73 MB/s (9.45 GB)
(02,03)     | 27.97 MB/s 29.65 MB/s 36.55 MB/s (3.95 GB)
(02,04)     | 20.64 MB/s 26.73 MB/s 33.34 MB/s (4.65 GB)
(02,05)     | 19.88 MB/s 35.94 MB/s 55.60 MB/s (4.30 GB)
(02,06)     | 27.13 MB/s 29.68 MB/s 56.03 MB/s (4.30 GB)
(02,07)     | 19.35 MB/s 29.75 MB/s 30.75 MB/s (3.95 GB)
(03,00)     | 25.92 MB/s 32.77 MB/s 42.75 MB/s (3.10 GB)
(03,01)     | 21.35 MB/s 31.39 MB/s 43.57 MB/s (5.65 GB)
(03,02)     | 20.29 MB/s 32.34 MB/s 38.91 MB/s (4.15 GB)
(03,04)     | 35.25 MB/s 59.73 MB/s 60.46 MB/s (3.45 GB)
(03,05)     | 26.99 MB/s 34.27 MB/s 50.60 MB/s (3.45 GB)
(03,06)     | 20.03 MB/s 37.08 MB/s 42.02 MB/s (3.80 GB)
(03,07)     | 43.82 MB/s 53.66 MB/s 82.59 MB/s (9.40 GB)
(04,00)     | 26.70 MB/s 37.78 MB/s 48.50 MB/s (7.05 GB)
(04,01)     | 20.01 MB/s 25.75 MB/s 54.05 MB/s (3.95 GB)
(04,02)     | 17.01 MB/s 30.15 MB/s 38.18 MB/s (4.30 GB)
(04,03)     | 19.65 MB/s 24.22 MB/s 31.63 MB/s (4.30 GB)
(04,05)     | 30.06 MB/s 45.21 MB/s 60.45 MB/s (9.45 GB)
(04,06)     | 22.11 MB/s 30.65 MB/s 37.39 MB/s (3.95 GB)
(04,07)     | 21.77 MB/s 27.99 MB/s 34.59 MB/s (4.30 GB)
(05,00)     | 18.43 MB/s 21.65 MB/s 28.90 MB/s (3.95 GB)
(05,01)     | 16.39 MB/s 24.25 MB/s 27.77 MB/s (3.95 GB)
(05,02)     | 19.00 MB/s 27.80 MB/s 41.88 MB/s (6.70 GB)
(05,03)     | 30.40 MB/s 33.86 MB/s 63.56 MB/s (4.30 GB)
(05,04)     | 32.38 MB/s 38.12 MB/s 49.22 MB/s (4.65 GB)
(05,06)     | 25.84 MB/s 44.11 MB/s 59.88 MB/s (7.05 GB)
(05,07)     | 17.45 MB/s 43.34 MB/s 55.06 MB/s (6.35 GB)
(06,00)     | 22.80 MB/s 29.27 MB/s 56.39 MB/s (5.70 GB)
(06,01)     | 20.88 MB/s 33.09 MB/s 46.56 MB/s (5.70 GB)
(06,02)     | 22.63 MB/s 33.25 MB/s 41.93 MB/s (7.40 GB)
(06,03)     | 18.70 MB/s 27.05 MB/s 33.61 MB/s (3.95 GB)
(06,04)     | 25.33 MB/s 35.18 MB/s 61.78 MB/s (10.50 GB)
(06,05)     | 24.70 MB/s 29.76 MB/s 42.74 MB/s (6.35 GB)
(06,07)     | 21.52 MB/s 26.05 MB/s 38.47 MB/s (3.95 GB)
(07,00)     | 16.39 MB/s 23.22 MB/s 33.17 MB/s (3.80 GB)
(07,01)     | 15.76 MB/s 23.19 MB/s 33.79 MB/s (4.65 GB)
(07,02)     | 20.98 MB/s 28.23 MB/s 41.44 MB/s (4.65 GB)
(07,03)     | 24.76 MB/s 52.00 MB/s 56.94 MB/s (5.15 GB)
(07,04)     | 18.56 MB/s 25.99 MB/s 58.61 MB/s (4.15 GB)
(07,05)     | 18.20 MB/s 43.64 MB/s 54.24 MB/s (4.15 GB)
(07,06)     | 20.18 MB/s 31.06 MB/s 36.45 MB/s (4.15 GB)

With UCX (Only NVLink Enabled; dgx14):

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 0,1,2,3,4,5,6,7
==========================
Total time  | 28.27 s
Total time  | 28.54 s
Total time  | 28.79 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(00,01)     | 254.00 MB/s 513.77 MB/s 838.71 MB/s (2.80 GB)
(00,02)     | 266.62 MB/s 302.47 MB/s 557.03 MB/s (5.55 GB)
(00,03)     | 334.01 MB/s 435.66 MB/s 4.36 GB/s (5.20 GB)
(00,04)     | 98.57 MB/s 124.89 MB/s 479.46 MB/s (5.55 GB)
(00,05)     | 82.38 MB/s 87.01 MB/s 97.09 MB/s (2.45 GB)
(00,06)     | 69.51 MB/s 74.75 MB/s 199.61 MB/s (4.15 GB)
(00,07)     | 83.66 MB/s 86.23 MB/s 109.52 MB/s (2.45 GB)
(01,00)     | 288.21 MB/s 526.00 MB/s 1.22 GB/s (3.50 GB)
(01,02)     | 192.12 MB/s 421.93 MB/s 726.91 MB/s (2.80 GB)
(01,03)     | 140.04 MB/s 265.66 MB/s 357.55 MB/s (5.20 GB)
(01,04)     | 46.03 MB/s 56.02 MB/s 108.98 MB/s (1.40 GB)
(01,05)     | 112.66 MB/s 146.25 MB/s 161.14 MB/s (2.10 GB)
(01,06)     | 56.56 MB/s 83.10 MB/s 101.28 MB/s (2.45 GB)
(01,07)     | 51.98 MB/s 61.95 MB/s 140.31 MB/s (4.50 GB)
(02,00)     | 147.37 MB/s 263.72 MB/s 536.50 MB/s (2.80 GB)
(02,01)     | 216.53 MB/s 416.35 MB/s 635.28 MB/s (3.15 GB)
(02,03)     | 247.11 MB/s 469.44 MB/s 844.11 MB/s (2.80 GB)
(02,04)     | 68.20 MB/s 109.48 MB/s 168.20 MB/s (6.90 GB)
(02,05)     | 73.38 MB/s 79.80 MB/s 100.35 MB/s (2.45 GB)
(02,06)     | 136.05 MB/s 141.95 MB/s 146.95 MB/s (4.15 GB)
(02,07)     | 69.77 MB/s 105.93 MB/s 112.89 MB/s (4.50 GB)
(03,00)     | 484.21 MB/s 1.50 GB/s 5.52 GB/s (8.30 GB)
(03,01)     | 283.58 MB/s 552.94 MB/s 1.31 GB/s (3.50 GB)
(03,02)     | 216.72 MB/s 647.79 MB/s 961.33 MB/s (4.20 GB)
(03,04)     | 59.10 MB/s 88.74 MB/s 113.32 MB/s (3.15 GB)
(03,05)     | 63.16 MB/s 75.99 MB/s 86.53 MB/s (2.80 GB)
(03,06)     | 49.91 MB/s 98.08 MB/s 130.07 MB/s (5.20 GB)
(03,07)     | 86.16 MB/s 118.22 MB/s 144.84 MB/s (3.85 GB)
(04,00)     | 77.73 MB/s 128.88 MB/s 193.89 MB/s (3.15 GB)
(04,01)     | 71.80 MB/s 75.98 MB/s 78.97 MB/s (1.75 GB)
(04,02)     | 47.95 MB/s 70.25 MB/s 89.96 MB/s (2.10 GB)
(04,03)     | 56.97 MB/s 76.66 MB/s 89.15 MB/s (2.45 GB)
(04,05)     | 186.71 MB/s 2.43 GB/s 4.69 GB/s (6.55 GB)
(04,06)     | 155.27 MB/s 190.28 MB/s 524.52 MB/s (2.80 GB)
(04,07)     | 163.34 MB/s 324.46 MB/s 1.35 GB/s (4.85 GB)
(05,00)     | 89.56 MB/s 98.86 MB/s 112.51 MB/s (2.45 GB)
(05,01)     | 97.85 MB/s 129.70 MB/s 253.26 MB/s (2.80 GB)
(05,02)     | 85.63 MB/s 95.64 MB/s 141.05 MB/s (2.45 GB)
(05,03)     | 84.74 MB/s 95.96 MB/s 111.97 MB/s (5.20 GB)
(05,04)     | 149.25 MB/s 181.48 MB/s 271.61 MB/s (2.80 GB)
(05,06)     | 109.16 MB/s 206.66 MB/s 361.80 MB/s (4.15 GB)
(05,07)     | 140.37 MB/s 163.69 MB/s 257.51 MB/s (2.80 GB)
(06,00)     | 60.86 MB/s 69.51 MB/s 89.35 MB/s (2.80 GB)
(06,01)     | 64.78 MB/s 72.00 MB/s 107.79 MB/s (3.15 GB)
(06,02)     | 92.29 MB/s 105.28 MB/s 213.23 MB/s (3.15 GB)
(06,03)     | 74.23 MB/s 103.20 MB/s 130.14 MB/s (7.25 GB)
(06,04)     | 174.92 MB/s 300.91 MB/s 562.39 MB/s (3.15 GB)
(06,05)     | 189.45 MB/s 648.71 MB/s 4.74 GB/s (5.55 GB)
(06,07)     | 115.96 MB/s 300.81 MB/s 430.57 MB/s (3.15 GB)
(07,00)     | 62.87 MB/s 90.63 MB/s 94.57 MB/s (2.45 GB)
(07,01)     | 68.59 MB/s 75.21 MB/s 101.45 MB/s (4.85 GB)
(07,02)     | 82.95 MB/s 90.17 MB/s 120.03 MB/s (2.80 GB)
(07,03)     | 109.30 MB/s 120.74 MB/s 140.04 MB/s (3.50 GB)
(07,04)     | 128.11 MB/s 191.59 MB/s 384.80 MB/s (2.10 GB)
(07,05)     | 142.40 MB/s 152.86 MB/s 664.38 MB/s (1.75 GB)
(07,06)     | 122.96 MB/s 756.28 MB/s 1.59 GB/s (5.20 GB)
Clone this wiki locally