Profile-Guided Optimization (PGO) benchmark report

[zamazan4ik]

Hi!

As I have done many times before, I decided to test the Profile-Guided Optimization (PGO) technique to optimize the library performance. For reference, results for other projects are available at https://github.com/zamazan4ik/awesome-pgo . Since PGO helped a lot for many other libraries, I decided to apply it on vrl to see if the performance win (or lose) can be achieved. Here are my benchmark results.

Test environment

• Fedora 40
• Linux kernel 6.9.12
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.79.0
• vrl version: main branch on commit 16889935bf08268547dd494054dc906705a1dfc7
• Disabled Turbo boost

Benchmark

For benchmark purposes, I use built-in into the project benchmarks. For PGO optimization I use cargo-pgo tool. Release bench results I got with taskset -c 0 cargo bench --workspace --all-features command. The PGO training phase is done with taskset -c 0 cargo pgo bench -- --workspace --all-features, PGO optimization phase - with taskset -c 0 cargo pgo optimize bench -- --workspace --all-features.

taskset -c 0 is used for reducing the OS scheduler's influence on the results. All measurements are done on the same machine, with the same background "noise" (as much as I can guarantee).

Results

I got the following results:

• Release: https://gist.github.com/zamazan4ik/4c603a62548f0c4edb3038f583912151
• PGO optimized compared to Release: https://gist.github.com/zamazan4ik/71e45a0493ad75514fe3254ad0378576
• (just for reference) PGO instrumented compared to Release: https://gist.github.com/zamazan4ik/09142c8031940ace3d8ee094cdf63471

According to the results, PGO measurably improves the libraries' performance in many cases.

Further steps

I understand that the steps above can be time-consuming and hard to implement in practice. At the very least, the library's users can find this performance report and decide to enable PGO for their applications if they care about vrl performance in their workloads. Maybe a small note somewhere in the documentation will be enough to raise awareness about this work.

Thank you.

[jszwedko]

Thanks for sharing this @zamazan4ik !