observability: integrate CPU & memory profiling with Grafana Pyroscope

[BugenZhao]

Proposed by @arkbriar.

Grafana Pyroscope is an open source software project for aggregating continuous profiling data.
Grafana Pyroscope is fully integrated with Grafana allowing you to correlate with other observability signals, like metrics, logs, and traces.

Its Rust client has built-in integration with pprof-rs. We may investigate if it's possible to integrate with jeprof as well.

[arkbriar]

I'm not sure but I think it might also be possible to integrate the jemalloc profiling 🤔

[fuyufjh]

In addition, how about integrating await tree as a backend? It might be more helpful than CPU profiling especially for IO-intensive cases.

https://github.com/grafana/pyroscope-rs/blob/main/pyroscope_backends/pyroscope_pprofrs/src/lib.rs#L158

The core part is the report() function, which looks easy for us to implement one for await tree

[BugenZhao]

After some investigation, I'm afraid await-tree is not that compatible with Pyroscope:

• The backends of Pyroscope should be sample-based. However, await-tree is not designed to be frequently captured as it'll acquire locks.
• await-tree yields trees, instead of stacks. Aggregating the results may not be feasible.
• await-tree does not output in a structured format now.

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.