Tracking: replace metrics-rs with prometheus client-rs

[flaneur2020]

Summary

This issue is an design RFC doc for this task #9422.

Background

Have discussed in #9422

Goal

The goal is to replace metrics-rs with a native prometheus client, this allows us to shorten an abstraction layer on the metrics path and easier to cover more metrics about histogram.

Considerations

Replacing metrics-rs with prometheus-rs is not only just replacing the metrics, we need some additional considerations about:

1. the convention on adding metrics: may have different conventions, it's a chance to find out a convention on making metrics.
2. integration with external library like opendal: opendal have it's prometheus metrics builtin, we need find a way to integrate the metrics together.
3. support the metrics_table: databend also contains a system table which allows us to dump the metrics in a table manner.
4. able to not affect performance on the fast path, able to plot metrics without memory allocation.

And of course, we need keep the existed metrics not changed.

tikv/rust-promtheus or prometheus/client-rust

tikv/rust-prometheus have been proven to be a production solution in many projects. But I'd prefer prometheus/client-rust here, because:

• prometheus/client-rust is the official client and could be considered to have more long-term support resources.
• prometheus/client-rust utilizes a strong type system for labels, whereas tikv/rust-prometheus adopts a more Go-like approach.

prometheus/client-rust has a quirk: it appends a "_total" suffix to all counter metrics. This could impact existing counter metrics. However, in my view, this is acceptable given the advantage of having long-term support.

TODOs

• check whether client_rs could dump the metrics
• observability: switch to prometheus's official rust client in metasrv #12648
• observability: replace metrics-rs with prometheus-client in databend-query's storage #12649
• observability: switch to prometheus-client in query execution #12688
• observability: replace the metrics_table with prometheus-client, and removes metrics-rs #12870
• refactor the dump metrics logic by a builtin iterator instead of parsing the prometheus output

References

• https://github.com/mosecorg/mosec/pull/335/files