fix(metrics): avoid calling with_label_values in every report for label guarded metrics

[wenym1]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Part of #13981.

For label guarded metrics, if we call with_label_values and drop the returned value label guarded metric immediately after we use it, the label will be seen as unused anymore. After the metrics are collected, the label will be removed, and later report on the same label will reinitialize the metrics, which may cause incorrect metrics (#13948).

In this PR, we remove all incorrect usages of with_label_values on the label guarded metrics, which uses the metrics in the form of metrics.with_label_values(...).inc(). Instead, we will store the value returned from with_label_values as field of some struct with longer lifetime so that the label will be regarded as being used and will not be repeatedly dropped and reinitialized.

Besides, the with_label_values of label guarded metrics is renamed to with_guarded_label_values so that all previous usages of with_label_values of label guarded metrics can be reviewed in this PR. And since we change to a new method name, when we are working on #12805, after we replace some usage of the raw metrics vec to the label guarded metrics vec, the code cannot be compiled directly. Any previous usage on with_label_values of raw metrics vec can be reviewed and change to be used in the correct way.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[fuyufjh]

LGTM

[chenzl25]

LGTM. Shall we cherry pick this pr to the 1.5.1-rc?

[codecov]

Codecov Report

Attention: 117 lines in your changes are missing coverage. Please review.

Comparison is base (028f0ae) 68.04% compared to head (fa4a8b1) 68.04%.

Files	Patch %	Lines
src/batch/src/monitor/stats.rs	0.00%	35 Missing ⚠️
...ream/src/common/log_store_impl/kv_log_store/mod.rs	0.00%	26 Missing ⚠️
src/stream/src/executor/monitor/streaming_stats.rs	0.00%	24 Missing ⚠️
src/stream/src/executor/dispatch.rs	81.39%	8 Missing ⚠️
src/common/src/monitor/connection.rs	46.15%	7 Missing ⚠️
src/batch/src/task/task_execution.rs	0.00%	6 Missing ⚠️
src/meta/src/stream/source_manager.rs	0.00%	6 Missing ⚠️
src/batch/src/task/context.rs	0.00%	2 Missing ⚠️
src/batch/src/executor/generic_exchange.rs	0.00%	1 Missing ⚠️
src/batch/src/executor/row_seq_scan.rs	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13997      +/-   ##
==========================================
- Coverage   68.04%   68.04%   -0.01%     
==========================================
  Files        1540     1540              
  Lines      265465   265529      +64     
==========================================
+ Hits       180637   180679      +42     
- Misses      84828    84850      +22     

Flag	Coverage Δ
rust	68.04% <44.28%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.