bug: potential metrics leak

[hzxa21]

Describe the bug

In our system, metrics label are widely used but I nonticed that the labeled metrics are not cleaned when they become unused. There are potential metrics leak for the following metrics:

• Per batch task metrics: should be cleaned when the task finishes.
• Per actor metrics: should be cleaned when the actor is dropped or moved. #9492
• Per source metrics: should be cleaned when the source is dropped.
• Exchange metrics: should be cleaned when the actor is dropped or moved.
• Per table storage metrics: should be cleand when the table id dropped. #9491
• Per host compaction metrics: shold be cleaned when the host is offline.
• Per compaction group metrics (meta side metrics are cleaned up): should be cleaned when the group is dropped.

To Reproduce

No response

Expected behavior

No response

Additional context

To fix the issue, we can cache the labeled metrics with_label_values and do remove_label_values when the metrics are no longer used. This has a side benefit to reduce label lookup when updating the metrics value.

[soundOfDestiny]

Per compaction group metrics (meta side metrics are cleaned up): should be cleaned when the group is dropped.

now it is dealt by remove_compaction_group_in_sst_stat which is called in sync_group

[hzxa21]

Per compaction group metrics (meta side metrics are cleaned up): should be cleaned when the group is dropped.

now it is dealt by remove_compaction_group_in_sst_stat which is called in sync_group

Yes, meta side compaction group metrics are already cleaned up correctly but the compactor side are still not.

[Gun9niR]

Cleaning the labels right after a short-living instance is dropped/finished will cause the corresponding metric value to be removed before it is collected. For example, a batch task can take only 1 sec, but Prometheus may collect once every 20s. Therefore, the stale labels can only be removed after they are collected, which means we need to implement our own Collector to be aware of the invocation of collect(). In #5770, I call reset() on all the metrics after they are collected, but it has a race condition that if data is written between collect() and reset(), it will be lost, tho it happens very rarely.

[lmatz]

Give an example:

The top ones come from a streaming query that has been dropped. But the metrics are kept.

[fuyufjh]

Need to call remove_label_values when actor is dropped.