Reduce metrics size

[fuyufjh]

Currently, according to a sample (metrics.zip) from the daily longevity test, we generate ~30k timeseries for compute-node and ~22k timeseries for meta node. This has caused a heavy burden for Prometheus.

$ grep -v '^#' compute.txt | wc -l                                                               
   30125
$ grep -v '^#' meta.txt | wc -l  
   21718

Analysis

Top timeseries (more than 500) on the compute node:

$ grep -v '^#' compute.txt | cut -d '{' -f 1 | cut -d ' ' -f 1 | sort | uniq -c | sort -nr | less
4579 stream_actor_output_buffer_blocking_duration_ns
4151 stream_exchange_frag_send_size
3343 state_store_mem_table_memory_size
3343 state_store_mem_table_item_count
2595 stream_actor_input_buffer_blocking_duration_ns
1160 stream_join_matched_join_keys_bucket
1034 stream_join_barrier_align_duration_bucket
 939 stream_executor_row_count
 880 state_store_sst_store_block_request_counts
 880 state_store_iter_scan_key_counts
 690 stream_memory_usage
 690 lru_evicted_watermark_time_ms
 660 state_store_read_req_positive_but_non_exist_counts
 660 state_store_read_req_check_bloom_filter_counts
 660 state_store_read_req_bloom_filter_positive_counts

Top timeseries (more than 500) on the meta node:

$ grep -v '^#' meta.txt | cut -d '{' -f 1 | cut -d ' ' -f 1 | sort | uniq -c | sort -nr | less
10020 table_info
7984 actor_info
 762 storage_version_stats
 680 storage_compact_task_size_bucket

Tasks

• bug: dispatcher_id should be downstream's fragment_id instead of actor_id #13592
  • stream_actor_output_buffer_blocking_duration_ns
  • stream_exchange_frag_send_size
  • stream_actor_input_buffer_blocking_duration_ns
• Reduce metrics of state_store_mem_table to fragment-level #13595
  • state_store_mem_table_memory_size
  • state_store_mem_table_item_count
• Reduce metrics of table_info & actor_info #13594
  • table_info
  • actor_info