feat(meta): optimize table stats report performance #15401
Conversation
Signed-off-by: Little-Wallace <[email protected]>
| table_stats_change: &PbTableStatsMap, | ||
| ) { | ||
| for (table_id, stats) in table_stats_change { | ||
| if stats.total_key_size == 0 && stats.total_value_size == 0 && stats.total_key_count == 0 { |
There was a problem hiding this comment.
Shall we need to check 0 after purge_prost_table_stats ?
|
|
||
| // apply version delta before we persist this change. If it causes panic we can | ||
| // recover to a correct state after restarting meta-node. | ||
| current_version.apply_version_delta(&version_delta); | ||
| if purge_prost_table_stats(&mut version_stats.table_stats, ¤t_version) { |
There was a problem hiding this comment.
IMO, This metric is just an approximation, we don't need to update it every time we report the compact task, we can reduce the overhead by reporting it less frequently, what do you think?
There was a problem hiding this comment.
This PR only report the table which has update
Signed-off-by: Little-Wallace <[email protected]>
There was a problem hiding this comment.
There are three differnet logics to report the table stats metrics in differnet places:
- init:
get_local_table_stats - report_compact_task:
trigger_local_table_stat - commit_epoch:
trigger_table_stat
This looks ad-hoc and not clean. Some ideas to improve:
- Maintain local table stats (with table_id -> Int instead of IntGauge) under versioning
- Update the local table stats under version write lock (which is already acquired) in report_compact_task and commit_epoch. We can do it incrementally and no reset is needed.
- Report local table stats to prometheus in HummockTimerEvent::Report
Signed-off-by: Little-Wallace <[email protected]>
I have refactor code to maintain local table stats (with table_id -> Int instead of IntGauge) under versioning. |
Did you forget to push? |
| @@ -265,7 +265,7 @@ async fn test_hummock_compaction_task() { | |||
| // Finish the task and succeed. | |||
|
|
|||
| assert!(hummock_manager | |||
| .report_compact_task(compact_task.task_id, TaskStatus::Success, vec![], None) | |||
| .report_compact_task(compact_task.task_id, TaskStatus::Success, vec![], None,) | |||
| @@ -1972,7 +1972,7 @@ async fn test_compaction_task_expiration_due_to_split_group() { | |||
| let version_1 = hummock_manager.get_current_version().await; | |||
| // compaction_task.task_status = TaskStatus::Success.into(); | |||
| assert!(!hummock_manager | |||
| .report_compact_task(compaction_task.task_id, TaskStatus::Success, vec![], None) | |||
| .report_compact_task(compaction_task.task_id, TaskStatus::Success, vec![], None,) | |||
Signed-off-by: Little-Wallace <[email protected]>
Signed-off-by: Little-Wallace <[email protected]>
Signed-off-by: Little-Wallace <[email protected]>
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
close #15153
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.