Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: stale metrics of meta leader election #16334

Merged
merged 2 commits into from
Apr 16, 2024
Merged

fix: stale metrics of meta leader election #16334

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c6e628
Reset metrics to clean the stale labels e.g. invalid lease ids
fuyufjh Apr 16, 2024
4e6dceb
minor
fuyufjh Apr 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions src/meta/src/rpc/metrics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,17 +47,17 @@ use crate::rpc::ElectionClientRef;

#[derive(Clone)]
pub struct MetaMetrics {
/// ********************************** Meta ************************************
// ********************************** Meta ************************************
/// The number of workers in the cluster.
pub worker_num: IntGaugeVec,
/// The roles of all meta nodes in the cluster.
pub meta_type: IntGaugeVec,

/// ********************************** gRPC ************************************
// ********************************** gRPC ************************************
/// gRPC latency of meta services
pub grpc_latency: HistogramVec,

/// ********************************** Barrier ************************************
// ********************************** Barrier ************************************
/// The duration from barrier injection to commit
/// It is the sum of inflight-latency, sync-latency and wait-commit-latency
pub barrier_latency: Histogram,
Expand All @@ -73,11 +73,11 @@ pub struct MetaMetrics {
/// The timestamp (UNIX epoch seconds) of the last committed barrier's epoch time.
pub last_committed_barrier_time: IntGauge,

/// ********************************** Recovery ************************************
// ********************************** Recovery ************************************
pub recovery_failure_cnt: IntCounter,
pub recovery_latency: Histogram,

/// ********************************** Hummock ************************************
// ********************************** Hummock ************************************
/// Max committed epoch
pub max_committed_epoch: IntGauge,
/// The smallest epoch that has not been `GCed`.
Expand Down Expand Up @@ -160,16 +160,16 @@ pub struct MetaMetrics {
pub compaction_event_consumed_latency: Histogram,
pub compaction_event_loop_iteration_latency: Histogram,

/// ********************************** Object Store ************************************
// ********************************** Object Store ************************************
// Object store related metrics (for backup/restore and version checkpoint)
pub object_store_metric: Arc<ObjectStoreMetrics>,

/// ********************************** Source ************************************
// ********************************** Source ************************************
/// supervisor for which source is still up.
pub source_is_up: LabelGuardedIntGaugeVec<2>,
pub source_enumerator_metrics: Arc<SourceEnumeratorMetrics>,

/// ********************************** Fragment ************************************
// ********************************** Fragment ************************************
/// A dummpy gauge metrics with its label to be the mapping from actor id to fragment id
pub actor_info: IntGaugeVec,
/// A dummpy gauge metrics with its label to be the mapping from table id to actor id
Expand Down Expand Up @@ -744,6 +744,10 @@ pub fn start_worker_info_monitor(
}
};

// Reset metrics to clean the stale labels e.g. invalid lease ids
meta_metrics.worker_num.reset();
meta_metrics.meta_type.reset();

for (worker_type, worker_num) in node_map {
meta_metrics
.worker_num
Expand Down
Loading