Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate alerts when cluster is in yellow state #506

Merged
merged 5 commits into from
Dec 4, 2024

Conversation

gabrielcocenza
Copy link
Member

When a cluster is in yellow state, it's because some shards are not active. This can be reached under heavy load, however if there are shards unassigned and the cluster is yellow, that means that some replicas are not allocated and it might be necessary to add new nodes in order to host all shards.

When a cluster is in yellow state, it's because some shards are
not active. This can be reached under heavy load, however if there
are shards unassigned and the cluster is yellow, that means that
some replicas are not allocated and it might be necessary to add
new nodes in order to host all shards.
phvalguima
phvalguima previously approved these changes Nov 21, 2024
Copy link
Contributor

@phvalguima phvalguima left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I don't have a strong opinion on per cluster or per instance, I think per instance makes sense as the user will receive multiple alerts but they will give you an itemized view of the servers in trouble.

Copy link
Contributor

@Mehdi-Bendriss Mehdi-Bendriss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Gabriel. I would rather rather have alerting on a cluster level to not spam with alerts instead of at the instance level.

src/alert_rules/prometheus/prometheus_alerts.yaml Outdated Show resolved Hide resolved
src/alert_rules/prometheus/prometheus_alerts.yaml Outdated Show resolved Hide resolved
src/alert_rules/prometheus/prometheus_alerts.yaml Outdated Show resolved Hide resolved
zmraul
zmraul previously approved these changes Dec 2, 2024
@gabrielcocenza gabrielcocenza requested a review from zmraul December 4, 2024 12:34
@gabrielcocenza gabrielcocenza merged commit 453fb0e into canonical:2/edge Dec 4, 2024
37 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants