feat(alerting): detect flapping alerts #148
Conversation
| - name: AlertRules | ||
| rules: | ||
| - alert: AlertRulesFlapping | ||
| expr: changes(ALERTS_FOR_STATE{severity = "warning"}[1h]) > 5 |
There was a problem hiding this comment.
TIL!
Apparently there also ALERTS, but the value there is "1", whereas for ALERTS_FOR_STATE it's timestamp.
How did you find out this exists?
There was a problem hiding this comment.
ALERTS is not working with changes for some reason, when I found this out I google and find out about ALERTS_FOR_STATE it is internal metrics which is used to track alert state in between prometheus restarts. According to official docs
There was a problem hiding this comment.
Would we need to replicate the alert per severity?
There was a problem hiding this comment.
BTW, should we move this alert to prometheus? Seems like a generic alert rule.
We probably don't want topology injected here.
(And also we don't want prometheus topology injected there.)
|
Hi, during next two day I'm going to test this rule with inhibitions rules, so please do not merge I might be able to find some useful improvements |
|
@err404r Any news? |
Issue
We are currently not providing any alert rules that would detect when an alert is transitioning to and from
pendingin a fast pace, which means we could potentially loose out on information from recurring intermittent errors.Solution
Detect whenever an alert is transitioning more than 5 times in the last hour.
Context
resolves #145
Testing Instructions
Upgrade Notes