Add new alert rules for throttling #509
Conversation
gabrielcocenza
commented
Nov 28, 2024
•
gabrielcocenza
commented
- If OpenSearch is throttling, this is an alert that optimizations are necessary like scaling the number of nodes or changing queries and indexing patterns
- It's recommended that OpenSearch runs with at least 3 nodes to have high availability - If OpenSearch is throttling, this is an alert that optimizations are necessary like scaling the number of nodes or changing queries and indexing patterns
| @@ -105,3 +105,23 @@ | |||
| "for": "1m" | |||
| "labels": | |||
| "severity": "alert" | |||
|
|
|||
| - "alert": "OpenSearchFewNodes" | |||
There was a problem hiding this comment.
In my opinion this alert is not too relevant. High availability requirements should be explained on docs, or deployment requisites, not trigger once integrated with observability.
There was a problem hiding this comment.
I kind agree. However, the generated metrics cannot ensure that a node is lost apart from checking if the upmetric is working or not. I created this as a warning just to trigger something not critical to operators know that the current cluster is not operating as expected.
There was a problem hiding this comment.
I've removed the node count alert. We can add in the future if necessary
There was a problem hiding this comment.
Hey @gabrielcocenza we cannot unfortunately limit ourselves to 3+ nodes. The scenario of 1-node and 2-node cluster is also supposed to work. On the other side, the throttling alert is a really good one!
gabrielcocenza
changed the title
- If OpenSearch is throttling, this is an alert that optimizations are necessary like scaling the number of nodes or changing queries and indexing patterns
