[BUG] Muting notification doesn't mute the notifications

[albgus]

What is the bug?
If you use the mute functionality the alerts aren't muted, they are queued. This means that if you want to mute an alert for whatever reason, such as the alert being misconfigured or just developing it, you will get hit by a massive tidal wave of alerts once you unmute the the destination.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Configure an Alert and a notification destination
2. Mute the notification destination
3. Make sure the alert is triggered
4. Unmute the alert destination
5. You will now recieve a delayed alert.

What is the expected behavior?
In all other monitoring tools I have used, terminology like "mute" means stopping the alerts because they are either expected during a maintenance window or something similar. I am not sure why I would like to manually pause an alert just to receive it hours later.

What is your host/environment?

• OpenSearch 2.11.0

[albgus]

Turns out that the root cause of alerts being sent was a misconfigured alert with a much longer period than intended. This made it seem like alerts from the entire last month was being sent all at once, while it actually was re-sent every rule evaluation.