Improve stable-testing blackhole routing of alerts (#1488)

Signed-off-by: Marcus Noble <[email protected]>

CHANGELOG.md

@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- Improved the blackhole routing for `stable-testing` MCs to silence more alerts related to test WCs
+
 ## [4.63.1] - 2023-12-12
 
 ### Fixed

files/templates/alertmanager/alertmanager.yaml

@@ -22,6 +22,10 @@ route:
     matchers:
     - cluster_type="workload_cluster"
     continue: false
+  - receiver: blackhole
+    matchers:
+    - cluster_id=~"t-.*"
+    continue: false
   - receiver: blackhole
     matchers:
     - alertname=~"WorkloadClusterApp.*"
@@ -30,6 +34,10 @@ route:
     matchers:
     - alertname="PrometheusMetaOperatorReconcileErrors"
     continue: false
+  - receiver: blackhole
+    matchers:
+    - alertname="ClusterUnhealthyPhase"
+    continue: false
   - receiver: blackhole
     matchers:
     - alertname="ClusterUnhealthyPhase"
@@ -41,6 +49,7 @@ route:
     matchers:
     - alertname="ManagementClusterAppFailed"
     - namespace=~"org-.*"
+    continue: false
   [[- end ]]
 
   # Falco noise Slack