Investigation: Alert Routing in Multi-Provider Setup

[Rotfuks]

Feedback from Jose:

On many of our PrometheusRules we page the team depending on the provider, using a variable like providerTeam that changes depending on the provider. This doesn't work for MCs supporting multiple providers. For example, we phoenix are getting paged by vsphere clusters on wallaby, just because wallaby is a capz MC. Has this been discussed? Are there potential solutions that we can apply to page the right team?

See: https://gigantic.slack.com/archives/CLPMFRVU6/p1734009831450799

We need to create a process on how we can do alert routing for giantswarm teams with multi-provider setups

Todo

• "Document" the process of how alert routing for giantswarm teams (in multi-provider setups) currently works for review
• List the alerts that are tricky in multi-provider setups having issues with routing to the right team
• Review the current process and Alerts and Investigate how we could improve the Alert routing

Outcome

• We know how we can improve the situation.