Migrate to Mimir Alertmanager

[Rotfuks]

Motivation

Mimir has its own alertmanager, which is better integrated with the overall Mimir-functionality. Therefore we should migrate from the current Prometheus Alertmanager to the new Mimir Alertmanager.

Stories

Preview Give feedback

1. Investigation: Switch to the Grafana UI to fully replace Alert Manager UI #3745
   team/atlas +1
   
2. Investigation: Migrate the existing alerts to mimir alertmanager #3746
   team/atlas +1
   
3. Investigation: migration of notification template to the new alert manager #3747
   team/atlas +1
   
4. Deploy and test the Mimir Alert Manager #3744
   team/atlas +1
   
5. Change the silence manager to work with the new alert manager #3748
   team/atlas +1
   
6. Deprecate the PMO that manages the Alertmanager until now #3749
   team/atlas +1
7. Release the new Mimir Alertmanager #3752
   team/atlas +1
8. Document how mimir alert manager works #3789
   team/atlas +1
9. https://github.com/giantswarm/giantswarm/issues/21287

Outcome

• We know have an Alertmanager that is better integrated to the overall observability platform and gives us improved functionality in the future

[TheoBrigitte]

Here is a migration plan we though about with sir @hervenicol :

1. Add Feature flag to disable each Alertmanager implementation individually: in pmo and o11y-op
2. Enable Mimir Alertmanager
3. Switch Mimir Ruler to Mimir Alertmanager
4. Ensure Alerts are coming through on OpsGenie and Slack
5. Disable old Alertmanager