Add self-monitoring integrations

[lucabello]

Enhancement Proposal

The coordinator should run Grafana Agent in a sidecar container.

The agent should scrape the workers related to the coordinator at the addresses that are already communicated via the mimir_cluster interface.

The Mimir coordinator should implement the Prometheus send-remote-write relation so that the sidecar Grafana Agent can remote write those metrics somewhere.

---

External Grafana Agent

graph LR

subgraph mimir
worker1 ---|mimir-cluster| coordinator
worker2 ---|mimir-cluster| coordinator
worker3 ---|mimir-cluster| coordinator
coordinator ---|logging| gagent
coordinator ---|scrape| gagent
coordinator ---|remote-write| gagent
coordinator ---|dashboards| gagent
coordinator ---|trace| gagent

end

subgraph cos-lite
gagent ---|logging| loki
gagent ---|remote-write| prometheus
gagent ---|dashboards| grafana
gagent ---|trace| tempo

end

Loading

Describing the self-monitoring more in detail:

• dashboards and alerts are sent to Grafana Agent just like any other charm does;
• metrics are supported over the metrics_scrape relation; the coordinator charm will configure a scrape job for itself and for all the workers (getting the URLs from the mimir-cluster relation data);
• logs are supported over the logging relation; Grafana Agent needs to send the endpoint to push to over relation data to the coordinator, which then forwards it to the workers through the mimir-cluster relation. The workers should use Pebble log forwarding to push logs to that endpoint;
• receiving remote write is supported through the receive-remote-write relation, and should exist to allow for internal metrics (and potentially anything else) to be written back into Mimir from the Grafana Agent.

Having a separate Grafana agent has some pros:

• code changes are simpler (more isolated) because no replication is needed;
• separate the nginx and Grafana Agent load.