Update scrape and remote_write libs for generic HostHealth rules

[MichaelThamm]

TODO

• Revert back to cosl@main (PYDEPS, requirements.txt, tox.ini)

Issue

Within the tandem cosl PR, we have a centralized (Prometheus rather than individual charms) way to "inject" alerts on the fly so we can extend Prometheus for generic up/absent rules.

Solution

The prometheus_scrape and prometheus_remote_write libs were updated by injecting generic HostHealth rules to the rules.

Documentation for implementation and testing

Context

In tandem with:

• Enable Generic Alert Rules cos-lib#115

Testing Instructions

Without Grafana Agent

1. Deploy Prometheus, Avalanche, and Alertmanager
2. Relate prom:metrics-endpoint avalanche and alertmanager prom:alertmanager
3. Scale Avalanche with juju add-unit avalanche -n 1
4. In the Prometheus UI, check for the new HostHealth rules or query Prometheus with juju show-unit prom/0 | yq -r '."prom/0"."relation-info"[1]."application-data"."alert_rules"' | jq
5. Artificially take down one Avalanche unit with pebble

juju exec --unit avalanche/0 -- \
  PEBBLE_SOCKET=/charm/containers/avalanche/pebble.socket \
  pebble stop avalanche

6. Check the Prometheus UI and notice the HostDown Alert firing (showing the specific unit Avalanche/0)
7. Artificially take down the second Avalanche unit with pebble and check for each Avalanche unit in the labels
8. Check the alerts arriving in Alertmanager for:
   • Host 'prom_8b073ff8-5456-492a-804b-7f9f15c996dc_avalanche_avalanche/0' is down. VALUE = 0 LABELS = map[__name__:up instance:prom_8b073ff8-5456-492a-804b-7f9f15c996dc_avalanche_avalanche/0 job:juju_prom_8b073ff8_avalanche_prometheus_scrape_avalanche-0 juju_application:avalanche juju_charm:avalanche-k8s juju_model:prom juju_model_uuid:8b073ff8-5456-492a-804b-7f9f15c996dc juju_unit:avalanche/0]

With Grafana Agent

Scrape

1. Deploy Grafana Agent, Prometheus, Avalanche, and Alertmanager
2. Relate gagent:metrics-endpoint avalanche, gagent:send-remote-write prom, and alertmanager prom:alertmanager
3. Scale Avalanche with juju add-unit avalanche -n 1
4. In the Prometheus UI, check for the new HostHealth and AggregatorHostHealth rules or query Grafana Agent with juju show-unit gagent/0 | yq -r '."gagent/0"."relation-info"[0]."application-data"."alert_rules"' | jq
5. Artificially take down one Avalanche unit with pebble
6. Check the Prometheus UI and notice the HostDown Alert firing (showing the specific unit Avalanche/0)
7. Artificially take down the second Avalanche unit with pebble and check for each Avalanche unit in the labels
8. Check the alerts arriving in Alertmanager (similar result as without Grafana Agent).

Remote write

1. Artificially take down Grafana Agent with pebble

juju exec --unit gagent/0 -- \
  PEBBLE_SOCKET=/charm/containers/agent/pebble.socket \
  pebble stop agent

2. Check the Prometheus UI and notice the HostUnavailable Alert firing in both the HostHealth and AggregatorHostHealth groups

Note

This does not show each unit, with 2 avalanche units, alert labels shows once per app:
alertname=HostUnavailablejuju_application=avalanchejuju_charm=avalanche-k8sjuju_model=promjuju_model_uuid=8b073ff8-5456-492a-804b-7f9f15c996dcseverity=critical

3. Check the alerts arriving in Alertmanager for:
   • Metrics not received from host ''. VALUE = 1 LABELS = map[juju_application:avalanche juju_model:prom juju_model_uuid:8b073ff8-5456-492a-804b-7f9f15c996dc]

Upgrade Notes

By fetching the new libs you would get a set of new alerts automatically. If charms already had up/absent alerts, this will result in duplication of alerts and rules. up/absent alerts are ubiquitous and are handled by the libs modified in this PR. Any custom alerts duplicating this behaviour can be removed.

With the new design introduced in this PR, you would get a separate HostUnavailable alert for Grafana Agent itself and each unit that is aggregated by it.

[sed-i]

Reminder to revert before merging here and elsewhere.