README.md

Send Pod metrics to a different backend

Overview

By default, the operator sends metrics to the corresponding Grafana Cloud stack, either via convention or configuration.

You can change this behaviour, setting a custom storage (remoteWrite). For instance, this may be useful to configure a Victoria Metrics backend to store your metrics:

Zoom | Diagram source | Shareable link

Configuring the backend

If you want to use this feature, you need to annotate your Namespace, with a remoteWrite configuration:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    # Ommitted for brevity
+   metrics.monitoring.adevinta.com/remote-write={namespace}/{secret-name}

In the above example, {secret-name} is a reference to a Secret created in the cluster.

Note

If the Secret is defined in the same namespace where it's being configured, then you do not need to namespace it. So, in that case, you may eliminate the {namespace}/ part.

The content of the secret is a remoteWrite spec as well as references to secrets that need to be used:

apiVersion: v1
kind: Secret
metadata:
    name: my-remote-write-secret
    annotations:
       monitor.adevinta.com/referenced-secrets: my-secret,my-secret-2
stringData:
   remote-write: |
      url: https://my-remote-write
      basicAuth:
         username:
            name: my-secret
            key: user-name
         password:
            name: my-secret-2
            key: password

Observability Operator then configures the tenant's Prometheus remoteWrite accordingly.