Problem with Grafana Cloud and max-global-series-per-user

[mueller-tobias]

Hi there,

I wanted to try out the Grafana Stack and connect my Lab Cluster to the Grafana Cloud. It is an RKE2 cluster without major deployments. I registered with Grafana Cloud and then installed the agent via Grafana Cloud -> Infrastructure -> Kubernetes -> Configuration and the Helm Command.

The problem is that I now get the following error and no metrics end up in the Grafana Cloud:

level=error msg="non-recoverable error" component=prometheus.remote_write.metrics_service subcomponent=rw remote_name=5956c2 url=https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push count=274 exemplarCount=0 err="server returned HTTP status 400 Bad Request: failed pushing to ingester ingester-zone-c-52: user=1422941: per-user series limit of 150000 exceeded (err-mimir-max-series-per-user). To adjust the related per-tenant limit, configure -ingester.max-global-series-per-user, or contact your service administrator. (sampled 1/10)

Here's my config. It the default from the grafana cloud ui:

helm repo add grafana https://grafana.github.io/helm-charts &&
  helm repo update &&
  helm upgrade --install --atomic --timeout 300s grafana-k8s-monitoring grafana/k8s-monitoring \
    --namespace "grafana-agent" --create-namespace --values - <<EOF
cluster:
  name: tm-lab
externalServices:
  prometheus:
    host: https://prometheus-prod-24-prod-eu-west-2.grafana.net
    basicAuth:
      username: "***"
      password: ***
  loki:
    host: https://logs-prod-012.grafana.net
    basicAuth:
      username: "***"
      password: ***
  tempo:
    host: https://tempo-prod-10-prod-eu-west-2.grafana.net:443
    basicAuth:
      username: "***"
      password: ***
metrics:
  enabled: true
  cost:
    enabled: true
  node-exporter:
    enabled: true
logs:
  enabled: true
  pod_logs:
    enabled: true
  cluster_events:
    enabled: true
traces:
  enabled: true
opencost:
  enabled: true
  opencost:
    exporter:
      defaultClusterId: lab
    prometheus:
      external:
        url: https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom
kube-state-metrics:
  enabled: true
prometheus-node-exporter:
  enabled: true
prometheus-operator-crds:
  enabled: true
grafana-agent: {}
grafana-agent-logs: {}
EOF

All i find wo max-global-series-per-user is Mimir related. But i can't configure Mimir due to its a limit from Grafana Cloud. Can i change a setting on the agent to get the metrics working?

Greetings
Tobias

[tpaschalis]

Looks like you're hitting some limits with the free tier of Grafana Cloud, so I don't think this is an agent-related issue.

Are you on the free tier of Grafana Cloud? Is there a way to filter some of the data either through the grafana-k8s-monitoring helm chart that you're using or via the Kubernetes Monitoring app in Grafana Cloud's generated config?