Use a non-global metrics registry in Teleport #50913
Conversation
|
I didn't want to add a metrics RFD, but it would be good to start using the process registry instead of the global one for the next features we build/metrics we add. |
hugoShaka
added
no-changelog
| // and the global registry (used by some Teleport services and many dependencies). | ||
| gatherers := prometheus.Gatherers{ | ||
| prometheus.DefaultGatherer, | ||
| process.metricsRegistry, |
There was a problem hiding this comment.
If conflicting metrics are registered I assume they'll be dropped, but unaffected metrics will keep working. Do you know if that's correct?
There was a problem hiding this comment.
If conflicting metrics are registered I assume they'll be dropped
Currently, registration conflicts in the global registry can cause:
- hard failure / error returned
- panics
- silent failure (metric does not get registered and we don't know about it)
Adding a local registry will not change the failure modes in case of conflict in the same registry. However, we are adding a new failure mode: metrics conflicting between the local and global registry. In this case, the global will prevail (I did this for backward compatibility reasons as everything is using the global registry today) the local registry will take precedence.
As we start using the local registry more, we might create such hard to detect conflicts. The situation is not strictly worse than today (we already have some racy metric registration with silent failure going on 😬). To ensure no conflict happen we can prefix new metrics by wrapping the registry when passing it to the service.
I think we would benefit from metrics guideline, setting the teleport component in the metric subsystem would reduce the probability of conflict.
There was a problem hiding this comment.
Thanks, very informative.
| // and the global registry (used by some Teleport services and many dependencies). | ||
| gatherers := prometheus.Gatherers{ | ||
| prometheus.DefaultGatherer, | ||
| process.metricsRegistry, |
There was a problem hiding this comment.
Thanks, very informative.
hugoShaka
force-pushed
the
hugo/teleport-use-non-global-metrics-registry
branch
from
853ac49 to
7be0540
Compare
hugoShaka
force-pushed
the
hugo/teleport-use-non-global-metrics-registry
branch
from
d153fda to
fa31b4a
Compare
This PR adds a new non-global per-process metrics registry in Teleport.
Using the global registry and global metrics causes conflicts in tests as we are starting multiple Teleport processes and/or other non-teleport processes (tbot, the operator, ...).
Having a new per-process metrics registry will allow Teleport services to register metrics scoped to their Teleport process. This will reduce the conflicts happening in tests.
To ensure backward compatibility, the Teleport metrics server serves both the process-scoped registry and the global registry.
Required for the autoupdate controller metrics PR.