[prometheus] "Error on ingesting out-of-order exemplars" message in logs

[puckpuck]

Bug Report

The following error continues to show up in the Prometheus logs:

prometheus  | ts=2024-11-26T04:44:06.414Z caller=write_handler.go:279 level=warn component=web msg="Error on ingesting out-of-order exemplars" num_dropped=44

This error started happening after upgrading flagd to version 0.11.4 in this PR

[puckpuck]

@beeme1mr This issue in particular did not exists with prior versions of flagd

I did an incremental change to 0.11.3, and saw this error stopped happening, so I suspect this is something very specific to the 0.11.3 -> 0.11.4 upgrade. The release notes for flagd, are not super obvious on why this error would occur outside of a prometheus client upgrade.

[beeme1mr]

We didn't make any telemetry-related changes in the last release but I'll look into it.

Are you sure this issue is caused by flagd? It's not clear from the attached logs.

Sorry, reread your comment.

[beeme1mr]

I can't get the demo app running locally due to an unrelated error.

 ⠋ Container otel-col          Creating                                                                                                                                                                      0.1s
Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "rprivate"
make: *** [Makefile:138: start] Error 1

Another user reported this already, but the fix has been reverted.

[beeme1mr]

Possibly related.

prometheus/prometheus#13933

[julianocosta89]

I can't get the demo app running locally due to an unrelated error.

 ⠋ Container otel-col          Creating                                                                                                                                                                      0.1s
Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "rprivate"
make: *** [Makefile:138: start] Error 1

Another user reported this already, but the fix has been reverted.

@beeme1mr the latest release doesn't have that anymore.
Have you pulled the latest?

[beeme1mr]

Yeah, I'm trying to run the latest version of the demo in Ubuntu on WSL.

[julianocosta89]

Ah, I see. Re-reading your message it actually makes sense.
We have removed the rslave param because it is not required anymore in the latest docker version.
It seemed to be an issue that happened in one single version.

Multiple users reported that they were facing issues to run with the rslave param.

Could you check if updating docker solves for you?
If not, maybe you could edit the docker-compose.yaml file locally.

But ideally the demo would run in all setups.

[beeme1mr]

I'm running the latest version available through apt-get. However, it's not the latest version according to the Docker release notes. I'll check again tomorrow.