-
Notifications
You must be signed in to change notification settings - Fork 158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
prometheus exporter: idle metrics not re-exported when becoming active again #372
Comments
I only register metrics and interact with the handle types. Let me do some testing and build some reproduction cases and I'll let you know if I find anything interesting. |
If you're using the handle types directly, then the linked issue is definitively the cause. |
I just saw that #394 was merged which would mean there's no work-around for this issue anyone. Just to clarify from your last message, here is how we we're currently using metrics. I believe this is "using the handle types directly". use metrics::register_counter;
let messages_received = register_counter!("messages_received", "stream_id" => id);
messages_received.increment(1); |
Just to clarify: your workaround _is_ still possible. Instead of
`register_counter!`, you'd now just call `counter!`.
…On Sun, Nov 26, 2023, 10:38 PM Ben Mitchell ***@***.***> wrote:
I just saw that #394 <#394> was
merged which would mean there's no work-around for this issue anyone.
Just to clarify from your last message, here is how we were using the
handle types before.
let sm = StreamMetrics {
messages_received: register_counter!("messages_received", "stream" => id);};
sm.messages_received.increment(1);
—
Reply to this email directly, view it on GitHub
<#372 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABWLF6ASSYRKCVJNV3SES3YGQDLZAVCNFSM6AAAAAAYOSBLAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRXGA3TOMZWGY>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
The not re-exported issue occurs when using the handle types ( I believe this is no longer possible as #394 because the current emission macro ( I might be confused on the recent changes or about what is a handler type vs emission macro, so maybe best to wait until the next release that includes #394 and I'll try it out then and let you know. (My previous message was just clarifying that |
I understand the confusion now. Under the hood, a metric always has to be registered before it can be used, whether you're holding on to it long-term (such as keeping it as a field in a struct) or just immediately performing the operation (increment, etc) and then moving on. When calling Now, circling back to my comment prior, the change in #394 just means that |
Issue
During periods of inactivity, after the
idle_timeout
period, metrics are correctly no longer exported. However, if these same metrics become active again, they are not re-exported.This is problematic for my use case since my server provides long-lived streams with periods of activity lasting 15 minutes and then long periods of inactivity (>90 minutes). Additionally, even without supporting long-lived streams, clients can connect anytime before data is available, which would trigger the same idle timeout -> not re-exported issue.
Example
Currently I'm constructing and installing a prometheus push gateway exporter (with VictoriaMetrics instead of Prometheus):
Timeline
stream_id=A
stream_id=A
start being exported (all 0s)stream_id=A
no longer exportedstream_id=A
starts receiving datastream_id=A
being exportedCrate Versions
I'm currently using the following (which are a tiny bit out of date if that matters):
The text was updated successfully, but these errors were encountered: