-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve cache for MetricLookup.linkable_elements_for_measure()
#1443
Conversation
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
lambda: f"Getting valid linkable elements for measure '{measure_reference.element_name}' took: {time.time() - start_time:.2f}s" | ||
) | ||
) | ||
# Don't cache the result when group-by-metrics are selected as there can be many of them and may significantly |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we remove the cache for group by metrics, doesn't that mean this will be super slow when we try to fetch group by metric options?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had some concerns about memory usage, so caching of group-by-metrics is handled later through a separate cache that's LRU and avoids GC issues.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it 👍
measure_reference=measure_reference, | ||
element_filter=element_filter, | ||
) | ||
logger.debug( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious why you removed this log? Seems helpful to see if there was improvement on performance for this method. Also seems like it would be a good one to make INFO
instead of DEBUG
so that we can see how it performs for customers with larger manifests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method can potentially be called a lot during query parsing, so it's easier to assess the runtime through DD tooling. e.g. a query with ~100 metrics, ~100 group-by items (including ones in filters) leads to a lot of log lines. Could leave a debug line in there though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added that debug line.
7469801
to
9cc2d57
Compare
…linkable_elements_for_measure()`.
9cc2d57
to
38e78d7
Compare
This PR changes the cache for `MetricLookup.linkable_elements_for_measure()` to allow caching all measures. Memory use remained reasonable and because it's bound to `MetricLookup`, it will be GCed when a given manifest is no longer used.
This PR changes the cache for
MetricLookup.linkable_elements_for_measure()
to allow caching all measures. Memory use remained reasonable and because it's bound toMetricLookup
, it will be GCed when a given manifest is no longer used.