You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is a large-scoped issue for all filter pushdown optimization requirements.
At a high level, we want to get a place where end users can specify filters in a way that is both natural and allows for easy hinting that filter predicates should be pushed down as far as they can go. This requires three things on the input interface side:
Allow users to split up predicates by semantic model in some natural way. This is done, actually, but we don't make use of it today. Users can, if they wish, provide a list of predicates which we will process as an intersection of filters (i.e., we will link them all with AND). For example, ["metric_time = '2021-01-01'", "user__country = 'CA' OR user__region = 'EMEA'"] is the same as metric_time = '2021-01-01' AND (user__country = 'CA' OR user__region = 'EMEA', but the former can have each statement pushed down to the constituent semantic models while the latter cannot (why not is a separate discussion, short version is we don't currently have the ability to do that level of semantic analysis).
Allow users to construct filters allowing us to properly apply range adjustments to time filters on metrics requiring expanded windows. For example, a cumulative metric with a 7-day aggregation window must include the week preceding the first date in the filter to return correct results for every requested output row.
Provide a way for users to ensure filters on partition columns are pushed down in such a manner that the underlying engine will trigger pruning.
Between our existing input interfaces and the addition of more robust partition specification options in the semantic manifest we should be able to provide all of these things. Indeed, we can deliver partial support for all of these scenarios without added updates to our input specifications. The sub-tasks listed here cover the major component pieces, and will be roughly addressed in the order in which they are created.
The text was updated successfully, but these errors were encountered:
tlento
changed the title
Add full support for predicate pushdown, including partition pruning
[SL-1627] Add full support for predicate pushdown, including partition pruning
Jan 29, 2024
This is a large-scoped issue for all filter pushdown optimization requirements.
At a high level, we want to get a place where end users can specify filters in a way that is both natural and allows for easy hinting that filter predicates should be pushed down as far as they can go. This requires three things on the input interface side:
["metric_time = '2021-01-01'", "user__country = 'CA' OR user__region = 'EMEA'"]
is the same asmetric_time = '2021-01-01' AND (user__country = 'CA' OR user__region = 'EMEA'
, but the former can have each statement pushed down to the constituent semantic models while the latter cannot (why not is a separate discussion, short version is we don't currently have the ability to do that level of semantic analysis).Between our existing input interfaces and the addition of more robust partition specification options in the semantic manifest we should be able to provide all of these things. Indeed, we can deliver partial support for all of these scenarios without added updates to our input specifications. The sub-tasks listed here cover the major component pieces, and will be roughly addressed in the order in which they are created.
SL-1627
Tasks
The text was updated successfully, but these errors were encountered: