Bug fix: For measures using join_to_timespine, apply filters after time spine join
#1056
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
courtneyholcomb
requested review from
tlento and
plypaul
and removed request for
tlento
courtneyholcomb
added
the
Run Tests With Other SQL Engines
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
github-actions
bot
removed
the
Run Tests With Other SQL Engines
tlento
approved these changes
Feb 29, 2024
Comment on lines
+48
to
+51
| metric_time__day BETWEEN '2020-01-03' AND '2020-01-05' | ||
| ) AND ( | ||
| metric_time__day > '2020-01-01' | ||
| ) |
There was a problem hiding this comment.
This is just an artifact of passing both a time constraint and a where filter on metric_time in the same query, right? Because if the filter in the query was metric_time__day = '2020-01-01' then this would return the empty set, but that's actually what the constraint combination would tell us to do.
There was a problem hiding this comment.
Yes. I knew it was weird when I wrote this test case but I just wanted to make sure both filters got rendered 😆
There was a problem hiding this comment.
But also it won't return an empty dataset since '2020-01-03' through '2020-01-05' is > '2020-01-01'!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #1039
Completes SL-1732
Description
For queries using measures that join to time spine, filters were being applied before the time spine join, but not after. This means that there might be additional rows added from the time spine join that should be removed by the filters, resulting in unexpected rows in the output data. Here, we update the DataFlow Plan to apply those filters again after the time spine join, ensuring there will be no unexpected rows in the result.
Note: this means the filters are applied multiple times, which is redundant for the final output but ideally results in a more efficient query.