Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug fix: For measures using join_to_timespine, apply filters after time spine join #1056

Merged
merged 6 commits into from
Feb 29, 2024
Merged

Bug fix: For measures using join_to_timespine, apply filters after time spine join #1056

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
52417b3
Changelog
courtneyholcomb Feb 28, 2024
aaecec1
Update DataFlow Plan to apply filters again after joining to time spine
courtneyholcomb Feb 28, 2024
670538c
Write tests
courtneyholcomb Feb 28, 2024
816696c
Update basic snapshots
courtneyholcomb Feb 28, 2024
a399940
Update snapshots for all engines
courtneyholcomb Feb 28, 2024
312307b
Merge main
courtneyholcomb Feb 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
352 changes: 352 additions & 0 deletions ...s_with_rendering.py/SqlQueryPlan/BigQuery/test_join_to_time_spine_with_filters__plan0.sql
View file
Open in desktop

Large diffs are not rendered by default.

52 changes: 52 additions & 0 deletions ...dering.py/SqlQueryPlan/BigQuery/test_join_to_time_spine_with_filters__plan0_optimized.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
-- Compute Metrics via Expressions
SELECT
metric_time__day
, COALESCE(bookings, 0) AS bookings_fill_nulls_with_0
FROM (
-- Constrain Output with WHERE
-- Constrain Time Range to [2020-01-03T00:00:00, 2020-01-05T00:00:00]
SELECT
metric_time__day
, bookings
FROM (
-- Join to Time Spine Dataset
SELECT
subq_19.metric_time__day AS metric_time__day
, subq_18.bookings AS bookings
FROM (
-- Time Spine
SELECT
ds AS metric_time__day
FROM ***************************.mf_time_spine subq_20
WHERE ds BETWEEN '2020-01-03' AND '2020-01-05'
) subq_19
LEFT OUTER JOIN (
-- Constrain Output with WHERE
-- Aggregate Measures
SELECT
metric_time__day
, SUM(bookings) AS bookings
FROM (
-- Read Elements From Semantic Model 'bookings_source'
-- Metric Time Dimension 'ds'
-- Constrain Time Range to [2020-01-03T00:00:00, 2020-01-05T00:00:00]
-- Pass Only Elements: ['bookings', 'metric_time__day']
SELECT
DATE_TRUNC(ds, day) AS metric_time__day
, 1 AS bookings
FROM ***************************.fct_bookings bookings_source_src_28000
WHERE DATE_TRUNC(ds, day) BETWEEN '2020-01-03' AND '2020-01-05'
) subq_16
WHERE metric_time__day > '2020-01-01'
GROUP BY
metric_time__day
) subq_18
ON
subq_19.metric_time__day = subq_18.metric_time__day
) subq_21
WHERE (
metric_time__day BETWEEN '2020-01-03' AND '2020-01-05'
) AND (
metric_time__day > '2020-01-01'
)
Comment on lines +48 to +51
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just an artifact of passing both a time constraint and a where filter on metric_time in the same query, right? Because if the filter in the query was metric_time__day = '2020-01-01' then this would return the empty set, but that's actually what the constraint combination would tell us to do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I knew it was weird when I wrote this test case but I just wanted to make sure both filters got rendered 😆

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But also it won't return an empty dataset since '2020-01-03' through '2020-01-05' is > '2020-01-01'!

) subq_23
352 changes: 352 additions & 0 deletions ...with_rendering.py/SqlQueryPlan/Databricks/test_join_to_time_spine_with_filters__plan0.sql
View file
Open in desktop

Large diffs are not rendered by default.

52 changes: 52 additions & 0 deletions ...ring.py/SqlQueryPlan/Databricks/test_join_to_time_spine_with_filters__plan0_optimized.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
-- Compute Metrics via Expressions
SELECT
metric_time__day
, COALESCE(bookings, 0) AS bookings_fill_nulls_with_0
FROM (
-- Constrain Output with WHERE
-- Constrain Time Range to [2020-01-03T00:00:00, 2020-01-05T00:00:00]
SELECT
metric_time__day
, bookings
FROM (
-- Join to Time Spine Dataset
SELECT
subq_19.metric_time__day AS metric_time__day
, subq_18.bookings AS bookings
FROM (
-- Time Spine
SELECT
ds AS metric_time__day
FROM ***************************.mf_time_spine subq_20
WHERE ds BETWEEN '2020-01-03' AND '2020-01-05'
) subq_19
LEFT OUTER JOIN (
-- Constrain Output with WHERE
-- Aggregate Measures
SELECT
metric_time__day
, SUM(bookings) AS bookings
FROM (
-- Read Elements From Semantic Model 'bookings_source'
-- Metric Time Dimension 'ds'
-- Constrain Time Range to [2020-01-03T00:00:00, 2020-01-05T00:00:00]
-- Pass Only Elements: ['bookings', 'metric_time__day']
SELECT
DATE_TRUNC('day', ds) AS metric_time__day
, 1 AS bookings
FROM ***************************.fct_bookings bookings_source_src_28000
WHERE DATE_TRUNC('day', ds) BETWEEN '2020-01-03' AND '2020-01-05'
) subq_16
WHERE metric_time__day > '2020-01-01'
GROUP BY
metric_time__day
) subq_18
ON
subq_19.metric_time__day = subq_18.metric_time__day
) subq_21
WHERE (
metric_time__day BETWEEN '2020-01-03' AND '2020-01-05'
) AND (
metric_time__day > '2020-01-01'
)
) subq_23
352 changes: 352 additions & 0 deletions ...s_with_rendering.py/SqlQueryPlan/Postgres/test_join_to_time_spine_with_filters__plan0.sql
View file
Open in desktop

Large diffs are not rendered by default.

52 changes: 52 additions & 0 deletions ...dering.py/SqlQueryPlan/Postgres/test_join_to_time_spine_with_filters__plan0_optimized.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
-- Compute Metrics via Expressions
SELECT
metric_time__day
, COALESCE(bookings, 0) AS bookings_fill_nulls_with_0
FROM (
-- Constrain Output with WHERE
-- Constrain Time Range to [2020-01-03T00:00:00, 2020-01-05T00:00:00]
SELECT
metric_time__day
, bookings
FROM (
-- Join to Time Spine Dataset
SELECT
subq_19.metric_time__day AS metric_time__day
, subq_18.bookings AS bookings
FROM (
-- Time Spine
SELECT
ds AS metric_time__day
FROM ***************************.mf_time_spine subq_20
WHERE ds BETWEEN '2020-01-03' AND '2020-01-05'
) subq_19
LEFT OUTER JOIN (
-- Constrain Output with WHERE
-- Aggregate Measures
SELECT
metric_time__day
, SUM(bookings) AS bookings
FROM (
-- Read Elements From Semantic Model 'bookings_source'
-- Metric Time Dimension 'ds'
-- Constrain Time Range to [2020-01-03T00:00:00, 2020-01-05T00:00:00]
-- Pass Only Elements: ['bookings', 'metric_time__day']
SELECT
DATE_TRUNC('day', ds) AS metric_time__day
, 1 AS bookings
FROM ***************************.fct_bookings bookings_source_src_28000
WHERE DATE_TRUNC('day', ds) BETWEEN '2020-01-03' AND '2020-01-05'
) subq_16
WHERE metric_time__day > '2020-01-01'
GROUP BY
metric_time__day
) subq_18
ON
subq_19.metric_time__day = subq_18.metric_time__day
) subq_21
WHERE (
metric_time__day BETWEEN '2020-01-03' AND '2020-01-05'
) AND (
metric_time__day > '2020-01-01'
)
) subq_23
Loading
Loading