Include Time Spine Nodes in Dataflow Plan #1548
Merged
+59,216
−31,423
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
courtneyholcomb
force-pushed
the
court/simp8
branch
from
87ef573 to
107dcb7
Compare
courtneyholcomb
changed the title
courtneyholcomb
force-pushed
the
court/simp8
branch
from
107dcb7 to
d1921c9
Compare
courtneyholcomb
added
the
Run Tests With Other SQL Engines
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
github-actions
bot
removed
the
Run Tests With Other SQL Engines
courtneyholcomb
force-pushed
the
court/simp8
branch
2 times, most recently
from
c0b009e to
fe9a711
Compare
courtneyholcomb
force-pushed
the
court/simp7
branch
from
84dffb9 to
2bb872b
Compare
courtneyholcomb
force-pushed
the
court/simp8
branch
from
fe9a711 to
5515aa1
Compare
courtneyholcomb
added
the
Run Tests With Other SQL Engines
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
courtneyholcomb
temporarily deployed
to
DW_INTEGRATION_TESTS
GitHub Actions
Inactive
github-actions
bot
removed
the
Run Tests With Other SQL Engines
plypaul
approved these changes
Dec 11, 2024
metricflow/dataset/sql_dataset.py
Outdated
| @@ -144,7 +145,7 @@ def instances_for_time_dimensions( | |||
| return instances_to_return | |||
|
|
|||
| def instance_for_time_dimension(self, time_dimension_spec: TimeDimensionSpec) -> TimeDimensionInstance: | |||
| """Given the name of the time dimension, return the instance associated with it in the data set.""" | |||
| """Given ta time dimension spec, return the instance associated with it in the data set.""" | |||
metricflow/dataset/sql_dataset.py
Outdated
| for instance in instances: | ||
| if instance.spec == spec: | ||
| return instance | ||
| raise RuntimeError(f"Did not find instance matching spec in dataset. Spec: {spec}\nInstances: {instances}") |
There was a problem hiding this comment.
For better formatting, you can do:
# Easier autocomplete.
raise RuntimeError(str(LazyFormat("Did not find instance matching spec in dataset.", spec=spec, instances=instances))
# Or call formatting function.
raise RuntimeError(mf_pformat_dict("Did not find instance matching spec in dataset.", {"spec": spec, "instances": instances))
There was a problem hiding this comment.
Ah good call! I need to change this habit.
| class AliasSpecsNode(DataflowPlanNode, ABC): | ||
| """Change the columns matching the key specs to match the value specs.""" | ||
|
|
||
| change_specs: Sequence[Tuple[InstanceSpec, InstanceSpec]] |
There was a problem hiding this comment.
Since this class is frozen, it's preferable to keep the field types immutable as well. i.e.
change_specs: Tuple[...]
Also helpful to have a dataclass for Tuple[InstanceSpec, InstanceSpec] so that you don't have to remember what index is what.
… for time spines The metric time nodes are used for resolving metric_time without metrics. The read SQL nodes will be used for time spine joins.
This will allow us to stop building time spine nodes on the fly in the dataflow to SQL logic.
Also fixes a small issue where we were prematurely filtering out columns in that node. This is what you see in the snapshot changes. It should not impact optimized queries.
courtneyholcomb
force-pushed
the
court/simp8
branch
from
d74a4c4 to
9be730c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, time spine nodes were built on the fly in the dataflow to SQL logic. This came with several limitations, including:
This PR adds logic to build time spine nodes in the dataflow plan, instead.