Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate metric constraints in DataflowPlanBuilder #393

Merged
merged 2 commits into from
Jan 13, 2023

Conversation

WilliamDee
Copy link
Contributor

Context

#352 (comment)

Changes

Essentially in the DataflowPlanBuilder, given a MetricSpec

  1. we get the Metric object corresponding to that MetricSpec and combine the where constraint defined in the Metric
  2. combine on the where constraint defined in the MetricSpec as well

Currently, it doesn't seem like there's a use case of doing both of these other than to cause slight confusion. At runtime when building the query, we just pass a list of MetricSpec to the builder that was initialized by the metric names only. So to remove the redundancy here, in the query parser we should attach the constraint to the MetricSpec that gets passed to the builder so that we don't have to retrieve the metric constraint from the Metric object directly downstream.

@WilliamDee WilliamDee force-pushed the will/cleanup-builder branch 4 times, most recently from 68ea691 to 5036dac Compare January 5, 2023 04:24
@WilliamDee WilliamDee temporarily deployed to DW_INTEGRATION_TESTS January 5, 2023 04:30 — with GitHub Actions Inactive
@WilliamDee WilliamDee temporarily deployed to DW_INTEGRATION_TESTS January 5, 2023 04:30 — with GitHub Actions Inactive
@WilliamDee WilliamDee temporarily deployed to DW_INTEGRATION_TESTS January 5, 2023 04:30 — with GitHub Actions Inactive
@WilliamDee WilliamDee had a problem deploying to DW_INTEGRATION_TESTS January 5, 2023 04:30 — with GitHub Actions Failure
@WilliamDee WilliamDee temporarily deployed to DW_INTEGRATION_TESTS January 5, 2023 05:21 — with GitHub Actions Inactive
@WilliamDee WilliamDee temporarily deployed to DW_INTEGRATION_TESTS January 5, 2023 05:21 — with GitHub Actions Inactive
@WilliamDee WilliamDee temporarily deployed to DW_INTEGRATION_TESTS January 5, 2023 05:22 — with GitHub Actions Inactive
@WilliamDee WilliamDee temporarily deployed to DW_INTEGRATION_TESTS January 5, 2023 05:22 — with GitHub Actions Inactive
@@ -243,6 +235,9 @@ def _build_metrics_output_node(
aggregated_measures_node=aggregated_measures_node,
)
)

assert len(compute_metrics_nodes) > 0, "ComputeMetricsNode was not properly constructed"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this assertion have existed before or is this now needed due to how the constraints are being bundled?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a sanity check that shouldve existed before, something wouldve went wrong if len == 0.

@WilliamDee WilliamDee requested review from plypaul and tlento January 5, 2023 20:46
Copy link
Contributor

@tlento tlento left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks less weird! I do think some added clarity around how we populate the spec constraint would be good to have before merge.

@@ -376,8 +377,26 @@ def _parse_and_validate_query(
metric_references=metric_references, time_dimension_specs=where_time_specs
)

metric_specs: List[MetricSpec] = []
for metric_reference in base_metric_references:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. It was a little hard to wrap my head around this.

The reason we do base_metric_references is because we only evaluate constraints on metrics at this point. The input_metrics to a DERIVED metric have the constraints pushed into the spec inside MetricSemantics.

It might be worth it to consolidate this further - right now we do top level spec construction in the query parser and upstream dependent spec construction inside MetricSemantics. But I also think that's likely opening up a can of worms, since MetricSemantics is in need of some restructuring itself.

So, I've convinced myself we should leave this here. But let's add a comment somewhere in this vicinity for why the base_metric_references are the things we care about here.

Also, I suggest initializing the local variable right above this block, it's not as important as putting it on line 233 makes it seem. We might even want to factor this all out into a helper method, then use the docstring to explain everything that's going on. I think I like that best.

@WilliamDee WilliamDee force-pushed the will/cleanup-builder branch from 5036dac to 73485c6 Compare January 10, 2023 18:17
@WilliamDee WilliamDee force-pushed the will/cleanup-builder branch from 73485c6 to 91f9f08 Compare January 10, 2023 19:22
@WilliamDee WilliamDee merged commit 34437ed into main Jan 13, 2023
@WilliamDee WilliamDee deleted the will/cleanup-builder branch January 13, 2023 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants