-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a Resolver for Saved-Query Dependencies #1152
Merged
Merged
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
9d73ca5
Move `LinkableElementProperty.`
plypaul 0fa7db7
Move query-related specs to separate file.
plypaul c270b8b
Remove `MetricFlowQuerySpec` dependency from `GroupByMetricSpec`.
plypaul 4367174
Change `MetricFlowQueryParser.parse_and_validate_saved_query()` to re…
plypaul 796b367
Split set classes from `spec_classes.py` into separate files.
plypaul 0673c22
Update `LinkableElementSet` to include semantic model and spec methods.
plypaul 9cb1650
Use `LinkableElementSet` instead of specs in group-by-item resolution…
plypaul 5ed211e
Associated import and method changes in the rest of the codebase.
plypaul 847a4d2
Add interface for `SavedQueryDependencyResolver`.
plypaul d31029e
Update snapshots for where filter changes.
plypaul 76a11f9
Update snapshots to include spec -> element set updates.
plypaul 4a19e0f
Add change log for #1155.
plypaul bcbf1bf
Move `_path_key_to_spec` to `ElementPathKey`.
plypaul 07761a3
Address comments.
plypaul File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Empty file.
Empty file.
75 changes: 75 additions & 0 deletions
75
metricflow-semantics/metricflow_semantics/api/v0_1/saved_query_dependency_resolver.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
from __future__ import annotations | ||
|
||
import logging | ||
from dataclasses import dataclass | ||
from typing import Tuple | ||
|
||
from dbt_semantic_interfaces.protocols import SemanticManifest | ||
from dbt_semantic_interfaces.references import ( | ||
SemanticModelReference, | ||
) | ||
|
||
from metricflow_semantics.model.semantic_manifest_lookup import SemanticManifestLookup | ||
from metricflow_semantics.query.query_parser import MetricFlowQueryParser | ||
from metricflow_semantics.specs.query_param_implementations import SavedQueryParameter | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
|
||
@dataclass(frozen=True) | ||
class SavedQueryDependencySet: | ||
"""The dependencies of a saved query. | ||
|
||
The primary use case is to handle creation of the cache item associated with the saved query. The dependencies | ||
listed in this class must be up-to-date before the cache associated with the saved query can be created. Otherwise, | ||
running the export / creating the cache may create a cache item that is out-of-date / unusable. | ||
""" | ||
|
||
# The semantic models that the saved query depends on. | ||
semantic_model_references: Tuple[SemanticModelReference, ...] | ||
|
||
|
||
class SavedQueryDependencyResolver: | ||
"""Resolves the dependencies of a saved query. Also see `SavedQueryDependencySet`.""" | ||
|
||
def __init__(self, semantic_manifest: SemanticManifest) -> None: # noqa: D107 | ||
self._semantic_manifest = semantic_manifest | ||
self._query_parser = MetricFlowQueryParser(SemanticManifestLookup(semantic_manifest)) | ||
|
||
def _resolve_dependencies(self, saved_query_name: str) -> SavedQueryDependencySet: | ||
parse_result = self._query_parser.parse_and_validate_saved_query( | ||
saved_query_parameter=SavedQueryParameter(saved_query_name), | ||
where_filter=None, | ||
limit=None, | ||
time_constraint_start=None, | ||
time_constraint_end=None, | ||
order_by_names=None, | ||
order_by_parameters=None, | ||
) | ||
|
||
return SavedQueryDependencySet( | ||
semantic_model_references=tuple( | ||
sorted( | ||
parse_result.queried_semantic_models, | ||
key=lambda reference: reference.semantic_model_name, | ||
) | ||
), | ||
) | ||
|
||
def resolve_dependencies(self, saved_query_name: str) -> SavedQueryDependencySet: | ||
"""Return the dependencies of the given saved query in the manifest.""" | ||
try: | ||
return self._resolve_dependencies(saved_query_name) | ||
except Exception: | ||
logger.exception( | ||
f"Got an exception while getting the dependencies of saved-query {repr(saved_query_name)}. " | ||
f"All semantic models will be returned instead for safety." | ||
) | ||
return SavedQueryDependencySet( | ||
semantic_model_references=tuple( | ||
sorted( | ||
(semantic_model.reference for semantic_model in self._semantic_manifest.semantic_models), | ||
key=lambda reference: reference.semantic_model_name, | ||
) | ||
), | ||
) |
Empty file.
Empty file.
25 changes: 25 additions & 0 deletions
25
...low-semantics/tests_metricflow_semantics/api/v0_1/test_saved_query_dependency_resolver.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
from __future__ import annotations | ||
|
||
import pytest | ||
from dbt_semantic_interfaces.implementations.semantic_manifest import ( | ||
PydanticSemanticManifest, | ||
) | ||
from dbt_semantic_interfaces.references import ( | ||
SemanticModelReference, | ||
) | ||
from metricflow_semantics.api.v0_1.saved_query_dependency_resolver import SavedQueryDependencyResolver | ||
|
||
|
||
@pytest.fixture(scope="session") | ||
def resolver( # noqa: D103 | ||
simple_semantic_manifest: PydanticSemanticManifest, | ||
) -> SavedQueryDependencyResolver: | ||
return SavedQueryDependencyResolver(simple_semantic_manifest) | ||
|
||
|
||
def test_saved_query_dependency_resolver(resolver: SavedQueryDependencyResolver) -> None: # noqa: D103 | ||
dependency_set = resolver.resolve_dependencies("p0_booking") | ||
assert tuple(dependency_set.semantic_model_references) == ( | ||
SemanticModelReference(semantic_model_name="bookings_source"), | ||
SemanticModelReference(semantic_model_name="listings_latest"), | ||
) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. We might need to do something about this log level for deployment - if it turns out to be common we'll want it to show up with level WARN or INFO instead of EXCEPTION - but I think we can do that via datadog instead of here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a fail-safe and shouldn't really be hit - if it is, it means there's a bug that we need to fix.