Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create metricflow-semantics Package #1151

Merged
merged 80 commits into from
Apr 26, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
34bb6ff
Add missing `__init__.py` files in `tests`.
plypaul Apr 23, 2024
86420c6
Add semantics module.
plypaul Apr 23, 2024
f208ee1
Rename errors.
plypaul Apr 23, 2024
c75352d
Rename dataset.
plypaul Apr 23, 2024
678b993
Rename specs.
plypaul Apr 23, 2024
e50a1e5
Add missing `__init__.py` in `inference`.
plypaul Apr 23, 2024
8929eec
Move `base_time_grain.py`.
plypaul Apr 23, 2024
614f0e7
Move patterns.
plypaul Apr 23, 2024
2b2ea34
Move spec_classes.py
plypaul Apr 23, 2024
8657abf
Move specs.
plypaul Apr 23, 2024
38f689e
Move naming.
plypaul Apr 23, 2024
e9e04f4
Move filters.
plypaul Apr 23, 2024
91529ec
Move model.
plypaul Apr 23, 2024
248474b
Move dag.
plypaul Apr 23, 2024
c952eb0
Move dataset.
plypaul Apr 23, 2024
0cc46a3
Move dataflow plan sub-modules.
plypaul Apr 23, 2024
4f28202
Move the rest of dataflow.
plypaul Apr 23, 2024
64cad82
Move some files out of plan_conversion.
plypaul Apr 23, 2024
fb53b89
Move plan_conversion.
plypaul Apr 23, 2024
d6cbc72
Move errors.
plypaul Apr 23, 2024
02eda1b
Move protocols.
plypaul Apr 23, 2024
e1160dd
Move query.
plypaul Apr 23, 2024
d80ebc3
Move sql.
plypaul Apr 23, 2024
246b3b0
Move time.
plypaul Apr 23, 2024
c7f6a04
Move top level.
plypaul Apr 23, 2024
298b45e
Move collection_helpers.
plypaul Apr 23, 2024
d2cd7be
Move mf_logging.
plypaul Apr 23, 2024
5de0abd
Move out time.
plypaul Apr 23, 2024
be48a6d
Move out sql.
plypaul Apr 23, 2024
68875ee
Move out protocols/sql_client.
plypaul Apr 23, 2024
8724f02
Move out plan_conversion.
plypaul Apr 23, 2024
9c094dc
Move out dataset.
plypaul Apr 23, 2024
73580a4
Move out dataflow.
plypaul Apr 23, 2024
249cd1b
Move out time.
plypaul Apr 23, 2024
89bef6f
Move out `data_warehouse_model_validator.py`.
plypaul Apr 24, 2024
e629e9d
Move `sql_bind_parameters.py.`
plypaul Apr 24, 2024
471a7bd
Update `linkable_spec_resolver.py` to use metric time from DSI.
plypaul Apr 24, 2024
33752c7
Separate and move `SqlJoinType`.
plypaul Apr 24, 2024
fce72e9
Move `sql_join_type.py`.
plypaul Apr 24, 2024
2ef2163
Remove `SemanticManifestLookup.time_spine_source`.
plypaul Apr 24, 2024
77bd647
Move semantic tests.
plypaul Apr 24, 2024
cfc2eef
Add `metricflow-semantics` package skeleton.
plypaul Apr 24, 2024
58f631c
Move metricflow.semantics.
plypaul Apr 24, 2024
51431b2
Add `tests_metricflow_semantics`.
plypaul Apr 24, 2024
9173de7
Update `snapshot_path_prefix` to handle new test directories.
plypaul Apr 24, 2024
f88ad4e
Add `py.typed` for `metricflow_semantics`.
plypaul Apr 24, 2024
be48ed5
Move fixtures from `setup_fixtures.py` to `sql_client_fixtures.py`.
plypaul Apr 24, 2024
d2094c0
Rename tests -> tests_metricflow.
plypaul Apr 24, 2024
7c17c1e
Update test module name in tests.
plypaul Apr 24, 2024
f755143
Split `test_helpers.py` into separate files.
plypaul Apr 24, 2024
5426a61
Move helpers into `test_helpers` module.
plypaul Apr 24, 2024
3d9aba3
Change signature of `assert_snapshot_text_equal` to use `SnapshotConf…
plypaul Apr 24, 2024
2b87c74
Move `load_semantic_manifest` to `manifest_helpers.py`.
plypaul Apr 24, 2024
0313849
Move `semantic_manifest_yamls` to `test_helpers`.
plypaul Apr 24, 2024
d6d3cfc
Add `DirectoryAnchor` and use new manifest YAML dir.
plypaul Apr 24, 2024
d58a953
Move `assert_*_snapshot*` to `snapshot_helpers`.
plypaul Apr 24, 2024
4c9dda2
Add snapshot methods that don't depend on a SQL client.
plypaul Apr 24, 2024
3e349ff
Change signaure of `assert*` methods to use `SnapshotConfiguration`.
plypaul Apr 24, 2024
7e9ee2c
Initial configuration for `metricflow-semantics` tests.
plypaul Apr 24, 2024
00129ac
Move a few tests to new locations.
plypaul Apr 24, 2024
6f21bb7
Remove `DunderColumnAssociationResolver` from `test_suggestions.py`.
plypaul Apr 24, 2024
b988db5
Move `metric_time_dimension.py` to `test_helpers`.
plypaul Apr 24, 2024
3da1773
Remove `DataSet` dependency from `metric_time_dimension.py`
plypaul Apr 24, 2024
76119b6
Separate dataflow validation from `SemanticModelJoinEvaluator`.
plypaul Apr 24, 2024
ec9d092
Move semantic tests to `tests_metricflow_semantics`.
plypaul Apr 24, 2024
9b468a9
Move `DunderColumnAssociationResolver` to `metricflow-semantics`.
plypaul Apr 24, 2024
d201599
Add `column_association_resolver` fixture.
plypaul Apr 24, 2024
ae485c0
Add missing `query_parser` fixture.
plypaul Apr 24, 2024
742b85b
Update `pyproject.toml`.
plypaul Apr 24, 2024
d80bf8a
Move tests to `metricflow-semantics`.
plypaul Apr 24, 2024
1699579
Fix pretty_printing for newer Pydantic versions.
plypaul Apr 24, 2024
dbf35f8
Update various build-related files.
plypaul Apr 24, 2024
f9d3544
Move ID-related objects to `test_helpers``.
plypaul Apr 24, 2024
b853369
Move snapshots.
plypaul Apr 24, 2024
d54a4ad
Add change log for #1150.
plypaul Apr 24, 2024
4afbd09
Lint fixes.
plypaul Apr 24, 2024
f6f64cb
Address comments.
plypaul Apr 25, 2024
9e2c131
Update `DirectoryPathAnchor` to not require `__file__`.
plypaul Apr 25, 2024
60773a9
Update / cleanup build configuration.
plypaul Apr 25, 2024
47d7057
Add package CI tests.
plypaul Apr 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

import pathlib
from dataclasses import dataclass

from metricflow_semantics.test_helpers.snapshot_helpers import SnapshotConfiguration
Expand All @@ -23,3 +24,23 @@ class MetricFlowTestConfiguration(SnapshotConfiguration):
# is created and persisted between runs. The source schema name includes a hash of the tables that should be in
# the schema, so
use_persistent_source_schema: bool


class DirectoryPathAnchor:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Writing a wrapper class around a builtin one-liner is cutting against one of Python's core design principles. Just make the constant use pathlib directly. pathlib.Path(__file__).parent.directory

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Actually, I was looking to get rid of __file__ to make it easier to use. See the added commit that does this. Could be made into a function though.
  • I wanted to use these anchors for each of the semantic manifests. In getting the new package test configuration setup, not being able to auto-complete and relying on strings was error-prone. This is the n-th iteration that we've had to adjust this.

"""Defines a directory inside the repo.

Using this object allows you to avoid using hard-coded paths and instead use objects that will be handled properly
during refactoring.
"""

def __init__(self, path_to_file_in_directory: str) -> None:
"""Initializer.

Args:
path_to_file_in_directory: To be used with `__file__`.
"""
self._directory = pathlib.Path(path_to_file_in_directory).parent

@property
def directory(self) -> pathlib.Path: # noqa: D102
return self._directory
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
from __future__ import annotations

import os
from typing import Dict, Optional

from dbt_semantic_interfaces.implementations.semantic_manifest import PydanticSemanticManifest
Expand All @@ -10,16 +9,26 @@
)
from dbt_semantic_interfaces.validations.semantic_manifest_validator import SemanticManifestValidator

from metricflow_semantics.test_helpers.semantic_manifest_yamls import SEMANTIC_MANIFEST_YAMLS_PATH_ANCHOR


def load_semantic_manifest(
relative_manifest_path: str,
template_mapping: Optional[Dict[str, str]] = None,
) -> SemanticManifestBuildResult:
"""Reads the manifest YAMLs from the standard location, applies transformations, runs validations."""
yaml_file_directory = os.path.join(os.path.dirname(__file__), f"semantic_manifest_yamls/{relative_manifest_path}")
yaml_file_directory = SEMANTIC_MANIFEST_YAMLS_PATH_ANCHOR.directory.joinpath(relative_manifest_path)
build_result = parse_directory_of_yaml_files_to_semantic_manifest(
yaml_file_directory, template_mapping=template_mapping
str(yaml_file_directory), template_mapping=template_mapping
)
validator = SemanticManifestValidator[PydanticSemanticManifest]()
validator.checked_validations(build_result.semantic_manifest)
return build_result


def load_named_manifest(template_mapping: Dict[str, str], manifest_name: str) -> PydanticSemanticManifest: # noqa: D103
try:
build_result = load_semantic_manifest(manifest_name, template_mapping)
return build_result.semantic_manifest
except Exception as e:
raise RuntimeError(f"Error while loading semantic manifest: {manifest_name}") from e
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
from __future__ import annotations

from metricflow_semantics.test_helpers.config_helpers import DirectoryPathAnchor

SEMANTIC_MANIFEST_YAMLS_PATH_ANCHOR = DirectoryPathAnchor(__file__)