Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create metricflow-semantics Package #1151

Merged
merged 80 commits into from
Apr 26, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
34bb6ff
Add missing `__init__.py` files in `tests`.
plypaul Apr 23, 2024
86420c6
Add semantics module.
plypaul Apr 23, 2024
f208ee1
Rename errors.
plypaul Apr 23, 2024
c75352d
Rename dataset.
plypaul Apr 23, 2024
678b993
Rename specs.
plypaul Apr 23, 2024
e50a1e5
Add missing `__init__.py` in `inference`.
plypaul Apr 23, 2024
8929eec
Move `base_time_grain.py`.
plypaul Apr 23, 2024
614f0e7
Move patterns.
plypaul Apr 23, 2024
2b2ea34
Move spec_classes.py
plypaul Apr 23, 2024
8657abf
Move specs.
plypaul Apr 23, 2024
38f689e
Move naming.
plypaul Apr 23, 2024
e9e04f4
Move filters.
plypaul Apr 23, 2024
91529ec
Move model.
plypaul Apr 23, 2024
248474b
Move dag.
plypaul Apr 23, 2024
c952eb0
Move dataset.
plypaul Apr 23, 2024
0cc46a3
Move dataflow plan sub-modules.
plypaul Apr 23, 2024
4f28202
Move the rest of dataflow.
plypaul Apr 23, 2024
64cad82
Move some files out of plan_conversion.
plypaul Apr 23, 2024
fb53b89
Move plan_conversion.
plypaul Apr 23, 2024
d6cbc72
Move errors.
plypaul Apr 23, 2024
02eda1b
Move protocols.
plypaul Apr 23, 2024
e1160dd
Move query.
plypaul Apr 23, 2024
d80ebc3
Move sql.
plypaul Apr 23, 2024
246b3b0
Move time.
plypaul Apr 23, 2024
c7f6a04
Move top level.
plypaul Apr 23, 2024
298b45e
Move collection_helpers.
plypaul Apr 23, 2024
d2cd7be
Move mf_logging.
plypaul Apr 23, 2024
5de0abd
Move out time.
plypaul Apr 23, 2024
be48a6d
Move out sql.
plypaul Apr 23, 2024
68875ee
Move out protocols/sql_client.
plypaul Apr 23, 2024
8724f02
Move out plan_conversion.
plypaul Apr 23, 2024
9c094dc
Move out dataset.
plypaul Apr 23, 2024
73580a4
Move out dataflow.
plypaul Apr 23, 2024
249cd1b
Move out time.
plypaul Apr 23, 2024
89bef6f
Move out `data_warehouse_model_validator.py`.
plypaul Apr 24, 2024
e629e9d
Move `sql_bind_parameters.py.`
plypaul Apr 24, 2024
471a7bd
Update `linkable_spec_resolver.py` to use metric time from DSI.
plypaul Apr 24, 2024
33752c7
Separate and move `SqlJoinType`.
plypaul Apr 24, 2024
fce72e9
Move `sql_join_type.py`.
plypaul Apr 24, 2024
2ef2163
Remove `SemanticManifestLookup.time_spine_source`.
plypaul Apr 24, 2024
77bd647
Move semantic tests.
plypaul Apr 24, 2024
cfc2eef
Add `metricflow-semantics` package skeleton.
plypaul Apr 24, 2024
58f631c
Move metricflow.semantics.
plypaul Apr 24, 2024
51431b2
Add `tests_metricflow_semantics`.
plypaul Apr 24, 2024
9173de7
Update `snapshot_path_prefix` to handle new test directories.
plypaul Apr 24, 2024
f88ad4e
Add `py.typed` for `metricflow_semantics`.
plypaul Apr 24, 2024
be48ed5
Move fixtures from `setup_fixtures.py` to `sql_client_fixtures.py`.
plypaul Apr 24, 2024
d2094c0
Rename tests -> tests_metricflow.
plypaul Apr 24, 2024
7c17c1e
Update test module name in tests.
plypaul Apr 24, 2024
f755143
Split `test_helpers.py` into separate files.
plypaul Apr 24, 2024
5426a61
Move helpers into `test_helpers` module.
plypaul Apr 24, 2024
3d9aba3
Change signature of `assert_snapshot_text_equal` to use `SnapshotConf…
plypaul Apr 24, 2024
2b87c74
Move `load_semantic_manifest` to `manifest_helpers.py`.
plypaul Apr 24, 2024
0313849
Move `semantic_manifest_yamls` to `test_helpers`.
plypaul Apr 24, 2024
d6d3cfc
Add `DirectoryAnchor` and use new manifest YAML dir.
plypaul Apr 24, 2024
d58a953
Move `assert_*_snapshot*` to `snapshot_helpers`.
plypaul Apr 24, 2024
4c9dda2
Add snapshot methods that don't depend on a SQL client.
plypaul Apr 24, 2024
3e349ff
Change signaure of `assert*` methods to use `SnapshotConfiguration`.
plypaul Apr 24, 2024
7e9ee2c
Initial configuration for `metricflow-semantics` tests.
plypaul Apr 24, 2024
00129ac
Move a few tests to new locations.
plypaul Apr 24, 2024
6f21bb7
Remove `DunderColumnAssociationResolver` from `test_suggestions.py`.
plypaul Apr 24, 2024
b988db5
Move `metric_time_dimension.py` to `test_helpers`.
plypaul Apr 24, 2024
3da1773
Remove `DataSet` dependency from `metric_time_dimension.py`
plypaul Apr 24, 2024
76119b6
Separate dataflow validation from `SemanticModelJoinEvaluator`.
plypaul Apr 24, 2024
ec9d092
Move semantic tests to `tests_metricflow_semantics`.
plypaul Apr 24, 2024
9b468a9
Move `DunderColumnAssociationResolver` to `metricflow-semantics`.
plypaul Apr 24, 2024
d201599
Add `column_association_resolver` fixture.
plypaul Apr 24, 2024
ae485c0
Add missing `query_parser` fixture.
plypaul Apr 24, 2024
742b85b
Update `pyproject.toml`.
plypaul Apr 24, 2024
d80bf8a
Move tests to `metricflow-semantics`.
plypaul Apr 24, 2024
1699579
Fix pretty_printing for newer Pydantic versions.
plypaul Apr 24, 2024
dbf35f8
Update various build-related files.
plypaul Apr 24, 2024
f9d3544
Move ID-related objects to `test_helpers``.
plypaul Apr 24, 2024
b853369
Move snapshots.
plypaul Apr 24, 2024
d54a4ad
Add change log for #1150.
plypaul Apr 24, 2024
4afbd09
Lint fixes.
plypaul Apr 24, 2024
f6f64cb
Address comments.
plypaul Apr 25, 2024
9e2c131
Update `DirectoryPathAnchor` to not require `__file__`.
plypaul Apr 25, 2024
60773a9
Update / cleanup build configuration.
plypaul Apr 25, 2024
47d7057
Add package CI tests.
plypaul Apr 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 13 additions & 10 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ ADDITIONAL_PYTEST_OPTIONS =

# Pytest that can populate the persistent source schema
USE_PERSISTENT_SOURCE_SCHEMA = --use-persistent-source-schema
POPULATE_PERSISTENT_SOURCE_SCHEMA = tests/source_schema_tools.py::populate_source_schema
TESTS_METRICFLOW = tests_metricflow
TESTS_METRICFLOW_SEMANTICS = tests_metricflow_semantics
POPULATE_PERSISTENT_SOURCE_SCHEMA = $(TESTS_METRICFLOW)/source_schema_tools.py::populate_source_schema

# Install Hatch package / project manager
.PHONY: install-hatch
Expand All @@ -19,32 +21,33 @@ install-hatch:
# Testing and linting
.PHONY: test
test:
hatch -v run dev-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) tests/
cd metricflow-semantics && hatch -v run dev-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) $(TESTS_METRICFLOW_SEMANTICS)/
hatch -v run dev-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) $(TESTS_METRICFLOW)/

.PHONY: test-postgresql
test-postgresql:
hatch -v run postgres-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) tests/
hatch -v run postgres-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) $(TESTS_METRICFLOW)/

# Engine-specific test environments.
.PHONY: test-bigquery
test-bigquery:
hatch -v run bigquery-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) tests/
hatch -v run bigquery-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) $(TESTS_METRICFLOW)/

.PHONY: populate-persistent-source-schema-bigquery
populate-persistent-source-schema-bigquery:
hatch -v run bigquery-env:pytest -vv $(ADDITIONAL_PYTEST_OPTIONS) $(USE_PERSISTENT_SOURCE_SCHEMA) $(POPULATE_PERSISTENT_SOURCE_SCHEMA)

.PHONY: test-databricks
test-databricks:
hatch -v run databricks-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) tests/
hatch -v run databricks-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) $(TESTS_METRICFLOW)/

.PHONY: populate-persistent-source-schema-databricks
populate-persistent-source-schema-databricks:
hatch -v run databricks-env:pytest -vv $(ADDITIONAL_PYTEST_OPTIONS) $(USE_PERSISTENT_SOURCE_SCHEMA) $(POPULATE_PERSISTENT_SOURCE_SCHEMA)

.PHONY: test-redshift
test-redshift:
hatch -v run redshift-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) tests/
hatch -v run redshift-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) $(TESTS_METRICFLOW)/

.PHONY: populate-persistent-source-schema-redshift
populate-persistent-source-schema-redshift:
Expand All @@ -53,15 +56,15 @@ populate-persistent-source-schema-redshift:

.PHONY: test-snowflake
test-snowflake:
hatch -v run snowflake-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) tests/
hatch -v run snowflake-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) $(TESTS_METRICFLOW)/

.PHONY: populate-persistent-source-schema-snowflake
populate-persistent-source-schema-snowflake:
hatch -v run snowflake-env:pytest -vv $(ADDITIONAL_PYTEST_OPTIONS) $(USE_PERSISTENT_SOURCE_SCHEMA) $(POPULATE_PERSISTENT_SOURCE_SCHEMA)

.PHONY: test-trino
test-trino:
hatch -v run trino-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) tests/
hatch -v run trino-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) $(TESTS_METRICFLOW)/

.PHONY: lint
lint:
Expand All @@ -79,12 +82,12 @@ trino:
# Re-generate test snapshots using all supported SQL engines.
.PHONY: regenerate-test-snapshots
regenerate-test-snapshots:
hatch -v run dev-env:python tests/generate_snapshots.py
hatch -v run dev-env:python tests_metricflow/generate_snapshots.py

# Populate persistent source schemas for all relevant SQL engines.
.PHONY: populate-persistent-source-schemas
populate-persistent-source-schemas:
hatch -v run dev-env:python tests/populate_persistent_source_schemas.py
hatch -v run dev-env:python $(TESTS_METRICFLOW)/populate_persistent_source_schemas.py

# Re-generate snapshots for the default SQL engine.
.PHONY: test-snap
Expand Down
5 changes: 3 additions & 2 deletions metricflow-semantics/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,12 @@ features = [

[tool.hatch.build.targets.sdist]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why we use exclude vs include?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I suspect this is a holdover from the original poetry configurations in the internal Transform repo pre-MetricFlow.

That said, I tried to switch to include when I split out the dbt dependencies and that kept producing broken builds for some reason, so I went back to exclude just to move on with my life.

Also, I don't think this applies right here right now, but there's stuff we're supposed to include in any distributions for legal reasons, and it's really easy to forget to update the packaging if we move one of those files or something.

exclude = [
"/.github",
"/.changes",
".github",
".changes",
".changie.yaml",
".gitignore",
".pre-commit-config.yaml",
"CONTRIBUTING.md",
"MAKEFILE",
"/tests_metricflow_semantics"
]
14 changes: 11 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,16 @@ trino-sql-client-packages = [

[tool.hatch.build.targets.sdist]
exclude = [
"/.github",
"/.changes",
".github",
".changes",
".changie.yaml",
".gitignore",
".pre-commit-config.yaml",
"CONTRIBUTING.md",
"Makefile",
"/dbt-metricflow",
"/local-data-warehouses",
"/metricflow-semantics/tests_metricflow_semantics",
"/scripts",
"/tests",
]
Expand All @@ -112,7 +113,8 @@ exclude = [
description = "Environment for development. Includes a DuckDB-backed client."

pre-install-commands = [
"pip install -e ./dbt-metricflow[duckdb]"
"pip install -e ./metricflow-semantics",
"pip install -e ./dbt-metricflow[duckdb]",
]

features = [
Expand All @@ -133,6 +135,7 @@ MF_TEST_ADAPTER_TYPE="postgres"
description = "Dev environment for working with Postgres adapter"

pre-install-commands = [
"pip install -e ./metricflow-semantics",
"pip install -e ./dbt-metricflow[postgres]"
]

Expand All @@ -151,6 +154,7 @@ MF_TEST_ADAPTER_TYPE="bigquery"
description = "Dev environment for working with the BigQuery adapter"

pre-install-commands = [
"pip install -e ./metricflow-semantics",
"pip install -e ./dbt-metricflow[bigquery]"
]

Expand All @@ -166,6 +170,7 @@ MF_TEST_ADAPTER_TYPE="databricks"
description = "Dev environment for working with the Databricks adapter"

pre-install-commands = [
"pip install -e ./metricflow-semantics",
"pip install -e ./dbt-metricflow[databricks]"
]

Expand All @@ -181,6 +186,7 @@ MF_TEST_ADAPTER_TYPE="redshift"
description = "Dev environment for working with the Redshift adapter"

pre-install-commands = [
"pip install -e ./metricflow-semantics",
"pip install -e ./dbt-metricflow[redshift]"
]

Expand All @@ -196,6 +202,7 @@ MF_TEST_ADAPTER_TYPE="snowflake"
description = "Dev environment for working with Snowflake adapter"

pre-install-commands = [
"pip install -e ./metricflow-semantics",
"pip install -e ./dbt-metricflow[snowflake]"
]

Expand All @@ -213,6 +220,7 @@ DBT_ENV_SECRET_CATALOG="memory"
description = "Dev environment for working with the Trino adapter"

pre-install-commands = [
"pip install -e ./metricflow-semantics",
"pip install -e ./dbt-metricflow[trino]"
]

Expand Down
2 changes: 1 addition & 1 deletion tests_metricflow/generate_snapshots.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
logger = logging.getLogger(__name__)


TEST_DIRECTORY = "tests"
TEST_DIRECTORY = "tests_metricflow"


class MetricFlowTestCredentialSet(FrozenBaseModel): # noqa: D101
Expand Down