Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test time spines for sub-daily granularity #1358

Merged
merged 6 commits into from
Jul 30, 2024
Merged

Add test time spines for sub-daily granularity #1358

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9cec4eb
Add test time spines for sub-daily granularity
courtneyholcomb Jul 27, 2024
d7ea994
Update snapshots - all node id changes due to new time spine source n…
courtneyholcomb Jul 27, 2024
fdcf983
Update time spine test fixture to support multiple
courtneyholcomb Jul 27, 2024
ed9555c
Update SQL engine snapshots
courtneyholcomb Jul 29, 2024
2ef8de8
Remove tests that require engine-specific syntax in metric definitions
courtneyholcomb Jul 29, 2024
ae263fd
Handle no-metric queries with sub-daily `metric_time` (#1359)
courtneyholcomb Jul 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions ...tricflow_semantics/test_helpers/semantic_manifest_yamls/shared/project_configuration.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,40 @@ project_configuration:
- location: $source_schema.mf_time_spine
column_name: ds
grain: day
time_spines:
- node_relation:
alias: mf_time_spine_nanosecond
schema_name: $source_schema
primary_column:
name: ts
time_granularity: nanosecond
- node_relation:
alias: mf_time_spine_microsecond
schema_name: $source_schema
primary_column:
name: ts
time_granularity: microsecond
- node_relation:
alias: mf_time_spine_millisecond
schema_name: $source_schema
primary_column:
name: ts
time_granularity: millisecond
- node_relation:
alias: mf_time_spine_second
schema_name: $source_schema
primary_column:
name: ts
time_granularity: second
- node_relation:
alias: mf_time_spine_minute
schema_name: $source_schema
primary_column:
name: ts
time_granularity: minute
- node_relation:
alias: mf_time_spine_hour
schema_name: $source_schema
primary_column:
name: ts
time_granularity: hour
42 changes: 42 additions & 0 deletions tests_metricflow/fixtures/source_table_snapshots/time_spine_table/mf_time_spine_hour.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
table_snapshot:
table_name: mf_time_spine_hour
column_definitions:
- name: ts
type: TIME
rows:
- ["2020-01-01 01:00:00"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh. I just realized, are we going to date_trunc the time spine input to the specified grain? I'm pretty sure we don't do it today, but there's a type for it (DATE). The spec calls for the end user to configure that correctly, so I'm inclined not to date_trunc right now, but it might be something we need to do.

Presumably most people are using packages to build these things so maybe we just rely on that. If we're worried but not very worried about this we could also set up a best-effort warehouse validation, or release a validation package for time spine models that people can use if they wish.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't apply that DATE_TRUNC in JoinToTimeSpineNode or JoinOverTimeRangeNode, But we do in ReadSqlSourceNode & MetricTimeDimensionTransformNode. This feature doesn't change that behavior so far, but we could discuss if we want to change it.
I think the warehouse validations will be a good idea so we can have more efficient queries.

- ["2020-01-01 02:00:00"]
- ["2020-01-01 03:00:00"]
- ["2020-01-01 04:00:00"]
- ["2020-01-01 05:00:00"]
- ["2020-01-01 06:00:00"]
- ["2020-01-01 07:00:00"]
- ["2020-01-01 08:00:00"]
- ["2020-01-01 09:00:00"]
- ["2020-01-01 010:00:00"]
- ["2020-01-01 11:00:00"]
- ["2020-01-01 12:00:00"]
- ["2020-01-02 01:00:00"]
- ["2020-01-02 02:00:00"]
- ["2020-01-02 03:00:00"]
- ["2020-01-02 04:00:00"]
- ["2020-01-02 05:00:00"]
- ["2020-01-02 06:00:00"]
- ["2020-01-02 07:00:00"]
- ["2020-01-02 08:00:00"]
- ["2020-01-02 09:00:00"]
- ["2020-01-02 010:00:00"]
- ["2020-01-02 11:00:00"]
- ["2020-01-02 12:00:00"]
- ["2020-01-03 01:00:00"]
- ["2020-01-03 02:00:00"]
- ["2020-01-03 03:00:00"]
- ["2020-01-03 04:00:00"]
- ["2020-01-03 05:00:00"]
- ["2020-01-03 06:00:00"]
- ["2020-01-03 07:00:00"]
- ["2020-01-03 08:00:00"]
- ["2020-01-03 09:00:00"]
- ["2020-01-03 010:00:00"]
- ["2020-01-03 11:00:00"]
- ["2020-01-03 12:00:00"]
37 changes: 37 additions & 0 deletions ...etricflow/fixtures/source_table_snapshots/time_spine_table/mf_time_spine_microsecond.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
table_snapshot:
table_name: mf_time_spine_microsecond
column_definitions:
- name: ts
type: TIME
rows:
- ["2020-01-01 00:00:00.000000"]
- ["2020-01-01 00:00:00.000001"]
- ["2020-01-01 00:00:00.000002"]
- ["2020-01-01 00:00:00.000003"]
- ["2020-01-01 00:00:00.000004"]
- ["2020-01-01 00:00:00.000005"]
- ["2020-01-01 00:00:00.000006"]
- ["2020-01-01 00:00:00.000007"]
- ["2020-01-01 00:00:00.000008"]
- ["2020-01-01 00:00:00.000009"]
- ["2020-01-01 00:00:00.000010"]
- ["2020-01-01 00:00:00.000011"]
- ["2020-01-01 00:00:00.000012"]
- ["2020-01-01 00:00:00.000013"]
- ["2020-01-01 00:00:00.000014"]
- ["2020-01-01 00:00:00.000015"]
- ["2020-01-01 00:00:00.000016"]
- ["2020-01-01 00:00:00.000017"]
- ["2020-01-01 00:00:00.000018"]
- ["2020-01-01 00:00:00.000019"]
- ["2020-01-01 00:00:00.000020"]
- ["2020-01-01 00:00:00.000021"]
- ["2020-01-01 00:00:00.000022"]
- ["2020-01-01 00:00:00.000023"]
- ["2020-01-01 00:00:00.000024"]
- ["2020-01-01 00:00:00.000025"]
- ["2020-01-01 00:00:00.000026"]
- ["2020-01-01 00:00:00.000027"]
- ["2020-01-01 00:00:00.000028"]
- ["2020-01-01 00:00:00.000029"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want values from multiple days, even though they won't be contiguous in the input.

I'm not sure if this really matters but I'm always wary of having test data pegged to a boundary (in this case, a year boundary).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough - I can update that tomorrow! Shouldn't impact any of the tests, just will need to repopulate the source schemas.

- ["2020-01-01 00:00:00.000030"]
36 changes: 36 additions & 0 deletions ...etricflow/fixtures/source_table_snapshots/time_spine_table/mf_time_spine_millisecond.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
table_snapshot:
table_name: mf_time_spine_millisecond
column_definitions:
- name: ts
type: TIME
rows:
- ["2020-01-01 00:00:00.001"]
- ["2020-01-01 00:00:00.002"]
- ["2020-01-01 00:00:00.003"]
- ["2020-01-01 00:00:00.004"]
- ["2020-01-01 00:00:00.005"]
- ["2020-01-01 00:00:00.006"]
- ["2020-01-01 00:00:00.007"]
- ["2020-01-01 00:00:00.008"]
- ["2020-01-01 00:00:00.009"]
- ["2020-01-01 00:00:00.010"]
- ["2020-01-01 00:00:00.011"]
- ["2020-01-01 00:00:00.012"]
- ["2020-01-01 00:00:00.013"]
- ["2020-01-01 00:00:00.014"]
- ["2020-01-01 00:00:00.015"]
- ["2020-01-01 00:00:00.016"]
- ["2020-01-01 00:00:00.017"]
- ["2020-01-01 00:00:00.018"]
- ["2020-01-01 00:00:00.019"]
- ["2020-01-01 00:00:00.020"]
- ["2020-01-01 00:00:00.021"]
- ["2020-01-01 00:00:00.022"]
- ["2020-01-01 00:00:00.023"]
- ["2020-01-01 00:00:00.024"]
- ["2020-01-01 00:00:00.025"]
- ["2020-01-01 00:00:00.026"]
- ["2020-01-01 00:00:00.027"]
- ["2020-01-01 00:00:00.028"]
- ["2020-01-01 00:00:00.029"]
- ["2020-01-01 00:00:00.030"]
37 changes: 37 additions & 0 deletions tests_metricflow/fixtures/source_table_snapshots/time_spine_table/mf_time_spine_minute.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
table_snapshot:
table_name: mf_time_spine_minute
column_definitions:
- name: ts
type: TIME
rows:
- ["2020-01-01 00:00:00"]
- ["2020-01-01 00:01:00"]
- ["2020-01-01 00:02:00"]
- ["2020-01-01 00:03:00"]
- ["2020-01-01 00:04:00"]
- ["2020-01-01 00:05:00"]
- ["2020-01-01 00:06:00"]
- ["2020-01-01 00:07:00"]
- ["2020-01-01 00:08:00"]
- ["2020-01-01 00:09:00"]
- ["2020-01-01 00:10:00"]
- ["2020-01-01 00:11:00"]
- ["2020-01-01 00:12:00"]
- ["2020-01-01 00:13:00"]
- ["2020-01-01 00:14:00"]
- ["2020-01-01 00:15:00"]
- ["2020-01-01 00:16:00"]
- ["2020-01-01 00:17:00"]
- ["2020-01-01 00:18:00"]
- ["2020-01-01 00:19:00"]
- ["2020-01-01 00:20:00"]
- ["2020-01-01 00:21:00"]
- ["2020-01-01 00:22:00"]
- ["2020-01-01 00:23:00"]
- ["2020-01-01 00:24:00"]
- ["2020-01-01 00:25:00"]
- ["2020-01-01 00:26:00"]
- ["2020-01-01 00:27:00"]
- ["2020-01-01 00:28:00"]
- ["2020-01-01 00:29:00"]
- ["2020-01-01 00:30:00"]
36 changes: 36 additions & 0 deletions ...metricflow/fixtures/source_table_snapshots/time_spine_table/mf_time_spine_nanosecond.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
table_snapshot:
table_name: mf_time_spine_nanosecond
column_definitions:
- name: ts
type: TIME
rows:
- ["2020-01-01 00:00:00.000000001"]
- ["2020-01-01 00:00:00.000000002"]
- ["2020-01-01 00:00:00.000000003"]
- ["2020-01-01 00:00:00.000000004"]
- ["2020-01-01 00:00:00.000000005"]
- ["2020-01-01 00:00:00.000000006"]
- ["2020-01-01 00:00:00.000000007"]
- ["2020-01-01 00:00:00.000000008"]
- ["2020-01-01 00:00:00.000000009"]
- ["2020-01-01 00:00:00.000000010"]
- ["2020-01-01 00:00:00.000000011"]
- ["2020-01-01 00:00:00.000000012"]
- ["2020-01-01 00:00:00.000000013"]
- ["2020-01-01 00:00:00.000000014"]
- ["2020-01-01 00:00:00.000000015"]
- ["2020-01-01 00:00:00.000000016"]
- ["2020-01-01 00:00:00.000000017"]
- ["2020-01-01 00:00:00.000000018"]
- ["2020-01-01 00:00:00.000000019"]
- ["2020-01-01 00:00:00.000000020"]
- ["2020-01-01 00:00:00.000000021"]
- ["2020-01-01 00:00:00.000000022"]
- ["2020-01-01 00:00:00.000000023"]
- ["2020-01-01 00:00:00.000000024"]
- ["2020-01-01 00:00:00.000000025"]
- ["2020-01-01 00:00:00.000000026"]
- ["2020-01-01 00:00:00.000000027"]
- ["2020-01-01 00:00:00.000000028"]
- ["2020-01-01 00:00:00.000000029"]
- ["2020-01-01 00:00:00.000000030"]
36 changes: 36 additions & 0 deletions tests_metricflow/fixtures/source_table_snapshots/time_spine_table/mf_time_spine_second.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
table_snapshot:
table_name: mf_time_spine_second
column_definitions:
- name: ts
type: TIME
rows:
- ["2020-01-01 00:00:01"]
- ["2020-01-01 00:00:02"]
- ["2020-01-01 00:00:03"]
- ["2020-01-01 00:00:04"]
- ["2020-01-01 00:00:05"]
- ["2020-01-01 00:00:06"]
- ["2020-01-01 00:00:07"]
- ["2020-01-01 00:00:08"]
- ["2020-01-01 00:00:09"]
- ["2020-01-01 00:00:10"]
- ["2020-01-01 00:00:11"]
- ["2020-01-01 00:00:12"]
- ["2020-01-01 00:00:13"]
- ["2020-01-01 00:00:14"]
- ["2020-01-01 00:00:15"]
- ["2020-01-01 00:00:16"]
- ["2020-01-01 00:00:17"]
- ["2020-01-01 00:00:18"]
- ["2020-01-01 00:00:19"]
- ["2020-01-01 00:00:20"]
- ["2020-01-01 00:00:21"]
- ["2020-01-01 00:00:22"]
- ["2020-01-01 00:00:23"]
- ["2020-01-01 00:00:24"]
- ["2020-01-01 00:00:25"]
- ["2020-01-01 00:00:26"]
- ["2020-01-01 00:00:27"]
- ["2020-01-01 00:00:28"]
- ["2020-01-01 00:00:29"]
- ["2020-01-01 00:00:30"]
Loading