Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace uses of DataFrame with MetricflowDataTable #1235

Merged
merged 15 commits into from
Jun 3, 2024
Merged

Replace uses of DataFrame with MetricflowDataTable #1235

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
91bb0fb
/* PR_START p--py312 04 */ Add assertion for intergration tests to ve…
plypaul May 31, 2024
b291444
Use `.text_format()` for data table to string conversion.
plypaul May 31, 2024
e464d74
Migrate various uses of dataframes to the data table.
plypaul May 28, 2024
eeb57c1
Update `SqlTableSnapshot` to use `MetricFlowDataTable`.
plypaul May 31, 2024
152b6ce
Update snapshots for new test-table format.
plypaul May 8, 2024
3ed8f0c
Rename strings similar to "dataframe".
plypaul May 8, 2024
95e0e7e
Update snapshots for `WriteToResultDataframeNode` rename.
plypaul May 8, 2024
ae53659
Rename `write_to_dataframe.py` to `write_to_data_table.py`.
plypaul May 8, 2024
47f4866
Rename `SelectSqlQueryToDataFrameTask` to `SelectSqlQueryToDataTableT…
plypaul May 29, 2024
1808631
Update snapshots for rename to `SelectSqlQueryToDataTableTask`.
plypaul May 29, 2024
b6a313a
Rename `as_df` to `as_data_table`.
plypaul May 29, 2024
a3c2110
Update `SnowflakeInferenceContextProvider` to use correct types.
plypaul May 28, 2024
0ece9ef
Update the CLI to use corresponding methods for the data table.
plypaul May 30, 2024
19beaba
Address comments.
plypaul Jun 3, 2024
027afbb
Update snapshots.
plypaul Jun 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions tests_metricflow/snapshots/test_cli.py/str/BigQuery/test_saved_query__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00 | 5 | 1 | 0 |
| 2019-12-18 00:00:00 | 4 | 4 | 2 |
| 2019-12-19 00:00:00 | 4 | 6 | 6 |
| 2019-12-19 00:00:00 | 5 | 2 | 0 |
| 2019-12-20 00:00:00 | 5 | 2 | 0 |
| 2020-01-01 00:00:00 | 4 | 2 | 1 |
| 2020-01-02 00:00:00 | 4 | 3 | 3 |
| 2020-01-02 00:00:00 | 5 | 1 | 0 |
| 2020-01-03 00:00:00 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh interesting. Is that because the datetime/dateutil objects don't format in 0 values for time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually due to a case in the text formatting logic. In retrospect, let me remove that.

2019-12-18 4 4 2
2019-12-19 4 6 6
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-01 4 2 1
2020-01-02 4 3 3
2020-01-02 5 1 0
2020-01-03 5 1 0
6 changes: 3 additions & 3 deletions ...napshots/test_cli.py/str/BigQuery/test_saved_query_with_cumulative_metric__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
| metric_time__day | trailing_2_months_revenue |
|:--------------------|----------------------------:|
| 2020-01-01 00:00:00 | 1000 |
metric_time__day trailing_2_months_revenue
------------------ ---------------------------
2020-01-01 1000
10 changes: 5 additions & 5 deletions ...metricflow/snapshots/test_cli.py/str/BigQuery/test_saved_query_with_limit__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00 | 5 | 1 | 0 |
| 2019-12-18 00:00:00 | 4 | 4 | 2 |
| 2019-12-19 00:00:00 | 4 | 6 | 6 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
14 changes: 7 additions & 7 deletions ...metricflow/snapshots/test_cli.py/str/BigQuery/test_saved_query_with_where__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00 | 5 | 1 | 0 |
| 2019-12-19 00:00:00 | 5 | 2 | 0 |
| 2019-12-20 00:00:00 | 5 | 2 | 0 |
| 2020-01-02 00:00:00 | 5 | 1 | 0 |
| 2020-01-03 00:00:00 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-02 5 1 0
2020-01-03 5 1 0
22 changes: 11 additions & 11 deletions tests_metricflow/snapshots/test_cli.py/str/Databricks/test_saved_query__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00+00:00 | 5 | 1 | 0 |
| 2019-12-18 00:00:00+00:00 | 4 | 4 | 2 |
| 2019-12-19 00:00:00+00:00 | 4 | 6 | 6 |
| 2019-12-19 00:00:00+00:00 | 5 | 2 | 0 |
| 2019-12-20 00:00:00+00:00 | 5 | 2 | 0 |
| 2020-01-01 00:00:00+00:00 | 4 | 2 | 1 |
| 2020-01-02 00:00:00+00:00 | 4 | 3 | 3 |
| 2020-01-02 00:00:00+00:00 | 5 | 1 | 0 |
| 2020-01-03 00:00:00+00:00 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-01 4 2 1
2020-01-02 4 3 3
2020-01-02 5 1 0
2020-01-03 5 1 0
6 changes: 3 additions & 3 deletions ...pshots/test_cli.py/str/Databricks/test_saved_query_with_cumulative_metric__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
| metric_time__day | trailing_2_months_revenue |
|:--------------------------|----------------------------:|
| 2020-01-01 00:00:00+00:00 | 1000 |
metric_time__day trailing_2_months_revenue
------------------ ---------------------------
2020-01-01 1000
10 changes: 5 additions & 5 deletions ...tricflow/snapshots/test_cli.py/str/Databricks/test_saved_query_with_limit__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00+00:00 | 5 | 1 | 0 |
| 2019-12-18 00:00:00+00:00 | 4 | 4 | 2 |
| 2019-12-19 00:00:00+00:00 | 4 | 6 | 6 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
14 changes: 7 additions & 7 deletions ...tricflow/snapshots/test_cli.py/str/Databricks/test_saved_query_with_where__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00+00:00 | 5 | 1 | 0 |
| 2019-12-19 00:00:00+00:00 | 5 | 2 | 0 |
| 2019-12-20 00:00:00+00:00 | 5 | 2 | 0 |
| 2020-01-02 00:00:00+00:00 | 5 | 1 | 0 |
| 2020-01-03 00:00:00+00:00 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-02 5 1 0
2020-01-03 5 1 0
22 changes: 11 additions & 11 deletions tests_metricflow/snapshots/test_cli.py/str/DuckDB/test_saved_query__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:-------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 | 5 | 1 | 0 |
| 2019-12-18 | 4 | 4 | 2 |
| 2019-12-19 | 4 | 6 | 6 |
| 2019-12-19 | 5 | 2 | 0 |
| 2019-12-20 | 5 | 2 | 0 |
| 2020-01-01 | 4 | 2 | 1 |
| 2020-01-02 | 4 | 3 | 3 |
| 2020-01-02 | 5 | 1 | 0 |
| 2020-01-03 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-01 4 2 1
2020-01-02 4 3 3
2020-01-02 5 1 0
2020-01-03 5 1 0
6 changes: 3 additions & 3 deletions .../snapshots/test_cli.py/str/DuckDB/test_saved_query_with_cumulative_metric__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
| metric_time__day | trailing_2_months_revenue |
|:--------------------|----------------------------:|
| 2020-01-01 00:00:00 | 1000 |
metric_time__day trailing_2_months_revenue
------------------ ---------------------------
2020-01-01 1000
10 changes: 5 additions & 5 deletions ...s_metricflow/snapshots/test_cli.py/str/DuckDB/test_saved_query_with_limit__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:-------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 | 5 | 1 | 0 |
| 2019-12-18 | 4 | 4 | 2 |
| 2019-12-19 | 4 | 6 | 6 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
14 changes: 7 additions & 7 deletions ...s_metricflow/snapshots/test_cli.py/str/DuckDB/test_saved_query_with_where__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:-------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 | 5 | 1 | 0 |
| 2019-12-19 | 5 | 2 | 0 |
| 2019-12-20 | 5 | 2 | 0 |
| 2020-01-02 | 5 | 1 | 0 |
| 2020-01-03 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-02 5 1 0
2020-01-03 5 1 0
22 changes: 11 additions & 11 deletions tests_metricflow/snapshots/test_cli.py/str/Postgres/test_saved_query__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00 | 5 | 1 | 0 |
| 2019-12-18 00:00:00 | 4 | 4 | 2 |
| 2019-12-19 00:00:00 | 4 | 6 | 6 |
| 2019-12-19 00:00:00 | 5 | 2 | 0 |
| 2019-12-20 00:00:00 | 5 | 2 | 0 |
| 2020-01-01 00:00:00 | 4 | 2 | 1 |
| 2020-01-02 00:00:00 | 4 | 3 | 3 |
| 2020-01-02 00:00:00 | 5 | 1 | 0 |
| 2020-01-03 00:00:00 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-01 4 2 1
2020-01-02 4 3 3
2020-01-02 5 1 0
2020-01-03 5 1 0
6 changes: 3 additions & 3 deletions ...napshots/test_cli.py/str/Postgres/test_saved_query_with_cumulative_metric__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
| metric_time__day | trailing_2_months_revenue |
|:--------------------|----------------------------:|
| 2020-01-01 00:00:00 | 1000.00 |
metric_time__day trailing_2_months_revenue
------------------ ---------------------------
2020-01-01 1000
10 changes: 5 additions & 5 deletions ...metricflow/snapshots/test_cli.py/str/Postgres/test_saved_query_with_limit__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00 | 5 | 1 | 0 |
| 2019-12-18 00:00:00 | 4 | 4 | 2 |
| 2019-12-19 00:00:00 | 4 | 6 | 6 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
14 changes: 7 additions & 7 deletions ...metricflow/snapshots/test_cli.py/str/Postgres/test_saved_query_with_where__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00 | 5 | 1 | 0 |
| 2019-12-19 00:00:00 | 5 | 2 | 0 |
| 2019-12-20 00:00:00 | 5 | 2 | 0 |
| 2020-01-02 00:00:00 | 5 | 1 | 0 |
| 2020-01-03 00:00:00 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-02 5 1 0
2020-01-03 5 1 0
22 changes: 11 additions & 11 deletions tests_metricflow/snapshots/test_cli.py/str/Redshift/test_saved_query__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00 | 5 | 1 | 0 |
| 2019-12-18 00:00:00 | 4 | 4 | 2 |
| 2019-12-19 00:00:00 | 4 | 6 | 6 |
| 2019-12-19 00:00:00 | 5 | 2 | 0 |
| 2019-12-20 00:00:00 | 5 | 2 | 0 |
| 2020-01-01 00:00:00 | 4 | 2 | 1 |
| 2020-01-02 00:00:00 | 4 | 3 | 3 |
| 2020-01-02 00:00:00 | 5 | 1 | 0 |
| 2020-01-03 00:00:00 | 5 | 1 | 0 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
2019-12-19 5 2 0
2019-12-20 5 2 0
2020-01-01 4 2 1
2020-01-02 4 3 3
2020-01-02 5 1 0
2020-01-03 5 1 0
6 changes: 3 additions & 3 deletions ...napshots/test_cli.py/str/Redshift/test_saved_query_with_cumulative_metric__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
| metric_time__day | trailing_2_months_revenue |
|:--------------------|----------------------------:|
| 2020-01-01 00:00:00 | 1000 |
metric_time__day trailing_2_months_revenue
------------------ ---------------------------
2020-01-01 1000
10 changes: 5 additions & 5 deletions ...metricflow/snapshots/test_cli.py/str/Redshift/test_saved_query_with_limit__cli_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
| metric_time__day | listing__capacity_latest | bookings | instant_bookings |
|:--------------------|---------------------------:|-----------:|-------------------:|
| 2019-12-01 00:00:00 | 5 | 1 | 0 |
| 2019-12-18 00:00:00 | 4 | 4 | 2 |
| 2019-12-19 00:00:00 | 4 | 6 | 6 |
metric_time__day listing__capacity_latest bookings instant_bookings
------------------ -------------------------- ---------- ------------------
2019-12-01 5 1 0
2019-12-18 4 4 2
2019-12-19 4 6 6
Loading