From 2ee3de1baf5d733c219320477e63a7ae7f19d401 Mon Sep 17 00:00:00 2001 From: Grace Goheen Date: Tue, 20 Feb 2024 15:49:32 -0700 Subject: [PATCH 1/6] add documentation for unit testing incremental models --- website/docs/docs/build/unit-tests.md | 67 +++++++++++++++++++ .../resource-properties/unit-tests.md | 4 ++ 2 files changed, 71 insertions(+) diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index 6c83e435290..0f20036b3ce 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -254,5 +254,72 @@ unit_tests: ``` +## Unit testing incremental models +When configuring your unit test, you can override the output of macros, vars, or environment variables for a given unit test. This enables you to unit test your incremental models - both in "full refresh" and "incremental" mode. +When testing an incremental model, the expected output is the __result of the materialization__ (i.e. what will be merged/inserted), not the resulting model itself (i.e. what the final table will look like after the merge/insert). + +Let's say I have an incremental model in my project: + + + +```sql + +{{ + config( + materialized='incremental' + ) +}} + +select * from {{ ref('events') }} +{% if is_incremental() %} +where event_time > (select max(event_time) from {{ this }}) +{% endif %} + +``` + +I can define unit tests on `my_incremental_model` to ensure my incremental logic is working as expected: + +```yml + +unit_tests: + - name: my_incremental_model_full_refresh_mode + model: my_incremental_model + overrides: + macros: + # unit test this model in "full refresh" mode + is_incremental: false + given: + - input: ref('events') + rows: + - {event_id: 1, event_time: 2020-01-01} + expect: + rows: + - {event_id: 1, event_time: 2020-01-01} + + - name: my_incremental_model_incremental_mode + model: my_incremental_model + overrides: + macros: + # unit test this model in "incremental" mode + is_incremental: true + given: + - input: ref('events') + rows: + - {event_id: 1, event_time: 2020-01-01} + - {event_id: 2, event_time: 2020-01-02} + - {event_id: 3, event_time: 2020-01-03} + - input: this + # contents of current my_incremental_model + rows: + - {event_id: 1, event_time: 2020-01-01} + expect: + # what will be inserted/merged into my_incremental_model + rows: + - {event_id: 2, event_time: 2020-01-02} + - {event_id: 3, event_time: 2020-01-03} + +``` + +There is currently no way to unit test whether the dbt framework actually inserted/merged the records into your existing model correctly or not - but [we're investigating a way to support this in the future](https://github.com/dbt-labs/dbt-core/issues/8664). \ No newline at end of file diff --git a/website/docs/reference/resource-properties/unit-tests.md b/website/docs/reference/resource-properties/unit-tests.md index 40c3414e373..6c112b99924 100644 --- a/website/docs/reference/resource-properties/unit-tests.md +++ b/website/docs/reference/resource-properties/unit-tests.md @@ -153,6 +153,10 @@ When `format: csv`, can either supply: - If you do supply an input for a seed, we will use that input instead. - You can also have “empty” inputs, by setting rows to an empty list `rows: []` +### Overrides + +When configuring your unit test, you can override the output of macros, vars, or environment variables for a given unit test. + ## Examples ```yml From 054fab985a547687e38ba5825e5fa31ac5eca2a3 Mon Sep 17 00:00:00 2001 From: Grace Goheen <53586774+graciegoheen@users.noreply.github.com> Date: Wed, 21 Feb 2024 10:44:21 -0700 Subject: [PATCH 2/6] Apply suggestions from code review Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/docs/docs/build/unit-tests.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index 0f20036b3ce..bc140f47d06 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -256,11 +256,11 @@ unit_tests: ## Unit testing incremental models -When configuring your unit test, you can override the output of macros, vars, or environment variables for a given unit test. This enables you to unit test your incremental models - both in "full refresh" and "incremental" mode. +When configuring your unit test, you can override the output of macros, vars, or environment variables. This enables you to unit test your incremental models in "full refresh" and "incremental" modes. -When testing an incremental model, the expected output is the __result of the materialization__ (i.e. what will be merged/inserted), not the resulting model itself (i.e. what the final table will look like after the merge/insert). +When testing an incremental model, the expected output is the __result of the materialization__ (what will be merged/inserted), not the resulting model itself (what the final table will look like after the merge/insert). -Let's say I have an incremental model in my project: +For example, say you have an incremental model in your project: @@ -279,7 +279,7 @@ where event_time > (select max(event_time) from {{ this }}) ``` -I can define unit tests on `my_incremental_model` to ensure my incremental logic is working as expected: +You can define unit tests on `my_incremental_model` to ensure your incremental logic is working as expected: ```yml From 1b7b31aad62f7be70b1c7a65aa344ec1b5f1d10e Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 21 Feb 2024 12:52:05 -0500 Subject: [PATCH 3/6] Update website/docs/docs/build/unit-tests.md --- website/docs/docs/build/unit-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index bc140f47d06..ac169c91673 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -322,4 +322,4 @@ unit_tests: ``` -There is currently no way to unit test whether the dbt framework actually inserted/merged the records into your existing model correctly or not - but [we're investigating a way to support this in the future](https://github.com/dbt-labs/dbt-core/issues/8664). \ No newline at end of file +There is currently no way to unit test whether the dbt framework inserted/merged the records into your existing model correctly, but [we're investigating support for this in the future](https://github.com/dbt-labs/dbt-core/issues/8664). \ No newline at end of file From 9e81f3488a884c3030b6fbdf24aee4d3769a159c Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 21 Feb 2024 13:31:30 -0500 Subject: [PATCH 4/6] Update unit-tests.md Missed closing the file --- website/docs/docs/build/unit-tests.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index ac169c91673..19dfebf7d9a 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -279,6 +279,8 @@ where event_time > (select max(event_time) from {{ this }}) ``` + + You can define unit tests on `my_incremental_model` to ensure your incremental logic is working as expected: ```yml @@ -322,4 +324,4 @@ unit_tests: ``` -There is currently no way to unit test whether the dbt framework inserted/merged the records into your existing model correctly, but [we're investigating support for this in the future](https://github.com/dbt-labs/dbt-core/issues/8664). \ No newline at end of file +There is currently no way to unit test whether the dbt framework inserted/merged the records into your existing model correctly, but [we're investigating support for this in the future](https://github.com/dbt-labs/dbt-core/issues/8664). From bf296d443199636b48e31d3a59f939458205f9d9 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 21 Feb 2024 13:46:14 -0500 Subject: [PATCH 5/6] Update website/docs/docs/build/unit-tests.md --- website/docs/docs/build/unit-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index 19dfebf7d9a..480ad565511 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -279,7 +279,7 @@ where event_time > (select max(event_time) from {{ this }}) ``` - + You can define unit tests on `my_incremental_model` to ensure your incremental logic is working as expected: From c987fc53cd4548cdfa17b39c08dd81caf8e7f135 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 21 Feb 2024 13:46:20 -0500 Subject: [PATCH 6/6] Update website/docs/docs/build/unit-tests.md --- website/docs/docs/build/unit-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index 480ad565511..7981108dc71 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -262,7 +262,7 @@ When testing an incremental model, the expected output is the __result of the ma For example, say you have an incremental model in your project: - + ```sql