From 9f53cff09ed00a8bc32b5b4bb338b651ce5aa081 Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Fri, 5 Jan 2024 19:23:40 -0700 Subject: [PATCH 01/10] User-defined custom incremental strategies --- website/docs/docs/build/incremental-models.md | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index cc45290ae15..42bf4126a72 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -450,5 +450,62 @@ The syntax depends on how you configure your `incremental_strategy`: +### Custom strategies + + + +Custom incremental strategies can be defined beginning in dbt v1.2. + + + + + +As an easier alternative to [creating an entirely new materialization](/guides/create-new-materializations), users can define and use their own "custom" user-defined incremental strategies by: + +1. defining a macro named `get_incremental_{STRATEGY}_sql` +2. configuring `incremental_strategy: {STRATEGY}` within an incremental model + + + + +```sql +{% macro get_incremental_insert_only_sql(arg_dict) %} + + {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %} + +{% endmacro %} + + +{% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %} + + {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%} + + insert into {{ target_relation }} ({{ dest_cols_csv }}) + ( + select {{ dest_cols_csv }} + from {{ temp_relation }} + ) + +{% endmacro %} +``` + + + + + +```sql +{{ config( + materialized="incremental", + incremental_strategy="insert_only", + ... +) }} + +... +``` + + + + + From 7768ec5218cbb36a5e68b557d64bc6f0d84437e1 Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Fri, 5 Jan 2024 19:54:19 -0700 Subject: [PATCH 02/10] Example of a custom incremental strategy --- website/docs/docs/build/incremental-models.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 42bf4126a72..b6434f7d9d4 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -466,6 +466,8 @@ As an easier alternative to [creating an entirely new materialization](/guides/c 2. configuring `incremental_strategy: {STRATEGY}` within an incremental model +For example, a user-defined strategy named `insert_only` can be defined and used with the following files: + ```sql From 0efb82d87384ae8a2dc6f1d8606487e21ba270c8 Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Sat, 6 Jan 2024 18:01:15 -0700 Subject: [PATCH 03/10] Using custom strategies from a package --- website/docs/docs/build/incremental-models.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index b6434f7d9d4..d31276b8a56 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -507,6 +507,16 @@ For example, a user-defined strategy named `insert_only` can be defined and used +### Custom strategies from a package + +To use the `merge_null_safe` custom incremental strategy from the `example` package, first [install the package](/build/packages#how-do-i-add-a-package-to-my-project), then add this macro to your project: + +```sql +{% macro get_incremental_merge_null_safe_sql(arg_dict) %} + {% do return(example.get_incremental_merge_null_safe_sql(arg_dict)) %} +{% endmacro %} +``` + From f5818389e77eedba12ee2ee4caada2c428edc404 Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Sat, 6 Jan 2024 18:02:52 -0700 Subject: [PATCH 04/10] Remove curlies --- website/docs/docs/build/incremental-models.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index d31276b8a56..6a94e957b12 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -462,8 +462,8 @@ Custom incremental strategies can be defined beginning in dbt v1.2. As an easier alternative to [creating an entirely new materialization](/guides/create-new-materializations), users can define and use their own "custom" user-defined incremental strategies by: -1. defining a macro named `get_incremental_{STRATEGY}_sql` -2. configuring `incremental_strategy: {STRATEGY}` within an incremental model +1. defining a macro named `get_incremental_STRATEGY_sql` +2. configuring `incremental_strategy: STRATEGY` within an incremental model For example, a user-defined strategy named `insert_only` can be defined and used with the following files: From 63104c9ff23d1b2f7366d6f58d882f13aa9dd96d Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Sat, 6 Jan 2024 18:04:02 -0700 Subject: [PATCH 05/10] Add an example file name --- website/docs/docs/build/incremental-models.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 6a94e957b12..b5c0a34e9a2 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -511,12 +511,16 @@ For example, a user-defined strategy named `insert_only` can be defined and used To use the `merge_null_safe` custom incremental strategy from the `example` package, first [install the package](/build/packages#how-do-i-add-a-package-to-my-project), then add this macro to your project: + + ```sql {% macro get_incremental_merge_null_safe_sql(arg_dict) %} {% do return(example.get_incremental_merge_null_safe_sql(arg_dict)) %} {% endmacro %} ``` + + From ea158040916146ba2326621114b8082c7c244299 Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Sat, 6 Jan 2024 18:22:43 -0700 Subject: [PATCH 06/10] Fix link --- website/docs/docs/build/incremental-models.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index b5c0a34e9a2..10435a39453 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -509,7 +509,7 @@ For example, a user-defined strategy named `insert_only` can be defined and used ### Custom strategies from a package -To use the `merge_null_safe` custom incremental strategy from the `example` package, first [install the package](/build/packages#how-do-i-add-a-package-to-my-project), then add this macro to your project: +To use the `merge_null_safe` custom incremental strategy from the `example` package, first [install the package](/docs/build/packages#how-do-i-add-a-package-to-my-project), then add this macro to your project: From 59345e440f828882bef20e2d87cda7564739e750 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 8 Jan 2024 15:46:56 +0000 Subject: [PATCH 07/10] Update incremental-models.md folding in changes --- website/docs/docs/build/incremental-models.md | 52 ++++++++++++++++++- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 10435a39453..319da658a40 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -236,7 +236,7 @@ Instead, whenever the logic of your incremental changes, execute a full-refresh ## About `incremental_strategy` -There are various ways (strategies) to implement the concept of an incremental materializations. The value of each strategy depends on: +There are various ways (strategies) to implement the concept of incremental materializations. The value of each strategy depends on: * the volume of data, * the reliability of your `unique_key`, and @@ -450,6 +450,53 @@ The syntax depends on how you configure your `incremental_strategy`: +### Built-in strategies and their corresponding macros + +Before diving into [custom strategies](#custom-strategies), it's important to understand the built-in incremental strategies in dbt and their corresponding macros: + +| `incremental_strategy` | Corresponding macro | +|------------------------|----------------------------------------| +| `append` | `get_incremental_append_sql` | +| `delete+insert` | `get_incremental_delete_insert_sql` | +| `merge` | `get_incremental_merge_sql` | +| `insert_overwrite` | `get_incremental_insert_overwrite_sql` | + + +For example, a built-in strategy for the `append` can be defined and used with the following files: + + + +```sql +{% macro get_incremental_append_sql(arg_dict) %} + + {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %} + +{% endmacro %} + + +{% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %} + + {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%} + + insert into {{ target_relation }} ({{ dest_cols_csv }}) + ( + select {{ dest_cols_csv }} + from {{ temp_relation }} + ) + +{% endmacro %} +``` +Define a model models/my_model.sql: + +```sql +{{ config( + materialized="incremental", + incremental_strategy="append", +) }} + +select * from {{ ref("some_model") }} +``` + ### Custom strategies @@ -462,9 +509,10 @@ Custom incremental strategies can be defined beginning in dbt v1.2. As an easier alternative to [creating an entirely new materialization](/guides/create-new-materializations), users can define and use their own "custom" user-defined incremental strategies by: -1. defining a macro named `get_incremental_STRATEGY_sql` +1. defining a macro named `get_incremental_STRATEGY_sql`. Note that `STRATEGY` is a placeholder and you should replace it with the name of your custom incremental strategy. 2. configuring `incremental_strategy: STRATEGY` within an incremental model +dbt won't validate user-defined strategies, it will just look for the macro by that name, and raise an error if it can't find one. For example, a user-defined strategy named `insert_only` can be defined and used with the following files: From f5579f900bdd84f0334dc8b15953f23bfcca491f Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 8 Jan 2024 15:47:28 +0000 Subject: [PATCH 08/10] Update website/docs/docs/build/incremental-models.md --- website/docs/docs/build/incremental-models.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 319da658a40..70dd6039b12 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -450,7 +450,7 @@ The syntax depends on how you configure your `incremental_strategy`: -### Built-in strategies and their corresponding macros +### Built-in strategies Before diving into [custom strategies](#custom-strategies), it's important to understand the built-in incremental strategies in dbt and their corresponding macros: From 285915a4f8a3549a1318bc91310cee616393fa95 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 8 Jan 2024 15:47:57 +0000 Subject: [PATCH 09/10] Update website/docs/docs/build/incremental-models.md --- website/docs/docs/build/incremental-models.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 70dd6039b12..927d4b93889 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -557,7 +557,9 @@ For example, a user-defined strategy named `insert_only` can be defined and used ### Custom strategies from a package -To use the `merge_null_safe` custom incremental strategy from the `example` package, first [install the package](/docs/build/packages#how-do-i-add-a-package-to-my-project), then add this macro to your project: +To use the `merge_null_safe` custom incremental strategy from the `example` package: +- [Install the package](/docs/build/packages#how-do-i-add-a-package-to-my-project) +- Then add the following macro to your project: From 89bc5e1b84925438f74eba62d6accf03896df865 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 8 Jan 2024 15:54:19 +0000 Subject: [PATCH 10/10] Update incremental-models.md --- website/docs/docs/build/incremental-models.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 927d4b93889..9f1c206f5fb 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -486,6 +486,8 @@ For example, a built-in strategy for the `append` can be defined and used with t {% endmacro %} ``` + + Define a model models/my_model.sql: ```sql @@ -570,7 +572,6 @@ To use the `merge_null_safe` custom incremental strategy from the `example` pack ``` -