Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User-defined custom incremental strategies #4716

Merged
merged 15 commits into from
Jan 8, 2024
Merged

User-defined custom incremental strategies #4716

Changes from 9 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
9f53cff
User-defined custom incremental strategies
dbeatty10 Jan 6, 2024
7768ec5
Example of a custom incremental strategy
dbeatty10 Jan 6, 2024
0efb82d
Using custom strategies from a package
dbeatty10 Jan 7, 2024
f581838
Remove curlies
dbeatty10 Jan 7, 2024
63104c9
Add an example file name
dbeatty10 Jan 7, 2024
ea15804
Fix link
dbeatty10 Jan 7, 2024
17d7838
Merge branch 'current' into dbeatty/custom-incremental-strategies
mirnawong1 Jan 8, 2024
3689f47
Merge branch 'current' into dbeatty/custom-incremental-strategies
mirnawong1 Jan 8, 2024
ebff49a
Merge branch 'current' into dbeatty/custom-incremental-strategies
mirnawong1 Jan 8, 2024
e78ffec
Merge branch 'current' into dbeatty/custom-incremental-strategies
mirnawong1 Jan 8, 2024
e9a0422
Merge branch 'current' into dbeatty/custom-incremental-strategies
mirnawong1 Jan 8, 2024
59345e4
Update incremental-models.md
mirnawong1 Jan 8, 2024
f5579f9
Update website/docs/docs/build/incremental-models.md
mirnawong1 Jan 8, 2024
285915a
Update website/docs/docs/build/incremental-models.md
mirnawong1 Jan 8, 2024
89bc5e1
Update incremental-models.md
mirnawong1 Jan 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions website/docs/docs/build/incremental-models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,5 +450,78 @@ The syntax depends on how you configure your `incremental_strategy`:

</VersionBlock>

### Custom strategies

<VersionBlock lastVersion="1.1">

Custom incremental strategies can be defined beginning in dbt v1.2.

</VersionBlock>

<VersionBlock firstVersion="1.2">

As an easier alternative to [creating an entirely new materialization](/guides/create-new-materializations), users can define and use their own "custom" user-defined incremental strategies by:

1. defining a macro named `get_incremental_STRATEGY_sql`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we saying the macro should always be named get_incremental_STRATEGY_sql?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or is STRATEGY a placeholder? if its a placeholder, we should maybe be more clear on that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, STRATEGY is a placeholder!

Any ideas how to make it more clear?

Along the lines of your suggestion about the table, you think adding adding it would make it more clear?

Screenshot 2024-01-08 at 12 10 43

Copy link
Contributor

@mirnawong1 mirnawong1 Jan 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok gotcha, thank you for clarifying! waht do you think about the following suggestion @dbeatty10 :

Built-in strategies and their corresponding macros

Before diving into custom strategies, it's important to understand the built-in incremental strategies in dbt and their corresponding macros:

incremental_strategy Corresponding macro
append get_incremental_append_sql
delete+insert get_incremental_delete_insert_sql
merge get_incremental_merge_sql
insert_overwrite get_incremental_insert_overwrite_sql

For example, a built-in strategy for the append can be defined and used with the following files:

{% macro get_incremental_append_sql(arg_dict) %}

  {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %}

{% endmacro %}


{% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %}

    {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%}

    insert into {{ target_relation }} ({{ dest_cols_csv }})
    (
        select {{ dest_cols_csv }}
        from {{ temp_relation }}
    )

{% endmacro %}

Define a model models/my_model.sql:

{{ config(
    materialized="incremental",
    incremental_strategy="append",
) }}

select * from {{ ref("some_model") }}

Custom strategies

Custom incremental strategies can be defined beginning in dbt v1.2.

As an easier alternative to creating an entirely new materialization,
users can define and use their own "custom" user-defined incremental strategies by:

  1. defining a macro named get_incremental_STRATEGY_sql. Note that STRATEGY is a placeholder and you should replace it with the name of your custom incremental strategy.
  2. configuring incremental_strategy: STRATEGY within an incremental model

dbt won't validate user-defined strategies, it will just look for the macro by that name, and raise an error if it can't find one.

For example,.....

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok @dbeatty10 proposing the above adn before i make the changes or sugg - please let me know if that makes sense to you 🙏

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good 🚀

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok cool, will make that change!

2. configuring `incremental_strategy: STRATEGY` within an incremental model


For example, a user-defined strategy named `insert_only` can be defined and used with the following files:

<File name='macros/my_custom_strategies.sql'>

```sql
{% macro get_incremental_insert_only_sql(arg_dict) %}

{% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %}

{% endmacro %}


{% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %}

{%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%}

insert into {{ target_relation }} ({{ dest_cols_csv }})
(
select {{ dest_cols_csv }}
from {{ temp_relation }}
)

{% endmacro %}
```

</File>

<File name='models/my_model.sql'>

```sql
{{ config(
materialized="incremental",
incremental_strategy="insert_only",
...
) }}

...
```

</File>

### Custom strategies from a package

To use the `merge_null_safe` custom incremental strategy from the `example` package, first [install the package](/docs/build/packages#how-do-i-add-a-package-to-my-project), then add this macro to your project:
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<File name='macros/my_custom_strategies.sql'>

```sql
{% macro get_incremental_merge_null_safe_sql(arg_dict) %}
{% do return(example.get_incremental_merge_null_safe_sql(arg_dict)) %}
{% endmacro %}
```

</File>

</VersionBlock>

<Snippet path="discourse-help-feed-header" />
<DiscourseHelpFeed tags="incremental"/>
Loading