-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CT-3183] [Feature] custom incremental strategy from dbt package #8769
Comments
Thanks for opening this @sudo-pradip ! This is essentially one of the "outstanding questions" listed here:
Let's suppose you add a custom macro named Did you already try overriding global macros within dispatch:
- macro_namespace: dbt
search_order: ['my_project', 'dbt_materializations', 'dbt'] I haven't tried this myself, so curious if you already tried it. |
Hello @dbeatty10 ,
|
Finding the right macro
You're probably onto something here! If that area of the code doesn't respect {% macro get_incremental_merge_null_safe_sql(arg_dict) %}
{{ dbt_materializations.get_incremental_merge_null_safe_sql(arg_dict) }}
{% endmacro %} Do you have any ideas / opinions about re-implementation of Usage of dispatchIn general, you'd need a separate dispatch:
- macro_namespace: dbt_utils
search_order: ['my_project', 'dbt_utils', 'dbt']
- macro_namespace: dbt_expectations
search_order: ['my_project', 'dbt_expectations', 'dbt'] Wanna give that dispatch config a try with |
Current
Expectation
My assumptions
Possible solutions
Solution 1 we can think, but we have to duplicate our logic at two places (i think it's all right ?), whats your input on this ? |
Thanks for doing this research @sudo-pradip 🧠 I'll either need to grab an extra set of expert eyes from an engineer on the dbt-core team do some deeper research myself. Labeling this as "Refinement" accordingly. All of us are time-constrained this week and next due to the upcoming release of dbt-core of 1.7.0 and the Coalesce conference next week, so it could be a few weeks (or more) until we can dive deeper into this. In the meantime, your best bet is the alternative you've already considered:
|
any updates ? |
@sudo-pradip we're not able to prioritize further research at this time. So we've opened dbt-labs/docs.getdbt.com#4716 to document the installation steps that work currently. |
[Preview](https://docs-getdbt-com-git-dbeatty-custom-incremental-d92d96-dbt-labs.vercel.app/docs/build/incremental-models#custom-strategies) ## What are you changing in this pull request and why? This addresses the "**For end users**" portion of #1761. The feature request in dbt-labs/dbt-core#5245 describes the value proposition as well as the previous and new behavior: #### Functional Requirement - Advanced users that wish to specify a custom incremental strategy must be able to do so. #### Previous behavior - Advanced dbt users who wished to specify a custom incremental strategy must override the same boilerplate Jinja macro by copy pasting it into their dbt project. #### New behavior - Advanced dbt users who wish to specify a custom incremental strategy will only need to create a macro that conforms to the naming convention `get_incremental_NAME_sql` that produces the correct SQL for the target warehouse. ## Also To address the questions raised in dbt-labs/dbt-core#8769, we also want to document how to utilize custom incremental macros that come from a package. For example, to use the `merge_null_safe` custom incremental strategy from the `example` package, first [install the package](/build/packages#how-do-i-add-a-package-to-my-project), then add this macro to your project: ```sql {% macro get_incremental_merge_null_safe_sql(arg_dict) %} {% do return(example.get_incremental_merge_null_safe_sql(arg_dict)) %} {% endmacro %} ``` ## 🎩 <img width="503" alt="image" src="https://github.com/dbt-labs/docs.getdbt.com/assets/44704949/51c3266e-e3fb-49bd-9428-7c43920a5412"> ## Checklist - [x] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [x] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content).
Is this your first time submitting a feature request?
Describe the feature
get_incremental_<name>_sql
, which will be invoke run time usingmodel_context
variable andget_incremental_strategy_macro
Describe alternatives you've considered
Who will this benefit?
Are you interested in contributing this feature?
Yes
Anything else?
context
variable, but didn't workThe text was updated successfully, but these errors were encountered: