Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Custom Incremental Strategy does not work with BigQuery adapter #1133

Closed
2 tasks done
matthieuvilatte opened this issue Mar 8, 2024 · 1 comment
Closed
2 tasks done
Labels
bug Something isn't working

Comments

@matthieuvilatte
Copy link

matthieuvilatte commented Mar 8, 2024

Is this a new bug in dbt-bigquery?

  • I believe this is a new bug in dbt-bigquery
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When trying to use Custom Incremental Strategies, dbt raises an error.

Expected Behavior

The macro which deals with incremental materialization should try to load get_incremental_STRATEGY_sql macros before telling that the strategy is invalid.

Steps To Reproduce

I followed the documentation for Custom strategies : https://docs.getdbt.com/docs/build/incremental-models#custom-strategies

Create the following macro :

{% macro get_incremental_insert_only_sql(arg_dict) %}

  {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %}

{% endmacro %}


{% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %}

    {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%}

    insert into {{ target_relation }} ({{ dest_cols_csv }})
    (
        select {{ dest_cols_csv }}
        from {{ temp_relation }}
    )

{% endmacro %}

And the follwing model :

{{ config(
    materialized="incremental",
    incremental_strategy="insert_only",
) }}

SELECT 1 AS foo

Relevant log output

Compilation Error in model my_model (models/my_model.sql)
Invalid incremental strategy provided: insert_only
Expected one of: 'merge', 'insert_overwrite'

> in macro dbt_bigquery_validate_get_incremental_strategy (macros/materializations/incremental.sql)
> called by macro materialization_incremental_bigquery (macros/materializations/incremental.sql)
> called by model my_model (models/my_model.sql)

Environment

- OS: Docker python:3.8-slim-bullseye
- Python: 3.8
- dbt-core: 1.7.9
- dbt-bigquery: 1.7.6

Additional Context

No response

@matthieuvilatte matthieuvilatte added bug Something isn't working triage labels Mar 8, 2024
@dbeatty10 dbeatty10 self-assigned this Mar 9, 2024
@dbeatty10
Copy link
Contributor

Thanks for reaching out @matthieuvilatte !

This hasn't yet been implemented yet, and dbt-labs/dbt-core#9290 covers the remaining work to be done.

Since we already have an issue open for this, I'm going to close this one as a duplicate.

@dbeatty10 dbeatty10 closed this as not planned Won't fix, can't repro, duplicate, stale Mar 9, 2024
@dbeatty10 dbeatty10 removed the triage label Mar 9, 2024
@dbeatty10 dbeatty10 removed their assignment Mar 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants