Skip to content

Commit

Permalink
dbt run for incremental models fail on second run
Browse files Browse the repository at this point in the history
Internal Issue Ticket: https://jira.cloudera.com/projects/DBT/issues/DBT-130 (#42)

Test plan:
Use the Impala demo project https://github.com/cloudera/dbt-impala-example
Generate the first set of fake data, and then run the models using dbt run.
Next generate the second set of fake data, and then run the models using dbt run.
The second run should terminate without any error, updating the the incremental model.

Synopsis:
For model definitions that define partition_by clause with a string, the adapter wrongly generates the insert overwrite statement,
resulting in an error being thrown by impala.
An example model configuration that will generate error (before applying this PR):
{{
    config(
        materialized='incremental',
        partition_by='report_date'
    )
}}

An example model configuration that will not generate error (before applying this PR):
{{
    config(
        materialized='incremental',
        partition_by=['report_date']
    )
}}

The current PR addresses the issue of defining model config either as a string or array of string.
  • Loading branch information
tovganesh authored Jul 15, 2022
1 parent 40a22d8 commit f47191f
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion dbt/include/impala/macros/insertoverwrite.sql
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,11 @@
{%- set partition_cols = config.get('partition_by', validator=validation.any[list]) -%}

{% if partition_cols is not none %}
{%- set partition_col = partition_cols[0] -%}
{% if partition_cols is string %}
{%- set partition_col = partition_cols -%}
{% else %}
{%- set partition_col = partition_cols[0] -%}
{% endif %}

{%- set dest_cols_csv = get_quoted_csv_exclude(dest_columns | map(attribute="name"), "") -%}
{%- set dest_cols_csv_exclude = get_quoted_csv_exclude(dest_columns | map(attribute="name"), partition_col) -%}
Expand Down

0 comments on commit f47191f

Please sign in to comment.