Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] copy_partitions with static partitions list produces empty statement #540

Open
2 tasks done
alexandru-pisarenco-dh opened this issue Sep 18, 2024 · 1 comment · May be fixed by dbt-labs/dbt-bigquery#1351
Open
2 tasks done
Labels
community A PR, or an issue with a PR, from a community member feature:incremental Issues related to incremental materializations pkg:dbt-bigquery Issue affects dbt-bigquery type:bug Something isn't working as documented

Comments

@alexandru-pisarenco-dh
Copy link

Is this a new bug in dbt-bigquery?

  • I believe this is a new bug in dbt-bigquery
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When an incremental load with copy_partitions=True, and partitions=[... list of partitions ...] is used, the macro bq_static_insert_overwrite_sql does not produce any SQL statements, which leads the BigQuery client to trigger an error.

On top of that, because this code:

  -- 2. clean up the temp table
  drop table if exists {{ tmp_relation }};

is only executed when copy_partitions=False, the temp table is not deleted.

Expected Behavior

A non-empty statement should be produced by the macro and sent to dbt. I figure that drop table if exists {{tmp_relation}}; should just be taken out of any conditional blocks and just always be there.

Steps To Reproduce

  1. Create a new model with the following config:
{{
  config(
    materialized="incremental",
    partition_by={
      "field": "partition_date",
      "data_type": "date",
      "granularity": "day",
      "copy_partitions": true,
    },
    incremental_strategy="insert_overwrite",
    partitions=["2024-08-13", "2024-08-14"]
  )
}}
  1. Run the model

Relevant log output

14:48:05  Completed with 1 error and 0 warnings:
14:48:05  
14:48:05    Database Error in model the_model (models/the_model.sql)
  Syntax error: Unexpected end of statement at [1:331]
  compiled Code at /Users/alex/the_project/target/run/the_project/models/the_model.sql

Environment

- OS: MacOS
- Python: 3.10.14
- dbt-core: 1.8.6
- dbt-bigquery: 1.8.2

Additional Context

No response

@alexandru-pisarenco-dh alexandru-pisarenco-dh added type:bug Something isn't working as documented triage:product In Product's queue labels Sep 18, 2024
@github-christophe-oudar
Copy link
Contributor

github-christophe-oudar commented Sep 21, 2024

It wasn't supported but I wrote dbt-labs/dbt-bigquery#1351 to do so.
As a workaround, you might just use the updated macros and use them as a macro override to check in your own dbt project.

@amychen1776 amychen1776 added feature:incremental Issues related to incremental materializations and removed triage:product In Product's queue labels Sep 26, 2024
@colin-rogers-dbt colin-rogers-dbt added the community A PR, or an issue with a PR, from a community member label Oct 11, 2024
@mikealfare mikealfare added the pkg:dbt-bigquery Issue affects dbt-bigquery label Jan 14, 2025
@mikealfare mikealfare transferred this issue from dbt-labs/dbt-bigquery Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community A PR, or an issue with a PR, from a community member feature:incremental Issues related to incremental materializations pkg:dbt-bigquery Issue affects dbt-bigquery type:bug Something isn't working as documented
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants