dbt-bigquery batch_id config

[dbeatty10]

Contributions

• I am a PM or subject matter expert at dbt who is responsible for this feature.

Where does this content belong?

https://docs.getdbt.com/reference/resource-configs/bigquery-configs

Link to source material

Here is the relevant issue & pull request:

dbt-labs/dbt-bigquery#671 / dbt-labs/dbt-bigquery#804 (review)

Additional context

None of the following needs to be documented, but for completeness sake:

To get dbt-bigquery is in a solid state for the RC for 1.6 and to avoid the issue described in dbt-labs/dbt-bigquery#822, we used dbt-labs/dbt-bigquery#826 to revert the original PR in dbt-labs/dbt-bigquery#727.

[nickozilla]

This is how we would likely use batch_id param in our python models - using an env var that is only present in our CICD environment as a conditional, else use the run start time, suffixed by the model name. This should ensure that the batch ID's are always unique and compliant against the dataproc batch ID naming conventions:

models:
  - name: technical_python_array
    description: A random table with a calculated column defined in python.
    config:
      enabled: true
      batch_id: |
        {%- if env_var('CICD_BUILD_ID', '') != '' -%} {{ env_var('CICD_BUILD_ID') }}-python-array
        {%- else -%} {{ run_started_at.strftime("%Y-%m-%d-%H-%M") }}-python-array
        {%- endif -%}
    columns:
      - name: A
        description: Column A
      - name: B
        description: Column B
      - name: C
        description: Column C

[matthewshaver]

Added this as part of #4431 to the existing examples.
I've opened a separate issue for us to address the need for more documentation of dataproc_batch #4433