[CT-1341] [Bug] Source not evaluated properly when used in a set block which is then assigned to a hook in a model file

[jeremyyeo]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

It looks like source doesn't work properly when it's being used in a set jinja block + used in a hook.

Expected Behavior

source() should always evaluate correctly everywhere no matter where we're calling it.

Steps To Reproduce

1. Setup profile.

# ~/.dbt/profiles.yml

snowflake:
  target: dev
  outputs:
    dev:
      database: development   # Just to show the default (I think is not relevant here).
      ...

2. dbt_project.yml - nothing special.

# dbt_project.yml

name: "my_dbt_project"
version: "1.0.0"
config-version: 2
profile: "snowflake"

models:
  my_dbt_project:
    +materialized: table

3. Sources.

# models/sources.yml

version: 2
sources:
  - name: my_sources
    database: analytics  # Override default of 'development'
    tables:
      - name: my_source

3. Setup model.

-- models/foo.sql

{% set my_var -%}
    update {{ source('my_sources', 'my_source') }} set foo = 'bar'
{%- endset %}

{{ config(post_hook = 'select 2 -- ' ~ my_var) }}

select 1 as id -- {{ source('my_sources', 'my_source') }}

4. Run our models:

$ dbt --debug run | grep -E 'select 1|select 2'

select 1 as id -- analytics.my_sources.my_source
select 2 -- update development.dbt_jyeo.foo set foo = 'bar'

Clearly {{ source('my_sources', 'my_source') }} evaluated correctly when it's in the body of the model but when we declare it in a set and then assign that to a hook, it becomes weird (evaluates to the {{ this }} for some reason).

Relevant log output

No response

Environment

- OS: macOS
- Python: 3.8.7
- dbt: 1.2.1

Which database adapter are you using with dbt?

snowflake

Additional Context

I know this is not best practice but more so for understanding why this is happening.

For readers, to make this work properly, we could do something like:

1. Add a macro:

-- macros/update.sql
{% macro update_something(arg) %}
    select 2 -- update {{ arg }} set foo = 'bar'
{% endmacro %}

2. Tweak our model:

-- models/foo.sql

{{ config(post_hook = "{{ update_something(source('my_sources', 'my_source')) }}") }}

select 1 as id -- {{ source('my_sources', 'my_source') }}

3. Run:

$ dbt --debug run | grep -E 'select 1|select 2'

select 1 as id -- analytics.my_sources.my_source
select 2 -- update analytics.my_sources.my_source set foo = 'bar'

[jtcohen6]

Thanks @jeremyyeo. I believe this is the same issue as #4606 (and #2793 before that).

In its first parse through the project, dbt captures all refs (to construct the DAG) but renders the code with a "dummy" value. That "dummy" value is, for no great reason, the current node reference.

Put another way: At parse time, we don't have the resolved form of that source yet. So {{ this }} is the placeholder used instead.

Pre- and post-hooks can be defined with double/nested curlies to force a re-render at execution time, which is what you've shown in the example by defining update_something as a custom macro and then wrapping it in an extra set of curly braces. https://docs.getdbt.com/docs/building-a-dbt-project/dont-nest-your-curlies#an-exception

This is a problem that crops up with some frequency, so it is worth figuring out what an actually good solution might be. Best I've come up with is finding a way to store the raw/unrendered values of all configs (especially hooks) at parse time—not a thing we can do today—and then re-rendering them at execution time. It's not a totally satisfying answer.

Closing this for now as a duplicate, but it shouldn't be far from our thoughts.