[Pre-regression] Error when serializing RunResults (variable results) with to_dict() during the on-run-end

[DmytroSly]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I log RunResults into a database table by parsing the results variable this way:

{% macro parse_dbt_results(results) %}
    -- Create a list of parsed results
    {%- set parsed_results = [] %}
    -- Flatten results and add to list
    {% for run_result in results %}
        -- Convert the run result object to a simple dictionary
        {% set run_result_dict = run_result.to_dict() %}
        {% do parsed_results.append(run_result_dict) %}
    {% endfor %}
    {{ return(parsed_results) }}
{% endmacro %}

After upgrading to dbt-core==1.8.0rc1 and dbt-snowflake==1.8.0b3 I get the following error:

Encountered an error:
Class RunResult has unresolved type reference agate in some of its fields
09:15:00  Traceback (most recent call last):
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 138, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 101, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 218, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 247, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 294, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 332, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\main.py", line 568, in run
    results = task.run()
              ^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\runnable.py", line 526, in run
    result = self.execute_with_hooks(selected_uids)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\runnable.py", line 488, in execute_with_hooks
    self.after_run(adapter, res)
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 484, in after_run
    self.safe_run_hooks(adapter, RunHookType.End, extras)
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 423, in safe_run_hooks
    self.run_hooks(adapter, hook_type, extra_context)
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 374, in run_hooks
    sql = self.get_hook_sql(adapter, hook, idx, num_hooks, extra_context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 326, in get_hook_sql
    compiled = self.compiler.compile_node(hook, self.manifest, extra_context)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\compilation.py", line 533, in compile_node
    node = self._compile_code(node, manifest, extra_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\compilation.py", line 421, in _compile_code
    node.compiled_code = jinja.get_rendered(
                         ^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - SIBASE_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 146, in get_rendered
    rendered = render_template(template, ctx, node)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 539, in render_template
    return template.render(ctx)
           ^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 144, in render
    return self.environment.handle_exception()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 8, in template
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 9, in template
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in __mashumaro_to_dict__
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 1140, in add_pack_method
    self._add_pack_method_lines(method_name)
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 836, in _add_pack_method_lines
    field_types = self.get_field_types(include_extras=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 221, in get_field_types
    return self.__get_field_types(include_extras=include_extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 188, in __get_field_types
    raise UnresolvedTypeReferenceError(self.cls, name) from None
mashumaro.exceptions.UnresolvedTypeReferenceError: Class RunResult has unresolved type reference agate in some of its fields

Expected Behavior

No error happens, just like in the previous versions of dbt-core

Steps To Reproduce

Just try to serialize the results variable during the on-run-end in the below way:

{% macro parse_dbt_results(results) %}
    -- Create a list of parsed results
    {%- set parsed_results = [] %}
    -- Flatten results and add to list
    {% for run_result in results %}
        -- Convert the run result object to a simple dictionary
        {% set run_result_dict = run_result.to_dict() %} -- the error happens here
        {% do parsed_results.append(run_result_dict) %}
    {% endfor %}
    {{ return(parsed_results) }}
{% endmacro %}

Relevant log output

Encountered an error:
Class RunResult has unresolved type reference agate in some of its fields
09:15:00  Traceback (most recent call last):
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 138, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 101, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 218, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 247, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 294, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\requires.py", line 332, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\cli\main.py", line 568, in run
    results = task.run()
              ^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\runnable.py", line 526, in run
    result = self.execute_with_hooks(selected_uids)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\runnable.py", line 488, in execute_with_hooks
    self.after_run(adapter, res)
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 484, in after_run
    self.safe_run_hooks(adapter, RunHookType.End, extras)
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 423, in safe_run_hooks
    self.run_hooks(adapter, hook_type, extra_context)
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 374, in run_hooks
    sql = self.get_hook_sql(adapter, hook, idx, num_hooks, extra_context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\task\run.py", line 326, in get_hook_sql
    compiled = self.compiler.compile_node(hook, self.manifest, extra_context)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\compilation.py", line 533, in compile_node
    node = self._compile_code(node, manifest, extra_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\compilation.py", line 421, in _compile_code
    node.compiled_code = jinja.get_rendered(
                         ^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - SIBASE_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 146, in get_rendered
    rendered = render_template(template, ctx, node)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 539, in render_template
    return template.render(ctx)
           ^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 144, in render
    return self.environment.handle_exception()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 8, in template
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt\clients\jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\dbt_common\clients\jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 9, in template
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\jinja2\sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in __mashumaro_to_dict__
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 1140, in add_pack_method
    self._add_pack_method_lines(method_name)
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 836, in _add_pack_method_lines
    field_types = self.get_field_types(include_extras=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 221, in get_field_types
    return self.__get_field_types(include_extras=include_extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\UserData\z004ttau\Documents\Repos\dbt - XXXXXX_X\venv\Lib\site-packages\mashumaro\core\meta\code\builder.py", line 188, in __get_field_types
    raise UnresolvedTypeReferenceError(self.cls, name) from None
mashumaro.exceptions.UnresolvedTypeReferenceError: Class RunResult has unresolved type reference agate in some of its fields

Environment

- OS: Windows 10 Enterprise
- Python: 3.11.4
- dbt-core: 1.8.0rc1

Which database adapter are you using with dbt?

dbt-snowflake==1.8.0b3

Additional Context

No response

[dbeatty10]

Thanks for reporting this @DmytroSly !

I was able to see the same difference as you between 1.7 and 1.8 (see "reprex" below for details). I didn't confirm or deny, but it might be related to the changes in #9744.

Reprex

Add the following to dbt_project.yml:

on-run-end: "{{ log_dbt_results(results) }}"

models/my_model.sql

select 1 as id

macros/log_dbt_results.sql

{% macro parse_dbt_results(results) %}
    -- Create a list of parsed results
    {%- set parsed_results = [] %}

    {{ log("results: " ~ results, True) }}

    -- Flatten results and add to list
    {% for run_result in results %}
        -- Convert the run result object to a simple dictionary
        {% set run_result_dict = run_result.to_dict() %}
        {% do parsed_results.append(run_result_dict) %}
    {% endfor %}
    {{ return(parsed_results) }}
{% endmacro %}


{% macro log_dbt_results(results) %}
    {% set parsed_results = parse_dbt_results(results) %}

    {% for run_result in results %}
        -- Convert the run result object to a simple dictionary
        {{ log("run_result: " ~ run_result, True) }}
    {% endfor %}

    -- dummy SQL to execute in the on-run-end hook
    select 2 as id
{% endmacro %}

[jtcohen6]

Thanks for the report @DmytroSly, and for the quick follow-up @dbeatty10!

I can confirm that updating this line to remove the conditional if TYPE_CHECKING makes the reproduction case pass. I suspect adding that back in would undo the startup performance improvement, but it might be our safest path toward avoiding this regression before the v1.8.0 final release.

For context, here is where we're adding results: List[RunResult] into the on-run-end Jinja context:

dbt-core/core/dbt/task/run.py

Line 470 in fb6dbc8

"results": results,

Alternative ideas:

• Exclude agate_table from the to_dict serialization of RunResult. (We haven't documented that it's included, and I believe it's only populated for tests, seeds, and dbt show.)
• Change the type of what's added into the on-run-end context from List[RunResult] (which includes node and agate_table) to List[RunResultOutput] (which doesn't, and exactly matches what's in run_results.json). This would be a breaking change, since we've documented that node is included: https://docs.getdbt.com/reference/dbt-classes#result-objects

[peterallenwebb]

@jtcohen6 It looks like agate_table is already excluded, in the sense that it always explicitly serializes to None, so it's unfortunate that mashumaro gets so upset that its annotated type isn't available.

There's a variant of your second suggestion that might work. Instead of RunResult, we could provide instances of its base class NodeResult to on-run-end, which does include node, but not agate_table.

[jtcohen6]

@peterallenwebb and I agreed that:

• The "undo" is fastest & safest fix for v1.8.0-final
• It would be great to redo this speedup, possibly by converting RunResult -> NodeResult before sticking in the on-run-end context (Peter's suggestion)
• We need to add a test case for this: "on-run-end: [{{ results[0].to_dict() }}] does not raise an UnresolvedTypeReferenceError"

[dwreeves]

Hi I'm the contributor of the original code that caused the mess. Wow! That's unfortunate. Good job finding the bug and its fix, and my apologies for introducing it. dbt-core is tricky sometimes. 😅

[dwreeves]

So I'm saying this without much knowledge of how this works (I've never used mashumaro), but maybe adding something like this works? That way you don't need to break the schema for RunResult.

@dataclass
class RunResult(NodeResult):

    def __pre_serialize__(self, context: Optional[Dict] = None):
        import agate
        global agate
        return self

    @classmethod
    def __pre_deserialize__(cls, d: Dict[Any, Any]) -> Dict[Any, Any]:
        import agate
        global agate
        return d

I'm pretty swamped at work (probably shouldn't even be messing around with this, lol), so cannot get to this until the weekend, but when I do, I will be able test this using @dbeatty10's reproduction.

I'm also a little interested, for my own edification. Does dbt always serialize a run result? If so, this really isn't saving time except for like, dbt --help, dbt docs serve, and maybe something like dbt parse. ☹️ That's unfortunate!