[Regression] Deprecation date check raises KeyError in dbt-core & dbt Cloud

[danlsn]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

dbt-core is failing to parse a project where a versioned model, with a deprecation date defined, is depended on by an exposure.

Inside the ManifestLoader class, the check_for_model_deprecations(self) method is raising a KeyError when checking child nodes.

dbt-core/core/dbt/parser/manifest.py

Lines 592 to 593 in bdb79e8

	for child_unique_id in child_nodes:
	child_node = self.manifest.nodes[child_unique_id]

As you can see from the error the code is using the fqn for an exposure to search the self.manifest.nodes dictionary, which it would never find.

Potential Fix

A simple solution would be to try and find the child_node in self.manifest.nodes using the .get() method instead. In this case when looking for an exposure in nodes, it will return None and continue at the isinstance() guard clause.

 child_nodes = self.manifest.child_map[node.unique_id]
                for child_unique_id in child_nodes:
                    child_node = self.manifest.nodes.get(child_unique_id)
                    if not isinstance(child_node, ModelNode):
                        continue
                    if node.is_past_deprecation_date:
                        event_cls = DeprecatedReference
                    else:
                        event_cls = UpcomingReferenceDeprecation

Expected Behavior

• dbt project parses without raising an error

Steps To Reproduce

1. using latest version of dbt-core from main branch or dbt Cloud versionless
2. run dbt parse on a project where a versioned model is used in an exposure

Relevant log output

23:30:49  Traceback (most recent call last):
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\cli\requires.py", line 153, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\cli\requires.py", line 103, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\cli\requires.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\cli\requires.py", line 264, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\cli\requires.py", line 311, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\cli\requires.py", line 327, in wrapper
    setup_manifest(ctx, write=write, write_perf_info=write_perf_info)
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\cli\requires.py", line 351, in setup_manifest
    ctx.obj["manifest"] = parse_manifest(
                          ^^^^^^^^^^^^^^^
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\parser\manifest.py", line 2003, in parse_manifest
    manifest = ManifestLoader.get_full_manifest(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\parser\manifest.py", line 308, in get_full_manifest
    manifest = loader.load()
               ^^^^^^^^^^^^^
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\parser\manifest.py", line 508, in load
    self.check_for_model_deprecations()
  File "c:\users\e124584\ideaprojects\dap-dbt-da-versionless\dbt-core\core\dbt\parser\manifest.py", line 593, in check_for_model_deprecations
    child_node = self.manifest.nodes[child_unique_id]
                 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'exposure.dap_dbt_da.aacsb_dataflow'

Environment

- OS: Windows 10
- Python: 3.11.9
- dbt: main branch (1.9.0-b2)

Which database adapter are you using with dbt?

snowflake

Additional Context

• This bug is related to Refactor code to properly handle reference deprecations #10852
• This is affecting our dbt Cloud projects as well

[danlsn]

I'm not familiar enough with the internals of dbt-core testing, but I feel like the tests should include more sophisticated/"real-world" examples of production-like dbt projects.

For example, versioned models with deprecation dates.

[dbeatty10]

Thank you for reporting this @danlsn !

I was able to reproduce this using the latest code in the main branch for dbt-core (bdb79e8) with the following files and commands:

Reprex

Create these files:

models/my_model.sql

select 1 as id

models/_properties.yml

models:
  - name: my_model
    deprecation_date: 1999-01-01 00:00:00.00+00:00

exposures:
  - name: my_exposure
    type: analysis
    owner: {email: [email protected]}
    depends_on:
      - ref("my_model")

Run these commands:

dbt parse --no-partial-parse

Get these error logs:

$ dbt parse --no-partial-parse
16:29:40  Running with dbt=1.9.0-b2
16:29:40  Registered adapter: postgres=1.9.0-b1
16:29:41  [WARNING]: Model my_model has passed its deprecation date of 1999-01-01T00:00:00+00:00. This model should be disabled or removed.
16:29:41  Encountered an error:
'exposure.my_project.my_exposure'
16:29:41  Traceback (most recent call last):
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/cli/requires.py", line 153, in wrapper
    result, success = func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/cli/requires.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/cli/requires.py", line 235, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/cli/requires.py", line 264, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/cli/requires.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/cli/requires.py", line 327, in wrapper
    setup_manifest(ctx, write=write, write_perf_info=write_perf_info)
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/cli/requires.py", line 351, in setup_manifest
    ctx.obj["manifest"] = parse_manifest(
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/parser/manifest.py", line 2003, in parse_manifest
    manifest = ManifestLoader.get_full_manifest(
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/parser/manifest.py", line 308, in get_full_manifest
    manifest = loader.load()
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/parser/manifest.py", line 508, in load
    self.check_for_model_deprecations()
  File "/Users/dbeatty/projects/environments/editable/dbt-core/core/dbt/parser/manifest.py", line 593, in check_for_model_deprecations
    child_node = self.manifest.nodes[child_unique_id]
KeyError: 'exposure.my_project.my_exposure'

[dbeatty10]

@danlsn I confirmed that when I used your code suggestion, that the error went away:

diff --git a/core/dbt/parser/manifest.py b/core/dbt/parser/manifest.py
index 493b562bb..d5fbef59a 100644
--- a/core/dbt/parser/manifest.py
+++ b/core/dbt/parser/manifest.py
@@ -590,7 +590,7 @@ class ManifestLoader:
                 # Get the child_nodes and check for deprecations.
                 child_nodes = self.manifest.child_map[node.unique_id]
                 for child_unique_id in child_nodes:
-                    child_node = self.manifest.nodes[child_unique_id]
+                    child_node = self.manifest.nodes.get(child_unique_id)
                     if not isinstance(child_node, ModelNode):
                         continue
                     if node.is_past_deprecation_date: