[CT-1197] [Bug] Schema Files not parsed for models at the right time

[emmyoop]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When we parse a model node, any configs set in the schema yaml file do not get incorporated until after the fact.

Specifically all nodes are considered enabled (if only disabled in the schema yaml file) while we parse through the sql files until we parse the schema file. So they end up as nodes in the manifest to be processed and are not in the disabled dict.

Specifically the value of node.config.enabled will always be True when the node was set to disabled only in the yaml file. It will probably be incorrectly set to False if it is set to False at the project level and overridden at the yaml level to be True.

dbt-core/core/dbt/parser/base.py

Lines 370 to 374 in f5a94fc

	def add_result_node(self, block: FileBlock, node: ManifestNodes):
	if node.config.enabled:
	self.manifest.add_node(block.file, node)
	else:
	self.manifest.add_disabled(block.file, node)

There is also a bit of hacky logic as follows to ensure we don't resolve refs on disabled nodes even though the node is in the manifest.

TODO: add link to process_refs in parser/manifest.py after #5868 gets merged

            # if the node is disabled, no need to resolve the refs
            if node.config.enabled:
                _process_refs_for_node(self.manifest, current_project, node)

The above conditional should be removed when this gets resolved since we would no longer be resolving refs for disabled nodes.

Expected Behavior

Account for configs set in schema files.

Steps To Reproduce

A barebones project with 3 models:

-- my_model.sql
select 1 as user

-- my_model_2.sql
select * from {{ ref('my_model') }}

-- my_model_3.sql
select * from {{ ref('my_model_2') }}

A schema.yml like so:

version: 2
models:
  - name: my_model
  - name: my_model_2
    config:
      enabled: false
  - name: my_model_3
    config:
      enabled: false

run dbt run and examine manifest.nodes and manifest.disabled

Additional Context

#5868 fixed the bugs users are experiencing, this is to fix the underlying issue

[jtcohen6]

@emmyoop Thanks for the thorough writeup!

The fundamental issue in the parsing order that you've identified here feels related to #4000 as well. We instantiate snapshot nodes after parsing the .sql file, perform dataclass validation, check for specific configs, if they're not supplied we raise an error, before actually parsing the .yml file that might contain those configs.

[emmyoop]

This really comes down to the order we have to process the files. "Fixing" this really means reworking parsing order, possibly saving files to parse over again? Not sure if it would really decrease complexity because we re-process the files.