[CT-627] [Feature] Allow references to input parameters, specifically for dependency handling

[smitsrr]

Is this your first time opening an issue?

• I have read the expectations for open source contributors

Describe the Feature

When dbt projects become large, models are used in many places, and even small failures can bring down business critical portions of the DAG.

To isolate risk, it would be useful if a reference (or source) could take a parameter for what to do if that model fails. Arguments could include, 'proceed', 'skip', where 'skip' is the current default behavior.

For instance, we have a minor ref table that failed and it caused our main fact to fail. However, the little ref is not critical to the fact table, and we would be happy to proceed without that data. However, that ref table is critical to another dimension, so we would want that DAG to skip.

Describe alternatives you've considered

I've considered the tests and built DAG alternatives, but that doesn't allow customization across portions of the DAG.

Who will this benefit?

This will benefit users who have large projects with complex dependencies.

Are you interested in contributing this feature?

No response

Anything else?

No response

[jtcohen6]

@smitsrr There's been a lot of new and recent conversation in #2142, including a comment I see from you! Prompted by those comments, and also your thoughtful issue here, I've just reopened it and left a new comment: #2142 (comment)

One thing I mentioned there:

I'm hesitant to support defining this configuration for the full matrix of parent-child relationships, as I think that can get unwieldy very quickly.

As the person writing the model, I see the intuitive appeal of being able to configure this right within the ref() / source() specification. When these models are actually running in production, I'm worried about the ballooning complexity that comes from so many different behaviors, which would require jumping back through the dbt source code to understand the cause.

I'd much prefer to reason clearly about a model that MUST block downstream builds (if we choose to configure this on the parent), or a model that MUST run (if we choose to configure this the child).

small failures can bring down business critical portions of the DAG

I'd push back slightly on "bring down." One important feature of dbt's modeling paradigm is that, in almost all cases, model builds ought to be idempotent and atomic. (There are precious few exceptions: snapshots; --full-refresh on certain adapters that don't support create or replace or rename-swapping.) If an upstream model build fails, the database views and tables built by downstream models do not go away. They do get stale, over time, if the upstream model continues to fail without recourse. Hopefully that's an unusual occasion, and one where a manual intervention is warranted, to run the critical models while excluding the failing non-critical ones.

I'm definitely open to discussing this more. For now, I see this idea as very closely related to the one in #2142, so I'm going to close this issue to keep the conversation centralized over there.