[Bug] Lack of migration path from user configs defined in profiles.yml to the 'flags' key in dbt_project.yml

[wherrmann]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Initially introduced with #9289 and released with dbt-core v1.8 (though I see from #9183 that it was also backported to 1.6 and 1.7), there isn't a viable migration path for moving from user configs defined in profiles.yml to the 'flags' key in dbt_project.yml.

The current behavior is that if user configs ARE defined in profiles.yml and the 'flags' key is NOT defined dbt_project.yml, dbt will yield this warning:

[WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.

if user configs ARE defined in profiles.yml and the 'flags' key is ALSO defined in dbt_project.yml, dbt will yield this error:

Runtime Error
  Do not specify both 'config' in profiles.yml and 'flags' in dbt_project.yml. Using 'config' in profiles.yml is deprecated.

The issue is that, as mentioned in dbt's documentation, profiles.yml is often not part of the dbt project and so not version-controlled:

This file generally lives outside of your dbt project to avoid sensitive credentials being checked in to version control

For an organization to make this change, then, each member of the organization would need to update their profiles.yml simultaneously with the creation of the flags key in dbt_project.yml in order to avoid a runtime error.

Expected Behavior

My expectation is that setting both user configs defined in profiles.yml and the 'flags' key in dbt_project.yml would yield a deprecation warning, with the user config defined in profiles.yml being ignored in favor of the 'flags' key in dbt_project.yml. This would allow users to opt-in to the new behavior, as intended, instead of making this, effectively, a hard deprecation for any organization with more than one user.

Steps To Reproduce

1. Use dbt-core v1.8.x
2. Define both a user config in profiles.yml:

config:
  use_colors: false

and a 'flags' key in dbt_project.yml

flags:
  use_colors: false

3. Run any dbt command (say, dbt debug)
4. See runtime error

Runtime Error
  Do not specify both 'config' in profiles.yml and 'flags' in dbt_project.yml. Using 'config' in profiles.yml is deprecated.

Relevant log output

No response

Environment

- OS: macOS-14.5-x86_64
- Python: 3.11.4
- dbt: 1.8.3

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[dbeatty10]

@wherrmann Appreciate you noticing this and taking the time to bring this to our attention 🤩

Acceptance criteria

@jtcohen6, @dataders, and myself discussed this today. We agree with your expected behavior that:

• setting both user configs defined in profiles.yml and the flags key in dbt_project.yml should be allowed but yield a deprecation warning
• any individual user config defined in both profiles.yml should be ignored in favor of the one under flags key in dbt_project.yml. e.g. similar to how dictionaries for the meta config are merged

The would allow multiple users of the same dbt project to add flags: to their shared dbt_project.yml file but update their local profiles.yml on their own individual timing.

Example

profiles.yml

config:
  use_colors: false
  printer_width: 80

dbt_project.yml

flags:
  use_colors: true
  partial_parse: true

After merging values from profiles.yml and dbt_project.yml (with the latter taking precedence), we'd get the following:

  use_colors: true
  partial_parse: true
  printer_width: 80

[dbeatty10]

We'd backport this into 1.8 (but not earlier since it wouldn't be relevant in earlier versions).

[mahiki]

this just bit me as well. my confusion is that the only user configs in profiles.yml are:

muh_project:
   target: dev
   outputs:
    dev:
      type: databricks
      method: http
      catalog: muh_team_catalog
      schema: dev
      threads: 8
      http_path: /sql/1.0/warehouses/5xxxxxxxxxxx
      host: muh-company.cloud.databricks.com
      token: "{{ env_var('DATABRICKS_TOKEN') }}"

What of this is the 'config' part? I don't think I can eliminate these from profiles.yml and the flags in dbt_project.yml are from the elementary package and if I turn those off it will reduce functionality.

edit:

Looks like an unrelated issue specific to databricks jobs. When databricks workflow job is improperly configured it provides a default profiles.yml file and runs the models without choosing a target.

I cant find what that generated databricks profile yaml looks like yet, but when the same project was configured to specify --project-dir and --profiles-dir the error disappeared.

dbt 1.8.6

[NortySpock]

Hey @mahiki, I'm not quite following how you implemented your workaround within Databricks.

Did you supply a templated profiles.yml from your repo and then modify it somehow? Was a profiles.yml file dropped into a folder from a key vault? Did you decide to add --project-dir to each of your command line flags? What indicates that a databricks workflow job is improperly configured? (Because possibly ours is 'improperly configured' and we just have been working around the quirks of the Databricks dbt job this whole time.)

My team also recently ran into this issue when trying to enable dbt_elementary in Databricks (dbt-core v1.7.x currently, hoping to upgrade to v1.8.x soon) using a Databricks dbt Job.

I assumed Databricks already specified the --project-dir (given that my dbt_project.yml is not at the root of my repo, I had to specify the subfolder), and I can see that dbt is reporting that the profiles directory is /tmp/tmp-dbt-run-703977762881311-profiles which was clearly automatically created for the purposes of the run...