[CT-3034] [Bug] relative path in DBT_PROFILES_DIR fails

[leo-schick]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I set globally the environment variable DBT_PROFILES_DIR to ~/.dbt which breaks the execution:

But it works when I set the absolute path e.g. DBT_PROFILES_DIR=/root/.dbt.

Expected Behavior

I would expect that dbt resolves the relative path in environment variable DBT_PROFILES_DIR.

Steps To Reproduce

1. run export DBT_PROFILES_DIR=~/.dbt
2. run dbt run

Relevant log output

No response

Environment

- OS:Debian 11
- Python: 3.9.2
- dbt: 1.6.1

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

• dbt-databricks 1.6.1

[leo-schick]

Note: Found an issue with my description and adjusted the issue accordingly. Relative paths seem to not working anymore since 1.6.1. Before I used dbt version 1.4.6 where this worked out fine.

[jtcohen6]

@leo-schick It looks like you're running dbt inside a Databricks notebook/workflow(?). I don't see this error while running dbt-core locally, including more recent versions. I'm wondering if this error is specific to running against DBFS? We've seen some other issues recently that fall into this category:

• [CT-3101] dbt logs not getting generated for dbt commands in Databricks #8608
• [CT-2457] [Bug] dbt deps commands error our in databricks DBFS  #7428

$ export DBT_PROFILES_DIR=~/.dbt
$ dbt run --profile sandbox-databricks
10:39:39  Running with dbt=1.6.1
10:39:40  Registered adapter: databricks=1.6.3
10:39:40  Found 1 model, 1 operation, 1 test, 0 sources, 0 exposures, 0 metrics, 686 macros, 0 groups, 0 semantic models
...

[leo-schick]

@jtcohen6 Yes this is running inside a Databricks Notebook

[aranke]

I tried and wasn't able to reproduce this.

[jtcohen6]

We're going to close this issue, and document it (along with other issues labeled dbfs) as a limitation of running dbt-core inside a Databricks notebook.

[jaklan]

@jtcohen6 it's not related to Databricks - dbt simply doesn't support tilde for neither --project-dir nor --profiles-dir:

Error: Invalid value for '--project-dir': Path '~/.local' does not exist.
Error: Invalid value for '--profiles-dir': Path '~/.local' does not exist.

The path of course exists. Tested with dbt-core 1.7.2.

EDIT: I have just tested environment variables - DBT_PROFILES_DIR and DBT_PROFILES_DIR. In their case tilde works.