AWS Databricks - 1.6.0 causing dbt commands to describe tables and schemas unrelated to the operation

[jjardis]

Describe the bug

Our jobs used to only scan tables related to a job, for example, dbt run --select my_model would show DESCRIBE statements for tables within the schema that this model was concerned with.

It is now describing tables within schemas the run doesn't manage, which is both adding execution time and causing errors when the user doesn't have access to the storage location of that data.

Steps To Reproduce

1. Create a basic SQL endpoint with default permissions in AWS
2. Create a simple DBT project and run it in databricks, preferrably with two different schemas that contain tables. Call this SCHEMA_A and SCHEMA_B
3. Run a DEEP COPY into a table in SCHEMA_B that is stored in an external storage location that databricks doesn't have access to by default.
4. Run the models for SCHEMA_A, this will show a failure.

Expected behavior

This should only run DESCRIBE on tables that have been selected to be ran by the job, not all tables or unrelated tables to the project.

System information

I'm aware this isn't the same command, but I don't have access to this directly.
The output of dbt --version:

+ dbt deps
14:08:06  Running with dbt=1.6.0
14:08:06  Installing dbt-labs/dbt_utils
14:08:07  Installed from version 0.9.2
14:08:07  Updated version available: 1.1.1
14:08:07  
14:08:07  Updates available for packages: ['dbt-labs/dbt_utils']                 
Update your versions in packages.yml, then run dbt deps

The operating system you're using:
Databricks jobs: 11.2 runtime version

The output of python --version:
3.9.5

Additional context

I believe that this has something to do with the following change: #326, it is the only change to this I could find in the release between dbt-core / dbt-databricks 1.6.0

[susodapop]

Thanks for opening this issue and for the reproduction steps. Investigating now.

[susodapop]

Yep, #326 is the culprit. Fix incoming.

[susodapop]

Okay the quick fix is that we're going to revert #326 and cut release 1.6.1 with it reverted. Then to solve the original issue that #326 tries to solve, we'll restore the behaviour behind a config so that AWS Glue Catalog users can turn it on as needed.

[susodapop]

Revert has been completed and release 1.6.1 is live on pypi and on github: https://github.com/databricks/dbt-databricks/releases/tag/v1.6.1

pip install dbt-databricks==1.6.1

[susodapop]

Please reopen the issue if the error persists.

[jjardis]

@susodapop The issue has persisted, we have resolved by rolling back to 1.5.4 as that was the last working version reported in our systems.

[jeremyyeo]

Just filed #442 before finding this issue - probably the same thing so see that issue for a reproduction.