Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spark__get_relations_by_pattern doesn't seem to work as expected #36

Open
VDFaller opened this issue Oct 27, 2023 · 1 comment
Open

Comments

@VDFaller
Copy link

VDFaller commented Oct 27, 2023

spark_utils==0.3.0

** Edit ** just realized the readme says that it doesn't have a shim for this, but it sure seems to.

Found this because I was trying to figure out this codegen issue

if I call

dbt run-operation generate_source --args '{schema_name: my_schema, database_name: my_database}' 

I don't get anything back

But if I call

dbt run-operation generate_source --args '{"schema_name": "my_schema", "database_name": "my_database", "table_pattern": ""*""}'

I do get stuff back, but I get the target.database instead my_database (they have the same schema).
I think that's because database=None

I'm guessing I could do something similar to this in order to get the table_pattern problem. codegen defaults it to '%'

Possible Workaround

I'm able to get it working if I add

    {% set table_pattern = table_pattern|replace('%', '*') %}
    {%- call statement('switch_database', fetch_result=False) %}
        USE CATALOG {{database}}
    {%- endcall -%}

to the top of spark__get_relations_by_pattern

But I don't know what kind of impact that might truly have. It's only the path I'm coming from.

@VDFaller
Copy link
Author

It's totally possible that this is half on codegen for defaulting table_pattern. And half spark_utils for the database thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant