Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup catalog string comparison by using ilike before equals #982

Merged
merged 7 commits into from
May 13, 2024

Conversation

aranke
Copy link
Member

@aranke aranke commented Apr 22, 2024

resolves #1035

Problem

dbt docs generate for a single model takes 30-90s

Solution

Speed up information_schema comparison using ilike to drop performance down to 3-4s.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

@cla-bot cla-bot bot added the cla:yes label Apr 22, 2024
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-snowflake contributing guide.

@aranke aranke marked this pull request as draft April 22, 2024 14:37
@aranke aranke marked this pull request as ready for review May 13, 2024 17:09
@aranke aranke requested a review from a team as a code owner May 13, 2024 17:09
@aranke aranke requested a review from mikealfare May 13, 2024 17:11
@colin-rogers-dbt colin-rogers-dbt merged commit 32b6d39 into main May 13, 2024
17 of 18 checks passed
@colin-rogers-dbt colin-rogers-dbt deleted the speedup_catalog_comparison branch May 13, 2024 20:06
@colin-rogers-dbt colin-rogers-dbt added backport 1.7.latest Tag for PR to be backported to the 1.7.latest branch backport 1.8.latest labels May 13, 2024
github-actions bot pushed a commit that referenced this pull request May 13, 2024
* Speedup catalog string comparison by using ilike before equals

* Add parens

* add changie

* Update Under the Hood-20240425-144556.yaml

(cherry picked from commit 32b6d39)
github-actions bot pushed a commit that referenced this pull request May 13, 2024
* Speedup catalog string comparison by using ilike before equals

* Add parens

* add changie

* Update Under the Hood-20240425-144556.yaml

(cherry picked from commit 32b6d39)
colin-rogers-dbt pushed a commit that referenced this pull request May 14, 2024
…#1036)

* Speedup catalog string comparison by using ilike before equals

* Add parens

* add changie

* Update Under the Hood-20240425-144556.yaml

(cherry picked from commit 32b6d39)

Co-authored-by: Kshitij Aranke <[email protected]>
colin-rogers-dbt added a commit that referenced this pull request May 15, 2024
…#1037)

* Speedup catalog string comparison by using ilike before equals

* Add parens

* add changie

* Update Under the Hood-20240425-144556.yaml

(cherry picked from commit 32b6d39)

Co-authored-by: Kshitij Aranke <[email protected]>
Co-authored-by: Colin Rogers <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.7.latest Tag for PR to be backported to the 1.7.latest branch backport 1.8.latest cla:yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] snowflake__get_catalog_schemas_where_clause_sql is very slow
3 participants