Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Update Dynamic table parameters on materialization #893

Closed
wants to merge 5 commits into from

Conversation

HenkvanDyk
Copy link
Contributor

@HenkvanDyk HenkvanDyk commented Jan 31, 2024

resolves #868
docs dbt-labs/docs.getdbt.com/#

Documentation updates here dbt-labs/docs.getdbt.com#4835

Problem

Dynamic tables were refreshed on each run in 1.6.x, but not in 1.7.x which resulted in 2 things:

  1. In 1.6.x the recreation of a dynamic table would trigger a refresh. Any downstream models which are recreated in the same run could (if the upstream target_lag = Downstream) trigger upstream models again. This resulted in redundant refreshes and delayed runs.
  2. Users who upgraded to 1.7.x who actually want to refresh a dynamic table on a run would need to add a macro to trigger a refresh (I'd want to do this to ensure my orchestration system checks that a table is up to date before downstream tables read from it)

Solution

Snowflake introduced new parameters which can be set on dynamic tables creation. Docs
The INITIALIZE feature allows one to refresh a dynamic table either only on schedule, or also on creation.
This could allow orchestration of refreshes to be done when a dbt run is invoked, which "creates" the table again.
This could resolve #859 as well, as DBT would not need to set a refresh strategy.

Alterations of these parameters are not included as these settings cannot be altered after creation. https://docs.snowflake.com/en/sql-reference/sql/alter-dynamic-table#parameters

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

@HenkvanDyk HenkvanDyk requested a review from a team as a code owner January 31, 2024 10:09
@cla-bot cla-bot bot added the cla:yes label Jan 31, 2024
@HenkvanDyk HenkvanDyk marked this pull request as draft January 31, 2024 10:11
@HenkvanDyk
Copy link
Contributor Author

I am having a hard time testing this locally to check the creation implements the features as expected.

I've tried installing dbt-snowflake but when running tests the import is failing - any guidance on the structure of my venv would be appreciated.

image

@HenkvanDyk HenkvanDyk marked this pull request as ready for review January 31, 2024 13:46
@mikealfare
Copy link
Contributor

Thanks for your contribution @HenkvanDyk! The implementation for object creation looks fine, but there are a few more items to address. At a high level:

  • update the describe query to pull back the new attributes
  • incorporate those new attributes when producing the changeset here and here
  • add/update tests

Adding the new fields to the describe macro will allow for them to be checked in the tests; the tests call the describe macro to inspect the object in the database (which in turn also tests the describe macro if you provide a non-default setting).

@mikealfare
Copy link
Contributor

I am having a hard time testing this locally to check the creation implements the features as expected.

I've tried installing dbt-snowflake but when running tests the import is failing - any guidance on the structure of my venv would be appreciated.

image

This looks like it might be a result of our decoupling work earlier this year (the error is familiar). We pulled what we affectionately referred to as "the adapters zone" into its own package dbt-adapters here, and then removed dbt-core as a functional dependency within the adapter implementations (e.g. dbt-snowflake). Depending on when you pulled down these branches, you might have gotten something that was still under construction. That period has since passed though, so you should be able to setup a fresh environment now.

@mikealfare
Copy link
Contributor

This PR will be continued in #1081. I'm closing this to avoid confusion, however the work will continue. Please refer to #1081 moving forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants