Skip to content

Commit

Permalink
Update snowflake-configs.md (#6231)
Browse files Browse the repository at this point in the history
  • Loading branch information
@mirnawong1
mirnawong1 authored Oct 17, 2024
2 parents ce4095a + d668969 commit 9ddc992
Showing 1 changed file with 0 additions and 264 deletions.
264 changes: 0 additions & 264 deletions website/docs/reference/resource-configs/snowflake-configs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -695,270 +695,6 @@ models:
</File>



The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about).
This materialization is specific to Snowflake, which means that any model configuration that
would normally come along for the ride from `dbt-core` (e.g. as with a `view`) may not be available
for dynamic tables. This gap will decrease in future patches and versions.
While this materialization is specific to Snowflake, it very much follows the implementation
of [materialized views](/docs/build/materializations#Materialized-View).
In particular, dynamic tables have access to the `on_configuration_change` setting.
Dynamic tables are supported with the following configuration parameters:

<VersionBlock lastVersion="1.8">

| Parameter | Type | Required | Default | Change Monitoring Support |
|--------------------|------------|----------|-------------|---------------------------|
| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `<string>` | no | `apply` | n/a |
| [`target_lag`](#target-lag) | `<string>` | yes | | alter |
| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `<string>` | yes | | alter |
</VersionBlock>

<VersionBlock firstVersion="1.9">

| Parameter | Type | Required | Default | Change Monitoring Support |
|--------------------|------------|----------|-------------|---------------------------|
| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `<string>` | no | `apply` | n/a |
| [`target_lag`](#target-lag) | `<string>` | yes | | alter |
| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `<string>` | yes | | alter |
| [`refresh_mode`](#refresh-mode) | `<string>` | no | `AUTO` | refresh |
| [`initialize`](#initialize) | `<string>` | no | `ON_CREATE` | n/a |

</VersionBlock>

<VersionBlock lastVersion="1.8">

<Tabs
groupId="config-languages"
defaultValue="project-yaml"
values={[
{ label: 'Project file', value: 'project-yaml', },
{ label: 'Property file', value: 'property-yaml', },
{ label: 'Config block', value: 'config', },
]
}>

<TabItem value="project-yaml">

<File name='dbt_project.yml'>

```yaml
models:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): dynamic_table
[+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
[+](/reference/resource-configs/plus-prefix)[target_lag](#target-lag): downstream | <time-delta>
[+](/reference/resource-configs/plus-prefix)[snowflake_warehouse](#configuring-virtual-warehouses): <warehouse-name>
```

</File>

</TabItem>


<TabItem value="property-yaml">

<File name='models/properties.yml'>

```yaml
version: 2
models:
- name: [<model-name>]
config:
[materialized](/reference/resource-configs/materialized): dynamic_table
[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
[target_lag](#target-lag): downstream | <time-delta>
[snowflake_warehouse](#configuring-virtual-warehouses): <warehouse-name>
```

</File>

</TabItem>


<TabItem value="config">

<File name='models/<model_name>.sql'>

```jinja
{{ config(
[materialized](/reference/resource-configs/materialized)="dynamic_table",
[on_configuration_change](/reference/resource-configs/on_configuration_change)="apply" | "continue" | "fail",
[target_lag](#target-lag)="downstream" | "<integer> seconds | minutes | hours | days",
[snowflake_warehouse](#configuring-virtual-warehouses)="<warehouse-name>",
) }}
```

</File>

</TabItem>

</Tabs>

</VersionBlock>

<VersionBlock firstVersion="1.9">

<Tabs
groupId="config-languages"
defaultValue="project-yaml"
values={[
{ label: 'Project file', value: 'project-yaml', },
{ label: 'Property file', value: 'property-yaml', },
{ label: 'Config block', value: 'config', },
]
}>

<TabItem value="project-yaml">

<File name='dbt_project.yml'>

```yaml
models:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): dynamic_table
[+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
[+](/reference/resource-configs/plus-prefix)[target_lag](#target-lag): downstream | <time-delta>
[+](/reference/resource-configs/plus-prefix)[snowflake_warehouse](#configuring-virtual-warehouses): <warehouse-name>
[+](/reference/resource-configs/plus-prefix)[refresh_mode](#refresh-mode): AUTO | FULL | INCREMENTAL
[+](/reference/resource-configs/plus-prefix)[initialize](#initialize): ON_CREATE | ON_SCHEDULE
```

</File>

</TabItem>


<TabItem value="property-yaml">

<File name='models/properties.yml'>

```yaml
version: 2
models:
- name: [<model-name>]
config:
[materialized](/reference/resource-configs/materialized): dynamic_table
[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
[target_lag](#target-lag): downstream | <time-delta>
[snowflake_warehouse](#configuring-virtual-warehouses): <warehouse-name>
[refresh_mode](#refresh-mode): AUTO | FULL | INCREMENTAL
[initialize](#initialize): ON_CREATE | ON_SCHEDULE
```

</File>

</TabItem>


<TabItem value="config">

<File name='models/<model_name>.sql'>

```jinja
{{ config(
[materialized](/reference/resource-configs/materialized)="dynamic_table",
[on_configuration_change](/reference/resource-configs/on_configuration_change)="apply" | "continue" | "fail",
[target_lag](#target-lag)="downstream" | "<integer> seconds | minutes | hours | days",
[snowflake_warehouse](#configuring-virtual-warehouses)="<warehouse-name>",
[refresh_mode](#refresh-mode)="AUTO" | "FULL" | "INCREMENTAL",
[initialize](#initialize)="ON_CREATE" | "ON_SCHEDULE",
) }}
```

</File>

</TabItem>

</Tabs>

</VersionBlock>

Learn more about these parameters in Snowflake's [docs](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table):

### Target lag

Snowflake allows two configuration scenarios for scheduling automatic refreshes:
- **Time-based** &mdash; Provide a value of the form `<int> { seconds | minutes | hours | days }`. For example, if the dynamic table needs to be updated every 30 minutes, use `target_lag='30 minutes'`.
- **Downstream** &mdash; Applicable when the dynamic table is referenced by other dynamic tables. In this scenario, `target_lag='downstream'` allows for refreshes to be controlled at the target, instead of at each layer.

Learn more about `target_lag` in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh#understanding-target-lag).

<VersionBlock firstVersion="1.9">

### Refresh mode

Snowflake allows three options for refresh mode:
- **AUTO** &mdash; Enforces an incremental refresh of the dynamic table by default. If the `CREATE DYNAMIC TABLE` statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode.
- **FULL** &mdash; Enforces a full refresh of the dynamic table, even if the dynamic table can be incrementally refreshed.
- **INCREMENTAL** &mdash; Enforces an incremental refresh of the dynamic table. If the query that underlies the dynamic table can’t perform an incremental refresh, dynamic table creation fails and displays an error message.

Learn more about `refresh_mode` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh).

### Initialize

Snowflake allows two options for initialize:
- **ON_CREATE** &mdash; Refreshes the dynamic table synchronously at creation. If this refresh fails, dynamic table creation fails and displays an error message.
- **ON_SCHEDULE** &mdash; Refreshes the dynamic table at the next scheduled refresh.

Learn more about `initialize` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh).

</VersionBlock>

### Limitations

As with materialized views on most data platforms, there are limitations associated with dynamic tables. Some worth noting include:

- Dynamic table SQL has a [limited feature set](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#query-constructs-not-currently-supported-in-dynamic-tables).
- Dynamic table SQL cannot be updated; the dynamic table must go through a `--full-refresh` (DROP/CREATE).
- Dynamic tables cannot be downstream from: materialized views, external tables, streams.
- Dynamic tables cannot reference a view that is downstream from another dynamic table.

Find more information about dynamic table limitations in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#dynamic-table-limitations-and-supported-functions).

For dbt limitations, these dbt features are not supported:
- [Model contracts](/docs/collaborate/govern/model-contracts)
- [Copy grants configuration](/reference/resource-configs/snowflake-configs#copying-grants)

<VersionBlock lastVersion="1.6">

#### Changing materialization to and from "dynamic_table"

Version `1.6.x` does not support altering the materialization from a non-dynamic table be a dynamic table and vice versa.
Re-running with the `--full-refresh` does not resolve this either.
The workaround is manually dropping the existing model in the warehouse prior to calling `dbt run`.
This only needs to be done once for the conversion.

For example, assume for the example model below, `my_model`, has already been materialized to the underlying data platform via `dbt run`.
If the model config is updated to `materialized="dynamic_table"`, dbt will return an error.
The workaround is to execute `DROP TABLE my_model` on the data warehouse before trying the model again.

<File name='my_model.sql'>

```yaml
{{ config(
materialized="table" # or any model type (e.g. view, incremental)
) }}
```

</File>

</VersionBlock>

## Source freshness known limitation

Snowflake calculates source freshness using information from the `LAST_ALTERED` column, meaning it relies on a field updated whenever any object undergoes modification, not only data updates. No action must be taken, but analytics teams should note this caveat.
Expand Down

0 comments on commit 9ddc992

Please sign in to comment.