From 357d269dc4be8622cababd6275672c453dc1fce3 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Fri, 27 Sep 2024 13:24:45 -0400
Subject: [PATCH 01/36] Beta documentation for Snowflake Iceberg tables
---
.../resource-configs/snowflake-configs.md | 400 ++++++++++++++++--
1 file changed, 361 insertions(+), 39 deletions(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index a59bc8dee00..2bb706b975f 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -9,6 +9,367 @@ To-do:
- use the reference doc structure for this article / split into separate articles
--->
+## Iceberg table format
+
+Snowflake Iceberg tables support for the dbt-snowflake adapter is available for three out-of-the-box materializations:
+- [Table](/docs/build/materializations#table)
+- [Incremental](/docs/build/materializations#view)
+- [Dynamic](#dynamic-tables)
+
+For now, Iceberg tables require a [behavior flag](/reference/global-configs/behavior-changes) due to performance impact related to using Iceberg tables. Snowflake does not support `is_iceberg` on the `Show Objects` queries, which dbt depends on for metadata.
+
+To use Iceberg, set the Iceberg Materialization Behavior flag to true in your dbt_project.yml:
+
+```yaml
+flags:
+ enable_iceberg_materializations: True
+```
+
+If you do not set this behavior flag, you will get an error, and dbt won’t materialize the table in the Iceberg format.
+
+The following configurations are supported:
+
+| Field | Type | Required | Description | Sample input |
+|-------|------|----------|-------------|-------|
+|Table Format | String | Yes | Configures the objects table format. | `iceberg` |
+|External volumne | String | Yes | Specifies the identifier (name) of the external volume where the Iceberg table stores its metadata files and data in Parquet format. If you don’t specify this parameter, the Iceberg table defaults to the external volume for the schema, database, or account. The schema takes precedence over the database, and the database takes precedence over the account.| `my_s3_bucket` |
+|Base location |String | No | Defines the directory path for Snowflake to write the table data and metadata files. You can’t change this after creating the table. If not specified, Snowflake will use the current schema. | `jaffle_marketing_folder` |
+
+Example of an Iceberg table materialization configuration:
+
+
+
+```sql
+
+{{
+ config(
+ materialized = "table",
+ table_format="iceberg",
+ external_volume="s3_iceberg_snow",
+ base_location_subpath="jaffle_shop_folder",
+ )
+}}
+
+select * from {{ ref('raw_orders') }}
+
+```
+
+
+
+There are some limitations to the implementation you need to be aware of:
+
+- You cannot create transient or temporary Iceberg tables on Snowflake.
+- Supplying an input to `base_location_subpath` will always be apprehended to your schema name. Currently, you cannot override this behavior, which ensures that dbt can differentiate Iceberg model builds based on the environment.
+- Snowflake has limitations for Dynamic Tables. Check out the [Snowflake docs]( https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#usage-notes) for more information. By default, we recommend leaving the `base_location_subpath` field blank, as each target has it’s own default path.
+- Copying rows from an existing default-format model into an Iceberg-format model is not currently supported. This means for incremental models, you will have to use `--full-refresh` to drop and recreate the table from scratch. An exception is thrown without `--full-refresh`.
+- Only `on_schema_change` strategies fail/ignore are supported at p0.
+
+
+## Dynamic tables
+
+The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about).
+This materialization is specific to Snowflake, which means that any model configuration that
+would normally come along for the ride from `dbt-core` (e.g. as with a `view`) may not be available
+for dynamic tables. This gap will decrease in future patches and versions.
+While this materialization is specific to Snowflake, it very much follows the implementation
+of [materialized views](/docs/build/materializations#Materialized-View).
+In particular, dynamic tables have access to the `on_configuration_change` setting.
+Dynamic tables are supported with the following configuration parameters:
+
+
+
+| Parameter | Type | Required | Default | Change Monitoring Support |
+|--------------------|------------|----------|-------------|---------------------------|
+| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `` | no | `apply` | n/a |
+| [`target_lag`](#target-lag) | `` | yes | | alter |
+| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `` | yes | | alter |
+
+
+
+
+| Parameter | Type | Required | Default | Change Monitoring Support |
+|--------------------|------------|----------|-------------|---------------------------|
+| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `` | no | `apply` | n/a |
+| [`target_lag`](#target-lag) | `` | yes | | alter |
+| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `` | yes | | alter |
+| [`refresh_mode`](#refresh-mode) | `` | no | `AUTO` | refresh |
+| [`initialize`](#initialize) | `` | no | `ON_CREATE` | n/a |
+
+
+
+
+
+
+
+
+
+
+
+```yaml
+models:
+ [](/reference/resource-configs/resource-path):
+ [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): dynamic_table
+ [+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
+ [+](/reference/resource-configs/plus-prefix)[target_lag](#target-lag): downstream |
+ [+](/reference/resource-configs/plus-prefix)[snowflake_warehouse](#configuring-virtual-warehouses):
+
+```
+
+
+
+
+
+
+
+
+
+
+```yaml
+version: 2
+
+models:
+ - name: []
+ config:
+ [materialized](/reference/resource-configs/materialized): dynamic_table
+ [on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
+ [target_lag](#target-lag): downstream |
+ [snowflake_warehouse](#configuring-virtual-warehouses):
+
+```
+
+
+
+
+
+
+
+
+
+
+```jinja
+
+{{ config(
+ [materialized](/reference/resource-configs/materialized)="dynamic_table",
+ [on_configuration_change](/reference/resource-configs/on_configuration_change)="apply" | "continue" | "fail",
+ [target_lag](#target-lag)="downstream" | " seconds | minutes | hours | days",
+ [snowflake_warehouse](#configuring-virtual-warehouses)="",
+
+) }}
+
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+```yaml
+models:
+ [](/reference/resource-configs/resource-path):
+ [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): dynamic_table
+ [+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
+ [+](/reference/resource-configs/plus-prefix)[target_lag](#target-lag): downstream |
+ [+](/reference/resource-configs/plus-prefix)[snowflake_warehouse](#configuring-virtual-warehouses):
+ [+](/reference/resource-configs/plus-prefix)[refresh_mode](#refresh-mode): AUTO | FULL | INCREMENTAL
+ [+](/reference/resource-configs/plus-prefix)[initialize](#initialize): ON_CREATE | ON_SCHEDULE
+
+```
+
+
+
+
+
+
+
+
+
+
+```yaml
+version: 2
+
+models:
+ - name: []
+ config:
+ [materialized](/reference/resource-configs/materialized): dynamic_table
+ [on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
+ [target_lag](#target-lag): downstream |
+ [snowflake_warehouse](#configuring-virtual-warehouses):
+ [refresh_mode](#refresh-mode): AUTO | FULL | INCREMENTAL
+ [initialize](#initialize): ON_CREATE | ON_SCHEDULE
+
+```
+
+
+
+
+
+
+
+
+
+
+```jinja
+
+{{ config(
+ [materialized](/reference/resource-configs/materialized)="dynamic_table",
+ [on_configuration_change](/reference/resource-configs/on_configuration_change)="apply" | "continue" | "fail",
+ [target_lag](#target-lag)="downstream" | " seconds | minutes | hours | days",
+ [snowflake_warehouse](#configuring-virtual-warehouses)="",
+ [refresh_mode](#refresh-mode)="AUTO" | "FULL" | "INCREMENTAL",
+ [initialize](#initialize)="ON_CREATE" | "ON_SCHEDULE",
+
+) }}
+
+```
+
+
+
+
+
+
+
+
+
+Learn more about these parameters in Snowflake's [docs](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table):
+
+### Target lag
+
+Snowflake allows two configuration scenarios for scheduling automatic refreshes:
+- **Time-based** — Provide a value of the form ` { seconds | minutes | hours | days }`. For example, if the dynamic table needs to be updated every 30 minutes, use `target_lag='30 minutes'`.
+- **Downstream** — Applicable when the dynamic table is referenced by other dynamic tables. In this scenario, `target_lag='downstream'` allows for refreshes to be controlled at the target, instead of at each layer.
+
+Learn more about `target_lag` in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh#understanding-target-lag).
+
+
+
+### Refresh mode
+
+Snowflake allows three options for refresh mode:
+- **AUTO** — Enforces an incremental refresh of the dynamic table by default. If the `CREATE DYNAMIC TABLE` statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode.
+- **FULL** — Enforces a full refresh of the dynamic table, even if the dynamic table can be incrementally refreshed.
+- **INCREMENTAL** — Enforces an incremental refresh of the dynamic table. If the query that underlies the dynamic table can’t perform an incremental refresh, dynamic table creation fails and displays an error message.
+
+Learn more about `refresh_mode` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh).
+
+### Initialize
+
+Snowflake allows two options for initialize:
+- **ON_CREATE** — Refreshes the dynamic table synchronously at creation. If this refresh fails, dynamic table creation fails and displays an error message.
+- **ON_SCHEDULE** — Refreshes the dynamic table at the next scheduled refresh.
+
+Learn more about `initialize` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh).
+
+
+
+### Limitations
+
+As with materialized views on most data platforms, there are limitations associated with dynamic tables. Some worth noting include:
+
+- Dynamic table SQL has a [limited feature set](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#query-constructs-not-currently-supported-in-dynamic-tables).
+- Dynamic table SQL cannot be updated; the dynamic table must go through a `--full-refresh` (DROP/CREATE).
+- Dynamic tables cannot be downstream from: materialized views, external tables, streams.
+- Dynamic tables cannot reference a view that is downstream from another dynamic table.
+
+Find more information about dynamic table limitations in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#dynamic-table-limitations-and-supported-functions).
+
+For dbt limitations, these dbt features are not supported:
+- [Model contracts](/docs/collaborate/govern/model-contracts)
+- [Copy grants configuration](/reference/resource-configs/snowflake-configs#copying-grants)
+
+
+
+#### Changing materialization to and from "dynamic_table"
+
+Version `1.6.x` does not support altering the materialization from a non-dynamic table be a dynamic table and vice versa.
+Re-running with the `--full-refresh` does not resolve this either.
+The workaround is manually dropping the existing model in the warehouse prior to calling `dbt run`.
+This only needs to be done once for the conversion.
+
+For example, assume for the example model below, `my_model`, has already been materialized to the underlying data platform via `dbt run`.
+If the model config is updated to `materialized="dynamic_table"`, dbt will return an error.
+The workaround is to execute `DROP TABLE my_model` on the data warehouse before trying the model again.
+
+
+
+```yaml
+
+{{ config(
+ materialized="table" # or any model type (e.g. view, incremental)
+) }}
+
+```
+
+
+
+
+
+## Temporary tables
+
+Incremental table merges for Snowflake prefer to utilize a `view` rather than a `temporary table`. The reasoning is to avoid the database write step that a temporary table would initiate and save compile time.
+
+However, some situations remain where a temporary table would achieve results faster or more safely. The `tmp_relation_type` configuration enables you to opt in to temporary tables for incremental builds. This is defined as part of the model configuration.
+
+To guarantee accuracy, an incremental model using the `delete+insert` strategy with a `unique_key` defined requires a temporary table; trying to change this to a view will result in an error.
+
+Defined in the project YAML:
+
+
+
+```yaml
+name: my_project
+
+...
+
+models:
+ :
+ +tmp_relation_type: table | view ## If not defined, view is the default.
+
+```
+
+
+
+In the configuration format for the model SQL file:
+
+
+
+```yaml
+
+{{ config(
+ tmp_relation_type="table | view", ## If not defined, view is the default.
+) }}
+
+```
+
+
+
+
## Transient tables
Snowflake supports the creation of [transient tables](https://docs.snowflake.net/manuals/user-guide/tables-temp-transient.html). Snowflake does not preserve a history for these tables, which can result in a measurable reduction of your Snowflake storage costs. Transient tables participate in time travel to a limited degree with a retention period of 1 day by default with no fail-safe period. Weigh these tradeoffs when deciding whether or not to configure your dbt models as `transient`. **By default, all Snowflake tables created by dbt are `transient`.**
@@ -299,46 +660,7 @@ models:
-## Temporary tables
-
-Incremental table merges for Snowflake prefer to utilize a `view` rather than a `temporary table`. The reasoning is to avoid the database write step that a temporary table would initiate and save compile time.
-
-However, some situations remain where a temporary table would achieve results faster or more safely. The `tmp_relation_type` configuration enables you to opt in to temporary tables for incremental builds. This is defined as part of the model configuration.
-
-To guarantee accuracy, an incremental model using the `delete+insert` strategy with a `unique_key` defined requires a temporary table; trying to change this to a view will result in an error.
-
-Defined in the project YAML:
-
-
-
-```yaml
-name: my_project
-
-...
-
-models:
- :
- +tmp_relation_type: table | view ## If not defined, view is the default.
-
-```
-
-
-
-In the configuration format for the model SQL file:
-
-
-```yaml
-
-{{ config(
- tmp_relation_type="table | view", ## If not defined, view is the default.
-) }}
-
-```
-
-
-
-## Dynamic tables
The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about).
This materialization is specific to Snowflake, which means that any model configuration that
From d71a8ec925921682499495d7a5fdf38d84845190 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Fri, 27 Sep 2024 13:27:42 -0400
Subject: [PATCH 02/36] Edits
---
.../docs/reference/resource-configs/snowflake-configs.md | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 2bb706b975f..b17d4529dfc 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -20,11 +20,17 @@ For now, Iceberg tables require a [behavior flag](/reference/global-configs/beha
To use Iceberg, set the Iceberg Materialization Behavior flag to true in your dbt_project.yml:
+
+
```yaml
+
flags:
enable_iceberg_materializations: True
+
```
+
+
If you do not set this behavior flag, you will get an error, and dbt won’t materialize the table in the Iceberg format.
The following configurations are supported:
From 42a7e0231aaac9d5b5926afe42a118c0d33ca944 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Mon, 30 Sep 2024 11:54:01 +0100
Subject: [PATCH 03/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index b17d4529dfc..78c6a571c79 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -18,7 +18,7 @@ Snowflake Iceberg tables support for the dbt-snowflake adapter is available for
For now, Iceberg tables require a [behavior flag](/reference/global-configs/behavior-changes) due to performance impact related to using Iceberg tables. Snowflake does not support `is_iceberg` on the `Show Objects` queries, which dbt depends on for metadata.
-To use Iceberg, set the Iceberg Materialization Behavior flag to true in your dbt_project.yml:
+To use Iceberg, set the Iceberg Materialization Behavior flag to `True` in your dbt_project.yml:
From ba39e90e079ce320874492876ae35367ab4cab4e Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 13:37:25 -0400
Subject: [PATCH 04/36] Apply suggestions from code review
Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Co-authored-by: Anders
---
.../reference/resource-configs/snowflake-configs.md | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 78c6a571c79..24390c79c3e 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -13,7 +13,7 @@ To-do:
Snowflake Iceberg tables support for the dbt-snowflake adapter is available for three out-of-the-box materializations:
- [Table](/docs/build/materializations#table)
-- [Incremental](/docs/build/materializations#view)
+- [Incremental](/docs/build/materializations#incremental)
- [Dynamic](#dynamic-tables)
For now, Iceberg tables require a [behavior flag](/reference/global-configs/behavior-changes) due to performance impact related to using Iceberg tables. Snowflake does not support `is_iceberg` on the `Show Objects` queries, which dbt depends on for metadata.
@@ -31,17 +31,18 @@ flags:
-If you do not set this behavior flag, you will get an error, and dbt won’t materialize the table in the Iceberg format.
+If you don't set this behavior flag, you will get an error and dbt won’t materialize the table in the Iceberg format.
The following configurations are supported:
| Field | Type | Required | Description | Sample input |
|-------|------|----------|-------------|-------|
|Table Format | String | Yes | Configures the objects table format. | `iceberg` |
-|External volumne | String | Yes | Specifies the identifier (name) of the external volume where the Iceberg table stores its metadata files and data in Parquet format. If you don’t specify this parameter, the Iceberg table defaults to the external volume for the schema, database, or account. The schema takes precedence over the database, and the database takes precedence over the account.| `my_s3_bucket` |
-|Base location |String | No | Defines the directory path for Snowflake to write the table data and metadata files. You can’t change this after creating the table. If not specified, Snowflake will use the current schema. | `jaffle_marketing_folder` |
+|External volume | String | Yes | Specifies the identifier (name) of the external volume where the Iceberg table stores its metadata files and data in Parquet format. If you don’t specify this parameter, the Iceberg table defaults to the external volume for the schema, database, or account. The schema takes precedence over the database, and the database takes precedence over the account.| `my_s3_bucket` |
+|Base location |String | No | Defines the directory path for Snowflake to write the table data and metadata files. You can’t change this after creating the table. If not specified, dbt-snowflake will chose one based on the model's current schema. | `jaffle_marketing_folder` |
-Example of an Iceberg table materialization configuration:
+### Example configuration
+To configure an Iceberg table materialization in dbt, refer to the example configuration:
@@ -62,6 +63,7 @@ select * from {{ ref('raw_orders') }}
+### Limitations
There are some limitations to the implementation you need to be aware of:
- You cannot create transient or temporary Iceberg tables on Snowflake.
From 676ee89bcf8433ad125fee38c656a6dda39fea81 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 13:38:37 -0400
Subject: [PATCH 05/36] Apply suggestions from code review
Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 24390c79c3e..95eb5865d72 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -67,7 +67,7 @@ select * from {{ ref('raw_orders') }}
There are some limitations to the implementation you need to be aware of:
- You cannot create transient or temporary Iceberg tables on Snowflake.
-- Supplying an input to `base_location_subpath` will always be apprehended to your schema name. Currently, you cannot override this behavior, which ensures that dbt can differentiate Iceberg model builds based on the environment.
+- The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
- Snowflake has limitations for Dynamic Tables. Check out the [Snowflake docs]( https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#usage-notes) for more information. By default, we recommend leaving the `base_location_subpath` field blank, as each target has it’s own default path.
- Copying rows from an existing default-format model into an Iceberg-format model is not currently supported. This means for incremental models, you will have to use `--full-refresh` to drop and recreate the table from scratch. An exception is thrown without `--full-refresh`.
- Only `on_schema_change` strategies fail/ignore are supported at p0.
From c6488c8d807083e2537363cea38836bff9f9f301 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 13:39:31 -0400
Subject: [PATCH 06/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 95eb5865d72..3279f5cd0a6 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -70,7 +70,7 @@ There are some limitations to the implementation you need to be aware of:
- The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
- Snowflake has limitations for Dynamic Tables. Check out the [Snowflake docs]( https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#usage-notes) for more information. By default, we recommend leaving the `base_location_subpath` field blank, as each target has it’s own default path.
- Copying rows from an existing default-format model into an Iceberg-format model is not currently supported. This means for incremental models, you will have to use `--full-refresh` to drop and recreate the table from scratch. An exception is thrown without `--full-refresh`.
-- Only `on_schema_change` strategies fail/ignore are supported at p0.
+- For `on_schema_change`, only the `fail` and `ignore` strategies are currently supported.
## Dynamic tables
From f9e14d74fe665bd8555beec6a37f5d4117b157a7 Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Mon, 30 Sep 2024 16:20:04 -0400
Subject: [PATCH 07/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 3279f5cd0a6..59015ad2de6 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -69,7 +69,7 @@ There are some limitations to the implementation you need to be aware of:
- You cannot create transient or temporary Iceberg tables on Snowflake.
- The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
- Snowflake has limitations for Dynamic Tables. Check out the [Snowflake docs]( https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#usage-notes) for more information. By default, we recommend leaving the `base_location_subpath` field blank, as each target has it’s own default path.
-- Copying rows from an existing default-format model into an Iceberg-format model is not currently supported. This means for incremental models, you will have to use `--full-refresh` to drop and recreate the table from scratch. An exception is thrown without `--full-refresh`.
+- You cannot incrementally update a preexisting incremental model to be an Iceberg table. To do so, you must fully rebuild the table with the `--full-refresh` flag.
- For `on_schema_change`, only the `fail` and `ignore` strategies are currently supported.
From bbe2c5cd06d5cb4b97ce9b41f3dfe9eec87a1c5c Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Mon, 30 Sep 2024 16:20:17 -0400
Subject: [PATCH 08/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
website/docs/reference/resource-configs/snowflake-configs.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 59015ad2de6..16d1f2f2f85 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -70,7 +70,6 @@ There are some limitations to the implementation you need to be aware of:
- The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
- Snowflake has limitations for Dynamic Tables. Check out the [Snowflake docs]( https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#usage-notes) for more information. By default, we recommend leaving the `base_location_subpath` field blank, as each target has it’s own default path.
- You cannot incrementally update a preexisting incremental model to be an Iceberg table. To do so, you must fully rebuild the table with the `--full-refresh` flag.
-- For `on_schema_change`, only the `fail` and `ignore` strategies are currently supported.
## Dynamic tables
From 6ff3ffd4c634f977ab07fc643bb000360124c564 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 16:21:29 -0400
Subject: [PATCH 09/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 16d1f2f2f85..ea7d509fc78 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -18,7 +18,7 @@ Snowflake Iceberg tables support for the dbt-snowflake adapter is available for
For now, Iceberg tables require a [behavior flag](/reference/global-configs/behavior-changes) due to performance impact related to using Iceberg tables. Snowflake does not support `is_iceberg` on the `Show Objects` queries, which dbt depends on for metadata.
-To use Iceberg, set the Iceberg Materialization Behavior flag to `True` in your dbt_project.yml:
+To use Iceberg, set the `enable_iceberg_materializations` flag to `True` in your dbt_project.yml:
From f0a066cc1e2806c1557fd16c4bb872f1ffe4920c Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 16:22:54 -0400
Subject: [PATCH 10/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index ea7d509fc78..d7c0fa66f2a 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -66,7 +66,7 @@ select * from {{ ref('raw_orders') }}
### Limitations
There are some limitations to the implementation you need to be aware of:
-- You cannot create transient or temporary Iceberg tables on Snowflake.
+- Using Iceberg tables with dbt, the result is that your query is materialized in Iceberg. However, often, dbt creates intermediary objects as temporary and transient tables for certain materializations, such as incremental ones. It is not possible to configure these temporary objects also to be Iceberg-formatted. You may see non-Iceberg tables created in the logs to support specific materializations, but they will be dropped after usage.
- The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
- Snowflake has limitations for Dynamic Tables. Check out the [Snowflake docs]( https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#usage-notes) for more information. By default, we recommend leaving the `base_location_subpath` field blank, as each target has it’s own default path.
- You cannot incrementally update a preexisting incremental model to be an Iceberg table. To do so, you must fully rebuild the table with the `--full-refresh` flag.
From 0f0b43947863b89e6c34c84af4801163764b0033 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 16:24:14 -0400
Subject: [PATCH 11/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index d7c0fa66f2a..881e00a4cae 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -68,7 +68,7 @@ There are some limitations to the implementation you need to be aware of:
- Using Iceberg tables with dbt, the result is that your query is materialized in Iceberg. However, often, dbt creates intermediary objects as temporary and transient tables for certain materializations, such as incremental ones. It is not possible to configure these temporary objects also to be Iceberg-formatted. You may see non-Iceberg tables created in the logs to support specific materializations, but they will be dropped after usage.
- The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
-- Snowflake has limitations for Dynamic Tables. Check out the [Snowflake docs]( https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#usage-notes) for more information. By default, we recommend leaving the `base_location_subpath` field blank, as each target has it’s own default path.
+- By default, we recommend leaving the `base_location_subpath` field blank, as each target has a default path.
- You cannot incrementally update a preexisting incremental model to be an Iceberg table. To do so, you must fully rebuild the table with the `--full-refresh` flag.
From 2bbb37f9583fd5ce383ad1dbaf6cb14bfdc0a831 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 17:22:53 -0400
Subject: [PATCH 12/36] Adding base location section
---
.../resource-configs/snowflake-configs.md | 22 ++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 881e00a4cae..90a556b1402 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -11,7 +11,8 @@ To-do:
## Iceberg table format
-Snowflake Iceberg tables support for the dbt-snowflake adapter is available for three out-of-the-box materializations:
+Snowflake Iceberg tables support for the dbt-snowflake adapter is available for three out-of-the-box materializations:
+
- [Table](/docs/build/materializations#table)
- [Incremental](/docs/build/materializations#incremental)
- [Dynamic](#dynamic-tables)
@@ -42,6 +43,7 @@ The following configurations are supported:
|Base location |String | No | Defines the directory path for Snowflake to write the table data and metadata files. You can’t change this after creating the table. If not specified, dbt-snowflake will chose one based on the model's current schema. | `jaffle_marketing_folder` |
### Example configuration
+
To configure an Iceberg table materialization in dbt, refer to the example configuration:
@@ -63,7 +65,25 @@ select * from {{ ref('raw_orders') }}
+### Base location
+
+If you were to create an Iceberg Table in the Snowflake query editor, you would be required to provide a `base_location` as an input. dbt Labs has chosen to abstract away from that. The default behavior in dbt is to provide a `base_location` string that follows this convention:
+
+_dbt/{SCHEMA_NAME}/{MODEL_NAME}.
+
+However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation.
+
+dbt does this to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The base location declares where to write the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the external volume declared and what `base_location` is provided. However, passing anything into the `base_location` field, including an empty string, is possible. An empty string could result in future technical debt because it will limit the ability to:
+
+- Navigate the underlying object store (S3/Azure blob)
+- Read Iceberg tables via an object-store integration
+- Grant schema-specific access to tables via object store
+- Use a crawler pointed at the tables within the external volume to build a new catalog with another tool
+
+In summary, dbt-snowflake does not support arbitrary definition of base_location for Iceberg tables. Instead, dbt, by default, writes your tables within a `_dbt/{SCHEMA_NAME}` prefix to ensure easier object-store observability and auditability.
+
### Limitations
+
There are some limitations to the implementation you need to be aware of:
- Using Iceberg tables with dbt, the result is that your query is materialized in Iceberg. However, often, dbt creates intermediary objects as temporary and transient tables for certain materializations, such as incremental ones. It is not possible to configure these temporary objects also to be Iceberg-formatted. You may see non-Iceberg tables created in the logs to support specific materializations, but they will be dropped after usage.
From 104b0d0cb244c97090bc7d4e084e062fa1def9ff Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Mon, 30 Sep 2024 17:28:09 -0400
Subject: [PATCH 13/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 90a556b1402..82a89a78dc0 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -32,7 +32,7 @@ flags:
-If you don't set this behavior flag, you will get an error and dbt won’t materialize the table in the Iceberg format.
+If you don't set this behavior flag, you will get an error and dbt will not materialize the table in the Iceberg format.
The following configurations are supported:
From 3080fd51a366ab51ef44e1ebceced1eec30fb7e9 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 17:52:34 -0400
Subject: [PATCH 14/36] Apply suggestions from code review
Co-authored-by: Amy Chen <46451573+amychen1776@users.noreply.github.com>
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 82a89a78dc0..ddbc66047a1 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -67,7 +67,7 @@ select * from {{ ref('raw_orders') }}
### Base location
-If you were to create an Iceberg Table in the Snowflake query editor, you would be required to provide a `base_location` as an input. dbt Labs has chosen to abstract away from that. The default behavior in dbt is to provide a `base_location` string that follows this convention:
+If you were to create an Iceberg Table in the Snowflake query editor, you would be required to provide a `base_location` as an input. dbt abstracts away from that. The default behavior in dbt is to provide a `base_location` string that follows this convention:
_dbt/{SCHEMA_NAME}/{MODEL_NAME}.
From 40e3cc682648b3dae23542aa73c5315d3552d151 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 17:58:35 -0400
Subject: [PATCH 15/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Amy Chen <46451573+amychen1776@users.noreply.github.com>
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index ddbc66047a1..3efcc1b47e3 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -73,7 +73,7 @@ _dbt/{SCHEMA_NAME}/{MODEL_NAME}.
However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation.
-dbt does this to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The base location declares where to write the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the external volume declared and what `base_location` is provided. However, passing anything into the `base_location` field, including an empty string, is possible. An empty string could result in future technical debt because it will limit the ability to:
+dbt does this to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The base location declares where to write in the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the external volume declared and what `base_location` is provided. However, passing anything into the `base_location` field, including an empty string, is possible. If a project does not enforce an input and always allows empty strings, this could result in future technical debt because it will limit the ability to:
- Navigate the underlying object store (S3/Azure blob)
- Read Iceberg tables via an object-store integration
From b6b89d7be8c8335476366107ea9bee0e15b185f7 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Mon, 30 Sep 2024 18:00:12 -0400
Subject: [PATCH 16/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Amy Chen <46451573+amychen1776@users.noreply.github.com>
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 3efcc1b47e3..99ceaa522f1 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -40,7 +40,7 @@ The following configurations are supported:
|-------|------|----------|-------------|-------|
|Table Format | String | Yes | Configures the objects table format. | `iceberg` |
|External volume | String | Yes | Specifies the identifier (name) of the external volume where the Iceberg table stores its metadata files and data in Parquet format. If you don’t specify this parameter, the Iceberg table defaults to the external volume for the schema, database, or account. The schema takes precedence over the database, and the database takes precedence over the account.| `my_s3_bucket` |
-|Base location |String | No | Defines the directory path for Snowflake to write the table data and metadata files. You can’t change this after creating the table. If not specified, dbt-snowflake will chose one based on the model's current schema. | `jaffle_marketing_folder` |
+|Base location Subpath |String | No | Defines the directory path for Snowflake to write the table data and metadata files. If you change this after creating the table, it will write a new table but leave the existing one. If not specified, dbt-snowflake will default to one based on the model's current schema. | `jaffle_marketing_folder` |
### Example configuration
From 4a3cc7b46f59a4f88a2c5fbdd9c997978bce838d Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 11:37:27 -0400
Subject: [PATCH 17/36] Moving requested text
---
.../docs/reference/resource-configs/snowflake-configs.md | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 99ceaa522f1..fa5d5ad7f65 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -67,11 +67,11 @@ select * from {{ ref('raw_orders') }}
### Base location
-If you were to create an Iceberg Table in the Snowflake query editor, you would be required to provide a `base_location` as an input. dbt abstracts away from that. The default behavior in dbt is to provide a `base_location` string that follows this convention:
+If you were to create an Iceberg table in the Snowflake query editor, you would be required to provide a `base_location` as an input. dbt abstracts away from that. The default behavior in dbt is to provide a `base_location` string that follows this convention:
_dbt/{SCHEMA_NAME}/{MODEL_NAME}.
-However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation.
+However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation. We recommend leaving the `base_location_subpath` field blank, as each target has a default path. The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
dbt does this to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The base location declares where to write in the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the external volume declared and what `base_location` is provided. However, passing anything into the `base_location` field, including an empty string, is possible. If a project does not enforce an input and always allows empty strings, this could result in future technical debt because it will limit the ability to:
@@ -87,8 +87,6 @@ In summary, dbt-snowflake does not support arbitrary definition of base_location
There are some limitations to the implementation you need to be aware of:
- Using Iceberg tables with dbt, the result is that your query is materialized in Iceberg. However, often, dbt creates intermediary objects as temporary and transient tables for certain materializations, such as incremental ones. It is not possible to configure these temporary objects also to be Iceberg-formatted. You may see non-Iceberg tables created in the logs to support specific materializations, but they will be dropped after usage.
-- The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
-- By default, we recommend leaving the `base_location_subpath` field blank, as each target has a default path.
- You cannot incrementally update a preexisting incremental model to be an Iceberg table. To do so, you must fully rebuild the table with the `--full-refresh` flag.
From a11d777a56c96a6d94e876b1122465f62194e905 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 12:42:14 -0400
Subject: [PATCH 18/36] Fixing syntax
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index fa5d5ad7f65..1a2d4a2967d 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -69,7 +69,7 @@ select * from {{ ref('raw_orders') }}
If you were to create an Iceberg table in the Snowflake query editor, you would be required to provide a `base_location` as an input. dbt abstracts away from that. The default behavior in dbt is to provide a `base_location` string that follows this convention:
-_dbt/{SCHEMA_NAME}/{MODEL_NAME}.
+`_dbt/{SCHEMA_NAME}/{MODEL_NAME} `
However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation. We recommend leaving the `base_location_subpath` field blank, as each target has a default path. The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
From f5c33c0485ca7754df4b58a43d43ebe4cde9e007 Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Tue, 1 Oct 2024 14:19:17 -0400
Subject: [PATCH 19/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 1a2d4a2967d..8ec56452819 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -38,7 +38,7 @@ The following configurations are supported:
| Field | Type | Required | Description | Sample input |
|-------|------|----------|-------------|-------|
-|Table Format | String | Yes | Configures the objects table format. | `iceberg` |
+|Table Format | String | Yes | Configures the object table format. | `iceberg` |
|External volume | String | Yes | Specifies the identifier (name) of the external volume where the Iceberg table stores its metadata files and data in Parquet format. If you don’t specify this parameter, the Iceberg table defaults to the external volume for the schema, database, or account. The schema takes precedence over the database, and the database takes precedence over the account.| `my_s3_bucket` |
|Base location Subpath |String | No | Defines the directory path for Snowflake to write the table data and metadata files. If you change this after creating the table, it will write a new table but leave the existing one. If not specified, dbt-snowflake will default to one based on the model's current schema. | `jaffle_marketing_folder` |
From 0749ffef864c55c4806084987f8f0db195cf8caf Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Tue, 1 Oct 2024 14:19:26 -0400
Subject: [PATCH 20/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 8ec56452819..cd17b7ee6e3 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -73,7 +73,7 @@ If you were to create an Iceberg table in the Snowflake query editor, you would
However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation. We recommend leaving the `base_location_subpath` field blank, as each target has a default path. The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
-dbt does this to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The base location declares where to write in the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the external volume declared and what `base_location` is provided. However, passing anything into the `base_location` field, including an empty string, is possible. If a project does not enforce an input and always allows empty strings, this could result in future technical debt because it will limit the ability to:
+dbt does this to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The base location declares where to write in the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the `external volume` declared and what `base_location` is provided. However, passing anything into the `base_location` field, including an empty string, is possible. If a project does not enforce an input and always allows empty strings, this could result in future technical debt because it will limit the ability to:
- Navigate the underlying object store (S3/Azure blob)
- Read Iceberg tables via an object-store integration
From 5da0ca206c2dc434707f155017c9bff319d674ae Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Tue, 1 Oct 2024 14:19:42 -0400
Subject: [PATCH 21/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index cd17b7ee6e3..0e3900046e7 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -80,6 +80,8 @@ dbt does this to enforce best practices. With Snowflake-managed Iceberg format t
- Grant schema-specific access to tables via object store
- Use a crawler pointed at the tables within the external volume to build a new catalog with another tool
+To maintain best practices, we enforce an input. Currently, we do not support overriding the default `base location` input but will consider it based on user feedback.
+
In summary, dbt-snowflake does not support arbitrary definition of base_location for Iceberg tables. Instead, dbt, by default, writes your tables within a `_dbt/{SCHEMA_NAME}` prefix to ensure easier object-store observability and auditability.
### Limitations
From d4208f87bdbbd1b826a034d2ab7ca7e94fc32e2c Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Tue, 1 Oct 2024 14:19:52 -0400
Subject: [PATCH 22/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 0e3900046e7..21958c0cc0a 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -40,7 +40,7 @@ The following configurations are supported:
|-------|------|----------|-------------|-------|
|Table Format | String | Yes | Configures the object table format. | `iceberg` |
|External volume | String | Yes | Specifies the identifier (name) of the external volume where the Iceberg table stores its metadata files and data in Parquet format. If you don’t specify this parameter, the Iceberg table defaults to the external volume for the schema, database, or account. The schema takes precedence over the database, and the database takes precedence over the account.| `my_s3_bucket` |
-|Base location Subpath |String | No | Defines the directory path for Snowflake to write the table data and metadata files. If you change this after creating the table, it will write a new table but leave the existing one. If not specified, dbt-snowflake will default to one based on the model's current schema. | `jaffle_marketing_folder` |
+|Base location Subpath |String | No | Defines the directory path for Snowflake to write the table data and metadata files. If you change this after creating the table, it will write a new table but leave the existing one. | `jaffle_marketing_folder` |
### Example configuration
From b20edbc413f3f448ec76028608e3f9282b85d356 Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Tue, 1 Oct 2024 14:20:00 -0400
Subject: [PATCH 23/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 21958c0cc0a..7951b69ced2 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -71,7 +71,7 @@ If you were to create an Iceberg table in the Snowflake query editor, you would
`_dbt/{SCHEMA_NAME}/{MODEL_NAME} `
-However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation. We recommend leaving the `base_location_subpath` field blank, as each target has a default path. The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
+However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation. We recommend leaving the `base_location_subpath` field blank by default. The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
dbt does this to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The base location declares where to write in the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the `external volume` declared and what `base_location` is provided. However, passing anything into the `base_location` field, including an empty string, is possible. If a project does not enforce an input and always allows empty strings, this could result in future technical debt because it will limit the ability to:
From d1de103b8c749d5a184b590677956573d30e1f04 Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Tue, 1 Oct 2024 14:29:21 -0400
Subject: [PATCH 24/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 7951b69ced2..471d955f8f4 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -11,7 +11,7 @@ To-do:
## Iceberg table format
-Snowflake Iceberg tables support for the dbt-snowflake adapter is available for three out-of-the-box materializations:
+The dbt-snowflake adapter supports the Iceberg table format. It is available for three of the Snowflake materializations:
- [Table](/docs/build/materializations#table)
- [Incremental](/docs/build/materializations#incremental)
From 265c624da4c8ef222e1a9b5ba137ecbf5e6aaf1e Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Tue, 1 Oct 2024 14:29:39 -0400
Subject: [PATCH 25/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 471d955f8f4..cf033bf8fe7 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -17,7 +17,7 @@ The dbt-snowflake adapter supports the Iceberg table format. It is available for
- [Incremental](/docs/build/materializations#incremental)
- [Dynamic](#dynamic-tables)
-For now, Iceberg tables require a [behavior flag](/reference/global-configs/behavior-changes) due to performance impact related to using Iceberg tables. Snowflake does not support `is_iceberg` on the `Show Objects` queries, which dbt depends on for metadata.
+For now, to create Iceberg tables, you must implement a [behavior flag](/reference/global-configs/behavior-changes) due to performance impact related to using Iceberg tables. Snowflake does not support `is_iceberg` on the `Show Objects` query, which dbt depends on for metadata.
To use Iceberg, set the `enable_iceberg_materializations` flag to `True` in your dbt_project.yml:
From 3c13e45c78e859058f2de90056d5b03fa32220ca Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Tue, 1 Oct 2024 14:43:21 -0400
Subject: [PATCH 26/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
website/docs/reference/resource-configs/snowflake-configs.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index cf033bf8fe7..e5d02ce773c 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -32,7 +32,6 @@ flags:
-If you don't set this behavior flag, you will get an error and dbt will not materialize the table in the Iceberg format.
The following configurations are supported:
From d27684c38bd536791be94d4c379b088348dabf1a Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 16:42:56 -0400
Subject: [PATCH 27/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
website/docs/reference/resource-configs/snowflake-configs.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index e5d02ce773c..42f90447540 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -72,7 +72,9 @@ If you were to create an Iceberg table in the Snowflake query editor, you would
However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation. We recommend leaving the `base_location_subpath` field blank by default. The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
-dbt does this to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The base location declares where to write in the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the `external volume` declared and what `base_location` is provided. However, passing anything into the `base_location` field, including an empty string, is possible. If a project does not enforce an input and always allows empty strings, this could result in future technical debt because it will limit the ability to:
+#### rationale
+
+dbt manages `base_location` on behalf of users to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The `base_ location` parameter declares where to write the data within the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the `external volume` declared and what `base_location` is provided. However, Snowflake permits passing anything into the `base_location` field, including an empty string, even reusing the same path across multiple tables. This could result in future technical debt because it will limit the ability to:
- Navigate the underlying object store (S3/Azure blob)
- Read Iceberg tables via an object-store integration
From bf676bfd96c7b3d061e7a654479d6177f038ff6f Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 16:43:16 -0400
Subject: [PATCH 28/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 42f90447540..d687682a97f 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -72,7 +72,7 @@ If you were to create an Iceberg table in the Snowflake query editor, you would
However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation. We recommend leaving the `base_location_subpath` field blank by default. The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
-#### rationale
+#### Rationale
dbt manages `base_location` on behalf of users to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The `base_ location` parameter declares where to write the data within the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the `external volume` declared and what `base_location` is provided. However, Snowflake permits passing anything into the `base_location` field, including an empty string, even reusing the same path across multiple tables. This could result in future technical debt because it will limit the ability to:
From 6bf494efa6668d12219e2410696eb6e4694bb1a1 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 16:44:16 -0400
Subject: [PATCH 29/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index d687682a97f..3786d86fcd5 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -74,7 +74,7 @@ However, dbt allows users to configure a `base_location_subpath` that is empty b
#### Rationale
-dbt manages `base_location` on behalf of users to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The `base_ location` parameter declares where to write the data within the external volume. The Snowflake Iceberg catalog will keep track of your Iceberg table regardless of where the data lives within the `external volume` declared and what `base_location` is provided. However, Snowflake permits passing anything into the `base_location` field, including an empty string, even reusing the same path across multiple tables. This could result in future technical debt because it will limit the ability to:
+dbt manages `base_location` on behalf of users to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The `base_ location` parameter declares where to write the data within the external volume. The Snowflake Iceberg catalog keeps track of your Iceberg table regardless of where the data lives within the `external volume` declared and the `base_location` provided. However, Snowflake permits passing anything into the `base_location` field, including an empty string, even reusing the same path across multiple tables. This behavior could result in future technical debt because it will limit the ability to:
- Navigate the underlying object store (S3/Azure blob)
- Read Iceberg tables via an object-store integration
From b84dbebcdde6c85b5c3159cbcfeee409b42397c4 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 16:47:43 -0400
Subject: [PATCH 30/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
.../docs/reference/resource-configs/snowflake-configs.md | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 3786d86fcd5..f25dd8cf9f6 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -70,7 +70,12 @@ If you were to create an Iceberg table in the Snowflake query editor, you would
`_dbt/{SCHEMA_NAME}/{MODEL_NAME} `
-However, dbt allows users to configure a `base_location_subpath` that is empty by default but, when provided, will be concatenated to the end of the previously described pattern for `base_location` string generation. We recommend leaving the `base_location_subpath` field blank by default. The `base_location_subpath` input is always appended to your schema name, and this behavior cannot be overridden. This ensures dbt can differentiate Iceberg model builds by environment.
+#### Base Location Subpath
+We recommend using dbt's auto-generated `base_location`. However, if you have a need to customize the resulting `base_location`, dbt allows users to configure a `base_location_subpath`. When specified, it will be concatenated to the end of the previously described pattern for `base_location` string generation.
+
+For example, `config(base_location_subpath="prod")` will generate a `base_location` of the form `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`
+
+A theoretical (not recommended) use case: re-using an `EXTERNAL VOLUME` while still maintaining isolation across development and production environments. This is not recommended as storage permissions should configured on the external volume and underlying storage, not paths that can be modified by any analytics engineer.
#### Rationale
From ddf2abc2d9bbb650dca11b5bd37c72101f8276b0 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 16:49:09 -0400
Subject: [PATCH 31/36] Apply suggestions from code review
Editorial changes
---
.../docs/reference/resource-configs/snowflake-configs.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index f25dd8cf9f6..8bfe421614e 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -71,11 +71,11 @@ If you were to create an Iceberg table in the Snowflake query editor, you would
`_dbt/{SCHEMA_NAME}/{MODEL_NAME} `
#### Base Location Subpath
-We recommend using dbt's auto-generated `base_location`. However, if you have a need to customize the resulting `base_location`, dbt allows users to configure a `base_location_subpath`. When specified, it will be concatenated to the end of the previously described pattern for `base_location` string generation.
+We recommend using dbt's auto-generated `base_location`. However, if you need to customize the resulting `base_location`, dbt allows users to configure a `base_location_subpath`. When specified, the subpath concatenates to the end of the previously described pattern for `base_location` string generation.
-For example, `config(base_location_subpath="prod")` will generate a `base_location` of the form `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`
+For example, `config(base_location_subpath="prod")` will generate a `base_location` of the form `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`.
-A theoretical (not recommended) use case: re-using an `EXTERNAL VOLUME` while still maintaining isolation across development and production environments. This is not recommended as storage permissions should configured on the external volume and underlying storage, not paths that can be modified by any analytics engineer.
+A theoretical (but not recommended) use case is re-using an `EXTERNAL VOLUME` while maintaining isolation across development and production environments. We recommend against this as storage permissions should configured on the external volume and underlying storage, not paths that any analytics engineer can modify.
#### Rationale
From 9b42ef46d99f2ba546127be142be59ef9ab0fb25 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 16:49:48 -0400
Subject: [PATCH 32/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 8bfe421614e..5c9ba51dd8c 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -54,7 +54,6 @@ To configure an Iceberg table materialization in dbt, refer to the example confi
materialized = "table",
table_format="iceberg",
external_volume="s3_iceberg_snow",
- base_location_subpath="jaffle_shop_folder",
)
}}
From 4a097c5d120b8f0230daecbe0122cddb304e5d98 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 16:50:09 -0400
Subject: [PATCH 33/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 5c9ba51dd8c..66a153a1aca 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -65,7 +65,7 @@ select * from {{ ref('raw_orders') }}
### Base location
-If you were to create an Iceberg table in the Snowflake query editor, you would be required to provide a `base_location` as an input. dbt abstracts away from that. The default behavior in dbt is to provide a `base_location` string that follows this convention:
+Snowflake's `CREATE ICEBERG TABLE` DDEL requires that a `base_location` be provided. dbt defines this parameter on behalf of the user to both streamline usage and enforce basic isolation of table data within the `EXTERNAL VOLUME`. The default behavior in dbt is to provide a `base_location` string that follows the below convention:
`_dbt/{SCHEMA_NAME}/{MODEL_NAME} `
From f571dbb88c6db13cb0b14ae1b458d9f1a2322118 Mon Sep 17 00:00:00 2001
From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
Date: Tue, 1 Oct 2024 16:51:41 -0400
Subject: [PATCH 34/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 66a153a1aca..78ee9665d5c 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -65,7 +65,7 @@ select * from {{ ref('raw_orders') }}
### Base location
-Snowflake's `CREATE ICEBERG TABLE` DDEL requires that a `base_location` be provided. dbt defines this parameter on behalf of the user to both streamline usage and enforce basic isolation of table data within the `EXTERNAL VOLUME`. The default behavior in dbt is to provide a `base_location` string that follows the below convention:
+Snowflake's `CREATE ICEBERG TABLE` DDEL requires that a `base_location` be provided. dbt defines this parameter on the user's behalf to streamline usage and enforce basic isolation of table data within the `EXTERNAL VOLUME`. The default behavior in dbt is to provide a `base_location` string that follows the below convention:
`_dbt/{SCHEMA_NAME}/{MODEL_NAME} `
From f354d11bc60399f8aa3af654cf1e9bb9740a2ccf Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Wed, 2 Oct 2024 09:16:36 -0400
Subject: [PATCH 35/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
website/docs/reference/resource-configs/snowflake-configs.md | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 78ee9665d5c..33f4a1cb48a 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -65,9 +65,7 @@ select * from {{ ref('raw_orders') }}
### Base location
-Snowflake's `CREATE ICEBERG TABLE` DDEL requires that a `base_location` be provided. dbt defines this parameter on the user's behalf to streamline usage and enforce basic isolation of table data within the `EXTERNAL VOLUME`. The default behavior in dbt is to provide a `base_location` string that follows the below convention:
-
-`_dbt/{SCHEMA_NAME}/{MODEL_NAME} `
+Snowflake's `CREATE ICEBERG TABLE` DDL requires that a `base_location` be provided. dbt defines this parameter on the user's behalf to streamline usage and enforce basic isolation of table data within the `EXTERNAL VOLUME`. The default behavior in dbt is to provide a `base_location` string of the form: `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`
#### Base Location Subpath
We recommend using dbt's auto-generated `base_location`. However, if you need to customize the resulting `base_location`, dbt allows users to configure a `base_location_subpath`. When specified, the subpath concatenates to the end of the previously described pattern for `base_location` string generation.
From 205e5c8fceb59e2a9641dc1ff0f8f47a2e5c43c0 Mon Sep 17 00:00:00 2001
From: Amy Chen <46451573+amychen1776@users.noreply.github.com>
Date: Wed, 2 Oct 2024 09:16:46 -0400
Subject: [PATCH 36/36] Update
website/docs/reference/resource-configs/snowflake-configs.md
Co-authored-by: Anders
---
website/docs/reference/resource-configs/snowflake-configs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md
index 33f4a1cb48a..809667ef685 100644
--- a/website/docs/reference/resource-configs/snowflake-configs.md
+++ b/website/docs/reference/resource-configs/snowflake-configs.md
@@ -85,7 +85,7 @@ dbt manages `base_location` on behalf of users to enforce best practices. With S
To maintain best practices, we enforce an input. Currently, we do not support overriding the default `base location` input but will consider it based on user feedback.
-In summary, dbt-snowflake does not support arbitrary definition of base_location for Iceberg tables. Instead, dbt, by default, writes your tables within a `_dbt/{SCHEMA_NAME}` prefix to ensure easier object-store observability and auditability.
+In summary, dbt-snowflake does not support arbitrary definition of `base_location` for Iceberg tables. Instead, dbt, by default, writes your tables within a `_dbt/{SCHEMA_NAME}/{TABLE_NAME}` prefix to ensure easier object-store observability and auditability.
### Limitations