Skip to content

Commit

Permalink
Merge branch 'current' into rpourzand-dbt-partner-source
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Dec 19, 2023
2 parents cf661bc + eb5911c commit 8214b79
Show file tree
Hide file tree
Showing 9 changed files with 165 additions and 72 deletions.
4 changes: 4 additions & 0 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,10 @@ exports.versionedPages = [
{
"page": "docs/build/saved-queries",
"firstVersion": "1.7",
},
{
"page": "reference/resource-configs/on_configuration_change",
"firstVersion": "1.6",
}
]

Expand Down
47 changes: 25 additions & 22 deletions website/docs/docs/build/materializations.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,27 +109,7 @@ When using the `table` materialization, your model is rebuilt as a <Term id="tab

### Materialized View

The `materialized view` materialization allows the creation and maintenance of materialized views
in the target database. This materialization makes use of the `on_configuration_change` config, which
aligns with the incremental nature of the namesake database object. This setting tells dbt to attempt to
make configuration changes directly to the object when possible, as opposed to completely recreating
the object to implement the updated configuration. Using `dbt-postgres` as an example, indexes can
be dropped and created on the materialized view without the need to recreate the materialized view itself.

The `on_configuration_change` config has three settings:
- `apply` (default) &mdash; attempt to update the existing database object if possible, avoiding a complete rebuild
- *Note:* if any individual configuration change requires a full refresh, a full refresh be performed in lieu of individual alter statements
- `continue` &mdash; allow runs to continue while also providing a warning that the object was left untouched
- *Note:* this could result in downstream failures as those models may depend on these unimplemented changes
- `fail` &mdash; force the entire run to fail if a change is detected

Materialized views are implemented following this "drop through" life cycle:
1. If an object does not exist, create a materialized view
2. If an object exists, other than a materialized view, that object is dropped and replaced with a materialized view
3. If `--full-refresh` is supplied, replace the materialized view regardless of changes and the `on_configuration_change` setting
4. If there are no configuration changes, refresh the materialized view
5. At this point there are configuration changes, proceed according to the `on_configuration_change` setting

The `materialized view` materialization allows the creation and maintenance of materialized views in the target database.
Materialized views are a combination of a view and a table, and serve use cases similar to incremental models.

* **Pros:**
Expand All @@ -145,7 +125,30 @@ less configuration options available, see your database platform's docs for more
* **Advice:**
* Consider materialized views for use cases where incremental models are sufficient, but you would like the data platform to manage the incremental logic and refresh.

**Note:** `dbt-snowflake` _does not_ support materialized views, it uses Dynamic Tables instead. For details, refer to [Snowflake specific configurations](/reference/resource-configs/snowflake-configs#dynamic-tables).
#### Configuration Change Monitoring

This materialization makes use of the [`on_configuration_change`](/reference/resource-configs/on_configuration_change)
config, which aligns with the incremental nature of the namesake database object. This setting tells dbt to attempt to
make configuration changes directly to the object when possible, as opposed to completely recreating
the object to implement the updated configuration. Using `dbt-postgres` as an example, indexes can
be dropped and created on the materialized view without the need to recreate the materialized view itself.

#### Scheduled Refreshes

In the context of a `dbt run` command, materialized views should be thought of as similar to views.
For example, a `dbt run` command is only needed if there is the potential for a change in configuration or sql;
it's effectively a deploy action.
By contrast, a `dbt run` command is needed for a table in the same scenarios *AND when the data in the table needs to be updated*.
This also holds true for incremental and snapshot models, whose underlying relations are tables.
In the table cases, the scheduling mechanism is either dbt Cloud or your local scheduler;
there is no built-in functionality to automatically refresh the data behind a table.
However, most platforms (Postgres excluded) provide functionality to configure automatically refreshing a materialized view.
Hence, materialized views work similarly to incremental models with the benefit of not needing to run dbt to refresh the data.
This assumes, of course, that auto refresh is turned on and configured in the model.

:::info
`dbt-snowflake` _does not_ support materialized views, it uses Dynamic Tables instead. For details, refer to [Snowflake specific configurations](/reference/resource-configs/snowflake-configs#dynamic-tables).
:::

## Python materializations

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/metrics-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ This page explains the different supported metric types you can add to your dbt
### Cumulative metrics
[Cumulative metrics](/docs/build/cumulative) aggregate a measure over a given window. If no window is specified, the window would accumulate the measure over all time. **Note**m, you will need to create the [time spine model](/docs/build/metricflow-time-spine) before you add cumulative metrics.
[Cumulative metrics](/docs/build/cumulative) aggregate a measure over a given window. If no window is specified, the window would accumulate the measure over all time. **Note**, you will need to create the [time spine model](/docs/build/metricflow-time-spine) before you add cumulative metrics.
```yaml
# Cumulative metrics aggregate a measure over a given window. The window is considered infinite if no window parameter is passed (accumulate the measure over all time)
Expand Down
34 changes: 17 additions & 17 deletions website/docs/reference/resource-configs/bigquery-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -725,18 +725,18 @@ The `grant_access_to` config is not thread-safe when multiple views need to be a
The BigQuery adapter supports [materialized views](https://cloud.google.com/bigquery/docs/materialized-views-intro)
with the following configuration parameters:

| Parameter | Type | Required | Default | Change Monitoring Support |
|-------------------------------------------------------------|------------------------|----------|---------|---------------------------|
| `on_configuration_change` | `<string>` | no | `apply` | n/a |
| [`cluster_by`](#clustering-clause) | `[<string>]` | no | `none` | drop/create |
| [`partition_by`](#partition-clause) | `{<dictionary>}` | no | `none` | drop/create |
| [`enable_refresh`](#auto-refresh) | `<boolean>` | no | `true` | alter |
| [`refresh_interval_minutes`](#auto-refresh) | `<float>` | no | `30` | alter |
| [`max_staleness`](#auto-refresh) (in Preview) | `<interval>` | no | `none` | alter |
| [`description`](/reference/resource-properties/description) | `<string>` | no | `none` | alter |
| [`labels`](#specifying-labels) | `{<string>: <string>}` | no | `none` | alter |
| [`hours_to_expiration`](#controlling-table-expiration) | `<integer>` | no | `none` | alter |
| [`kms_key_name`](#using-kms-encryption) | `<string>` | no | `none` | alter |
| Parameter | Type | Required | Default | Change Monitoring Support |
|----------------------------------------------------------------------------------|------------------------|----------|---------|---------------------------|
| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `<string>` | no | `apply` | n/a |
| [`cluster_by`](#clustering-clause) | `[<string>]` | no | `none` | drop/create |
| [`partition_by`](#partition-clause) | `{<dictionary>}` | no | `none` | drop/create |
| [`enable_refresh`](#auto-refresh) | `<boolean>` | no | `true` | alter |
| [`refresh_interval_minutes`](#auto-refresh) | `<float>` | no | `30` | alter |
| [`max_staleness`](#auto-refresh) (in Preview) | `<interval>` | no | `none` | alter |
| [`description`](/reference/resource-properties/description) | `<string>` | no | `none` | alter |
| [`labels`](#specifying-labels) | `{<string>: <string>}` | no | `none` | alter |
| [`hours_to_expiration`](#controlling-table-expiration) | `<integer>` | no | `none` | alter |
| [`kms_key_name`](#using-kms-encryption) | `<string>` | no | `none` | alter |

<Tabs
groupId="config-languages"
Expand All @@ -757,7 +757,7 @@ with the following configuration parameters:
models:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): materialized_view
[+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail
[+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
[+](/reference/resource-configs/plus-prefix)[cluster_by](#clustering-clause): <field-name> | [<field-name>]
[+](/reference/resource-configs/plus-prefix)[partition_by](#partition-clause):
- field: <field-name>
Expand Down Expand Up @@ -794,7 +794,7 @@ models:
- name: [<model-name>]
config:
[materialized](/reference/resource-configs/materialized): materialized_view
on_configuration_change: apply | continue | fail
[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
[cluster_by](#clustering-clause): <field-name> | [<field-name>]
[partition_by](#partition-clause):
- field: <field-name>
Expand Down Expand Up @@ -827,7 +827,7 @@ models:
```jinja
{{ config(
[materialized](/reference/resource-configs/materialized)='materialized_view',
on_configuration_change="apply" | "continue" | "fail",
[on_configuration_change](/reference/resource-configs/on_configuration_change)="apply" | "continue" | "fail",
[cluster_by](#clustering-clause)="<field-name>" | ["<field-name>"],
[partition_by](#partition-clause)={
"field": "<field-name>",
Expand Down Expand Up @@ -868,7 +868,7 @@ models:
Many of these parameters correspond to their table counterparts and have been linked above.
The set of parameters unique to materialized views covers [auto-refresh functionality](#auto-refresh).

Find more information about these parameters in the BigQuery docs:
Learn more about these parameters in BigQuery's docs:
- [CREATE MATERIALIZED VIEW statement](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_materialized_view_statement)
- [materialized_view_option_list](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#materialized_view_option_list)

Expand All @@ -886,7 +886,7 @@ BigQuery only officially supports the configuration of the frequency (the "once
however, there is a feature in preview that allows for the configuration of the staleness (the "5 minutes" refresh).
dbt will monitor these parameters for changes and apply them using an `ALTER` statement.

Find more information about these parameters in the BigQuery docs:
Learn more about these parameters in BigQuery's docs:
- [materialized_view_option_list](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#materialized_view_option_list)
- [max_staleness](https://cloud.google.com/bigquery/docs/materialized-views-create#max_staleness)

Expand Down
86 changes: 86 additions & 0 deletions website/docs/reference/resource-configs/on_configuration_change.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
resource_types: [models]
description: "on_configuration_change - Read this in-depth guide to learn about configuration change monitoring in dbt."
datatype: "string"
---

:::info
This functionality is currently only supported for [materialized views](/docs/build/materializations#materialized-view) on a subset of adapters.
:::

The `on_configuration_change` config has three settings:
- `apply` (default) &mdash; Attempt to update the existing database object if possible, avoiding a complete rebuild.
- *Note:* If any individual configuration change requires a full refresh, a full refresh be performed in lieu of individual alter statements.
- `continue` &mdash; Allow runs to continue while also providing a warning that the object was left untouched.
- *Note:* This could result in downstream failures as those models may depend on these unimplemented changes.
- `fail` &mdash; Force the entire run to fail if a change is detected.

<Tabs
groupId="config-languages"
defaultValue="project-yaml"
values={[
{ label: 'Project file', value: 'project-yaml', },
{ label: 'Property file', value: 'property-yaml', },
{ label: 'Config block', value: 'config', },
]
}>


<TabItem value="project-yaml">

<File name='dbt_project.yml'>

```yaml
models:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): <materialization_name>
[+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail
```
</File>
</TabItem>
<TabItem value="property-yaml">
<File name='models/properties.yml'>
```yaml
version: 2

models:
- name: [<model-name>]
config:
[materialized](/reference/resource-configs/materialized): <materialization_name>
on_configuration_change: apply | continue | fail
```
</File>
</TabItem>
<TabItem value="config">
<File name='models/<model_name>.sql'>
```jinja
{{ config(
[materialized](/reference/resource-configs/materialized)="<materialization_name>",
on_configuration_change="apply" | "continue" | "fail"
) }}
```

</File>

</TabItem>

</Tabs>

Materializations are implemented following this "drop through" life cycle:
1. If a model does not exist with the provided path, create the new model.
2. If a model exists, but has a different type, drop the existing model and create the new model.
3. If `--full-refresh` is supplied, replace the existing model regardless of configuration changes and the `on_configuration_change` setting.
4. If there are no configuration changes, perform the default action for that type (e.g. apply refresh for a materialized view).
5. Determine whether to apply the configuration changes according to the `on_configuration_change` setting.
17 changes: 8 additions & 9 deletions website/docs/reference/resource-configs/postgres-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,10 @@ models:
The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html)
with the following configuration parameters:

| Parameter | Type | Required | Default | Change Monitoring Support |
|---------------------------|--------------------|----------|---------|---------------------------|
| `on_configuration_change` | `<string>` | no | `apply` | n/a |
| [`indexes`](#indexes) | `[{<dictionary>}]` | no | `none` | alter |
| Parameter | Type | Required | Default | Change Monitoring Support |
|----------------------------------------------------------------------------------|--------------------|----------|---------|---------------------------|
| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `<string>` | no | `apply` | n/a |
| [`indexes`](#indexes) | `[{<dictionary>}]` | no | `none` | alter |

<Tabs
groupId="config-languages"
Expand All @@ -135,7 +135,7 @@ with the following configuration parameters:
models:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): materialized_view
[+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail
[+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
[+](/reference/resource-configs/plus-prefix)[indexes](#indexes):
- columns: [<column-name>]
unique: true | false
Expand All @@ -158,7 +158,7 @@ models:
- name: [<model-name>]
config:
[materialized](/reference/resource-configs/materialized): materialized_view
on_configuration_change: apply | continue | fail
[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail
[indexes](#indexes):
- columns: [<column-name>]
unique: true | false
Expand All @@ -177,7 +177,7 @@ models:
```jinja
{{ config(
[materialized](/reference/resource-configs/materialized)="materialized_view",
on_configuration_change="apply" | "continue" | "fail",
[on_configuration_change](/reference/resource-configs/on_configuration_change)="apply" | "continue" | "fail",
[indexes](#indexes)=[
{
"columns": ["<column-name>"],
Expand All @@ -198,8 +198,7 @@ The [`indexes`](#indexes) parameter corresponds to that of a table, as explained
It's worth noting that, unlike tables, dbt monitors this parameter for changes and applies the changes without dropping the materialized view.
This happens via a `DROP/CREATE` of the indexes, which can be thought of as an `ALTER` of the materialized view.

Find more information about materialized view parameters in the Postgres docs:
- [CREATE MATERIALIZED VIEW](https://www.postgresql.org/docs/current/sql-creatematerializedview.html)
Learn more about these parameters in Postgres's [docs](https://www.postgresql.org/docs/current/sql-creatematerializedview.html).

<VersionBlock firstVersion="1.6" lastVersion="1.6">

Expand Down
Loading

0 comments on commit 8214b79

Please sign in to comment.