From 406b40851a34268a194b2ae0e15ca5c180c09e92 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Tue, 5 Dec 2023 13:10:15 -0500 Subject: [PATCH 01/13] add materialized view configuration for postgres --- .../resource-configs/postgres-configs.md | 46 ++++++++++++++----- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index 97a695ee12e..f9bd4e11cbc 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -108,21 +108,43 @@ models: ## Materialized views -The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html). -Indexes are the only configuration that is specific to `dbt-postgres`. -The remaining configuration follows the general [materialized view](/docs/build/materializations#materialized-view) configuration. -There are also some limitations that we hope to address in the next version. +The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) +with the following configuration parameters: -### Monitored configuration changes +| Parameter | Type | Required | Default | Change Monitoring Support | Reference | +|---------------------------|------------|----------|-----------|---------------------------|---------------------| +| `on_configuration_change` | STRING | NO | `'apply'` | N/A | | +| `indexes` | LIST[DICT] | NO | | ALTER | [Indexes](#indexes) | -The settings below are monitored for changes applicable to `on_configuration_change`. +#### Sample model file: -#### Indexes + -Index changes (`CREATE`, `DROP`) can be applied without the need to rebuild the materialized view. -This differs from a table model, where the table needs to be dropped and re-created to update the indexes. -If the `indexes` portion of the `config` block is updated, the changes will be detected and applied -directly to the materialized view in place. +```sql +{{ config( + materialized='materialized_view', + on_configuration_change='{ apply | continue | fail }' + indexes=[ + { + 'columns': ['', ...], + 'unique': }, + 'type': '{ HASH | B-TREE | GIST | SP-GIST | GIN | BRIN }', + ] +) }} + +select * from {{ ref('my_base_table') }} +``` + + + +The `indexes` parameter corresponds to that of a table, as linked above. +It's worth noting that, unlike with tables, dbt will monitor this parameter for changes and apply the changes without dropping the materialized view. +This happens via a `DROP/CREATE` of the indexes, which could be thought of as a `ALTER` of the materialized view. + +Find more information about materialized view parameters in the Postgres docs: +- [CREATE MATERIALIZED VIEW](https://www.postgresql.org/docs/current/sql-creatematerializedview.html) + + ### Limitations @@ -138,3 +160,5 @@ If the user changes the model's config to `materialized="materialized_view"`, th The solution is to execute `DROP TABLE my_model` on the data warehouse before trying the model again. + + From 54b29418138dfae2e7c153a83a3b1e8b9d1ae26c Mon Sep 17 00:00:00 2001 From: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Date: Wed, 6 Dec 2023 17:36:24 -0500 Subject: [PATCH 02/13] Update website/docs/reference/resource-configs/postgres-configs.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/resource-configs/postgres-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index f9bd4e11cbc..855c17d8883 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -127,7 +127,7 @@ with the following configuration parameters: indexes=[ { 'columns': ['', ...], - 'unique': }, + 'unique': , 'type': '{ HASH | B-TREE | GIST | SP-GIST | GIN | BRIN }', ] ) }} From 964e12563623525ccffc3a52ab6b2b2795f957f2 Mon Sep 17 00:00:00 2001 From: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Date: Wed, 6 Dec 2023 17:36:39 -0500 Subject: [PATCH 03/13] Update website/docs/reference/resource-configs/postgres-configs.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/resource-configs/postgres-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index 855c17d8883..109bd595bb9 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -129,7 +129,7 @@ with the following configuration parameters: 'columns': ['', ...], 'unique': , 'type': '{ HASH | B-TREE | GIST | SP-GIST | GIN | BRIN }', - ] + }] ) }} select * from {{ ref('my_base_table') }} From 0d4dd63ad767fcb480ca39f9ee8d2999e920a1d1 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 6 Dec 2023 21:40:59 -0500 Subject: [PATCH 04/13] incorporate style standards --- .../resource-configs/postgres-configs.md | 86 +++++++++++++++---- 1 file changed, 70 insertions(+), 16 deletions(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index 109bd595bb9..98ef0d56d27 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -111,32 +111,86 @@ models: The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) with the following configuration parameters: -| Parameter | Type | Required | Default | Change Monitoring Support | Reference | -|---------------------------|------------|----------|-----------|---------------------------|---------------------| -| `on_configuration_change` | STRING | NO | `'apply'` | N/A | | -| `indexes` | LIST[DICT] | NO | | ALTER | [Indexes](#indexes) | +| Parameter | Type | Required | Default | Change Monitoring Support | +|---------------------------|------------------|----------|---------|---------------------------| +| `on_configuration_change` | | no | `apply` | n/a | +| [`indexes`](#indexes) | [{}] | no | `none` | alter | + + + + -#### Sample model file: + - +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): materialized_view + [+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail + [+](/reference/resource-configs/plus-prefix)[indexes](#indexes): + - columns: [] + unique: true | false + type: hash | btree +``` -```sql + + + + + + + + +```yaml +version: 2 + +models: + - name: [] + config: + [materialized](/reference/resource-configs/materialized): materialized_view + on_configuration_change: apply | continue | fail + [indexes](#indexes): + - columns: [] + unique: true | false + type: hash | btree +``` + + + + + + + + + +```jinja {{ config( - materialized='materialized_view', - on_configuration_change='{ apply | continue | fail }' - indexes=[ + [materialized](/reference/resource-configs/materialized)="materialized_view", + on_configuration_change="apply" | "continue" | "fail" + [indexes](#indexes)=[ { - 'columns': ['', ...], - 'unique': , - 'type': '{ HASH | B-TREE | GIST | SP-GIST | GIN | BRIN }', - }] + "columns": [""], + "unique": true | false, + "type": "hash" | "btree", + } + ] ) }} - -select * from {{ ref('my_base_table') }} ``` + + + + The `indexes` parameter corresponds to that of a table, as linked above. It's worth noting that, unlike with tables, dbt will monitor this parameter for changes and apply the changes without dropping the materialized view. This happens via a `DROP/CREATE` of the indexes, which could be thought of as a `ALTER` of the materialized view. From bd6c4fc146678f0aeae9a9e15f15b2591cdd4bf8 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 6 Dec 2023 22:08:42 -0500 Subject: [PATCH 05/13] incorporate style standards --- website/docs/reference/resource-configs/postgres-configs.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index 98ef0d56d27..d5960231c66 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -145,6 +145,7 @@ models: + @@ -167,6 +168,7 @@ models: + @@ -174,7 +176,7 @@ models: ```jinja {{ config( [materialized](/reference/resource-configs/materialized)="materialized_view", - on_configuration_change="apply" | "continue" | "fail" + on_configuration_change="apply" | "continue" | "fail", [indexes](#indexes)=[ { "columns": [""], From 5e904f13d54f8228f2c89848f8ca8a0440b5f026 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 6 Dec 2023 22:34:56 -0500 Subject: [PATCH 06/13] remove tabs to test build --- .../resource-configs/postgres-configs.md | 77 ------------------- 1 file changed, 77 deletions(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index d5960231c66..c4d60ba905d 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -116,83 +116,6 @@ with the following configuration parameters: | `on_configuration_change` | | no | `apply` | n/a | | [`indexes`](#indexes) | [{}] | no | `none` | alter | - - - - - - -```yaml -models: - [](/reference/resource-configs/resource-path): - [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): materialized_view - [+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail - [+](/reference/resource-configs/plus-prefix)[indexes](#indexes): - - columns: [] - unique: true | false - type: hash | btree -``` - - - - - - - - - - -```yaml -version: 2 - -models: - - name: [] - config: - [materialized](/reference/resource-configs/materialized): materialized_view - on_configuration_change: apply | continue | fail - [indexes](#indexes): - - columns: [] - unique: true | false - type: hash | btree -``` - - - - - - - - - - -```jinja -{{ config( - [materialized](/reference/resource-configs/materialized)="materialized_view", - on_configuration_change="apply" | "continue" | "fail", - [indexes](#indexes)=[ - { - "columns": [""], - "unique": true | false, - "type": "hash" | "btree", - } - ] -) }} -``` - - - - - - - The `indexes` parameter corresponds to that of a table, as linked above. It's worth noting that, unlike with tables, dbt will monitor this parameter for changes and apply the changes without dropping the materialized view. This happens via a `DROP/CREATE` of the indexes, which could be thought of as a `ALTER` of the materialized view. From 76df337ea6acb03da8fcd1ab313deb6ed0e4e4c3 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 6 Dec 2023 22:43:16 -0500 Subject: [PATCH 07/13] remove quotes to test build --- website/docs/reference/resource-configs/postgres-configs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index c4d60ba905d..de084f9eb02 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -111,10 +111,10 @@ models: The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) with the following configuration parameters: -| Parameter | Type | Required | Default | Change Monitoring Support | -|---------------------------|------------------|----------|---------|---------------------------| +| Parameter | Type | Required | Default | Change Monitoring Support | +|-------------------------|------------------|----------|---------|---------------------------| | `on_configuration_change` | | no | `apply` | n/a | -| [`indexes`](#indexes) | [{}] | no | `none` | alter | +| [indexes](#indexes) | [{}] | no | `none` | alter | The `indexes` parameter corresponds to that of a table, as linked above. It's worth noting that, unlike with tables, dbt will monitor this parameter for changes and apply the changes without dropping the materialized view. From b744ea85e0b24fcbad5a73d4c2fc4d8deba1e206 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 6 Dec 2023 22:53:13 -0500 Subject: [PATCH 08/13] test build --- .../reference/resource-configs/postgres-configs.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index de084f9eb02..e230641c37a 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -111,10 +111,10 @@ models: The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) with the following configuration parameters: -| Parameter | Type | Required | Default | Change Monitoring Support | -|-------------------------|------------------|----------|---------|---------------------------| -| `on_configuration_change` | | no | `apply` | n/a | -| [indexes](#indexes) | [{}] | no | `none` | alter | +| Parameter | Type | Required | Default | Change Monitoring Support | +|---------------------------|--------------|----------|---------|---------------------------| +| `on_configuration_change` | | no | `apply` | n/a | +| [indexes](#indexes) | | no | `none` | alter | The `indexes` parameter corresponds to that of a table, as linked above. It's worth noting that, unlike with tables, dbt will monitor this parameter for changes and apply the changes without dropping the materialized view. @@ -123,7 +123,7 @@ This happens via a `DROP/CREATE` of the indexes, which could be thought of as a Find more information about materialized view parameters in the Postgres docs: - [CREATE MATERIALIZED VIEW](https://www.postgresql.org/docs/current/sql-creatematerializedview.html) - + ### Limitations From fb00255b8588c8df6a324f00780adc85858978b1 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 6 Dec 2023 22:59:24 -0500 Subject: [PATCH 09/13] test build --- .../docs/reference/resource-configs/postgres-configs.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index e230641c37a..d4e53efbc1d 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -111,10 +111,10 @@ models: The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) with the following configuration parameters: -| Parameter | Type | Required | Default | Change Monitoring Support | -|---------------------------|--------------|----------|---------|---------------------------| -| `on_configuration_change` | | no | `apply` | n/a | -| [indexes](#indexes) | | no | `none` | alter | +| Parameter | Type | Required | Default | Change Monitoring Support | +|---------------------------|------------|----------|---------|---------------------------| +| `on_configuration_change` | string | no | `apply` | n/a | +| [indexes](#indexes) | dictionary | no | `none` | alter | The `indexes` parameter corresponds to that of a table, as linked above. It's worth noting that, unlike with tables, dbt will monitor this parameter for changes and apply the changes without dropping the materialized view. From d19ea838bfd9d1538e7c1e264886e0620feebada Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 6 Dec 2023 23:08:08 -0500 Subject: [PATCH 10/13] add tabs back in --- .../resource-configs/postgres-configs.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index d4e53efbc1d..d47aaad7559 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -116,6 +116,84 @@ with the following configuration parameters: | `on_configuration_change` | string | no | `apply` | n/a | | [indexes](#indexes) | dictionary | no | `none` | alter | + + + + + + + +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): materialized_view + [+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail + [+](/reference/resource-configs/plus-prefix)[indexes](#indexes): + - columns: [] + unique: true | false + type: hash | btree +``` + + + + + + + + + + +```yaml +version: 2 + +models: + - name: [] + config: + [materialized](/reference/resource-configs/materialized): materialized_view + on_configuration_change: apply | continue | fail + [indexes](#indexes): + - columns: [] + unique: true | false + type: hash | btree +``` + + + + + + + + + + +```jinja +{{ config( + [materialized](/reference/resource-configs/materialized)="materialized_view", + on_configuration_change="apply" | "continue" | "fail", + [indexes](#indexes)=[ + { + "columns": [""], + "unique": true | false, + "type": "hash" | "btree", + } + ] +) }} +``` + + + + + + + The `indexes` parameter corresponds to that of a table, as linked above. It's worth noting that, unlike with tables, dbt will monitor this parameter for changes and apply the changes without dropping the materialized view. This happens via a `DROP/CREATE` of the indexes, which could be thought of as a `ALTER` of the materialized view. From 139046b04d6d55fae6336ede64dbdf8afa7a7556 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 6 Dec 2023 23:33:23 -0500 Subject: [PATCH 11/13] add formatting back --- .../docs/reference/resource-configs/postgres-configs.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index d47aaad7559..c904802d598 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -111,10 +111,10 @@ models: The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) with the following configuration parameters: -| Parameter | Type | Required | Default | Change Monitoring Support | -|---------------------------|------------|----------|---------|---------------------------| -| `on_configuration_change` | string | no | `apply` | n/a | -| [indexes](#indexes) | dictionary | no | `none` | alter | +| Parameter | Type | Required | Default | Change Monitoring Support | +|---------------------------|--------------------|----------|---------|---------------------------| +| `on_configuration_change` | `` | no | `apply` | n/a | +| [`indexes`](#indexes) | `[{}]` | no | `none` | alter | Date: Wed, 13 Dec 2023 14:14:16 -0500 Subject: [PATCH 12/13] Apply suggestions from code review --- website/docs/reference/resource-configs/postgres-configs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index c904802d598..c01ea9ea718 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -194,9 +194,9 @@ models: -The `indexes` parameter corresponds to that of a table, as linked above. -It's worth noting that, unlike with tables, dbt will monitor this parameter for changes and apply the changes without dropping the materialized view. -This happens via a `DROP/CREATE` of the indexes, which could be thought of as a `ALTER` of the materialized view. +The [`indexes`](#indexes) parameter corresponds to that of a table, as explained above. +It's worth noting that, unlike with tables, dbt monitors this parameter for changes and applies the changes without dropping the materialized view. +This happens via a `DROP/CREATE` of the indexes, which can be thought of as an `ALTER` of the materialized view. Find more information about materialized view parameters in the Postgres docs: - [CREATE MATERIALIZED VIEW](https://www.postgresql.org/docs/current/sql-creatematerializedview.html) From e4b780cefa4fd7852ca593488faed5c150beb7d3 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 13 Dec 2023 14:14:42 -0500 Subject: [PATCH 13/13] Update website/docs/reference/resource-configs/postgres-configs.md --- website/docs/reference/resource-configs/postgres-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index c01ea9ea718..8465a5cbb31 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -195,7 +195,7 @@ models: The [`indexes`](#indexes) parameter corresponds to that of a table, as explained above. -It's worth noting that, unlike with tables, dbt monitors this parameter for changes and applies the changes without dropping the materialized view. +It's worth noting that, unlike tables, dbt monitors this parameter for changes and applies the changes without dropping the materialized view. This happens via a `DROP/CREATE` of the indexes, which can be thought of as an `ALTER` of the materialized view. Find more information about materialized view parameters in the Postgres docs: