From 50998fe803b6c48f46dab4a8cc930d8c33a41b01 Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Fri, 30 Aug 2024 11:41:12 +0100 Subject: [PATCH 01/63] Updated constraints doc following slack --- .../resource-properties/constraints.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b8111ef0adb..74078d7541f 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -303,6 +303,67 @@ select +### Custom constraints on models for advanced configuration of tables + +In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + +Masking policies conceal sensitive data from unauthorized access, while still allowing authorized users to retrieve the data during query execution. + +Contracts and constraints support tag-based masking policies using the following syntax: + + + +```shell + +models: + +- name: my_model +config: +contract: {enforced: true} +materialized: table +columns: + - name: id + data_type: int + constraints: + - type: custom + expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. + +``` + + + +Using this syntax requires defining all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. + +To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). + +Alternatively, you can add a masking policy without tags: + + + +```shell + +models: + - name: my_model + config: + contract: {enforced: true} + materialized: table + columns: + - name: id + data_type: int + constraints: + - type: custom + expression: "masking policy my_policy" + +``` + + + +:::info + +Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example for [DBX](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) for [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). + +::: +
From dc70f56591115867f24672ef262d98e30e93558a Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 13:14:12 +0100 Subject: [PATCH 02/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 74078d7541f..7d1e5d0a9e5 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -307,7 +307,6 @@ select In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). -Masking policies conceal sensitive data from unauthorized access, while still allowing authorized users to retrieve the data during query execution. Contracts and constraints support tag-based masking policies using the following syntax: From 1732a00296018a2c42410ed049f306fe0b906722 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 13:14:28 +0100 Subject: [PATCH 03/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 7d1e5d0a9e5..f3388febc43 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -312,7 +312,7 @@ Contracts and constraints support tag-based masking policies using the following -```shell +```yaml models: From 9368790e51b9f8fde212a574e3298887b2da9954 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 14:40:01 +0100 Subject: [PATCH 04/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index f3388febc43..8129535eb72 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -331,7 +331,7 @@ columns: -Using this syntax requires defining all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. +Using this syntax requires configuring all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). From 3ac1d4d2822ac63474dc131426d60dc331983a0c Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 14:42:57 +0100 Subject: [PATCH 05/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 8129535eb72..956d03dfa72 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -357,7 +357,6 @@ models: -:::info Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example for [DBX](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) for [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). From 70460bbfd1a712c0387a236f1d7fe67a3a3a07c2 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 14:43:40 +0100 Subject: [PATCH 06/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 956d03dfa72..b181340f337 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -358,7 +358,6 @@ models: -Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example for [DBX](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) for [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). ::: From 3ea40bd01a7cded1f7df68734c020f98ab197685 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 14:47:46 +0100 Subject: [PATCH 07/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b181340f337..f4694c1875b 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -305,7 +305,9 @@ select ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). +In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + +Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". Contracts and constraints support tag-based masking policies using the following syntax: From fa43fff6590f3ac861dcdeafc1e3973980865c49 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:07:58 +0100 Subject: [PATCH 08/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index f4694c1875b..5af9c90528f 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -320,7 +320,7 @@ models: - name: my_model config: -contract: {enforced: true} + contract: {enforced: true} materialized: table columns: - name: id From 4a176427ed7df522a7c9cb751231f74cc350705a Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:08:25 +0100 Subject: [PATCH 09/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 5af9c90528f..f6098f08a2c 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -321,7 +321,7 @@ models: - name: my_model config: contract: {enforced: true} -materialized: table + materialized: table columns: - name: id data_type: int From 0d834a73ef0085ee42e681a68b3e1e6e61a5d89b Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:10:28 +0100 Subject: [PATCH 10/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index f6098f08a2c..5359327008d 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -322,7 +322,7 @@ models: config: contract: {enforced: true} materialized: table -columns: + columns: - name: id data_type: int constraints: From a8e26c4bab36051dfb4779a98d704779a2b5616d Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:12:23 +0100 Subject: [PATCH 11/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 5359327008d..082eae9f18d 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -324,7 +324,7 @@ config: materialized: table columns: - name: id - data_type: int + data_type: int constraints: - type: custom expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. From e6fe96b5b809552c32328a6cb5cca75bfdf0c6aa Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:14:11 +0100 Subject: [PATCH 12/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 082eae9f18d..1e56862fcec 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -325,7 +325,7 @@ config: columns: - name: id data_type: int - constraints: + constraints: - type: custom expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. From cdf0aded955f87f324091608a10d0eda4ff559d4 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:14:37 +0100 Subject: [PATCH 13/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 1e56862fcec..7d8895ad7d6 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -327,7 +327,7 @@ config: data_type: int constraints: - type: custom - expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. + expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. ``` From 9da9e5d34429a040ab68ec0ec7758199717eabe3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:22:41 +0100 Subject: [PATCH 14/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 7d8895ad7d6..6f06d848fd8 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -310,7 +310,7 @@ In dbt Cloud, you can use custom constraints on models for advanced configuratio Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". -Contracts and constraints support tag-based masking policies using the following syntax: +Here are a few examples of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 85abbd2e92bac540dbeb6ad94ce384c9c9ced458 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:29:48 +0100 Subject: [PATCH 15/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 6f06d848fd8..aaba6a3c851 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -341,7 +341,7 @@ Alternatively, you can add a masking policy without tags: -```shell +```yaml models: - name: my_model From 03fc2a2f704a4617a0044006a1e34c3fca72cc78 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:30:11 +0100 Subject: [PATCH 16/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index aaba6a3c851..6fe9b199c08 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -361,7 +361,6 @@ models: -:::
From 8851b141655181f4e17011cdd26a022718f4a26a Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Mon, 2 Sep 2024 16:29:05 +0100 Subject: [PATCH 17/63] Updated doc, moved section to new location --- .../resource-properties/constraints.md | 116 +++++++++--------- 1 file changed, 57 insertions(+), 59 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 6fe9b199c08..d65398ae005 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -303,65 +303,6 @@ select -### Custom constraints on models for advanced configuration of tables - -In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - -Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". - - -Here are a few examples of how to implement tag-based masking policies with contracts and constraints using the following syntax: - - - -```yaml - -models: - -- name: my_model -config: - contract: {enforced: true} - materialized: table - columns: - - name: id - data_type: int - constraints: - - type: custom - expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. - -``` - - - -Using this syntax requires configuring all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. - -To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). - -Alternatively, you can add a masking policy without tags: - - - -```yaml - -models: - - name: my_model - config: - contract: {enforced: true} - materialized: table - columns: - - name: id - data_type: int - constraints: - - type: custom - expression: "masking policy my_policy" - -``` - - - - - -
@@ -577,3 +518,60 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0
+ +### Custom constraints on models for advanced configuration of tables + +In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + +Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". + +Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: + + + +```shell + +models: + +- name: my_model +config: + contract: {enforced: true} + materialized: table + columns: + - name: id + data_type: int + constraints: + - type: custom + expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. + +``` + + + +Using this syntax requires configuring all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. + +To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). + +Alternatively, you can add a masking policy without tags: + + + +```shell + +models: + - name: my_model + config: + contract: {enforced: true} + materialized: table + columns: + - name: id + data_type: int + constraints: + - type: custom + expression: "masking policy my_policy" + +``` + + + + From 7fc710aa35dd444ca18ebddca666487d1f769d6a Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Mon, 2 Sep 2024 16:56:43 +0100 Subject: [PATCH 18/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index d65398ae005..7098a55e311 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -523,7 +523,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). -Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". +Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 8f7cd16960c42f05dc1e054ea04b621d8aba18c3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Mon, 2 Sep 2024 16:57:15 +0100 Subject: [PATCH 19/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 7098a55e311..301dd2f7ed0 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -529,7 +529,7 @@ Here's an example of how to implement tag-based masking policies with contracts -```shell +```yaml models: From 43d6cf5de4dee15f8c04e387df3ac9a73927e707 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Mon, 2 Sep 2024 16:57:32 +0100 Subject: [PATCH 20/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 301dd2f7ed0..673af5277c4 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -556,7 +556,7 @@ Alternatively, you can add a masking policy without tags: -```shell +```yaml models: - name: my_model From 8295508329a32349ec8c8f6cb11c6d03f297c433 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:18:21 +0100 Subject: [PATCH 21/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 673af5277c4..09f5f6076e9 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,7 @@ Each data warehouse has its own set of parameters that can be set for columns in Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: - + ```yaml From f8cce3344bbc924b9b646c0e89583f4f77e7b1b7 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:18:54 +0100 Subject: [PATCH 22/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 09f5f6076e9..079dd88106e 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -554,7 +554,7 @@ To generate a YAML with all the columns, you could also use `generate_model_yaml Alternatively, you can add a masking policy without tags: - + ```yaml From 04bf56d9daf18e3a0d3ea6f7abe58e014ed7e93c Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:34:33 +0100 Subject: [PATCH 23/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 079dd88106e..b31923c2693 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -533,7 +533,7 @@ Here's an example of how to implement tag-based masking policies with contracts models: -- name: my_model + - name: my_model config: contract: {enforced: true} materialized: table From 2455fb5804538906d8d09a20dcf8ed1bd687ce70 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:35:26 +0100 Subject: [PATCH 24/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b31923c2693..838e502d714 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -534,7 +534,7 @@ Here's an example of how to implement tag-based masking policies with contracts models: - name: my_model -config: + config: contract: {enforced: true} materialized: table columns: From 70ad2605a99021fafb8bd5649ce1bbf913e1ca25 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:41:58 +0100 Subject: [PATCH 25/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 838e502d714..d7c5c53dd61 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -537,7 +537,7 @@ models: config: contract: {enforced: true} materialized: table - columns: + columns: - name: id data_type: int constraints: From 72420bdf2a90f7aafd1638e9cafe4db1b2a91be5 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:43:31 +0100 Subject: [PATCH 26/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index d7c5c53dd61..a502a0b87d4 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -533,7 +533,7 @@ Here's an example of how to implement tag-based masking policies with contracts models: - - name: my_model + - name: my_model config: contract: {enforced: true} materialized: table From 7d2287d7e2443896cf362a08a6234744b67f2c4d Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:44:08 +0100 Subject: [PATCH 27/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index a502a0b87d4..a26a2ebb917 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -537,7 +537,7 @@ models: config: contract: {enforced: true} materialized: table - columns: + columns: - name: id data_type: int constraints: From e10b8269d95f6443b1f0b821feafc07338fe01b9 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:51:47 +0100 Subject: [PATCH 28/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index a26a2ebb917..b044b4ba670 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -535,7 +535,7 @@ models: - name: my_model config: - contract: {enforced: true} + contract: {enforced: true} materialized: table columns: - name: id From 244e2f407c1e5ef57ea95674114a85106c84945a Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:52:02 +0100 Subject: [PATCH 29/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b044b4ba670..d038465a687 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -536,7 +536,7 @@ models: - name: my_model config: contract: {enforced: true} - materialized: table + materialized: table columns: - name: id data_type: int From 8258226b8a62e226e159181558df39cd6fbceb62 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:53:15 +0100 Subject: [PATCH 30/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index d038465a687..555766e13d7 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -537,7 +537,7 @@ models: config: contract: {enforced: true} materialized: table - columns: + columns: - name: id data_type: int constraints: From 2cf8771dfd8afd71bc59d7fc54ea9fde3cab3133 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 10:35:17 +0100 Subject: [PATCH 31/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 555766e13d7..891cff8c9e8 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,7 +521,9 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). +In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Different data warehouses will support different syntax and different capabilities, but custom constraints will always allow you to add configuration to specific columns. + +For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). From 15f76640be92a19273610878ca94cb3f8302d308 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 12:00:36 +0100 Subject: [PATCH 32/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 891cff8c9e8..2457972ad86 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,6 +527,11 @@ For example, custom constraints allow you to set [masking policies](https://docs Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). +There are some different scenarios of implementing contracts and constraints: + +- [Tag-based masking policies](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2310772737). +- [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). + Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 05afe4ece7b40f71b58c63f9416f08af3a034568 Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Wed, 4 Sep 2024 13:46:37 +0100 Subject: [PATCH 33/63] updated doc --- .../resource-properties/constraints.md | 35 ++++++++++--------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 2457972ad86..8f2746c497c 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,13 +521,13 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Different data warehouses will support different syntax and different capabilities, but custom constraints will always allow you to add configuration to specific columns. +In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses will support different syntax and capabilities, but custom constraints will always allow you to add configuration to specific columns. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). -Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). +[Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are also other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. -There are some different scenarios of implementing contracts and constraints: +There are different scenarios of implementing contracts and constraints: - [Tag-based masking policies](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2310772737). - [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). @@ -539,25 +539,25 @@ Here's an example of how to implement tag-based masking policies with contracts ```yaml models: - - name: my_model - config: - contract: {enforced: true} - materialized: table - columns: - - name: id - data_type: int - constraints: - - type: custom - expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. +config: null +contract: + enforced: true +materialized: table +columns: + - name: id + data_type: int + constraints: + - type: custom + expression: tag (my_tag = 'my_value') # A custom SQL expression used to enforce a specific constraint on a column. ``` -Using this syntax requires configuring all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. +Using this syntax requires configuring all the columns and their types as it’s the only way to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. -To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). +To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). Alternatively, you can add a masking policy without tags: @@ -568,14 +568,15 @@ Alternatively, you can add a masking policy without tags: models: - name: my_model config: - contract: {enforced: true} + contract: + enforced: true materialized: table columns: - name: id data_type: int constraints: - type: custom - expression: "masking policy my_policy" + expression: masking policy my_policy ``` From d9290f0da08641875d7534e6a4cd2db24381baf3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:17:35 +0100 Subject: [PATCH 34/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 8f2746c497c..fc4b52103eb 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,7 +521,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses will support different syntax and capabilities, but custom constraints will always allow you to add configuration to specific columns. +In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). From 3fea39018e7741ae6af661becd2b12a828ff3254 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:17:48 +0100 Subject: [PATCH 35/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index fc4b52103eb..2aeaeb37e3c 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,7 @@ For example, custom constraints allow you to set [masking policies](https://docs [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are also other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. -There are different scenarios of implementing contracts and constraints: +You can implement constraints in a couple of different ways: - [Tag-based masking policies](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2310772737). - [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). From 278f3d5da3a6a993f81fd39899511654da0d0465 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:18:45 +0100 Subject: [PATCH 36/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 2aeaeb37e3c..b8b857405ce 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -523,7 +523,9 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. -For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + + - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) ar other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are also other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. From f3bced0a19ac7908d8dab997ad9f932c03cd5219 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:19:29 +0100 Subject: [PATCH 37/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b8b857405ce..abde4c13f94 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,6 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) ar other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. -[Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are also other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways: From 88f16f07773fbbdb38ff0e852f3433b4d016c8fc Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:23:25 +0100 Subject: [PATCH 38/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index abde4c13f94..1b3ba2db7a3 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -577,7 +577,7 @@ models: data_type: int constraints: - type: custom - expression: masking policy my_policy + expression: "masking policy my_policy" ``` From bb6c5ea042d0edc74c2fb9e84207870de6c659cd Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:32:43 +0100 Subject: [PATCH 39/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 1b3ba2db7a3..ab5a83aa64a 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -530,7 +530,7 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: -- [Tag-based masking policies](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2310772737). +- ### Custom constraints with tags - [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From fb6e44d5cb12350a29a6e21230d8ffe9f6202156 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:33:23 +0100 Subject: [PATCH 40/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index ab5a83aa64a..702da092b73 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -531,7 +531,7 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: - ### Custom constraints with tags -- [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). +- ### Custom constraints without tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 9cfc253db79a8a40c35b30ce266c74205a4e2f35 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:54:08 +0100 Subject: [PATCH 41/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 702da092b73..2d4ac244715 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -531,7 +531,6 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: - ### Custom constraints with tags -- ### Custom constraints without tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 1a7b84e618769cf783c41a76ed3ff91b0276ac7c Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:55:13 +0100 Subject: [PATCH 42/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 2d4ac244715..552ffbd918d 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -559,6 +559,9 @@ Using this syntax requires configuring all the columns and their types as it’s To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). + +### Custom constraints without tags + Alternatively, you can add a masking policy without tags: From 84ec68c10a816d9320fd9f6c8a0cf0b7ff844680 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:55:36 +0100 Subject: [PATCH 43/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 552ffbd918d..58fa5b999a6 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -530,7 +530,7 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: -- ### Custom constraints with tags +### Custom constraints with tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From b4285e79dade96dd4fb6aa307494f17bfa0e129c Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Wed, 4 Sep 2024 16:24:12 +0100 Subject: [PATCH 44/63] update yaml --- .../resource-properties/constraints.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 8f2746c497c..9170f23dbd2 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -540,16 +540,16 @@ Here's an example of how to implement tag-based masking policies with contracts models: - name: my_model -config: null -contract: - enforced: true -materialized: table -columns: - - name: id - data_type: int - constraints: - - type: custom - expression: tag (my_tag = 'my_value') # A custom SQL expression used to enforce a specific constraint on a column. + config: null + contract: + enforced: true + materialized: table + columns: + - name: id + data_type: int + constraints: + - type: custom + expression: tag (my_tag = 'my_value') # A custom SQL expression used to enforce a specific constraint on a column. ``` From 5217d03149f33f96a39a697a163559f668e28bb9 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 16:41:08 +0100 Subject: [PATCH 45/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 0d4c71c7b69..bcae091afde 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -525,7 +525,7 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) ar other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. + - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways: From 3885b93c00c3aaec9dfadb9314fd0319dd0ac3e8 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 17:37:19 +0100 Subject: [PATCH 46/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index bcae091afde..f047ecadfff 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -530,6 +530,8 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: +- [Custom constraints with tags](#custom-constraints-with-tags) +- [Custom constraints without tags](#custom-constraints-without-tags) ### Custom constraints with tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From ecff6de1c6800e423173e0ff58021ba79a203be5 Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Tue, 10 Sep 2024 17:00:11 +0100 Subject: [PATCH 47/63] Updated project splitting section --- .../how-we-structure/5-the-rest-of-the-project.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index c7522bf12eb..07489ab67cd 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -102,9 +102,9 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec ### Project splitting -One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Our present stance on this for most projects, particularly for teams starting out, is straightforward: you should avoid it unless you have no other option or it saves you from an even more complex workaround. If you do have the need to split up your project, it’s completely possible through the use of private packages, but the added complexity and separation is, for most organizations, a hindrance, not a help, at present. That said, this is very likely subject to change! [We want to create a world where it’s easy to bring lots of dbt projects together into a cohesive lineage](https://github.com/dbt-labs/dbt-core/discussions/5244). In a world where it’s simple to break up monolithic dbt projects into multiple connected projects, perhaps inside of a modern mono repo, the calculus will be different, and the below situations we recommend against may become totally viable. So watch this space! +One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting out, is fairly simple: in most cases, we recommend doing so! With the introduction of [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro), organizations can more effectively handle complexity by utilizing multiple interconnected dbt projects rather than a single large, monolithic one. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! -- ❌ **Business groups or departments.** Conceptual separations within the project are not a good reason to split up your project. Splitting up, for instance, marketing and finance modeling into separate projects will not only add unnecessary complexity but destroy the unifying effect of collaborating across your organization on cohesive definitions and business logic. +- ✅ **Business groups or departments.** Conceptual separations within the project are a good reason to split up your project. For more information about dbt mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. From 55f9a4c757f5fbb1251a9afc6443be0b02b312f7 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 10 Sep 2024 21:05:42 +0100 Subject: [PATCH 48/63] Update website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com> --- .../how-we-structure/5-the-rest-of-the-project.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index 07489ab67cd..172dbd149b4 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -104,7 +104,7 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting out, is fairly simple: in most cases, we recommend doing so! With the introduction of [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro), organizations can more effectively handle complexity by utilizing multiple interconnected dbt projects rather than a single large, monolithic one. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! -- ✅ **Business groups or departments.** Conceptual separations within the project are a good reason to split up your project. For more information about dbt mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). +- ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using dbt Mesh. For more information about dbt Mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. From b5d2a4514909b06ed760ecdb5a396fe41b6097c4 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 10 Sep 2024 21:06:45 +0100 Subject: [PATCH 49/63] Update website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- .../how-we-structure/5-the-rest-of-the-project.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index 172dbd149b4..f60e0c844f2 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -102,7 +102,9 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec ### Project splitting -One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting out, is fairly simple: in most cases, we recommend doing so! With the introduction of [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro), organizations can more effectively handle complexity by utilizing multiple interconnected dbt projects rather than a single large, monolithic one. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! +One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting, is fairly simple: in most cases, we recommend doing so with [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro)! dbt Mesh allows organizations to handle complexity by connecting several dbt projects rather than relying on one big, monolithic project. This approach is designed to speed up development while maintaining governance. + +As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! - ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using dbt Mesh. For more information about dbt Mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. From f40d5b3d1ea7e1a456f0e6c1ad0068e1749ae9bb Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Tue, 10 Sep 2024 21:12:59 +0100 Subject: [PATCH 50/63] Changed order of paragraph around --- .../how-we-structure/5-the-rest-of-the-project.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index 07489ab67cd..117a78d8c67 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -105,9 +105,9 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting out, is fairly simple: in most cases, we recommend doing so! With the introduction of [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro), organizations can more effectively handle complexity by utilizing multiple interconnected dbt projects rather than a single large, monolithic one. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! - ✅ **Business groups or departments.** Conceptual separations within the project are a good reason to split up your project. For more information about dbt mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). -- ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. +- ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. ## Final considerations From 63c9b4630ea44afd81e5d1120090de023c5ba759 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 10 Sep 2024 21:25:54 +0100 Subject: [PATCH 51/63] Update website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md --- .../best-practices/how-we-structure/5-the-rest-of-the-project.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index 81151582c17..9358b507acc 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -107,7 +107,6 @@ One important, growing consideration in the analytics engineering ecosystem is h As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! - ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using dbt Mesh. For more information about dbt Mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). -- ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. From 1207ce20f26acf13c0a29da846e38b80bd8355c3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 12:27:47 +0100 Subject: [PATCH 52/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index f047ecadfff..229d83bbc58 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -551,7 +551,7 @@ models: data_type: int constraints: - type: custom - expression: tag (my_tag = 'my_value') # A custom SQL expression used to enforce a specific constraint on a column. + expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. ``` From b7cd5067c76a17fde71625c3898740563163052b Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 13:37:16 +0100 Subject: [PATCH 53/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 229d83bbc58..3a4d432dfe8 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,7 +521,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. +In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). From d7eb83035f145aca1a3f3e2caa6279acfcd5ab38 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 13:51:30 +0100 Subject: [PATCH 54/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 3a4d432dfe8..880b56b6710 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -519,7 +519,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 -### Custom constraints on models for advanced configuration of tables +### Custom constraints In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. From 6f7b622081718e8cf88ae98e1143fb6120a24d3f Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 13:51:45 +0100 Subject: [PATCH 55/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 880b56b6710..3c0b3683ab3 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,7 +521,9 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints -In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. +In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. + +Custom constraints allow you to add configuration to specific columns: - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). From b72d530a0c71ede771f5261ced43338acaae76b7 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:02:37 +0100 Subject: [PATCH 56/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 3c0b3683ab3..a53842dc5e9 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -544,7 +544,7 @@ Here's an example of how to implement tag-based masking policies with contracts models: - name: my_model - config: null + config: contract: enforced: true materialized: table From 261d61f12ced36ec78f10c263bbba42334d765a3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:03:24 +0100 Subject: [PATCH 57/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index a53842dc5e9..34bc56b947a 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -523,7 +523,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. -Custom constraints allow you to add configuration to specific columns: +Custom constraints allow you to add configuration to specific columns. For example: - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). From 42746780b317074fcfc8a624c1a90b6b9863a24b Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:03:39 +0100 Subject: [PATCH 58/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 34bc56b947a..0e7c3d25cc1 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -525,7 +525,7 @@ In dbt Cloud and dbt Core, you can use custom constraints on models for the adva Custom constraints allow you to add configuration to specific columns. For example: - - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + - Set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. From 86038b67d5c617eaf5268b0104d3e2ae1c570fbc Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:04:18 +0100 Subject: [PATCH 59/63] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 0e7c3d25cc1..99a232f4c23 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,7 @@ Custom constraints allow you to add configuration to specific columns. For examp - Set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. + - Other data warehouses (such as [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)) have their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways: From a9b4418a3b5f95add69bf0c802b85d6a80f5c2d3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:04:44 +0100 Subject: [PATCH 60/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 99a232f4c23..54391ed58f4 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,7 @@ Custom constraints allow you to add configuration to specific columns. For examp - Set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - - Other data warehouses (such as [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)) have their own set of parameters that can be set for columns in their CTAS statements. + - Other data warehouses (such as [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) have their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways: From c9b93bf24b6ed7e262137d73f92d1244ed96b304 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 12 Sep 2024 17:58:57 +0100 Subject: [PATCH 61/63] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 54391ed58f4..12691158e73 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -519,7 +519,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 -### Custom constraints +## Custom constraints In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. From 3ae24bd57d0f80dc6d8e7c59feb61f96d236cadc Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 4 Oct 2024 09:36:13 +0100 Subject: [PATCH 62/63] Update constraints.md --- .../docs/reference/resource-properties/constraints.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index de429ea71bf..353e56cd1f8 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -15,7 +15,7 @@ Constraints require the declaration and enforcement of a model [contract](/refer Constraints may be defined for a single column, or at the model level for one or more columns. As a general rule, we recommend defining single-column constraints directly on those columns. -If you are defining multiple `primary_key` constraints for a single model, those _must_ be defined at the model level. Defining multiple `primary_key` constraints at the column level is not supported. +If you define multiple `primary_key` constraints for a single model, those _must_ be defined at the model level. Defining multiple `primary_key` constraints at the column level is not supported. The structure of a constraint is: - `type` (required): one of `not_null`, `unique`, `primary_key`, `foreign_key`, `check`, `custom` @@ -588,7 +588,8 @@ You can implement constraints in a couple of different ways: - [Custom constraints with tags](#custom-constraints-with-tags) - [Custom constraints without tags](#custom-constraints-without-tags) -### Custom constraints with tags + + Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: @@ -616,9 +617,9 @@ models: Using this syntax requires configuring all the columns and their types as it’s the only way to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). + - -### Custom constraints without tags + Alternatively, you can add a masking policy without tags: @@ -642,5 +643,5 @@ models: ``` - + From 06384c611188838795be6617dcd17ed43aacc3bd Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 4 Oct 2024 09:44:13 +0100 Subject: [PATCH 63/63] Update constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 353e56cd1f8..4dced11f337 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -619,7 +619,7 @@ Using this syntax requires configuring all the columns and their types as it’s To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). - + Alternatively, you can add a masking policy without tags: