From 78ac2ba685903137d79ac873ec77026470494531 Mon Sep 17 00:00:00 2001 From: Ly Nguyen Date: Mon, 18 Sep 2023 12:02:18 -0700 Subject: [PATCH 1/9] Updates for CI phase 3 --- .../04-Sept-2023/ci-updates-phase3-rn.md | 16 +++++++ website/docs/docs/deploy/ci-jobs.md | 47 +++++-------------- .../docs/deploy/continuous-integration.md | 2 +- 3 files changed, 29 insertions(+), 36 deletions(-) create mode 100644 website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md diff --git a/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md new file mode 100644 index 00000000000..4e59c6dfcf7 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md @@ -0,0 +1,16 @@ +--- +title: "Update: Improvements to dbt Cloud continuous integration" +description: "September 2023: Improved deletion of temporary schemas" +sidebar_label: "Update: Improved deletion of temporary schemas" +tags: [Sept-2023, CI] +date: 2023-09-18 +sidebar_position: 08 +--- + +Temporary schemas are now being deleted (dropped) for all adapters (like Databricks), PrivateLink connections, and environment variables in connection strings. + +dbt Labs has rearchitected how schema deletion works for [continuous integration (CI)](/docs/deploy/continuous-integration) runs. We created a new service to delete any schema with a prefix of `dbt_cloud_pr` that's been generated by a PR run. + +However, temporary schemas will not be automatically deleted if: +- Your project overrides the `generate_schema_name` macro but it doesn't contain the prefix `dbt_cloud_pr`. +- You're using a [non-native Git integration](/docs/deploy/ci-jobs#trigger-a-ci-job-with-the-api). This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations. diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index 27229671cc4..f77bc81c10f 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -91,10 +91,20 @@ The green checkmark means the dbt build and tests were successful. Clicking on t If you're experiencing any issues, review some of the common questions and answers below. +
+ Temporary schemas aren't dropping +
+
If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates you have overriden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix. To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example: +

+ • ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)

+ • ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

+
+
+
Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps
-
If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud Slim CI.



+
If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud CI.



First, make sure you have the native GitHub authentication, native GitLab authentication, or native Azure DevOps authentication set up depending on which git provider you use. After you have gone through those steps, go to Account Settings, select Projects and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.



Once you're in the repository page, select Edit and then Disconnect Repository at the bottom.

@@ -119,7 +129,7 @@ If you're experiencing any issues, review some of the common questions and answe
Production job runs failing at the Clone Git Repository step
-
dbt Cloud can only checkout commits that belong to the original repository. dbt Cloud _cannot_ checkout commits that belong to a fork of that repository.



+
dbt Cloud can only check out commits that belong to the original repository. dbt Cloud cannot checkout commits that belong to a fork of that repository.



If you receive the following error message at the Clone Git Repository step of your job run:

@@ -144,39 +154,6 @@ If you're experiencing any issues, review some of the common questions and answe
-### Temp PR schema limitations If your temporary pull request schemas aren't dropping after a merge or close of the PR, it's likely due to the below scenarios. Open and review the toggles below for recommendations on how to resolve this: -
- You used dbt Cloud environment variables in your connection settings page -
-
To resolve this, remove environment variables in your connections settings.
-
-
-
- You have an empty/blank default schema -
-
To change this, edit and fill in your default schema.
-
-
-
- You have overridden the generate_schema_name macro -
-
To resolve this, change your macro so that the temporary PR schema name contains the default prefix and review the guidance below: -

- • ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)

- • ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

-
-
-
-
- You have overridden the generate_database_name macro -
-
If you assume that the project's default connection is to a database named analytics, review the guidance below to resolve this: -

- • ✅ Database remains the same as the connection default (like analytics)

- • ❌ Database has changed from the default connection (like dev).

-
-
-
diff --git a/website/docs/docs/deploy/continuous-integration.md b/website/docs/docs/deploy/continuous-integration.md index 779eafbb62c..23ed37afaa0 100644 --- a/website/docs/docs/deploy/continuous-integration.md +++ b/website/docs/docs/deploy/continuous-integration.md @@ -24,7 +24,7 @@ dbt Cloud builds and tests the models affected by the code change in a temporary When the CI run completes, you can view the run status directly from within the pull request. dbt Cloud updates the pull request in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run. The status message states whether the models and tests ran successfully or not. -dbt Cloud deletes the temporary schema from your  when you close or merge the pull request. If your project has database or schema customization using the [generate_database_name](/docs/build/custom-databases#generate_database_name) or [generate_schema_name](/docs/build/custom-schemas#how-does-dbt-generate-a-models-schema-name) macros, dbt Cloud might not drop the temporary schema from your data warehouse. For more information, refer to [Temp PR schema limitations](/docs/deploy/ci-jobs#temp-pr-schema-limitations). +dbt Cloud deletes the temporary schema from your  when you close or merge the pull request. If your project has schema customization using the [generate_schema_name](/docs/build/custom-schemas#how-does-dbt-generate-a-models-schema-name) macro, dbt Cloud might not drop the temporary schema from your data warehouse. For more information, refer to [Troubleshooting](/docs/deploy/ci-jobs#troubleshooting). ## Differences between CI jobs and other deployment jobs From 4ceb27319e95d29f1ab9d7a6ed7b48bc769b1005 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> Date: Mon, 18 Sep 2023 14:12:50 -0700 Subject: [PATCH 2/9] Update website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md Co-authored-by: Aaron Raff --- .../release-notes/04-Sept-2023/ci-updates-phase3-rn.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md index 4e59c6dfcf7..279de9a71d7 100644 --- a/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md +++ b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md @@ -12,5 +12,5 @@ Temporary schemas are now being deleted (dropped) for all adapters (like Databri dbt Labs has rearchitected how schema deletion works for [continuous integration (CI)](/docs/deploy/continuous-integration) runs. We created a new service to delete any schema with a prefix of `dbt_cloud_pr` that's been generated by a PR run. However, temporary schemas will not be automatically deleted if: -- Your project overrides the `generate_schema_name` macro but it doesn't contain the prefix `dbt_cloud_pr`. +- Your project overrides the `generate_schema_name` macro but it doesn't contain the prefix `dbt_cloud_pr_`. - You're using a [non-native Git integration](/docs/deploy/ci-jobs#trigger-a-ci-job-with-the-api). This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations. From 6ceb430c6cdd1cbc42d78c6089d36d9a5de34b28 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> Date: Mon, 18 Sep 2023 14:46:30 -0700 Subject: [PATCH 3/9] Update website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md Co-authored-by: Aaron Raff --- .../release-notes/04-Sept-2023/ci-updates-phase3-rn.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md index 279de9a71d7..c6bc4718b3e 100644 --- a/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md +++ b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md @@ -9,7 +9,7 @@ sidebar_position: 08 Temporary schemas are now being deleted (dropped) for all adapters (like Databricks), PrivateLink connections, and environment variables in connection strings. -dbt Labs has rearchitected how schema deletion works for [continuous integration (CI)](/docs/deploy/continuous-integration) runs. We created a new service to delete any schema with a prefix of `dbt_cloud_pr` that's been generated by a PR run. +dbt Labs has rearchitected how schema deletion works for [continuous integration (CI)](/docs/deploy/continuous-integration) runs. We created a new service to delete any schema with a prefix of `dbt_cloud_pr_` that's been generated by a PR run. However, temporary schemas will not be automatically deleted if: - Your project overrides the `generate_schema_name` macro but it doesn't contain the prefix `dbt_cloud_pr_`. From 7a3da8ae0ff2b896435464fea4b4b9b828b64b27 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> Date: Mon, 18 Sep 2023 14:50:57 -0700 Subject: [PATCH 4/9] Update website/docs/docs/deploy/ci-jobs.md --- website/docs/docs/deploy/ci-jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index f77bc81c10f..3317a71adf0 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -94,7 +94,7 @@ If you're experiencing any issues, review some of the common questions and answe
Temporary schemas aren't dropping
-
If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates you have overriden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix. To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example: +
If your temporary schemas aren't dropping after a PR merges or closes, it typically indicates you have overriden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix. To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example:

• ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)

• ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

From 1b73a03cc0818d83c5e7b4de1910e783d650fe30 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> Date: Mon, 18 Sep 2023 14:53:28 -0700 Subject: [PATCH 5/9] Update website/docs/docs/deploy/ci-jobs.md --- website/docs/docs/deploy/ci-jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index 3317a71adf0..67a2e60d8ac 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -94,7 +94,7 @@ If you're experiencing any issues, review some of the common questions and answe
Temporary schemas aren't dropping
-
If your temporary schemas aren't dropping after a PR merges or closes, it typically indicates you have overriden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix. To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example: +
If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates you have overridden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix. To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example:

• ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)

• ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

From 25643cac988e97d321fa49a102f26b7500c95433 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> Date: Mon, 18 Sep 2023 15:02:35 -0700 Subject: [PATCH 6/9] Update website/docs/docs/deploy/ci-jobs.md --- website/docs/docs/deploy/ci-jobs.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index 67a2e60d8ac..33d23e68699 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -155,5 +155,4 @@ If you're experiencing any issues, review some of the common questions and answe
-If your temporary pull request schemas aren't dropping after a merge or close of the PR, it's likely due to the below scenarios. Open and review the toggles below for recommendations on how to resolve this: From 8a09d60f1a0d5b44a868309c9d45db01e42cba11 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> Date: Wed, 20 Sep 2023 08:54:06 -0700 Subject: [PATCH 7/9] Update website/docs/docs/deploy/ci-jobs.md Co-authored-by: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/deploy/ci-jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index 33d23e68699..e08327a53e8 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -94,7 +94,7 @@ If you're experiencing any issues, review some of the common questions and answe
Temporary schemas aren't dropping
-
If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates you have overridden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix. To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example: +
If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates you have overridden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix.



To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example:

• ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)

• ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

From 3294290e5eb351fba2ffb60d74edc04b1d70bc60 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> Date: Wed, 20 Sep 2023 08:55:16 -0700 Subject: [PATCH 8/9] Update website/docs/docs/deploy/ci-jobs.md Co-authored-by: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/deploy/ci-jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index e08327a53e8..fb603e2864e 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -95,7 +95,7 @@ If you're experiencing any issues, review some of the common questions and answe Temporary schemas aren't dropping
If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates you have overridden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix.



To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example: -

+



• ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)

• ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

From 9996e3e7ff92a3c0e7abcc6a51d425649e79fca4 Mon Sep 17 00:00:00 2001 From: Ly Nguyen Date: Wed, 20 Sep 2023 09:20:27 -0700 Subject: [PATCH 9/9] Fold in feedback --- .../release-notes/04-Sept-2023/ci-updates-phase3-rn.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md index c6bc4718b3e..174de2bdaaf 100644 --- a/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md +++ b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md @@ -1,16 +1,16 @@ --- title: "Update: Improvements to dbt Cloud continuous integration" description: "September 2023: Improved deletion of temporary schemas" -sidebar_label: "Update: Improved deletion of temporary schemas" +sidebar_label: "Update: Improved automatic deletion of temporary schemas" tags: [Sept-2023, CI] date: 2023-09-18 sidebar_position: 08 --- -Temporary schemas are now being deleted (dropped) for all adapters (like Databricks), PrivateLink connections, and environment variables in connection strings. +Temporary schemas are now being automatically deleted (dropped) for all adapters (like Databricks), PrivateLink connections, and environment variables in connection strings. dbt Labs has rearchitected how schema deletion works for [continuous integration (CI)](/docs/deploy/continuous-integration) runs. We created a new service to delete any schema with a prefix of `dbt_cloud_pr_` that's been generated by a PR run. However, temporary schemas will not be automatically deleted if: -- Your project overrides the `generate_schema_name` macro but it doesn't contain the prefix `dbt_cloud_pr_`. +- Your project overrides the [generate_schema_name macro](/docs/build/custom-schemas) but it doesn't contain the required prefix `dbt_cloud_pr_`. For details, refer to [Troubleshooting](/docs/deploy/ci-jobs#troubleshooting). - You're using a [non-native Git integration](/docs/deploy/ci-jobs#trigger-a-ci-job-with-the-api). This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations.