diff --git a/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md new file mode 100644 index 00000000000..174de2bdaaf --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-updates-phase3-rn.md @@ -0,0 +1,16 @@ +--- +title: "Update: Improvements to dbt Cloud continuous integration" +description: "September 2023: Improved deletion of temporary schemas" +sidebar_label: "Update: Improved automatic deletion of temporary schemas" +tags: [Sept-2023, CI] +date: 2023-09-18 +sidebar_position: 08 +--- + +Temporary schemas are now being automatically deleted (dropped) for all adapters (like Databricks), PrivateLink connections, and environment variables in connection strings. + +dbt Labs has rearchitected how schema deletion works for [continuous integration (CI)](/docs/deploy/continuous-integration) runs. We created a new service to delete any schema with a prefix of `dbt_cloud_pr_` that's been generated by a PR run. + +However, temporary schemas will not be automatically deleted if: +- Your project overrides the [generate_schema_name macro](/docs/build/custom-schemas) but it doesn't contain the required prefix `dbt_cloud_pr_`. For details, refer to [Troubleshooting](/docs/deploy/ci-jobs#troubleshooting). +- You're using a [non-native Git integration](/docs/deploy/ci-jobs#trigger-a-ci-job-with-the-api). This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations. diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index 27229671cc4..fb603e2864e 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -91,10 +91,20 @@ The green checkmark means the dbt build and tests were successful. Clicking on t If you're experiencing any issues, review some of the common questions and answers below. +
+ Temporary schemas aren't dropping +
+
If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates you have overridden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix.



To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example: +



+ • ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)

+ • ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

+
+
+
Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps
-
If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud Slim CI.



+
If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud CI.



First, make sure you have the native GitHub authentication, native GitLab authentication, or native Azure DevOps authentication set up depending on which git provider you use. After you have gone through those steps, go to Account Settings, select Projects and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.



Once you're in the repository page, select Edit and then Disconnect Repository at the bottom.

@@ -119,7 +129,7 @@ If you're experiencing any issues, review some of the common questions and answe
Production job runs failing at the Clone Git Repository step
-
dbt Cloud can only checkout commits that belong to the original repository. dbt Cloud _cannot_ checkout commits that belong to a fork of that repository.



+
dbt Cloud can only check out commits that belong to the original repository. dbt Cloud cannot checkout commits that belong to a fork of that repository.



If you receive the following error message at the Clone Git Repository step of your job run:

@@ -144,39 +154,5 @@ If you're experiencing any issues, review some of the common questions and answe
-### Temp PR schema limitations -If your temporary pull request schemas aren't dropping after a merge or close of the PR, it's likely due to the below scenarios. Open and review the toggles below for recommendations on how to resolve this: -
- You used dbt Cloud environment variables in your connection settings page -
-
To resolve this, remove environment variables in your connections settings.
-
-
-
- You have an empty/blank default schema -
-
To change this, edit and fill in your default schema.
-
-
-
- You have overridden the generate_schema_name macro -
-
To resolve this, change your macro so that the temporary PR schema name contains the default prefix and review the guidance below: -

- • ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)

- • ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

-
-
-
-
- You have overridden the generate_database_name macro -
-
If you assume that the project's default connection is to a database named analytics, review the guidance below to resolve this: -

- • ✅ Database remains the same as the connection default (like analytics)

- • ❌ Database has changed from the default connection (like dev).

-
-
-
diff --git a/website/docs/docs/deploy/continuous-integration.md b/website/docs/docs/deploy/continuous-integration.md index 779eafbb62c..23ed37afaa0 100644 --- a/website/docs/docs/deploy/continuous-integration.md +++ b/website/docs/docs/deploy/continuous-integration.md @@ -24,7 +24,7 @@ dbt Cloud builds and tests the models affected by the code change in a temporary When the CI run completes, you can view the run status directly from within the pull request. dbt Cloud updates the pull request in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run. The status message states whether the models and tests ran successfully or not. -dbt Cloud deletes the temporary schema from your  when you close or merge the pull request. If your project has database or schema customization using the [generate_database_name](/docs/build/custom-databases#generate_database_name) or [generate_schema_name](/docs/build/custom-schemas#how-does-dbt-generate-a-models-schema-name) macros, dbt Cloud might not drop the temporary schema from your data warehouse. For more information, refer to [Temp PR schema limitations](/docs/deploy/ci-jobs#temp-pr-schema-limitations). +dbt Cloud deletes the temporary schema from your  when you close or merge the pull request. If your project has schema customization using the [generate_schema_name](/docs/build/custom-schemas#how-does-dbt-generate-a-models-schema-name) macro, dbt Cloud might not drop the temporary schema from your data warehouse. For more information, refer to [Troubleshooting](/docs/deploy/ci-jobs#troubleshooting). ## Differences between CI jobs and other deployment jobs