From 9a869c6ace3338c3ce82202348cf592aabe584ef Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 3 Sep 2024 17:45:19 -0400 Subject: [PATCH 01/24] Updating multi cell migration page (#6002) ## What are you changing in this pull request and why? Updating the migration checklist based on recent changes. Many older manual items are now automated, so customers need to take a minimum of actions. ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --------- Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/docs/cloud/migration.md | 51 +++++++++++++++++----------- 1 file changed, 31 insertions(+), 20 deletions(-) diff --git a/website/docs/docs/cloud/migration.md b/website/docs/docs/cloud/migration.md index 8bdf47eae5a..3aec1956297 100644 --- a/website/docs/docs/cloud/migration.md +++ b/website/docs/docs/cloud/migration.md @@ -7,34 +7,45 @@ pagination_next: null pagination_prev: null --- -dbt Labs is in the process of migrating dbt Cloud to a new _cell-based architecture_. This architecture will be the foundation of dbt Cloud for years to come, and will bring improved scalability, reliability, and security to all customers and users of dbt Cloud. +dbt Labs is in the process of rolling out a new cell-based architecture for dbt Cloud. This architecture provides the foundation of dbt Cloud for years to come, and brings improved reliability, performance, and consistency to users of dbt Cloud. -There is some preparation required to ensure a successful migration. +We're scheduling migrations by account. When we're ready to migrate your account, you will receive a banner or email communication with your migration date. If you have not received this communication, then you don't need to take action at this time. dbt Labs will share information about your migration with you, with appropriate advance notice, when applicable to your account. -Migrations are being scheduled on a per-account basis. _If you haven't received any communication (either with a banner or by email) about a migration date, you don't need to take any action at this time._ dbt Labs will share migration date information with you, with appropriate advance notice, before we complete any migration steps in the dbt Cloud backend. +Your account will be automatically migrated on its scheduled date. However, if you use certain features, you must take action before that date to avoid service disruptions. -This document outlines the steps that you must take to prevent service disruptions before your environment is migrated over to the cell-based architecture. This will impact areas such as login, IP restrictions, and API access. +## Recommended actions -## Pre-migration checklist +We highly recommended you take these actions: -Prior to your migration date, your dbt Cloud account admin will need to make some changes to your account. Most of your configurations will be migrated automatically, but a few will require manual intervention. +- Ensure pending user invitations are accepted or note outstanding invitations. Pending user invitations will be voided during the migration and must be resent after it is complete. +- Commit unsaved changes in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud). Unsaved changes will be lost during migration. +- Export and download [audit logs](/docs/cloud/manage-access/audit-log) older than 90 days, as they will be lost during migration. If you lose critical logs older than 90 days during the migration, you will have to work with the dbt Labs Customer Support team to recover. -If your account is scheduled for migration, you will see a banner indicating your migration date when you log in. If you don't see a banner, you don't need to take any action. +## Required actions -1. **IP addresses** — dbt Cloud will be using new IPs to access your warehouse after the migration. Make sure to allow inbound traffic from these IPs in your firewall and include it in any database grants. All six of the IPs below should be added to allowlists. - * Old IPs: `52.45.144.63`, `54.81.134.249`, `52.22.161.231` - * New IPs: `52.3.77.232`, `3.214.191.130`, `34.233.79.135` -2. **User invitations** — Any pending user invitations will be invalidated during the migration. You can resend the invitations after the migration is complete. -3. **SSO integrations** — If you've completed the Auth0 migration, your account SSO configurations will be automatically transferred. If you haven't completed the Auth0 migration, dbt Labs recommends doing that before starting the mult-cell migration to avoid service disruptions. -4. **IDE sessions** — Any unsaved changes in the IDE might be lost during migration. dbt Labs _strongly_ recommends committing all changes in the IDE before your scheduled migration time. +These actions are required to prevent users from losing access dbt Cloud: -## Post-migration +- If you still need to, complete [Auth0 migration for SSO](/docs/cloud/manage-access/auth0-migration) before your scheduled migration date to avoid service disruptions. If you've completed the Auth0 migration, your account SSO configurations will be transferred automatically. +- Update your IP allow lists. dbt Cloud will be using new IPs to access your warehouse post-migration. Allow inbound traffic from all of the following new IPs in your firewall and include them in any database grants: -After migration, if you completed all the [Pre-migration checklist](#pre-migration-checklist) items, your dbt Cloud resources and jobs will continue to work as they did before. + - `52.3.77.232` + - `3.214.191.130` + - `34.233.79.135` -You have the option to log in to dbt Cloud at a different URL: - * If you were previously logging in at `cloud.getdbt.com`, you should instead plan to login at `us1.dbt.com`. The original URL will still work, but you’ll have to click through to be redirected upon login. - * You may also log in directly with your account’s unique [access URL](/docs/cloud/about-cloud/access-regions-ip-addresses#accessing-your-account). + Keep the old dbt Cloud IPs listed until the migration is complete. -:::info Login with GitHub -Users who previously used the "Login with GitHub" functionality will no longer be able to use this method to login to dbt Cloud after migration. To continue accessing your account, you can use your existing email and password. +## Post-migration​ + +Complete all of these items to ensure your dbt Cloud resources and jobs will continue working without interruption. + +Use one of these two URL login options: + +- `us1.dbt.com.` If you were previously logging in at `cloud.getdbt.com`, you should instead plan to log in at us1.dbt.com. The original URL will still work, but you’ll have to click through to be redirected upon login. +- `ACCOUNT_PREFIX.us1.dbt.com`: A unique URL specifically for your account. If you belong to multiple accounts, each will have a unique URL available as long as they have been migrated to multi-cell. +Check out [access, regions, and IP addresses](/docs/cloud/about-cloud/access-regions-ip-addresses) for more information. + +Remove the following old IP addresses from your firewall and database grants: + +- `52.45.144.63` +- `54.81.134.249` +- `52.22.161.231` From 331c95a62b932cd6385ae92debf84d20cea6d8c5 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 3 Sep 2024 19:15:50 -0400 Subject: [PATCH 02/24] Updating beta banner (#6004) ## What are you changing in this pull request and why? Based on some recent changes we've made (and may continue to make) around versions, updating the banner to be more generic and some of the language ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --------- Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/src/theme/DocRoot/Layout/Main/index.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/src/theme/DocRoot/Layout/Main/index.js b/website/src/theme/DocRoot/Layout/Main/index.js index 7303e484863..29e7efdc253 100644 --- a/website/src/theme/DocRoot/Layout/Main/index.js +++ b/website/src/theme/DocRoot/Layout/Main/index.js @@ -71,7 +71,7 @@ export default function DocRootLayoutMain({ } else { setPreData({ showisPrereleaseBanner: true, - isPrereleaseBannerText: `You are currently viewing v${dbtVersion}, which is a prerelease of dbt Core. The latest stable version is v${latestStableRelease}`, + isPrereleaseBannerText: `You are viewing the docs for a prerelease version of dbt Core. There may be features described that are still in development, incomplete, or unstable.`, }); } // If EOLDate not set for version, do not show banner From 53227181c9f31d3eceea4cef85186c9a0f982284 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 4 Sep 2024 11:17:49 -0400 Subject: [PATCH 03/24] Update EOL banners (#6009) ## What are you changing in this pull request and why? Updating the EOL banner texts, removing versions and fixing links in the text. ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --------- Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/src/theme/DocRoot/Layout/Main/index.js | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/src/theme/DocRoot/Layout/Main/index.js b/website/src/theme/DocRoot/Layout/Main/index.js index 29e7efdc253..458cb9d8716 100644 --- a/website/src/theme/DocRoot/Layout/Main/index.js +++ b/website/src/theme/DocRoot/Layout/Main/index.js @@ -71,7 +71,7 @@ export default function DocRootLayoutMain({ } else { setPreData({ showisPrereleaseBanner: true, - isPrereleaseBannerText: `You are viewing the docs for a prerelease version of dbt Core. There may be features described that are still in development, incomplete, or unstable.`, + isPrereleaseBannerText: `You are viewing the docs for a prerelease version of dbt Core. There may be features described that are still in development, incomplete, or unstable. For the latest generally available features, install the latest stable version`, }); } // If EOLDate not set for version, do not show banner @@ -86,12 +86,12 @@ export default function DocRootLayoutMain({ if (new Date() > new Date(EOLDate)) { setEOLData({ showEOLBanner: true, - EOLBannerText: `This version of dbt Core is no longer supported. No patch releases will be made, even for critical security issues. For better performance, improved security, and new features, you should upgrade to ${latestStableRelease}, the latest stable version.`, + EOLBannerText: `This version of dbt Core is no longer supported. There will be no more patches or security fixes. For improved performance, security, and features, upgrade to the latest stable version.`, }); } else if (new Date() > threeMonths) { setEOLData({ showEOLBanner: true, - EOLBannerText: `This version of dbt Core is nearing the end of its critical support period. For better performance, improved security, and new features, you should upgrade to ${latestStableRelease}, the latest stable version.`, + EOLBannerText: `This version of dbt Core is nearing the end of its critical support period. For improved perfomance, security, and features, upgrade to the latest stable version.`, }); } else { setEOLData({ From 54fce192191cca139413b1f8ea265c9ac9c9f329 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 4 Sep 2024 16:17:18 -0400 Subject: [PATCH 04/24] Removing github login (#6010) ## What are you changing in this pull request and why? Removing GitHub login from dbt Cloud - Removed/edited content on the GitHub integration page - Edited links - Added a release note ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --------- Co-authored-by: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> --- website/docs/docs/cloud/about-cloud/browsers.md | 2 +- website/docs/docs/cloud/git/connect-github.md | 9 ++++----- website/docs/docs/dbt-versions/release-notes.md | 1 + 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/website/docs/docs/cloud/about-cloud/browsers.md b/website/docs/docs/cloud/about-cloud/browsers.md index 12665bc7b72..1e26d3a6d59 100644 --- a/website/docs/docs/cloud/about-cloud/browsers.md +++ b/website/docs/docs/cloud/about-cloud/browsers.md @@ -27,4 +27,4 @@ To improve your experience using dbt Cloud, we suggest that you turn off ad bloc A session is a period of time during which you’re signed in to a dbt Cloud account from a browser. If you close your browser, it will end your session and log you out. You'll need to log in again the next time you try to access dbt Cloud. -If you've logged in using [SSO](/docs/cloud/manage-access/sso-overview) or [OAuth](/docs/cloud/git/connect-github#personally-authenticate-with-github), you can customize your maximum session duration, which might vary depending on your identity provider (IdP). +If you've logged in using [SSO](/docs/cloud/manage-access/sso-overview), you can customize your maximum session duration, which might vary depending on your identity provider (IdP). diff --git a/website/docs/docs/cloud/git/connect-github.md b/website/docs/docs/cloud/git/connect-github.md index 4dc4aaf73e9..f230f70e1f6 100644 --- a/website/docs/docs/cloud/git/connect-github.md +++ b/website/docs/docs/cloud/git/connect-github.md @@ -7,7 +7,6 @@ sidebar_label: "Connect to GitHub" Connecting your GitHub account to dbt Cloud provides convenience and another layer of security to dbt Cloud: -- Log into dbt Cloud using OAuth through GitHub. - Import new GitHub repositories with a couple clicks during dbt Cloud project setup. - Clone repos using HTTPS rather than SSH. - Trigger [Continuous integration](/docs/deploy/continuous-integration)(CI) builds when pull requests are opened in GitHub. @@ -48,15 +47,15 @@ To connect your dbt Cloud account to your GitHub account: - Read and write access to Workflows 6. Once you grant access to the app, you will be redirected back to dbt Cloud and shown a linked account success state. You are now personally authenticated. -7. Ask your team members to [personally authenticate](/docs/cloud/git/connect-github#personally-authenticate-with-github) by connecting their GitHub profiles. +7. Ask your team members to individually authenticate by connecting their [personal GitHub profiles](#authenticate-your-personal-github-account). ## Limiting repository access in GitHub If you are your GitHub organization owner, you can also configure the dbt Cloud GitHub application to have access to only select repositories. This configuration must be done in GitHub, but we provide an easy link in dbt Cloud to start this process. -## Personally authenticate with GitHub +## Authenticate your personal GitHub account -Once the dbt Cloud admin has [set up a connection](/docs/cloud/git/connect-github#installing-dbt-cloud-in-your-github-account) to your organization GitHub account, you need to personally authenticate, which improves the security of dbt Cloud by enabling you to log in using OAuth through GitHub. +After the dbt Cloud administrator [sets up a connection](/docs/cloud/git/connect-github#installing-dbt-cloud-in-your-github-account) to your organization's GitHub account, you need to authenticate using your personal account. You must connect your personal GitHub profile to dbt Cloud to use the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) and [CLI](/docs/cloud/cloud-cli-installation) and verify your read and write access to the repository. :::info GitHub profile connection @@ -77,7 +76,7 @@ To connect a personal GitHub account: 4. Once you approve authorization, you will be redirected to dbt Cloud, and you should now see your connected account. -The next time you log into dbt Cloud, you will be able to do so via OAuth through GitHub, and if you're on the Enterprise plan, you're ready to use the dbt Cloud IDE or dbt Cloud CLI. +You can now use the dbt Cloud IDE or dbt Cloud CLI. ## FAQs diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index a9db34334ad..57aadba284d 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,6 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## August 2024 +- **Behavior change:** GitHub is no longer supported for OAuth login to dbt Cloud. Use a supported [SSO or OAuth provider](/docs/cloud/manage-access/sso-overview) to securely manage access to your dbt Cloud account. - **New**: You can now configure metrics at granularities at finer time grains, such as hour, minute, or even by the second. This is particularly useful for more detailed analysis and for datasets where high-resolution time data is required, such as minute-by-minute event tracking. Refer to [dimensions](/docs/build/dimensions) for more information about time granularity. ## July 2024 From 3b596ebf18e54894134416fbbb41c0be0714a498 Mon Sep 17 00:00:00 2001 From: Lana <100206579+Lana-hl@users.noreply.github.com> Date: Thu, 5 Sep 2024 02:41:19 -0600 Subject: [PATCH 05/24] Update set-up-snowflake-oauth.md (#6015) Adding quotes to `OAUTH_REDIRECT_URI` value as it doesn't work without the quotes ## What are you changing in this pull request and why? ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --- website/docs/docs/cloud/manage-access/set-up-snowflake-oauth.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/manage-access/set-up-snowflake-oauth.md b/website/docs/docs/cloud/manage-access/set-up-snowflake-oauth.md index 3b3b9c2d870..e9c4236438e 100644 --- a/website/docs/docs/cloud/manage-access/set-up-snowflake-oauth.md +++ b/website/docs/docs/cloud/manage-access/set-up-snowflake-oauth.md @@ -43,7 +43,7 @@ CREATE OR REPLACE SECURITY INTEGRATION DBT_CLOUD ENABLED = TRUE OAUTH_CLIENT = CUSTOM OAUTH_CLIENT_TYPE = 'CONFIDENTIAL' - OAUTH_REDIRECT_URI = LOCATED_REDIRECT_URI + OAUTH_REDIRECT_URI = 'LOCATED_REDIRECT_URI' OAUTH_ISSUE_REFRESH_TOKENS = TRUE OAUTH_REFRESH_TOKEN_VALIDITY = 7776000; ``` From a923185b853f5700b364bbeb02451d5d580e59bb Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Thu, 5 Sep 2024 02:51:19 -0600 Subject: [PATCH 06/24] Fix config names for hooks for `properties.yml` (#6007) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [Preview](https://docs-getdbt-com-git-dbeatty10-patch-1-dbt-labs.vercel.app/reference/seed-configs#general-configurations) - General configurations > Property file ## What are you changing in this pull request and why? Fixes per https://github.com/dbt-labs/dbt-core/issues/10650#issuecomment-2328348338 ## 🎩 image ## Checklist - [x] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [x] The topic I'm writing about applies to all dbt versions Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/seed-configs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/seed-configs.md b/website/docs/reference/seed-configs.md index dd733795eef..5d5c39071d6 100644 --- a/website/docs/reference/seed-configs.md +++ b/website/docs/reference/seed-configs.md @@ -113,8 +113,8 @@ seeds: config: [enabled](/reference/resource-configs/enabled): true | false [tags](/reference/resource-configs/tags): | [] - [pre-hook](/reference/resource-configs/pre-hook-post-hook): | [] - [post-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [pre_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [post_hook](/reference/resource-configs/pre-hook-post-hook): | [] [database](/reference/resource-configs/database): [schema](/reference/resource-properties/schema): [alias](/reference/resource-configs/alias): From 4d1591cd13b9302949e648663ff61ea6052318ed Mon Sep 17 00:00:00 2001 From: Perth Ngarmtrakulchol Date: Thu, 5 Sep 2024 19:01:49 +1000 Subject: [PATCH 07/24] Add a text in docs to prevent using -- for comment. (#6006) The error message (local variable 'connection' referenced before assignment) when using `--` doesn't indicate the actual syntax error. It is also not documented ## What are you changing in this pull request and why? Stumbled upon error from this issue and tried to debug it for few hours until I realised this error. Adding text in the docs to prevent other developers finding the same issue. ## Checklist - [x] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [x] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/build/jinja-macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/jinja-macros.md b/website/docs/docs/build/jinja-macros.md index fc4a0cad3e8..bc91e3674c9 100644 --- a/website/docs/docs/build/jinja-macros.md +++ b/website/docs/docs/build/jinja-macros.md @@ -74,7 +74,7 @@ group by 1 You can recognize Jinja based on the delimiters the language uses, which we refer to as "curlies": - **Expressions `{{ ... }}`**: Expressions are used when you want to output a string. You can use expressions to reference [variables](/reference/dbt-jinja-functions/var) and call [macros](/docs/build/jinja-macros#macros). - **Statements `{% ... %}`**: Statements don't output a string. They are used for control flow, for example, to set up `for` loops and `if` statements, to [set](https://jinja.palletsprojects.com/en/3.1.x/templates/#assignments) or [modify](https://jinja.palletsprojects.com/en/3.1.x/templates/#expression-statement) variables, or to define macros. -- **Comments `{# ... #}`**: Jinja comments are used to prevent the text within the comment from executing or outputing a string. +- **Comments `{# ... #}`**: Jinja comments are used to prevent the text within the comment from executing or outputing a string. Don't use `--` for comment. When used in a dbt model, your Jinja needs to compile to a valid query. To check what SQL your Jinja compiles to: * **Using dbt Cloud:** Click the compile button to see the compiled SQL in the Compiled SQL pane From 2399e86f9b279ffad18f307a7be2f893f19110cb Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 5 Sep 2024 10:21:20 +0100 Subject: [PATCH 08/24] add model paths callout (#6011) adding callout to SL best practices and SL quickstart so users know that sm and metrics should be under the model-path directory. raised in [internal slack thread](https://dbt-labs.slack.com/archives/C05K4R7KZ5Z/p1724945059555389?thread_ts=1724777237.590479&cid=C05K4R7KZ5Z) --- .../semantic-layer-7-semantic-structure.md | 4 ++++ website/docs/guides/sl-snowflake-qs.md | 11 ++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-7-semantic-structure.md b/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-7-semantic-structure.md index 295d86e9c20..5bfbea82dda 100644 --- a/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-7-semantic-structure.md +++ b/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-7-semantic-structure.md @@ -20,6 +20,10 @@ The first thing you need to establish is how you’re going to consistently stru It’s not terribly difficult to shift between these (it can be done with some relatively straightforward shell scripting), and this is purely a decision based on your developers’ preference (i.e. it has no impact on execution or performance), so don’t feel locked in to either path. Just pick the one that feels right and you can always shift down the road if you change your mind. +:::tip +Make sure to save all semantic models and metrics under the directory defined in the [`model-paths`](/reference/project-configs/model-paths) (or a subdirectory of it, like `models/semantic_models/`). If you save them outside of this path, it will result in an empty `semantic_manifest.json` file, and your semantic models or metrics won't be recognized. +::: + ## Naming Next, establish your system for consistent file naming: diff --git a/website/docs/guides/sl-snowflake-qs.md b/website/docs/guides/sl-snowflake-qs.md index 6d9f88ab159..7d42aecabc2 100644 --- a/website/docs/guides/sl-snowflake-qs.md +++ b/website/docs/guides/sl-snowflake-qs.md @@ -619,6 +619,11 @@ select * from final In the following steps, semantic models enable you to define how to interpret the data related to orders. It includes entities (like ID columns serving as keys for joining data), dimensions (for grouping or filtering data), and measures (for data aggregations). 1. In the `metrics` sub-directory, create a new file `fct_orders.yml`. + +:::tip +Make sure to save all semantic models and metrics under the directory defined in the [`model-paths`](/reference/project-configs/model-paths) (or a subdirectory of it, like `models/semantic_models/`). If you save them outside of this path, it will result in an empty `semantic_manifest.json` file, and your semantic models or metrics won't be recognized. +::: + 2. Add the following code to that newly created file: @@ -765,7 +770,11 @@ There are different types of metrics you can configure: Once you've created your semantic models, it's time to start referencing those measures you made to create some metrics: -Add metrics to your `fct_orders.yml` semantic model file: +1. Add metrics to your `fct_orders.yml` semantic model file: + +:::tip +Make sure to save all semantic models and metrics under the directory defined in the [`model-paths`](/reference/project-configs/model-paths) (or a subdirectory of it, like `models/semantic_models/`). If you save them outside of this path, it will result in an empty `semantic_manifest.json` file, and your semantic models or metrics won't be recognized. +::: From a451264d28b67620ab69bf83104f41b6499e525a Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Thu, 5 Sep 2024 04:00:10 -0600 Subject: [PATCH 09/24] General examples of calling a macro in a hook (#5972) [Preview: Calling a macro in a hook](https://docs-getdbt-com-git-dbeatty10-patch-2-dbt-labs.vercel.app/docs/build/hooks-operations#calling-a-macro-in-a-hook) [Preview: pre-hook & post-hook ](https://docs-getdbt-com-git-dbeatty10-patch-2-dbt-labs.vercel.app/reference/resource-configs/pre-hook-post-hook) ## What are you changing in this pull request and why? We have a couple concrete examples of calling a macro in a hook (like [here](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end#call-a-macro-to-grant-privileges) and [here](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook#apache-spark-analyze-tables-after-creation)). But we don't currently have any co-located canonical examples of the different locations where hooks can be configured: 1. SQL model config 1. `properties.yml` \* 1. `dbt_project.yml` \* Calling a macro in a hook within a properties YAML file was fixed in v1.8+ by https://github.com/dbt-labs/dbt-core/pull/10603. ## Other pages to review Once https://github.com/dbt-labs/dbt-core/issues/7128 is resolved, we should also review these pages to ensure they include examples of configs across `dbt_project` YAML files, properties YAML files, and node-level config in SQL files: - https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook - https://docs.getdbt.com/reference/model-configs#apply-configurations-to-one-model-only - https://docs.getdbt.com/reference/seed-configs#apply-the-schema-configuration-to-one-seed-only ## Additional context Inspired by https://github.com/dbt-labs/dbt-core/issues/10522 ## Checklist - [x] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. --------- Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/build/hooks-operations.md | 35 ++++++++++++++++++ .../resource-configs/pre-hook-post-hook.md | 36 +++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/website/docs/docs/build/hooks-operations.md b/website/docs/docs/build/hooks-operations.md index 9ed20291c34..6cec2a673c0 100644 --- a/website/docs/docs/build/hooks-operations.md +++ b/website/docs/docs/build/hooks-operations.md @@ -72,6 +72,41 @@ You can use hooks to provide database-specific functionality not available out-o You can also use a [macro](/docs/build/jinja-macros#macros) to bundle up hook logic. Check out some of the examples in the reference sections for [on-run-start and on-run-end hooks](/reference/project-configs/on-run-start-on-run-end) and [pre- and post-hooks](/reference/resource-configs/pre-hook-post-hook). + + +```sql +{{ config( + pre_hook=[ + "{{ some_macro() }}" + ] +) }} +``` + + + + + +```yaml +models: + - name: + config: + pre_hook: + - "{{ some_macro() }}" +``` + + + + + +```yaml +models: + : + +pre-hook: + - "{{ some_macro() }}" +``` + + + ## About operations Operations are [macros](/docs/build/jinja-macros#macros) that you can run using the [`run-operation`](/reference/commands/run-operation) command. As such, operations aren't actually a separate resource in your dbt project — they are just a convenient way to invoke a macro without needing to run a model. diff --git a/website/docs/reference/resource-configs/pre-hook-post-hook.md b/website/docs/reference/resource-configs/pre-hook-post-hook.md index bf4375c9490..e1e7d67f02e 100644 --- a/website/docs/reference/resource-configs/pre-hook-post-hook.md +++ b/website/docs/reference/resource-configs/pre-hook-post-hook.md @@ -45,6 +45,18 @@ select ... ``` + + + + +```yml +models: + - name: [] + config: + [pre_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [post_hook](/reference/resource-configs/pre-hook-post-hook): | [] +``` + @@ -66,6 +78,18 @@ seeds: + + +```yml +seeds: + - name: [] + config: + [pre_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [post_hook](/reference/resource-configs/pre-hook-post-hook): | [] +``` + + + @@ -102,6 +126,18 @@ select ... + + +```yml +snapshots: + - name: [] + config: + [pre_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [post_hook](/reference/resource-configs/pre-hook-post-hook): | [] +``` + + + From 9850338758e6c26b4b4f9cd6414efdac91f637fe Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 5 Sep 2024 11:05:45 +0100 Subject: [PATCH 10/24] add search heavy frontmatter (#5994) adding a search_weight frontmatter option and keywords to the custome ci/cd pipeline page. thisis because this guide isn't listed in the initial search results went searching in algolia. raised in slack channel: https://dbt-labs.slack.com/archives/C02NCQ9483C/p1725010557552679 [notion doc](https://www.notion.so/dbtlabs/DocSearch-Weight-Component-Documentation-58aaa3aca7ff42938887e6159f63e7da) --- website/blog/2022-04-14-add-ci-cd-to-bitbucket.md | 2 ++ website/docs/guides/custom-cicd-pipelines.md | 4 +++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/website/blog/2022-04-14-add-ci-cd-to-bitbucket.md b/website/blog/2022-04-14-add-ci-cd-to-bitbucket.md index eae8d595ca5..e871687d8cd 100644 --- a/website/blog/2022-04-14-add-ci-cd-to-bitbucket.md +++ b/website/blog/2022-04-14-add-ci-cd-to-bitbucket.md @@ -10,6 +10,8 @@ hide_table_of_contents: false date: 2022-05-06 is_featured: true +keywords: + - dbt core pipeline, slim ci pipeline, slim cd pipeline, bitbucket --- diff --git a/website/docs/guides/custom-cicd-pipelines.md b/website/docs/guides/custom-cicd-pipelines.md index 59a7767c69b..be23524d096 100644 --- a/website/docs/guides/custom-cicd-pipelines.md +++ b/website/docs/guides/custom-cicd-pipelines.md @@ -10,6 +10,9 @@ hide_table_of_contents: true tags: ['dbt Cloud', 'Orchestration', 'CI'] level: 'Intermediate' recently_updated: true +search_weight: "heavy" +keywords: + - bitbucket pipeline, custom pipelines, github, gitlab, azure devops, ci/cd custom pipeline ---
@@ -19,7 +22,6 @@ One of the core tenets of dbt is that analytic code should be version controlled A note on parlance in this article since each code hosting platform uses different terms for similar concepts. The terms `pull request` (PR) and `merge request` (MR) are used interchangeably to mean the process of merging one branch into another branch. - ### What are pipelines? Pipelines (which are known by many names, such as workflows, actions, or build steps) are a series of pre-defined jobs that are triggered by specific events in your repository (PR created, commit pushed, branch merged, etc). Those jobs can do pretty much anything your heart desires assuming you have the proper security access and coding chops. From 8eceeda49b75e47b065fc5f75fcc306e21732d80 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 5 Sep 2024 15:25:48 +0100 Subject: [PATCH 11/24] update ip doc and add clarification (#6020) This PR updates the IP restrictions doc and addresses the issues raised in #5727. Resolves #5727 --- website/docs/docs/cloud/secure/ip-restrictions.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/website/docs/docs/cloud/secure/ip-restrictions.md b/website/docs/docs/cloud/secure/ip-restrictions.md index 034b3a6c144..d39960dab42 100644 --- a/website/docs/docs/cloud/secure/ip-restrictions.md +++ b/website/docs/docs/cloud/secure/ip-restrictions.md @@ -13,7 +13,7 @@ import SetUpPages from '/snippets/_available-tiers-iprestrictions.md'; IP Restrictions help control which IP addresses are allowed to connect to dbt Cloud. IP restrictions allow dbt Cloud customers to meet security and compliance controls by only allowing approved IPs to connect to their dbt Cloud environment. This feature is supported in all regions across NA, Europe, and Asia-Pacific, but contact us if you have questions about availability. -## Configuring IP Restrictions +## Configuring IP restrictions To configure IP restrictions, go to **Account Settings** → **IP Restrictions**. IP restrictions provide two methods for determining which IPs can access dbt Cloud: an allowlist and a blocklist. IPs in the allowlist are allowed to access dbt Cloud, and IPs in the deny list will be blocked from accessing dbt Cloud. IP Restrictions can be used for a range of use cases, including: @@ -29,7 +29,7 @@ For any version control system integrations (Github, Gitlab, ADO, etc.) inbound To add an IP to the allowlist, from the **IP Restrictions** page: -1. Click **edit** +1. Click **Edit** 2. Click **Add Rule** 3. Add name and description for the rule - For example, Corporate VPN CIDR Range @@ -39,7 +39,9 @@ To add an IP to the allowlist, from the **IP Restrictions** page: - You can add multiple ranges in the same rule. 6. Click **Save** -Note that simply adding the IP Ranges will not enforce IP restrictions. For more information, see the section “Enabling Restrictions.” +Add multiple IP ranges by clicking the **Add IP range** button to create a new text field. + +Note that simply adding the IP Ranges will not enforce IP restrictions. For more information, see the [Enabling restrictions](#enabling-restrictions) section. If you only want to allow the IP ranges added to this list and deny all other requests, adding a denylist is not necessary. By default, if only an allow list is added, dbt Cloud will only allow IPs in the allowable range and deny all other IPs. However, you can add a denylist if you want to deny specific IP addresses within your allowlist CIDR range. @@ -65,9 +67,9 @@ It is possible to put an IP range on one list and then a sub-range or IP address ::: -## Enabling Restrictions +## Enabling restrictions -Once you are done adding all your ranges, IP restrictions can be enabled by selecting the **Enable IP restrictions** button and clicking **Save**. If your IP address is in any of the denylist ranges, you won’t be able to save or enable IP restrictions - this is done to prevent accidental account lockouts. If you do get locked out due to IP changes on your end, please reach out to support@dbtlabs.com +Once you are done adding all your ranges, IP restrictions can be enabled by selecting the **Enable IP restrictions** button and clicking **Save**. If your IP address is in any of the denylist ranges, you won’t be able to save or enable IP restrictions - this is done to prevent accidental account lockouts. If you do get locked out due to IP changes on your end, please reach out to support@getdbt.com Once enabled, when someone attempts to access dbt Cloud from a restricted IP, they will encounter one of the following messages depending on whether they use email & password or SSO login. From 9ae02b8713f6f7fdc8b2546549b37061d18bbd3a Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 5 Sep 2024 12:19:58 -0400 Subject: [PATCH 12/24] Removing versions (#6021) ## What are you changing in this pull request and why? Now that we have the option for dbt Cloud in the docs, we are removing the v1.9 beta content until the beta is actually out. This removes the whole upgrade guide since the sidebar is autogenerated. ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --- website/dbt-versions.js | 4 --- .../core-upgrade/06-upgrading-to-v1.9.md | 27 ------------------- 2 files changed, 31 deletions(-) delete mode 100644 website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md diff --git a/website/dbt-versions.js b/website/dbt-versions.js index e5a2b9f4290..0f322100a45 100644 --- a/website/dbt-versions.js +++ b/website/dbt-versions.js @@ -16,10 +16,6 @@ exports.versions = [ version: "1.9.1", customDisplay: "Cloud (Versionless)", }, - { - version: "1.9", - isPrerelease: true, - }, { version: "1.8", EOLDate: "2025-04-15", diff --git a/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md b/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md deleted file mode 100644 index 544590b18df..00000000000 --- a/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: "Upgrading to v1.9 (beta)" -id: upgrading-to-v1.9 -description: New features and changes in dbt Core v1.9 -displayed_sidebar: "docs" ---- - -## Resources - -- Changelog (coming soon) -- [dbt Core CLI Installation guide](/docs/core/installation-overview) -- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud) — dbt Cloud is now versionless. dbt v1.9 will not appear in the version dropdown. Select **Versionless** to get all the latest features and functionality in your dbt Cloud account. - -## What to know before upgrading - -dbt Labs is committed to providing backward compatibility for all versions 1.x, except for any changes explicitly mentioned on this page. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new). - - -## New and changed features and functionality - -Features and functionality new in dbt v1.9. - -**Coming soon** - -## Quick hits - -**Coming soon** \ No newline at end of file From 03507dd449da1ecb1d6f82fce817266961684c4e Mon Sep 17 00:00:00 2001 From: Jeremy Cohen Date: Thu, 5 Sep 2024 21:24:44 +0200 Subject: [PATCH 13/24] Update "Behavior changes" (#6019) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What are you changing in this pull request and why? - Rename "Legacy behaviors" to "Behavior changes" — this is how we've been referring to them - Add definition of what we consider to be a behavior change, as well as examples of what does / doesn't constitute a behavior change --------- Co-authored-by: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../core-upgrade/07-upgrading-to-v1.8.md | 8 ++--- .../docs/docs/dbt-versions/release-notes.md | 6 ++-- .../global-configs/about-global-configs.md | 6 ++-- ...egacy-behaviors.md => behavior-changes.md} | 29 ++++++++++++++++--- .../reference/global-configs/project-flags.md | 2 +- website/sidebars.js | 2 +- website/vercel.json | 5 ++++ 7 files changed, 42 insertions(+), 16 deletions(-) rename website/docs/reference/global-configs/{legacy-behaviors.md => behavior-changes.md} (75%) diff --git a/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md b/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md index dd22329668c..9163047e7e0 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md +++ b/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md @@ -98,13 +98,13 @@ The ability for installed packages to override built-in materializations without ### Managing changes to legacy behaviors -dbt Core v1.8 has introduced flags for [managing changes to legacy behaviors](/reference/global-configs/legacy-behaviors). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`. +dbt Core v1.8 has introduced flags for [managing changes to legacy behaviors](/reference/global-configs/behavior-changes). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: -- (Mature, enabled by default) [Require explicit package overrides for builtin materializations](/reference/global-configs/legacy-behaviors#require_explicit_package_overrides_for_builtin_materializations) -- (Introduced, disabled by default) [Require resource names without spaces](https://docs.getdbt.com/reference/global-configs/legacy-behaviors#require_resource_names_without_spaces) -- (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](/reference/global-configs/legacy-behaviors#source_freshness_run_project_hooks) +- (Mature, enabled by default) [Require explicit package overrides for builtin materializations](/reference/global-configs/behavior-changes#require_explicit_package_overrides_for_builtin_materializations) +- (Introduced, disabled by default) [Require resource names without spaces](/reference/global-configs/behavior-changes#require_resource_names_without_spaces) +- (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](/reference/global-configs/behavior-changes#source_freshness_run_project_hooks) ## Quick hits diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 57aadba284d..1ab09b3839c 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -148,7 +148,7 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show -- **Behavior change:** Introduced the `require_resource_names_without_spaces` flag, opt-in and disabled by default. If set to `True`, dbt will raise an exception if it finds a resource name containing a space in your project or an installed package. This will become the default in a future version of dbt. Read [No spaces in resource names](/reference/global-configs/legacy-behaviors#no-spaces-in-resource-names) for more information. +- **Behavior change:** Introduced the `require_resource_names_without_spaces` flag, opt-in and disabled by default. If set to `True`, dbt will raise an exception if it finds a resource name containing a space in your project or an installed package. This will become the default in a future version of dbt. Read [No spaces in resource names](/reference/global-configs/behavior-changes#no-spaces-in-resource-names) for more information. ## April 2024 @@ -160,7 +160,7 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show -- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and "Versionless" dbt Cloud). Read [Package override for built-in materialization](/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization) for more information. +- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and "Versionless" dbt Cloud). Read [Package override for built-in materialization](/reference/global-configs/behavior-changes#package-override-for-built-in-materialization) for more information. **dbt Semantic Layer** - **New**: Use Saved selections to [save your query selections](/docs/cloud-integrations/semantic-layer/gsheets#using-saved-selections) within the [Google Sheets application](/docs/cloud-integrations/semantic-layer/gsheets). They can be made private or public and refresh upon loading. @@ -182,7 +182,7 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show - **Fix:** `dbt parse` no longer shows an error when you use a list of filters (instead of just a string filter) on a metric. - **Fix:** `join_to_timespine` now properly gets applied to conversion metric input measures. - **Fix:** Fixed an issue where exports in Redshift were not always committing to the DWH, which also had the side-effect of leaving table locks open. -- **Behavior change:** Introduced the `source_freshness_run_project_hooks` flag, opt-in and disabled by default. If set to `True`, dbt will include `on-run-*` project hooks in the `source freshness` command. This will become the default in a future version of dbt. Read [Project hooks with source freshness](/reference/global-configs/legacy-behaviors#project-hooks-with-source-freshness) for more information. +- **Behavior change:** Introduced the `source_freshness_run_project_hooks` flag, opt-in and disabled by default. If set to `True`, dbt will include `on-run-*` project hooks in the `source freshness` command. This will become the default in a future version of dbt. Read [Project hooks with source freshness](/reference/global-configs/behavior-changes#project-hooks-with-source-freshness) for more information. ## February 2024 diff --git a/website/docs/reference/global-configs/about-global-configs.md b/website/docs/reference/global-configs/about-global-configs.md index bbbe63ac439..3708b8c96be 100644 --- a/website/docs/reference/global-configs/about-global-configs.md +++ b/website/docs/reference/global-configs/about-global-configs.md @@ -16,7 +16,7 @@ There is a significant overlap between dbt's flags and dbt's command line option ### Setting flags There are multiple ways of setting flags, which depend on the use case: -- **[Project-level `flags` in `dbt_project.yml`](/reference/global-configs/project-flags):** Define version-controlled defaults for everyone running this project. Preserve [legacy behaviors](/reference/global-configs/legacy-behaviors) until their slated deprecation. +- **[Project-level `flags` in `dbt_project.yml`](/reference/global-configs/project-flags):** Define version-controlled defaults for everyone running this project. Also, opt in or opt out of [behavior changes](/reference/global-configs/behavior-changes) to manage your migration off legacy functionality. - **[Environment variables](/reference/global-configs/environment-variable-configs):** Define different behavior in different runtime environments (development vs. production vs. [continuous integration](/docs/deploy/continuous-integration), or different behavior for different users in development (based on personal preferences). - **[CLI options](/reference/global-configs/command-line-options):** Define behavior specific to _this invocation_. Supported for all dbt commands. @@ -41,7 +41,7 @@ dbt run --no-fail-fast # set to False There are two categories of exceptions: 1. **Flags setting file paths:** Flags for file paths that are relevant to runtime execution (for example, `--log-path` or `--state`) cannot be set in `dbt_project.yml`. To override defaults, pass CLI options or set environment variables (`DBT_LOG_PATH`, `DBT_STATE`). Flags that tell dbt where to find project resources (for example, `model-paths`) are set in `dbt_project.yml`, but as a top-level key, outside the `flags` dictionary; these configs are expected to be fully static and never vary based on the command or execution environment. -2. **Opt-in flags:** Flags opting into [legacy dbt behaviors](/reference/global-configs/legacy-behaviors) can _only_ be defined in `dbt_project.yml`. These are intended to be set in version control and migrated via pull/merge request. Their values should not diverge indefinitely across invocations, environments, or users. +2. **Opt-in flags:** Flags opting in or out of [behavior changes](/reference/global-configs/behavior-changes) can _only_ be defined in `dbt_project.yml`. These are intended to be set in version control and migrated via pull/merge request. Their values should not diverge indefinitely across invocations, environments, or users. ### Accessing flags @@ -84,7 +84,7 @@ Because the values of `flags` can differ across invocations, we strongly advise | [quiet](/reference/global-configs/logs#suppress-non-error-logs-in-output) | boolean | False | ❌ | `DBT_QUIET` | `--quiet` | ✅ | | [resource-type](/reference/global-configs/resource-type) (v1.8+) | string | None | ❌ | `DBT_RESOURCE_TYPES`

`DBT_EXCLUDE_RESOURCE_TYPES` | `--resource-type`

`--exclude-resource-type` | ✅ | | [send_anonymous_usage_stats](/reference/global-configs/usage-stats) | boolean | True | ✅ | `DBT_SEND_ANONYMOUS_USAGE_STATS` | `--send-anonymous-usage-stats`, `--no-send-anonymous-usage-stats` | ❌ | -| [source_freshness_run_project_hooks](/reference/global-configs/legacy-behaviors#source_freshness_run_project_hooks) | boolean | False | ✅ | ❌ | ❌ | ❌ | +| [source_freshness_run_project_hooks](/reference/global-configs/behavior-changes#source_freshness_run_project_hooks) | boolean | False | ✅ | ❌ | ❌ | ❌ | | [state](/reference/node-selection/defer) | path | none | ❌ | `DBT_STATE`, `DBT_DEFER_STATE` | `--state`, `--defer-state` | ❌ | | [static_parser](/reference/global-configs/parsing#static-parser) | boolean | True | ✅ | `DBT_STATIC_PARSER` | `--static-parser`, `--no-static-parser` | ❌ | | [store_failures](/reference/resource-configs/store_failures) | boolean | False | ✅ (as resource config) | `DBT_STORE_FAILURES` | `--store-failures`, `--no-store-failures` | ✅ | diff --git a/website/docs/reference/global-configs/legacy-behaviors.md b/website/docs/reference/global-configs/behavior-changes.md similarity index 75% rename from website/docs/reference/global-configs/legacy-behaviors.md rename to website/docs/reference/global-configs/behavior-changes.md index 1450fda1459..20f5722b944 100644 --- a/website/docs/reference/global-configs/legacy-behaviors.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -1,7 +1,7 @@ --- -title: "Legacy behaviors" -id: "legacy-behaviors" -sidebar: "Legacy behaviors" +title: "Behavior changes" +id: "behavior-changes" +sidebar: "Behavior changes" --- Most flags exist to configure runtime behaviors with multiple valid choices. The right choice may vary based on the environment, user preference, or the specific invocation. @@ -12,10 +12,31 @@ Another category of flags provides existing projects with a migration window for - Providing maintainability of dbt software. Every fork in behavior requires additional testing & cognitive overhead that slows future development. These flags exist to facilitate migration from "current" to "better," not to stick around forever. These flags go through three phases of development: -1. **Introduction (disabled by default):** dbt adds logic to support both 'old' + 'new' behaviors. The 'new' behavior is gated behind a flag, disabled by default, preserving the old behavior. +1. **Introduction (disabled by default):** dbt adds logic to support both 'old' and 'new' behaviors. The 'new' behavior is gated behind a flag, disabled by default, preserving the old behavior. 2. **Maturity (enabled by default):** The default value of the flag is switched, from `false` to `true`, enabling the new behavior by default. Users can preserve the 'old' behavior and opt out of the 'new' behavior by setting the flag to `false` in their projects. They may see deprecation warnings when they do so. 3. **Removal (generally enabled):** After marking the flag for deprecation, we remove it along with the 'old' behavior it supported from the dbt codebases. We aim to support most flags indefinitely, but we're not committed to supporting them forever. If we choose to remove a flag, we'll offer significant advance notice. +## What is a behavior change? + +The same dbt project code and the same dbt commands return one result before the behavior change, and they return a different result after the behavior change. + +Examples of behavior changes: +- dbt begins raising a validation _error_ that it didn't previously. +- dbt changes the signature of a built-in macro. Your project has a custom reimplementation of that macro. This could lead to errors, because your custom reimplementation will be passed arguments it cannot accept. +- A dbt adapter renames or removes a method that was previously available on the `{{ adapter }}` object in the dbt-Jinja context. +- dbt makes a breaking change to contracted metadata artifacts by deleting a required field, changing the name or type of an existing field, or removing the default value of an existing field ([README](https://github.com/dbt-labs/dbt-core/blob/37d382c8e768d1e72acd767e0afdcb1f0dc5e9c5/core/dbt/artifacts/README.md#breaking-changes)). +- dbt removes one of the fields from [structured logs](/reference/events-logging#structured-logging). + +The following are **not** behavior changes: +- Fixing a bug where the previous behavior was defective, undesirable, or undocumented. +- dbt begins raising a _warning_ that it didn't previously. +- dbt updates the language of human-friendly messages in log events. +- dbt makes a non-breaking change to contracted metadata artifacts by adding a new field with a default, or deleting a field with a default ([README](https://github.com/dbt-labs/dbt-core/blob/37d382c8e768d1e72acd767e0afdcb1f0dc5e9c5/core/dbt/artifacts/README.md#non-breaking-changes)). + +The vast majority of changes are not behavior changes. Because introducing these changes does not require any action on the part of users, they are included in continuous releases of dbt Cloud and patch releases of dbt Core. + +By contrast, behavior change migrations happen slowly, over the course of months, facilitated by behavior change flags. The flags are loosely coupled to the specific dbt runtime version. By setting flags, users have control over opting in (and later opting out) of these changes. + ## Behavior change flags These flags _must_ be set in the `flags` dictionary in `dbt_project.yml`. They configure behaviors closely tied to project code, which means they should be defined in version control and modified through pull or merge requests, with the same testing and peer review. diff --git a/website/docs/reference/global-configs/project-flags.md b/website/docs/reference/global-configs/project-flags.md index 896276d9735..cdbe3463b14 100644 --- a/website/docs/reference/global-configs/project-flags.md +++ b/website/docs/reference/global-configs/project-flags.md @@ -17,7 +17,7 @@ flags: Reference the [table of all flags](/reference/global-configs/about-global-configs#available-flags) to see which global configs are available for setting in [`dbt_project.yml`](/reference/dbt_project.yml). -The `flags` dictionary is the _only_ place you can opt out of [behavior changes](/reference/global-configs/legacy-behaviors), while the legacy behavior is still supported. +The `flags` dictionary is the _only_ place you can opt out of [behavior changes](/reference/global-configs/behavior-changes), while the legacy behavior is still supported. diff --git a/website/sidebars.js b/website/sidebars.js index b3b752990dd..69ba286228d 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -1073,6 +1073,7 @@ const sidebarSettings = { }, items: [ "reference/global-configs/about-global-configs", + "reference/global-configs/behavior-changes", { type: "category", label: "Setting flags", @@ -1091,7 +1092,6 @@ const sidebarSettings = { "reference/global-configs/failing-fast", "reference/global-configs/indirect-selection", "reference/global-configs/json-artifacts", - "reference/global-configs/legacy-behaviors", "reference/global-configs/parsing", "reference/global-configs/print-output", "reference/global-configs/record-timing-info", diff --git a/website/vercel.json b/website/vercel.json index 8fdf311e72f..f79fc959187 100644 --- a/website/vercel.json +++ b/website/vercel.json @@ -599,6 +599,11 @@ "destination": "/reference/global-configs/command-line-options", "permanent": true }, + { + "source": "/reference/global-configs/legacy-behaviors", + "destination": "/reference/global-configs/behavior-changes", + "permanent": true + }, { "source": "/reference/global-configs/yaml-configurations", "destination": "/reference/global-configs/project-flags", From 1059f0a39eb3a691bd099b7bb6407659da36d5e9 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 5 Sep 2024 20:15:51 -0400 Subject: [PATCH 14/24] Adding guidance for custom display (#6022) ## What are you changing in this pull request and why? Adding clarifying text about customDisplay ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --- website/dbt-versions.js | 2 ++ 1 file changed, 2 insertions(+) diff --git a/website/dbt-versions.js b/website/dbt-versions.js index 0f322100a45..871c3ce601e 100644 --- a/website/dbt-versions.js +++ b/website/dbt-versions.js @@ -10,6 +10,8 @@ * @property {string} EOLDate "End of Life" date which is used to show the EOL banner * @property {boolean} isPrerelease Boolean used for showing the prerelease banner * @property {string} customDisplay Allows setting a custom display name for the current version + * + * customDisplay for dbt Cloud should be a version ahead of latest dbt Core release (GA or beta). */ exports.versions = [ { From 6a1a7bc2b5a18f998f7b8186fbf192bf7da50fb2 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 6 Sep 2024 10:04:22 +0100 Subject: [PATCH 15/24] needs PM review - update data health tile (#6018) After PM sync, updated [data health tile doc](https://docs.getdbt.com/docs/collaborate/data-tile) to make it clearer users can embed data health tile url or iframe in any tool. turn tableau into example. [ ] Needs PM review --------- Co-authored-by: nataliefiann <120089939+nataliefiann@users.noreply.github.com> --- website/docs/docs/collaborate/data-tile.md | 34 +++++++++++++--------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/website/docs/docs/collaborate/data-tile.md b/website/docs/docs/collaborate/data-tile.md index f40f21ebe18..446922acb92 100644 --- a/website/docs/docs/collaborate/data-tile.md +++ b/website/docs/docs/collaborate/data-tile.md @@ -9,9 +9,11 @@ image: /img/docs/collaborate/dbt-explorer/data-tile-pass.jpg # Embed data health tile in dashboards With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. This trust signal allows teams to immediately go back into Explorer to see more details and investigate issues. + :::info Available in beta Data health tile is currently available in open beta. ::: + The data health tile: - Distills trust signals for data consumers. @@ -19,6 +21,8 @@ The data health tile: - Provides richer information and makes it easier to debug. - Revamps the existing, [job-based tiles](#job-based-data-health). +Data health tiles rely on [exposures](/docs/build/exposures) to surface trust signals in your dashboards. When you configure exposures in your dbt project, you are explicitly defining how specific outputs—like dashboards or reports—depend on your data models. + ## Prerequisites @@ -34,43 +38,45 @@ First, be sure to enable [source freshness](/docs/deploy/source-freshness) in 1. Navigate to dbt Explorer by clicking on the **Explore** link in the navigation. 2. In the main **Overview** page, go to the left navigation. -3. Under the **Resources** tab, click on **Exposures** to view the exposures list. +3. Under the **Resources** tab, click on **Exposures** to view the [exposures](/docs/build/exposures) list. 4. Select a dashboard exposure and go to the **General** tab to view the data health information. -5. In this tab, you’ll see: - - Data health status: Data freshness passed, Data quality passed, Data may be stale, Data quality degraded - - Name of the exposure. +5. In this tab, you’ll see: + - Name of the exposure. + - Data health status: Data freshness passed, Data quality passed, Data may be stale, Data quality degraded. - Resource type (model, source, and so on). - Dashboard status: Failure, Pass, Stale. - You can also see the last check completed, the last check time, and the last check duration. -6. You can also click the **Open Dashboard** button on the upper right to immediately view this in your analytics tool. +6. You can click the **Open Dashboard** button on the upper right to immediately view this in your analytics tool. ## Embed in your dashboard -Once you’ve navigated to the auto-exposure in dbt Explorer, you’ll need to set up your dashboard status tile and [service token](/docs/dbt-cloud-apis/service-tokens): +Once you’ve navigated to the auto-exposure in dbt Explorer, you’ll need to set up your data health tile and [service token](/docs/dbt-cloud-apis/service-tokens). You can embed data health tile to any analytics tool that supports URL or iFrame embedding. + +Follow these steps to set up your data health tile: 1. Go to **Account settings** in dbt Cloud. 2. Select **API tokens** in the left sidebar and then **Service tokens**. 3. Click on **Create service token** and give it a name. -4. Select the [**Metadata Only** permission](/docs/dbt-cloud-apis/service-tokens). This token will be used to embed the exposure tile in your dashboard in the later steps. +4. Select the [**Metadata Only**](/docs/dbt-cloud-apis/service-tokens) permission. This token will be used to embed the tile in your dashboard in the later steps. -5. Copy the **Metadata Only token** and save it in a secure location. You'll need it token in the next steps. +5. Copy the **Metadata Only** token and save it in a secure location. You'll need it token in the next steps. 6. Navigate back to dbt Explorer and select an exposure. 7. Below the **Data health** section, expand on the toggle for instructions on how to embed the exposure tile (if you're an account admin with develop permissions). 8. In the expanded toggle, you'll see a text field where you can paste your **Metadata Only token**. -9. Once you’ve pasted your token, you can select either **URL** or **iFrame** depending on which you need to install into your dashboard. +9. Once you’ve pasted your token, you can select either **URL** or **iFrame** depending on which you need to add to your dashboard. If your analytics tool supports iFrames, you can embed the dashboard tile within it. -### Embed data health tile in Tableau -To embed the data health tile in Tableau, follow these steps: +#### Tableau example +Here’s an example with Tableau, where you can embed the iFrame in a web page object: -1. Ensure you've copied the embed iFrame content in dbt Explorer. -2. For the revamped environment-based exposure tile you can insert these fields into the following iFrame, and then embed them with your dashboard. This is the iFrame that is available from the **Exposure details** page in dbt Explorer. +- Ensure you've copied the embed iFrame snippet from the dbt Explorer **Data health** section. +- **For the revamped environment-based exposure tile** — Insert the following fields into the following iFrame. Then embed them with your dashboard. This is the iFrame available from the **Exposure details** page in dbt Explorer. `