From d8f0ef2d5ad1cd42563c38ebca2b7eb7bfc356a9 Mon Sep 17 00:00:00 2001 From: Jeremy Cohen Date: Thu, 2 May 2024 03:11:30 +0200 Subject: [PATCH] Revise docs for behavior change flags (#5389) ## What are you changing in this pull request and why? **Legacy behaviors:** - Update explanations - Rename behavior change flags - Add table with dates for Intro / Maturity **v1.8 guide:** - Reorganize + update with links to the legacy behavior page **Depends on changes tracked here:** - https://github.com/dbt-labs/dbt-core/issues/10062 --------- Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- .../docs/dbt-versions/2024-release-notes.md | 9 +- .../core-upgrade/01-upgrading-to-v1.8.md | 26 +++--- .../global-configs/legacy-behaviors.md | 83 ++++++++++++++++--- 3 files changed, 95 insertions(+), 23 deletions(-) diff --git a/website/docs/docs/dbt-versions/2024-release-notes.md b/website/docs/docs/dbt-versions/2024-release-notes.md index a381d7f49de..31130cee886 100644 --- a/website/docs/docs/dbt-versions/2024-release-notes.md +++ b/website/docs/docs/dbt-versions/2024-release-notes.md @@ -9,17 +9,23 @@ pagination_prev: null dbt Cloud release notes for 2024. +## May 2024 + +- **Behavior change:** Introduced the `require_resource_names_without_spaces` flag, opt-in and disabled by default. If set to `True`, dbt will raise an exception if it finds a resource name containing a space in your project or an installed package. This will become the default in a future version of dbt. Read [No spaces in resource names](/reference/global-configs/legacy-behaviors#no-spaces-in-resource-names) for more information. + ## April 2024 - - You can now set up a continuous deployment (CD) workflow for your projects natively in dbt Cloud. You can now access a beta release of [Merge jobs](/docs/deploy/merge-jobs), which is a new [job type](/docs/deploy/jobs), that enables you to trigger dbt job runs as soon as changes (via Git pull requests) merge into production. + You can now set up a continuous deployment (CD) workflow for your projects natively in dbt Cloud. You can now access a beta release of [Merge jobs](/docs/deploy/merge-jobs), which is a new [job type](/docs/deploy/jobs), that enables you to trigger dbt job runs as soon as changes (via Git pull requests) merge into production. + +- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and dbt Cloud "Keep on latest version"). Read [Package override for built-in materialization](/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization) for more information. **dbt Semantic Layer** - **New**: Use Saved selections to [save your query selections](/docs/use-dbt-semantic-layer/gsheets#using-saved-selections) within the [Google Sheets application](/docs/use-dbt-semantic-layer/gsheets). They can be made private or public and refresh upon loading. @@ -41,6 +47,7 @@ dbt Cloud release notes for 2024. - **Fix:** `dbt parse` no longer shows an error when you use a list of filters (instead of just a string filter) on a metric. - **Fix:** `join_to_timespine` now properly gets applied to conversion metric input measures. - **Fix:** Fixed an issue where exports in Redshift were not always committing to the DWH, which also had the side-effect of leaving table locks open. +- **Behavior change:** Introduced the `source_freshness_run_project_hooks` flag, opt-in and disabled by default. If set to `True`, dbt will include `on-run-*` project hooks in the `source freshness` command. This will become the default in a future version of dbt. Read [Project hooks with source freshness](/reference/global-configs/legacy-behaviors#project-hooks-with-source-freshness) for more information. ## February 2024 diff --git a/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md b/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md index caf80840def..bf87434a646 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md +++ b/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md @@ -4,7 +4,7 @@ id: upgrading-to-v1.8 description: New features and changes in dbt Core v1.8 displayed_sidebar: "docs" --- - + ## Resources - Changelog (coming soon) @@ -31,9 +31,9 @@ Features and functionality new in dbt v1.8. ### New dbt Core adapter installation procedure -Before v1.8, when you installed an adapter, you would automatically get `dbt-core` installed along with the adapter package (if you didn’t already have an existing, compatible version of dbt-core). +Before dbt Core v1.8, whenever you would `pip install` a data warehouse adapter for dbt, `pip` would automatically install `dbt-core` alongside it. The dbt adapter directly depended on components of `dbt-core`, and `dbt-core` depended on the adapter for execution. This bidirectional dependency made it difficult to develop adapters independent of `dbt-core`. -Beginning in v1.8, the [dbt adapters and dbt Core have been decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87). As a result, you must install _both_ dbt-core and the desired adapter. A new `pip` installation needs to look like this: +Beginning in v1.8, [`dbt-core` and adapters are decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87). Going forward, your installations should explicitly include _both_ `dbt-core` _and_ the desired adapter. The new `pip` installation command should look like this: ```shell pip install dbt-core dbt-ADAPTER_NAME @@ -44,6 +44,8 @@ For example, you would use the following command if you use Snowflake: pip install dbt-core dbt-snowflake ``` +For the time being, we have maintained install-time dependencies to avoid breaking existing scripts in surprising ways; `pip install dbt-snowflake` will continue to install the latest versions of both `dbt-core` and 1dbt-snowflake`. Given that we may remove this implicit dependency in future versions, we strongly encourage you to update install scripts **now**. + ### Unit Tests Historically, dbt's test coverage was confined to [“data” tests](/docs/build/data-tests), assessing the quality of input data or resulting datasets' structure. @@ -78,23 +80,23 @@ models: ``` -### The `--empty` flag +#### The `--empty` flag The [`run`](/reference/commands/run#the-`--empty`-flag) and [`build`](/reference/commands/build#the---empty-flag) commands now support the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. -### Spaces in dbt model names +### Managing changes to legacy behaviors + +dbt Core v1.8 has introduced a flags has been created for [managing changes to legacy behaviors](/reference/global-configs/legacy-behaviors). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`. -We will begin deprecating support for spaces in dbt model names in v1.8 (raising a warning) before removing support entirely in v1.9 (raising an error). Reasons for removing spaces in model names include: -- Spaces in a model name make it impossible to `--select` the model name because the argument gets split into pieces over spaces very early in the pipeline. -- Most warehouses do not accept a table, or other object, with a space in its name. +You can read more about each of these behavior changes in the following links: -To upgrade, replace any spaces in the model file name with an underscore and update any associated YAML that contains the model name to match. You can keep spaces in the database table name by configuring a [custom `alias`](/docs/build/custom-aliases#usage). +- (Mature, enabled by default) [Require explicit package overrides for builtin materializations](/reference/global-configs/legacy-behaviors#require_explicit_package_overrides_for_builtin_materializations) +- (Introduced, disabled by default) [Require resource names without spaces](https://docs.getdbt.com/reference/global-configs/legacy-behaviors#require_resource_names_without_spaces) +- (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](/reference/global-configs/legacy-behaviors#source_freshness_run_project_hooks) ## Quick hits -- [Global config flags](/reference/global-configs/about-global-configs) are deprecated from the [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file and should be moved to the [`dbt_project.yml`](/reference/dbt_project.yml). -- A new subcategory of flags has been created for [legacy behaviors](/reference/global-configs/legacy-behaviors). -- The [`--indirect_selection`](/reference/global-configs/indirect-selection) flag used with `dbt test` or `dbt build` configures which tests to run for the nodes you specify. +- Custom defaults of [global config flags](/reference/global-configs/about-global-configs) should be set in the `flags` dictionary in [`dbt_project.yml`](/reference/dbt_project.yml), instead of in [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml). Support for `profiles.yml` has been deprecated. - New CLI flag [`--resource-type`/`--exclude-resource-type`](/reference/global-configs/resource-type) for including/excluding resources from dbt `build`, `run`, and `clone`. - To improve performance, dbt now issues a single (batch) query when calculating `source freshness` through metadata, instead of executing a query per source. - Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. diff --git a/website/docs/reference/global-configs/legacy-behaviors.md b/website/docs/reference/global-configs/legacy-behaviors.md index d1ef606b149..cd898d55187 100644 --- a/website/docs/reference/global-configs/legacy-behaviors.md +++ b/website/docs/reference/global-configs/legacy-behaviors.md @@ -12,15 +12,82 @@ Another category of flags provides existing projects with a migration window for - Providing maintainability of dbt software. Every fork in behavior requires additional testing & cognitive overhead that slows future development. These flags exist to facilitate migration from "current" to "better," not to stick around forever. These flags go through three phases of development: -1. **Introduction:** The flag is introduced, and the logic added within dbt to support both 'old' + 'new' behaviours. -2. **Maturity:** The default value of the flag is switched, from 'old' to 'new.' -3. **Deprecation:** The flag, having been marked for deprecation (with user warnings), is removed. The logic to support the ‘old’ behaviour is removed from dbt codebases. +1. **Introduction (disabled by default):** dbt adds logic to support both 'old' + 'new' behaviors. The 'new' behavior is gated behind a flag, disabled by default, preserving the old behavior. +2. **Maturity (enabled by default):** The default value of the flag is switched, from `false` to `true`, enabling the new behavior by default. Users can preserve the 'old' behavior and opt out of the 'new' behavior by setting the flag to `false` in their projects. They may see deprecation warnings when they do so. +3. **Removal (generally enabled):** After marking the flag for deprecation, we remove it along with the 'old' behavior it supported from the dbt codebases. We aim to support most flags indefinitely, but we're not committed to supporting them forever. If we choose to remove a flag, we'll offer significant advance notice. -These flags can _only_ be set in `dbt_project.yml`. They configure behaviors closely tied to project code. They should be defined in version control, and changed via pull/merge request, with the same testing and peer review. +## Behavior change flags -## Behaviors +These flags _must_ be set in the `flags` dictionary in `dbt_project.yml`. They configure behaviors closely tied to project code, which means they should be defined in version control and modified through pull or merge requests, with the same testing and peer review. -### `source_freshness_run_project_hooks` +The following example displays the current flags and their current default values in the latest dbt Cloud and dbt Core versions. To opt out of a specific behavior change, set the values of the flag to `False` in `dbt_project.yml`. You'll continue to see warnings for legacy behaviors that you have opted out of explicitly until you either resolve them (switch the flag to `True`) or choose to silence the warnings using the `warn_error_options.silence` flag. + + + +```yml +flags: + require_explicit_package_overrides_for_builtin_materializations: False + require_model_names_without_spaces: False + source_freshness_run_project_hooks: False +``` + + + +When we use dbt Cloud in the following table, we're referring to "[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)." + +| Flag | dbt Cloud: Intro | dbt Cloud: Maturity | dbt Core: Intro | dbt Core: Maturity | +|-----------------------------------------------------------------|------------------|---------------------|-----------------|--------------------| +| require_explicit_package_overrides_for_builtin_materializations | 2024.04.141 | 2024.05.TBD | 1.6.14, 1.7.14 | 1.8.0 | +| require_resource_names_without_spaces | 2024.05.TBD | 2024.06.TBD | 1.8.0 | 1.9.0 | +| source_freshness_run_project_hooks | 2024.03.61 | 2024.06.TBD | 1.8.0 | 1.9.0 | + + + +### Package override for built-in materialization + +Setting the `require_explicit_package_overrides_for_builtin_materializations` flag to `True` prevents this automatic override. + +We have deprecated the behavior where installed packages could override built-in materializations without your explicit opt-in. When this flag is set to `True`, a materialization defined in a package that matches the name of a built-in materialization will no longer be included in the search and resolution order. Unlike macros, materializations don't use the `search_order` defined in the project `dispatch` config. + +The built-in materializations are `'view'`, `'table'`, `'incremental'`, `'materialized_view'` for models as well as `'test'`, `'unit'`, `'snapshot'`, `'seed'`, and `'clone'`. + +You can still explicitly override built-in materializations, in favor of a materialization defined in a package, by reimplementing the built-in materialization in your root project and wrapping the package implementation. + + + +```sql +{% materialization view, snowflake %} + {{ return(my_installed_package_name.materialization_view_snowflake()) }} +{% endmaterialization %} +``` + + + +In the future, we may extend the project-level [`dispatch` configuration](/reference/project-configs/dispatch-config) to support a list of authorized packages for overriding built-in materialization. + + + +The following flags were introduced in a future version of dbt Core. If you're still using an older version, then you have the legacy behavior by default (when each flag is `False`). + + + +### No spaces in resource names + +The `require_resource_names_without_spaces` flag enforces using resource names without spaces. + +The names of dbt resources (models, sources, etc) should contain letters, numbers, and underscores. We highly discourage the use of other characters, especially spaces. To that end, we have deprecated support for spaces in resource names. When the `require_resource_names_without_spaces` flag is set to `True`, dbt will raise an exception (instead of a deprecation warning) if it detects a space in a resource name. + + + +```sql +-- This model file should be renamed to model_name_with_underscores.sql +``` + + + +### Project hooks with source freshness + +Set the `source_freshness_run_project_hooks` flag to `True` to include "project hooks" ([`on-run-start` / `on-run-end`](/reference/project-configs/on-run-start-on-run-end)) in the `dbt source freshness` command execution. If you have specific project [`on-run-start` / `on-run-end`](/reference/project-configs/on-run-start-on-run-end) hooks that should not run before/after `source freshness` command, you can add a conditional check to those hooks: @@ -31,7 +98,3 @@ on-run-start: - '{{ ... if flags.WHICH != 'freshness' }}' ``` - -You can configure the `dbt source freshness` command to include on-run-start / on-run-end hooks in v1.8 with a planned removal in v1.9. The default value is `True`. - -In the meantime, you can set `source_freshness_run_project_hooks: False` to preserve the legacy behavior.