diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 0534dd916cb..193e989b875 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -19,5 +19,5 @@ Uncomment when publishing docs for a prerelease version of dbt: Adding or removing pages (delete if not applicable): - [ ] Add/remove page in `website/sidebars.js` - [ ] Provide a unique filename for new pages -- [ ] Add an entry for deleted pages in `website/static/_redirects` +- [ ] Add an entry for deleted pages in `website/vercel.json` - [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages diff --git a/website/dbt-versions.js b/website/dbt-versions.js index 0bc4d778f8a..85d96f1156f 100644 --- a/website/dbt-versions.js +++ b/website/dbt-versions.js @@ -30,6 +30,10 @@ exports.versions = [ ] exports.versionedPages = [ + { + "page": "reference/global-configs/indirect-selection", + "firstVersion": "1.8", + }, { "page": "reference/resource-configs/store_failures_as", "firstVersion": "1.7", diff --git a/website/docs/docs/core/connect-data-platform/profiles.yml.md b/website/docs/docs/core/connect-data-platform/profiles.yml.md index 18ca4e86f68..dacdc23fcd8 100644 --- a/website/docs/docs/core/connect-data-platform/profiles.yml.md +++ b/website/docs/docs/core/connect-data-platform/profiles.yml.md @@ -13,6 +13,14 @@ If you're using dbt Cloud, you can [connect to your data platform](/docs/cloud/c This section identifies the parts of your `profiles.yml` that aren't specific to a particular data platform. For specific connection details, refer to the relevant page for your data platform. + + +:::warning Global configs + +Starting in dbt v1.8, global configs have been deprecated from the `profiles.yml` file and should be configured in the [`dbt_project.yml`](/reference/dbt_project.yml) file instead. + +::: + ```yml @@ -28,6 +36,7 @@ This section identifies the parts of your `profiles.yml` that aren't specific to [debug](/reference/global-configs/logs#log-level): [version_check](/reference/global-configs/version-compatibility): [fail_fast](/reference/global-configs/failing-fast): + [indirect_selection](/reference/global-configs/indirect-selection): [use_experimental_parser](/reference/global-configs/parsing): [static_parser](/reference/global-configs/parsing): [cache_selected_only](/reference/global-configs/cache): @@ -54,6 +63,37 @@ This section identifies the parts of your `profiles.yml` that aren't specific to + + + + + + +```yml + +: + target: # this is the default target + outputs: + : + type: + schema: + threads: + + ### database-specific connection details + ... + + : # additional targets + ... + +: # additional profiles + ... + +``` + + + + + ## User config -You can set default values of global configs for all projects that you run using your local machine. See the docs on [global configs](/reference/global-configs/about-global-configs) for details. +You can set default values of global configs for all projects that you run using your local machine. Refer to [About global configs](/reference/global-configs/about-global-configs) for details. diff --git a/website/docs/docs/dbt-cloud-apis/sl-manifest.md b/website/docs/docs/dbt-cloud-apis/sl-manifest.md index ef4e330a38e..dcbb87228bb 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-manifest.md +++ b/website/docs/docs/dbt-cloud-apis/sl-manifest.md @@ -19,7 +19,7 @@ dbt creates an [artifact](/reference/artifacts/dbt-artifacts) file called the _S By using the semantic manifest produced by dbt Core, MetricFlow will instantiate a data flow plan and generate SQL from Semantic Layer query requests. It's a valuable reference that you can use to understand the structure and details of your data models. -Similar to the [`manifest.json` file](/reference/artifacts/manifest-json), the `semantic_manifest.json` file also lives in the [target directory](/reference/project-configs/target-path) of your dbt project where dbt stores various artifacts (such as compiled models and tests) generated during the execution of your project. +Similar to the [`manifest.json` file](/reference/artifacts/manifest-json), the `semantic_manifest.json` file also lives in the [target directory](/reference/global-configs/json-artifacts) of your dbt project where dbt stores various artifacts (such as compiled models and tests) generated during the execution of your project. ## How it's produced diff --git a/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.2.md b/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.2.md index dd9cb4742ea..a7942fd22af 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.2.md +++ b/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.2.md @@ -33,7 +33,7 @@ See GitHub discussion [dbt-labs/dbt-core#5468](https://github.com/dbt-labs/dbt-c - **[Metrics](/docs/build/build-metrics-intro)** now support an `expression` type (metrics-on-metrics), as well as a `metric()` function to use when referencing metrics from within models, macros, or `expression`-type metrics. For more information on how to use expression metrics, check out the [**`dbt_metrics` package**](https://github.com/dbt-labs/dbt_metrics) - **[dbt-Jinja functions](/reference/dbt-jinja-functions)** now include the [`itertools` Python module](/reference/dbt-jinja-functions/modules#itertools), as well as the [set](/reference/dbt-jinja-functions/set) and [zip](/reference/dbt-jinja-functions/zip) functions. - **[Node selection](/reference/node-selection/syntax)** includes a [file selection method](/reference/node-selection/methods#the-file-method) (`-s model.sql`), and [yaml selector](/reference/node-selection/yaml-selectors) inheritance. -- **[Global configs](/reference/global-configs/about-global-configs)** now include CLI flag and environment variable settings for [`target-path`](/reference/project-configs/target-path) and [`log-path`](/reference/project-configs/log-path), which can be used to override the values set in `dbt_project.yml` +- **[Global configs](/reference/global-configs/about-global-configs)** now include CLI flag and environment variable settings for [`target-path`](/reference/global-configs/json-artifacts) and [`log-path`](/reference/global-configs/logs), which can be used to override the values set in `dbt_project.yml` ### Specific adapters diff --git a/website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.0.md b/website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.0.md index 013d70b311c..f3d330dcc8e 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.0.md +++ b/website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.0.md @@ -70,7 +70,7 @@ Several under-the-hood changes from past minor versions, tagged with deprecation - Add [metrics](/docs/build/build-metrics-intro), a new node type - [Generic tests](/best-practices/writing-custom-generic-tests) can be defined in `tests/generic` (new), in addition to `macros/` (as before) - [Parsing](/reference/parsing): partial parsing and static parsing have been turned on by default. -- [Global configs](/reference/global-configs/about-global-configs) have been standardized. Related updates to [global CLI flags](/reference/global-cli-flags) and [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml). +- [Global configs](/reference/global-configs/about-global-configs) have been standardized. Related updates to [global CLI flags](/reference/global-configs/about-global-configs) and [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml). - [The `init` command](/reference/commands/init) has a whole new look and feel. It's no longer just for first-time users. - Add `result:` subselectors for smarter reruns when dbt models have errors and tests fail. See examples: [Pro-tips for Workflows](/best-practices/best-practice-workflows#pro-tips-for-workflows) - Secret-prefixed [env vars](/reference/dbt-jinja-functions/env_var) are now allowed only in `profiles.yml` + `packages.yml` diff --git a/website/docs/reference/artifacts/dbt-artifacts.md b/website/docs/reference/artifacts/dbt-artifacts.md index 58074c8b426..369c07d472c 100644 --- a/website/docs/reference/artifacts/dbt-artifacts.md +++ b/website/docs/reference/artifacts/dbt-artifacts.md @@ -30,7 +30,7 @@ Most dbt commands (and corresponding RPC methods) produce artifacts: ## Where are artifacts produced? -By default, artifacts are written to the `/target` directory of your dbt project. You can configure the location using the [`target-path`](/reference/project-configs/target-path#configuration). +By default, artifacts are written to the `/target` directory of your dbt project. You can configure the location using the [`target-path` flag](/reference/global-configs/json-artifacts). ## Common metadata diff --git a/website/docs/reference/dbt_project.yml.md b/website/docs/reference/dbt_project.yml.md index ae911200b40..446878c408a 100644 --- a/website/docs/reference/dbt_project.yml.md +++ b/website/docs/reference/dbt_project.yml.md @@ -42,8 +42,8 @@ The following example is a list of all available configurations in the `dbt_proj [docs-paths](/reference/project-configs/docs-paths): [directorypath] [asset-paths](/reference/project-configs/asset-paths): [directorypath] -[target-path](/reference/project-configs/target-path): directorypath -[log-path](/reference/project-configs/log-path): directorypath +[target-path](/reference/global-configs/json-artifacts): directorypath +[log-path](/reference/global-configs/logs): directorypath [packages-install-path](/reference/project-configs/packages-install-path): directorypath [clean-targets](/reference/project-configs/clean-targets): [directorypath] @@ -52,6 +52,9 @@ The following example is a list of all available configurations in the `dbt_proj [require-dbt-version](/reference/project-configs/require-dbt-version): version-range | [version-range] +[flags](/reference/global-configs/project-flags): + [](/reference/global-configs/project-flags) + [dbt-cloud](/docs/cloud/cloud-cli-installation): [project-id](/docs/cloud/configure-cloud-cli#configure-the-dbt-cloud-cli): project_id # Required [defer-env-id](/docs/cloud/about-cloud-develop-defer#defer-in-dbt-cloud-cli): environment_id # Optional @@ -120,8 +123,8 @@ vars: [docs-paths](/reference/project-configs/docs-paths): [directorypath] [asset-paths](/reference/project-configs/asset-paths): [directorypath] -[target-path](/reference/project-configs/target-path): directorypath -[log-path](/reference/project-configs/log-path): directorypath +[target-path](/reference/global-configs/json-artifacts): directorypath +[log-path](/reference/global-configs/logs): directorypath [packages-install-path](/reference/project-configs/packages-install-path): directorypath [clean-targets](/reference/project-configs/clean-targets): [directorypath] @@ -193,8 +196,8 @@ vars: [docs-paths](/reference/project-configs/docs-paths): [directorypath] [asset-paths](/reference/project-configs/asset-paths): [directorypath] -[target-path](/reference/project-configs/target-path): directorypath -[log-path](/reference/project-configs/log-path): directorypath +[target-path](/reference/global-configs/json-artifacts): directorypath +[log-path](/reference/global-configs/logs): directorypath [packages-install-path](/reference/project-configs/packages-install-path): directorypath [clean-targets](/reference/project-configs/clean-targets): [directorypath] diff --git a/website/docs/reference/events-logging.md b/website/docs/reference/events-logging.md index e3349530cb7..603a60f2640 100644 --- a/website/docs/reference/events-logging.md +++ b/website/docs/reference/events-logging.md @@ -4,7 +4,7 @@ title: "Events and logs" As dbt runs, it generates events. The most common way to see those events is as log messages, written in real time to two places: - The command line terminal (`stdout`), to provide interactive feedback while running dbt. -- The debug log file (`logs/dbt.log`), to enable detailed [debugging of errors](/guides/debug-errors) when they occur. The text-formatted log messages in this file include all `DEBUG`-level events, as well as contextual information, such as log level and thread name. The location of this file can be configured via [the `log_path` config](/reference/project-configs/log-path). +- The debug log file (`logs/dbt.log`), to enable detailed [debugging of errors](/guides/debug-errors) when they occur. The text-formatted log messages in this file include all `DEBUG`-level events, as well as contextual information, such as log level and thread name. The location of this file can be configured via [the `log-path` flag](/reference/global-configs/logs). diff --git a/website/docs/reference/global-cli-flags.md b/website/docs/reference/global-cli-flags.md deleted file mode 100644 index 88e4f6417b3..00000000000 --- a/website/docs/reference/global-cli-flags.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: "Global CLI flags" -id: "global-cli-flags" ---- - -dbt commands, such as `run` or `test`, support their own command-specific CLI flags. In addition, the dbt command itself supports "global" flags applicable to *all* subcommands. - -As of v1.0, the vast majority of global CLI flags map to [global configs](/reference/global-configs/about-global-configs), which can also be configured via environment variables or in the `profiles.yml`. - -The `--version` and `--record-timing-info` flags remain available to the CLI only. - - -## Record timing info - -The `-r` or `--record-timing-info` flag saves performance profiling information to a file. This file can be visualized with `snakeviz` to understand the performance characteristics of a dbt invocation - - - -```text -$ dbt -r timing.txt run -... - -$ snakeviz timing.txt -``` - - diff --git a/website/docs/reference/global-configs/about-global-configs.md b/website/docs/reference/global-configs/about-global-configs.md index 9d1691812b5..39875f04dbe 100644 --- a/website/docs/reference/global-configs/about-global-configs.md +++ b/website/docs/reference/global-configs/about-global-configs.md @@ -1,18 +1,98 @@ --- -title: "About global configs" +title: "About flags (global configs)" id: "about-global-configs" -sidebar: "About global configs" +sidebar: "About flags (global configs)" +pagination_next: null --- -Global configs enable you to fine-tune _how_ dbt runs projects on your machine—whether your personal laptop, an orchestration tool running remotely, or (in some cases) dbt Cloud. In general, they differ from most [project configs](/reference/dbt_project.yml) and [resource configs](/reference/configs-and-properties), which tell dbt _what_ to run. +In dbt, "flags" (also called "global configs") are configurations for fine-tuning _how_ dbt runs your project. They differ from [resource-specific configs](/reference/configs-and-properties) that tell dbt about _what_ to run. -Global configs control things like the visual output of logs, the manner in which dbt parses your project, and what to do when dbt finds a version mismatch or a failing model. These configs are "global" because they are available for all dbt commands, and because they can be set for all projects running on the same machine or in the same environment. +Flags control things like the visual output of logs, whether to treat specific warning messages as errors, or whether to "fail fast" after encountering the first error. Flags are "global" configs because they are available for all dbt commands and they can be set in multiple places. -### Global config precedence +There is a significant overlap between dbt's flags and dbt's command line options, but there are differences: +- Certain flags can only be set in `dbt_project.yml` and cannot be overridden for specific invocations via CLI options. +- If a CLI option is supported by specific commands, rather than supported by all commands ("global"), it is generally not considered to be a "flag". -Starting in v1.0, you can set global configs in three places. dbt will evaluate the configs in the following order: -1. [user config](https://docs.getdbt.com/reference/global-configs/yaml-configurations) -1. [environment variable](https://docs.getdbt.com/reference/global-configs/environment-variable-configs) -1. [CLI flag](https://docs.getdbt.com/reference/global-configs/command-line-flags) +### Setting flags -Each config is prioritized over the previous one. For example, if all three are provided, then the CLI flag takes precedence. +There are multiple ways of setting flags, which depend on the use case: +- **[Project-level `flags` in `dbt_project.yml`](/reference/global-configs/project-flags):** Define version-controlled defaults for everyone running this project. Preserve [legacy behaviors](/reference/global-configs/legacy-behaviors) until their slated deprecation. +- **[Environment variables](/reference/global-configs/environment-variable-configs):** Define different behavior in different runtime environments (development vs. production vs. [continuous integration](/docs/deploy/continuous-integration), or different behavior for different users in development (based on personal preferences). +- **[CLI options](/reference/global-configs/command-line-options):** Define behavior specific to _this invocation_. Supported for all dbt commands. + +The most specific setting "wins." If you set the same flag in all three places, the CLI option will take precedence, followed by the environment variable, and finally, the value in `dbt_project.yml`. If you set the flag in none of those places, it will use the default value defined within dbt. + +Most flags can be set in all three places: +```yaml +# dbt_project.yml +flags: + # set default for running this project -- anywhere, anytime, by anyone + fail_fast: true +``` +```bash +# set this environment variable to 'True' (bash syntax) +export DBT_FAIL_FAST=1 +dbt run +``` +```bash +dbt run --fail-fast # set to True for this specific invocation +dbt run --no-fail-fast # set to False +``` + +There are two categories of exceptions: +1. **Flags setting file paths:** Flags for file paths that are relevant to runtime execution (for example, `--log-path` or `--state`) cannot be set in `dbt_project.yml`. To override defaults, pass CLI options or set environment variables (`DBT_LOG_PATH`, `DBT_STATE`). Flags that tell dbt where to find project resources (for example, `model-paths`) are set in `dbt_project.yml`, but as a top-level key, outside the `flags` dictionary; these configs are expected to be fully static and never vary based on the command or execution environment. +2. **Opt-in flags:** Flags opting into [legacy dbt behaviors](/reference/global-configs/legacy-behaviors) can _only_ be defined in `dbt_project.yml`. These are intended to be set in version control and migrated via pull/merge request. Their values should not diverge indefinitely across invocations, environments, or users. + +### Accessing flags + +Custom user-defined logic, written in Jinja, can check the values of flags using [the `flags` context variable](/reference/dbt-jinja-functions/flags). + +```yaml +# dbt_project.yml + +on-run-start: + - '{{ log("I will stop at the first sign of trouble", info = true) if flags.FAIL_FAST }}' +``` + +Because the values of `flags` can differ across invocations, we strongly advise against using `flags` as an input to configurations or dependencies (`ref` + `source`) that dbt resolves [during parsing](/reference/parsing#known-limitations). + +## Available flags + +| Flag name | Type | Default | Supported in project? | Environment variable | Command line option | Supported in Cloud CLI? | +|-----------|------|---------|-----------------------|----------------------|---------------------|-------------------------| +| [cache_selected_only](/reference/global-configs/cache) | boolean | False | ✅ | `DBT_CACHE_SELECTED_ONLY` | `--cache-selected-only`, `--no-cache-selected-only` | ✅ | +| [debug](/reference/global-configs/logs#debug-level-logging) | boolean | False | ✅ | `DBT_DEBUG` | `--debug`, `--no-debug` | ✅ | +| [defer](/reference/node-selection/defer) | boolean | False | ❌ | `DBT_DEFER` | `--defer`, `--no-defer` | ✅ (enabled by default) | +| [defer_state](/reference/node-selection/defer) | path | None | ❌ | `DBT_DEFER_STATE` | `--defer-state` | ❌ | +| [fail_fast](/reference/global-configs/failing-fast) | boolean | False | ✅ | `DBT_FAIL_FAST` | `--fail-fast`, `-x`, `--no-fail-fast` | ✅ | +| [full_refresh](/reference/resource-configs/full_refresh) | boolean | False | ✅ (as resource config) | `DBT_FULL_REFRESH` | `--full-refresh`, `--no-full-refresh` | ✅ | +| [indirect_selection](/reference/node-selection/test-selection-examples#syntax-examples) | enum | eager | ✅ | `DBT_INDIRECT_SELECTION` | `--indirect-selection` | ❌ | +| [introspect](/reference/commands/compile#introspective-queries) | boolean | True | ❌ | `DBT_INTROSPECT` | `--introspect`, `--no-introspect` | ❌ | +| [log_cache_events](/reference/global-configs/logs#logging-relational-cache-events) | boolean | False | ❌ | `DBT_LOG_CACHE_EVENTS` | `--log-cache-events`, `--no-log-cache-events` | ❌ | +| [log_format_file](/reference/global-configs/logs#log-formatting) | enum | default (text) | ✅ | `DBT_LOG_FORMAT_FILE` | `--log-format-file` | ❌ | +| [log_format](/reference/global-configs/logs#log-formatting) | enum | default (text) | ✅ | `DBT_LOG_FORMAT` | `--log-format` | ❌ | +| [log_level_file](/reference/global-configs/logs#log-level) | enum | debug | ✅ | `DBT_LOG_LEVEL_FILE` | `--log-level-file` | ❌ | +| [log_level](/reference/global-configs/logs#log-level) | enum | info | ✅ | `DBT_LOG_LEVEL` | `--log-level` | ❌ | +| [log_path](/reference/global-configs/logs) | path | None (uses `logs/`) | ❌ | `DBT_PROFILES_DIR` | `--profiles-dir` | ❌ | +| [partial_parse](/reference/global-configs/parsing#partial-parsing) | boolean | True | ✅ | `DBT_PARTIAL_PARSE` | `--partial-parse`, `--no-partial-parse` | ✅ | +| [populate_cache](/reference/global-configs/cache) | boolean | True | ✅ | `DBT_POPULATE_CACHE` | `--populate-cache`, `--no-populate-cache` | ✅ | +| [print](/reference/global-configs/print-output#suppress-print-messages-in-stdout) | boolean | True | ❌ | `DBT_PRINT` | `--print` | ❌ | +| [printer_width](/reference/global-configs/print-output#printer-width) | int | 80 | ✅ | `DBT_PRINTER_WIDTH` | `--printer-width` | ❌ | +| [profile](/docs/core/connect-data-platform/connection-profiles#about-profiles) | string | None | ✅ (as top-level key) | `DBT_PROFILE` | `--profile` | ❌ | +| [profiles_dir](/docs/core/connect-data-platform/connection-profiles#about-profiles) | path | None (current dir, then HOME dir) | ❌ | `DBT_PROFILES_DIR` | `--profiles-dir` | ❌ | +| [project_dir](/reference/dbt_project.yml) | path | | ❌ | `DBT_PROJECT_DIR` | `--project-dir` | ❌ | +| [quiet](/reference/global-configs/logs#suppress-non-error-logs-in-output) | boolean | False | ❌ | `DBT_QUIET` | `--quiet` | ✅ | +| [send_anonymous_usage_stats](/reference/global-configs/usage-stats) | boolean | True | ✅ | `DBT_SEND_ANONYMOUS_USAGE_STATS` | `--send-anonymous-usage-stats`, `--no-send-anonymous-usage-stats` | ❌ | +| [source_freshness_run_project_hooks](/reference/global-configs/legacy-behaviors#source_freshness_run_project_hooks) | boolean | False | ✅ | ❌ | ❌ | ❌ | +| [state](/reference/node-selection/defer) | path | none | ❌ | `DBT_STATE`, `DBT_DEFER_STATE` | `--state`, `--defer-state` | ❌ | +| [static_parser](/reference/global-configs/parsing#static-parser) | boolean | True | ✅ | `DBT_STATIC_PARSER` | `--static-parser`, `--no-static-parser` | ❌ | +| [store_failures](/reference/resource-configs/store_failures) | boolean | False | ✅ (as resource config) | `DBT_STORE_FAILURES` | `--store-failures`, `--no-store-failures` | ✅ | +| [target_path](/reference/global-configs/json-artifacts) | path | none | ❌ | `DBT_TARGET_PATH` | `--target-path` | ❌ | +| [target](/docs/core/connect-data-platform/connection-profiles#about-profiles) | path | None (uses `target/`) | ❌ | `DBT_TARGET` | `--target` | ❌ | +| [use_colors_file](/reference/global-configs/logs#color) | boolean | True | ✅ | `DBT_USE_COLORS_FILE` | `--use-colors-file`, `--no-use-colors-file` | ❌ | +| [use_colors](/reference/global-configs/print-output#print-color) | boolean | True | ✅ | `DBT_USE_COLORS` | `--use-colors`, `--no-use-colors` | ❌ | +| [use_experimental_parser](/reference/global-configs/parsing#experimental-parser) | boolean | False | ✅ | `DBT_USE_EXPERIMENTAL_PARSER` | `--use-experimental-parser`, `--no-use-experimental-parser` | ❌ | +| [version_check](/reference/global-configs/version-compatibility) | boolean | varies | ✅ | `DBT_VERSION_CHECK` | `--version-check`, `--no-version-check` | ❌ | +| [warn_error_options](/reference/global-configs/warnings) | dict | {} | ✅ | `DBT_WARN_ERROR_OPTIONS` | `--warn-error-options` | ✅ | +| [warn_error](/reference/global-configs/warnings) | boolean | False | ✅ | `DBT_WARN_ERROR` | `--warn-error`, `--no-warn-error` | ✅ | +| [write_json](/reference/global-configs/json-artifacts) | boolean | True | ✅ | `DBT_WRITE_JSON` | `--write-json`, `--no-write-json` | ✅ | diff --git a/website/docs/reference/global-configs/command-line-flags.md b/website/docs/reference/global-configs/command-line-options.md similarity index 95% rename from website/docs/reference/global-configs/command-line-flags.md rename to website/docs/reference/global-configs/command-line-options.md index fbe89ce28f1..058aecb6cd5 100644 --- a/website/docs/reference/global-configs/command-line-flags.md +++ b/website/docs/reference/global-configs/command-line-options.md @@ -1,7 +1,7 @@ --- -title: "Command line flags" -id: "command-line-flags" -sidebar: "Command line flags" +title: "Command line options" +id: "command-line-options" +sidebar: "Command line options" --- For consistency, command-line interface (CLI) flags should come right after the `dbt` prefix and its subcommands. This includes "global" flags (supported for all commands). When set, CLI flags override environment variables and profile configs. diff --git a/website/docs/reference/global-configs/indirect-selection.md b/website/docs/reference/global-configs/indirect-selection.md new file mode 100644 index 00000000000..e543ee1a690 --- /dev/null +++ b/website/docs/reference/global-configs/indirect-selection.md @@ -0,0 +1,53 @@ +--- +title: "Indirect selection" +id: "indirect-selection" +sidebar: "Indirect selection" +--- + +Use the `--indirect_selection` flag to `dbt test` or `dbt build` to configure which tests to run for the nodes you specify. You can set this as a CLI flag or an environment variable. In dbt Core, you can also configure user configurations in [YAML selectors](/reference/node-selection/yaml-selectors) or in the `flags:` block of `dbt_project.yml`, which sets project-level flags. + +When all flags are set, the order of precedence is as follows. Refer to [About global configs](/reference/global-configs/about-global-configs) for more details: + +1. CLI configurations +1. Environment variables +1. User configurations + +You can set the flag to: `empty`, `buildable`, `cautious`, or `eager` (default). By default, dbt indirectly selects all tests if they touch any resource you select. Learn more about these options in [Indirect selection in Test selection examples](/reference/node-selection/test-selection-examples?indirect-selection-mode=empty#indirect-selection). + +For example, you can run tests that only refer to selected nodes using a CLI configuration: + + + +```shell +dbt test --indirect-selection cautious +``` + + + +Or you can run tests that only refer to selected nodes using an environment variable: + + + +```text + +$ export DBT_INDIRECT_SELECTION=cautious +dbt run + +``` + + + +You can also run tests that only refer to selected nodes using `dbt_project.yml` project-level flags: + + + +```yaml + +flags: + indirect_selection: cautious + +``` + + + + diff --git a/website/docs/reference/global-configs/json-artifacts.md b/website/docs/reference/global-configs/json-artifacts.md index d8948b20d8e..57940512ef2 100644 --- a/website/docs/reference/global-configs/json-artifacts.md +++ b/website/docs/reference/global-configs/json-artifacts.md @@ -4,7 +4,9 @@ id: "json-artifacts" sidebar: "JSON artifacts" --- -The `WRITE_JSON` config determines whether dbt writes JSON artifacts (eg. `manifest.json`, `run_results.json`) to the `target/` directory. JSON serialization can be slow, and turning this flag off _might_ make invocations of dbt faster. Alternatively, you might disable this config if you want to perform a dbt operation and avoid overwriting artifacts from a previous run step. +### Write JSON artifacts + +The `WRITE_JSON` config determines whether dbt writes [JSON artifacts](/reference/artifacts/dbt-artifacts) (for example, `manifest.json`, `run_results.json`) to the `target/` directory. JSON serialization can be slow, and turning this flag off _might_ make invocations of dbt faster. Alternatively, you can disable this config to perform a dbt operation and avoid overwriting artifacts from a previous run step. @@ -13,3 +15,13 @@ dbt --no-write-json run ``` + + + +### Target path + +By default, dbt will write JSON artifacts and compiled SQL files to a directory named `target/`. This directory is located relative to `dbt_project.yml` of the active project. + +Just like other global configs, it is possible to override these values for your environment or invocation by using the CLI option (`--target-path`) or environment variables (`DBT_TARGET_PATH`). + + diff --git a/website/docs/reference/global-configs/legacy-behaviors.md b/website/docs/reference/global-configs/legacy-behaviors.md new file mode 100644 index 00000000000..7626240b71f --- /dev/null +++ b/website/docs/reference/global-configs/legacy-behaviors.md @@ -0,0 +1,36 @@ +--- +title: "Legacy behaviors" +id: "legacy-behaviors" +sidebar: "Legacy behaviors" +--- + +Most flags exist to configure runtime behaviors with multiple valid choices. The right choice may vary based on the environment, user preference, or the specific invocation. + +Another category of flags provides existing projects with a migration window for runtime behaviors that are changing in newer releases of dbt. These flags help us achieve a balance between these goals, which can otherwise be in tension, by: +- Providing a better, more sensible, and more consistent default behavior for new users/projects. +- Providing a migration window for existing users/projects — nothing changes overnight without warning. +- Providing maintainability of dbt software. Every fork in behavior requires additional testing & cognitive overhead that slows future development. These flags exist to facilitate migration from "current" to "better," not to stick around forever. + +These flags go through three phases of development: +1. **Introduction:** The flag is introduced, and the logic added within dbt to support both 'old' + 'new' behaviours. +2. **Maturity:** The default value of the flag is switched, from 'old' to 'new.' +3. **Deprecation:** The flag, having been marked for deprecation (with user warnings), is removed. The logic to support the ‘old’ behaviour is removed from dbt codebases. + +These flags can _only_ be set in `dbt_project.yml`. They configure behaviors closely tied to project code. They should be defined in version control, and changed via pull/merge request, with the same testing and peer review. + +## Behaviors + +### `source_freshness_run_project_hooks` +- Should the 'dbt source freshness' command include [on-run-start / on-run-end hooks](/reference/project-configs/on-run-start-on-run-end)? +- Planned introduction in v1.8, with default 'True' +- Planned removal in v1.9 + +If you have specific project `on-run-start` / `on-run-end` hooks that should not run before/after `source freshness` command, you can add a conditional check to those hooks: +```yaml +# dbt_project.yml + +on-run-start: + - '{{ ... if flags.WHICH != 'freshness' }}' +``` + +In the meantime, you can set `source_freshness_run_project_hooks: False` to preserve the legacy behavior. diff --git a/website/docs/reference/global-configs/logs.md b/website/docs/reference/global-configs/logs.md index 69dd57b1db4..7dfadb902c6 100644 --- a/website/docs/reference/global-configs/logs.md +++ b/website/docs/reference/global-configs/logs.md @@ -83,20 +83,9 @@ dbt --debug run ### Log and target paths -By default, dbt will write logs to a directory named `logs/`, and all other artifacts to a directory named `target/`. Both of those directories are located relative to `dbt_project.yml` of the active project—that is, the root directory from which dbt is run. +By default, dbt will write logs to a directory named `logs/`, and all other artifacts to a directory named `target/`. Both of those directories are located relative to `dbt_project.yml` of the active project. -Just like other global configs, it is possible to override these values for your environment or invocation by using CLI flags (`--target-path`, `--log-path`) or environment variables (`DBT_TARGET_PATH`, `DBT_LOG_PATH`). - -Unlike the other global configs documented on this page, which can be set in `profiles.yml`, the project paths are configured in `dbt_project.yml`. This is because `profiles.yml` and `dbt_project.yml` are most often located in separate file systems on your machine, and the log and artifact paths are always defined relative to the location of `dbt_project.yml`. - - - -```yaml -[target-path](target-path): "other-target" -[log-path](log-path): "other-logs" -``` - - +Just like other global configs, it is possible to override these values for your environment or invocation by using CLI options (`--target-path`, `--log-path`) or environment variables (`DBT_TARGET_PATH`, `DBT_LOG_PATH`). diff --git a/website/docs/reference/global-configs/parsing.md b/website/docs/reference/global-configs/parsing.md index b8fbf432652..744a7a00eab 100644 --- a/website/docs/reference/global-configs/parsing.md +++ b/website/docs/reference/global-configs/parsing.md @@ -44,15 +44,4 @@ config: ### Experimental parser -With the `USE_EXPERIMENTAL_PARSER` config, you can opt into the latest and greatest experimental version of the static parser, which is still being sampled for 100% correctness. See [the docs on parsing](/reference/parsing#experimental-parser) for more details. - - - -```yaml - -config: - use_experimental_parser: true - -``` - - +Not currently in use. diff --git a/website/docs/reference/global-configs/project-flags.md b/website/docs/reference/global-configs/project-flags.md new file mode 100644 index 00000000000..d20292508b0 --- /dev/null +++ b/website/docs/reference/global-configs/project-flags.md @@ -0,0 +1,47 @@ +--- +title: "Project flags" +id: "project-flags" +sidebar: "Project flags" +--- + + + +```yaml + +flags: + : + +``` + + + +Reference the [table of all flags](/reference/global-configs/about-global-configs#available-flags) to see which global configs are available for setting in [`dbt_project.yml`](/reference/dbt_project.yml). + +The `flags` dictionary is the _only_ place you can opt out of [behavior changes](/reference/global-configs/legacy-behaviors), while the legacy behavior is still supported. + + + +:::warning Deprecated functionality +In older versions of dbt, custom default values of flags (global configs) were set in `profiles.yml`. Starting in v1.8, these configs are set in `dbt_project.yml` instead. +::: + +For most global configurations, you can set "user profile" configurations in the `config:` block of `profiles.yml`. This style of configuration sets default values for all projects using this profile directory — usually, all projects running on your local machine. + + + +```yaml + +config: + : true + +``` + + + + + + + +The exception: Some global configurations are actually set in `dbt_project.yml`, instead of `profiles.yml`, because they control where dbt places logs and artifacts. Those file paths are always relative to the location of `dbt_project.yml`. For more details, refer to [Log and target paths](/reference/global-configs/logs#log-and-target-paths). + + diff --git a/website/docs/reference/global-configs/record-timing-info.md b/website/docs/reference/global-configs/record-timing-info.md new file mode 100644 index 00000000000..503c8de3fe9 --- /dev/null +++ b/website/docs/reference/global-configs/record-timing-info.md @@ -0,0 +1,17 @@ +--- +title: "Record timing info" +id: "record-timing-info" +--- + +The `-r` or `--record-timing-info` flag saves performance profiling information to a file. This file can be visualized with `snakeviz` to understand the performance characteristics of a dbt invocation. + + + +```text +$ dbt -r timing.txt run +... + +$ snakeviz timing.txt +``` + + diff --git a/website/docs/reference/global-configs/version-compatibility.md b/website/docs/reference/global-configs/version-compatibility.md index fdc9bc1d48e..9f009d64084 100644 --- a/website/docs/reference/global-configs/version-compatibility.md +++ b/website/docs/reference/global-configs/version-compatibility.md @@ -4,7 +4,7 @@ id: "version-compatibility" sidebar: "Version compatibility" --- -Projects are recommended to set [dbt version requirements](/reference/project-configs/require-dbt-version), especially if they use features that are newer, or which may break in future versions of dbt Core. By default, if you run a project with an incompatible dbt version, dbt will raise an error. +For the first several years of dbt Core's development, breaking changes were more common. For this reason, we encouraged setting [dbt version requirements](/reference/project-configs/require-dbt-version) — especially if they use features that are newer or which may break in future versions of dbt Core. By default, if you run a project with an incompatible dbt version, dbt will raise an error. You can use the `VERSION_CHECK` config to disable this check and suppress the error message: diff --git a/website/docs/reference/global-configs/yaml-configurations.md b/website/docs/reference/global-configs/yaml-configurations.md deleted file mode 100644 index 9bac8063c1f..00000000000 --- a/website/docs/reference/global-configs/yaml-configurations.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: "YAML configurations" -id: "yaml-configurations" -sidebar: "YAML configurations" ---- - -For most global configurations, you can set "user profile" configurations in the `config:` block of `profiles.yml`. This style of configuration sets default values for all projects using this profile directory—usually, all projects running on your local machine. - - - -```yaml - -config: - : true - -``` - - - - - -The exception: Some global configurations are actually set in `dbt_project.yml`, instead of `profiles.yml`, because they control where dbt places logs and artifacts. Those file paths are always relative to the location of `dbt_project.yml`. For more details, refer to [Log and target paths](/reference/global-configs/logs#log-and-target-paths). - - diff --git a/website/docs/reference/parsing.md b/website/docs/reference/parsing.md index 6eed4c96af0..1cb3490554a 100644 --- a/website/docs/reference/parsing.md +++ b/website/docs/reference/parsing.md @@ -5,8 +5,8 @@ description: "Read this guide to understand the project parsing configuration in ## Related documentation - The `dbt parse` [command](/reference/commands/parse) -- Partial parsing [profile config](/docs/core/connect-data-platform/profiles.yml#partial_parse) and [CLI flags](/reference/global-cli-flags#partial-parsing) -- Experimental parser [CLI flag](/reference/global-cli-flags#experimental-parser) +- Partial parsing [profile config](/docs/core/connect-data-platform/profiles.yml#partial_parse) and [CLI flags](/reference/global-configs/parsing) +- Experimental parser [CLI flag](/reference/global-configs/parsing) ## What is parsing? diff --git a/website/docs/reference/project-configs/clean-targets.md b/website/docs/reference/project-configs/clean-targets.md index 8ca4065ed75..711939ca77d 100644 --- a/website/docs/reference/project-configs/clean-targets.md +++ b/website/docs/reference/project-configs/clean-targets.md @@ -16,7 +16,7 @@ clean-targets: [directorypath] Optionally specify a custom list of directories to be removed by the `dbt clean` [command](/reference/commands/clean). As such, you should only include directories containing artifacts (e.g. compiled files, logs, installed packages) in this list. ## Default -If this configuration is not included in your `dbt_project.yml` file, the `clean` command will remove files in your [target-path](/reference/project-configs/target-path). +If this configuration is not included in your `dbt_project.yml` file, the `clean` command will remove files in your [target-path](/reference/global-configs/json-artifacts). ## Examples diff --git a/website/docs/reference/project-configs/log-path.md b/website/docs/reference/project-configs/log-path.md deleted file mode 100644 index 29cad35d120..00000000000 --- a/website/docs/reference/project-configs/log-path.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -datatype: directorypath -description: "Read this guide to understand the log-path configuration in dbt." -default_value: logs ---- - - -```yml -log-path: directorypath -``` - - - -## Definition -Optionally specify a custom directory where dbt will write logs. - -## Default -By default, dbt will write to the `logs` directory, i.e. `log-path: logs` - - - -## Configuration - -In the manner of a ["global" config](/reference/global-configs/about-global-configs), the log path can be set in three places: -1. `--log-path` CLI flag -2. `DBT_LOG_PATH` environment variable -3. `log-path` in `dbt_project.yml` - - - -:::warning Feature deprecation - -As of dbt version 1.5, setting the `log-path` in the `dbt_project.yml` is deprecated. Backward compatibility is still supported in 1.5 but will be removed in a future update. Migrate to the CLI flag or environment variable methods to avoid potential errors or disruptions. - -::: - -The precedence order is: CLI flag > env var > `dbt_project.yml(deprecated)` - - - - - -The precedence order is: CLI flag > env var > `dbt_project.yml` - - - - - -## Examples -### Specify subdirectory using the project config file - - - -```yml -log-path: dbt_logs -``` - - - - -### Specify subdirectory from the command line - -```bash -dbt --log-path dbt_logs run -``` - - diff --git a/website/docs/reference/project-configs/target-path.md b/website/docs/reference/project-configs/target-path.md deleted file mode 100644 index fddc5a93c5e..00000000000 --- a/website/docs/reference/project-configs/target-path.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -datatype: directorypath -default_value: target ---- - - -```yml -target-path: directorypath -``` - - - -# Definition -Optionally specify a custom directory where compiled files (e.g. compiled models and tests) will be written when you run the `dbt run`, `dbt compile`, or `dbt test` command. - - - -## Default -By default, dbt will write compiled files to the `target` directory, i.e. `target-path: target` - - - -## Configuration - -In the manner of a ["global" config](/reference/global-configs/about-global-configs), the target path can be set in three places: -1. `--target-path` CLI flag -2. `DBT_TARGET_PATH` environment variable -3. `target-path` in `dbt_project.yml` - - - -:::warning Feature deprecation - -As of dbt version 1.5, setting the `target-path` in the `dbt_project.yml` is deprecated. Backward compatibility is still supported in 1.5 but will be removed in a future update. Migrate to the CLI flag or environment variable methods to avoid potential errors or disruptions. - -::: - -The precedence order is: CLI flag > env var > `dbt_project.yml(deprecated)` - - - - - -The precedence order is: CLI flag > env var > `dbt_project.yml` - - - - - -## Examples -### Specify subdirectory using the project config file - - - -```yml -target-path: "compiled_files" -``` - - - - - -### Specify subdirectory from the command line - -```bash -dbt run --target-path compiled_files -``` - - \ No newline at end of file diff --git a/website/docs/reference/resource-configs/severity.md b/website/docs/reference/resource-configs/severity.md index 25bab9647d6..51413029baa 100644 --- a/website/docs/reference/resource-configs/severity.md +++ b/website/docs/reference/resource-configs/severity.md @@ -21,7 +21,7 @@ Here's how those play in practice: - If `severity: error`, dbt will check the `error_if` condition first. If the error condition is met, the test returns an error. If it's not met, dbt will then check the `warn_if` condition (defaulted to `!=0`). If it's not specified or the warn condition is met, the test warns; if it's not met, the test passes. - If `severity: warn`, dbt will skip the `error_if` condition entirely and jump straight to the `warn_if` condition. If the warn condition is met, the test warns; if it's not met, the test passes. -Note that test warn statuses will return errors instead if the [`--warn-error`](/reference/global-cli-flags#warnings-as-errors) flag is passed. Unless dbt is told to treat warnings as errors, a test with `warn` severity will never return an error. +Note that test warn statuses will return errors instead if the [`--warn-error`](/reference/global-configs/warnings) flag is passed. Unless dbt is told to treat warnings as errors, a test with `warn` severity will never return an error.