-
Notifications
You must be signed in to change notification settings - Fork 976
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
First draft: global configs --> flags
- Loading branch information
Showing
11 changed files
with
174 additions
and
59 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
59 changes: 49 additions & 10 deletions
59
website/docs/reference/global-configs/about-global-configs.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,18 +1,57 @@ | ||
| --- | ||
| title: "About global configs" | ||
| title: "About flags (global configs)" | ||
| id: "about-global-configs" | ||
| sidebar: "About global configs" | ||
| sidebar: "About flags (global configs)" | ||
| --- | ||
|
|
||
| Global configs enable you to fine-tune _how_ dbt runs projects on your machine—whether your personal laptop, an orchestration tool running remotely, or (in some cases) dbt Cloud. In general, they differ from most [project configs](/reference/dbt_project.yml) and [resource configs](/reference/configs-and-properties), which tell dbt _what_ to run. | ||
| In dbt, "flags" (also called "global configs") are configurations for fine-tuning _how_ dbt runs your project. They differ from [resource-specific configs](/reference/configs-and-properties) that tell dbt about _what_ to run. | ||
|
|
||
| Global configs control things like the visual output of logs, the manner in which dbt parses your project, and what to do when dbt finds a version mismatch or a failing model. These configs are "global" because they are available for all dbt commands, and because they can be set for all projects running on the same machine or in the same environment. | ||
| Flags control things like the visual output of logs, whether to treat specific warning messages as errors, or whether to "fail fast" after encountering the first error. These flags are "global" because they are available for all dbt commands, and because they can be set in multiple places. | ||
|
|
||
| ### Global config precedence | ||
| There is significant overlap between dbt's flags and dbt's command line options, but they are not the same: | ||
| - Certain flags can only be set in `dbt_project.yml`, and cannot be overridden for specific invocations via CLI option. | ||
| - If a CLI option is supported by specific commands, rather than supported by all commands ("global"), it is generally not considered to be a "flag". | ||
|
|
||
| Starting in v1.0, you can set global configs in three places. dbt will evaluate the configs in the following order: | ||
| 1. [user config](https://docs.getdbt.com/reference/global-configs/yaml-configurations) | ||
| 1. [environment variable](https://docs.getdbt.com/reference/global-configs/environment-variable-configs) | ||
| 1. [CLI flag](https://docs.getdbt.com/reference/global-configs/command-line-flags) | ||
| ### Setting flags | ||
|
|
||
| There are multiple ways of setting flags, which depend on the use case: | ||
| - **[Project-level `flags` (`dbt_project.yml`)](https://docs.getdbt.com/reference/global-configs/project-flags):** Define version-controlled defaults for everyone running this project. Preserve legacy behaviors until their slated deprecation. | ||
| - **[Environment variables](https://docs.getdbt.com/reference/global-configs/environment-variable-configs):** Define behavior that should be different in different runtime environments (development vs. production vs. [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration), or different for different users in development (based on personal preferences). | ||
| - **[CLI options](https://docs.getdbt.com/reference/global-configs/command-line-options):** Define behavior specific to _this invocation_. Supported for all dbt commands. | ||
|
|
||
| The precedence order is CLI > Env Var > Project. | ||
|
|
||
| Most flags can be set in all three places: | ||
| ```yaml | ||
| # dbt_project.yml | ||
| flags: | ||
| # set default for running this project -- anywhere, anytime, by anyone | ||
| fail_fast: true | ||
| ``` | ||
| ```bash | ||
| # set this environment variable to 'True' (bash syntax) | ||
| export DBT_FAIL_FAST=1 | ||
| dbt run | ||
| ``` | ||
| ```bash | ||
| dbt run --fail-fast # set to True for this specific invocation | ||
| dbt run --no-fail-fast # set to False | ||
| ``` | ||
|
|
||
| However, there are exceptions: | ||
| - Flags for impermanent file paths (e.g. `--log-path` or `--state-path`) cannot be set in `dbt_project.yml`. To override defaults, pass CLI options or set environment variables (`DBT_LOG_PATH`, `DBT_STATE_PATH`). | ||
| - Flags opting into legacy dbt behaviors can _only_ be defined in `dbt_project.yml`. These are intended to be set in version control, and migrated via pull/merge request. Their values should not diverge indefinitely across invocations, environments, or users. | ||
|
|
||
| ### Accessing flags | ||
|
|
||
| Custom user-defined logic, written in Jinja, can check the values of flags using [the `flags` context variable](https://docs.getdbt.com/reference/dbt-jinja-functions/flags). | ||
|
|
||
| ```yaml | ||
| # dbt_project.yml | ||
|
|
||
| on-run-start: | ||
| - '{{ log("I will stop at the first sign of trouble", info = true) if flags.FAIL_FAST }}' | ||
| ``` | ||
| Because the values of `flags` can differ across invocations, we strongly advise against using `flags` as an input to configurations or dependencies (`ref` + `source`) that dbt resolves [during parsing](https://docs.getdbt.com/reference/parsing#known-limitations). | ||
|
|
||
| Each config is prioritized over the previous one. For example, if all three are provided, then the CLI flag takes precedence. |
6 changes: 3 additions & 3 deletions
6
...ence/global-configs/command-line-flags.md → ...ce/global-configs/command-line-options.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| --- | ||
| title: "Legacy behaviors" | ||
| id: "legacy-behaviors" | ||
| sidebar: "Legacy behaviors" | ||
| --- | ||
|
|
||
| Most flags exist to configure runtime behaviors with multiple valid choices. The right choice may vary based on the environment, user preference, or the specific invocation. | ||
|
|
||
| Another category of flags provides existing projects with a migration window for runtime behaviors that are changing in newer releases of dbt. These flags help us achieve a balance between these goals, which can otherwise be in tension: | ||
| 1. Providing a better, more sensible, and more consistent default behaviour for new users/projects going forward | ||
| 2. Providing a migration window for existing users/projects — nothing changes overnight without warning | ||
| 3. Maintainability of dbt software. Every fork in behaviour requires additional testing & cognitive overhead that slows future development. These flags exist to facilitate migration from "current" to "better," not to stick around forever. | ||
|
|
||
| These flags go through three phases of development: | ||
| 1. **Introduction:** The flag is introduced, and the logic added within dbt to support both 'old' + 'new' behaviours. | ||
| 2. **Maturity:** The default value of the flag is switched, from 'old' to 'new.' | ||
| 3. **Deprecation:** The flag, having been marked for deprecation (with user warnings), is removed. The logic to support the ‘old’ behaviour is removed from dbt codebases. | ||
|
|
||
| These flags can _only_ be set in `dbt_project.yml`. They configure behaviors closely tied to project code. They should be defined in version control, and changed via pull/merge request, with the same testing and peer review. | ||
|
|
||
| ## Behaviors | ||
|
|
||
| ### `source_freshness_run_project_hooks` | ||
| - Should the 'dbt source freshness' command include [on-run-start / on-run-end hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end)? | ||
| - Planned introduction in v1.8, with default 'True' | ||
| - Planned removal in v1.9 | ||
|
|
||
| If you have specific project `on-run-start` / `on-run-end` hooks that should not run before/after `source freshness` command, you can add a conditional check to those hooks: | ||
| ```yaml | ||
| # dbt_project.yml | ||
|
|
||
| on-run-start: | ||
| - '{{ ... if flags.WHICH != 'freshness' }}' | ||
| ``` | ||
| In the meantime, you can set `source_freshness_run_project_hooks: False` to preserve the legacy behavior. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
17 changes: 17 additions & 0 deletions
17
website/docs/reference/global-configs/record-timing-info.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| --- | ||
| title: "Record timing info" | ||
| id: "record-timing-info" | ||
| --- | ||
|
|
||
| The `-r` or `--record-timing-info` flag saves performance profiling information to a file. This file can be visualized with `snakeviz` to understand the performance characteristics of a dbt invocation | ||
|
|
||
| <File name='Usage'> | ||
|
|
||
| ```text | ||
| $ dbt -r timing.txt run | ||
| ... | ||
| $ snakeviz timing.txt | ||
| ``` | ||
|
|
||
| </File> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters