Skip to content

Commit

Permalink
Merge branch 'current' into dbeatty/do-not-track-environment-variable
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Jan 8, 2024
2 parents 8d349f8 + e5b6505 commit 40cf449
Show file tree
Hide file tree
Showing 19 changed files with 463 additions and 77 deletions.
2 changes: 2 additions & 0 deletions website/docs/best-practices/how-we-mesh/mesh-1-intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ dbt Cloud is designed to coordinate the features above and simplify the complexi

If you're just starting your dbt journey, don't worry about building a multi-project architecture right away. You can _incrementally_ adopt the features in this guide as you scale. The collection of features work effectively as independent tools. Familiarizing yourself with the tooling and features that make up a multi-project architecture, and how they can apply to your organization will help you make better decisions as you grow.

For additional information, refer to the [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-4-faqs).

## Learning goals

- Understand the **purpose and tradeoffs** of building a multi-project architecture.
Expand Down
317 changes: 317 additions & 0 deletions website/docs/best-practices/how-we-mesh/mesh-4-faqs.md

Large diffs are not rendered by default.

43 changes: 22 additions & 21 deletions website/docs/docs/cloud/dbt-cloud-ide/lint-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Linters analyze code for errors, bugs, and style issues, while formatters fix st
</details>


In the dbt Cloud IDE, you have the capability to perform linting, auto-fix, and formatting on five different file types:
In the dbt Cloud IDE, you can perform linting, auto-fix, and formatting on five different file types:

- SQL &mdash; [Lint](#lint) and fix with SQLFluff, and [format](#format) with sqlfmt
- YAML, Markdown, and JSON &mdash; Format with Prettier
Expand Down Expand Up @@ -146,7 +146,7 @@ The Cloud IDE formatting integrations take care of manual tasks like code format

To format your SQL code, dbt Cloud integrates with [sqlfmt](http://sqlfmt.com/), which is an uncompromising SQL query formatter that provides one way to format the SQL query and Jinja.

By default, the IDE uses sqlfmt rules to format your code, making the **Format** button available and convenient to use right away. However, if you have a file named .sqlfluff in the root directory of your dbt project, the IDE will default to SQLFluff rules instead.
By default, the IDE uses sqlfmt rules to format your code, making the **Format** button available and convenient to use immediately. However, if you have a file named .sqlfluff in the root directory of your dbt project, the IDE will default to SQLFluff rules instead.

To enable sqlfmt:

Expand Down Expand Up @@ -189,10 +189,8 @@ To format your Python code, dbt Cloud integrates with [Black](https://black.read

## FAQs

<details>
<summary>When should I use SQLFluff and when should I use sqlfmt?</summary>

SQLFluff and sqlfmt are both tools used for formatting SQL code, but there are some differences that may make one preferable to the other depending on your use case. <br />
<detailsToggle alt_header="When should I use SQLFluff and when should I use sqlfmt?">
SQLFluff and sqlfmt are both tools used for formatting SQL code, but some differences may make one preferable to the other depending on your use case. <br />

SQLFluff is a SQL code linter and formatter. This means that it analyzes your code to identify potential issues and bugs, and follows coding standards. It also formats your code according to a set of rules, which are [customizable](#customize-linting), to ensure consistent coding practices. You can also use SQLFluff to keep your SQL code well-formatted and follow styling best practices. <br />

Expand All @@ -204,34 +202,37 @@ You can use either SQLFluff or sqlfmt depending on your preference and what work

- Use sqlfmt to only have your code well-formatted without analyzing it for errors and bugs. You can use sqlfmt out of the box, making it convenient to use right away without having to configure it.

</details>
</detailsToggle>

<details>
<summary>Can I nest <code>.sqlfluff</code> files?</summary>
<detailsToggle alt_header="Can I nest `.sqlfluff` files?">

To ensure optimal code quality, consistent code, and styles &mdash; it's highly recommended you have one main `.sqlfluff` configuration file in the root folder of your project. Having multiple files can result in various different SQL styles in your project. <br /><br />

However, you can customize and include an additional child `.sqlfluff` configuration file within specific subfolders of your dbt project. <br /><br />By nesting a `.sqlfluff` file in a subfolder, SQLFluff will apply the rules defined in that subfolder's configuration file to any files located within it. The rules specified in the parent `.sqlfluff` file will be used for all other files and folders outside of the subfolder. This hierarchical approach allows for tailored linting rules while maintaining consistency throughout your project. Refer to [SQLFluff documentation](https://docs.sqlfluff.com/en/stable/configuration.html#configuration-files) for more info.

</details>
</detailsToggle>

<details>
<summary>Can I run SQLFluff commands from the terminal?</summary>
<detailsToggle alt_header="Can I run SQLFluff commands from the terminal?">

Currently, running SQLFluff commands from the terminal isn't supported.
</details>
</detailsToggle>

<details>
<summary>Why am I unable to see the <bold>Lint</bold> or <bold>Format</bold> button?</summary>
<detailsToggle alt_header="Why am I unable to see the Lint or Format button?">

Make sure you're on a development branch. Formatting or Linting isn't available on "main" or "read-only" branches.
</details>
</detailsToggle>

<details>
<summary>Why is there inconsistent SQLFluff behavior when running outside the dbt Cloud IDE (such as a GitHub Action)?</summary>
&mdash; Double-check your SQLFluff version matches the one in dbt Cloud IDE (found in the <b>Code Quality</b> tab after a lint operation). <br /><br />
&mdash; If your lint operation passes despite clear rule violations, confirm you're not linting models with ephemeral models. Linting doesn't support ephemeral models in dbt v1.5 and lower.
</details>
<detailsToggle alt_header="Why is there inconsistent SQLFluff behavior when running outside the dbt Cloud IDE?">
- Double-check that your SQLFluff version matches the one in dbt Cloud IDE (found in the <b>Code Quality</b> tab after a lint operation). <br /><br />
- If your lint operation passes despite clear rule violations, confirm you're not linting models with ephemeral models. Linting doesn't support ephemeral models in dbt v1.5 and lower.
</detailsToggle>

<detailsToggle alt_header="What are some considerations when using dbt Cloud linting?">
Currently, the dbt Cloud IDE can lint or fix files up to a certain size and complexity. If you attempt to lint or fix files that are too large, taking more than 60 seconds for the dbt Cloud backend to process, you will see an 'Unable to complete linting this file' error.

To avoid this, break up your model into smaller models (files) so that they are less complex to lint or fix. Note that linting is simpler than fixing so there may be cases where a file can be linted but not fixed.

</detailsToggle>

## Related docs

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
title: "New: Native support for partial parsing"
description: "December 2023: For faster run times with your dbt invocations, configure dbt Cloud to parse only the changed files in your project."
sidebar_label: "New: Native support for partial parsing"
sidebar_position: 09
tags: [Jan-2024]
date: 2024-01-03
---

By default, dbt parses all the files in your project at the beginning of every dbt invocation. Depending on the size of your project, this operation can take a long time to complete. With the new partial parsing feature in dbt Cloud, you can reduce the time it takes for dbt to parse your project. When enabled, dbt Cloud parses only the changed files in your project instead of parsing all the project files. As a result, your dbt invocations will take less time to run.

To learn more, refer to [Partial parsing](/docs/deploy/deploy-environments#partial-parsing).

<Lightbox src="/img/docs/deploy/example-account-settings.png" width="85%" title="Example of the Partial parsing option" />

Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ Now available for dbt Cloud Enterprise plans is a new option to enable Git repos

To learn more, refer to [Repo caching](/docs/deploy/deploy-environments#git-repository-caching).

<Lightbox src="/img/docs/deploy/example-repo-caching.png" width="85%" title="Example of the Repository caching option" />
<Lightbox src="/img/docs/deploy/example-account-settings.png" width="85%" title="Example of the Repository caching option" />
2 changes: 1 addition & 1 deletion website/docs/guides/create-new-materializations.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ recently_updated: true

## Introduction

The model <Term id="materialization">materializations</Term> you're familiar with, `table`, `view`, and `incremental` are implemented as macros in a package that's distributed along with dbt. You can check out the [source code for these materializations](https://github.com/dbt-labs/dbt-core/tree/main/core/dbt/include/global_project/macros/materializations). If you need to create your own materializations, reading these files is a good place to start. Continue reading below for a deep-dive into dbt materializations.
The model <Term id="materialization">materializations</Term> you're familiar with, `table`, `view`, and `incremental` are implemented as macros in a package that's distributed along with dbt. You can check out the [source code for these materializations](https://github.com/dbt-labs/dbt-core/tree/main/core/dbt/adapters/include/global_project/macros/materializations). If you need to create your own materializations, reading these files is a good place to start. Continue reading below for a deep-dive into dbt materializations.

:::caution

Expand Down
6 changes: 4 additions & 2 deletions website/docs/reference/parsing.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ The [`PARTIAL_PARSE` global config](/reference/global-configs/parsing) can be en

Parse-time attributes (dependencies, configs, and resource properties) are resolved using the parse-time context. When partial parsing is enabled, and certain context variables change, those attributes will _not_ be re-resolved, and are likely to become stale.

In particular, you may see **incorrect results** if these attributes depend on "volatile" context variables, such as [`run_started_at`](/reference/dbt-jinja-functions/run_started_at), [`invocation_id`](/reference/dbt-jinja-functions/invocation_id), or [flags](/reference/dbt-jinja-functions/flags). These variables are likely (or even guaranteed!) to change in each invocation. We _highly discourage_ you from using these variables to set parse-time attributes (dependencies, configs, and resource properties).
In particular, you may see incorrect results if these attributes depend on "volatile" context variables, such as [`run_started_at`](/reference/dbt-jinja-functions/run_started_at), [`invocation_id`](/reference/dbt-jinja-functions/invocation_id), or [flags](/reference/dbt-jinja-functions/flags). These variables are likely (or even guaranteed!) to change in each invocation. dbt Labs _strongly discourages_ you from using these variables to set parse-time attributes (dependencies, configs, and resource properties).

Starting in v1.0, dbt _will_ detect changes in environment variables. It will selectively re-parse only the files that depend on that [`env_var`](/reference/dbt-jinja-functions/env_var) value. (If the env var is used in `profiles.yml` or `dbt_project.yml`, a full re-parse is needed.) However, dbt will _not_ re-render **descriptions** that include env vars. If your descriptions include frequently changing env vars (this is highly uncommon), we recommend that you fully re-parse when generating documentation: `dbt --no-partial-parse docs generate`.

Expand All @@ -51,7 +51,9 @@ If certain inputs change between runs, dbt will trigger a full re-parse. The res
- `dbt_project.yml` content (or `env_var` values used within)
- installed packages
- dbt version
- certain widely-used macros, e.g. [builtins](/reference/dbt-jinja-functions/builtins) overrides or `generate_x_name` for `database`/`schema`/`alias`
- certain widely-used macros (for example, [builtins](/reference/dbt-jinja-functions/builtins), overrides, or `generate_x_name` for `database`/`schema`/`alias`)

If you're triggering [CI](/docs/deploy/continuous-integration) job runs, the benefits of partial parsing are not applicable to new pull requests (PR) or new branches. However, they are applied on subsequent commits to the new PR or branch.

If you ever get into a bad state, you can disable partial parsing and trigger a full re-parse by setting the `PARTIAL_PARSE` global config to false, or by deleting `target/partial_parse.msgpack` (e.g. by running `dbt clean`).

Expand Down
2 changes: 1 addition & 1 deletion website/docs/reference/resource-configs/full_refresh.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Optionally set a resource to always or never full-refresh.

</VersionBlock>

This logic is encoded in the [`should_full_refresh()`](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/include/global_project/macros/materializations/configs.sql#L6) macro.
This logic is encoded in the [`should_full_refresh()`](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/adapters/include/global_project/macros/materializations/configs.sql#L6) macro.

## Usage

Expand Down
2 changes: 1 addition & 1 deletion website/docs/reference/resource-configs/store_failures.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Optionally set a test to always or never store its failures in the database.
- If the `store_failures` config is `none` or omitted, the resource will use the value of the `--store-failures` flag.
- When true, `store_failures` save all the record(s) that failed the test only if [limit](/reference/resource-configs/limit) is not set or if there are fewer records than the limit. `store_failures` are saved in a new table with the name of the test. By default, `store_failures` use a schema named `dbt_test__audit`, but, you can [configure](/reference/resource-configs/schema#tests) the schema to a different value.

This logic is encoded in the [`should_store_failures()`](https://github.com/dbt-labs/dbt-core/blob/98c015b7754779793e44e056905614296c6e4527/core/dbt/include/global_project/macros/materializations/helpers.sql#L77) macro.
This logic is encoded in the [`should_store_failures()`](https://github.com/dbt-labs/dbt-core/blob/77632122974b28967221758b4a470d7dfb608ac2/core/dbt/adapters/include/global_project/macros/materializations/configs.sql#L15) macro.



Expand Down
4 changes: 2 additions & 2 deletions website/docs/reference/resource-configs/strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,8 @@ This is a **required configuration**. There is no default value.

### Advanced: define and use custom snapshot strategy
Behind the scenes, snapshot strategies are implemented as macros, named `snapshot_<strategy>_strategy`
* [Source code](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/include/global_project/macros/materializations/snapshots/strategies.sql#L65) for the timestamp strategy
* [Source code](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/include/global_project/macros/materializations/snapshots/strategies.sql#L131) for the check strategy
* [Source code](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/adapters/include/global_project/macros/materializations/snapshots/strategies.sql#L52) for the timestamp strategy
* [Source code](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/adapters/include/global_project/macros/materializations/snapshots/strategies.sql#L136) for the check strategy

It's possible to implement your own snapshot strategy by adding a macro with the same naming pattern to your project. For example, you might choose to create a strategy which records hard deletes, named `timestamp_with_deletes`.

Expand Down
2 changes: 1 addition & 1 deletion website/docs/reference/resource-configs/where.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ tests:
The rendering context for the `where` config is the same as for all configurations defined in `.yml` files. You have access to `{{ var() }}` and `{{ env_var() }}`, but you **do not** have access to custom macros for setting this config. If you do want to use custom macros to template out the `where` filter for certain tests, there is a workaround.

As of v0.21, dbt defines a [`get_where_subquery` macro](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/include/global_project/macros/materializations/tests/where_subquery.sql).
As of v0.21, dbt defines a [`get_where_subquery` macro](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/adapters/include/global_project/macros/materializations/tests/where_subquery.sql).

dbt replaces `{{ model }}` in generic test definitions with `{{ get_where_subquery(relation) }}`, where `relation` is a `ref()` or `source()` for the resource being tested. The default implementation of this macro returns:
- `{{ relation }}` when the `where` config is not defined (`ref()` or `source()`)
Expand Down
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -1046,6 +1046,7 @@ const sidebarSettings = {
items: [
"best-practices/how-we-mesh/mesh-2-structures",
"best-practices/how-we-mesh/mesh-3-implementation",
"best-practices/how-we-mesh/mesh-4-faqs",
],
},
{
Expand Down
Loading

0 comments on commit 40cf449

Please sign in to comment.