Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add defer page to cloud develop docs #4229

Merged
merged 27 commits into from
Oct 11, 2023
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
4f895d5
add defer page to cloud develop docs
dave-connors-3 Oct 5, 2023
5c052aa
Merge branch 'mwong-cloud-cli-links' into dc-defer-in-cloud
mirnawong1 Oct 9, 2023
3d28be8
Update cloud-cli-installation.md
mirnawong1 Oct 9, 2023
c6be8e1
Merge branch 'mwong-cloud-cli-links' into dc-defer-in-cloud
mirnawong1 Oct 9, 2023
1d9243a
Merge branch 'cloud-cli-2' into dc-defer-in-cloud
mirnawong1 Oct 10, 2023
112a29e
Merge branch 'cloud-cli-2' into dc-defer-in-cloud
mirnawong1 Oct 10, 2023
959222c
Merge branch 'cloud-cli-2' into dc-defer-in-cloud
mirnawong1 Oct 10, 2023
9d585af
Merge branch 'cloud-cli-2' into dc-defer-in-cloud
mirnawong1 Oct 10, 2023
3aa2fd3
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 10, 2023
7bc788c
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 10, 2023
f764515
Update about-cloud-develop-defer.md
mirnawong1 Oct 10, 2023
faef573
Update about-cloud-develop-defer.md
mirnawong1 Oct 10, 2023
ac9f38e
Merge branch 'cloud-cli-2' into dc-defer-in-cloud
mirnawong1 Oct 11, 2023
94ffe6e
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
868fb65
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
34ee784
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
39b43cf
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
e44e66b
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
b5628d3
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
56df191
Update about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
e5a3d66
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
aeb68f0
add defer screenshot
mirnawong1 Oct 11, 2023
02207fe
better pic
mirnawong1 Oct 11, 2023
a31f454
Update website/docs/docs/cloud/about-cloud-develop-defer.md
mirnawong1 Oct 11, 2023
d4c2655
Merge branch 'cloud-cli-2' into dc-defer-in-cloud
mirnawong1 Oct 11, 2023
b696215
Update configure-cloud-cli.md
mirnawong1 Oct 11, 2023
e3e453f
Update cloud-cli-installation.md
mirnawong1 Oct 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions website/docs/docs/cloud/about-cloud-develop-defer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: Using defer in dbt Cloud
id: about-cloud-develop-defer
description: "Learn how to leverage defer to prod when developing with dbt Cloud."
sidebar_label: "Using defer in dbt Cloud"
pagination_next: "docs/cloud/cloud-cli-installation"
---


[Defer](/reference/node-selection/defer) is a powerful feature that allows developers to only build and run and test models they've edited without having to first run and build these models' upstream parents. This is powered by having a production manifest to compare to, and dbt will resolve the `{{ ref() }}` function with upstream production artifacts.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

Both the dbt Cloud IDE and the dbt Cloud CLI allow users to natively defer to production metadata directly in their development workflows, dramatically reducing development time and warehouse spend by preventing unecessary model builds.

## Required setup

Set your production environment using the [production environment checkbox](/docs/deploy/deploy-environments#set-as-production-environment-beta) in the environment settings page. This can be set for one deployment environment per dbt Cloud project. When using defer, the artifacts from the latest successful, non-CI job execution in the production environment will be compared to.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also does the user need to perform a one job run?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, there will need to be a successful job run first!

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

### Defer in the dbt Cloud IDE

Defer in the IDE is as simple as toggling the `Defer to production` switch on the command bar! When enabled, dbt Cloud will:
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

1. Pull down the most recent manifest from the Production environment for comparison
2. Pass the `--defer` flag to the command (for any command that accepts the flag)
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

For example, if you were to start developing with on a new branch with [nothing in your development schema](/reference/node-selection/defer#usage), make an edit to a single model, and run `dbt build -s state:modified` you would see *just* that model run, with any ` {{ ref() }}` functions resolve to the production location of the reffed models!
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

### Defer in dbt Cloud CLI

One key difference between using `--defer` in the Cloud CLI vs the IDE is that defer is *automatically on by default for all invocations*, using artifacts from your production evironment to compare. You can pass the `--no-defer` flag to your dbt command to disable this behavior. This flag is unique to the dbt Cloud CLI only.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

Cloud CLI offers additional flexibility for declaring which environment is used as the source of deferral artifacts. You can set a `defer-env-id` key in either your `dbt_project.yml` file or your `dbt_cloud.yml` file.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

```yml
# in dbt_cloud.yml
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
dever-env-id: '123456'
```

```yml
# in dbt_project.yml
dbt_cloud:
dever-env-id: '123456'
```
2 changes: 2 additions & 0 deletions website/docs/docs/cloud/about-cloud-develop.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,7 @@ dbt Cloud offers a fast and reliable way to work on your dbt project. It runs db

The following sections provide detailed instructions on setting up the dbt Cloud CLI and dbt Cloud IDE. To get started with dbt development, you'll need a [developer](/docs/cloud/manage-access/seats-and-users) account. For a more comprehensive guide about developing in dbt, refer to our [quickstart guides](/quickstarts).


---------
**Note**: The dbt Cloud CLI and the open-sourced dbt Core are both command line tools that let you run dbt commands. The key distinction is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its [features](/docs/cloud/about-cloud/dbt-cloud-features).

4 changes: 3 additions & 1 deletion website/docs/docs/cloud/cloud-cli-installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ dbt Cloud natively supports developing using a command line (CLI), empowering te
dbt commands are run against dbt Cloud's infrastructure and benefit from:

* Secure credential storage in the dbt Cloud platform.
* Automatic deferral of build artifacts to your Cloud project's production environment.
* [Automatic deferral](/docs/cloud/about-cloud-develop-defer) of build artifacts to your Cloud project's production environment.
* Speedier, lower-cost builds.
* Support for dbt Mesh ([cross-project `ref`](/docs/collaborate/govern/project-dependencies)),
* Significant platform improvements, to be released over the coming months.
Expand Down Expand Up @@ -65,6 +65,7 @@ This tab is for users who already have a Python environment configured.
```

</TabItem>


<TabItem value="brew" label="macOS (brew)">

Expand Down Expand Up @@ -145,6 +146,7 @@ During the public preview period, we recommend updating before filing a bug repo


<Tabs>

<TabItem value="existing" label="Existing dbt Core users (pip)">

To update, run `pip install --upgrade dbt`.
Expand Down
6 changes: 0 additions & 6 deletions website/docs/docs/cloud/configure-cloud-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ import CloudCLIFlag from '/snippets/_cloud-cli-flag.md';
Once you install the dbt Cloud CLI, you need to configure it to connect to a dbt Cloud project.

1. Ensure you meet the prerequisites above.

2. Download your credentials from dbt Cloud by clicking on the **Try the dbt Cloud CLI** banner on the dbt Cloud homepage.
3. Follow the banner instructions and download the config file to `~/.dbt/dbt_cloud.yml`. The config file looks like:

Expand All @@ -42,15 +41,10 @@ Once you install the dbt Cloud CLI, you need to configure it to connect to a dbt
- project-id: "<project-id>"
account-host: "<account-host>"
api-key: "<user-api-key>"

```

4. After downloading the config file, navigate to a dbt project in your terminal:

```bash
cd ~/dbt-projects/jaffle_shop
```

5. In your `dbt_project.yml` file, ensure you have or include a `dbt-cloud` section with a `project-id` field. The `project-id` field contains the dbt Cloud project ID you want to use.

```yaml
Expand Down
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ const sidebarSettings = {
link: { type: "doc", id: "docs/cloud/about-cloud-develop" },
items: [
"docs/cloud/about-cloud-develop",
"docs/cloud/about-cloud-develop-defer",
{
type: "category",
label: "dbt Cloud CLI",
Expand Down