Skip to content

Commit

Permalink
Preview launch of Advanced CI
Browse files Browse the repository at this point in the history
  • Loading branch information
nghi-ly committed Aug 7, 2024
1 parent f1ac857 commit 0115497
Show file tree
Hide file tree
Showing 5 changed files with 69 additions and 29 deletions.
14 changes: 14 additions & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,20 @@ Release notes are grouped by month for both multi-tenant and virtual private clo

\* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability.

## August 2024

- <Expandable alt_header="New: Advanced CI" lifecycle="preview">

[PLACEHOLDER STUB for content]

Advanced CI is now available as a [preview](/docs/dbt-versions/product-lifecycles#dbt-cloud)! You can enable the Compare changes option when creating new CI jobs. PR comments LOREM IPSUM

To learn more, check out [Advanced CI](/docs/dbt-versions/product-lifecycles) and [Set up CI jobs](/docs/dbt-versions/product-lifecycles). LOREM IPSUM

If you're working with already-existing CI jobs, you can easily enable these new options by editing the job's settings. Subsequent runs will include those feature options. LOREM IPSUM

</Expandable>

## July 2024
- **New:** [Connections](/docs/cloud/connect-data-platform/about-connections#connection-management) are now available under **Account settings** as a global setting. Previously, they were found under **Project settings**. This is being rolled out in phases over the coming weeks.
- **New:** Admins can now assign [environment-level permissions](/docs/cloud/manage-access/environment-permissions) to groups for specific roles.
Expand Down
34 changes: 18 additions & 16 deletions website/docs/docs/deploy/ci-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,40 +21,42 @@ To make CI job creation easier, many options on the **CI job** page are set to d

1. On your deployment environment page, click **Create job** > **Continuous integration job** to create a new CI job.

2. Options in the **Job settings** section:
1. Options in the **Job settings** section:
- **Job name** &mdash; Specify the name for this CI job.
- **Description** &mdash; Provide a description about the CI job.
- **Environment** &mdash; By default, it’s set to the environment you created the CI job from.
- **Environment** &mdash; By default, it’s set to the environment you created the CI job from. Use the dropdown to change the default setting.

1. Options in the **Git trigger** section:
- **Triggered by pull requests** &mdash; By default, it’s enabled. Every time a developer opens up a pull request or pushes a commit to an existing pull request, this job will get triggered to run.
- **Run on Draft Pull Request** &mdash; Enable this option if you want to also trigger the job to run every time a developer opens up a draft pull request or pushes a commit to that draft pull request.
- **Run on draft pull request** &mdash; Enable this option if you want to also trigger the job to run every time a developer opens up a draft pull request or pushes a commit to that draft pull request.

3. Options in the **Execution settings** section:
1. Options in the **Execution settings** section:
- **Commands** &mdash; By default, it includes the `dbt build --select state:modified+` command. This informs dbt Cloud to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click **Add command** to add more [commands](/docs/deploy/job-commands) that you want to be invoked when this job runs.
- **Run compare changes** &mdash; LOREM IPSUM Enable to compare changes between job runs. When enabled, you can review [Compare](/docs/deploy/run-visibility#compare-tab) dashboard. Refer to [Advanced CI](/docs/deploy/continuous-integration#advanced-ci) for more details. LOREM IPSUM
- **Compare changes against an environment (Deferral)** &mdash; By default, it’s set to the **Production** environment if you created one. This option allows dbt Cloud to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG.

:::info
Older versions of dbt Cloud only allow you to defer to a specific job instead of an environment. Deferral to a job compares state against the project code that was run in the deferred job's last successful run. While deferral to an environment is more efficient as dbt Cloud will compare against the project representation (which is stored in the `manifest.json`) of the last successful deploy job run that executed in the deferred environment. By considering _all_ [deploy jobs](/docs/deploy/deploy-jobs) that run in the deferred environment, dbt Cloud will get a more accurate, latest project representation state.
:::
:::info
Older versions of dbt Cloud only allow you to defer to a specific job instead of an environment. Deferral to a job compares state against the project code that was run in the deferred job's last successful run. While deferral to an environment is more efficient as dbt Cloud will compare against the project representation (which is stored in the `manifest.json`) of the last successful deploy job run that executed in the deferred environment. By considering _all_ [deploy jobs](/docs/deploy/deploy-jobs) that run in the deferred environment, dbt Cloud will get a more accurate, latest project representation state.
:::

- **Run timeout** &mdash; Cancel this CI job if the run time exceeds the timeout value. You can use this option to help ensure that a CI check doesn't consume too much of your warehouse resources.


- **Generate docs on run** &mdash; Enable this option if you want to [generate project docs](/docs/collaborate/build-and-view-your-docs) when this job runs. This option is disabled by default since most teams do not want to test doc generation on every CI check.

4. (optional) Options in the **Advanced settings** section:
1. (optional) Options in the **Advanced settings** section:
- **Environment variables** &mdash; Define [environment variables](/docs/build/environment-variables) to customize the behavior of your project when this CI job runs. You can specify that a CI job is running in a _Staging_ or _CI_ environment by setting an environment variable and modifying your project code to behave differently, depending on the context. It's common for teams to process only a subset of data for CI runs, using environment variables to branch logic in their dbt project code.
- **Target name** &mdash; Define the [target name](/docs/build/custom-target-names). Similar to **Environment Variables**, this option lets you customize the behavior of the project. You can use this option to specify that a CI job is running in a _Staging_ or _CI_ environment by setting the target name and modifying your project code to behave differently, depending on the context.
- **Run timeout** &mdash; Cancel this CI job if the run time exceeds the timeout value. You can use this option to help ensure that a CI check doesn't consume too much of your warehouse resources.
- **dbt version** &mdash; By default, it’s set to inherit the [dbt version](/docs/dbt-versions/core) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior.
- **Threads** &mdash; By default, it’s set to 4 [threads](/docs/core/connect-data-platform/connection-profiles#understanding-threads). Increase the thread count to increase model execution concurrency.
- **Generate docs on run** &mdash; Enable this option if you want to [generate project docs](/docs/collaborate/build-and-view-your-docs) when this job runs. This option is disabled by default since most teams do not want to test doc generation on every CI check.
- **Run source freshness** &mdash; Enable this option to invoke the `dbt source freshness` command before running this CI job. Refer to [Source freshness](/docs/deploy/source-freshness) for more details.

### Examples

- Example of creating a CI job:
<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/create-ci-job.png" title="Example of CI Job page in dbt Cloud UI"/>

- Example of GitHub pull request. The green checkmark means the dbt build and tests were successful. Clicking on the dbt Cloud section navigates you to the relevant CI run in dbt Cloud.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-github-pr.png" title="GitHub pull request example"/>
### Example of GitHub pull request
The following is an example of GitHub pull request. The green checkmark means the dbt build and tests were successful. Clicking on the dbt Cloud section navigates you to the relevant CI run in dbt Cloud.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-github-pr.png" width="60%" title="GitHub pull request example"/>

## Trigger a CI job with the API

Expand Down
24 changes: 17 additions & 7 deletions website/docs/docs/deploy/continuous-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,7 @@ dbt Cloud deletes the temporary schema from your <Term id="data-warehouse" /> w

The [dbt Cloud scheduler](/docs/deploy/job-scheduler) executes CI jobs differently from other deployment jobs in these important ways:

- **Concurrent CI checks** &mdash; CI runs triggered by the same dbt Cloud CI job execute concurrently (in parallel), when appropriate
- **Smart cancellation of stale builds** &mdash; Automatically cancels stale, in-flight CI runs when there are new commits to the PR
- **Run slot treatment** &mdash; CI runs don't consume a run slot

### Concurrent CI checks
<Expandable alt_header="Concurrent CI checks">

When you have teammates collaborating on the same dbt project creating pull requests on the same dbt repository, the same CI job will get triggered. Since each run builds into a dedicated, temporary schema that’s tied to the pull request, dbt Cloud can safely execute CI runs _concurrently_ instead of _sequentially_ (differing from what is done with deployment dbt Cloud jobs). Because no one needs to wait for one CI run to finish before another one can start, with concurrent CI checks, your whole team can test and integrate dbt code faster.

Expand All @@ -44,12 +40,26 @@ Below describes the conditions when CI checks are run concurrently and when they
- CI runs with the _same_ PR number and _different_ commit SHAs execute serially because they’re building into the same schema. dbt Cloud will run the latest commit and cancel any older, stale commits. For details, refer to [Smart cancellation of stale builds](#smart-cancellation).
- CI runs with the same PR number and same commit SHA, originating from different dbt Cloud projects will execute jobs concurrently. This can happen when two CI jobs are set up in different dbt Cloud projects that share the same dbt repository.

### Smart cancellation of stale builds {#smart-cancellation}
</Expandable>

<Expandable alt_header="Smart cancellation of stale builds">

When you push a new commit to a PR, dbt Cloud enqueues a new CI run for the latest commit and cancels any CI run that is (now) stale and still in flight. This can happen when you’re pushing new commits while a CI build is still in process and not yet done. By cancelling runs in a safe and deliberate way, dbt Cloud helps improve productivity and reduce data platform spend on wasteful CI runs.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-smart-cancel-job.png" width="70%" title="Example of an automatically canceled run"/>

### Run slot treatment <Lifecycle status="team,enterprise" />
</Expandable>

<Expandable alt_header="Run slot treatment" lifecycle="team,enterprise">

CI runs don't consume run slots. This guarantees a CI check will never block a production run.

</Expandable>

## How Advanced CI works {#advanced-ci}

[PLACEHOLDER STUB for content]

- Compare changes LOREM IPSUM
- PR comments LOREM IPSUM

26 changes: 20 additions & 6 deletions website/docs/docs/deploy/run-visibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ You can view the history of your runs and the model timing dashboard to help ide

## Run history

The **Run history** dashboard in dbt Cloud helps you monitor the health of your dbt project. It provides a detailed overview of all of your project's job runs and empowers you with a variety of filters to help you focus on specific aspects. You can also use it to review recent runs, find errored runs, and track the progress of runs in progress. You can access it on the top navigation menu by clicking **Deploy** and then **Run history**.
The **Run history** dashboard in dbt Cloud helps you monitor the health of your dbt project. It provides a detailed overview of all of your project's job runs and empowers you with a variety of filters to help you focus on specific aspects. You can also use it to review recent runs, find errored runs, and track the progress of runs in progress. You can access it from the top navigation menu by clicking **Deploy** and then **Run history**.

The dashboard displays your full run history, including job name, status, associated environment, job trigger, commit SHA, schema, and timing info.

dbt Cloud developers can access their run history for the last 365 days through the dbt Cloud user interface (UI) and API.

We limit self-service retrieval of run history metadata to 365 days to improve dbt Cloud's performance.
dbt Labs limits self-service retrieval of run history metadata to 365 days to improve dbt Cloud's performance.

<Lightbox src="/img/docs/dbt-cloud/deployment/run-history.png" width="85%" title="Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more."/>

Expand All @@ -29,16 +29,30 @@ An example of a completed run with a configuration for a [job completion trigger

<Lightbox src="/img/docs/dbt-cloud/deployment/example-job-details.png" width="65%" title="Example of run details" />

### Access logs
### Run summary tab

You can view or download in-progress and historical logs for your dbt runs. This makes it easier for the team to debug errors more efficiently.
You can view or download in-progress and historical access logs for your dbt runs. This makes it easier for the team to debug errors more efficiently.

<Lightbox src="/img/docs/dbt-cloud/deployment/access-logs.gif" width="85%" title="Access logs for run steps" />

### Model timing <Lifecycle status="team,enterprise" />
### Lineage tab

The **Model timing** dashboard displays the composition, order, and time taken by each model in a job run. The visualization appears for successful jobs and highlights the top 1% of model durations. This helps you identify bottlenecks in your runs, so you can investigate them and potentially make changes to improve their performance.
[PLACEHOLDER STUB for content]

### Model timing tab <Lifecycle status="team,enterprise" />

The **Model timing** tab displays the composition, order, and time taken by each model in a job run. The visualization appears for successful jobs and highlights the top 1% of model durations. This helps you identify bottlenecks in your runs, so you can investigate them and potentially make changes to improve their performance.

You can find the dashboard on the [job's run details](#job-run-details).

<Lightbox src="/img/docs/dbt-cloud/model-timing.png" width="85%" title="The Model timing tab displays the top 1% of model durations and visualizes model bottlenecks" />

### Artifacts tab

[PLACEHOLDER STUB for content]

### Compare tab

[PLACEHOLDER STUB for content]

LOREM IPSOM add content about the different tabs, tables, Explorer link, etc
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 0115497

Please sign in to comment.