Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fast-follows for CI phase 2 #4077

Merged
merged 73 commits into from
Oct 2, 2023
Merged
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
a4607b4
Fast-follows for CI phase 2
nghi-ly Sep 14, 2023
01a907a
This branch was auto-updated!
github-actions[bot] Sep 15, 2023
0003368
This branch was auto-updated!
github-actions[bot] Sep 15, 2023
21d6be1
This branch was auto-updated!
github-actions[bot] Sep 18, 2023
f0e6fe5
This branch was auto-updated!
github-actions[bot] Sep 18, 2023
5f48e3d
This branch was auto-updated!
github-actions[bot] Sep 18, 2023
53046aa
This branch was auto-updated!
github-actions[bot] Sep 18, 2023
60a73a2
This branch was auto-updated!
github-actions[bot] Sep 18, 2023
d44463a
This branch was auto-updated!
github-actions[bot] Sep 19, 2023
2881c20
This branch was auto-updated!
github-actions[bot] Sep 19, 2023
a8e4b6d
This branch was auto-updated!
github-actions[bot] Sep 19, 2023
b7039f2
This branch was auto-updated!
github-actions[bot] Sep 19, 2023
63361d7
This branch was auto-updated!
github-actions[bot] Sep 19, 2023
56b37c6
This branch was auto-updated!
github-actions[bot] Sep 20, 2023
4f099f0
Fold in feedback
nghi-ly Sep 20, 2023
4ec1588
Update website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-u…
nghi-ly Sep 20, 2023
455f2d8
Reword last bullet a bit to capture nuance
nghi-ly Sep 20, 2023
2841ef2
This branch was auto-updated!
github-actions[bot] Sep 20, 2023
7cae387
This branch was auto-updated!
github-actions[bot] Sep 21, 2023
86bbd63
This branch was auto-updated!
github-actions[bot] Sep 21, 2023
e821742
This branch was auto-updated!
github-actions[bot] Sep 21, 2023
f795460
This branch was auto-updated!
github-actions[bot] Sep 22, 2023
9a074aa
This branch was auto-updated!
github-actions[bot] Sep 22, 2023
25fb785
This branch was auto-updated!
github-actions[bot] Sep 22, 2023
ff7a565
This branch was auto-updated!
github-actions[bot] Sep 22, 2023
c0ae97a
This branch was auto-updated!
github-actions[bot] Sep 22, 2023
a42651b
This branch was auto-updated!
github-actions[bot] Sep 25, 2023
6327069
This branch was auto-updated!
github-actions[bot] Sep 25, 2023
29603ea
This branch was auto-updated!
github-actions[bot] Sep 25, 2023
25e04c7
This branch was auto-updated!
github-actions[bot] Sep 25, 2023
eec1efd
This branch was auto-updated!
github-actions[bot] Sep 25, 2023
c98939c
This branch was auto-updated!
github-actions[bot] Sep 25, 2023
6670a92
This branch was auto-updated!
github-actions[bot] Sep 26, 2023
355a14b
This branch was auto-updated!
github-actions[bot] Sep 26, 2023
77077f6
This branch was auto-updated!
github-actions[bot] Sep 26, 2023
fdee0f7
This branch was auto-updated!
github-actions[bot] Sep 26, 2023
68fc25f
This branch was auto-updated!
github-actions[bot] Sep 27, 2023
ea6c84c
This branch was auto-updated!
github-actions[bot] Sep 27, 2023
4cab59c
This branch was auto-updated!
github-actions[bot] Sep 27, 2023
e5349b0
This branch was auto-updated!
github-actions[bot] Sep 27, 2023
9a1fe60
This branch was auto-updated!
github-actions[bot] Sep 27, 2023
8a439c3
This branch was auto-updated!
github-actions[bot] Sep 27, 2023
682910c
This branch was auto-updated!
github-actions[bot] Sep 27, 2023
c91640a
This branch was auto-updated!
github-actions[bot] Sep 27, 2023
d47b7a1
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
9d58e7a
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
315f5c6
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
9c5011b
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
74e3234
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
e006d74
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
3b462c8
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
eeb6d06
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
5b70e1a
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
89a221c
This branch was auto-updated!
github-actions[bot] Sep 28, 2023
c8c1d1f
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
1bbdc18
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
5137580
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
61f0103
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
ace2f38
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
8bc5067
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
171364d
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
9040ced
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
7d09535
This branch was auto-updated!
github-actions[bot] Sep 29, 2023
eb08049
This branch was auto-updated!
github-actions[bot] Oct 2, 2023
140871d
This branch was auto-updated!
github-actions[bot] Oct 2, 2023
c788416
This branch was auto-updated!
github-actions[bot] Oct 2, 2023
19a6f09
This branch was auto-updated!
github-actions[bot] Oct 2, 2023
3066aaf
This branch was auto-updated!
github-actions[bot] Oct 2, 2023
4e9b235
Add new screenshot
nghi-ly Oct 2, 2023
bd0a977
Update website/docs/docs/dbt-versions/release-notes/04-Sept-2023/ci-u…
nghi-ly Oct 2, 2023
a72bc88
This branch was auto-updated!
github-actions[bot] Oct 2, 2023
f36ec64
Minor formatting update
nghi-ly Oct 2, 2023
b790e86
Merge branch 'ly-docs-fast-follow-ci-phase2' of github.com:dbt-labs/d…
nghi-ly Oct 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
---
title: "Update: Improvements to dbt Cloud continuous integration"
description: "September 2023: dbt Cloud now has two types of jobs — deploy jobs and CI jobs — with streamlined setup and improved efficiency. "
description: "September 2023: dbt Cloud now has two types of jobs -- deploy jobs and CI jobs -- with streamlined setup and improved efficiency. "
sidebar_label: "Update: Improvements to dbt jobs"
tags: [Sept-2023, CI]
date: 2023-09-11
date: 2023-09-15
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved
sidebar_position: 10
---

dbt Cloud now has two distinct job types: [deploy jobs](/docs/deploy/deploy-jobs) for building production data assets, and [CI jobs](/docs/deploy/ci-jobs) for checking code changes. These jobs perform fundamentally different tasks so dbt Labs improved the setup experience with better defaults for each.
dbt Cloud now has two distinct job types: [deploy jobs](/docs/deploy/deploy-jobs) for building production data assets, and [continuous integration (CI) jobs](/docs/deploy/ci-jobs) for checking code changes. These jobs perform fundamentally different tasks so dbt Labs improved the setup experience with better defaults for each.

With two types of jobs, instead of one generic type, we can better guide you through the setup flow. Best practices are built into the default settings so you can go from curious to being set up in seconds.

<Lightbox src="/img/docs/release-notes/ci-job-setup.gif" title="Example of setting up a CI job"/>
<Lightbox src="/img/docs/release-notes/ci-job-setup.gif" width="60%" title="Example of setting up a CI job"/>

And, we now have more efficient state comparisons on CI checks: never waste a build or test on code that hasn’t been changed. We now diff between the Git PR code and what’s running in production more efficiently with the introduction of deferral to an environment versus a job. To learn more, refer to [Continuous integration in dbt Cloud](/docs/deploy/continuous-integration) and [Get started with continuous integration tests](/guides/orchestration/set-up-ci/overview).
And, we now have more efficient state comparisons on CI checks: never waste a build or test on code that hasn’t been changed. We now diff between the Git pull request (PR) code and what’s running in production more efficiently with the introduction of deferral to an environment versus a job. To learn more, refer to [Continuous integration in dbt Cloud](/docs/deploy/continuous-integration).

Below is a comparison table that describes how deploy jobs and CI jobs behave differently:

Expand All @@ -29,4 +29,14 @@ Below is a comparison table that describes how deploy jobs and CI jobs behave di

## What you need to update

If you previously set up a job using the [Create Job](/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint before September 11, 2023, you must re-create the job as described in [Trigger a CI job with the API](/docs/deploy/ci-jobs#trigger-a-ci-job-with-the-api). This is because you must set the `job_type` to be `ci`.
- If you want to set up a CI environment for your jobs, dbt Labs recommends that you create your CI job in a dedicated [deployment environment](/docs/deploy/deploy-environments#create-a-deployment-environment) that's connected to a staging database. To learn more about these environment best practices, refer to the guide [Get started with continuous integration tests](/guides/orchestration/set-up-ci/overview).

- If you had set up a CI job before October 2, 2023, the job might've been misclassified as a deploy job with this update. Below describes how to fix the job type:

- If you used the [Create Job](/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint but didn't set `"triggers":triggers.git_provider_webhook`, the job was misclassified as a deploy job and you must re-create it as described in [Trigger a CI job with the API](/docs/deploy/ci-jobs#trigger-a-ci-job-with-the-api).

- If you used the dbt Cloud UI but didn't enable the **Run on Pull Requests** option that was in the **Continuous Integration** (CI) tab, the job was misclassified as a deploy job and you must re-create it as described in [Set up CI jobs](/docs/deploy/ci-jobs#set-up-ci-jobs).

To check for the job type, review your CI jobs in dbt Cloud's [Run History](/docs/deploy/run-visibility#run-history) and check for the **CI Job** tag below the job name. If it doesn't have this tag, it was misclassified and you need to re-create the job.

<Lightbox src="/img/docs/release-notes/ci-job-tag.png" width="60%" title="Example of a correct CI job type"/>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading