Skip to content

Commit

Permalink
Merge branch 'current' into patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Dec 23, 2024
2 parents ea49a60 + 567d744 commit bdd3ca5
Show file tree
Hide file tree
Showing 299 changed files with 4,115 additions and 1,135 deletions.
12 changes: 6 additions & 6 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@
* @dbt-labs/product-docs

# Adapter & Package Development Docs
/website/docs/docs/supported-data-platforms.md @dbt-labs/product-docs @dataders
/website/docs/reference/warehouse-setups @dbt-labs/product-docs @dataders
/website/docs/docs/supported-data-platforms.md @dbt-labs/product-docs @amychen1776
/website/docs/reference/warehouse-setups @dbt-labs/product-docs @amychen1776
# `resource-configs` contains more than just warehouse setups
/website/docs/reference/resource-configs/*-configs.md @dbt-labs/product-docs @dataders
/website/docs/guides/advanced/adapter-development @dbt-labs/product-docs @dataders @dbeatty10
/website/docs/reference/resource-configs/*-configs.md @dbt-labs/product-docs @amychen1776
/website/docs/guides/advanced/adapter-development @dbt-labs/product-docs @amychen1776

/website/docs/guides/building-packages @dbt-labs/product-docs @amychen1776 @dataders @dbeatty10
/website/docs/guides/creating-new-materializations @dbt-labs/product-docs @dataders @dbeatty10
/website/docs/guides/building-packages @dbt-labs/product-docs @amychen1776
/website/docs/guides/creating-new-materializations @dbt-labs/product-docs

# Require approval from the Multicell team when making
# changes to the public facing migration documentation.
Expand Down
7 changes: 3 additions & 4 deletions .github/workflows/vale.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,11 @@ jobs:
python-version: '3.x'

- name: Install Vale
run: pip install vale==2.27.0 # Install a stable version of Vale
run: pip install vale==3.9.1.0 # Install a stable version of Vale

- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@v34
uses: tj-actions/changed-files@v45
with:
files: |
website/**/*.md
Expand Down Expand Up @@ -63,10 +63,9 @@ jobs:
uses: errata-ai/vale-action@reviewdog
with:
token: ${{ secrets.GITHUB_TOKEN }}
reporter: github-check
reporter: github-pr-review
files: ${{ steps.changed-files.outputs.all_changed_and_modified_files }}
separator: ' '
version: '2.27.0'

# - name: Post summary comment
# if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,5 @@ N/A
pseudocolumn
yml
values=
dbt v\\d+\\.\\d+
v\\d+\\.\\d+
File renamed without changes.
2 changes: 2 additions & 0 deletions styles/custom/Typos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,5 @@ tokens:
- '\w+/\w+|\w+-\w+|n/a'
- 'n/a'
- 'N/A'
- 'dbt v\\d+\\.\\d+'
- 'v\\d+\\.\\d+ '
3 changes: 2 additions & 1 deletion website/blog/2021-11-23-how-to-upgrade-dbt-versions.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ is_featured: true
It's been a few years since dbt-core turned 1.0! Since then, we've committed to releasing zero breaking changes whenever possible and it's become much easier to upgrade dbt Core versions.

In 2024, we're taking this promise further by:

- Stabilizing interfaces for everyone — adapter maintainers, metadata consumers, and (of course) people writing dbt code everywhere — as discussed in [our November 2023 roadmap update](https://github.com/dbt-labs/dbt-core/blob/main/docs/roadmap/2023-11-dbt-tng.md).
- Introducing **Versionless** in dbt Cloud. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. For more details, refer to [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud).
- Introducing [Release tracks](/docs/dbt-versions/cloud-release-tracks) (formerly known as Versionless) to dbt Cloud. No more manual upgrades and no need for _a second sandbox project_ just to try out new features in development. For more details, refer to [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud).

We're leaving the rest of this post as is, so we can all remember how it used to be. Enjoy a stroll down memory lane.

Expand Down
2 changes: 1 addition & 1 deletion website/blog/2022-04-14-add-ci-cd-to-bitbucket.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ Reading the file over, you can see that we:

In summary, anytime anything is pushed to main, we’ll ensure our production database reflects the dbt transformation, and we’ve saved the resulting artifacts to defer to.

> ❓ **What are artifacts and why should I defer to them?** dbt artifacts are metadata of the last run - what models and tests were defined, which ones ran successfully, and which failed. If a future dbt run is set to ***defer*** to this metadata, it means that it can select models and tests to run based on their state, including and especially their difference from the reference metadata. See [Artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts), [Selection methods: “state”](https://docs.getdbt.com/reference/node-selection/methods#the-state-method), and [Caveats to state comparison](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats) for details.
> ❓ **What are artifacts and why should I defer to them?** dbt artifacts are metadata of the last run - what models and tests were defined, which ones ran successfully, and which failed. If a future dbt run is set to ***defer*** to this metadata, it means that it can select models and tests to run based on their state, including and especially their difference from the reference metadata. See [Artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts), [Selection methods: “state”](https://docs.getdbt.com/reference/node-selection/methods#state), and [Caveats to state comparison](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats) for details.

### Slim Continuous Integration: Retrieve the artifacts and do a state-based run

Expand Down
2 changes: 1 addition & 1 deletion website/blog/2024-04-22-extended-attributes.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ All you need to do is configure an environment as staging and enable the **Defer

## Upgrading on a curve

Lastly, let’s consider a more specialized use case. Imagine we have a "tiger team" (consisting of a lone analytics engineer named Dave) tasked with upgrading from dbt version 1.6 to the new **Versionless** setting, to take advantage of added stability and feature access. We want to keep the rest of the data team being productive in dbt 1.6 for the time being, while enabling Dave to upgrade and do his work in the new versionless mode.
Lastly, let’s consider a more specialized use case. Imagine we have a "tiger team" (consisting of a lone analytics engineer named Dave) tasked with upgrading from dbt version 1.6 to the new **[Latest release track](/docs/dbt-versions/cloud-release-tracks)**, to take advantage of new features and performance improvements. We want to keep the rest of the data team being productive in dbt 1.6 for the time being, while enabling Dave to upgrade and do his work with Latest (and greatest) dbt.

### Development environment

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "How we're making sure you can confidently go \"Versionless\" in dbt Cloud"
title: "How we're making sure you can confidently switch to the \"Latest\" release track in dbt Cloud"
description: "Over the past 6 months, we've laid a stable foundation for continuously improving dbt."
slug: latest-dbt-stability

Expand All @@ -12,23 +12,27 @@ date: 2024-05-02
is_featured: true
---

import Latest from '/snippets/_release-stages-from-versionless.md'

<Latest/>

As long as dbt Cloud has existed, it has required users to select a version of dbt Core to use under the hood in their jobs and environments. This made sense in the earliest days, when dbt Core minor versions often included breaking changes. It provided a clear way for everyone to know which version of the underlying runtime they were getting.

However, this came at a cost. While bumping a project's dbt version *appeared* as simple as selecting from a dropdown, there was real effort required to test the compatibility of the new version against existing projects, package dependencies, and adapters. On the other hand, putting this off meant foregoing access to new features and bug fixes in dbt.

But no more. Today, we're ready to announce the general availability of a new option in dbt Cloud: [**"Versionless."**](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless)
But no more. Today, we're ready to announce the general availability of a new option in dbt Cloud: [**the "Latest" release track.**](/docs/dbt-versions/cloud-release-tracks)

<!--truncate-->

For customers, this means less maintenance overhead, faster access to bug fixes and features, and more time to focus on what matters most: building trusted data products. This will be our stable foundation for improvement and innovation in dbt Cloud.

But we wanted to go a step beyond just making this option available to you. In this blog post, we aim to shed a little light on the extensive work we've done to ensure that using "Versionless" is a stable, reliable experience for the thousands of customers who rely daily on dbt Cloud.
But we wanted to go a step beyond just making this option available to you. In this blog post, we aim to shed a little light on the extensive work we've done to ensure that using the "Latest" release track is a stable and reliable experience for the thousands of customers who rely daily on dbt Cloud.

## How we safely deploy dbt upgrades to Cloud

We've put in place a rigorous, best-in-class suite of tests and control mechanisms to ensure that all changes to dbt under the hood are fully vetted before they're deployed to customers of dbt Cloud.

This pipeline has in fact been in place since January! It's how we've already been shipping continuous changes to the hundreds of customers who've selected "Versionless" while it's been in Beta and Preview. In that time, this process has enabled us to prevent multiple regressions before they were rolled out to any customers.
This pipeline has in fact been in place since January! It's how we've already been shipping continuous changes to the hundreds of customers who've selected the "Latest" release track while it's been in Beta and Preview. In that time, this process has enabled us to prevent multiple regressions before they were rolled out to any customers.

We're very confident in the robustness of this process**. We also know that we'll need to continue building trust with time.** We're sharing details about this work in the spirit of transparency and to build that trust.

Expand Down Expand Up @@ -82,9 +86,9 @@ All incidents are retrospected to make sure we not only identify and fix the roo

:::

The outcome of this process is that, when you select "Versionless" in dbt Cloud, the time between an improvement being made to dbt Core and you *safely* getting access to it in your projects is a matter of days — rather than months of waiting for the next dbt Core release, on top of any additional time it may have taken to actually carry out the upgrade.
The outcome of this process is that, when you select the "Latest" release track in dbt Cloud, the time between an improvement being made to dbt Core and you *safely* getting access to it in your projects is a matter of days — rather than months of waiting for the next dbt Core release, on top of any additional time it may have taken to actually carry out the upgrade.

We’re pleased to say that since the beta launch of “Versionless” in dbt Cloud in March, **we have not had any functional regressions reach customers**, while we’ve also been shipping multiple improvements to dbt functionality every day. This is a foundation that we aim to build on for the foreseeable future.
We’re pleased to say that, at the time of writing (May 2, 2024), since the beta launch of the "Latest" release track in dbt Cloud in March, **we have not had any functional regressions reach customers**, while we’ve also been shipping multiple improvements to dbt functionality every day. This is a foundation that we aim to build on for the foreseeable future.

## Stability as a feature

Expand All @@ -98,7 +102,7 @@ The adapter interface — i.e. how dbt Core actually connects to a third-party d

To solve that, we've released a new set of interfaces that are entirely independent of the `dbt-core` library: [`dbt-adapters==1.0.0`](https://github.com/dbt-labs/dbt-adapters). From now on, any changes to `dbt-adapters` will be backward and forward-compatible. This also decouples adapter maintenance from the regular release cadence of dbt Core — meaning maintainers get full control over when they ship implementations of new adapter-powered features.

Note that adapters running in dbt Cloud **must** be [migrated to the new decoupled architecture](https://github.com/dbt-labs/dbt-adapters/discussions/87) as a baseline in order to support the new "Versionless" option.
Note that adapters running in dbt Cloud **must** be [migrated to the new decoupled architecture](https://github.com/dbt-labs/dbt-adapters/discussions/87) as a baseline in order to support the new "Latest" release track.

### Managing behavior changes: stability as a feature

Expand All @@ -118,7 +122,7 @@ We’ve now [formalized our development best practices](https://github.com/dbt-l

In conclusion, we’re putting a lot of new muscle behind our commitments to dbt Cloud customers, the dbt Community, and the broader ecosystem:

- **Continuous updates**: "Versionless" dbt Cloud simplifies the update process, ensuring you always have the latest features and bug fixes without the maintenance overhead.
- **Continuous updates**: The "Latest" release track in dbt Cloud simplifies the update process, ensuring you always have the latest features and bug fixes without the maintenance overhead.
- **A rigorous new testing and deployment process**: Our new testing pipeline ensures that every update is carefully vetted against documented interfaces, Cloud-supported adapters, and popular packages before it reaches you. This process minimizes the risk of regressions — and has now been successful at entirely preventing them for hundreds of customers over multiple months.
- **A commitment to stability**: We’ve reworked our approaches to adapter interfaces, behaviour change management, and metadata artifacts to give you more stability and control.

Expand Down
14 changes: 7 additions & 7 deletions website/blog/2024-06-12-putting-your-dag-on-the-internet.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ date: 2024-06-14
is_featured: true
---

**New in dbt: allow Snowflake Python models to access the internet**
## New in dbt: allow Snowflake Python models to access the internet

With dbt 1.8, dbt released support for Snowflake’s [external access integrations](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) further enabling the use of dbt + AI to enrich your data. This allows querying of external APIs within dbt Python models, a functionality that was required for dbt Cloud customer, [EQT AB](https://eqtgroup.com/). Learn about why they needed it and how they helped build the feature and get it shipped!

Expand Down Expand Up @@ -45,7 +45,7 @@ This API is open and if it requires an API key, handle it similarly to managing
For simplicity’s sake, we will show how to create them using [pre-hooks](/reference/resource-configs/pre-hook-post-hook) in a model configuration yml file:


```
```yml
models:
- name: external_access_sample
config:
Expand All @@ -57,7 +57,7 @@ models:
Then we can simply use the new external_access_integrations configuration parameter to use our network rule within a Python model (called external_access_sample.py):
```
```python
import snowflake.snowpark as snowpark
def model(dbt, session: snowpark.Session):
dbt.config(
Expand All @@ -75,7 +75,7 @@ def model(dbt, session: snowpark.Session):
The result is a model with some json I can parse, for example, in a SQL model to extract some information:


```
```sql
{{
config(
materialized='incremental',
Expand Down Expand Up @@ -108,12 +108,12 @@ The result is a model that will keep track of dbt invocations, and the current U

This is a very new area to Snowflake and dbt -- something special about SQL and dbt is that it’s very resistant to external entropy. The second we rely on API calls, Python packages and other external dependencies, we open up to a lot more external entropy. APIs will change, break, and your models could fail.

Traditionally dbt is the T in ELT (dbt overview [here](https://docs.getdbt.com/terms/elt)), and this functionality unlocks brand new EL capabilities for which best practices do not yet exist. What’s clear is that EL workloads should be separated from T workloads, perhaps in a different modeling layer. Note also that unless using incremental models, your historical data can easily be deleted. dbt has seen a lot of use cases for this, including this AI example as outlined in this external [engineering blog post](https://klimmy.hashnode.dev/enhancing-your-dbt-project-with-large-language-models).
Traditionally dbt is the T in ELT (dbt overview [here](https://docs.getdbt.com/terms/elt)), and this functionality unlocks brand new EL capabilities for which best practices do not yet exist. What’s clear is that EL workloads should be separated from T workloads, perhaps in a different modeling layer. Note also that unless using incremental models, your historical data can easily be deleted. dbt has seen a lot of use cases for this, including this AI example as outlined in this external [engineering blog post](https://klimmy.hashnode.dev/enhancing-your-dbt-project-with-large-language-models).

**A few words about the power of Commercial Open Source Software**
## A few words about the power of Commercial Open Source Software

In order to get this functionality shipped quickly, EQT opened a pull request, Snowflake helped with some problems we had with CI and a member of dbt Labs helped write the tests and merge the code in!

dbt now features this functionality in dbt 1.8+ or the “Versionless” option of dbt Cloud (dbt overview [here](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless)).
dbt now features this functionality in dbt 1.8+ and all [Release tracks](/docs/dbt-versions/cloud-release-tracks) in dbt Cloud.

dbt Labs staff and community members would love to chat more about it in the [#db-snowflake](https://getdbt.slack.com/archives/CJN7XRF1B) slack channel.
Loading

0 comments on commit bdd3ca5

Please sign in to comment.