From 9f8686a45c71409b7ab20eeae6e67c9b70f87ca4 Mon Sep 17 00:00:00 2001 From: Grace Goheen Date: Thu, 12 Oct 2023 10:44:36 -0600 Subject: [PATCH 1/5] updated 1.7 migration guide to call out a behavior change --- .../versions/00-upgrading-to-v1.7.md | 22 ++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index ba079e6a0fb..8a5659c7f2d 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -14,11 +14,22 @@ description: New features and changes in dbt Core v1.7 dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new). +### Behavior changes + +1.7 expands the amount of sources folks can configure freshness for. Previously, freshness was limited to sources that had a `loaded_at_field`; but now, freshness can be generated from warehouse metadata tables when available. + +As part of this change, the `loaded_at_field` is **no longer required** in order to generate source freshness. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: +- if a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior) +- if a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new!) + +This is technically a small behavior change comapared to previous versons. In order to _not_ calculate freshness for a source: +- don't add a `freshness:` block +- explicity set `freshness: null` + ## New and changed features and functionality - [`dbt docs generate`](/reference/commands/cmd-docs) now supports `--select` to generate documentation for a subset of your project. Currently available for Snowflake and Postgres only, but other adapters are coming soon. -- [Source freshness](/docs/deploy/source-freshness) can now be generated from warehouse metadata tables, currently snowflake only, but other adapters that have metadata tables are coming soon. If you configure source freshness without a `loaded_at_field`, dbt will try to determine freshness from warehouse metadata tables. -- The nodes dictionary in the `catalog.json` can now be "partial" if `dbt docs generate` is run with a selector. +- [Source freshness](/docs/deploy/source-freshness) can now be generated from warehouse metadata tables, currently Snowflake only, but other adapters that have metadata tables are coming soon. ### MetricFlow enhancements @@ -32,8 +43,9 @@ dbt Labs is committed to providing backward compatibility for all versions 1.x, - The [manifest](/reference/artifacts/manifest-json) schema version has been updated to v11. - The [run_results](/reference/artifacts/run-results-json) schema version has been updated to v5. -- Added [node attributes](/reference/artifacts/run-results-json) related to compilation (`compiled`, `compiled_code`, `relation_name`). - +- There were a handful of specific changes to the [catalog.json](/reference/artifacts/catalog-json): + - Added [node attributes](/reference/artifacts/run-results-json) related to compilation (`compiled`, `compiled_code`, `relation_name`) to the `catalog.json`. + - The nodes dictionary in the `catalog.json` can now be "partial" if `dbt docs generate` is run with a selector. ### Model governance @@ -49,4 +61,4 @@ dbt Core v1.5 introduced model governance which we're continuing to refine. v1. With these quick hits, you can now: - Configure a `delimiter` for a seed file. - Use packages with the same git repo and unique subdirectory. -- Moved the `date_spine` macro from dbt-utils to dbt-core. +- Access the `date_spine` macro directly from dbt-core (moved over from dbt-utils). From 6f78a15d1a51cc40eaff0333903e7f1831434e9d Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:04:51 -0400 Subject: [PATCH 2/5] Apply suggestions from code review --- .../guides/migration/versions/00-upgrading-to-v1.7.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index 8a5659c7f2d..60218969266 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -16,11 +16,11 @@ dbt Labs is committed to providing backward compatibility for all versions 1.x, ### Behavior changes -1.7 expands the amount of sources folks can configure freshness for. Previously, freshness was limited to sources that had a `loaded_at_field`; but now, freshness can be generated from warehouse metadata tables when available. +dbt Core v1.7 expands the amount of sources you can configure freshness for. Previously, freshness was limited to sources with a `loaded_at_field`; now, freshness can be generated from warehouse metadata tables when available. -As part of this change, the `loaded_at_field` is **no longer required** in order to generate source freshness. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: -- if a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior) -- if a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new!) +As part of this change, the `loaded_at_field` is no longer required to generate source freshness. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: +- If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior). +- If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior). This is technically a small behavior change comapared to previous versons. In order to _not_ calculate freshness for a source: - don't add a `freshness:` block From 1d57c34a7bac8c64e8af50eb10778d2a4f1da015 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:07:05 -0400 Subject: [PATCH 3/5] Apply suggestions from code review --- .../docs/guides/migration/versions/00-upgrading-to-v1.7.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index 60218969266..6a1d828bff4 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -22,9 +22,9 @@ As part of this change, the `loaded_at_field` is no longer required to generate - If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior). - If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior). -This is technically a small behavior change comapared to previous versons. In order to _not_ calculate freshness for a source: -- don't add a `freshness:` block -- explicity set `freshness: null` +This is a relatively small behavior change compared to previous versions. To exclude a source from calculating source freshness: +- Don't add a `freshness:` block. +- Explicitly set `freshness: null` ## New and changed features and functionality From 6237a8e340dc72e56a99f5d9744decafff33c667 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:10:23 -0400 Subject: [PATCH 4/5] Update website/docs/guides/migration/versions/00-upgrading-to-v1.7.md --- website/docs/guides/migration/versions/00-upgrading-to-v1.7.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index 6a1d828bff4..c92fdaeed4a 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -43,7 +43,7 @@ This is a relatively small behavior change compared to previous versions. To exc - The [manifest](/reference/artifacts/manifest-json) schema version has been updated to v11. - The [run_results](/reference/artifacts/run-results-json) schema version has been updated to v5. -- There were a handful of specific changes to the [catalog.json](/reference/artifacts/catalog-json): +- There are a few specific changes to the [catalog.json](/reference/artifacts/catalog-json): - Added [node attributes](/reference/artifacts/run-results-json) related to compilation (`compiled`, `compiled_code`, `relation_name`) to the `catalog.json`. - The nodes dictionary in the `catalog.json` can now be "partial" if `dbt docs generate` is run with a selector. From b771380baa23728f276b7e5fa342bf57c2d7ed70 Mon Sep 17 00:00:00 2001 From: Grace Goheen <53586774+graciegoheen@users.noreply.github.com> Date: Thu, 12 Oct 2023 12:20:56 -0600 Subject: [PATCH 5/5] Update website/docs/guides/migration/versions/00-upgrading-to-v1.7.md --- website/docs/guides/migration/versions/00-upgrading-to-v1.7.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md index c92fdaeed4a..a85dd0e69e0 100644 --- a/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md +++ b/website/docs/guides/migration/versions/00-upgrading-to-v1.7.md @@ -22,7 +22,7 @@ As part of this change, the `loaded_at_field` is no longer required to generate - If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior). - If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior). -This is a relatively small behavior change compared to previous versions. To exclude a source from calculating source freshness: +This is a relatively small behavior change, but worth calling out in case you notice that dbt is calculating freshness for _more_ sources than before. To exclude a source from freshness calculations, you have two options: - Don't add a `freshness:` block. - Explicitly set `freshness: null`