From 3a389b8ba6c65d974d05ae679ce51c698ed00bf4 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 18 Jan 2024 14:52:14 -0500 Subject: [PATCH 1/7] Updating freshness ref page --- .../resource-properties/freshness.md | 26 ++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index f332f5a1b8f..52733bd9611 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -37,6 +37,30 @@ A freshness block is used to define the acceptable amount of time between the mo In the `freshness` block, one or both of `warn_after` and `error_after` can be provided. If neither is provided, then dbt will not calculate freshness snapshots for the tables in this source. + + +Source freshness is calculated for all tables by default. + +If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: +- If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (behavior prior to v1.7). +- If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new in v1.7). + +To exclude a source from freshness calculations, you have two options: +- Don't add a `freshness:` block. +- Explicitly set `freshness: null`. + +Freshness blocks are applied hierarchically: +- A `freshness` property added to a source will be applied to all all tables defined in that source +- A `freshness` property added to a source _table_ will override any properties applied to the source. + +This is useful when all of the tables in a source have the same `loaded_at_field`. + +## loaded_at_field +(Optional) + + + + Additionally, the `loaded_at_field` is required to calculate freshness for a table. If a `loaded_at_field` is not provided, then dbt will not calculate freshness for the table. Freshness blocks are applied hierarchically: @@ -47,7 +71,7 @@ This is useful when all of the tables in a source have the same `loaded_at_field ## loaded_at_field (Required) - + A column name (or expression) that returns a timestamp indicating freshness. If using a date field, you may have to cast it to a timestamp: From 3dab9c18ddad3ca0f83db5f33a76d396cc38a9c3 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 18 Jan 2024 16:10:15 -0500 Subject: [PATCH 2/7] Update website/docs/reference/resource-properties/freshness.md --- website/docs/reference/resource-properties/freshness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 52733bd9611..09b1d2e75e6 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -50,7 +50,7 @@ To exclude a source from freshness calculations, you have two options: - Explicitly set `freshness: null`. Freshness blocks are applied hierarchically: -- A `freshness` property added to a source will be applied to all all tables defined in that source +- A `freshness` property added to a source will be applied to all tables defined in that source. - A `freshness` property added to a source _table_ will override any properties applied to the source. This is useful when all of the tables in a source have the same `loaded_at_field`. From 792cd82a2db73bd3e9d12ede27b0a89dc211e32f Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 18 Jan 2024 16:11:29 -0500 Subject: [PATCH 3/7] Update website/docs/reference/resource-properties/freshness.md --- website/docs/reference/resource-properties/freshness.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 09b1d2e75e6..47d38f80090 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -39,7 +39,6 @@ In the `freshness` block, one or both of `warn_after` and `error_after` can be p -Source freshness is calculated for all tables by default. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: - If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (behavior prior to v1.7). From 161463f3d91d5bfdbb4068e769a8da1638b7cf82 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 18 Jan 2024 16:13:58 -0500 Subject: [PATCH 4/7] Update website/docs/reference/resource-properties/freshness.md --- website/docs/reference/resource-properties/freshness.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 47d38f80090..745d1edd949 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -52,7 +52,6 @@ Freshness blocks are applied hierarchically: - A `freshness` property added to a source will be applied to all tables defined in that source. - A `freshness` property added to a source _table_ will override any properties applied to the source. -This is useful when all of the tables in a source have the same `loaded_at_field`. ## loaded_at_field (Optional) From 7dee599ce15ddc9bc79734b5088164f999e8407a Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 18 Jan 2024 16:38:43 -0500 Subject: [PATCH 5/7] Update freshness.md --- .../docs/reference/resource-properties/freshness.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 745d1edd949..0c07185fc04 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -39,10 +39,19 @@ In the `freshness` block, one or both of `warn_after` and `error_after` can be p +In most cases, the `loaded_at_field` is required. Some adapters support calculating source freshness from the warehouse metadata tables and can exclude the `loaded_at_field`. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: - If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (behavior prior to v1.7). -- If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new in v1.7). +- If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new in v1.7 on supported adapters). + +Currently, calculating freshness from warehouse metadata tables is supported on: +- [Snowflake](/reference/resource-configs/snowflake-configs) + +Support is coming soon to the following adapters: +- [Redshift](/reference/resource-configs/redshift-configs) +- [BigQuery](/reference/resource-configs/bigquery-configs) +- [Spark](/reference/resource-configs/spark-configs) To exclude a source from freshness calculations, you have two options: - Don't add a `freshness:` block. @@ -54,7 +63,7 @@ Freshness blocks are applied hierarchically: ## loaded_at_field -(Optional) +(Required. Optional on adaters that support pulling freshness from warehouse metadata tables.) From daa6876995d77bdc3f1018c0c7c2e09719344c8e Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 18 Jan 2024 16:41:20 -0500 Subject: [PATCH 6/7] Update freshness.md --- .../docs/reference/resource-properties/freshness.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 0c07185fc04..514854b217a 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -53,15 +53,16 @@ Support is coming soon to the following adapters: - [BigQuery](/reference/resource-configs/bigquery-configs) - [Spark](/reference/resource-configs/spark-configs) +Freshness blocks are applied hierarchically: +- a `freshness` and `loaded_at_field` property added to a source will be applied to all all tables defined in that source +- a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source. + +This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case. + To exclude a source from freshness calculations, you have two options: - Don't add a `freshness:` block. - Explicitly set `freshness: null`. -Freshness blocks are applied hierarchically: -- A `freshness` property added to a source will be applied to all tables defined in that source. -- A `freshness` property added to a source _table_ will override any properties applied to the source. - - ## loaded_at_field (Required. Optional on adaters that support pulling freshness from warehouse metadata tables.) From 6eccc9c5552227a0e8463a60c2064b345f0afa26 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Fri, 19 Jan 2024 09:51:45 -0500 Subject: [PATCH 7/7] Update website/docs/reference/resource-properties/freshness.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/resource-properties/freshness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 514854b217a..0b017991d68 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -64,7 +64,7 @@ To exclude a source from freshness calculations, you have two options: - Explicitly set `freshness: null`. ## loaded_at_field -(Required. Optional on adaters that support pulling freshness from warehouse metadata tables.) +(Optional on adapters that support pulling freshness from warehouse metadata tables, required otherwise.)