From bac8f0ac4efad94aaabecf8897d696d93ccff4a8 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 14 Mar 2024 15:55:35 +0000 Subject: [PATCH 1/6] add redshift and bq to supported adapters for freshness Resolves #5061 --- website/docs/reference/resource-properties/freshness.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 513544d61c7..c9220f26f98 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -45,12 +45,12 @@ If a source has a `freshness:` block, dbt will attempt to calculate freshness fo - If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (behavior prior to v1.7). - If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new in v1.7 on supported adapters). -Currently, calculating freshness from warehouse metadata tables is supported on: +Currently, calculating freshness from warehouse metadata tables is supported on the following adapters: - [Snowflake](/reference/resource-configs/snowflake-configs) - -Support is coming soon to the following adapters: - [Redshift](/reference/resource-configs/redshift-configs) - [BigQuery](/reference/resource-configs/bigquery-configs) +- +Support is coming soon to the following adapter: - [Spark](/reference/resource-configs/spark-configs) Freshness blocks are applied hierarchically: From d48d0a23c099d61e09567636b6d314cebe39bb40 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:07:26 +0000 Subject: [PATCH 2/6] Update freshness.md --- website/docs/reference/resource-properties/freshness.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index c9220f26f98..c1cbfddf708 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -49,12 +49,12 @@ Currently, calculating freshness from warehouse metadata tables is supported on - [Snowflake](/reference/resource-configs/snowflake-configs) - [Redshift](/reference/resource-configs/redshift-configs) - [BigQuery](/reference/resource-configs/bigquery-configs) -- + Support is coming soon to the following adapter: - [Spark](/reference/resource-configs/spark-configs) Freshness blocks are applied hierarchically: -- a `freshness` and `loaded_at_field` property added to a source will be applied to all all tables defined in that source +- a `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source - a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source. This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case. @@ -72,7 +72,7 @@ To exclude a source from freshness calculations, you have two options: Additionally, the `loaded_at_field` is required to calculate freshness for a table. If a `loaded_at_field` is not provided, then dbt will not calculate freshness for the table. Freshness blocks are applied hierarchically: -- a `freshness` and `loaded_at_field` property added to a source will be applied to all all tables defined in that source +- a `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source - a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source. This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case. @@ -117,7 +117,7 @@ This filter *only* applies to dbt's source freshness queries - it will not impac This is particularly useful if: - You are using BigQuery and your source tables are [partitioned tables](https://cloud.google.com/bigquery/docs/partitioned-tables) -- You are using Snowflake, Databricks or Spark with large tables, and this results in a performance benefit +- You are using Snowflake, Databricks, or Spark with large tables, and this results in a performance benefit ## Examples From 7c1cb4899f038cd1887b8e094a1d017dca6312bc Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Mar 2024 16:55:43 +0000 Subject: [PATCH 3/6] Update website/docs/reference/resource-properties/freshness.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/docs/reference/resource-properties/freshness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index c1cbfddf708..3b83395cbf3 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -50,7 +50,7 @@ Currently, calculating freshness from warehouse metadata tables is supported on - [Redshift](/reference/resource-configs/redshift-configs) - [BigQuery](/reference/resource-configs/bigquery-configs) -Support is coming soon to the following adapter: +Support is coming soon to the [Spark](/reference/resource-configs/spark-configs) adapter. - [Spark](/reference/resource-configs/spark-configs) Freshness blocks are applied hierarchically: From 3197da2afa3cbef793497ac5da654b3b358c131d Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Mar 2024 16:55:51 +0000 Subject: [PATCH 4/6] Update website/docs/reference/resource-properties/freshness.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/docs/reference/resource-properties/freshness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 3b83395cbf3..5ed5a00190b 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -72,7 +72,7 @@ To exclude a source from freshness calculations, you have two options: Additionally, the `loaded_at_field` is required to calculate freshness for a table. If a `loaded_at_field` is not provided, then dbt will not calculate freshness for the table. Freshness blocks are applied hierarchically: -- a `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source +- A `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source - a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source. This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case. From bd20ee0430eeb729af96a66c7cc71e8997c2f378 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Mar 2024 16:55:59 +0000 Subject: [PATCH 5/6] Update website/docs/reference/resource-properties/freshness.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/docs/reference/resource-properties/freshness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 5ed5a00190b..0d90b7cebc9 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -73,7 +73,7 @@ Additionally, the `loaded_at_field` is required to calculate freshness for a tab Freshness blocks are applied hierarchically: - A `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source -- a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source. +- A `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source. This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case. From 46b0ea807fc0fb51ab47eb8ab1f2ed77b6259f21 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 19 Mar 2024 12:56:20 -0400 Subject: [PATCH 6/6] Update website/docs/reference/resource-properties/freshness.md --- website/docs/reference/resource-properties/freshness.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/freshness.md b/website/docs/reference/resource-properties/freshness.md index 0d90b7cebc9..a41c909a4fc 100644 --- a/website/docs/reference/resource-properties/freshness.md +++ b/website/docs/reference/resource-properties/freshness.md @@ -51,7 +51,6 @@ Currently, calculating freshness from warehouse metadata tables is supported on - [BigQuery](/reference/resource-configs/bigquery-configs) Support is coming soon to the [Spark](/reference/resource-configs/spark-configs) adapter. -- [Spark](/reference/resource-configs/spark-configs) Freshness blocks are applied hierarchically: - a `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source