Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add redshift and bq to supported adapters for freshness #5080

Merged
merged 34 commits into from
Mar 19, 2024
Merged
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
bac8f0a
add redshift and bq to supported adapters for freshness
mirnawong1 Mar 14, 2024
b4212f0
Merge branch 'current' into mirnawong1-patch-26
mirnawong1 Mar 14, 2024
66e20a0
Merge branch 'current' into mirnawong1-patch-26
mirnawong1 Mar 14, 2024
d48d0a2
Update freshness.md
mirnawong1 Mar 14, 2024
55dffc1
Merge branch 'current' into mirnawong1-patch-26
mirnawong1 Mar 15, 2024
ff1a1fe
This branch was auto-updated!
github-actions[bot] Mar 15, 2024
5b30075
This branch was auto-updated!
github-actions[bot] Mar 15, 2024
71a97fe
This branch was auto-updated!
github-actions[bot] Mar 15, 2024
3c19a9e
This branch was auto-updated!
github-actions[bot] Mar 15, 2024
9d00ba1
This branch was auto-updated!
github-actions[bot] Mar 15, 2024
0935f3d
This branch was auto-updated!
github-actions[bot] Mar 15, 2024
8c531cd
This branch was auto-updated!
github-actions[bot] Mar 15, 2024
4b74a0c
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
4abe3ef
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
74558c6
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
5a0dc19
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
b1bd0ff
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
8e543ee
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
18dab1b
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
e7db516
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
4d87c42
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
192fa05
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
bc8e284
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
5efea6a
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
9a8f8c9
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
326f075
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
84cb5f9
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
7c1cb48
Update website/docs/reference/resource-properties/freshness.md
mirnawong1 Mar 19, 2024
3197da2
Update website/docs/reference/resource-properties/freshness.md
mirnawong1 Mar 19, 2024
bd20ee0
Update website/docs/reference/resource-properties/freshness.md
mirnawong1 Mar 19, 2024
46b0ea8
Update website/docs/reference/resource-properties/freshness.md
matthewshaver Mar 19, 2024
99baaa2
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
8b0478b
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
e0e4f81
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions website/docs/reference/resource-properties/freshness.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,16 +45,15 @@ If a source has a `freshness:` block, dbt will attempt to calculate freshness fo
- If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (behavior prior to v1.7).
- If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new in v1.7 on supported adapters).

Currently, calculating freshness from warehouse metadata tables is supported on:
Currently, calculating freshness from warehouse metadata tables is supported on the following adapters:
- [Snowflake](/reference/resource-configs/snowflake-configs)

Support is coming soon to the following adapters:
- [Redshift](/reference/resource-configs/redshift-configs)
- [BigQuery](/reference/resource-configs/bigquery-configs)
- [Spark](/reference/resource-configs/spark-configs)

Support is coming soon to the [Spark](/reference/resource-configs/spark-configs) adapter.

Freshness blocks are applied hierarchically:
- a `freshness` and `loaded_at_field` property added to a source will be applied to all all tables defined in that source
- a `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source
- a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source.

This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case.
Expand All @@ -72,8 +71,8 @@ To exclude a source from freshness calculations, you have two options:
Additionally, the `loaded_at_field` is required to calculate freshness for a table. If a `loaded_at_field` is not provided, then dbt will not calculate freshness for the table.

Freshness blocks are applied hierarchically:
- a `freshness` and `loaded_at_field` property added to a source will be applied to all all tables defined in that source
- a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source.
- A `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source
- A `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source.

This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case.

Expand Down Expand Up @@ -117,7 +116,7 @@ This filter *only* applies to dbt's source freshness queries - it will not impac

This is particularly useful if:
- You are using BigQuery and your source tables are [partitioned tables](https://cloud.google.com/bigquery/docs/partitioned-tables)
- You are using Snowflake, Databricks or Spark with large tables, and this results in a performance benefit
- You are using Snowflake, Databricks, or Spark with large tables, and this results in a performance benefit


## Examples
Expand Down
Loading