Skip to content

Commit

Permalink
add redshift and bq to supported adapters for freshness (#5080)
Browse files Browse the repository at this point in the history
this PR adds redshift and bigquery as adapters that support calculating
freshness from warehouse metadata tables.

Resolves #5061
  • Loading branch information
mirnawong1 authored Mar 19, 2024
2 parents b2d11a1 + e0e4f81 commit 976a908
Showing 1 changed file with 7 additions and 8 deletions.
15 changes: 7 additions & 8 deletions website/docs/reference/resource-properties/freshness.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,16 +45,15 @@ If a source has a `freshness:` block, dbt will attempt to calculate freshness fo
- If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (behavior prior to v1.7).
- If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new in v1.7 on supported adapters).

Currently, calculating freshness from warehouse metadata tables is supported on:
Currently, calculating freshness from warehouse metadata tables is supported on the following adapters:
- [Snowflake](/reference/resource-configs/snowflake-configs)

Support is coming soon to the following adapters:
- [Redshift](/reference/resource-configs/redshift-configs)
- [BigQuery](/reference/resource-configs/bigquery-configs)
- [Spark](/reference/resource-configs/spark-configs)

Support is coming soon to the [Spark](/reference/resource-configs/spark-configs) adapter.

Freshness blocks are applied hierarchically:
- a `freshness` and `loaded_at_field` property added to a source will be applied to all all tables defined in that source
- a `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source
- a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source.

This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case.
Expand All @@ -72,8 +71,8 @@ To exclude a source from freshness calculations, you have two options:
Additionally, the `loaded_at_field` is required to calculate freshness for a table. If a `loaded_at_field` is not provided, then dbt will not calculate freshness for the table.

Freshness blocks are applied hierarchically:
- a `freshness` and `loaded_at_field` property added to a source will be applied to all all tables defined in that source
- a `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source.
- A `freshness` and `loaded_at_field` property added to a source will be applied to all tables defined in that source
- A `freshness` and `loaded_at_field` property added to a source _table_ will override any properties applied to the source.

This is useful when all of the tables in a source have the same `loaded_at_field`, as is often the case.

Expand Down Expand Up @@ -117,7 +116,7 @@ This filter *only* applies to dbt's source freshness queries - it will not impac

This is particularly useful if:
- You are using BigQuery and your source tables are [partitioned tables](https://cloud.google.com/bigquery/docs/partitioned-tables)
- You are using Snowflake, Databricks or Spark with large tables, and this results in a performance benefit
- You are using Snowflake, Databricks, or Spark with large tables, and this results in a performance benefit


## Examples
Expand Down

0 comments on commit 976a908

Please sign in to comment.