[Applied State] Implement batch strategy for metadata-based source freshness computation

[MichelleArk]

Housekeeping

• I am a maintainer of dbt-adapters

Short description

It should be possible to provide the Adapter object a list of sources (BaseRelation), and receive a mapping of BaseRelation to FreshnessResponses, if there is a performant (faster than calling existing metadata support sequentially) means for the adapter to do so (indicated by TableLastModifiedMetadataBatch). If there is no performant implementation possible, the adapter should be able to indicate this via TableLastModifiedMetadataBatch -- not supported.

Context

Core Epic: dbt-labs/dbt-core#9425
Core Issue: dbt-labs/dbt-core#8705

Acceptance criteria

Tasks

Preview Give feedback

1. A new Capability, TableLastModifiedMetadataBatch exists and can be used to infer whether the adapter supports a performant batch alternative to TableLastModifiedMetadata
2. A new method on the BaseAdapter, calculate_freshness_from_metadata_batch is implemented that accepts a list of BaseRelation sources and returns a mapping of the BaseRelation to the FreshnessResult for the source

Testing

Tasks

Preview Give feedback

1. Add tests for adapters that can support TableLastModifiedMetadataBatch
2. existing TableLastModifiedMetadata capability should not be modified as verified by existing tests

Security

N/A

Docs

N/A

Consequences

Core Issue: dbt-labs/dbt-core#8705

[adamcunnington-mlg]

Hello! So glad to see this issue progressing as part of the overall FR. I just wanted to check whether it is likely that this (and the downstream BQ implementation) will be backported to v1.8 in time for June?

I'm currently planning a hands-on week for my cross-europe team to meet in person and carry out a number of upgrades. It sure would be awesome if we could add this to the list. Our hourly source freshness now taking circa 15 mins - I think this feature will bring it closer to 15 seconds!

Thanks