From b1f9c688f04ef473670014cad9bf448157c1f65e Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 29 Oct 2024 23:18:15 +0000 Subject: [PATCH] add microbatch updates (#6351) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit this pr adds updates to incremental microbatch per core prs: - [#10878](https://github.com/dbt-labs/dbt-core/pull/10878) - makes it so --event-time-start adn --event-time-end are mutually required. - [#10876](https://github.com/dbt-labs/dbt-core/pull/10876) - changes lookback default window to 1 (from 0) [ X ] dbt Core PRs must get merged first before docs pr is merged --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-update-microbatch-dbt-labs.vercel.app/docs/build/incremental-microbatch --- website/docs/docs/build/incremental-microbatch.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/build/incremental-microbatch.md b/website/docs/docs/build/incremental-microbatch.md index 38e0edddfd6..6d80007e2d8 100644 --- a/website/docs/docs/build/incremental-microbatch.md +++ b/website/docs/docs/build/incremental-microbatch.md @@ -165,7 +165,7 @@ Several configurations are relevant to microbatch models, and some are required: | `event_time` | Column (required) | The column indicating "at what time did the row occur." Required for your microbatch model and any direct parents that should be filtered. | N/A | | `begin` | Date (required) | The "beginning of time" for the microbatch model. This is the starting point for any initial or full-refresh builds. For example, a daily-grain microbatch model run on `2024-10-01` with `begin = '2023-10-01` will process 366 batches (it's a leap year!) plus the batch for "today." | N/A | | `batch_size` | String (required) | The granularity of your batches. Supported values are `hour`, `day`, `month`, and `year` | N/A | -| `lookback` | Integer (optional) | Process X batches prior to the latest bookmark to capture late-arriving records. | `0` | +| `lookback` | Integer (optional) | Process X batches prior to the latest bookmark to capture late-arriving records. | `1` | @@ -192,12 +192,15 @@ During standard incremental runs, dbt will process batches according to the curr Whether to fix erroneous source data or retroactively apply a change in business logic, you may need to reprocess a large amount of historical data. -Backfilling a microbatch model is as simple as selecting it to run or build, and specifying a "start" and "end" for `event_time`. As always, dbt will process the batches between the start and end as independent queries. +Backfilling a microbatch model is as simple as selecting it to run or build, and specifying a "start" and "end" for `event_time`. Note that `--event-time-start` and `--event-time-end` are mutually necessary, meaning that if you specify one, you must specify the other. + +As always, dbt will process the batches between the start and end as independent queries. ```bash dbt run --event-time-start "2024-09-01" --event-time-end "2024-09-04" ``` + ### Retry