Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarification for incremental-microbatch.md #6260

Merged
merged 3 commits into from
Oct 8, 2024
Merged

Clarification for incremental-microbatch.md #6260

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dd31647
Clarification for incremental-microbatch.md
ernestoongaro Oct 8, 2024
df4d3c3
Update website/docs/docs/build/incremental-microbatch.md
mirnawong1 Oct 8, 2024
04b754e
Merge branch 'current' into feature/clarify_microbatch_config
mirnawong1 Oct 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions website/docs/docs/build/incremental-microbatch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,15 @@ A `sessions` model is aggregating and enriching data that comes from two other m

The `page_view_start` column in `page_views` is configured as that model's `event_time`. The `customers` model does not configure an `event_time`. Therefore, each batch of `sessions` will filter `page_views` to the equivalent time-bounded batch, and it will not filter `customers` (a full scan for every batch).

<File name="models/staging/page_views.yml">

```yaml
models:
- name: page_views
config:
event_time: page_view_start
```

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
We run the `sessions` model on October 1, 2024, and then again on October 2. It produces the following queries:

<Tabs>
Expand Down
Loading