Skip to content

Commit

Permalink
Merge branch 'main' into chore/debug-lookup-join
Browse files Browse the repository at this point in the history
  • Loading branch information
yezizp2012 authored Jun 11, 2024
2 parents 9b1c79e + e6b8e12 commit b7046f8
Showing 1 changed file with 16 additions and 0 deletions.
16 changes: 16 additions & 0 deletions docs/backfill.md
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,22 @@ and arrangement backfill will consume this historical data snapshot:
| 1 | 'Jack' | 29 |
| 2 | 'Jill' | 30 |

#### Initialization

Something to note is that for the first snapshot,
upstream may not have finished committing data in that epoch to s3.

Additionally, we have not replicated any upstream records
during that epoch, only in the subsequent ones.

As such, we must wait for that first checkpoint to be committed,
before reading, or we risk missing the uncommitted data in our backfill.

This is supported internally inside `init_epoch` for replicated state table.
```shell
upstream_table.init_epoch(first_epoch).await?;
```

### Recovery

TODO
Expand Down

0 comments on commit b7046f8

Please sign in to comment.