Skip to content

Commit

Permalink
doc(stream): add docs on initializing the replicated state table (#17027
Browse files Browse the repository at this point in the history
)
  • Loading branch information
kwannoel authored Jun 11, 2024
1 parent 21babd5 commit e6b8e12
Showing 1 changed file with 16 additions and 0 deletions.
16 changes: 16 additions & 0 deletions docs/backfill.md
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,22 @@ and arrangement backfill will consume this historical data snapshot:
| 1 | 'Jack' | 29 |
| 2 | 'Jill' | 30 |

#### Initialization

Something to note is that for the first snapshot,
upstream may not have finished committing data in that epoch to s3.

Additionally, we have not replicated any upstream records
during that epoch, only in the subsequent ones.

As such, we must wait for that first checkpoint to be committed,
before reading, or we risk missing the uncommitted data in our backfill.

This is supported internally inside `init_epoch` for replicated state table.
```shell
upstream_table.init_epoch(first_epoch).await?;
```

### Recovery

TODO
Expand Down

0 comments on commit e6b8e12

Please sign in to comment.