Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize the snapshot read of CDC backfill #15812

Closed
Tracked by #11079
StrikeW opened this issue Mar 20, 2024 · 0 comments
Closed
Tracked by #11079

Optimize the snapshot read of CDC backfill #15812

StrikeW opened this issue Mar 20, 2024 · 0 comments
Assignees
Milestone

Comments

@StrikeW
Copy link
Contributor

StrikeW commented Mar 20, 2024

Existing implementation of cdc backfill will start a new snapshot upon receiving a barrier message. And to ensure make progress of the backfill, we guarantee to consume and emit a snapshot chunk for each epoch, which could cause barrier piles up in some scenario for example the wide table use case.

image

As comparison, here is the metric that we alter the barrier interval to 5 mins instead of apply the patch #15684, the barrier doesn't pile up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant