Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize backfill: Make backfill vnode level within each partition #14905

Open
Tracked by #10225
kwannoel opened this issue Jan 31, 2024 · 1 comment
Open
Tracked by #10225

Comments

@kwannoel
Copy link
Contributor

kwannoel commented Jan 31, 2024

We can read from storage at a vnode level, and spawn different threads for each vnode to handle backfilling.

This increases the concurrency of backfilling, and for finished vnode partitions, we no longer have to re-read s3 again.

@github-actions github-actions bot added this to the release-1.7 milestone Jan 31, 2024
@kwannoel kwannoel changed the title Parallelize state completion for arrangement backfill Parallelize backfill: Make backfill vnode level within each partition Mar 15, 2024
@kwannoel kwannoel removed this from the release-1.9 milestone May 14, 2024
Copy link
Contributor

github-actions bot commented Aug 1, 2024

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant