Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avoid pinning a hummock snapshot for mv creation #8810

Closed
hzxa21 opened this issue Mar 28, 2023 · 1 comment
Closed

avoid pinning a hummock snapshot for mv creation #8810

hzxa21 opened this issue Mar 28, 2023 · 1 comment
Assignees
Milestone

Comments

@hzxa21
Copy link
Collaborator

hzxa21 commented Mar 28, 2023

Currently, we will pin a hummock snapshot on mv creation and unpin it when mv creation completes.

IIUC, after #6341, we no longger need to pin hummock snapshot since the backfill executor always scan the latest snapshot

@github-actions github-actions bot added this to the release-0.19 milestone Mar 28, 2023
@wcy-fdu wcy-fdu self-assigned this Apr 4, 2023
@wcy-fdu
Copy link
Contributor

wcy-fdu commented Apr 10, 2023

Previously, we introduced a SnapshotManager to pin/unpin a shapshot when creating materialize view, which fix #5592. And later, with the implementation of #6341, chain/backfill executor will execute NoWait read, and seems that we don't need to pin shapshot anymore.

However, there is an issues: current implementation require strong binding between mv and chain. If we decouple chain and mv, chain executor need to wait epoch, but it may cause live lock.

If barrier 1 is creating mv, chain need to wait epoch 1. But if chain can wait epoch 1, epoch 1 must have flowed through the global. So actually chain buffer all the data behind epoch 1, including upstream change. And this is conflict with concurrent checkpoint.

I'm not sure whether will we decouple chain and materialize executor, so I'd like to draft a PR to avoid pin snapshot, and to ensure correctness.

It's ok to remove snapshot manager in barrier now.

@wcy-fdu wcy-fdu closed this as completed Apr 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants