Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: use 1 for vnode count of singletons #18753

Merged
merged 9 commits into from
Oct 25, 2024
Merged

Conversation

BugenZhao
Copy link
Member

@BugenZhao BugenZhao commented Sep 29, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This is a progress towards #15900.

Try to address #18515 (comment), i.e., use 1 for vnode count of singletons. There are several cases to consider:

  • Use 1 for vnode count for all singleton fragments or tables to create in the current version.
    • There can be (effectively) singleton table within a distributed fragment, typically the state table of Source executor. In this case, we still treat it as singleton and assign it with 1 as the vnode count.
    • As a result, the vnode_count of a state table is not always the same as the length of the vnode_bitmap of its associated fragment. This should be correctly handled while performing batch scans over them.
  • VnodeCountCompat::vnode_count should return 1 for structures persisted before, if they are believed to be singleton.
  • For SQL backend, correctly perform the migration to fill 1 for the vnode_count field of singletons. The logic here must be consistent with the one in VnodeCountCompat::vnode_count, as under SQL backend, we still have some table catalogs persisted in form of protobuf, like the internal table catalogs in plan nodes.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Copy link
Member Author

BugenZhao commented Sep 29, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @BugenZhao and the rest of your teammates on Graphite Graphite

@BugenZhao
Copy link
Member Author

This reveals a severe bug that we persist and notify about the internal tables before the vnode counts are correctly resolved, leading to inconsistency between the stream nodes and the catalog.

@BugenZhao
Copy link
Member Author

BugenZhao commented Sep 29, 2024

This reveals a severe bug

Will first fix in #18759

@BugenZhao BugenZhao force-pushed the bz/var-vnode-singleton-1 branch from b427b00 to 048b217 Compare September 29, 2024 09:38
@BugenZhao BugenZhao changed the base branch from bz/var-vnode-system-tables to bz/var-vnode-table-catalog-vnode-count September 29, 2024 09:38
@BugenZhao BugenZhao force-pushed the bz/var-vnode-table-catalog-vnode-count branch from eb7d2cf to beba88d Compare September 30, 2024 03:05
@BugenZhao BugenZhao force-pushed the bz/var-vnode-singleton-1 branch from 048b217 to 5716e51 Compare September 30, 2024 03:05
@BugenZhao BugenZhao force-pushed the bz/var-vnode-table-catalog-vnode-count branch from beba88d to 23a1d96 Compare October 8, 2024 05:40
@BugenZhao BugenZhao force-pushed the bz/var-vnode-singleton-1 branch from 5716e51 to 778ca89 Compare October 8, 2024 05:41
@BugenZhao BugenZhao changed the title refactor: use 1 for vnode count of singletons for backward compatibility refactor: use 1 for vnode count of singletons Oct 8, 2024
Base automatically changed from bz/var-vnode-table-catalog-vnode-count to main October 8, 2024 08:50
@BugenZhao BugenZhao force-pushed the bz/var-vnode-singleton-1 branch from 778ca89 to 602c622 Compare October 12, 2024 03:46
@BugenZhao BugenZhao changed the base branch from main to bz/tmp-hack-sqlite-mem-backend October 12, 2024 03:46
@BugenZhao BugenZhao force-pushed the bz/tmp-hack-sqlite-mem-backend branch from ccf26b4 to 8e6c87d Compare October 12, 2024 04:30
@BugenZhao BugenZhao force-pushed the bz/var-vnode-singleton-1 branch from 09fe68d to 94dadd9 Compare October 12, 2024 04:31
@BugenZhao BugenZhao marked this pull request as ready for review October 12, 2024 04:34
@BugenZhao BugenZhao requested a review from chenzl25 October 12, 2024 06:34
@BugenZhao BugenZhao force-pushed the bz/var-vnode-singleton-1 branch from 8451b92 to 3e73343 Compare October 23, 2024 05:50
@BugenZhao
Copy link
Member Author

This PR has been significantly reworked. Please review it again based on the updated PR body. 🥰

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can get this PR into release 2.1, we don't need a new migration step but only to update this existing one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decide to include this PR in v2.2 instead. Switch to a new migration file.

Base automatically changed from bz/var-vnode-check-replace to main October 25, 2024 04:17
@BugenZhao BugenZhao force-pushed the bz/var-vnode-singleton-1 branch from 3e73343 to 0a9cb10 Compare October 25, 2024 06:07
@BugenZhao BugenZhao enabled auto-merge October 25, 2024 06:42
@BugenZhao BugenZhao added this pull request to the merge queue Oct 25, 2024
Merged via the queue into main with commit 58ecec2 Oct 25, 2024
33 of 35 checks passed
@BugenZhao BugenZhao deleted the bz/var-vnode-singleton-1 branch October 25, 2024 07:49
BugenZhao added a commit that referenced this pull request Oct 28, 2024
github-merge-queue bot pushed a commit that referenced this pull request Oct 28, 2024
github-merge-queue bot pushed a commit that referenced this pull request Oct 28, 2024
BugenZhao added a commit that referenced this pull request Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants