feat: try to remove parallel unit mapping in frontend #16205
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shanicky
force-pushed
the
peng/fe-remove-pu
branch
2 times, most recently
from
75c32cb to
738db9a
Compare
shanicky
changed the title
shanicky
force-pushed
the
peng/fe-remove-pu
branch
2 times, most recently
from
cce0081 to
55bdf78
Compare
shanicky
changed the title
9 tasks
shanicky
force-pushed
the
peng/fe-remove-pu
branch
from
55bdf78 to
be9203d
Compare
6 tasks
shanicky
force-pushed
the
peng/fe-remove-pu
branch
2 times, most recently
from
480a376 to
ff0cccb
Compare
Refactor fragment mappings and cleanup code Refactor worker_node, enhance vnode logic, update imports & mapping Refactor: Disable `FragmentParallelUnitMapping` in protobuf and Rust code Refactor/gen: Improve formatting in `risingwave_common` tests Removed proto comments; updated Rust imports and tests. Remove `FragmentParallelUnitMapping` import and extra lines in `observer_manager.rs` Refactor FragManager, cleanup proto fields & comments Remove commented-out alternative vnode placement code Refactor: Clean up imports in fragment and streaming_job Refactor worker mapping in join operation Refactor fields() output in db lib for brevity Refine comments on migration and worker ID terms
shanicky
force-pushed
the
peng/fe-remove-pu
branch
from
ff0cccb to
353adec
Compare
… & import management Signed-off-by: Shanicky Chen <[email protected]>
shanicky
force-pushed
the
peng/fe-remove-pu
branch
from
353adec to
724d037
Compare
chenzl25
reviewed
Apr 18, 2024
Yes, this PR simply removes the frontend's dependency on the parallel unit mapping. To fully eliminate the parallel unit, changes need to be made in both the scheduler and the scaling system, which will be addressed in updates #16167 and #16324. |
yezizp2012
approved these changes
Apr 22, 2024
| let fragment_mappings = get_fragment_mappings_by_jobs(&txn, streaming_jobs.clone()).await?; | ||
|
|
||
| let fragment_mappings = fragment_mappings |
There was a problem hiding this comment.
Can we provide a helper function to wrap it, since this part will be used as long as the fragment mappings is transformed?
src/meta/src/controller/catalog.rs
Outdated
| @@ -274,8 +278,30 @@ impl CatalogController { | |||
| .into_tuple() | |||
| .all(&txn) | |||
| .await?; | |||
|
|
|||
| let parallel_unit_to_worker = Self::get_parallel_unit_to_worker_map(&txn).await?; | |||
There was a problem hiding this comment.
Not related to this PR, we might need to cache it as well when we dully deprecate parallel units and maintain the fragment mapping in memory.
Signed-off-by: Shanicky Chen <[email protected]>
3 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
This PR attempts to remove the dependency on parallel unit mapping in the frontend and aims to minimize the occurrence of parallel units in the frontend as much as possible.
By replacing ParallelUnit with Worker, we can forcefully substitute ParallelUnitMapping with WorkerMapping. Essentially, the frontend locates the parallel unit through these vnodes, and then maps the parallel unit to the worker. Therefore, we can directly provide worker mapping.
Checklist
./risedev check(or alias,./risedev c)