Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Replace the parallel unit mapping with worker slot mapping in frontend. #16801

Merged
merged 32 commits into from
May 28, 2024
Merged
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
decfccf
Add WorkerMapping to protos; refactor to worker ID usage.
shanicky Apr 7, 2024
5947afb
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu-2
shanicky Apr 16, 2024
724d037
Refactor observer, notification, & catalog files for better structure…
shanicky Apr 16, 2024
0916ac8
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
shanicky Apr 17, 2024
8f7f9b3
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
shanicky Apr 18, 2024
e40d0c8
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
shanicky Apr 22, 2024
b4673e6
Refactor parallel unit-worker map method to utils module.
shanicky Apr 22, 2024
b50031c
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
shanicky Apr 22, 2024
3b48ac9
Deprecate fields in MetaSnapshot & reserve tags.
shanicky Apr 22, 2024
36937ac
Merge branch 'main' into peng/fe-remove-pu
shanicky Apr 24, 2024
6125cab
update proto
shanicky Apr 24, 2024
0c8b540
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
shanicky May 6, 2024
ff4e3fa
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
shanicky May 9, 2024
ca47e5b
Fixed typos, clean up imports and whitespace in Rust files
shanicky May 11, 2024
f107855
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
shanicky May 11, 2024
b6e2700
Refactor serving.rs for efficient worker-to-vnode mapping, update imp…
shanicky May 11, 2024
e3de9ff
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
shanicky May 13, 2024
78622ad
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
May 14, 2024
bc2b8c5
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu
May 16, 2024
40c8a7e
Refactor protobufs, Rust types, rename worker mappings
May 17, 2024
ee15f5d
Update vnode placement tests and unique counts
May 17, 2024
3f4ac77
Remove assert from LocalLookupJoinExecBuilder
May 17, 2024
93cb568
Refactor WorkerMapping to WorkerSlotMapping in protobuf and Rust code
May 17, 2024
5a0c927
Update proto defs, error msg, rename vars, type changes
May 17, 2024
f54bb6c
Remove typecast for ROOT_TASK_* constants
May 17, 2024
95ded68
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu-slot
May 17, 2024
b85ec7a
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu-slot
May 20, 2024
56eba87
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu-slot
May 21, 2024
366020b
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu-slot
May 27, 2024
b79c38b
Merge remote-tracking branch 'origin/main' into peng/fe-remove-pu-slot
May 27, 2024
22edf32
Refactor `catalog.rs` imports, drop `visit_stream_node_cont`
May 27, 2024
6eda8c6
Refactor: Simplify `visit_stream_git_node_cont_mut` import in `catalo…
May 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refactor catalog.rs imports, drop visit_stream_node_cont
Shanicky Chen committed May 27, 2024
commit 22edf32d3280e30461dc27b8cb563d30ab061805
5 changes: 3 additions & 2 deletions src/meta/src/controller/catalog.rs
Original file line number Diff line number Diff line change
@@ -20,8 +20,9 @@ use anyhow::anyhow;
use itertools::Itertools;
use risingwave_common::catalog::{TableOption, DEFAULT_SCHEMA_NAME, SYSTEM_SCHEMAS};
use risingwave_common::hash::ParallelUnitMapping;
use risingwave_common::util::stream_graph_visitor::visit_stream_node_cont;
use risingwave_common::util::stream_graph_visitor::visit_stream_node_cont_mut;
use risingwave_common::util::stream_graph_visitor::{
visit_stream_node_cont_mut,
};
use risingwave_common::{bail, current_cluster_version};
use risingwave_connector::source::UPSTREAM_SOURCE_KEY;
use risingwave_meta_model_v2::object::ObjectType;

Unchanged files with check annotations Beta

message TaskId {
string query_id = 1;
uint32 stage_id = 2;
uint64 task_id = 3;

Check failure on line 228 in proto/batch_plan.proto

GitHub Actions / Check breaking changes in Protobuf files

Field "3" with name "task_id" on message "TaskId" changed type from "uint32" to "uint64". See https://developers.google.com/protocol-buffers/docs/proto3#updating for wire compatibility rules and https://developers.google.com/protocol-buffers/docs/proto3#json for JSON compatibility rules.
}
// Every task will create N buffers (channels) for parent operators to fetch results from,
message TaskOutputId {
TaskId task_id = 1;
// The id of output channel to fetch from
uint64 output_id = 2;

Check failure on line 236 in proto/batch_plan.proto

GitHub Actions / Check breaking changes in Protobuf files

Field "2" with name "output_id" on message "TaskOutputId" changed type from "uint32" to "uint64". See https://developers.google.com/protocol-buffers/docs/proto3#updating for wire compatibility rules and https://developers.google.com/protocol-buffers/docs/proto3#json for JSON compatibility rules.
}
message LocalExecutePlan {
repeated uint32 inner_side_key = 4;
uint32 lookup_prefix_len = 5;
plan_common.StorageTableDesc inner_side_table_desc = 6;
repeated uint64 inner_side_vnode_mapping = 7;

Check failure on line 273 in proto/batch_plan.proto

GitHub Actions / Check breaking changes in Protobuf files

Field "7" with name "inner_side_vnode_mapping" on message "LocalLookupJoinNode" changed type from "uint32" to "uint64". See https://developers.google.com/protocol-buffers/docs/proto3#updating for wire compatibility rules and https://developers.google.com/protocol-buffers/docs/proto3#json for JSON compatibility rules.
repeated int32 inner_side_column_ids = 8;
repeated uint32 output_indices = 9;
repeated common.WorkerNode worker_nodes = 10;