-
Notifications
You must be signed in to change notification settings - Fork 591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: allow customize pulsar subscription name #18379
Conversation
Signed-off-by: tabVersion <[email protected]>
Signed-off-by: tabVersion <[email protected]>
.subscription_name_prefix | ||
.unwrap_or(PULSAR_DEFAULT_SUBSCRIPTION_PREFIX.to_string()), | ||
source_ctx.fragment_id, | ||
source_ctx.actor_id |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is actor_id needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we use exclusive subscription here, which mean a partition can only be consumed by one subscription. So actor-id is essential to avoid collision.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 pulsar split corresponds to 1 topic. In one source (i.e., one fragment), each actor will be assigned to different topics. So actor_id is not necessary. There's no collision.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 pulsar split corresponds to 1 topic. In one source (i.e., one fragment), each actor will be assigned to different topics.
I dont get the logic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The assignment looks like this. Where's the collision? Correct me if I'm wrong
- fragment 6 (topic
t
)- actor 1001 - topic
t-partition-1
, subscriptionrw-consumer-6
- actor 1002 - topic
t-partition-2
, subscriptionrw-consumer-6
- actor 1003 - topic
t-partition-3
, subscriptionrw-consumer-6
- ...
- actor 1001 - topic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am afraid things not work in this way
The exclusive type is a subscription type that only allows a single consumer to attach to the subscription. If multiple consumers subscribe to a topic using the same subscription, an error occurs. Note that if the topic is partitioned, all partitions will be consumed by the single consumer allowed to be connected to the subscription. source
We cannot attach multiple consumers to one subscription in exclusive
mode. Since we want to manage the data parallelism, ie. partitions, in RisingWave, we have to separate the consumers from each parallelism into different subscriptions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe each subscription belongs to a different (sub)topic... They are not "one subscription"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the example above, rw-consumer-6
are 3 different subscriptions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We already separated the consumers by subscribing to different sub topics. It's not by separate them into different subscriptions..
Signed-off-by: tabVersion <[email protected]>
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
resolve #18377
Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.