Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow >1 paralellism for pubsub source to increase ingestion performance #16572

Closed
lmatz opened this issue May 2, 2024 · 1 comment · Fixed by #16733
Closed

allow >1 paralellism for pubsub source to increase ingestion performance #16572

lmatz opened this issue May 2, 2024 · 1 comment · Fixed by #16733
Assignees
Milestone

Comments

@lmatz
Copy link
Contributor

lmatz commented May 2, 2024

more discussion can be found here: https://risingwave-labs.slack.com/archives/C0606NNR74P/p1714493567060129

In essence, it is hard to make pubsub source exactly-once and other products only provide at-least-once semantics too, so 1 parallelism only does not help anyways.

We should make it parallel.

cc: @fuyufjh

@github-actions github-actions bot added this to the release-1.9 milestone May 2, 2024
@xxchan xxchan self-assigned this May 8, 2024
@lmatz
Copy link
Contributor Author

lmatz commented May 8, 2024

Is it possible to have a rough ETA? (not necessarily need to be done very urgently)
as we are about to have a call with the user

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants