Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scale: reconsider how to handle no-shuffle exchange #8653

Closed
Tracked by #3750
BugenZhao opened this issue Mar 20, 2023 · 1 comment
Closed
Tracked by #3750

scale: reconsider how to handle no-shuffle exchange #8653

BugenZhao opened this issue Mar 20, 2023 · 1 comment
Labels
component/meta Meta related issue. component/streaming Stream processing related issue. type/enhancement Improvements to existing implementation.

Comments

@BugenZhao
Copy link
Member

Previously in #4614, we decided to remove the no-shuffle exchange as much as possible and leave a single case of MV on MV (#5728), to make the scaling of each fragment independent and simplify the complexity of rescheduling plan resolving. However, as more and more operators are introduced (like lookup join and temporal join), we find that it's inevitable that the distribution of some operators needs to be coupled.

Therefore, we should reconsider how to handle no-shuffle exchange in scaling and make the implementation more general.

@BugenZhao BugenZhao added type/enhancement Improvements to existing implementation. component/streaming Stream processing related issue. component/meta Meta related issue. labels Mar 20, 2023
@github-actions github-actions bot added this to the release-0.19 milestone Mar 20, 2023
@BugenZhao
Copy link
Member Author

BugenZhao commented Apr 4, 2023

Luckily the scaling for temporal join and delta join (#8694) can be supported with a few modifications to the current logic, which is originally designed for MV-on-MV only. 😄 Let's wait for other novel streaming graph patterns before deciding whether we need this.

@BugenZhao BugenZhao removed this from the release-0.19 milestone Apr 4, 2023
@xxchan xxchan closed this as not planned Won't fix, can't repro, duplicate, stale Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/meta Meta related issue. component/streaming Stream processing related issue. type/enhancement Improvements to existing implementation.
Projects
None yet
Development

No branches or pull requests

2 participants