scale: reconsider how to handle no-shuffle exchange #8653
Labels
component/meta
Meta related issue.
component/streaming
Stream processing related issue.
type/enhancement
Improvements to existing implementation.
Previously in #4614, we decided to remove the no-shuffle exchange as much as possible and leave a single case of MV on MV (#5728), to make the scaling of each fragment independent and simplify the complexity of rescheduling plan resolving. However, as more and more operators are introduced (like lookup join and temporal join), we find that it's inevitable that the distribution of some operators needs to be coupled.
Therefore, we should reconsider how to handle no-shuffle exchange in scaling and make the implementation more general.
The text was updated successfully, but these errors were encountered: