perf(stream): optimize sink latency when sink pk mismatch

[st1page]

Currently, all data in a barrier will be buffered in the sink executor and some operations to prevent the disorder incorrect result of upsert sink. But it makes the e2e latency to barrier latency and the event can not be sinked earlier.
we can buffered the data but sink the event at time, but on barrier coming, send other operations to "fix" the wrong result.

[lmatz]

Does it mean that the user needs to be aware of the "wrong" results and somehow also be able to tolerate a temporary wrong result in the user's application?

Just try to understand the scope of this improvement that can be applied to

[st1page]

Does it mean that the user needs to be aware of the "wrong" results and somehow also be able to tolerate a temporary wrong result in the user's application?

Just try to understand the scope of this improvement that can be applied to

1. the downstream system must be an "upsert" system, in other words, when the system receives a new inserted value, it should overwrite the exist value in the system with the same key.
2. the result of a specific will be "eventually correct" but maybe some key will lost for a while... (I think maybe there could be another solution when I write this. maybe we can delay the delete operation to achieve a better experience for user)
3. the order of different key's update during a barrier is not guaranteed. And that's not guaranteed in our current implementation too.

[github-actions]

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄