Move rate limit from flow control to within source executors (snapshot read, source, dml)

[kwannoel]

No description provided.

[kwannoel]

We can do it step by step, per executor.
Also, to make sure alter rate limit is still supported, since this rate limit occurs upstream of barrier, we need a another channel to capture alter rate limit changes.

[stdrc]

Also, to make sure alter rate limit is still supported, since this rate limit occurs upstream of barrier, we need a another channel to capture alter rate limit changes.

If risingwavelabs/rfcs#81 this RFC is going to be implemented, seems this can be not a problem. Though we don't have a clear timeline for this now.

[tabVersion]

We can do it step by step, per executor. Also, to make sure alter rate limit is still supported, since this rate limit occurs upstream of barrier, we need a another channel to capture alter rate limit changes.

I vote for the overall goal to move rate limit from flow control to within source executors, spreading data into more epochs. But can you elaborate more on "we need a another channel to capture alter rate limit changes".
I think the source exec can receive all mutations and apply them to the inner part. That's how split change and split migration work.

[StrikeW]

May I know the motivation of the issue? What's the limitation of wrapping source executors with the FlowControlExecutor?

[kwannoel]

May I know the motivation of the issue? What's the limitation of wrapping source executors with the FlowControlExecutor?

It stems from a specific edge case: rate limiting smaller than chunk size with flow control executor will just cause barrier latency to spike. This is because rate limiting a chunk would mean the barrier before it will be blocked, until the entire chunk is processed.

As such, we add some chunk splitting logic upstream, inside executors themselves, before the data stream gets merged with the barrier stream from meta.

Now we have the logic for rate limiting split across flow control, and each executor. Additionally, the way it works is unintuitive (and perhaps unreliable sometimes), it relies on upstream to bias to barrier side to let it bypass.

It is simpler and more intuitive to relocate the rate limit logic into executors. Now all the logic is in one place, and rate limit happens at the data source, so it is clear that it won't block barriers.

[tabVersion]

under developing in #15948