feat(streaming): adapt backfill rate limit according to barrier latency

[kwannoel]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

wip.

• Show grafana monitoring results.
• Add more scenarios: Join with Agg after, UDF high latency.
• Include more description on the alternative solutions.
• Add frontend support for setting adaptive rate limit.

We have a few test scenarios:

Amplification is medium x100

Backpressure can handle this case well. Adaptive Rate Limit does not have any discernible effect.

Amplification is extreme, e.g. x500

In this case, adaptive rate limit will not be able to keep barrier latency low.
Since barrier latency is a lagging metric, it means that rate limit will only be adjusted much later, and will take a while for it to take effect. In such a scenario, barrier latency will remain high.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[kwannoel]

The default adaptive rate limit strategy doesn't work well when I test it with the high amplification case.

1. The rate limit increases too fast (since it scales rate limit exponentially 2x).
2. We cannot simply get "current barrier latency". The previous barrier may not have been collected yet.
3. Even if we have threshold = 0, and we do linear increments, this linear increment of rate limit can also be too large (1000).

Then we end up having too high a rate limit due to 1 and 2. At that point, when adaptive rate limit kicks in, it's too slow as well, since its barrier latency is high (as rate limit is too high).
After we return to normalcy, threshold becomes 0, and the rate limit fluctuates between 1000, and the initial rate limit (1). At 1000, the rate limit is too high as well in amplification cases. Leading to high barrier latency still.

[github-actions]

This PR has been open for 60 days with no activity.

If it's blocked by code review, feel free to ping a reviewer or ask someone else to review it.

If you think it is still relevant today, and have time to work on it in the near future, you can comment to update the status, or just manually remove the no-pr-activity label.

You can also confidently close this PR to keep our backlog clean. (If no further action taken, the PR will be automatically closed after 7 days. Sorry! 🙏)
Don't worry if you think the PR is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄

[kwannoel]

@hzxa21 suggests using the time lapse between barriers received by the backfill executor to adapt rate limit.

[kwannoel]

Some other thoughts: Adaptive rate limit should not attempt to handle extreme amplification cases. Just handle trends where the barrier latency slowly climbs up.