feat(connector): partition source reader batch_size by rate_limit
#13800
Task list completed / task-list-completed
Started
2024-02-23 07:29:01
ago
0 / 8 tasks completed
8 tasks still to be completed
Details
Required Tasks
| Task | Status |
|---|---|
| For the granularity of rate limit, currently only non-transaction sources will be able to rate limit smaller than chunk size. | Incomplete |
| For transaction sources (e.g. debezium), the granularity would still be chunk size, so we don't break the transaction semantics. We have to support dynamic config for transaction size for this feature. | Incomplete |
The configuration of rate_limit is done on stream job initialization, it cannot be altered yet. Need a further PR for it. |
Incomplete |
| I have written necessary rustdoc comments | Incomplete |
| I have added necessary unit tests and integration tests | Incomplete |
| I have added test labels as necessary. See details. | Incomplete |
| I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934). | Incomplete |
| My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future). | Incomplete |
All checks passed in ./risedev check (or alias, ./risedev c) |
Incomplete |
| My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details) | Incomplete |
| My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users) | Incomplete |
| snapshot read | Incomplete |
| source read | Incomplete |
| Support partitioning of the snapshot read / source read messages. | Incomplete |
| Support dynamically altering the partition sizes (slightly tricker for source read). | Incomplete |
| Understand the implications of revoking this secret by investigating where it is used in your code. | Incomplete |
| Replace and store your secret safely. Learn here the best practices. | Incomplete |
| Revoke and rotate this secret. | Incomplete |
| If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data. | Incomplete |
| following these best practices for managing and storing secrets including API keys and other credentials | Incomplete |
| install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation. | Incomplete |
In most of time when rate_limit is unset or set to a extremely large value, the drain branch won't be executed. |
Incomplete |
If the rate_limit is small, there must be other more significant bottlenecks in the system. The overhead here pales in comparison. |
Incomplete |
Loading