ducktape: Verify using consumer groups in the MPT #24656
Conversation
Instead of using a single consumer use multiple consumers for the verification. This avoids ultra wide reads (think 1k+ partitions per request). These are not realistic and make us run into CORE-8659 which makes the test timeout with increased partition density. We use one group/reader per 5k partitions which is still far from realistic but at least keeps that dimension somewhat tested.
|
/cdt |
|
/cdt |
|
/cdt |
Not sure about that, pretty sure people end up doing that, either in a sustained way but probably more often in a transient way when consumers are entering or leaving groups which have a large total number of partitions (but this case is probably fine since the consumers will get "unstuck" when more join and the problem stops occurring). |
| self.test_context, | ||
| self.redpanda, | ||
| target_topic, | ||
| 0, | ||
| readers=math.ceil(scale.partition_limit / 5000), |
There was a problem hiding this comment.
Doesn't this result in 5000 partitions per reader still? I thought that was way too many per the cover letter, i.e., we would still be in "(think 1k+ partitions per request)" territory?
There was a problem hiding this comment.
It results in 5000 / 6 (broker count) per reader so not entirely 1k+ but still high. As mentioned in the cover letter I still want to keep this around a little bit but just not extreme enough to hit the edge case.
Instead of using a single consumer use multiple consumers for the
verification.
This avoids ultra wide reads (think 1k+ partitions per request). These
are not realistic and make us run into CORE-8659 which makes the test
timeout with increased partition density.
We use one group/reader per 5k partitions which is still far from
realistic but at least keeps that dimension somewhat tested.
Backports Required
Release Notes