Compute Node OOM in longevity test with 20000 throughput #13326
Labels
type/bug
Something isn't working
Milestone
Comments
|
Previously, I tried to fix this by adjusting the parameters of fetch queue of rdkafka, but didn't work. https://github.com/risingwavelabs/kube-bench/compare/main...eric/limit_rdkakfa_fetch_queue_size If anyone is interested, may continue to work on this. To run with this kube-bench branch, set the env var of BuildKite job: also related #10888 |
|
Update: the occasional oom under 1w throughput is also caused by this(rdfaka). |
|
Can we close this issue? cc @fuyufjh |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
CN oom in longevity test with 2w throughput. We originally thought it was caused by the mem table of the materialized operator being too large, but now it turns out that it is caused by the memory consumed by rdkfaka being too high.
Currently the memory of materialized executor is no longer very high. We need to further solve the problem of rdkafka.
heapfile
2w-oom-cn1-0918.collapsed.zip
Related to #11977 #13060
Error message/log
No response
To Reproduce
No response
Expected behavior
No response
How did you deploy RisingWave?
No response
The version of RisingWave
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: