bug(oom): rdkafka uses too much memory

[fuyufjh]

Describe the bug

https://buildkite.com/risingwave-test/longevity-kubebench/builds/429#0188fb4e-838b-45f3-85ad-ae0b0e6f9277

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[fuyufjh]

Reproduced with MemTable statistics. Seems that MemTable is not the cause.

Grafana Link

[fuyufjh]

rdkafka takes so much memory? (~6GB)

cn.1.15.i15_04-47-01.heap.collapsed.zip

Captured before OOM:

[fuyufjh]

After I tweaked some rd_kafka parameters, the memory usage became quite stable:

(However, another bug #11170 came out... 🤣)

I am going to make it configurable in production.

[fuyufjh]

Let me explain the problem and solution here.

The problem was discussed in confluentinc/librdkafka#2076. In short, rdkafka keep the buffered messages in fetchq (fetch queue) more than queued.min.messages, while during this, the memory usage explodes.

I tried to decrease the queued.min.messages to 1/5 of default value but that reduced the performance drastically (basically 1/5).

I have a few ideas to continue the tweak, but that would be very time-consuming. Better to do it after we have statistics (#11187). For now, if users encounter this issue, they may use #11203 as a work-around.

[tabVersion]

since #11273 is merged, let's see if it helps

[fuyufjh]

since #11273 is merged, let's see if it helps

Reran here: https://buildkite.com/risingwave-test/longevity-kubebench/builds/542

[fuyufjh]

During the last test, it seems that CN can run without OOM, so let me close this for now.

https://buildkite.com/risingwave-test/longevity-kubebench/builds/550