Longevity test 2024-01-01 OOM

[lmatz]

Describe the bug

Today's date:2024-01-01
Result               FAIL                
Pipeline Message     @Nightly run all nexmark (8 sets of nexmark queries) with 10k throughput
TestBed              kubebench/3264g-medium-3cn-all-affinity
RW Version           nightly-20231231    
Test Start time      2023-12-31 15:16:11 
Test End time        2024-01-01 01:08:34 
Namespace            reglngvty-20231231-150225
Queries              nexmark_q0,nexmark_q1,nexmark_q2,nexmark_q3,nexmark_q4,nexmark_q5,nexmark_q6_group_top1,nexmark_q7,nexmark_q8,nexmark_q9,nexmark_q10,nexmark_q12,nexmark_q14,nexmark_q15,nexmark_q16,nexmark_q17,nexmark_q18,nexmark_q19,nexmark_q20,nexmark_q21,nexmark_q22,nexmark_q101,nexmark_q102,nexmark_q103,nexmark_q104,nexmark_q105
Grafana Metric       https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=reglngvty-20231231-150225&from=1704035771000&to=1704071314000
Grafana Logs         https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=reglngvty-20231231-150225&from=1704035771000&to=1704071314000
Memory Dumps to S3   https://s3.console.aws.amazon.com/s3/buckets/test-useast1-mgmt-bucket-archiver?region=us-east-1&prefix=k8s/reglngvty-20231231-150225/&showversions=false
Buildkite Job        https://buildkite.com/risingwave-test/longevity-test/builds/879
Crash Container logs https://rw-qa-artifacts-public.s3.us-east-1.amazonaws.com/longevity/879_logs.txt
Report               https://rw-qa-artifacts-public.s3.us-east-1.amazonaws.com/longevity/879_report.txt

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[lmatz]

collapsed.zip

Looks like still rdkafka, similar to #14280