Longevity test 2023-12-29 OOM

[lmatz]

Describe the bug

run all nexmark (8 sets of nexmark queries) with 10k throughput

RW Version:
nightly-20231228

Grafana Metric:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=reglngvty-20231228-150237&from=1703776532000&to=1703812055000

Grafana Logs:
https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=reglngvty-20231228-150237&from=1703776532000&to=1703812055000

Memory Dumps to S3:
https://s3.console.aws.amazon.com/s3/buckets/test-useast1-mgmt-bucket-archiver?region=us-east-1&prefix=k8s/reglngvty-20231228-150237/&showversions=false

Buildkite Job:https://buildkite.com/risingwave-test/longevity-test/builds/877

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[lmatz]

collapsed.zip

Even though we used materialized view instead of source before creating nexmark's materialized views, the memory consumption of rdkafka exceeds the expectation.

The left bottom corner of the flame graph implies that there is no single component in the streaming engine that is using too much memory.

More dumps can be found in the zip file

[lmatz]

We remark that this setting used to run always OK without OOM.