reglngvty-20231128-150243 OOM

[lmatz]

OOM

setting: Nightly run all nexmark (16 sets of nexmark queries) with 10000 throughput

https://risingwave-labs.slack.com/archives/C048NM5LNKX/p1701220086545809

Grafana Metric https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=reglngvty-20231128-150243&from=1701185131000&to=1701220086000
Grafana Logs https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=reglngvty-20231128-150243&from=1701185131000&to=1701220086000
Buildkite Job https://buildkite.com/risingwave-test/longevity-test/builds/826

Add some context:

Notice the "16 sets of nexmark queries" means every query is run for 16 times, that is, totally 16 * 25 queries running in the testing instance. This is the first time we ran such test.

[lmatz]

looks like no single component using too much memory...

anyone has idea?

[fuyufjh]

Already contains #13648.

I have tested #13648 with longevity test (nexmark all queries), and it should pass.

Let me take a look.

[fuyufjh]

Well, the memory manager already tries its best to evict cache, so it's not its fault.

[fuyufjh]

It's because too many queries running in the instance: 16 sets of nexmark queries

This seems to be expected. When we have 400 queries running parallelly, there will be many fragmented memory usage, added up together and blow off the memory.

looks like no single component using too much memory...

Yes. This is actually a good sign, otherwise, we have something to fix...

[fuyufjh]

I think this should not be considered a bug.