meta node OOM in longevity test with shared source

[xxchan]

Execution URL: https://buildkite.com/risingwave-test/longevity-test/builds/1357
Grafana: https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1714670735000&to=1714714072000&var-namespace=usrlngvty-20240502-171919
Log: https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&from=1714670735000&to=1714714072000&var-namespace=usrlngvty-20240502-171919

Quite strange

This test is triggered with

ENABLE_CREATE_MV_NEXMARK_TABLE="false"
ENABLE_SHARED_SOURCE="true"

[xxchan]

HEAPS_S3_URL="s3://test-useast1-mgmt-bucket-archiver/k8s/usrlngvty-20240502-171919/"
RW_VERSION="nightly-20240501"

https://buildkite.com/risingwave-test/generate-collapsed-heap-files/builds/49#018f3dd4-ab5c-46ec-916e-b85f389758cd

Looks like a large proto

[xxchan]

Hmmm, from nexmark, I see no much difference for whether or not shared source is enabled.

It may be possibly caused by ENABLE_CREATE_MV_NEXMARK_TABLE="false". (i.e., CREATE VIEW person/bid/auction, otherwise it's MATERIALIZED VIEW in the longevity).
Longevity also failed in this setting and with non shared source. https://buildkite.com/risingwave-test/longevity-test/builds/1358 But it didn't produce a memory dump

[xxchan]

ColumnDesc of SouceBackfillNode is 300MB? 😮

[xxchan]

@yezizp2012 not sure if you have any quick thoughts about this. 🤔

[yezizp2012]

@yezizp2012 not sure if you have any quick thoughts about this. 🤔

ColumnDesc shouldn't be very large. The parallelism of shared source streaming job should be 1, right?

[xxchan]

RW_META_STORE=postgresql seems ok...

https://buildkite.com/risingwave-test/longevity-test/builds/1400

[github-actions]

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄