Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize meta memory usage #18012

Open
xxchan opened this issue Aug 13, 2024 · 2 comments
Open

Optimize meta memory usage #18012

xxchan opened this issue Aug 13, 2024 · 2 comments
Assignees
Milestone

Comments

@xxchan
Copy link
Member

xxchan commented Aug 13, 2024

Background: https://www.notion.so/risingwave-labs/Jindo-Meta-OOM-752dedb74fac4e668fe4863c62318fbb?pvs=4

Meta consumed a lot of memory. And can consume a lot higher memory during some processes than the stable usage and OOM.

image

A known memory allocation involves duplicating all actors and distributing them to CNs for the recovery process.
by @yezizp2012

@github-actions github-actions bot added this to the release-2.0 milestone Aug 13, 2024
@xxchan
Copy link
Member Author

xxchan commented Aug 13, 2024

Can SQL backend reduce memory usage? 🤔

@yezizp2012 yezizp2012 self-assigned this Aug 13, 2024
@yezizp2012
Copy link
Member

Can SQL backend reduce memory usage? 🤔

Yes, meta will not maintain a cache of table fragments, this part of the memory that is always resident will be saved. For recovery, I had a offline discussion with @shanicky , we gonna fully deprecate table fragments and reduce the memory usage and RPC request body in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants