Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ets: snapshots / memorydump_to_file #9245

Open
vans163 opened this issue Dec 24, 2024 · 0 comments
Open

ets: snapshots / memorydump_to_file #9245

vans163 opened this issue Dec 24, 2024 · 0 comments
Assignees
Labels
enhancement team:VM Assigned to OTP team VM

Comments

@vans163
Copy link
Contributor

vans163 commented Dec 24, 2024

Is your feature request related to a problem? Please describe.
No way to snapshot a ETS state and/or dump a tables memory to file (if same ERTS is used to load)

Describe the solution you'd like

  1. A way to quickly CoW in-memory a table and have 2 copies, writes to one do not reflect to the other, reads to the other go to the original, unless a change occurred.
  2. A way to dump the memory directly to disk, right now tab2file reads every term sequentially + does term_to_binary on it, this gets linearly slower once the table surpasses 1m keys. The solution would be to structure the memory layout of a table making it friendly to be dumped to disk, then just read from disk directly into memory. A flag can be set that this is a very risky / insecure operation and untrusted tables should not be loaded this way. Assume that zstd or other compression is default, so large patches of zerod memory for example will compress really well.

Describe alternatives you've considered
No alternatives ATM. Mnesia disk writes are 10seconds+ delayed and enabling immediate flush mode on Mnesia makes it 100x slower.

EDIT: Funny posting this just gave me an idea how to implement ETS CoW purely on the managed side (abite with a large factor of less efficiency), without digging into ERTS sources.

@jhogberg jhogberg added the team:VM Assigned to OTP team VM label Dec 27, 2024
@jhogberg jhogberg self-assigned this Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement team:VM Assigned to OTP team VM
Projects
None yet
Development

No branches or pull requests

2 participants