Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write latest finalized state in chunks #9026

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

tbenr
Copy link
Contributor

@tbenr tbenr commented Jan 22, 2025

  • serializes beaconState into chunks via SszByteArrayChunksWriter (currently hardcoded to 8MiB chunk sizes)
  • on DB side introduced a KvStoreUnchunkedVariable which stores chunks in a similarly to what we do with columns:
    • the base key (id)->byte stores a byte with the number of chunks (max 255 chunks)
    • chunks are stored as(id,chunkIndex) -> chunk bytes

There is an automatic backward compatibility behavior: if the base (id)->byte returns something bigger than a single byte, than it is considered as a big chunk (essentially the previous storage mode)

TODO:

  • verify that this actually improves memory allocations (no more big 256mb single byte array)
  • implement an iterator when reading chunks
  • trimming: if the variable becomes smaller, previous additional chunks remains on DB (we could not care for BeaconState, it always grows)
  • rocksDB implementation
  • tests

fixes #9018

Documentation

  • I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

  • I thought about adding a changelog entry, and added one if I deemed necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support serializing state in chunks when saving it on DB
1 participant