-
Notifications
You must be signed in to change notification settings - Fork 125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add compact in-memory representation of RecordRef #2053
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clever use of the fact that Field
alignment is less than Field
size!
Unfortunately there are some changes we need to revert.
I'm far from proficient in such memory tricks so also left some questions here, mostly to help me understand the implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wonderful!
Because alignment of `i128` and `u128` is 8 on linux x86_64 and 16 on M1
This adds a compact
RecordRef
representation, significantly reducing theper-field overhead for records. There are some more improvements that can be
made here, namely not storing the schema per-record and removing the extra pointer
indirection for the field data.
Overhead
Memory measurements
5M records, 4 integers, 1 float (near-ideal case)
5M records, 1 integers, 2 strings