Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream cache and use mem-map-supported format #173

Open
k1o0 opened this issue Jan 8, 2025 · 0 comments
Open

Stream cache and use mem-map-supported format #173

k1o0 opened this issue Jan 8, 2025 · 0 comments
Labels
enhancement New feature or request question Further information is requested

Comments

@k1o0
Copy link
Collaborator

k1o0 commented Jan 8, 2025

Initial tests with converting parquet to feather then loading in mem-mapped mode did not improve loading times, although this was not profiled rigorously. Mem-mapping will become essential as our full dataset grows.

Given that we're moving away from downloading cache tables by default in ONE3, streaming cache tables may not be worth the effort.

@k1o0 k1o0 added enhancement New feature or request question Further information is requested labels Jan 8, 2025
@k1o0 k1o0 mentioned this issue Jan 8, 2025
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant