Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for TensorStore for Zarr opening #224

Closed
jacobbieker opened this issue Aug 15, 2023 · 1 comment
Closed

Add support for TensorStore for Zarr opening #224

jacobbieker opened this issue Aug 15, 2023 · 1 comment
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@jacobbieker
Copy link
Member

Tensorstore looks like it might be able to open our large zarrs faster. @dfulu mentioned up to twice as fast for some of our NWP data. This could also help speed up the satellite loading, which is one of the slowest parts of the data creation time right now.

Context

Zarr-Python is not very fast to open zarrs, and the slow reading from zarr is a bottleneck for us. Zarr-rust is being worked on that would speed things up possible a lot more, but in the meantime, this could be a relatively small change for significant gains in speed.

Possible Implementation

https://github.com/google/xarray-tensorstore should be a drop in replacement, with the possible need to open each NWP file separately, but that needs looking into. It would be a relatively small change to make.

@jacobbieker jacobbieker added enhancement New feature or request good first issue Good for newcomers labels Aug 15, 2023
@jacobbieker
Copy link
Member Author

Forgot to search beforehand, duplicate (mostly) of #198

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
No open projects
Status: No status
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant