You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Tensorstore looks like it might be able to open our large zarrs faster. @dfulu mentioned up to twice as fast for some of our NWP data. This could also help speed up the satellite loading, which is one of the slowest parts of the data creation time right now.
Context
Zarr-Python is not very fast to open zarrs, and the slow reading from zarr is a bottleneck for us. Zarr-rust is being worked on that would speed things up possible a lot more, but in the meantime, this could be a relatively small change for significant gains in speed.
Possible Implementation
https://github.com/google/xarray-tensorstore should be a drop in replacement, with the possible need to open each NWP file separately, but that needs looking into. It would be a relatively small change to make.
The text was updated successfully, but these errors were encountered:
Tensorstore looks like it might be able to open our large zarrs faster. @dfulu mentioned up to twice as fast for some of our NWP data. This could also help speed up the satellite loading, which is one of the slowest parts of the data creation time right now.
Context
Zarr-Python is not very fast to open zarrs, and the slow reading from zarr is a bottleneck for us. Zarr-rust is being worked on that would speed things up possible a lot more, but in the meantime, this could be a relatively small change for significant gains in speed.
Possible Implementation
https://github.com/google/xarray-tensorstore should be a drop in replacement, with the possible need to open each NWP file separately, but that needs looking into. It would be a relatively small change to make.
The text was updated successfully, but these errors were encountered: