Skip to content
This repository has been archived by the owner on Sep 11, 2023. It is now read-only.

Convert NWP grib files to Zarr intermediate #344

Closed
Tracked by #393
JackKelly opened this issue Nov 8, 2021 · 0 comments · Fixed by #357
Closed
Tracked by #393

Convert NWP grib files to Zarr intermediate #344

JackKelly opened this issue Nov 8, 2021 · 0 comments · Fixed by #357
Assignees
Labels
data New data source or feature; or modification of existing data source enhancement New feature or request

Comments

@JackKelly
Copy link
Member

JackKelly commented Nov 8, 2021

Detailed Description

#122 found that it'll be way too slow to load NWPs from native files directly. So we need to convert to Zarr first.

All of "file 3" listed in this Met Office PDF look useful for helping to predict the evolution of clouds.

Related issues:

Also see:

Something like this should work:

ds = xr.open_dataset(
    filename,
    engine="cfgrib",

    # Wind direction and wind speed are the only variables which are at 
    # 10 meters above ground in the UKV GRIB files.
    #backend_kwargs={'filter_by_keys': {'level': 10}}
    backend_kwargs=dict(filter_by_keys={'stepType': 'instant'})
)

And load twice: Once with stepType=instant and one with stepType=accum. And watch for all the fun exceptions that cfgrib throws :)

This looks like it'll be quite slow. So definitely need to parallelise. If it's super-slow then might be best to FTP new grib files to the cloud and do the processing in parallel in the cloud.

@JackKelly JackKelly added enhancement New feature or request data New data source or feature; or modification of existing data source labels Nov 8, 2021
@JackKelly JackKelly moved this to Todo in Nowcasting Nov 8, 2021
@JackKelly JackKelly self-assigned this Nov 8, 2021
Repository owner moved this from Todo to Done in Nowcasting Nov 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
data New data source or feature; or modification of existing data source enhancement New feature or request
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant