Proposed Recipes for kerchunking National Water Model gridded 1km output

[rsignell-usgs]

Dataset Name

The National Water Model CONUS Retrospective 1km gridded output, LDASOUT variables

Dataset URL

https://registry.opendata.aws/nwm-archive/

Description

The NOAA National Water Model Retrospective dataset contains input and output from multi-decade CONUS retrospective simulations. These simulations used meteorological input fields from meteorological retrospective datasets.

License

Unknown

Data Format

NetCDF

Data Format (other)

No response

Access protocol

S3

Source File Organization

One time step per file. 3 hourly output. 1 spatial chunk for entire CONUS.

Example URLs

s3://noaa-nwm-retrospective-2-1-pds/model_output/2017/201701010000.LDASOUT_DOMAIN1.comp

Authorization

No; data are fully public

Transformation / Processing

Files should be concatenated along the time dimension

Target Format

Reference Filesystem (Kerchunk)

Comments

I have a working notebook to kerchunk these files using Dask:
https://nbviewer.org/gist/3c178532adffef5201af3fb0a970a05a

This is part of the AWS ARCO project: #208

[rsignell-usgs]

@sharkinsspatial did I capture this correctly?

[sharkinsspatial]

Looks good @rsignell-usgs 👍 I'll link this back to the main list in #208

[rsignell-usgs]

I have a series of notebooks I've used to kerchunk the 1km gridded data:

• Create JSONs for each NetCDF file in parallel, using Dask bag, and checking to see which JSON files have already been created (in case need to restart): 02a_kerchunk_single.ipynb
• combine all the JSON files for each year: 02b_kerchunk_multi_to_years.ipynb
• combine all the yearly JSONS into decades, and then into one JSON: 02c_kerchunk_multi_from_years.ipynb

[rsignell-usgs]

Okay, this is embarrassing: I thought when I looked a few months ago on AWS open data these 1km NWM NetCDF files had not been converted to Zarr.

But I just looked again, and the Zarr datasets are there!

Was this just done recently perhaps?

Nope! :

Why isn't there an egg-on-face emoticon? 😕

This dataset was useful for testing out big kerchunk use cases, but it will be more compelling to find a different one that hasn't already been zarrified, so closing this.