Grib Reference Recipe

[norlandrhagen]

PR building on #390. Credit to darothen.
Based on discussion from #387.

• Minor update from (WIP/DO NOT SUBMIT) Re-factor HDFReferenceRecipe to expose kerchunk types #390, fixes s3 reading issue.
• Adds example for GRIBReference Recipe in docs

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[norlandrhagen]

TODO:

• Add GribReferenceRecipe testing.
• Add documentation for GribReferenceRecipe example.

[andersy005]

@norlandrhagen, it's my understanding that the tests are failing due to this line

pangeo-forge-recipes/tests/recipe_tests/test_XarrayZarrRecipe.py

Line 172 in fe5d1f7

rec = RecipeClass(file_pattern, open_input_with_kerchunk=True, **kwargs)

setting open_input_with_kerchunk=False seems to be one solution but i don't quite understand why this stopped working in this PR. Perhaps that @cisaacstern has some insights since he is the last person to have touched this part of the codebase :)

[cisaacstern]

setting open_input_with_kerchunk=False seems to be one solution

IIUC based on the docstring

pangeo-forge-recipes/tests/recipe_tests/test_XarrayZarrRecipe.py

Line 169 in fe5d1f7

"""Same as above, but use fsspec references for opening the files."""

This is explicitly testing open_input_with_kerchunk=True so setting that to False is not the solution here.

but i don't quite understand why this stopped working in this PR

I don't have an immediate answer but questions that come to mind:

1. What was the last PR where this test passed?
2. Has the version of kerchunk used in CI changed since then?
3. Has anything that touches open_input_with_kerchunk changed since then?

[andersy005]

It's likely this PR isn't the culprit because

• [pre-commit.ci] pre-commit autoupdate #440 is failing also

Has anything that touches open_input_with_kerchunk changed since then?

I doubt it. It appears the latest merge into main succeeded: https://github.com/pangeo-forge/pangeo-forge-recipes/actions/runs/3415831179/jobs/5685406660

[cisaacstern]

Per @norlandrhagen and my discussion this afternoon:

• beam-refactor remains on the near horizon, so any work that goes into this should be (a) as narrowly-scoped as possible; and (b) done with the awareness that it will need to be re-done before too long.
• The basics of what's required to make this work are simple enough that I believe it's worth continuing to pursue now, given that Raphael is interested (and it has relevance to his work), and he/we are open to using it as a learning experience to entrain him further into pangeo-forge-recipes development, and inform what will be required for a similar feature for the beam-refactor.
• Re: design, a mergable version of this should follow @rabernat's suggestion in add GRIB2ReferenceRecipe  #387 (comment), to use a single ReferenceRecipe object, which can infer file type from the user-passed file_pattern.file_type (as does XarrayZarrRecipe).
• Re: testing, Raphael is going to run the tests against master locally, and assuming they work, may simply start a new PR from there.

Thanks Raphael for your engagement on this!

[norlandrhagen]

Replaced with #452