Failing Tests

[norlandrhagen]

Issue

I'm getting some failures when running the pytest suite locally. For reference, I'm running this on an M1 mac and following the development guide.

Details

I've forked pangeo-forge-recipes to https://github.com/norlandrhagen/pangeo-forge-recipes. My fork seems to be in sync with the main repo.

On the master/main branch, I create a development environment with:

conda env create --file ci/py3.9.yml
conda activate pangeo-forge-recipes

and then run pytest:

py.test tests -v

For both the 3.8 and 3.9, I get 43 failed and 1027 passed.

short test summary info:

================================================================================== short test summary info ==================================================================================
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_function-netcdf_local_paths_sequential_2d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_generator-netcdf_local_paths_sequential_2d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_dask-netcdf_local_paths_sequential_2d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path ''"
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_beam-netcdf_local_paths_sequential_2d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path '' [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_0...
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential-execute_recipe_function-netcdf_local_paths_sequential_2d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential-execute_recipe_generator-netcdf_local_paths_sequential_2d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential-execute_recipe_dask-netcdf_local_paths_sequential_2d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path ''"
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential-execute_recipe_beam-netcdf_local_paths_sequential_2d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path '' [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_0...
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect-netcdf_local_paths_sequential_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect_dask-netcdf_local_paths_sequential_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect_wrapper-netcdf_local_paths_sequential_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect-netcdf_local_paths_sequential_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect_dask-netcdf_local_paths_sequential_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect_wrapper-netcdf_local_paths_sequential_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_function-netcdf_local_paths_sequential_1d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_generator-netcdf_local_paths_sequential_1d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_dask-netcdf_local_paths_sequential_1d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path ''"
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_beam-netcdf_local_paths_sequential_1d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path '' [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_0...
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect-netcdf_local_paths_sequential_1d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect_dask-netcdf_local_paths_sequential_1d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-execute_recipe_prefect_wrapper-netcdf_local_paths_sequential_1d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_function-netcdf_local_paths_sequential_multivariable_2d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_generator-netcdf_local_paths_sequential_multivariable_2d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_dask-netcdf_local_paths_sequential_multivariable_2d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path ''"
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_beam-netcdf_local_paths_sequential_multivariable_2d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path '' [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_0...
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_function-netcdf_local_paths_sequential_multivariable_2d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_generator-netcdf_local_paths_sequential_multivariable_2d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_dask-netcdf_local_paths_sequential_multivariable_2d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path ''"
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_beam-netcdf_local_paths_sequential_multivariable_2d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path '' [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_0...
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect-netcdf_local_paths_sequential_multivariable_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect_dask-netcdf_local_paths_sequential_multivariable_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect_wrapper-netcdf_local_paths_sequential_multivariable_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect-netcdf_local_paths_sequential_multivariable_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect_dask-netcdf_local_paths_sequential_multivariable_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect_wrapper-netcdf_local_paths_sequential_multivariable_2d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_function-netcdf_local_paths_sequential_multivariable_1d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_generator-netcdf_local_paths_sequential_multivariable_1d-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_dask-netcdf_local_paths_sequential_multivariable_1d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path ''"
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_beam-netcdf_local_paths_sequential_multivariable_1d-0] - zarr.errors.GroupNotFoundError: group not found at path "group not found at path '' [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_0...
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect-netcdf_local_paths_sequential_multivariable_1d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect_dask-netcdf_local_paths_sequential_multivariable_1d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_with_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-execute_recipe_prefect_wrapper-netcdf_local_paths_sequential_multivariable_1d-0] - ValueError: Prefect flow run failed with message Some reference tasks failed.
FAILED tests/recipe_tests/test_graph_memory.py::test_memory_usage - Failed: Timeout >90.0s
========================================================== 43 failed, 1027 passed, 130 skipped, 519 warnings in 537.27s (0:08:57) ===========================================================

Running with single pytest mark selected:

If pytest is running using only the executor_function mark:

pytest -m executor_function

================================================================================== short test summary info ==================================================================================
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-netcdf_local_paths_sequential_2d-execute_recipe_function-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential-netcdf_local_paths_sequential_2d-execute_recipe_function-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential-netcdf_local_paths_sequential_1d-execute_recipe_function-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-netcdf_local_paths_sequential_multivariable_2d-execute_recipe_function-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_subset_recipe-netcdf_local_file_pattern_sequential_multivariable-netcdf_local_paths_sequential_multivariable_2d-execute_recipe_function-0] - zarr.errors.GroupNotFoundError: group not found at path ''
FAILED tests/recipe_tests/test_XarrayZarrRecipe.py::test_recipe_with_references[execute_recipe_no_prefect-netCDFtoZarr_recipe-netcdf_local_file_pattern_sequential_multivariable-netcdf_local_paths_sequential_multivariable_1d-execute_recipe_function-0] - zarr.errors.GroupNotFoundError: group not found at path ''
===================================================== 6 failed, 204 passed, 18 skipped, 972 deselected, 28 warnings in 94.47s (0:01:34) =====================================================

cc @cisaacstern

Related issue: #438 (comment)

[cisaacstern]

@norlandrhagen thanks so much for raising this, and documenting your process so clearly.

As we discussed yesterday, my first hypothesis would be that this is a dependency versions issue, given that the tests passed on the last merge commit to master.

I just built a new environment from ci/py3.9.yml locally, and compared the versions built into that env from

https://github.com/pangeo-forge/pangeo-forge-recipes/actions/runs/3415831179/jobs/5685363741#step:10:48

These are the packages for which I get upgraded versions locally, as compared to the above-linked Actions env:

aiosignal 1.2.0 -> 1.3.1
apache-beam 2.42.0 -> 2.43.0
arrow-cpp 7.0.1 -> 9.0.0
asttokens 2.1.0 -> 2.2.1
attr 2.5.1 -> 22.1.0
blosc 1.21.1 -> 1.21.2
ca-certificates 2022.9.24 -> 2022.12.7
certifi 2022.9.24 -> 2022.12.7
cfitsio 4.1.0 -> 4.2.0
tiledb 2.11.3 -> 2.13.0
tornado 6.1 -> 6.2
traitlets 5.5.0 -> 5.7.1
tzcode 2022f -> 2022g
tzdata 2022f -> 2022g
urllib3 1.26.11 -> 1.26.13
wheel 0.38.2 -> 0.38.4
xarray 2022.11.0 -> 2022.12.0
zipp 3.10.0 -> 3.11.0

None of these immediately strike me as involved with the failing test (xarray possibly, but downgrading that did not allow the tests to pass).

This remains a mystery to me. We should certainly try to fix it before you continue with your ReferenceRecipe generalization, as the test is relevant to that part of the code.

[cisaacstern]

With @derekocallaghan's recently hitting this issue as well, I'll just note for the record that my bandwidth is almost entirely consumed by #450 currently, so I don't have plans to try to figure this out in the next few weeks at least.

Certainly it would be good to know why the tests are failing, and any further development on-top of the existing release of pangeo-forge-recipes will greatly benefit from this. (That is, I can envision continuing to patch the 0.9.x series for some time, even after the beam-refactor is merged.)

I welcome anyone interested in trying to work on this, and will do my best to support as needed.

[rabernat]

I will figure this out and fix it.

[norlandrhagen]

Awesome, thanks @rabernat!