-
Notifications
You must be signed in to change notification settings - Fork 59
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
<!--Please ensure the PR fulfills the following requirements! --> <!-- If this is your first PR, make sure to add your details to the AUTHORS.rst! --> ### Pull Request Checklist: - [x] This PR addresses an already opened issue (for bug fixes / features) - This PR relies on changes to be merged in Ouranosinc/xclim-testdata#29 - [x] Tests for the changes have been added (for bug fixes / features) - [x] (If applicable) Documentation has been added / updated (for bug fixes / features) - [x] CHANGELOG.rst has been updated (with summary of main changes) - [x] Link to issue (:issue:`number`) and pull request (:pull:`number`) has been added ### What kind of change does this PR introduce? * Replaces the logic for file gathering and caching from the in-house developed version to instead use `pooch`. * In order to fetch testing data, one can now use the following: ```python from xclim.testing.utils import nimbus n = nimbus() # from a fork of xclim-testdata: nimbus(repo="https://github.com/Me/My_Repo", branch="my_test_branch") file = n.fetch("some_folder/some_data.nc") ``` * Removes the remote GitHub calls for every file request (which was performed by `_get()`). * Exports most of the file request and cache handling to `pooch`, while maintaining a relatively unchanged API for users. * (To be confirmed) Speeds up the delivery of test data to tests by reducing the amount of redundant calls to fixtures and relying on a single pooch instance of pooch to prevent multiple setup stages. ### Does this PR introduce a breaking change? Absolutely. `get_file` and `open_dataset` no longer fetch remote files from GitHub. Instead, a locally-stored `registry.txt` file contains all the checksums of all files needed to run the tests and returns the appropriate file from a locally-held cache. If the file checksum does not match the expected value, it will attempt to replace it from the remote storage. Additionally, the `md5` files that accompanied all testing data files are now obsolete thanks to the use of the registry. The testing data is now versioned according to the `xclim-testdata` version/tag. All the `prefetch` logic baked into the `pytest` calls has been removed, making the setup code much easier to follow. There is no longer a need to run `$ xclim prefetch_testing_data` unless users are running on Windows (for the very first run of `pytest` only). There are now three environment variables to help developers: - XCLIM_TESTDATA_BRANCH - Controls the branch name of `xclim-testdata`. - XCLIM_TESTDATA_CACHE_DIR - Controls the local folder to be used when fetching the test data. - XCLIM_TESTDATA_REPO_URL - Controls the repository URL for `xclim-testdata` (for forks) `platformdirs` is no longer a hard dependency. The default cache directory will only be determined if `pooch` is installed. ### Other information: There is still a lot of potential here to tighten this up; I'd like to land on a design that is clean and easily portable to other projects. What is unchanged is that `pytest` will still do the following on every run: 1. Check that a locally stored copy of the test data exists in a platform-dependent default location and, if not found, will fetch a copy. 2. Each worker of `pytest` creates its own copy of the test data, which is delivered by its own `pooch` instance, written to a threadsafe temporary directory 3. The equivalent to the `get_file()` fixture is now `nimbus.fetch()`, providing the absolute paths to files, respective of platform and workers. Many tests related to testing the file accessors have also been removed (as these are now out of scope).
- Loading branch information
Showing
26 changed files
with
773 additions
and
838 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -28,9 +28,9 @@ dependencies: | |
- flox | ||
- lmoments3 # Required for some Jupyter notebooks | ||
# Testing and development dependencies | ||
- black ==24.4.2 | ||
- black ==24.8.0 | ||
- blackdoc ==0.3.9 | ||
- bump-my-version >=0.24.3 | ||
- bump-my-version >=0.25.4 | ||
- cairosvg | ||
- codespell ==2.3.0 | ||
- coverage >=7.5.0 | ||
|
@@ -54,7 +54,6 @@ dependencies: | |
- nc-time-axis >=1.4.1 | ||
- notebook | ||
- pandas-stubs >=2.2 | ||
- platformdirs >=3.2 | ||
- pooch >=1.8.0 | ||
- pre-commit >=3.7 | ||
- pybtex >=0.24.0 | ||
|
@@ -74,7 +73,7 @@ dependencies: | |
- tokenize-rt >=5.2.0 | ||
- tox >=4.16.0 | ||
# - tox-conda # Will be added when a [email protected]+ compatible plugin is released. | ||
- vulture # ==2.11 # The conda-forge version is out of date. | ||
- vulture ==2.11 | ||
- xdoctest >=1.1.5 | ||
- yamllint >=1.35.1 | ||
- pip >=24.0 | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.