-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
<!-- Please ensure the PR fulfills the following requirements! --> <!-- If this is your first PR, make sure to add your details to the AUTHORS.rst! --> ### Pull Request Checklist: - [ ] This PR addresses an already opened issue (for bug fixes / features) - This PR fixes #xyz - [x] (If applicable) Documentation has been added / updated (for bug fixes / features). - [x] (If applicable) Tests have been added. - [x] This PR does not seem to break the templates. - [x] CHANGELOG.rst has been updated (with summary of main changes). - [x] Link to issue (:issue:`number`) and pull request (:pull:`number`) has been added. ### What kind of change does this PR introduce? * `zip_directory` and `unzip_directory` to do what their name hints to. Useful for working with zipped zarrs. I compared different tools and using python's built-in `zipfile` was only slightly slower than subprocessing something like `7z` or `zip`. So I chose that module for maximum compatibility (not tested on windows though). * `DataCatalog.copy_files` : copy everything from a catalog to a new root folder. Can unzip on the way, can recreate a structure or not. At first I didn't want to do this in-place, but the result of `search_data_catalogs` is not easily copiable. - I use this to transfer my inputs from a network to a local storage on narval because I know my workflow (indicators) will read the input multiple times. * Allow string `directories` in `parse_directory`. Wow did I wonder why my parsing was taking so much time. When passing a single string, it is still treated as a sequence so my path starting with `/tank` was first looking through `/ `... * `indicators.get_indicator_output` : a helper for something we often do when checking if indicators exist in a catalog. Returns the output frequency and variable names of an indicator. * Option `rechunk_for_resample` for `compute_indicators` (and corresponding util). Not 100% how much this changes the performance, but the idea is to have chunks that fit the resampling period before calling the indicator. * `indicators.fix_semiannual` to fix the *?!&# problem with "2QS-OCT" where pandas doesn't want to respect the anchor. (and pandas doesn't have a 6-month period). * `xs.catutils.patterns_from_schema` to help when running `parse_directory` on a folder that was filled with `build_path`. Taking the name of the pattern as in the config gile, returns all possible patterns. ### Does this PR introduce a breaking change? Shouldn't. ### Other information: Pour les intéressées, voir mon utilisation des ces choses dans le script `1_indicators.py` et le script `slurm_indicators.sh` dans ma branche de Portraits : https://github.com/Ouranosinc/portraits_climatiques/tree/ind-in-pc/preparation .
- Loading branch information
Showing
11 changed files
with
419 additions
and
73 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
Oops, something went wrong.