Skip to content

Commit

Permalink
Skip "root" dirs too (#479)
Browse files Browse the repository at this point in the history
<!-- Please ensure the PR fulfills the following requirements! -->
<!-- If this is your first PR, make sure to add your details to the
AUTHORS.rst! -->
### Pull Request Checklist:
- [ ] This PR addresses an already opened issue (for bug fixes /
features)
    - This PR fixes #xyz
- [ ] (If applicable) Documentation has been added / updated (for bug
fixes / features).
- [ ] (If applicable) Tests have been added.
- [ ] This PR does not seem to break the templates.
- [x] CHANGELOG.rst has been updated (with summary of main changes).
- [x] Link to issue (:issue:`number`) and pull request (:pull:`number`)
has been added.

### What kind of change does this PR introduce?

Small fix upon #478. Skips even if a directory to skip is one of the
entries in `directory` or a parent of one of the entries.
I implemented it because I misunderstood an error I was having, but
still, this could be used in the specific scenario where you are running
`parse_directory` through an incremental stack of xscen configurations.
To remove en entry of the `directories` list of a previous
configuration, you would only need to add a `skip_dirs` entry, and not
rewrite the filtered `directories`.

This also adds logging calls when skipping. And a precision to the doc
about the absoluteness of paths to skip.

### Does this PR introduce a breaking change?
No.

### Other information:
  • Loading branch information
aulemahal authored Oct 21, 2024
2 parents 62fe16f + 531a09d commit 0af9f5a
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 4 deletions.
2 changes: 1 addition & 1 deletion CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ New features and enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* ``xs.io.make_toc`` now includes the global attributes of the dataset after the information about the variables. (:pull:`473`).
* New function ``xs.get_warming_level_from_period`` to get the warming level associated with a given time horizon. (:pull:`474`).
* Added ability to skip whole folders to ``xs.parse_directory`` with argument ``skip_dirs``.
* Added ability to skip whole folders to ``xs.parse_directory`` with argument ``skip_dirs``. (:pull:`478`, :pull:`479`).

Breaking changes
^^^^^^^^^^^^^^^^
Expand Down
13 changes: 10 additions & 3 deletions src/xscen/catutils.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,10 +148,12 @@ def _find_assets(
# Split zarr subdirectories from next iteration
zarrs = []
for dr in deepcopy(alldirs):
fdr = Path(top).joinpath(dr)
if dr.endswith(".zarr"):
zarrs.append(dr)
alldirs.remove(dr)
if Path(top).joinpath(dr) in skip_dirs:
if fdr in skip_dirs:
logger.debug("Skipping %s", fdr)
alldirs.remove(dr)

if (
Expand Down Expand Up @@ -323,6 +325,12 @@ def _parse_dir( # noqa: C901
exts = {Path(patt).suffix for patt in patterns}
comp_patterns = list(map(_compile_pattern, patterns))
checks = checks or []
parsed = []

root = Path(root)
if any([(skd in root.parents) or (skd == root) for skd in (skip_dirs or [])]):
logger.debug("Skipping %s", root)
return parsed

# Multithread, communicating via FIFO queues.
# This thread walks the directory
Expand All @@ -334,7 +342,6 @@ def _parse_dir( # noqa: C901
# Usually, the walking is the bottleneck.
q_found = queue.Queue()
q_checked = queue.Queue()
parsed = []

def check_worker():
# Worker that processes the checks.
Expand Down Expand Up @@ -517,7 +524,7 @@ def parse_directory( # noqa: C901
A glob pattern for path matching to accelerate the parsing of a directory tree if only a subtree is needed.
Only folders matching the pattern are parsed to find datasets.
skip_dirs : list of str or Paths, optional
A list of folders that will be removed from the search.
A list of folders that will be removed from the search, should be absolute.
xr_open_kwargs: dict
If needed, arguments to send xr.open_dataset() when opening the file to read the attributes.
only_official_columns: bool
Expand Down

0 comments on commit 0af9f5a

Please sign in to comment.