Add pytest-xdist (#464)

### What kind of change does this PR introduce?

* Adjusts the code base to potentially benefit from `pytest-xdist`
* Removes several old `tox-conda` configurations (sadly, the library is
no longer maintained).

### Does this PR introduce a breaking change?

`pytest-xdist` is now a required development dependency, even if users
don't use it.

### Other information:

https://pytest-xdist.readthedocs.io/en/stable/

`pytest-xdist` is a tool for distributing tests across workers. Assuming
that the setup is not too heavy, this greatly reduces the amount of time
that `pytest` needs to run tests. The testing code base is now free of
potential data races, so there should be no issues.

`pytest-xdist` is disabled by default as the spin-up time doesn't merit
using it yet, but once tests become more numerous, it may be worthwhile
to enable it.

.github/workflows/main.yml

@@ -161,7 +161,7 @@ jobs:
           python -m pip check || true
       - name: Test with pytest
         run: |
-          python -m pytest --cov xscen
+          python -m pytest
       - name: Report Coverage
         run: |
           python -m coveralls

CHANGELOG.rst

@@ -22,6 +22,7 @@ New features and enhancements
 * Convenience functions ``xs.io.zip_directory`` and ``xs.io.unzip_directory`` (for zarrs). (:pull:`431`).
 * New argument ``compute_indicators``: ``rechunk_input`` to rechunk the inputs to resample-appropriate chunks before calling xclim. (:pull:`431`).
 * New ``xs.indicators.get_indicator_outputs`` to retrieve what variable name(s) and frequency to expect from an xclim indicator. (:pull:`431`).
+* `xscen` now supports launches tests from `pytest` with the `--numprocesses` option. See the `pytest-xdist documentation <https://pytest-xdist.readthedocs.io/en/stable/>`_ for more information. (:pull:`464`).
 
 Bug fixes
 ^^^^^^^^^
@@ -45,6 +46,7 @@ Breaking changes
 * `attrs_to_remove` and `remove_all_attrs_except` in ``clean_up`` now use real regex. It should not be too breaking since a `fullmatch()` is used, but `*` is now `.*`. (:pull:`450`).
 * Python 3.9 is no longer supported. (:pull:`456`).
 * Functions and arguments that were deprecated in `xscen` v0.8.0 or earlier have been removed. (:pull:`461`).
+* `pytest-xdist` is now a development dependency. (:pull:`464`).
 
 Internal changes
 ^^^^^^^^^^^^^^^^

environment-dev.yml

@@ -57,10 +57,11 @@ dependencies:
   - pre-commit >=3.5.0
   - pytest >=8.3.2
   - pytest-cov >=5.0.0
+  - pytest-xdist >=3.2.0
   - ruff >=0.5.7
   - setuptools >=65.0.0
   - setuptools-scm >=8.0.0
-  - sphinx
+  - sphinx >=7.0.0
   - sphinx-autoapi
   - sphinx-rtd-theme >=1.0
   - sphinxcontrib-napoleon

pyproject.toml

@@ -88,6 +88,7 @@ dev = [
   "pre-commit >=3.3.2",
   "pytest-cov >=5.0.0",
   "pytest >=8.3.2",
+  "pytest-xdist[psutil] >=3.2.0",
   "ruff >=0.5.7",
   "tox >=4.18.0",
   "watchdog >=4.0.0",
@@ -249,8 +250,12 @@ addopts = [
   "--color=yes",
   "--cov=xscen",
   "--ignore-glob='*.ipynb_checkpoints'",
+  "--strict-config",
   "--strict-markers",
-  "--verbose"
+  "--verbose",
+  "--numprocesses=0",
+  "--maxprocesses=8",
+  "--dist=worksteal"
 ]
 filterwarnings = ["ignore::UserWarning"]
 testpaths = "tests"

tests/test_biasadjust.py

@@ -129,15 +129,15 @@ def test_basic(
         self, periods, to_level, bias_adjust_institution, bias_adjust_project
     ):
         dtrain = xs.train(
-            self.dref,
+            self.dref.copy(),
             self.dsim.sel(time=slice("2001", "2003")),
             var="tas",
             period=["2001", "2003"],
         )
 
         out = xs.adjust(
             dtrain,
-            self.dsim,
+            self.dsim.copy(),
             periods=periods,
             to_level=to_level,
             bias_adjust_institution=bias_adjust_institution,
@@ -175,9 +175,9 @@ def test_basic(
                 np.concatenate([np.ones(365 * 1) * 1, np.ones(365 * 1) * 3]),
             )
 
-    def test_write_train(self):
+    def test_write_train(self, tmpdir):
         dtrain = xs.train(
-            self.dref,
+            self.dref.copy(),
             self.dsim.sel(time=slice("2001", "2003")),
             var="tas",
             period=["2001", "2003"],
@@ -186,15 +186,15 @@ def test_write_train(self):
             jitter_under={"thresh": "2 K"},
         )
 
-        root = str(notebooks / "_data")
+        root = str(tmpdir / "_data")
         xs.save_to_zarr(dtrain, f"{root}/test.zarr", mode="o")
         dtrain2 = xr.open_dataset(
             f"{root}/test.zarr", chunks={"dayofyear": 365, "quantiles": 15}
         )
 
         out = xs.adjust(
             dtrain,
-            self.dsim,
+            self.dsim.copy(),
             periods=["2001", "2006"],
             xclim_adjust_args={
                 "detrend": {
@@ -205,7 +205,7 @@ def test_write_train(self):
 
         out2 = xs.adjust(
             dtrain2,
-            self.dsim,
+            self.dsim.copy(),
             periods=["2001", "2006"],
             xclim_adjust_args={
                 "detrend": {

tox.ini

@@ -21,8 +21,6 @@ python =
 description = Check for Code Compliance and missing french translations
 skip_install = True
 download = true
-conda_channels =
-conda_env =
 deps =
     black[jupyter] ==24.8.0
     blackdoc ==0.3.9
@@ -41,8 +39,6 @@ allowlist_externals =
 
 [testenv:docs{,-esmpy}]
 description = Run Build of xscen Documentation
-conda_deps =
-conda_env = environment-dev.yml
 extras =
     docs
 commands =
@@ -52,8 +48,6 @@ allowlist_externals =
 
 [testenv:doctests{,-esmpy}]
 description = Run documentation linters and doctests with pytest under {basepython}
-conda_deps =
-conda_env = environment-dev.yml
 extras =
     dev
     docs
@@ -65,28 +59,21 @@ commands =
 description = Run tests with pytest under {basepython} (Anaconda distribution)
 setenv =
     COV_CORE_SOURCE =
-    PYTEST_ADDOPTS = --color=yes --cov=xscen --strict-markers --verbose
     PYTHONPATH = {toxinidir}
 passenv =
+    CI
     COVERALLS_*
     ESMFMKFILE
     ESMF_VERSION
     GITHUB_*
+    XCLIM_*
 download = true
 deps =
     coveralls: coveralls
     esmpy: git+https://github.com/esmf-org/esmf.git@v{env:ESMF_VERSION}\#subdirectory=src/addon/esmpy
     upstream: -rrequirements_upstream.txt
 extras =
     dev
-conda_channels =
-    conda-forge
-    defaults
-conda_deps =
-    pytest
-    pytest-cov
-    xdoctest
-conda_env = environment.yml
 install_command = python -m pip install --no-user {opts} {packages}
 commands_pre =
     pip list