Offline testing and test data fetching improvements

.github/workflows/main.yml

@@ -20,10 +20,10 @@ on:
       - submitted
 
 env:
-  XCLIM_TESTDATA_BRANCH: v2023.6.28
+  XCLIM_TESTDATA_BRANCH: v2023.9.12
 
 jobs:
-  black:
+  lint:
     name: Black (Python${{ matrix.python-version }})
     runs-on: ubuntu-latest
     if: |
@@ -49,11 +49,11 @@ jobs:
       - name: Run pylint
         run: pylint --rcfile=pylintrc --disable=import-error --exit-zero xclim
       - name: Run linting suite
-        run: tox -e black
+        run: tox -e lint
 
   test-py39:
     name: test-${{ matrix.tox-env }} (Python${{ matrix.python-version }})
-    needs: black
+    needs: lint
     if: |
       (github.event_name == 'pull_request') && !contains(github.event.pull_request.labels.*.name, 'approved')
     runs-on: ubuntu-latest
@@ -76,7 +76,7 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
   test-pypi:
-    needs: black
+    needs: lint
     name: test-${{ matrix.tox-env }} (Python${{ matrix.python-version }})
     if: |
       contains(github.event.pull_request.labels.*.name, 'approved') ||
@@ -88,12 +88,20 @@ jobs:
         include:
           - tox-env: py38-coverage-eofs
             python-version: "3.8"
+            markers: -m 'not slow'
           - tox-env: py39-coverage-sbck-eofs
             python-version: "3.9"
-          - tox-env: notebooks_doctests
+            markers: -m 'not slow'
+          - tox-env: py310-coverage  # No markers -- includes slow tests
             python-version: "3.10"
           - tox-env: py311-coverage-sbck
             python-version: "3.11"
+            markers: -m 'not slow'
+          - tox-env: notebooks_doctests
+            python-version: "3.10"
+          - tox-env: offline-prefetch
+            python-version: "3.11"
+            markers: -m 'not slow and not requires_internet'
     steps:
       - uses: actions/[email protected]
       - name: Install Eigen3
@@ -108,15 +116,15 @@ jobs:
       - name: Install tox
         run: pip install tox~=4.0
       - name: Test with tox
-        run: tox -e ${{ matrix.tox-env }}
+        run: tox -e ${{ matrix.tox-env }} -- ${{ matrix.markers }}
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           COVERALLS_FLAG_NAME: run-{{ matrix.tox-env }}
           COVERALLS_PARALLEL: true
           COVERALLS_SERVICE_NAME: github
 
   test-conda:
-    needs: black
+    needs: lint
     name: test-conda-${{ matrix.tox-env }} (Python${{ matrix.python-version }})
     if: |
       contains(github.event.pull_request.labels.*.name, 'approved') ||

CHANGES.rst

@@ -2,13 +2,32 @@
 Changelog
 =========
 
-v0.46.0
+v0.46.0 (unreleased)
 --------------------
-Contributors to this version: Éric Dupuis (:user:`coxipi`).
+Contributors to this version: Éric Dupuis (:user:`coxipi`), Trevor James Smith (:user:`Zeitsperre`).
+
+New features and enhancements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+* `xclim` now has a dedicated console command for prefetching testing data from `xclim-testdata` with branch options (e.g.: `$ xclim prefetch_testing_data --branch some_development_branch`). This command can be used to download the testing data to a local cache, which can then be used to run the testing suite without internet access or in "offline" mode. For more information, see the contributing documentation section for `Updating Testing Data`. (:issue:`1468`, :pull:`1473`).
+* The testing suite now offers a means of running tests in "offline" mode (using `pytest-socket <https://github.com/miketheman/pytest-socket>`_ to block external connections). This requires a local copy of `xclim-testdata` to be present in the user's home cache directory and for certain `pytest` options and markers to be set when invoked. For more information, see the contributing documentation section for `Running Tests in Offline Mode`. (:issue:`1468`, :pull:`1473`).
+
+Bug fixes
+^^^^^^^^^
+* Fixed an error in the `pytest` configuration that prevented copying of testing data to thread-safe caches of workers under certain conditions (this should always occur). (:pull:`1473`).
+  * Coincidentally, this also fixes an error that caused `pytest` to error-out when invoked without an active internet connection. Running `pytest` without network access is now supported (requires cached testing data). (:issue:`1468`).
+
+Breaking changes
+^^^^^^^^^^^^^^^^
+* `pytest-socket` is now a required development dependency for running `"offline"` tests or the `"offline"` configuration of the `tox` testing suite. This has been added to the `dev` installation recipe. (:issue:`1468`, :pull:`1473`).
+* For better transparency and control in development, the `tox` configuration has been adapted to allow passing of markers directly to the `pytest` call. Positional arguments must be passed to tox after the `--` separator to select/deselect tests (e.g. ``'tox -e py38 -- -m "not slow"'``). (:pull:`1473`).
+* For better accuracy, the `tox -e black` recipe has been renamed to `tox -e lint`, as this configuration already included several other linting checks. (:pull:`1473`).
 
 Internal changes
 ^^^^^^^^^^^^^^^^
-* Change "degK" to "K" used to design Kelvin units. (:pull:`1475`).
+* Changed "degK" to "K" (used to designate Kelvin units). (:pull:`1475`).
+* Added a `pytest` marker (``pytest.mark.requires_internet``) to allow for skipping of tests that depend on remote network calls to function properly. (:pull:`1473`).
+* Added handling for `pytest-socket`'s ``SocketBlockedError`` in ``xclim.testing.open_dataset`` when attempting to fetch md5 validation files for cached testing data while explicitly disabling internet sockets. (:issue:`1468`, :pull:`1473`).
+* Updated the testing data used in the `analogs.ipynb` notebook to use the testing data now found in `Ouranosinc/xclim-testdata`'s main branch. (`xclim-testdata PR/26 <https://github.com/Ouranosinc/xclim-testdata/pull/26>`_, :pull:`1473`).
 
 v0.45.0 (2023-09-05)
 --------------------

CONTRIBUTING.rst

@@ -146,15 +146,35 @@ Ready to contribute? Here's how to set up `xclim` for local development.
     $ pydocstyle --config=setup.cfg xclim xclim
     $ yamllint --config-file .yamllint.yaml xclim
 
-6. When unit/doc tests are added or notebooks updated, use ``$ pytest`` to run them. Alternatively, one can use ``$ tox`` to run all testing suites as would github do when the PR is submitted and new commits are pushed::
+6. When features or bug fixes have been contributed, unit tests and doctests have been added, or notebooks have been updated, use ``$ pytest`` to test them::
 
     $ pytest --no-cov --nbval --dist=loadscope --rootdir=tests/ docs/notebooks --ignore=docs/notebooks/example.ipynb  # for notebooks, exclusively.
     $ pytest --no-cov --rootdir=tests/ --xdoctest xclim  # for doctests, exclusively.
     $ pytest  # for all unit tests, excluding doctests and notebooks.
-    $ tox  # run all testing suites
+    $ pytest -m "not slow"  # for all unit tests, excluding doctests, notebooks, and "slow" marked tests.
+
+  Alternatively, one can use ``$ tox`` to run very specific testing configurations, as GitHub Workflows would do when a Pull Request is submitted and new commits are pushed::
+
+    $ tox -e py38  # run tests on Python 3.8
+    $ tox -e py39-upstream-doctest  # run tests on Python 3.9, including doctests, with upstream dependencies
+    $ tox -e py310 -- -m "not slow  # run tests on Python 3.10, excluding "slow" marked tests
+    $ tox -e py311  # run tests on Python 3.11
+    $ tox -e notebooks_doctests  # run tests using the base Python on doctests and evaluate all notebooks
+    $ tox -e offline  # run tests using the base Python, excluding tests requiring internet access
+
+    $ tox -m test  # run all builds listed above
+
+.. warning::
+    Starting from `xclim` v0.46.0, when running tests with `tox`, any `pytest` markers passed to `pyXX` builds (e.g. `-m "not slow"`) must be passed to `tox` directly. This can be done as follows::
+
+        $ tox -e py38 -- -m "not slow"
+
+    The exceptions to this rule are:
+      `notebooks_doctests`: this configuration does not pass test  markers to its `pytest` call.
+      `offline`: this configuration runs by default with the `-m "not requires_internet"` test marker. Be aware that running `tox` and manually setting a `pytest` marker will override this default.
 
 .. note::
-    `xclim` tests are organized to support the `pytest-xdist <https://pytest-xdist.readthedocs.io/en/latest/>`_ plugin for distributed testing across workers or CPUs. In order to benefit from multiple processes, add the flag `--numprocesses=auto` or `-n auto` to your `pytest` calls.
+    `xclim` tests are organized to support the `pytest-xdist`_ plugin for distributed testing across workers or CPUs. In order to benefit from multiple processes, add the flag `--numprocesses=auto` or `-n auto` to your `pytest` calls.
 
     When running tests via `tox`, `numprocesses` is set to the number of logical cores available (`numprocesses=logical`), with a maximum amount of `8`.
 
@@ -234,8 +254,7 @@ Before you submit a pull request, please follow these guidelines:
 Updating Testing Data
 ~~~~~~~~~~~~~~~~~~~~~
 
-If your code changes require changes to the testing data of `xclim` (i.e.: modifications to existing datasets or new datasets),
-these changes must be made via a Pull Request at https://github.com/Ouranosinc/xclim-testdata.
+If your code changes require changes to the testing data of `xclim` (i.e.: modifications to existing datasets or new datasets), these changes must be made via a Pull Request at the `xclim-testdata repository`_.
 
 `xclim` allows for developers to test specific branches/versions of `xclim-testdata` via the `XCLIM_TESTDATA_BRANCH` environment variable, either through export, e.g.::
 
@@ -253,6 +272,19 @@ or by setting the variable at runtime::
 
 This will ensure that tests load the testing data from this branch before running.
 
+If you anticipate not having internet access, we suggest prefetching the testing data from `xclim-testdata repository`_ and storing it in your local cache. This can be done by running the following console command::
+
+    $ xclim prefetch_testing_data
+
+If your development branch relies on a specific branch of `Ouranosinc/xclim-testdata`, you can specify this using environment variables::
+
+    $ export XCLIM_TESTDATA_BRANCH="my_new_branch_of_testing_data"
+    $ xclim prefetch_testing_data
+
+or, alternatively, with the `--branch` option::
+
+    $ xclim prefetch_testing_data --branch my_new_branch_of_testing_data
+
 If you wish to test a specific branch using GitHub CI, this can be set in `.github/workflows/main.yml`:
 
 .. code-block:: yaml
@@ -261,9 +293,24 @@ If you wish to test a specific branch using GitHub CI, this can be set in `.gith
       XCLIM_TESTDATA_BRANCH: my_new_branch_of_testing_data
 
 .. warning::
-    In order for a Pull Request to be allowed to merge to main development branch, this variable must match the latest tagged commit name on `Ouranosinc/xclim-testdata`.
+    In order for a Pull Request to be allowed to merge to main development branch, this variable must match the latest tagged commit name on `xclim-testdata repository`_.
     We suggest merging changed testing data first, tagging a new version of `xclim-testdata`, then re-running tests on your Pull Request at `Ouranosinc/xclim` with the newest tag.
 
+Running Tests in Offline Mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`xclim` testing is designed with the assumption that the machine running the tests has internet access. Many calls to `xclim` functions will attempt to download data or verify checksums from the `Ouranosinc/xclim-testdata` repository. This can be problematic for developers working on features where internet access is not reliably available.
+
+If you wish to ensure that your feature or bugfix can be developed without internet access, `xclim` leverages the `pytest-socket`_ plugin so that testing can be run in "offline" mode by invoking pytest with the following options::
+
+    $ pytest --disable-socket --allow-unix-socket -m "not requires_internet"
+
+or, alternatively, using `tox` ::
+
+    $ tox -e offline
+
+These options will disable all network calls and skip tests marked with the `requires_internet` marker. The `--allow-unix-socket` option is required to allow the `pytest-xdist`_ plugin to function properly.
+
 Tips
 ----
 
@@ -351,7 +398,7 @@ When publishing on GitHub, maintainers will need to generate the release notes f
 The Manual Approach
 ~~~~~~~~~~~~~~~~~~~
 
-The manual approach to library packaging for general support (pip wheels) requires that the `flit <https://flit.pypa.io/en/stable/index.html>`_ library is installed.
+The manual approach to library packaging for general support (pip wheels) requires that the `flit`_ library is installed.
 
 From the command line on your Linux distribution, simply run the following from the clone's main dev branch::
 
@@ -385,8 +432,12 @@ Before updating the main conda-forge recipe, we *strongly* suggest performing th
 
 .. _`GitHub Repository`: https://github.com/Ouranosinc/xclim
 .. _`PEP8`: https://peps.python.org/pep-0008/
+.. _`flit`: https://flit.pypa.io/en/stable/index.html
 .. _`numpydoc`: https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard
+.. _`pytest-socket`: https://github.com/miketheman/pytest-socket
+.. _`pytest-xdist`: https://pytest-xdist.readthedocs.io/en/latest/
 .. _`reStructuredText (ReST)`: https://www.jetbrains.com/help/pycharm/using-docstrings-to-specify-types.html
 .. _`reStructuredText Primer`: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
 .. _`sphinxcontrib-bibtex`: https://sphinxcontrib-bibtex.readthedocs.io
 .. _`xclim on TestPyPI`: https://test.pypi.org/project/xclim/
+.. _`xclim-testdata repository`: https://github.com/Ouranosinc/xclim-testdata

docs/notebooks/analogs.ipynb

@@ -49,7 +49,6 @@
    "source": [
     "sim = open_dataset(\n",
     "    \"SpatialAnalogs/CanESM2_ScenGen_Chibougamau_2041-2070.nc\",\n",
-    "    branch=\"spatial-analogs-nb\",\n",
     "    decode_timedelta=False,\n",
     ")\n",
     "sim"
@@ -74,7 +73,6 @@
    "source": [
     "obs = open_dataset(\n",
     "    \"SpatialAnalogs/NRCAN_SECan_1981-2010.nc\",\n",
-    "    branch=\"spatial-analogs-nb\",\n",
     "    decode_timedelta=False,\n",
     ")\n",
     "obs"
@@ -88,7 +86,7 @@
    "outputs": [],
    "source": [
     "obs.tg_mean.isel(time=0).plot()\n",
-    "plt.plot(sim.lon, sim.lat, \"ro\");  # Plot a point over chibougamau"
+    "plt.plot(sim.lon, sim.lat, \"ro\");  # Plot a point over Chibougamau"
    ]
   },
   {

docs/notebooks/cli.ipynb

@@ -93,7 +93,6 @@
     "from __future__ import annotations\n",
     "\n",
     "import warnings\n",
-    "from pathlib import Path\n",
     "\n",
     "import numpy as np\n",
     "import pandas as pd\n",

environment.yml

@@ -55,6 +55,7 @@ dependencies:
     - pylint
     - pytest
     - pytest-cov
+    - pytest-socket
     - pytest-xdist>=3.2
     - sphinx
     - sphinx-autodoc-typehints

pyproject.toml

@@ -75,6 +75,7 @@ dev = [
   "pylint",
   "pytest",
   "pytest-cov",
+  "pytest-socket",
   "pytest-xdist[psutil] >=3.2",
   "tokenize-rt",
   "tox",
@@ -205,5 +206,6 @@ usefixtures = "xdoctest_namespace"
 doctest_optionflags = ["NORMALIZE_WHITESPACE", "IGNORE_EXCEPTION_DETAIL", "NUMBER", "ELLIPSIS"]
 markers = [
   "slow: marks tests as slow (deselect with '-m \"not slow\"')",
-  "requires_docs: mark tests that can only be run with documentation present (deselect with '-m \"not requires_docs\"')"
+  "requires_docs: mark tests that can only be run with documentation present (deselect with '-m \"not requires_docs\"')",
+  "requires_internet: mark tests that require internet access (deselect with '-m \"not requires_internet\"')"
 ]

tests/conftest.py

@@ -434,13 +434,19 @@ def gather_session_data(threadsafe_data_dir, worker_id, xdoctest_namespace):
     """Gather testing data on pytest run.
 
     When running pytest with multiple workers, one worker will copy data remotely to _default_cache_dir while
-    other workers wait using lockfile. Once the lock is released, all workers will copy data to their local
-    threadsafe_data_dir."""
+    other workers wait using lockfile. Once the lock is released, all workers will then copy data to their local
+    threadsafe_data_dir.As this fixture is scoped to the session, it will only run once per pytest run.
+
+    Additionally, this fixture is also used to generate the `atmosds` synthetic testing dataset as well as add the
+    example file paths to the xdoctest_namespace, used when running doctests.
+    """
 
     if (
         not _default_cache_dir.joinpath(helpers.TESTDATA_BRANCH).exists()
         or helpers.PREFETCH_TESTING_DATA
     ):
+        if helpers.PREFETCH_TESTING_DATA:
+            print("`XCLIM_PREFETCH_TESTING_DATA` set. Prefetching testing data...")
         if worker_id in "master":
             helpers.populate_testing_data(branch=helpers.TESTDATA_BRANCH)
         else:
@@ -451,7 +457,7 @@ def gather_session_data(threadsafe_data_dir, worker_id, xdoctest_namespace):
                 helpers.populate_testing_data(branch=helpers.TESTDATA_BRANCH)
                 _default_cache_dir.joinpath(".data_written").touch()
             fl.acquire()
-        shutil.copytree(_default_cache_dir, threadsafe_data_dir)
+    shutil.copytree(_default_cache_dir, threadsafe_data_dir)
     helpers.generate_atmos(threadsafe_data_dir)
     xdoctest_namespace.update(helpers.add_example_file_paths(threadsafe_data_dir))