Linting with Ruff (#1504)

### What kind of change does this PR introduce?

* Removed the configuration blocks in setup.cfg for `pydocstyle` and
`pycodestyle`, and removed most of the checking performed by `flake8`
(flake8 still needed for `flake8-alphabetize` and `flake8-rst`).
* Updated `pyproject.toml` with Ruff-based equivalent linting
configuration
* Updated `tox.ini` and `Makefile` to use Ruff. Reordered order of
linting calls to trigger more typical violations first.
* Updated dependencies.

### Does this PR introduce a breaking change?

Yes. Ruff is now a dev dependency, while `pycodestyle` and `pydocstyle`
are no longer required. The checks that were being performed by those
libraries are now redundant thanks to the new configuration.

### Other information:

Ruff is a linter for Python (built with Rust) that is blazingly fast and
has been in development for around 1.5 years. [Ruff v0.1.0 was released
today](https://astral.sh/blog/ruff-v0.1.0).

.pre-commit-config.yaml

@@ -36,22 +36,25 @@ repos:
   rev: 23.9.1
   hooks:
   - id: black
+- repo: https://github.com/PyCQA/isort
+  rev: 5.12.0
+  hooks:
+  - id: isort
+    args: ['--settings-file=pyproject.toml']
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.1.0
+  hooks:
+  - id: ruff
 - repo: https://github.com/pycqa/flake8
   rev: 6.1.0
   hooks:
   - id: flake8
     additional_dependencies: ['flake8-alphabetize', 'flake8-rst-docstrings']
     args: ['--config=setup.cfg']
-- repo: https://github.com/PyCQA/isort
-  rev: 5.12.0
-  hooks:
-  - id: isort
-    args: ['--settings-file=pyproject.toml', "--add_imports='from __future__ import annotations'"]
 - repo: https://github.com/nbQA-dev/nbQA
   rev: 1.7.0
   hooks:
   - id: nbqa-black
-    args: ['--target-version=py38']
   - id: nbqa-pyupgrade
     args: ['--py38-plus']
   - id: nbqa-isort
@@ -62,11 +65,6 @@ repos:
   - id: nbstripout
     files: '.ipynb'
     args: [ "--extra-keys", "metadata.kernelspec" ]
-- repo: https://github.com/pycqa/pydocstyle
-  rev: 6.3.0
-  hooks:
-  - id: pydocstyle
-    args: ['--config=setup.cfg']
 - repo: https://github.com/keewis/blackdoc
   rev: v0.3.8
   hooks:

CHANGES.rst

@@ -38,6 +38,7 @@ Breaking changes
 * For better accuracy, the `tox -e black` recipe has been renamed to `tox -e lint`, as this configuration already included several other linting checks. (:pull:`1473`).
 * ``xclim.indices.winter_storm`` renamed to ``xclim.indices.snd_storm_days``.  (:pull:`1505`).
 * Default threshold in ``xclim.indices.snw_season_{start|length|end}`` changed form `20 kg m-2` to `4 kg m-2`. (:pull:`1505`).
+* `xclim` development dependencies now include `ruff`. `pycodestyle` and `pydocstyle` have been replaced by `ruff` and removed from the `dev` installation recipe. (:pull:`1504`).
 
 Internal changes
 ^^^^^^^^^^^^^^^^
@@ -56,6 +57,7 @@ Internal changes
 * Added basic keywords on most indicators for easier searching in the docs. Extracted climate indicators API to its own page for faster loading. (:pull:`1502`, :issue:`1433`).
 * `nbstripout` now removes 'metadata.kernelspec' in notebook cells. (:pull:`1407`).
 * Deprecation wrapper ``xclim.core.utils.deprecated`` are added to help with deprecation warnings. (:pull:`1505`).
+* `xclim` now uses `ruff` to format the codebase with `make lint` and `pre-commit`. `flake8` is still used for the time being, solely to enforce docstring linting (with `flake8-rst-docstrings`) and alphabetical `__all__` entries (with `flake8-alphabetize`). (:pull:`1504`).
 
 v0.45.0 (2023-09-05)
 --------------------

Makefile

@@ -53,14 +53,13 @@ clean-test: ## remove test and coverage artifacts
 	rm -fr .pytest_cache
 
 lint: ## check style with flake8 and black
-	pycodestyle --config=setuyp.cfg xclim tests
-	pydocstyle --config=setup.cfg xclim tests
-	flake8 --config=setup.cfg xclim tests
 	black --check xclim tests
+	isort --check xclim tests
+	ruff xclim tests
+	flake8 --config=setup.cfg xclim tests
 	nbqa black --check docs
 	blackdoc --check --exclude=xclim/indices/__init__.py xclim
 	blackdoc --check docs
-	isort --check xclim tests
 	yamllint --config-file=.yamllint.yaml xclim
 
 test: ## run tests quickly with the default Python

README.rst

@@ -9,7 +9,7 @@ xclim: Climate services library |logo|
 +----------------------------+-----------------------------------------------------+
 | Open Source                | |license| |fair| |zenodo| |pyOpenSci| |joss|        |
 +----------------------------+-----------------------------------------------------+
-| Coding Standards           | |black| |pre-commit| |security| |fossa|             |
+| Coding Standards           | |black| |ruff| |pre-commit| |security| |fossa|      |
 +----------------------------+-----------------------------------------------------+
 | Development Status         | |status| |build| |coveralls|                        |
 +----------------------------+-----------------------------------------------------+
@@ -183,6 +183,10 @@ This package was created with Cookiecutter_ and the `audreyfeldroy/cookiecutter-
         :target: https://results.pre-commit.ci/latest/github/Ouranosinc/xclim/master
         :alt: pre-commit.ci status
 
+.. |ruff| image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
+    :target: https://github.com/astral-sh/ruff
+    :alt: Ruff
+
 .. |status| image:: https://www.repostatus.org/badges/latest/active.svg
         :target: https://www.repostatus.org/#active
         :alt: Project Status: Active – The project has reached a stable, usable state and is being actively developed.

docs/rstjinja.py

@@ -2,9 +2,7 @@
 
 
 def rstjinja(app, docname, source):
-    """
-    Render our pages as a jinja template for fancy templating goodness.
-    """
+    """Render our pages as a jinja template for fancy templating goodness."""
     # Make sure we're outputting HTML
     if app.builder.format != "html":
         return

environment.yml

@@ -3,37 +3,37 @@ channels:
     - conda-forge
     - defaults
 dependencies:
-    - python>=3.8
+    - python >=3.8
     - astroid
-    - boltons>=20.1
-    - bottleneck>=1.3.1
-    - cf_xarray>=0.6.1
-    - cftime>=1.4.1
+    - boltons >=20.1
+    - bottleneck >=1.3.1
+    - cf_xarray >=0.6.1
+    - cftime >=1.4.1
     - Click >=8.1
-    - dask>=2.6.0
+    - dask >=2.6.0
     - importlib-resources  # For Python3.8
     - jsonpickle
     - lmoments3
     - numba
-    - numpy>=1.16
-    - pandas>=0.23
-    - pint>=0.9
-    - poppler>=0.67
+    - numpy >=1.16
+    - pandas >=0.23
+    - pint >=0.9
+    - poppler >=0.67
     - pyyaml
-    - scikit-learn>=0.21.3
-    - scipy>=1.2
+    - scikit-learn >=0.21.3
+    - scipy >=1.2
     - statsmodels
-    - xarray>=2022.06.0
+    - xarray >=2022.06.0
     # Extras
     - eofs
     - flox
   # Testing and development dependencies
-    - black>=22.12
+    - black >=22.12
     - blackdoc
     - bump2version
     - cairosvg
     - coverage
-    - distributed>=2.0
+    - distributed >=2.0
     - filelock
     - flake8
     - flake8-rst-docstrings
@@ -47,23 +47,23 @@ dependencies:
     - nbsphinx
     - nbval
     - nc-time-axis
-    - netCDF4>=1.4
+    - netCDF4 >=1.4
     - notebook
     - platformdirs
     - pooch
     - pre-commit
     - pybtex
-    - pydocstyle
     - pylint
     - pytest
     - pytest-cov
     - pytest-socket
-    - pytest-xdist>=3.2
+    - pytest-xdist >=3.2
+    - ruff >=0.1.0
     - sphinx
     - sphinx-autodoc-typehints
     - sphinx-codeautolink
     - sphinx-copybutton
-    - sphinx_rtd_theme>=1.0
+    - sphinx_rtd_theme >=1.0
     - sphinxcontrib-bibtex
     - tokenize-rt
     - tox

pyproject.toml

@@ -71,13 +71,13 @@ dev = [
   "netCDF4 >=1.4",
   "platformdirs >=3.2",
   "pre-commit >=2.9",
-  "pydocstyle >=5.1.1",
   "pybtex",
   "pylint",
   "pytest",
   "pytest-cov",
   "pytest-socket",
   "pytest-xdist[psutil] >=3.2",
+  "ruff >=0.1.0",
   "tokenize-rt",
   "tox",
   # "tox-conda",  # Will be added when a [email protected]+ compatible plugin is released.
@@ -210,3 +210,68 @@ markers = [
   "requires_docs: mark tests that can only be run with documentation present (deselect with '-m \"not requires_docs\"')",
   "requires_internet: mark tests that require internet access (deselect with '-m \"not requires_internet\"')"
 ]
+
+[tool.ruff]
+src = ["xclim"]
+line-length = 150
+target-version = "py38"
+exclude = [
+  ".git",
+  "docs",
+  "build",
+  ".eggs",
+  "tests"
+]
+ignore = [
+  "D205",
+  "D400",
+  "D401"
+]
+select = [
+  "C9",
+  "D",
+  "E",
+  "F",
+  "W"
+]
+
+[tool.ruff.flake8-bandit]
+check-typed-exception = true
+
+[tool.ruff.flake8-import-conventions.aliases]
+"matplotlib.pyplot" = "plt"
+"xclim.indices" = "xci"
+numpy = "np"
+pandas = "pd"
+scipy = "sp"
+xarray = "xr"
+
+[tool.ruff.format]
+line-ending = "auto"
+
+[tool.ruff.isort]
+known-first-party = ["xclim"]
+case-sensitive = true
+detect-same-package = false
+lines-after-imports = 1
+no-lines-before = ["future", "standard-library"]
+
+[tool.ruff.mccabe]
+max-complexity = 15
+
+[tool.ruff.per-file-ignores]
+"docs/*.py" = ["D100", "D101", "D102", "D103"]
+"tests/*.py" = ["D100", "D101", "D102", "D103"]
+"xclim/**/__init__.py" = ["F401", "F403"]
+"xclim/core/indicator.py" = ["D214", "D405", "D406", "D407", "D411"]
+"xclim/core/locales.py" = ["E501", "W505"]
+"xclim/core/missing" = ["D103"]
+"xclim/indices/_agro.py" = ["E501"]
+"xclim/indices/fire/_cffwis.py" = ["D103"]
+"xclim/sdba/utils.py" = ["D103"]
+
+[tool.ruff.pycodestyle]
+max-doc-length = 180
+
+[tool.ruff.pydocstyle]
+convention = "numpy"

setup.cfg

@@ -3,13 +3,13 @@ current_version = 0.45.18-beta
 commit = True
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+).(?P<patch>\d+)(\-(?P<release>[a-z]+))?
-serialize = 
+serialize =
 	{major}.{minor}.{patch}-{release}
 	{major}.{minor}.{patch}
 
 [bumpversion:part:release]
 optional_value = gamma
-values = 
+values =
 	beta
 	gamma
 
@@ -18,40 +18,27 @@ search = __version__ = "{current_version}"
 replace = __version__ = "{new_version}"
 
 [flake8]
-exclude = 
+exclude =
 	.git,
 	docs,
 	build,
 	.eggs,
 	tests
-max-line-length = 88
-max-complexity = 12
-ignore = 
+ignore =
 	AZ100,
 	AZ200,
 	AZ300,
-	C901,
-	D107,
-	D205,
-	D400,
-	D401,
-	E203,
-	E501,
-	F401,
-	F403,
+	C,
+	D,
+    E,
+    F,
 	W503
-per-file-ignores = 
-	tests/*:E402
+per-file-ignores =
 	xclim/core/locales.py:RST399
-	xclim/core/missing:D103
-	xclim/indices/fire/_cffwis.py:D103
-	xclim/indices/fire/_ffdi.py:D403
-	xclim/sdba/utils.py:D103
-	xclim/testing/*:D103
-rst-directives = 
+rst-directives =
 	bibliography,
 	autolink-skip
-rst-roles = 
+rst-roles =
 	doc,
 	mod,
 	py:attr,
@@ -70,24 +57,3 @@ rst-roles =
 	cite:p,
 	cite:t,
 	cite:ts
-
-[pycodestyle]
-count = False
-exclude = tests
-ignore = 
-	E203,
-	E226,
-	E402,
-	E501,
-	W503
-max-line-length = 120
-statistics = True
-
-[pydocstyle]
-convention = numpy
-add-ignore = 
-	D205,
-	D400,
-	D401,
-	D403
-match = ((?!(test_|conftest|conf|rstjinja|autodoc_indicator)).)*\.py

tests/conftest.py

@@ -440,7 +440,6 @@ def gather_session_data(threadsafe_data_dir, worker_id, xdoctest_namespace):
     Additionally, this fixture is also used to generate the `atmosds` synthetic testing dataset as well as add the
     example file paths to the xdoctest_namespace, used when running doctests.
     """
-
     if (
         not _default_cache_dir.joinpath(helpers.TESTDATA_BRANCH).exists()
         or helpers.PREFETCH_TESTING_DATA

tests/test_analog.py

@@ -20,7 +20,6 @@ def matlab_sample(n=30):
     Climate Analogs" to compare against the functions here. The sample
     created here is identical to the sample used to drive the Matlab code.
 
-
     Parameters
     ----------
     n : int
@@ -31,7 +30,6 @@ def matlab_sample(n=30):
     2D array, 2D array
        Synthetic samples (3, n)
     """
-
     z = 1.0 * (np.arange(n) + 1) / n - 0.5
 
     x = np.vstack([z * 2 + 30, z * 3 + 40, z]).T
@@ -318,7 +316,6 @@ def test_different_sample_size(self, random):
     #
     def test_mvnormal(self, random):
         """Compare the results to the figure 2 in the paper."""
-
         n = 30000
         p = random.normal(0, 1, size=(n, 2))
         q = random.multivariate_normal([0.5, -0.5], [[0.5, 0.1], [0.1, 0.3]], size=n)