Merge remote-tracking branch 'upstream/main' into unpin_rio

.github/workflows/pip-checks.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       matrix:
         os: ["ubuntu-latest", "macos-latest", "windows-latest"]
-        python-version: ["3.9", "3.10", "3.11"]
+        python-version: ["3.10", "3.11", "3.12"]
 
     # Run all shells using bash (including Windows)
     defaults:

.github/workflows/python-publish.yml

@@ -26,7 +26,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v5
       with:
-        python-version: '3.11'
+        python-version: '3.12'
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip

.github/workflows/python-tests.yml

@@ -16,8 +16,8 @@ jobs:
 
     strategy:
       matrix:
-        os: ["ubuntu-latest", "macos-latest", "windows-latest"]
-        python-version: ["3.9", "3.10", "3.11"]
+        os: ["ubuntu-latest", "macos-latest"]
+        python-version: ["3.10", "3.11", "3.12"]
 
     # Run all shells using bash (including Windows)
     defaults:

.pre-commit-config.yaml

@@ -17,7 +17,7 @@ repos:
           hooks:
                 - id: codespell
                   args: [
-                    '--ignore-words-list', 'nd,alos,inout',
+                    '--ignore-words-list', 'nd,alos,inout,theses',
                     '--ignore-regex', '\bhist\b',
                     '--'
                   ]

CONTRIBUTING.md

@@ -20,11 +20,11 @@ The technical steps to contributing to xDEM are:
 
 ## Development environment
 
-xDEM currently supports Python versions of 3.9 to 3.11 (see `dev-environment.yml` for detailed dependencies), which are
+xDEM currently supports Python versions of 3.10 to 3.12 (see `dev-environment.yml` for detailed dependencies), which are
 tested in a continuous integration (CI) workflow running on GitHub Actions.
 
 When you open a PR on xDEM, a single linting action and 9 test actions will automatically start, corresponding to all
-supported Python versions (3.9, 3.10 and 3.11) and OS (Ubuntu, Mac, Windows). The coverage change of the tests will also
+supported Python versions (3.10, 3.11 and 3.12) and OS (Ubuntu, Mac, Windows). The coverage change of the tests will also
 be reported by CoverAlls.
 
 ### Setup
@@ -50,6 +50,8 @@ pytest
 
 Running `pytest` will trigger a script that automatically downloads test data from [https://github.com/GlacioHack/xdem-data](https://github.com/GlacioHack/xdem-data) used to run all tests.
 
+RichDEM should only be used for testing purposes within the xDEM project. The functionality of xDEM must not depend on RichDEM.
+
 ### Documentation
 
 If your changes need to be reflected in the documentation, update the related pages located in `doc/source/`. The documentation is written in MyST markdown syntax, similar to GitHub's default Markdown (see [MyST-NB](https://myst-nb.readthedocs.io/en/latest/authoring/text-notebooks.html) for details).

README.md

@@ -18,9 +18,9 @@
 
 It aims at **providing modular and robust tools for the most common analyses needed with DEMs**, including both geospatial
 operations specific to DEMs and a wide range of 3D alignment and correction methods from published, peer-reviewed studies.
-The core manipulation of DEMs (e.g., vertical alignment, terrain analysis) are **conveniently centered around `DEM` and `dDEM` classes** (that, notably, re-implements all tools
+The core manipulation of DEMs (e.g., vertical alignment, terrain analysis) are **conveniently centered around a `DEM` class** (that, notably, re-implements all tools
 of [gdalDEM](https://gdal.org/programs/gdaldem.html)). More complex pipelines (e.g., 3D rigid coregistration, bias corrections, filtering) are **built around
-modular `Coreg`, `BiasCorr` and `Filter` classes that easily interface between themselves**. Finally, xDEM includes advanced
+modular `Coreg`, `BiasCorr` classes that easily interface between themselves**. Finally, xDEM includes advanced
 uncertainty analysis tools based on spatial statistics of [SciKit-GStat](https://scikit-gstat.readthedocs.io/en/latest/).
 
 Additionally, xDEM inherits many convenient functionalities from [GeoUtils](https://github.com/GlacioHack/geoutils) such as
@@ -49,7 +49,7 @@ See [mamba's documentation](https://mamba.readthedocs.io/en/latest/) to install
 
 When using a method implemented in xDEM, please **cite both the package and the related study**:
 
-Citing xDEM: [![Zenodo](https://zenodo.org/badge/doi/10.5281/zenodo.4809697.svg)](https://zenodo.org/record/4809698)
+Citing xDEM: [![Zenodo](https://zenodo.org/badge/doi/10.5281/zenodo.4809697.svg)](https://zenodo.org/doi/10.5281/zenodo.4809697)
 
 Citing the related study:
 

dev-environment.yml

@@ -2,7 +2,7 @@ name: xdem-dev
 channels:
   - conda-forge
 dependencies:
-  - python>=3.9,<3.12
+  - python>=3.10,<3.13
   - geopandas>=0.12.0
   - numba=0.*
   - numpy=1.*
@@ -13,14 +13,13 @@ dependencies:
   - tqdm
   - scikit-image=0.*
   - scikit-gstat>=1.0.18,<1.1
-  - geoutils=0.1.9
+  - geoutils=0.1.10
 
   # Development-specific, to mirror manually in setup.cfg [options.extras_require].
   - pip
 
   # Optional dependencies
   - pytransform3d
-  - richdem
 
   # Test dependencies
   - gdal  # To test against GDAL
@@ -29,6 +28,7 @@ dependencies:
   - pyyaml
   - flake8
   - pylint
+  - richdem
 
   # Doc dependencies
   - sphinx

doc/source/_static/css/custom.css

@@ -0,0 +1,8 @@
+/*  Work around to wrong dark-mode for toggle button: https://github.com/executablebooks/MyST-NB/issues/523 */
+div.cell details.hide > summary {
+    background-color: var(--pst-color-surface);
+}
+
+div.cell details[open].above-input div.cell_input {
+    border-top: None;
+}

doc/source/about_xdem.md

@@ -2,72 +2,32 @@
 
 # About xDEM
 
+## What is xDEM?
 
-xDEM is a [Python](https://www.python.org/) package for the analysis of DEMs, with name standing for _cross-DEM analysis_[^sn1]
-and echoing its dependency on [xarray](https://docs.xarray.dev/en/stable/). It is designed for all Earth and planetary
-observation science, although our group currently has a strong focus on glaciological applications.
+xDEM is a Python package for the analysis of elevation data, and in particular that of digital elevation models (DEMs),
+with name standing for _cross-DEM analysis_[^sn1] and echoing its dependency on [Xarray](https://docs.xarray.dev/en/stable/).
 
-[^sn1]: The core features of xDEM rely on cross-analysis of surface elevation, for example for DEM alignment or error analysis.
+[^sn1]: Several core features of xDEM, in particular coregistration and uncertainty analysis, rely specifically on cross-analysis of elevation data over static surfaces.
 
+## Why use xDEM?
 
-```{epigraph}
-The core mission of xDEM is to be **easy-of-use**, **modular**, **robust**, **reproducible** and **fully open**.
+xDEM implements a wide range of high-level operations required for analyzing elevation data in a consistent framework
+tested to ensure the accuracy of these operations.
 
-Additionally, xDEM aims to be **efficient**, **scalable** and **state-of-the-art**.
-```
+It has three main focus points:
 
-```{important}
-:class: margin
-xDEM is in early stages of development and its features might evolve rapidly. Note the version you are working on for
-**reproducibility**!
-We are working on making features fully consistent for the first long-term release ``v0.1`` (planned early 2024).
-```
+1. Having an **easy and intuitive interface** based on the principle of least knowledge,
+2. Providing **statistically robust methods** for reliable quantitative analysis,
+3. Allowing **modular user input** to adapt to most applications.
 
-In details, those mean:
+Although modularity can sometimes hamper performance, we also aim to **preserve scalibility** as much as possible[^sn2].
 
-- **Ease-of-use:** all DEM basic operations or methods from published works should only require a few lines of code to be performed;
+[^sn2]: Out-of-memory, parallelizable computations relying on Dask are planned for late 2024!
 
-- **Modularity:** all DEM methods should be fully customizable, to allow both flexibility and inter-comparison;
+We particularly take to heart to verify the accuracy of our methods. For instance, our terrain attributes
+which have their own modular Python-based implementation, are tested to match exactly
+[gdaldem](https://gdal.org/programs/gdaldem.html) (slope, aspect, hillshade, roughness) and
+[RichDEM](https://richdem.readthedocs.io/en/latest/) (curvatures).
 
-- **Robustness:** all DEM methods should be tested within our continuous integration test-suite, to enforce that they always perform as expected;
-
-- **Reproducibility:** all code should be version-controlled and release-based, to ensure consistency of dependent
-  packages and works;
-
-- **Open-source:** all code should be accessible and re-usable to anyone in the community, for transparency and open governance.
-
-```{note}
-:class: margin
-Additional mission points, in particular **scalability**, are partly developed but not a priority until our first long-term release ``v0.1`` is reached. Those will be further developed specifically in a subsequent version ``v0.2``.
-```
-
-And, additionally:
-
-- **Efficiency**: all methods should be optimized at the lower-level, to function with the highest performance offered by Python packages;
-
-- **Scalability**: all methods should support both lazy processing and distributed parallelized processing, to work with high-resolution data on local machines as well as on HPCs;
-
-- **State-of-the-art**: all methods should be at the cutting edge of remote sensing science, to provide users with the most reliable and up-to-date tools.
-
-
-# The people behind xDEM
-
-```{margin}
-<sup>2</sup>More on our GlacioHack founder at [adehecq.github.io](https://adehecq.github.io/)!
-```
-
-xDEM was created during the [GlacioHack](https://github.com/GlacioHack) hackaton event, that was initiated by
-Amaury Dehecq<sup>2</sup> and took place online on November 8, 2020.
-
-```{margin}
-<sup>3</sup>Check-out [glaciology.ch](https://glaciology.ch) on our founding group of VAW glaciology!
-```
-
-The initial core development of xDEM was performed by members of the Glaciology group of the Laboratory of Hydraulics, Hydrology and
-Glaciology (VAW) at ETH Zürich<sup>3</sup>, with contributions by members of the University of Oslo, the University of Washington, and University
-Grenobles Alpes.
-
-We are not software developers but geoscientists, and we try our best to offer tools that can be useful to a larger group,
-documented, reliable and maintained. All development and maintenance is made on a voluntary basis and we welcome
-any new contributors. See some information on how to contribute in the dedicated page of our
-[GitHub repository](https://github.com/GlacioHack/xdem/blob/main/CONTRIBUTING.md).
+More details about the people behind xDEM and the package's objectives can be found on the **{ref}`background` reference
+page**.

doc/source/accuracy_precision.md

@@ -0,0 +1,116 @@
+(accuracy-precision)=
+
+# Grasping accuracy and precision
+
+Below, a small guide explaining what accuracy and precision are, and their relation to elevation data (or any spatial data!).
+
+## Why do we need to understand accuracy and precision?
+
+Elevation data comes from a wide range of instruments (optical, radar, lidar) acquiring in different conditions (ground,
+airborne, spaceborne) and relying on specific post-processing techniques (e.g., photogrammetry, interferometry).
+
+While some complexities are specific to certain instruments and methods, all elevation data generally possesses:
+
+- a [ground sampling distance](https://en.wikipedia.org/wiki/Ground_sample_distance), or footprint, **that does not necessarily represent the underlying spatial resolution of the observations**,
+- a [georeferencing](https://en.wikipedia.org/wiki/Georeferencing) **that can be subject to shifts, tilts or other deformations** due to inherent instrument errors, noise, or associated processing schemes,
+- a large number of [outliers](https://en.wikipedia.org/wiki/Outlier) **that remain difficult to filter** as they can originate from various sources (e.g., blunders, clouds).
+
+All of these factors lead to difficulties in assessing the reliability of elevation data, required to
+perform additional quantitative analysis, which calls for defining the concepts relating to accuracy and precision for elevation data.
+
+## Accuracy and precision of elevation data
+
+### What are accuracy and precision?
+
+[Accuracy and precision](https://en.wikipedia.org/wiki/Accuracy_and_precision) describe systematic and random errors, respectively.
+A more accurate measurement is on average closer to the true value (less systematic error), while a more precise measurement has
+less spread in measurement error (less random error), as shown in the simple schematic below.
+
+*Note: sometimes "accuracy" is also used to describe both types of errors, while "trueness" refers to systematic errors, as defined
+in* [ISO 5725-1](https://www.iso.org/obp/ui/#iso:std:iso:5725:-1:ed-1:v1:en) *. Here, we use accuracy for systematic
+errors as, to our knowledge, it is a more commonly used terminology for remote sensing applications.*
+
+:::{figure} imgs/precision_accuracy.png
+:width: 80%
+
+Source: [antarcticglaciers.org](http://www.antarcticglaciers.org/glacial-geology/dating-glacial-sediments2/precision-and-accuracy-glacial-geology/), accessed 29.06.21.
+:::
+
+### Translating these concepts for elevation data
+
+However, elevation data rarely consists of a single independent measurement but of a **series of measurements** (image grid,
+ground track) **related to a spatial support** (horizontal georeferencing, independent of height), which complexifies
+the notion of accuracy and precision.
+
+Due to this, spatially consistent systematic errors can arise in elevation data independently of the error in elevation itself,
+such as **affine biases** (systematic georeferencing shifts), in addition to **specific biases** known to exist locally
+(e.g., signal penetration in land cover type).
+
+For random errors, a variability in error magnitude or **heteroscedasticity** is common in elevation data (e.g.,
+large errors on steep slopes). Finally, spatially structured yet random patterns of errors (e.g., along-track undulations)
+also exist and force us to consider the **spatial correlation of random errors (sometimes called structured errors)**.
+
+Translating the accuracy and precision concepts to elevation data, we can thus define:
+
+- **Elevation accuracy** (systematic error) describes how close an elevation data is to the true elevation on the Earth's surface, both for errors **common to the entire spatial support**
+(DEM grid, altimetric track) and errors **specific to a location** (pixel, footprint),
+- **Elevation precision** (random error) describes the random spread of elevation error in measurement, independently of a possible bias from the true positioning, both for errors **structured over the spatial support** and **specific to a location**.
+
+These categories are depicted in the figure below.
+
+:::{figure} imgs/accuracy_precision_dem.png
+:width: 100%
+
+Source: [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922).
+:::
+
+### Absolute or relative elevation accuracy
+
+Accuracy is generally considered from two focus points:
+
+- **Absolute elevation accuracy** describes systematic errors to the true positioning, usually important when analysis focuses on the exact location of topographic features at a specific epoch.
+- **Relative elevation accuracy** describes systematic errors with reference to other elevation data that does not necessarily match the true positioning, important for analyses interested in topographic change over time.
+
+## How to get the best out of your elevation data?
+
+### Quantifying and improving accuracy
+
+Misalignments due to poor absolute or relative accuracy are common in elevation datasets, and are usually assessed and
+corrected by performing **three-dimensional elevation coregistration and bias corrections to an independent source
+of elevation data**.
+
+In the case of absolute accuracy, this independent source must be precise and accurate, such as altimetric data from
+[ICESat](https://icesat.gsfc.nasa.gov/icesat/) and [ICESat-2](https://icesat-2.gsfc.nasa.gov/) elevations, or coarser yet
+quality-controlled DEMs themselves aligned on altimetric data such as the
+[Copernicus DEM](https://portal.opentopography.org/raster?opentopoID=OTSDEM.032021.4326.3).
+
+To use coregistration and bias correction pipelines in xDEM, see the **feature pages on {ref}`coregistration` and {ref}`biascorr`**.
+
+### Quantifying and improving precision
+
+While assessing accuracy is fairly straightforward as it consists of computing the mean differences (biases) between
+two or multiple datasets, assessing precision of elevation data can be much more complex. The spread in measurement
+errors cannot be quantified by a difference at single data point, and require statistical inference.
+
+Assessing precision usually means applying **spatial statistics combined to uncertainty quantification**,
+to account for the spatial variability and the spatial correlation in errors. For this it is usually necessary, as
+for coregistration, to **rely on an independent source of elevation data on static surfaces similarly**.
+
+To use spatial statistics for quantifying precision in xDEM, see **the feature page on {ref}`uncertainty`**.
+
+Additionally, improving the precision of elevation data is sometimes possible by correcting random structured
+errors, such as pseudo-periodic errors (undulations). For this, one can **also use {ref}`biascorr`**.
+
+----------------
+
+:::{admonition} References and more reading
+:class: tip
+
+More background on structured random errors is available on the **{ref}`spatial-stats` guide page**.
+
+**References:**
+
+- [ISO 5725-1 (1994)](https://www.iso.org/obp/ui/#iso:std:iso:5725:-1:ed-1:v1:en), Accuracy (trueness and precision) of measurement methods and results — Part 1: General principles and definitions,
+- [Mittaz et al. (2019)](http://dx.doi.org/10.1088/1681-7575/ab1705), Applying principles of metrology to historical Earth observations from satellites,
+- [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922), Uncertainty analysis of digital elevation models by spatial inference from stable terrain.
+:::