diff --git a/.github/workflows/pip-checks.yml b/.github/workflows/pip-checks.yml
index 3b8a8977..996d02e7 100644
--- a/.github/workflows/pip-checks.yml
+++ b/.github/workflows/pip-checks.yml
@@ -16,7 +16,7 @@ jobs:
strategy:
matrix:
os: ["ubuntu-latest", "macos-latest", "windows-latest"]
- python-version: ["3.9", "3.10", "3.11"]
+ python-version: ["3.10", "3.11", "3.12"]
# Run all shells using bash (including Windows)
defaults:
diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml
index 68ae93ef..c0ba22e8 100644
--- a/.github/workflows/python-publish.yml
+++ b/.github/workflows/python-publish.yml
@@ -26,7 +26,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
- python-version: '3.11'
+ python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
diff --git a/.github/workflows/python-tests.yml b/.github/workflows/python-tests.yml
index 012b281a..8940c235 100644
--- a/.github/workflows/python-tests.yml
+++ b/.github/workflows/python-tests.yml
@@ -16,8 +16,8 @@ jobs:
strategy:
matrix:
- os: ["ubuntu-latest", "macos-latest", "windows-latest"]
- python-version: ["3.9", "3.10", "3.11"]
+ os: ["ubuntu-latest", "macos-latest"]
+ python-version: ["3.10", "3.11", "3.12"]
# Run all shells using bash (including Windows)
defaults:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index b716f6a8..e6336c9f 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -17,7 +17,7 @@ repos:
hooks:
- id: codespell
args: [
- '--ignore-words-list', 'nd,alos,inout',
+ '--ignore-words-list', 'nd,alos,inout,theses',
'--ignore-regex', '\bhist\b',
'--'
]
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index d4987aed..5c55f021 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -20,11 +20,11 @@ The technical steps to contributing to xDEM are:
## Development environment
-xDEM currently supports Python versions of 3.9 to 3.11 (see `dev-environment.yml` for detailed dependencies), which are
+xDEM currently supports Python versions of 3.10 to 3.12 (see `dev-environment.yml` for detailed dependencies), which are
tested in a continuous integration (CI) workflow running on GitHub Actions.
When you open a PR on xDEM, a single linting action and 9 test actions will automatically start, corresponding to all
-supported Python versions (3.9, 3.10 and 3.11) and OS (Ubuntu, Mac, Windows). The coverage change of the tests will also
+supported Python versions (3.10, 3.11 and 3.12) and OS (Ubuntu, Mac, Windows). The coverage change of the tests will also
be reported by CoverAlls.
### Setup
@@ -50,6 +50,8 @@ pytest
Running `pytest` will trigger a script that automatically downloads test data from [https://github.com/GlacioHack/xdem-data](https://github.com/GlacioHack/xdem-data) used to run all tests.
+RichDEM should only be used for testing purposes within the xDEM project. The functionality of xDEM must not depend on RichDEM.
+
### Documentation
If your changes need to be reflected in the documentation, update the related pages located in `doc/source/`. The documentation is written in MyST markdown syntax, similar to GitHub's default Markdown (see [MyST-NB](https://myst-nb.readthedocs.io/en/latest/authoring/text-notebooks.html) for details).
diff --git a/README.md b/README.md
index 2c8ea9dc..c8c79866 100644
--- a/README.md
+++ b/README.md
@@ -18,9 +18,9 @@
It aims at **providing modular and robust tools for the most common analyses needed with DEMs**, including both geospatial
operations specific to DEMs and a wide range of 3D alignment and correction methods from published, peer-reviewed studies.
-The core manipulation of DEMs (e.g., vertical alignment, terrain analysis) are **conveniently centered around `DEM` and `dDEM` classes** (that, notably, re-implements all tools
+The core manipulation of DEMs (e.g., vertical alignment, terrain analysis) are **conveniently centered around a `DEM` class** (that, notably, re-implements all tools
of [gdalDEM](https://gdal.org/programs/gdaldem.html)). More complex pipelines (e.g., 3D rigid coregistration, bias corrections, filtering) are **built around
-modular `Coreg`, `BiasCorr` and `Filter` classes that easily interface between themselves**. Finally, xDEM includes advanced
+modular `Coreg`, `BiasCorr` classes that easily interface between themselves**. Finally, xDEM includes advanced
uncertainty analysis tools based on spatial statistics of [SciKit-GStat](https://scikit-gstat.readthedocs.io/en/latest/).
Additionally, xDEM inherits many convenient functionalities from [GeoUtils](https://github.com/GlacioHack/geoutils) such as
@@ -49,7 +49,7 @@ See [mamba's documentation](https://mamba.readthedocs.io/en/latest/) to install
When using a method implemented in xDEM, please **cite both the package and the related study**:
-Citing xDEM: [](https://zenodo.org/record/4809698)
+Citing xDEM: [](https://zenodo.org/doi/10.5281/zenodo.4809697)
Citing the related study:
diff --git a/dev-environment.yml b/dev-environment.yml
index 848ee9e3..fb682146 100644
--- a/dev-environment.yml
+++ b/dev-environment.yml
@@ -2,7 +2,7 @@ name: xdem-dev
channels:
- conda-forge
dependencies:
- - python>=3.9,<3.12
+ - python>=3.10,<3.13
- geopandas>=0.12.0
- numba=0.*
- numpy=1.*
@@ -13,14 +13,13 @@ dependencies:
- tqdm
- scikit-image=0.*
- scikit-gstat>=1.0.18,<1.1
- - geoutils=0.1.9
+ - geoutils=0.1.10
# Development-specific, to mirror manually in setup.cfg [options.extras_require].
- pip
# Optional dependencies
- pytransform3d
- - richdem
# Test dependencies
- gdal # To test against GDAL
@@ -29,6 +28,7 @@ dependencies:
- pyyaml
- flake8
- pylint
+ - richdem
# Doc dependencies
- sphinx
diff --git a/doc/source/_static/css/custom.css b/doc/source/_static/css/custom.css
new file mode 100644
index 00000000..43a5b3e2
--- /dev/null
+++ b/doc/source/_static/css/custom.css
@@ -0,0 +1,8 @@
+/* Work around to wrong dark-mode for toggle button: https://github.com/executablebooks/MyST-NB/issues/523 */
+div.cell details.hide > summary {
+ background-color: var(--pst-color-surface);
+}
+
+div.cell details[open].above-input div.cell_input {
+ border-top: None;
+}
diff --git a/doc/source/about_xdem.md b/doc/source/about_xdem.md
index c5cb1d8d..78b92163 100644
--- a/doc/source/about_xdem.md
+++ b/doc/source/about_xdem.md
@@ -2,72 +2,32 @@
# About xDEM
+## What is xDEM?
-xDEM is a [Python](https://www.python.org/) package for the analysis of DEMs, with name standing for _cross-DEM analysis_[^sn1]
-and echoing its dependency on [xarray](https://docs.xarray.dev/en/stable/). It is designed for all Earth and planetary
-observation science, although our group currently has a strong focus on glaciological applications.
+xDEM is a Python package for the analysis of elevation data, and in particular that of digital elevation models (DEMs),
+with name standing for _cross-DEM analysis_[^sn1] and echoing its dependency on [Xarray](https://docs.xarray.dev/en/stable/).
-[^sn1]: The core features of xDEM rely on cross-analysis of surface elevation, for example for DEM alignment or error analysis.
+[^sn1]: Several core features of xDEM, in particular coregistration and uncertainty analysis, rely specifically on cross-analysis of elevation data over static surfaces.
+## Why use xDEM?
-```{epigraph}
-The core mission of xDEM is to be **easy-of-use**, **modular**, **robust**, **reproducible** and **fully open**.
+xDEM implements a wide range of high-level operations required for analyzing elevation data in a consistent framework
+tested to ensure the accuracy of these operations.
-Additionally, xDEM aims to be **efficient**, **scalable** and **state-of-the-art**.
-```
+It has three main focus points:
-```{important}
-:class: margin
-xDEM is in early stages of development and its features might evolve rapidly. Note the version you are working on for
-**reproducibility**!
-We are working on making features fully consistent for the first long-term release ``v0.1`` (planned early 2024).
-```
+1. Having an **easy and intuitive interface** based on the principle of least knowledge,
+2. Providing **statistically robust methods** for reliable quantitative analysis,
+3. Allowing **modular user input** to adapt to most applications.
-In details, those mean:
+Although modularity can sometimes hamper performance, we also aim to **preserve scalibility** as much as possible[^sn2].
-- **Ease-of-use:** all DEM basic operations or methods from published works should only require a few lines of code to be performed;
+[^sn2]: Out-of-memory, parallelizable computations relying on Dask are planned for late 2024!
-- **Modularity:** all DEM methods should be fully customizable, to allow both flexibility and inter-comparison;
+We particularly take to heart to verify the accuracy of our methods. For instance, our terrain attributes
+which have their own modular Python-based implementation, are tested to match exactly
+[gdaldem](https://gdal.org/programs/gdaldem.html) (slope, aspect, hillshade, roughness) and
+[RichDEM](https://richdem.readthedocs.io/en/latest/) (curvatures).
-- **Robustness:** all DEM methods should be tested within our continuous integration test-suite, to enforce that they always perform as expected;
-
-- **Reproducibility:** all code should be version-controlled and release-based, to ensure consistency of dependent
- packages and works;
-
-- **Open-source:** all code should be accessible and re-usable to anyone in the community, for transparency and open governance.
-
-```{note}
-:class: margin
-Additional mission points, in particular **scalability**, are partly developed but not a priority until our first long-term release ``v0.1`` is reached. Those will be further developed specifically in a subsequent version ``v0.2``.
-```
-
-And, additionally:
-
-- **Efficiency**: all methods should be optimized at the lower-level, to function with the highest performance offered by Python packages;
-
-- **Scalability**: all methods should support both lazy processing and distributed parallelized processing, to work with high-resolution data on local machines as well as on HPCs;
-
-- **State-of-the-art**: all methods should be at the cutting edge of remote sensing science, to provide users with the most reliable and up-to-date tools.
-
-
-# The people behind xDEM
-
-```{margin}
-2More on our GlacioHack founder at [adehecq.github.io](https://adehecq.github.io/)!
-```
-
-xDEM was created during the [GlacioHack](https://github.com/GlacioHack) hackaton event, that was initiated by
-Amaury Dehecq2 and took place online on November 8, 2020.
-
-```{margin}
-3Check-out [glaciology.ch](https://glaciology.ch) on our founding group of VAW glaciology!
-```
-
-The initial core development of xDEM was performed by members of the Glaciology group of the Laboratory of Hydraulics, Hydrology and
-Glaciology (VAW) at ETH Zürich3, with contributions by members of the University of Oslo, the University of Washington, and University
-Grenobles Alpes.
-
-We are not software developers but geoscientists, and we try our best to offer tools that can be useful to a larger group,
-documented, reliable and maintained. All development and maintenance is made on a voluntary basis and we welcome
-any new contributors. See some information on how to contribute in the dedicated page of our
-[GitHub repository](https://github.com/GlacioHack/xdem/blob/main/CONTRIBUTING.md).
+More details about the people behind xDEM and the package's objectives can be found on the **{ref}`background` reference
+page**.
diff --git a/doc/source/accuracy_precision.md b/doc/source/accuracy_precision.md
new file mode 100644
index 00000000..fbd41763
--- /dev/null
+++ b/doc/source/accuracy_precision.md
@@ -0,0 +1,116 @@
+(accuracy-precision)=
+
+# Grasping accuracy and precision
+
+Below, a small guide explaining what accuracy and precision are, and their relation to elevation data (or any spatial data!).
+
+## Why do we need to understand accuracy and precision?
+
+Elevation data comes from a wide range of instruments (optical, radar, lidar) acquiring in different conditions (ground,
+airborne, spaceborne) and relying on specific post-processing techniques (e.g., photogrammetry, interferometry).
+
+While some complexities are specific to certain instruments and methods, all elevation data generally possesses:
+
+- a [ground sampling distance](https://en.wikipedia.org/wiki/Ground_sample_distance), or footprint, **that does not necessarily represent the underlying spatial resolution of the observations**,
+- a [georeferencing](https://en.wikipedia.org/wiki/Georeferencing) **that can be subject to shifts, tilts or other deformations** due to inherent instrument errors, noise, or associated processing schemes,
+- a large number of [outliers](https://en.wikipedia.org/wiki/Outlier) **that remain difficult to filter** as they can originate from various sources (e.g., blunders, clouds).
+
+All of these factors lead to difficulties in assessing the reliability of elevation data, required to
+perform additional quantitative analysis, which calls for defining the concepts relating to accuracy and precision for elevation data.
+
+## Accuracy and precision of elevation data
+
+### What are accuracy and precision?
+
+[Accuracy and precision](https://en.wikipedia.org/wiki/Accuracy_and_precision) describe systematic and random errors, respectively.
+A more accurate measurement is on average closer to the true value (less systematic error), while a more precise measurement has
+less spread in measurement error (less random error), as shown in the simple schematic below.
+
+*Note: sometimes "accuracy" is also used to describe both types of errors, while "trueness" refers to systematic errors, as defined
+in* [ISO 5725-1](https://www.iso.org/obp/ui/#iso:std:iso:5725:-1:ed-1:v1:en) *. Here, we use accuracy for systematic
+errors as, to our knowledge, it is a more commonly used terminology for remote sensing applications.*
+
+:::{figure} imgs/precision_accuracy.png
+:width: 80%
+
+Source: [antarcticglaciers.org](http://www.antarcticglaciers.org/glacial-geology/dating-glacial-sediments2/precision-and-accuracy-glacial-geology/), accessed 29.06.21.
+:::
+
+### Translating these concepts for elevation data
+
+However, elevation data rarely consists of a single independent measurement but of a **series of measurements** (image grid,
+ground track) **related to a spatial support** (horizontal georeferencing, independent of height), which complexifies
+the notion of accuracy and precision.
+
+Due to this, spatially consistent systematic errors can arise in elevation data independently of the error in elevation itself,
+such as **affine biases** (systematic georeferencing shifts), in addition to **specific biases** known to exist locally
+(e.g., signal penetration in land cover type).
+
+For random errors, a variability in error magnitude or **heteroscedasticity** is common in elevation data (e.g.,
+large errors on steep slopes). Finally, spatially structured yet random patterns of errors (e.g., along-track undulations)
+also exist and force us to consider the **spatial correlation of random errors (sometimes called structured errors)**.
+
+Translating the accuracy and precision concepts to elevation data, we can thus define:
+
+- **Elevation accuracy** (systematic error) describes how close an elevation data is to the true elevation on the Earth's surface, both for errors **common to the entire spatial support**
+(DEM grid, altimetric track) and errors **specific to a location** (pixel, footprint),
+- **Elevation precision** (random error) describes the random spread of elevation error in measurement, independently of a possible bias from the true positioning, both for errors **structured over the spatial support** and **specific to a location**.
+
+These categories are depicted in the figure below.
+
+:::{figure} imgs/accuracy_precision_dem.png
+:width: 100%
+
+Source: [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922).
+:::
+
+### Absolute or relative elevation accuracy
+
+Accuracy is generally considered from two focus points:
+
+- **Absolute elevation accuracy** describes systematic errors to the true positioning, usually important when analysis focuses on the exact location of topographic features at a specific epoch.
+- **Relative elevation accuracy** describes systematic errors with reference to other elevation data that does not necessarily match the true positioning, important for analyses interested in topographic change over time.
+
+## How to get the best out of your elevation data?
+
+### Quantifying and improving accuracy
+
+Misalignments due to poor absolute or relative accuracy are common in elevation datasets, and are usually assessed and
+corrected by performing **three-dimensional elevation coregistration and bias corrections to an independent source
+of elevation data**.
+
+In the case of absolute accuracy, this independent source must be precise and accurate, such as altimetric data from
+[ICESat](https://icesat.gsfc.nasa.gov/icesat/) and [ICESat-2](https://icesat-2.gsfc.nasa.gov/) elevations, or coarser yet
+quality-controlled DEMs themselves aligned on altimetric data such as the
+[Copernicus DEM](https://portal.opentopography.org/raster?opentopoID=OTSDEM.032021.4326.3).
+
+To use coregistration and bias correction pipelines in xDEM, see the **feature pages on {ref}`coregistration` and {ref}`biascorr`**.
+
+### Quantifying and improving precision
+
+While assessing accuracy is fairly straightforward as it consists of computing the mean differences (biases) between
+two or multiple datasets, assessing precision of elevation data can be much more complex. The spread in measurement
+errors cannot be quantified by a difference at single data point, and require statistical inference.
+
+Assessing precision usually means applying **spatial statistics combined to uncertainty quantification**,
+to account for the spatial variability and the spatial correlation in errors. For this it is usually necessary, as
+for coregistration, to **rely on an independent source of elevation data on static surfaces similarly**.
+
+To use spatial statistics for quantifying precision in xDEM, see **the feature page on {ref}`uncertainty`**.
+
+Additionally, improving the precision of elevation data is sometimes possible by correcting random structured
+errors, such as pseudo-periodic errors (undulations). For this, one can **also use {ref}`biascorr`**.
+
+----------------
+
+:::{admonition} References and more reading
+:class: tip
+
+More background on structured random errors is available on the **{ref}`spatial-stats` guide page**.
+
+**References:**
+
+- [ISO 5725-1 (1994)](https://www.iso.org/obp/ui/#iso:std:iso:5725:-1:ed-1:v1:en), Accuracy (trueness and precision) of measurement methods and results — Part 1: General principles and definitions,
+- [Mittaz et al. (2019)](http://dx.doi.org/10.1088/1681-7575/ab1705), Applying principles of metrology to historical Earth observations from satellites,
+- [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922), Uncertainty analysis of digital elevation models by spatial inference from stable terrain.
+:::
diff --git a/doc/source/api.md b/doc/source/api.md
index 6f7465cb..e46532cc 100644
--- a/doc/source/api.md
+++ b/doc/source/api.md
@@ -20,28 +20,27 @@ documentation.
```{important}
A {class}`~xdem.DEM` inherits all raster methods and attributes from the {class}`~geoutils.Raster` object of GeoUtils.
-Below, we only repeat the core attributes and methods of GeoUtils, see
+Below, we only repeat some core attributes and methods of GeoUtils, see
[the Raster API in GeoUtils](https://geoutils.readthedocs.io/en/latest/api.html#raster) for the full list.
```
-### Opening or saving a DEM
+### Opening or saving
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
DEM
- DEM.info
DEM.save
```
-### Plotting a DEM
+### Plotting or summarize info
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- DEM
+ DEM.info
DEM.plot
```
@@ -68,6 +67,7 @@ Below, we only repeat the core attributes and methods of GeoUtils, see
DEM.crs
DEM.transform
DEM.nodata
+ DEM.area_or_point
```
#### Specific to {class}`~xdem.DEM`
@@ -79,6 +79,23 @@ Below, we only repeat the core attributes and methods of GeoUtils, see
DEM.vcrs
```
+### Other attributes
+
+#### Inherited from {class}`~geoutils.Raster`
+
+See the full list in [the Raster API of GeoUtils](https://geoutils.readthedocs.io/en/latest/api.html#raster).
+
+```{eval-rst}
+.. autosummary::
+ :toctree: gen_modules/
+
+ DEM.res
+ DEM.bounds
+ DEM.width
+ DEM.height
+ DEM.shape
+```
+
### Georeferencing
#### Inherited from {class}`~geoutils.Raster`
@@ -87,6 +104,9 @@ Below, we only repeat the core attributes and methods of GeoUtils, see
.. autosummary::
:toctree: gen_modules/
+ DEM.set_nodata
+ DEM.set_area_or_point
+ DEM.info
DEM.reproject
DEM.crop
```
@@ -113,19 +133,8 @@ See the full list of vector methods in [GeoUtils' documentation](https://geoutil
DEM.polygonize
DEM.proximity
-```
-
-### Coregistration
-
-```{tip}
-To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d`, see the API of {ref}`api-geo-handle`.
-```
-
-```{eval-rst}
-.. autosummary::
- :toctree: gen_modules/
-
- DEM.coregister_3d
+ DEM.to_pointcloud
+ DEM.interp_points
```
### Terrain attributes
@@ -148,24 +157,35 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
DEM.fractal_roughness
```
-## dDEM
+Or to get multiple related terrain attributes at once (for performance):
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- dDEM
+ DEM.get_terrain_attribute
```
-## DEMCollection
+### Coregistration and bias corrections
-## dDEM
+```{tip}
+To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d`, see the API of {ref}`api-geo-handle`.
+```
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- DEMCollection
+ DEM.coregister_3d
+```
+
+### Uncertainty analysis
+
+```{eval-rst}
+.. autosummary::
+ :toctree: gen_modules/
+
+ DEM.estimate_uncertainty
```
(api-geo-handle)=
@@ -175,8 +195,8 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
**Overview of co-registration class structure**:
```{eval-rst}
-.. inheritance-diagram:: xdem.coreg.base xdem.coreg.affine xdem.coreg.biascorr
- :top-classes: xdem.Coreg
+.. inheritance-diagram:: xdem.coreg.base.Coreg xdem.coreg.affine xdem.coreg.biascorr
+ :top-classes: xdem.coreg.Coreg
```
### Coregistration, pipeline and blockwise
@@ -185,9 +205,9 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
.. autosummary::
:toctree: gen_modules/
- xdem.coreg.Coreg
- xdem.coreg.CoregPipeline
- xdem.coreg.BlockwiseCoreg
+ coreg.Coreg
+ coreg.CoregPipeline
+ coreg.BlockwiseCoreg
```
#### Fitting and applying transforms
@@ -196,105 +216,185 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
.. autosummary::
:toctree: gen_modules/
- xdem.coreg.Coreg.fit
- xdem.coreg.Coreg.apply
+ coreg.Coreg.fit_and_apply
+ coreg.Coreg.fit
+ coreg.Coreg.apply
```
-#### Other functionalities
+#### Extracting metadata
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.coreg.Coreg.residuals
+ coreg.Coreg.info
+ coreg.Coreg.meta
```
-### Affine coregistration methods
+### Affine coregistration
+#### Parent object (to define custom methods)
-**Generic parent class:**
+```{eval-rst}
+.. autosummary::
+ :toctree: gen_modules/
+
+ coreg.AffineCoreg
+```
+
+#### Coregistration methods
+
+```{eval-rst}
+.. autosummary::
+ :toctree: gen_modules/
+
+ coreg.VerticalShift
+ coreg.NuthKaab
+ coreg.DhMinimize
+ coreg.ICP
+```
+
+#### Manipulating affine transforms
+
+```{eval-rst}
+.. autosummary::
+ :toctree: gen_modules/
+
+ coreg.AffineCoreg.from_matrix
+ coreg.AffineCoreg.to_matrix
+ coreg.AffineCoreg.from_translations
+ coreg.AffineCoreg.to_translations
+ coreg.AffineCoreg.from_rotations
+ coreg.AffineCoreg.to_rotations
+
+ coreg.apply_matrix
+ coreg.invert_matrix
+```
+
+### Bias-correction
+
+#### Parent object (to define custom methods)
+
+```{eval-rst}
+.. autosummary::
+ :toctree: gen_modules/
+
+ coreg.BiasCorr
+```
+
+#### Bias-correction methods
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.coreg.AffineCoreg
+ coreg.Deramp
+ coreg.DirectionalBias
+ coreg.TerrainBias
```
-**Convenience classes for specific coregistrations:**
+## Uncertainty analysis
+
+```{important}
+Several uncertainty functionalities of xDEM are being implemented directly in SciKit-GStat for spatial statistics
+(e.g., fitting a sum of variogram models, pairwise subsampling for grid data). This will allow to simplify several
+function inputs and outputs, by relying on a single {func}`~skgstat.Variogram` object.
+
+This will trigger API changes in future package versions.
+```
+
+### Core routines for heteroscedasticity, spatial correlations, error propagation
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.coreg.VerticalShift
- xdem.coreg.NuthKaab
- xdem.coreg.ICP
+ spatialstats.infer_heteroscedasticity_from_stable
+ spatialstats.infer_spatial_correlation_from_stable
+ spatialstats.spatial_error_propagation
```
-### Bias-correction (including non-affine coregistration) methods
+### Sub-routines for heteroscedasticity
-**Generic parent class:**
+```{eval-rst}
+.. autosummary::
+ :toctree: gen_modules/
+
+ spatialstats.nd_binning
+ spatialstats.interp_nd_binning
+ spatialstats.two_step_standardization
+```
+
+### Sub-routines for spatial correlations
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.coreg.BiasCorr
+ spatialstats.sample_empirical_variogram
+ spatialstats.fit_sum_model_variogram
+ spatialstats.correlation_from_variogram
```
-**Convenience classes for specific corrections:**
+### Sub-routines for error propagation
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.coreg.Deramp
- xdem.coreg.DirectionalBias
- xdem.coreg.TerrainBias
+ spatialstats.number_effective_samples
```
-## Terrain attributes
+### Empirical validation
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.terrain
+ spatialstats.patches_method
```
-## Volume integration methods
+### Plotting for uncertainty analysis
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.volume
+ spatialstats.plot_variogram
+ spatialstats.plot_1d_binning
+ spatialstats.plot_2d_binning
```
-## Fitting methods
+## Stand-alone functions (moved)
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.fit
+ spatialstats.nmad
+```
+
+
+## Development classes (removal or re-factoring)
+
+```{caution}
+The {class}`xdem.dDEM` and {class}`xdem.DEMCollection` classes will be removed or re-factored in the near future.
```
-## Filtering methods
+### dDEM
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.filters
+ dDEM
```
-## Spatial statistics methods
+### DEMCollection
```{eval-rst}
.. autosummary::
:toctree: gen_modules/
- xdem.spatialstats
+ DEMCollection
```
diff --git a/doc/source/background.md b/doc/source/background.md
new file mode 100644
index 00000000..ba7e3081
--- /dev/null
+++ b/doc/source/background.md
@@ -0,0 +1,65 @@
+(background)=
+
+# Background
+
+Below, some more information on the people behind the package, and its mission.
+
+## The people behind xDEM
+
+```{margin}
+2More on our GlacioHack founder at [adehecq.github.io](https://adehecq.github.io/)!
+```
+
+xDEM was created during the **[GlacioHack](https://github.com/GlacioHack) hackathon**, that was initiated by
+Amaury Dehecq2 and took place online on November 8, 2020.
+
+```{margin}
+3Check-out [glaciology.ch](https://glaciology.ch) on our founding group of VAW glaciology!
+```
+
+The initial core development of xDEM was performed by members of the Glaciology group of the Laboratory of Hydraulics, Hydrology and
+Glaciology (VAW) at ETH Zürich3, with contributions by members of the University of Oslo, the University of Washington, and University
+Grenoble Alpes.
+
+We are not software developers but geoscientists, and we try our best to offer tools that can be useful to a larger group,
+documented, reliable and maintained. All development and maintenance is made on a voluntary basis and we welcome
+any new contributors! See some information on how to contribute in the dedicated page of our
+[GitHub repository](https://github.com/GlacioHack/xdem/blob/main/CONTRIBUTING.md).
+
+## Mission
+
+```{epigraph}
+The core mission of xDEM is to be **easy-of-use**, **modular** and **robust**.
+
+It also attempts to be as **efficient**, **scalable** and **state-of-the-art** as possible.
+
+Finally, as an open source package, it aspires to foster **reproducibility** and **open science**.
+```
+
+In details, those mean:
+
+- **Ease-of-use:** all basic operations or methods from published works should only require a few lines of code to be performed;
+
+- **Modularity:** all methods should be fully customizable, to allow both flexibility and inter-comparison;
+
+- **Robustness:** all methods should be tested within our continuous integration test-suite, to enforce that they always perform as expected;
+
+```{note}
+:class: margin
+**Scalability** is currently being improved towards a first major release ``v1.0``.
+```
+
+And, additionally:
+
+- **Efficiency**: all methods should be optimized at the lower-level, to function with the highest performance offered by Python packages;
+
+- **Scalability**: all methods should support both lazy processing and distributed parallelized processing, to work with high-resolution data on local machines as well as on HPCs;
+
+- **State-of-the-art**: all methods should be at the cutting edge of remote sensing science, to provide users with the most reliable and up-to-date tools.
+
+And finally:
+
+- **Reproducibility:** all code should be version-controlled and release-based, to ensure consistency of dependent
+ packages and works;
+
+- **Open-source:** all code should be accessible and re-usable to anyone in the community, for transparency and open governance.
diff --git a/doc/source/biascorr.md b/doc/source/biascorr.md
index 44e3ac3a..d2c6380b 100644
--- a/doc/source/biascorr.md
+++ b/doc/source/biascorr.md
@@ -1,5 +1,7 @@
---
file_format: mystnb
+mystnb:
+ execution_timeout: 90
jupytext:
formats: md:myst
text_representation:
@@ -15,146 +17,375 @@ kernelspec:
# Bias correction
-In xDEM, bias-correction methods correspond to non-rigid transformations that cannot be described as a 3-dimensional
-affine function (see {ref}`coregistration`).
+In xDEM, bias-correction methods correspond to **transformations that cannot be described as a 3-dimensional
+affine function** (see {ref}`coregistration`), and aim at correcting both systematic elevation errors and
+spatially-structured random errors.
-Contrary to rigid coregistration methods, bias corrections are not limited to the information in the DEMs. They can be
-passed any external variables (e.g., land cover type, processing metric) to attempt to identify and correct biases in
-the DEM. Still, many methods rely either on coordinates (e.g., deramping, along-track corrections) or terrain
-(e.g., curvature- or elevation-dependant corrections), derived solely from the DEM.
+Contrary to affine coregistration methods, bias corrections are **not limited to the information in the elevation data**. They can be
+passed any external variables (e.g., land cover type, processing metric) to attempt to identify and correct biases.
+Still, many methods rely either on coordinates (e.g., deramping, along-track corrections) or terrain
+(e.g., curvature- or elevation-dependant corrections), derived solely from the elevation data.
-## The {class}`~xdem.BiasCorr` object
+## Quick use
-Each bias-correction method in xDEM inherits their interface from the {class}`~xdem.Coreg` class (see {ref}`coreg_object`).
-This implies that bias-correction methods can be combined in a {class}`~xdem.CoregPipeline` with any other methods, or
-applied in a block-wise manner through {class}`~xdem.BlockwiseCoreg`.
+Bias-correction methods are **used the same way as coregistrations**:
+
+```{code-cell} ipython3
+:tags: [remove-cell]
+
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 600
+pyplot.rcParams['savefig.dpi'] = 600
+```
+
+```{code-cell} ipython3
+import xdem
+import numpy as np
+
+# Create a bias-correction
+biascorr = xdem.coreg.DirectionalBias(angle=45, fit_or_bin="bin", bin_sizes=200, bin_apply_method="per_bin", bin_statistic=np.mean)
+```
+
+Bias correction can estimate and correct the bias **by a parametric fit** using `fit_or_bin="fit"` linked to `fit_` parameters, **by applying
+a binned statistic** using `fit_or_bin="bin"` linked to `bin_` parameters, or **by a parametric fit on the binned data** using `fit_or_bin="bin_and_fit"`
+linked to all parameters.
+
+Predefined bias corrections usually take additional arguments such as `angle` for {class}`~xdem.coreg.DirectionalBias`,
+`poly_order` for {class}`~xdem.coreg.Deramp` and `attribute` for {class}`~xdem.coreg.TerrainBias`.
+
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for opening example files"
+: code_prompt_hide: "Hide the code for opening example files"
+
+import geoutils as gu
+import numpy as np
+import matplotlib.pyplot as plt
+
+# Open a reference and to-be-aligned DEM
+ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
+tba_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
+```
+
+Once defined, they can be applied the same two ways as for coregistration (using {func}`~xdem.coreg.Coreg.fit` and
+{func}`~xdem.coreg.Coreg.apply` separately allows to re-apply the same correction to different elevation data).
+
+```{code-cell} ipython3
+# Coregister with bias correction by calling the DEM method
+corrected_dem = tba_dem.coregister_3d(ref_dem, biascorr)
+# (Equivalent) Or by calling the fit and apply steps
+corrected_dem = biascorr.fit_and_apply(ref_dem, tba_dem)
+```
+
+Additionally, **bias corrections can be customized to use any number of variables to correct simultaneously**,
+by defining `bias_var_names` in {class}`~xdem.coreg.BiasCorr` and passing a `bias_vars` dictionary arrays or rasters
+to {func}`~xdem.coreg.Coreg.fit` and {func}`~xdem.coreg.Coreg.apply`. See {ref}`custom-biascorr` for more details.
+
+
+## The modular {class}`~xdem.coreg.BiasCorr` object
+
+### Inherited from {class}`~xdem.coreg.Coreg`
+
+Each bias-correction method in xDEM inherits their interface from the {class}`~xdem.coreg.Coreg` class (see {ref}`coreg_object`).
+This implies that bias-correction methods can be combined in a {class}`~xdem.coreg.CoregPipeline` with any other methods, or
+applied in a block-wise manner through {class}`~xdem.coreg.BlockwiseCoreg`.
**Inheritance diagram of co-registration and bias corrections:**
```{eval-rst}
-.. inheritance-diagram:: xdem.coreg.base xdem.coreg.affine xdem.coreg.biascorr
- :top-classes: xdem.Coreg
+.. inheritance-diagram:: xdem.coreg.base.Coreg xdem.coreg.affine xdem.coreg.biascorr
+ :top-classes: xdem.coreg.Coreg
```
-As a result, each bias-correction approach has the following methods:
-
-- {func}`~xdem.BiasCorr.fit` for estimating the bias.
-- {func}`~xdem.BiasCorr.apply` for correcting the bias on a DEM.
+The main difference with {class}`~xdem.coreg.Coreg` is that a {class}`~xdem.coreg.BiasCorr` has a new `bias_var_names`
+argument which allows to declare the names of N bias-correction variables that will be passed, which **corresponds to the
+number of simultaneous dimensions in which the bias correction is performed**.
+This step is implicit for predefined methods such as {class}`~xdem.coreg.DirectionalBias`.
-## Modular estimators
+### Modular estimation
-Bias-correction methods have 3 main ways of estimating and correcting a bias, both relying on one or several variables:
+Bias-correction methods have three ways of estimating and correcting a bias in N-dimensions:
-- **Performing a binning of the data** along variables with a statistic (e.g., median), and applying the statistics in each bin,
-- **Fitting a parametric function** to the variables, and applying that function,
-- **(Recommended1) Fitting a parametric function on a data binning** of the variable, and applying that function.
+- **Performing a binning of the data** along variables with a statistic (e.g., median), then applying the statistics in each bin,
+- **Fitting a parametric function** to the variables, then applying that function,
+- **(Recommended1) Fitting a parametric function on a data binning** of the variable, then applying that function.
```{margin}
-1DEM alignment is a big data problem often plagued by outliers, greatly **simplified** and **accelerated** by binning with robust estimators.
+1DEM correction is a big data problem plagued by outliers, more robust and computationally efficient when binning with robust estimators.
+
+See the **{ref}`robust-estimators` guide page** for more details.
```
-To define the parameters related to fitting and/or binning, every {func}`~xdem.BiasCorr` is instantiated with the same arguments:
+The parameters related to fitting or binning are the same for every {func}`~xdem.coreg.BiasCorr` method:
-- `fit_or_bin` to either fit a parametric model to the bias by passing "fit", perform an empirical binning of the bias by passing "bin", or to fit a parametric model to the binning with "bin_and_fit" **(recommended)**,
+- `fit_or_bin` to either fit a parametric model to the bias by passing **"fit"**, perform an empirical binning of the bias by passing **"bin"**, or to fit a parametric model to the binning with **"bin_and_fit" (recommended)**,
- `fit_func` to pass any parametric function to fit to the bias,
- `fit_optimizer` to pass any optimizer function to perform the fit minimization,
- `bin_sizes` to pass the size or edges of the bins for each variable,
- `bin_statistic` to pass the statistic to compute in each bin,
- `bin_apply_method` to pass the method to apply the binning for correction.
-```{code-cell} ipython3
-:tags: [hide-input, hide-output]
+For predefined methods, the default values of these parameters differ. For instance, a {class}`~xdem.coreg.Deramp` generally performs well
+with a **"fit"** estimation on a subsample, and thus has a fixed `fit_func` (2D polynomial) solved by the classic optimizer {func}`scipy.optimize.curve_fit`.
+In contrast, a {class}`~xdem.coreg.TerrainBias` is generally hard to model parametrically, and thus defaults to a **"bin"** estimation.
-import geoutils as gu
-import numpy as np
+Finally, each bias-correction approach has the following methods:
-import xdem
+- {func}`~xdem.coreg.Coreg.fit` for estimating the bias, which expects a new `bias_vars` dictionary **except for predefined methods** such as {class}`~xdem.coreg.DirectionalBias`,
+- {func}`~xdem.coreg.Coreg.apply` for correcting the bias on a DEM, which also expects a `bias_vars` dictionary **except for predefined methods**.
-# Open a reference DEM from 2009
-ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
-# Open a to-be-aligned DEM from 1990
-tba_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem")).reproject(ref_dem, silent=True)
+### Good practices
-# Open glacier polygons from 1990, corresponding to unstable ground
-glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
-# Create an inlier mask of terrain outside the glacier polygons
-inlier_mask = glacier_outlines.create_mask(ref_dem)
-```
+Several good practices help performing a successful bias correction:
-(biascorr-deramp)=
+- **Avoid using "fit" with a subsample size larger than 1,000,000:** Otherwise the optimizer will be extremely slow and might fail with a memory error; consider using "bin_and_fit" instead to reduce the data size before the optimization which still allows to utilize all the data,
+- **Avoid using "fit" or "bin_and_fit" for more than 2 dimensions (input variables):** Fitting a parametric form in more than 2 dimensions is quite delicate, consider using "bin" or sequential 1D corrections instead,
+- **Use at least 1000 bins for all dimensions, being mindful about dimension number:** Using a small bin size is generally too rough, but a large bin size will grow exponentially with the number of bias variables,
+- **Use customized bin edges for data with large extreme values:** Passing simply a bin size will set the min/max of the data as the full binning range, which can be impractical (e.g., most curvatures lie between -2/2 but can take values of 10000+).
-## Deramping
+## Bias-correction methods
-{class}`xdem.coreg.Deramp`
+```{important}
+Below we **create biased elevation data to examplify the different methods** in relation to their type of correction.
-- **Performs:** Correct biases with a 2D polynomial of degree N.
-- **Supports weights** Yes.
-- **Recommended for:** Residuals from camera model.
+See bias correction on real data in the **{ref}`examples-basic` and {ref}`examples-advanced` gallery examples**!
+```
-Deramping works by estimating and correcting for an N-degree polynomial over the entire dDEM between a reference and the DEM to be aligned.
-This may be useful for correcting small rotations in the dataset, or nonlinear errors that for example often occur in structure-from-motion derived optical DEMs (e.g. Rosnell and Honkavaara [2012](https://doi.org/10.3390/s120100453); Javernick et al. [2014](https://doi.org/10.1016/j.geomorph.2014.01.006); Girod et al. [2017](https://doi.org/10.5194/tc-11-827-2017)).
+(deramp)=
+### Deramping
-### Limitations
+{class}`xdem.coreg.Deramp`
-Deramping does not account for horizontal (X/Y) shifts, and should most often be used in conjunction with other methods.
+- **Performs:** Correction with a 2D polynomial of degree N.
+- **Supports weights:** Yes.
+- **Pros:** Can help correct a large category of biases (lens deformations, camera positioning), and runs fast.
+- **Cons:** Overfits with limited static surfaces.
-1st order deramping is not perfectly equivalent to a rotational correction: values are simply corrected in the vertical direction, and therefore includes a horizontal scaling factor, if it would be expressed as a transformation matrix.
-For large rotational corrections, [ICP] is recommended.
+Deramping works by estimating and correcting for an N-degree polynomial over the entire elevation difference.
-### Example
```{code-cell} ipython3
-from xdem import coreg
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for adding a ramp bias"
+: code_prompt_hide: "Hide the code for adding a ramp bias"
+
+# Get grid coordinates
+xx, yy = np.meshgrid(np.arange(0, ref_dem.shape[1]), np.arange(0, ref_dem.shape[0]))
+
+# Create a ramp bias and add to the DEM
+cx = ref_dem.shape[1] / 2
+cy = ref_dem.shape[0] / 2
+synthetic_bias = 20 * ((xx - cx)**2 + (yy - cy)**2) / (cx * cy)
+synthetic_bias -= np.median(synthetic_bias)
+tbc_dem_ramp = ref_dem + synthetic_bias
+```
-# Instantiate a 1st order deramping
-deramp = coreg.Deramp(poly_order=1)
-# Fit the data to a suitable polynomial solution
-deramp.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)
+```{code-cell} ipython3
+# Instantiate a 2nd order 2D deramping
+deramp = xdem.coreg.Deramp(poly_order=2)
+# Fit and apply
+corrected_dem = deramp.fit_and_apply(ref_dem, tbc_dem_ramp)
+```
-# Apply the transformation
-corrected_dem = deramp.apply(tba_dem)
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before deramp")
+(tbc_dem_ramp - ref_dem).plot(cmap='RdYlBu', ax=ax[0])
+ax[1].set_title("After deramp")
+(corrected_dem - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
```
-## Directional biases
+### Directional biases
{class}`xdem.coreg.DirectionalBias`
-- **Performs:** Correct biases along a direction of the DEM.
-- **Supports weights** Yes.
-- **Recommended for:** Undulations or jitter, common in both stereo and radar DEMs.
+- **Performs:** Correct biases along a direction.
+- **Supports weights:** Yes.
+- **Pros:** Correcting undulations or jitter, common in both stereo and radar DEMs, or strips common in scanned imagery.
+- **Cons:** Long optimization when fitting a sum of sinusoids.
-The default optimizer for directional biases optimizes a sum of sinusoids using 1 to 3 different frequencies, and keeping the best performing fit.
+For strip-like errors, performing an empirical correction using only a binning with `fit_or_bin="bin"` allows more
+flexibility than a parametric form, but requires a large amount of static surfaces.
-### Example
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for adding a strip bias"
+: code_prompt_hide: "Hide the code for adding a strip bias"
+
+# Get rotated coordinates along an angle
+angle = 60
+xx = gu.raster.get_xy_rotated(ref_dem, along_track_angle=angle)[0]
+
+# Create a strip bias and add to the DEM
+synthetic_bias = np.zeros(np.shape(ref_dem.data))
+xmin = np.min(xx)
+synthetic_bias[np.logical_and((xx - xmin)<1200, (xx - xmin)>800)] = 20
+synthetic_bias[np.logical_and((xx - xmin)<2800, (xx - xmin)>2500)] = -10
+synthetic_bias[np.logical_and((xx - xmin)<5300, (xx - xmin)>5000)] = 10
+synthetic_bias[np.logical_and((xx - xmin)<15000, (xx - xmin)>14500)] = 5
+synthetic_bias[np.logical_and((xx - xmin)<21000, (xx - xmin)>20000)] = -15
+tbc_dem_strip = ref_dem + synthetic_bias
+```
```{code-cell} ipython3
-# Instantiate a directional bias correction
-dirbias = coreg.DirectionalBias(angle=65)
-# Fit the data
-dirbias.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)
+# Define a directional bias correction at a certain angle (degrees), for a binning of 1000 bins
+dirbias = xdem.coreg.DirectionalBias(angle=60, fit_or_bin="bin", bin_sizes=1000)
+# Fit and apply
+corrected_dem = dirbias.fit_and_apply(ref_dem, tbc_dem_strip)
+```
-# Apply the transformation
-corrected_dem = dirbias.apply(tba_dem)
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before directional\nde-biasing")
+(tbc_dem_strip - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[0])
+ax[1].set_title("After directional\nde-biasing")
+(corrected_dem - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
```
-## Terrain biases
+### Terrain biases
{class}`xdem.coreg.TerrainBias`
-- **Performs:** Correct biases along a terrain attribute of the DEM.
-- **Supports weights** Yes.
-- **Recommended for:** Different native resolution between DEMs.
+- **Performs:** Correct biases along a terrain attribute.
+- **Supports weights:** Yes.
+- **Pros:** Useful to correct for instance curvature-related bias due to different native resolution between elevation data.
+- **Cons:** For curvature-related biases, only works for elevation data with relatively close native resolution.
-The default optimizer for terrain biases optimizes a 1D polynomial with an order from 1 to 6, and keeping the best performing fit.
+The default optimizer for terrain biases optimizes a 1D polynomial with an order from 1 to 6,
+and keeps the best performing fit.
-### Example
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for adding a curvature bias"
+: code_prompt_hide: "Hide the code for adding a curvature bias"
+
+# Get maximum curvature
+maxc = ref_dem.maximum_curvature()
+
+# Create a bias depending on bins
+synthetic_bias = np.zeros(np.shape(ref_dem.data))
+
+# For each bin, add the curvature bias
+bin_edges = np.array((-1, -0.5, -0.1, 0.1, 0.5, 2, 5))
+bias_per_bin = np.array((-10, -5, 0, 5, 10, 20))
+for i in range(len(bin_edges) - 1):
+ synthetic_bias[np.logical_and(maxc.data >= bin_edges[i], maxc.data < bin_edges[i + 1])] = bias_per_bin[i]
+tbc_dem_curv = ref_dem + synthetic_bias
+```
+
+```{code-cell} ipython3
+# Instantiate a 1st order terrain bias correction for curvature
+terbias = xdem.coreg.TerrainBias(terrain_attribute="maximum_curvature",
+ bin_sizes={"maximum_curvature": np.linspace(-5, 5, 1000)},
+ bin_apply_method="per_bin")
+
+# We have to pass the original curvature here
+corrected_dem = terbias.fit_and_apply(ref_dem, tbc_dem_curv, bias_vars={"maximum_curvature": maxc})
+```
```{code-cell} ipython3
-# Instantiate a 1st order terrain bias correction
-terbias = coreg.TerrainBias(terrain_attribute="maximum_curvature")
-# Fit the data
-terbias.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before terrain\nde-biasing")
+(tbc_dem_curv - ref_dem).plot(cmap='RdYlBu', vmin=-10, vmax=10, ax=ax[0])
+ax[1].set_title("After terrain\nde-biasing")
+(corrected_dem - ref_dem).plot(cmap='RdYlBu', vmin=-10, vmax=10, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+```
+
+
+(custom-biascorr)=
+### Using custom variables
+
+All bias-corrections methods are inherited from generic classes that perform corrections in 1-, 2- or N-D. Having these
+separate helps the user navigating the dimensionality of the functions, optimizer, binning or variables used.
-# Apply the transformation
-corrected_dem = terbias.apply(tba_dem)
+{class}`xdem.coreg.BiasCorr`
+
+- **Performs:** Correct biases with any function and optimizer, or any binning, in 1-, 2- or N-D.
+- **Supports weights:** Yes.
+- **Pros:** Versatile.
+- **Cons:** Needs more setting up!
+
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code for creating inlier mask and coregistration"
+: code_prompt_hide: "Hide code for creating inlier mask and coregistration"
+
+import geoutils as gu
+
+# Open glacier outlines as vector
+glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
+
+# Create a stable ground mask (not glacierized) to mark "inlier data"
+inlier_mask = ~glacier_outlines.create_mask(ref_dem)
+
+# We align the two DEMs before doing any bias correction
+tba_dem_nk = tba_dem.coregister_3d(ref_dem, xdem.coreg.NuthKaab())
+```
+
+```{code-cell} ipython3
+# Create a bias correction defining three custom variable names that will be passed later
+# We force a binning method, more simple in 3D
+biascorr = xdem.coreg.BiasCorr(bias_var_names=["aspect", "slope", "elevation"], fit_or_bin="bin", bin_sizes=5)
+
+# Derive curvature and slope
+aspect, slope = ref_dem.get_terrain_attribute(["aspect", "slope"])
+
+# Pass the variables to the fit_and_apply function matching the names declared above
+corrected_dem = biascorr.fit_and_apply(
+ ref_dem,
+ tba_dem_nk,
+ inlier_mask=inlier_mask,
+ bias_vars={"aspect": aspect, "slope": slope, "elevation": ref_dem}
+)
+```
+
+```{warning}
+Using any custom variables, and especially in many dimensions, **can lead to over-correction and introduce new errors**.
+For instance, elevation-dependent corrections (as shown below) typically introduce new errors (due to more high curvatures
+at high elevation such as peaks, and low curvatures at low elevation with flat terrain).
+
+For this reason, it is important to check the sanity of elevation differences after correction!
+```
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before 3D\nde-biasing")
+(tba_dem_nk - ref_dem).plot(cmap='RdYlBu', vmin=-10, vmax=10, ax=ax[0])
+ax[1].set_title("After 3D\nde-biasing")
+(corrected_dem - ref_dem).plot(cmap='RdYlBu', vmin=-10, vmax=10, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
```
diff --git a/doc/source/cheatsheet.md b/doc/source/cheatsheet.md
new file mode 100644
index 00000000..bf47f725
--- /dev/null
+++ b/doc/source/cheatsheet.md
@@ -0,0 +1,266 @@
+---
+file_format: mystnb
+mystnb:
+ execution_timeout: 60
+jupytext:
+ formats: md:myst
+ text_representation:
+ extension: .md
+ format_name: myst
+kernelspec:
+ display_name: xdem-env
+ language: python
+ name: xdem
+---
+(cheatsheet)=
+
+# Cheatsheet: How to correct... ?
+
+In elevation data analysis, the problem generally starts with identifying what correction method to apply when
+observing a specific pattern of error in your own data.
+
+Below, we summarize a cheatsheet that links what method is likely to correct a pattern of error you can visually
+identify on **a map of elevation differences with another elevation dataset (looking at static surfaces)**!
+
+## Cheatsheet
+
+The patterns of errors categories listed in this spreadsheet **are linked to visual examples further below** that
+you use to compare to your own elevation differences.
+
+```{list-table}
+ :widths: 1 2 2 2
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Pattern
+ - Description
+ - Cause and correction
+ - Notes
+ * - {ref}`sharp-landforms`
+ - Positive and negative errors that are larger near high slopes and symmetric with opposite slope orientation, making landforms appear visually.
+ - Likely horizontal shift due to geopositioning errors, use a {ref}`coregistration` such as {class}`~xdem.coreg.NuthKaab`.
+ - Even a tiny horizontal misalignment can be visually identified! To not confuse with {ref}`peak-cavity`.
+ * - {ref}`smooth-large-field`
+ - Smooth offsets varying at scale of 10 km+, often same sign (either positive or negative).
+ - Likely wrong {ref}`vertical-ref`, can set and transform with {func}`~xdem.DEM.set_vcrs` and {func}`~xdem.DEM.to_vcrs`.
+ - Vertical references often only exists in a user guide, they are not coded in the raster CRS and need to be set manually.
+ * - {ref}`ramp-or-dome`
+ - Ramping errors, often near the edge of the data extent, sometimes with a center dome.
+ - Likely ramp/rotations due to camera errors, use either a {ref}`coregistration` such as {class}`~xdem.coreg.ICP` or a {ref}`biascorr` such as {class}`~xdem.coreg.Deramp`.
+ - Can sometimes be more rigorously fixed ahead of DEM generation with bundle adjustment.
+ * - {ref}`undulations`
+ - Positive and negative errors undulating patterns at one or several frequencies well larger than pixel size.
+ - Likely jitter-type errors, use a {ref}`biascorr` such as {class}`~xdem.coreg.DirectionalBias`.
+ - Can sometimes be more rigorously fixed ahead of DEM generation with jitter correction.
+ * - {ref}`peak-cavity`
+ - Positive and negative errors, with one sign located exclusively near peaks and the other exclusively near cavities.
+ - Likely resolution-type errors, use a {ref}`biascorr` such as {class}`~xdem.coreg.TerrainBias`.
+ - Can be over-corrected, sometimes better to simply ignore during analysis. Or avoid by downsampling all elevation data to the lowest resolution, rather than upsampling to the highest.
+ * - {ref}`point-alternating`
+ - Point data errors that alternate between negative and positive, higher on steeper slopes.
+ - Likely wrong point-raster comparison, use [point interpolation or reduction on the raster instead](https://geoutils.readthedocs.io/en/stable/raster_vector_point.html#rasterpoint-operations) such as {func}`~xdem.DEM.interp_points`.
+ - Rasterizing point data introduces spatially correlated random errors, instead it is recommended to interpolate raster data at the point coordinates.
+```
+
+## Visual patterns of errors
+
+```{important}
+The patterns of errors below are **created synthetically to examplify the effect of a single source of error**.
+In your own elevation differences, those will be mixed together and with random errors inherent to the data.
+
+For examples on real data, see the **{ref}`examples-basic` and {ref}`examples-advanced` gallery examples**!
+```
+
+```{code-cell} ipython3
+:tags: [remove-cell]
+
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 600
+pyplot.rcParams['savefig.dpi'] = 600
+pyplot.rcParams['font.size'] = 9 # Default 10 is a bit too big for coregistration plots
+```
+
+It is often crucial to relate the location of your errors on static surfaces to the terrain distribution
+(in particular, its slope and curvature), which can usually be inferred visually from a hillshade.
+
+
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for opening example file"
+: code_prompt_hide: "Hide the code for opening example file"
+
+import xdem
+import geoutils as gu
+import geopandas as gpd
+import numpy as np
+import matplotlib.pyplot as plt
+
+# Open an example DEM
+dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
+```
+
+```{code-cell} ipython3
+hs = dem.hillshade()
+hs.plot(cmap="Greys_r", cbar_title="Hillshade")
+```
+
+(sharp-landforms)=
+### Sharp landforms
+
+Example of sharp landforms appearing with a horizontal shift due to geolocation errors. We here translate the DEM
+horizontally by 1/10th of a pixel, for a pixel resolution of 20 m.
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code to simulate horizontal shift errors"
+: code_prompt_hide: "Hide code to simulate horizontal shift errors"
+
+# Simulate a translation of 1/10th of a pixel
+x_shift = 0.1
+y_shift = 0.1
+dem_shift = dem.translate(x_shift, y_shift, distance_unit="pixel")
+
+# Resample and plot the elevation differences of the horizontal shift
+dh = dem - dem_shift.reproject(dem)
+dh.plot(cmap='RdYlBu', vmin=-3, vmax=3, cbar_title="Elevation differences of\nhorizontal shift (m)")
+```
+
+(smooth-large-field)=
+### Smooth-field offset
+
+Example of smooth large offset field created by a wrong vertical CRS. We here show the difference due to the EGM96
+geoid added on top of the ellipsoid.
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code to simulate vertical referencing errors"
+: code_prompt_hide: "Hide code to simulate vertical referencing errors"
+
+# Set current vertical CRS as ellipsoid
+dem.set_vcrs("Ellipsoid")
+# Transform vertical reference to geoid
+trans_dem = dem.to_vcrs("EGM96")
+
+# Plot the elevation differences of the vertical transformation
+dh = dem - trans_dem
+dh.plot(cmap='RdYlBu', cbar_title="Elevation differences of\nvertical transform (m)")
+```
+
+(ramp-or-dome)=
+### Ramp or dome
+
+Example of ramp created by a rotation due to camera errors. We here show just a slight rotation of 0.02 degrees.
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code to simulate rotation errors"
+: code_prompt_hide: "Hide code to simulate rotation errors"
+
+# Apply a rotation of 0.02 degrees
+rotation = np.deg2rad(0.02)
+# Affine matrix for 3D transformation
+matrix = np.array(
+ [
+ [1, 0, 0, 0],
+ [0, np.cos(rotation), -np.sin(rotation), 0],
+ [0, np.sin(rotation), np.cos(rotation), 0],
+ [0, 0, 0, 1],
+ ]
+)
+# Select a centroid
+centroid = [dem.bounds.left + 5000, dem.bounds.top - 2000, np.median(dem) + 100]
+# We rotate the elevation data
+dem_rotated = xdem.coreg.apply_matrix(dem, matrix, centroid=centroid)
+
+# Plot the elevation differences of the rotation
+dh = dem - dem_rotated
+dh.plot(cmap='RdYlBu', cbar_title="Elevation differences of\nrotation (m)")
+```
+
+
+(undulations)=
+### Undulations
+
+Example of undulations resembling jitter errors. We here artificially create a sinusoidal signal at a certain angle.
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code to simulate undulation errors"
+: code_prompt_hide: "Hide code to simulate undulation errors"
+
+# Get rotated coordinates along an angle
+angle = -20
+xx = gu.raster.get_xy_rotated(dem, along_track_angle=angle)[0]
+
+# One sinusoid: amplitude, phases and frequencies
+params = np.array([(2, 3000, np.pi)]).flatten()
+
+# Create a sinusoidal bias and add to the DEM
+from xdem.fit import sumsin_1d
+synthetic_bias_arr = sumsin_1d(xx.flatten(), *params).reshape(np.shape(dem.data))
+
+# Plot the elevation differences of the undulations
+synthetic_bias = dem.copy(new_array=synthetic_bias_arr)
+synthetic_bias.plot(cmap='RdYlBu', vmin=-3, vmax=3, cbar_title="Elevation differences of\nundulations (m)")
+```
+
+(peak-cavity)=
+### Peak cuts and cavity fills
+
+Example of peak cutting and cavity filling errors. We here downsampled our DEM from 20 m to 100 m to simulate a lower
+native resolution, then upsample it again to 20 m, to show the errors affect areas near high curvatures such as
+peaks and cavities.
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code to simulate resolution errors"
+: code_prompt_hide: "Hide code to simulate resolution errors"
+
+# Downsample DEM (bilinear)
+dem_100m = dem.reproject(res=100)
+
+# Upsample (bilinear again) and compare
+dh = dem - dem_100m.reproject(dem)
+dh.plot(cmap='RdYlBu', vmin=-40, vmax=40, cbar_title="Elevation differences of\nresolution change (m)")
+```
+
+(point-alternating)=
+### Point alternating
+
+An example of alternating point errors created by wrong point-raster comparison by rasterization of the points,
+which are especially large around steep slopes.
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code to simulate point-raster comparison errors"
+: code_prompt_hide: "Hide code to simulate point-raster comparison errors"
+
+# Simulate swath coordinates of an elevation point cloud
+x = np.linspace(dem.bounds.left, dem.bounds.right, 100)
+y = np.linspace(dem.bounds.top - 5000, dem.bounds.bottom + 5000, 100)
+
+# Interpolate DEM at these coordinates to build the point cloud
+# (to approximate the real elevation at these coordinates,
+# which has negligible impact compared to rasterization)
+z = dem.interp_points((x,y))
+epc = gu.Vector(gpd.GeoDataFrame(geometry=gpd.points_from_xy(x=x, y=y, crs=dem.crs), data={"z": z}))
+
+# Rasterize point cloud back on the DEM grid
+epc_rast = epc.rasterize(dem, in_value=z, out_value=np.nan)
+
+# For easier plotting, convert the valid dh values to points
+dh = dem - epc_rast
+dh_pc = dh.to_pointcloud(data_column_name="dh")
+
+# Plot the elevation differences of the rasterization on top of a hillshade
+hs.plot(cmap="Greys_r", add_cbar=False)
+dh_pc.plot(column="dh", cmap='RdYlBu', vmin=-10, vmax=10, legend=True, cbar_title="Elevation differences of\npoint-raster differencing (m)")
+```
diff --git a/doc/source/citation.md b/doc/source/citation.md
new file mode 100644
index 00000000..7966b760
--- /dev/null
+++ b/doc/source/citation.md
@@ -0,0 +1,114 @@
+(citation)=
+
+# Citing and method overview
+
+When using a method implemented in xDEM, one should cite both the package and the original study behind the method (if there is any)!
+
+## Citing xDEM
+
+To cite the package, use the Zenodo DOI: [](https://zenodo.org/doi/10.5281/zenodo.4809697).
+
+## Method overview
+
+For citation and other purposes, here's an overview of all methods implemented in the package and their reference, if it exists.
+More details are available on each feature page!
+
+### Terrain attributes
+
+```{list-table}
+ :widths: 1 1
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Method
+ - Reference
+ * - Slope, aspect and hillshade
+ - [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) or [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107)
+ * - Curvatures
+ - [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107)
+ * - Topographic position index
+ - [Weiss (2001)](http://www.jennessent.com/downloads/TPI-poster-TNC_18x22.pdf)
+ * - Terrain ruggedness index
+ - [Riley et al. (1999)](http://download.osgeo.org/qgis/doc/reference-docs/Terrain_Ruggedness_Index.pdf) or [Wilson et al. (2007)](http://dx.doi.org/10.1080/01490410701295962)
+ * - Roughness
+ - [Dartnell (2000)](https://environment.sfsu.edu/node/11292)
+ * - Rugosity
+ - [Jenness (2004)]()
+ * - Fractal roughness
+ - [Taud and Parrot (2005)](https://doi.org/10.4000/geomorphologie.622)
+```
+
+### Coregistration
+
+```{list-table}
+ :widths: 1 1
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Method
+ - Reference
+ * - Nuth and Kääb
+ - [Nuth and Kääb (2011)](https://doi.org/10.5194/tc-5-271-2011)
+ * - Dh minimization
+ - N/A
+ * - Iterative closest point
+ - [Besl and McKay (1992)](https://doi.org/10.1117/12.57955)
+ * - Vertical shift
+ - N/A
+```
+
+### Bias-correction
+
+```{list-table}
+ :widths: 1 1
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Method
+ - Reference
+ * - Deramp
+ - N/A
+ * - Directional bias (sum of sinuoids)
+ - [Girod et al. (2017)](https://doi.org/10.3390/rs9070704)
+ * - Directional bias (other)
+ - N/A
+ * - Terrain bias (maximum curvature)
+ - [Gardelle et al. (2012)](https://doi.org/10.3189/2012JoG11J175)
+ * - Terrain bias (elevation)
+ - [Nuth and Kääb (2011)](https://doi.org/10.5194/tc-5-271-2011)
+ * - Terrain bias (other)
+ - N/A
+ * - Vertical shift
+ - N/A
+```
+
+### Gap-filling
+
+```{list-table}
+ :widths: 1 1
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Method
+ - Reference
+ * - Bilinear
+ - N/A
+ * - Local and regional hypsometric
+ - [Arendt et al. (2002)](https://doi.org/10.1126/science.1072497), [McNabb et al. (2019)](https://tc.copernicus.org/articles/13/895/2019/)
+```
+
+
+### Uncertainty analysis
+
+```{list-table}
+ :widths: 2 1
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Method
+ - Reference
+ * - R2009 (multiple correlation ranges, circular approximation)
+ - [Rolstad et al. (2009)](http://dx.doi.org/10.3189/002214309789470950)
+ * - H2022 (heteroscedasticity, multiple correlation ranges, spatial propagation approximation)
+ - [Hugonnet et al. (2022)](http://dx.doi.org/10.1109/JSTARS.2022.3188922)
+```
diff --git a/doc/source/code/comparison_plot_spatial_interpolation.py b/doc/source/code/comparison_plot_spatial_interpolation.py
index c80e83e7..62ee7a2a 100644
--- a/doc/source/code/comparison_plot_spatial_interpolation.py
+++ b/doc/source/code/comparison_plot_spatial_interpolation.py
@@ -15,7 +15,7 @@
# Introduce 50000 nans randomly throughout the dDEM.
ddem.data.mask.ravel()[np.random.default_rng(42).choice(ddem.data.size, 50000, replace=False)] = True
-ddem.interpolate(method="linear")
+ddem.interpolate(method="idw")
ylim = (300, 100)
xlim = (800, 1050)
diff --git a/doc/source/code/coregistration_plot_nuth_kaab.py b/doc/source/code/coregistration_plot_nuth_kaab.py
deleted file mode 100644
index dae51c53..00000000
--- a/doc/source/code/coregistration_plot_nuth_kaab.py
+++ /dev/null
@@ -1,39 +0,0 @@
-"""Plot the comparison between a dDEM before and after Nuth and Kääb (2011) coregistration."""
-import geoutils as gu
-import matplotlib.pyplot as plt
-import numpy as np
-
-import xdem
-
-dem_2009 = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
-dem_1990 = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
-outlines_1990 = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
-inlier_mask = ~outlines_1990.create_mask(dem_2009)
-
-nuth_kaab = xdem.coreg.NuthKaab()
-nuth_kaab.fit(dem_2009, dem_1990, inlier_mask=inlier_mask)
-dem_coreg = nuth_kaab.apply(dem_1990)
-
-ddem_pre = dem_2009 - dem_1990
-ddem_post = dem_2009 - dem_coreg
-
-nmad_pre = xdem.spatialstats.nmad(ddem_pre[inlier_mask])
-nmad_post = xdem.spatialstats.nmad(ddem_post[inlier_mask])
-
-vlim = 20
-plt.figure(figsize=(8, 5))
-plt.subplot2grid((1, 15), (0, 0), colspan=7)
-plt.title(f"Before coregistration. NMAD={nmad_pre:.1f} m")
-plt.imshow(ddem_pre.data, cmap="coolwarm_r", vmin=-vlim, vmax=vlim)
-plt.axis("off")
-plt.subplot2grid((1, 15), (0, 7), colspan=7)
-plt.title(f"After coregistration. NMAD={nmad_post:.1f} m")
-img = plt.imshow(ddem_post.data, cmap="coolwarm_r", vmin=-vlim, vmax=vlim)
-plt.axis("off")
-plt.subplot2grid((1, 15), (0, 14), colspan=1)
-cbar = plt.colorbar(img, fraction=0.4, ax=plt.gca())
-cbar.set_label("Elevation change (m)")
-plt.axis("off")
-
-plt.tight_layout()
-plt.show()
diff --git a/doc/source/code/intricacies_datatypes.py b/doc/source/code/intricacies_datatypes.py
new file mode 100644
index 00000000..35b5fb9f
--- /dev/null
+++ b/doc/source/code/intricacies_datatypes.py
@@ -0,0 +1,58 @@
+"""Plot example of elevation data types for guide page."""
+import matplotlib
+import matplotlib.pyplot as plt
+import numpy as np
+
+import xdem
+
+# Open reference DEM and crop to small area
+ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
+ref_dem = ref_dem.crop(
+ (ref_dem.bounds.left, ref_dem.bounds.bottom, ref_dem.bounds.left + 1000, ref_dem.bounds.bottom + 1000)
+)
+
+# Get point cloud with 100 points
+ref_epc = ref_dem.to_pointcloud(subsample=100, random_state=42)
+
+f, ax = plt.subplots(2, 2, squeeze=False, sharex=True, sharey=True)
+# Plot 1: DEM
+ax[0, 0].set_title("DEM")
+ref_dem.plot(cmap="terrain", ax=ax[0, 0], vmin=280, vmax=420, cbar_title="Elevation (m)")
+plt.gca().set_xticklabels([])
+plt.gca().set_yticklabels([])
+plt.gca().set_aspect("equal")
+
+# Plot 2: EPC
+ax[0, 1].set_title("Elevation\npoint cloud")
+point = ref_epc.plot(column="b1", cmap="terrain", ax=ax[0, 1], vmin=280, vmax=420, cbar_title="Elevation (m)")
+plt.gca().set_xticklabels([])
+plt.gca().set_yticklabels([])
+plt.gca().set_aspect("equal")
+
+# Plot 3: TIN
+ax[1, 1].set_title("Elevation TIN")
+triang = matplotlib.tri.Triangulation(ref_epc.geometry.x.values, ref_epc.geometry.y.values)
+ax[1, 1].triplot(triang, color="gray", marker=".")
+scat = ax[1, 1].scatter(
+ ref_epc.geometry.x.values, ref_epc.geometry.y.values, c=ref_epc["b1"].values, cmap="terrain", vmin=280, vmax=420
+)
+plt.colorbar(mappable=scat, ax=ax[1, 1], label="Elevation (m)", pad=0.02)
+ax[1, 1].set_xticklabels([])
+ax[1, 1].set_yticklabels([])
+ax[1, 1].set_aspect("equal")
+
+# Plot 4: Contour
+ax[1, 0].set_title("Elevation contour")
+coords = ref_dem.coords(grid=False)
+cont = ax[1, 0].contour(
+ np.flip(coords[0]), coords[1], np.flip(ref_dem.get_nanarray()), levels=15, cmap="terrain", vmin=280, vmax=420
+)
+plt.colorbar(mappable=cont, ax=ax[1, 0], label="Elevation (m)", pad=0.02)
+ax[1, 0].set_xticklabels([])
+ax[1, 0].set_yticklabels([])
+ax[1, 0].set_aspect("equal")
+
+plt.suptitle("Types of elevation data")
+
+plt.tight_layout()
+plt.show()
diff --git a/doc/source/code/robust_mean_std.py b/doc/source/code/robust_mean_std.py
new file mode 100644
index 00000000..29fa0257
--- /dev/null
+++ b/doc/source/code/robust_mean_std.py
@@ -0,0 +1,94 @@
+"""Plot example of NMAD/median as robust estimators for guide page."""
+import matplotlib.pyplot as plt
+import numpy as np
+
+import xdem
+
+# Create example distribution
+dh_inliers = np.random.default_rng(42).normal(loc=-5, scale=3, size=10**6)
+
+# Add outliers
+dh_outliers = np.concatenate(
+ (
+ np.repeat(-34, 600),
+ np.repeat(-33, 1800),
+ np.repeat(-32, 3600),
+ np.repeat(-31, 8500),
+ np.repeat(-30, 15000),
+ np.repeat(-29, 9000),
+ np.repeat(-28, 3800),
+ np.repeat(-27, 1900),
+ np.repeat(-26, 700),
+ )
+)
+dh_all = np.concatenate((dh_inliers, dh_outliers))
+
+# Get traditional and robust statistics on all data
+mean_dh = np.nanmean(dh_all)
+median_dh = np.nanmedian(dh_all)
+
+std_dh = np.nanstd(dh_all)
+nmad_dh = xdem.spatialstats.nmad(dh_all)
+
+# Get traditional and robust statistics on inlier data
+mean_dh_in = np.nanmean(dh_inliers)
+median_dh_in = np.nanmedian(dh_inliers)
+
+std_dh_in = np.nanstd(dh_inliers)
+nmad_dh_in = xdem.spatialstats.nmad(dh_inliers)
+
+# Plot
+fig, ax = plt.subplots()
+h1 = ax.hist(dh_inliers, bins=np.arange(-40, 25), density=False, color="gray", label="Inlier data")
+h2 = ax.hist(dh_outliers, bins=np.arange(-40, 25), density=False, color="red", label="Outlier data")
+
+max_count = max(h1[0])
+ax.vlines(x=[mean_dh_in, median_dh_in], ymin=0, ymax=max_count, colors=["tab:gray", "black"])
+ax.vlines(
+ x=[mean_dh_in - std_dh_in, mean_dh_in + std_dh_in, median_dh_in - nmad_dh_in, median_dh_in + nmad_dh_in],
+ ymin=0,
+ ymax=max_count,
+ colors=["gray", "gray", "black", "black"],
+ linestyles="dashed",
+)
+
+ax.vlines(x=[mean_dh, median_dh], ymin=0, ymax=max_count, colors=["red", "darkred"])
+ax.vlines(
+ x=[mean_dh - std_dh, mean_dh + std_dh, median_dh - nmad_dh, median_dh + nmad_dh],
+ ymin=0,
+ ymax=max_count,
+ colors=["red", "red", "darkred", "darkred"],
+ linestyles="dashed",
+)
+
+ax.set_xlim((-40, 25))
+ax.set_xlabel("Elevation differences (m)")
+ax.set_ylabel("Count")
+
+from matplotlib.patches import Rectangle
+
+handles = [
+ Rectangle((0, 0), 1, 1, color=h1[-1][0].get_facecolor(), alpha=1),
+ Rectangle((0, 0), 1, 1, color=h2[-1][0].get_facecolor(), alpha=1),
+]
+labels = ["Inlier data", "Outlier data"]
+
+data_legend = ax.legend(handles=handles, labels=labels, loc="upper right")
+ax.add_artist(data_legend)
+
+# Legends
+p1 = plt.plot([], [], color="red", label=f"Mean: {np.round(mean_dh, 2)} m")
+p2 = plt.plot([], [], color="red", linestyle="dashed", label=f"±STD: {np.round(std_dh, 2)} m")
+p3 = plt.plot([], [], color="darkred", label=f"Median: {np.round(median_dh, 2)} m")
+p4 = plt.plot([], [], color="darkred", linestyle="dashed", label=f"±NMAD: {np.round(nmad_dh, 2)} m")
+first_legend = ax.legend(handles=[p[0] for p in [p1, p2, p3, p4]], loc="center right", title="All data")
+ax.add_artist(first_legend)
+
+p1 = plt.plot([], [], color="gray", label=f"Mean: {np.round(mean_dh_in, 2)} m")
+p2 = plt.plot([], [], color="gray", linestyle="dashed", label=f"±STD: {np.round(std_dh_in, 2)} m")
+p3 = plt.plot([], [], color="black", label=f"Median: {np.round(median_dh_in, 2)} m")
+p4 = plt.plot([], [], color="black", linestyle="dashed", label=f"±NMAD: {np.round(nmad_dh_in, 2)} m")
+second_legend = ax.legend(handles=[p[0] for p in [p1, p2, p3, p4]], loc="center left", title="Inlier data")
+ax.add_artist(second_legend)
+
+ax.set_title("Effect of outliers on estimating\ncentral tendency and dispersion")
diff --git a/doc/source/code/robust_vario.py b/doc/source/code/robust_vario.py
new file mode 100644
index 00000000..d0d82aa2
--- /dev/null
+++ b/doc/source/code/robust_vario.py
@@ -0,0 +1,63 @@
+"""Plot example of Dowd variogram as robust estimator for guide page."""
+import matplotlib.pyplot as plt
+import numpy as np
+from skgstat import OrdinaryKriging, Variogram
+
+import xdem
+
+# Inspired by test_variogram in skgstat
+# Generate some random but spatially correlated data with a range of ~20
+np.random.seed(42)
+c = np.random.default_rng(41).random((50, 2)) * 60
+np.random.seed(42)
+v = np.random.default_rng(42).normal(10, 4, 50)
+
+V = Variogram(c, v).describe()
+V["effective_range"] = 20
+OK = OrdinaryKriging(V, coordinates=c, values=v)
+
+c = np.array(np.meshgrid(np.arange(60), np.arange(60).T)).reshape(2, 60 * 60).T
+dh = OK.transform(c)
+dh = dh.reshape((60, 60))
+
+# Add outliers
+dh_outliers = dh.copy()
+dh_outliers[0:6, 0:6] = -20
+
+# Derive empirical variogram for Dowd and Matheron
+df_inl_matheron = xdem.spatialstats.sample_empirical_variogram(
+ dh, estimator="matheron", gsd=1, random_state=42, subsample=2000
+)
+df_inl_dowd = xdem.spatialstats.sample_empirical_variogram(dh, estimator="dowd", gsd=1, random_state=42, subsample=2000)
+
+df_all_matheron = xdem.spatialstats.sample_empirical_variogram(
+ dh_outliers, estimator="matheron", gsd=1, random_state=42, subsample=2000
+)
+df_all_dowd = xdem.spatialstats.sample_empirical_variogram(
+ dh_outliers, estimator="dowd", gsd=1, random_state=42, subsample=2000
+)
+
+fig, ax = plt.subplots()
+
+ax.plot(df_inl_matheron.lags, df_inl_matheron.exp, color="black", marker="x")
+ax.plot(df_inl_dowd.lags, df_inl_dowd.exp, color="black", linestyle="dashed", marker="x")
+ax.plot(df_all_matheron.lags, df_all_matheron.exp, color="red", marker="x")
+ax.plot(df_all_dowd.lags, df_all_dowd.exp, color="red", linestyle="dashed", marker="x")
+
+
+p1 = plt.plot([], [], color="darkgrey", label="Matheron", marker="x")
+p2 = plt.plot([], [], color="darkgrey", linestyle="dashed", label="Dowd", marker="x")
+first_legend = ax.legend(handles=[p[0] for p in [p1, p2]], loc="lower right")
+ax.add_artist(first_legend)
+
+p1 = plt.plot([], [], color="black", label="Inlier data")
+p2 = plt.plot([], [], color="red", label="Inlier data + outlier data \n(1% of data replaced by 10 NMAD)")
+second_legend = ax.legend(handles=[p[0] for p in [p1, p2]], loc="upper left")
+ax.add_artist(second_legend)
+
+ax.set_xlabel("Spatial lag (m)")
+ax.set_ylabel("Variance of elevation changes (m²)")
+ax.set_ylim((0, 15))
+ax.set_xlim((0, 40))
+
+ax.set_title("Effect of outliers on estimating\nspatial correlation")
diff --git a/doc/source/code/spatialstats_heterosc_slope.py b/doc/source/code/spatialstats_heterosc_slope.py
index d2d554cf..1854d4fb 100644
--- a/doc/source/code/spatialstats_heterosc_slope.py
+++ b/doc/source/code/spatialstats_heterosc_slope.py
@@ -26,4 +26,4 @@
list_var_bins=30,
)
-xdem.spatialstats.plot_1d_binning(df_ns, "slope", "nmad", "Slope (degrees)", "Elevation error ($1\\sigma$, m)")
+xdem.spatialstats.plot_1d_binning(df_ns, "slope", "nmad", "Slope (degrees)", "Random elevation error\n($1\\sigma$, m)")
diff --git a/doc/source/code/spatialstats_stationarity_assumption.py b/doc/source/code/spatialstats_stationarity_assumption.py
index 35d3d963..02e81900 100644
--- a/doc/source/code/spatialstats_stationarity_assumption.py
+++ b/doc/source/code/spatialstats_stationarity_assumption.py
@@ -23,7 +23,7 @@
# Stationary mean and variance
ax1.plot(x, y_rand1, color="tab:blue", linewidth=0.5)
-ax1.hlines(0, xmin=0, xmax=1, color="black", label="Mean", linestyle="dashed")
+ax1.hlines(0, xmin=0, xmax=1, color="black", label="Mean")
ax1.hlines(
[-2 * sig, 2 * sig],
xmin=0,
@@ -45,7 +45,7 @@
# Non-stationary mean and stationary variance
ax2.plot(x, y_rand2 + y_mean, color="tab:olive", linewidth=0.5)
-ax2.plot(x, y_mean, color="black", label="Mean", linestyle="dashed")
+ax2.plot(x, y_mean, color="black", label="Mean")
ax2.plot(x, y_mean + 2 * sig, color="tab:gray", label="Dispersion (2$\\sigma$)", linestyle="dashed")
ax2.plot(x, y_mean - 2 * sig, color="tab:gray", linestyle="dashed")
ax2.set_xlim((0, 1))
@@ -61,7 +61,7 @@
# Stationary mean and non-stationary variance
ax3.plot(x, y_rand3 * fac_y_std, color="tab:orange", linewidth=0.5)
-ax3.hlines(0, xmin=0, xmax=1, color="black", label="Mean", linestyle="dashed")
+ax3.hlines(0, xmin=0, xmax=1, color="black", label="Mean")
ax3.plot(x, 2 * sig * fac_y_std, color="tab:gray", linestyle="dashed")
ax3.plot(x, -2 * sig * fac_y_std, color="tab:gray", linestyle="dashed")
ax3.set_xlim((0, 1))
diff --git a/doc/source/comparison.md b/doc/source/comparison.md
deleted file mode 100644
index 39a69d55..00000000
--- a/doc/source/comparison.md
+++ /dev/null
@@ -1,127 +0,0 @@
----
-file_format: mystnb
-jupytext:
- formats: md:myst
- text_representation:
- extension: .md
- format_name: myst
-kernelspec:
- display_name: xdem-env
- language: python
- name: xdem
----
-# Differencing and volume change
-
-**Example data**
-
-Example data in this chapter are loaded as follows:
-
-```{code-cell} ipython3
-from datetime import datetime
-
-import geoutils as gu
-import numpy as np
-
-import xdem
-
-# Load a reference DEM from 2009
-dem_2009 = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"), datetime=datetime(2009, 8, 1))
-# Load a DEM from 1990
-dem_1990 = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"), datetime=datetime(1990, 8, 1))
-# Load glacier outlines from 1990.
-glaciers_1990 = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
-glaciers_2010 = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines_2010"))
-
-# Make a dictionary of glacier outlines where the key represents the associated date.
-outlines = {
- datetime(1990, 8, 1): glaciers_1990,
- datetime(2009, 8, 1): glaciers_2010,
-}
-```
-
-## dDEM interpolation
-
-There are many approaches to interpolate a dDEM.
-A good comparison study for glaciers is McNabb et al., ([2019](https://doi.org/10.5194/tc-13-895-2019)).
-So far, xDEM has three types of interpolation:
-
-- Linear spatial interpolation
-- Local hypsometric interpolation
-- Regional hypsometric interpolation
-
-Let's first create a {class}`xdem.ddem.dDEM` object to experiment on:
-
-```{code-cell} ipython3
-ddem = xdem.dDEM(raster=dem_2009 - dem_1990, start_time=dem_1990.datetime, end_time=dem_2009.datetime)
-
-# The example DEMs are void-free, so let's make some random voids.
-# Introduce 50000 nans randomly throughout the dDEM.
-mask = np.zeros_like(ddem.data, dtype=bool)
-mask.ravel()[(np.random.choice(ddem.data.size, 50000, replace=False))] = True
-ddem.set_mask(mask)
-```
-
-### Linear spatial interpolation
-
-Linear spatial interpolation (also often called bilinear interpolation) of dDEMs is arguably the simplest approach: voids are filled by a an average of the surrounding pixels values, weighted by their distance to the void pixel.
-
-```{code-cell} ipython3
-ddem.interpolate(method="linear")
-```
-
-```{eval-rst}
-.. plot:: code/comparison_plot_spatial_interpolation.py
-
-```
-
-### Local hypsometric interpolation
-
-This approach assumes that there is a relationship between the elevation and the elevation change in the dDEM, which is often the case for glaciers.
-Elevation change gradients in late 1900s and 2000s on glaciers often have the signature of large melt in the lower parts, while the upper parts might be less negative, or even positive.
-This relationship is strongly correlated for a specific glacier, and weakly correlated on regional scales (see [Regional hypsometric interpolation]).
-With the local (glacier specific) hypsometric approach, elevation change gradients are estimated for each glacier separately.
-This is simply a linear or polynomial model estimated with the dDEM and a reference DEM.
-Then, voids are interpolated by replacing them with what "should be there" at that elevation, according to the model.
-
-```{code-cell} ipython3
-ddem.interpolate(method="local_hypsometric", reference_elevation=dem_2009, mask=glaciers_1990)
-```
-
-```{eval-rst}
-.. plot:: code/comparison_plot_local_hypsometric_interpolation.py
-
-```
-
-*Caption: The elevation dependent elevation change of Scott Turnerbreen on Svalbard from 1990--2009. The width of the bars indicate the standard deviation of the bin. The light blue background bars show the area distribution with elevation.*
-
-### Regional hypsometric interpolation
-
-Similarly to [Local hypsometric interpolation], the elevation change is assumed to be largely elevation-dependent.
-With the regional approach (often also called "global"), elevation change gradients are estimated for all glaciers in an entire region, instead of estimating one by one.
-This is advantageous in respect to areas where voids are frequent, as not even a single dDEM value has to exist on a glacier in order to reconstruct it.
-Of course, the accuracy of such an averaging is much lower than if the local hypsometric approach is used (assuming it is possible).
-
-```{code-cell} ipython3
-ddem.interpolate(method="regional_hypsometric", reference_elevation=dem_2009, mask=glaciers_1990)
-```
-
-```{eval-rst}
-.. plot:: code/comparison_plot_regional_hypsometric_interpolation.py
-
-```
-
-*Caption: The regional elevation dependent elevation change in central Svalbard from 1990--2009. The width of the bars indicate the standard deviation of the bin. The light blue background bars show the area distribution with elevation.*
-
-## The DEMCollection object
-
-Keeping track of multiple DEMs can be difficult when many different extents, resolutions and CRSs are involved, and {class}`xdem.demcollection.DEMCollection` is xDEM's answer to make this simple.
-We need metadata on the timing of these products.
-The DEMs can be provided with the `datetime=` argument upon instantiation, or the attribute could be set later.
-Multiple outlines are provided as a dictionary in the shape of `{datetime: outline}`.
-
-```{eval-rst}
-.. minigallery:: xdem.DEMCollection
- :add-heading:
-```
-
-[See here for the outline filtering syntax](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html).
diff --git a/doc/source/conf.py b/doc/source/conf.py
index 12578468..af0066c4 100644
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -17,6 +17,7 @@
sys.path.append(os.path.abspath("../.."))
sys.path.append(os.path.abspath("../../xdem/"))
sys.path.append(os.path.abspath(".."))
+sys.path.insert(0, os.path.dirname(__file__))
from sphinx_gallery.sorting import ExplicitOrder
@@ -25,8 +26,8 @@
# -- Project information -----------------------------------------------------
project = "xDEM"
-copyright = "2021, Erik Mannerfelt, Romain Hugonnet, Amaury Dehecq and others"
-author = "Erik Mannerfelt, Romain Hugonnet, Amaury Dehecq and others"
+copyright = "2020, GlacioHack"
+author = "xDEM contributors"
# The full version, including alpha/beta/rc tags
release = xdem.__version__
@@ -64,13 +65,14 @@
nb_kernel_rgx_aliases = {".*xdem.*": "python3"}
nb_execution_raise_on_error = True # To fail documentation build on notebook execution error
nb_execution_show_tb = True # To show full traceback on notebook execution error
+nb_output_stderr = "warn" # To warn if an error is raised in a notebook cell (if intended, override to "show" in cell)
# autosummary_generate = True
intersphinx_mapping = {
"python": ("https://docs.python.org/", None),
- "geoutils": ("https://geoutils.readthedocs.io/en/latest", None),
- "rasterio": ("https://rasterio.readthedocs.io/en/latest", None),
+ "geoutils": ("https://geoutils.readthedocs.io/en/stable", None),
+ "rasterio": ("https://rasterio.readthedocs.io/en/stable", None),
"numpy": ("https://numpy.org/doc/stable", None),
"matplotlib": ("https://matplotlib.org/stable", None),
"pyproj": ("https://pyproj4.github.io/pyproj/stable", None),
@@ -97,6 +99,11 @@
# os.path.join(os.path.dirname(__file__), "../", "../", "examples", "advanced")])
"remove_config_comments": True,
# To remove comments such as sphinx-gallery-thumbnail-number (only works in code, not in text)
+ "reset_modules": (
+ "matplotlib",
+ "sphinxext.reset_mpl",
+ ),
+ # To reset matplotlib for each gallery (and run custom function that fixes the default DPI)
}
extlinks = {
@@ -105,7 +112,7 @@
}
# For matplotlib figures generate with sphinx plot: (suffix, dpi)
-plot_formats = [(".png", 400)]
+plot_formats = [(".png", 600)]
# To avoid long path names in inheritance diagrams
inheritance_alias = {
@@ -166,10 +173,14 @@ def setup(app):
"notebook_interface": "jupyterlab",
# For launching Binder in Jupyterlab to open MD files as notebook (downloads them otherwise)
},
- "show_toc_level": 2, # To show more levels on the right sidebar TOC
+ "show_toc_level": 3, # To show more levels on the right sidebar TOC
"logo": {
"image_dark": "_static/xdem_logo_dark.svg",
},
+ "announcement": (
+ "⚠️ Our 0.1 release refactored several early-development functions for long-term stability, "
+ 'to update your code see here. ⚠️'
+ ),
}
# For dark mode
@@ -183,3 +194,7 @@ def setup(app):
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ["imgs", "_static"] # Commented out as we have no custom static data
+
+html_css_files = [
+ "css/custom.css",
+]
diff --git a/doc/source/config.md b/doc/source/config.md
new file mode 100644
index 00000000..7bb42533
--- /dev/null
+++ b/doc/source/config.md
@@ -0,0 +1,66 @@
+---
+file_format: mystnb
+jupytext:
+ formats: md:myst
+ text_representation:
+ extension: .md
+ format_name: myst
+kernelspec:
+ display_name: xdem-env
+ language: python
+ name: xdem
+---
+# Configuration
+
+xDEM allows to configure the **verbosity level** and the **default behaviour of certain operations on elevation data** (such as
+resampling method for reprojection, or pixel interpretation) directly at the package level.
+
+(verbosity)=
+## Verbosity level
+
+To configure the verbosity level (or logging) for xDEM, you can utilize Python's built-in `logging` module. This module
+has five levels of verbosity that are, in ascending order of severity: `DEBUG`, `INFO`, `WARNING`, `ERROR` and `CRITICAL`.
+Setting a level prints output from that level and all other of higher severity. Logging also allows you to specify other aspects,
+such as the destination of the output (console, file).
+
+```{important}
+**The default verbosity level is `WARNING`, implying that `INFO` and `DEBUG` do not get printed**. Use the basic configuration
+as below to setup an `INFO` level.
+```
+
+To specify the verbosity level, set up a logging configuration at the start of your script:
+
+```{code-cell} ipython3
+import logging
+
+# Basic configuration to simply print info
+logging.basicConfig(level=logging.INFO)
+```
+
+Optionally, you can specify the logging date, format, and handlers (destinations).
+
+```{code-cell} ipython3
+
+# More advanced configuration
+logging.basicConfig(
+ level=logging.INFO,
+ format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+ datefmt='%Y-%m-%d %H:%M:%S',
+ handlers=[
+ logging.FileHandler('app.log'), # Log messages will be saved to this file
+ logging.StreamHandler() # Log messages will also be printed to the console
+ ])
+```
+
+The above configuration will log messages with a severity level of `INFO` and above, including timestamps, logger names, and
+log levels in the output. You can change the logging level as needed.
+
+
+## Raster–vector–point operations
+
+To change the configuration at the package level regarding operations for rasters, vectors and points, see
+[GeoUtils' configuration](https://geoutils.readthedocs.io/en/stable/config.html).
+
+For instance, this allows to define a preferred resampling algorithm used when interpolating and reprojecting
+(e.g., bilinear, cubic), or the default behaviour linked to pixel interpretation during point–raster comparison.
+These changes will then apply to all your operations in xDEM, such as coregistration.
diff --git a/doc/source/coregistration.md b/doc/source/coregistration.md
index 687f2594..efc4e930 100644
--- a/doc/source/coregistration.md
+++ b/doc/source/coregistration.md
@@ -1,7 +1,7 @@
---
file_format: mystnb
mystnb:
- execution_timeout: 90
+ execution_timeout: 150
jupytext:
formats: md:myst
text_representation:
@@ -16,250 +16,473 @@ kernelspec:
# Coregistration
-Coregistration between DEMs correspond to aligning the digital elevation models in three dimensions.
+xDEM implements a wide range of **coregistration algorithms and pipelines for 3-dimensional alignment** from the
+peer-reviewed literature often tailored specifically to elevation data, aiming at correcting systematic elevation errors.
-Transformations that can be described by a 3-dimensional [affine](https://en.wikipedia.org/wiki/Affine_transformation) function are included in coregistration methods.
-Those transformations include for instance:
+Two categories of alignment are generally differentiated: 3D [affine transformations](https://en.wikipedia.org/wiki/Affine_transformation)
+described below, and other alignments that possibly rely on external variables, described in {ref}`biascorr`.
-- vertical and horizontal translations,
-- rotations, reflections,
-- scalings.
+Affine transformations can include vertical and horizontal translations, rotations and reflections, and scalings.
-## Quick use
+:::{admonition} More reading
+:class: tip
-Coregistrations are defined using either a single method or pipeline of {class}`~xdem.coreg.Coreg` methods, that are listed below.
+Coregistration heavily relies on the use of static surfaces, which you can read more about on the **{ref}`static-surfaces` guide page**.
-Performing the coregistration on a pair of DEM is done either by using {func}`xdem.DEM.coregister_3d` from the DEM that will be aligned, or
-by specifying the {func}`xdem.coreg.Coreg.fit` and {func}`xdem.coreg.Coreg.apply` steps, which allows array inputs and
-to apply the same fitted transformation to several objects (e.g., horizontal shift of both a stereo DEM and its ortho-image).
+:::
-## Introduction
+## Quick use
-Coregistration of a DEM is performed when it needs to be compared to a reference, but the DEM does not align with the reference perfectly.
-There are many reasons for why this might be, for example: poor georeferencing, unknown coordinate system transforms or vertical datums, and instrument- or processing-induced distortion.
+Coregistration pipelines are defined by combining {class}`~xdem.coreg.Coreg` objects:
-A main principle of all coregistration approaches is the assumption that all or parts of the portrayed terrain are unchanged between the reference and the DEM to be aligned.
-This *stable ground* can be extracted by masking out features that are assumed to be unstable.
-Then, the DEM to be aligned is translated, rotated and/or bent to fit the stable surfaces of the reference DEM as well as possible.
-In mountainous environments, unstable areas could be: glaciers, landslides, vegetation, dead-ice terrain and human structures.
-Unless the entire terrain is assumed to be stable, a mask layer is required.
+```{code-cell} ipython3
+:tags: [remove-cell]
-There are multiple approaches for coregistration, and each have their own strengths and weaknesses.
-Below is a summary of how each method works, and when it should (and should not) be used.
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 600
+pyplot.rcParams['savefig.dpi'] = 600
+pyplot.rcParams['font.size'] = 9 # Default 10 is a bit too big for coregistration plots
+```
-**Example data**
+```{code-cell} ipython3
+import xdem
-Examples are given using data close to Longyearbyen on Svalbard. These can be loaded as:
+# Create a coregistration pipeline
+my_coreg_pipeline = xdem.coreg.ICP() + xdem.coreg.NuthKaab()
+
+# Or use a single method
+my_coreg_pipeline = xdem.coreg.NuthKaab()
+```
+
+Then, coregistering a pair of elevation data can be done by calling {func}`xdem.DEM.coregister_3d` from the DEM that should be aligned.
```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for opening example files"
+: code_prompt_hide: "Hide the code for opening example files"
+
import geoutils as gu
import numpy as np
+import matplotlib.pyplot as plt
-import xdem
-
-# Open a reference DEM from 2009
+# Open a reference and to-be-aligned DEM
ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
-# Open a to-be-aligned DEM from 1990
-tba_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem")).reproject(ref_dem, silent=True)
+tba_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
+```
-# Open glacier polygons from 1990, corresponding to unstable ground
-glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
-# Create an inlier mask of terrain outside the glacier polygons
-inlier_mask = glacier_outlines.create_mask(ref_dem)
+```{code-cell} ipython3
+# Coregister by calling the DEM method
+aligned_dem = tba_dem.coregister_3d(ref_dem, my_coreg_pipeline)
```
-(coreg_object)=
-## The {class}`~xdem.Coreg` object
+Alternatively, the coregistration can be applied by calling {func}`~xdem.coreg.Coreg.fit_and_apply`, or sequentially
+calling the {func}`~xdem.coreg.Coreg.fit` and {func}`~xdem.coreg.Coreg.apply` steps,
+which allows a broader variety of arguments at each step, and re-using the same transformation to several objects
+(e.g., horizontal shift of both a stereo DEM and its ortho-image).
+
+```{code-cell} ipython3
+# (Equivalent) Or use fit and apply
+aligned_dem = my_coreg_pipeline.fit_and_apply(ref_dem, tba_dem)
+```
-Each coregistration approach in xDEM inherits their interface from the {class}`~xdem.Coreg` class1.
+Information about the coregistration inputs and outputs is summarized in {func}`~xdem.coreg.Coreg.info`.
-```{margin}
-1In a style resembling [scikit-learn's pipelines](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn-linear-model-linearregression).
+```{tip}
+Often, an `inlier_mask` has to be passed to {func}`~xdem.coreg.Coreg.fit` to isolate static surfaces to utilize during coregistration (for instance removing vegetation, snow, glaciers). This mask can be easily derived using {func}`~geoutils.Vector.create_mask`.
```
-Each coregistration approach has the following methods:
+## Summary of supported methods
+
+```{list-table}
+ :widths: 1 1 1
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Affine method
+ - Description
+ - Reference
+ * - {ref}`nuthkaab`
+ - Horizontal and vertical translations
+ - [Nuth and Kääb (2011)](https://doi.org/10.5194/tc-5-271-2011)
+ * - {ref}`dh-minimize`
+ - Horizontal and vertical translations
+ - N/A
+ * - {ref}`icp`
+ - Translation and rotations
+ - [Besl and McKay (1992)](https://doi.org/10.1117/12.57955)
+ * - {ref}`vshift`
+ - Vertical translation
+ - N/A
+```
-- {func}`~xdem.Coreg.fit` for estimating the transform.
-- {func}`~xdem.Coreg.apply` for applying the transform to a DEM.
-- {func}`~xdem.Coreg.apply_pts` for applying the transform to a set of 3D points.
-- {func}`~xdem.Coreg.to_matrix()` to convert the transform to a 4x4 transformation matrix, if possible.
+## Using a coregistration
-First, {func}`~xdem.Coreg.fit()` is called to estimate the transform, and then this transform can be used or exported using the subsequent methods.
+(coreg_object)=
+### The {class}`~xdem.coreg.Coreg` object
+
+Each coregistration method implemented in xDEM inherits their interface from the {class}`~xdem.coreg.Coreg` class1, and has the following methods:
+- {func}`~xdem.coreg.Coreg.fit_and_apply` for estimating the transformation and applying it in one step,
+- {func}`~xdem.coreg.Coreg.info` for plotting the metadata, including inputs and outputs of the coregistration.
+
+For more details on the input and output metadata stored by coregistration methods, see the **{ref}`coreg-meta` section**.
+
+The two above methods cover most uses. More specific methods are also available:
+- {func}`~xdem.coreg.Coreg.fit` for estimating the transformation without applying it,
+- {func}`~xdem.coreg.Coreg.apply` for applying an estimated transformation,
+- {func}`~xdem.coreg.AffineCoreg.to_matrix` to convert the transform to a 4x4 transformation matrix, if possible,
+- {func}`~xdem.coreg.AffineCoreg.from_matrix` to create a coregistration from a 4x4 transformation matrix.
+
+```{margin}
+1In a style inspired by [scikit-learn's pipelines](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn-linear-model-linearregression).
+```
**Inheritance diagram of implemented coregistrations:**
```{eval-rst}
-.. inheritance-diagram:: xdem.coreg.base xdem.coreg.affine xdem.coreg.biascorr
+.. inheritance-diagram:: xdem.coreg.base.Coreg xdem.coreg.affine xdem.coreg.biascorr
:top-classes: xdem.coreg.Coreg
```
-See {ref}`biascorr` for more information on non-rigid transformations ("bias corrections").
+**See {ref}`biascorr`** for more information on non-rigid transformations.
+
+## Coregistration methods
-(coregistration-nuthkaab)=
+```{important}
+Below we **create misaligned elevation data to examplify the different methods** in relation to their type of affine transformation.
-## Nuth and Kääb (2011)
+See coregistration on real data in the **{ref}`examples-basic` and {ref}`examples-advanced` gallery examples**!
+```
+
+(nuthkaab)=
+### Nuth and Kääb (2011)
{class}`xdem.coreg.NuthKaab`
-- **Performs:** translation and vertical shift.
-- **Supports weights** (soon)
-- **Recommended for:** Noisy data with low rotational differences.
+- **Performs:** Horizontal and vertical shifts.
+- **Supports weights:** Planned.
+- **Pros:** Refines sub-pixel horizontal shifts accurately, with fast convergence.
+- **Cons:** Diverges on flat terrain, as landforms are required to constrain the fit with aspect and slope.
-The Nuth and Kääb ([2011](https://doi.org/10.5194/tc-5-271-2011)) coregistration approach is named after the paper that first implemented it.
-It estimates translation iteratively by solving a cosine equation to model the direction at which the DEM is most likely offset.
-First, the DEMs are compared to get a dDEM, and slope/aspect maps are created from the reference DEM.
-Together, these three products contain the information about in which direction the offset is.
-A cosine function is solved using these products to find the most probable offset direction, and an appropriate horizontal shift is applied to fix it.
-This is an iterative process, and cosine functions with suggested shifts are applied in a loop, continuously refining the total offset.
-The loop stops either when the maximum iteration limit is reached, or when the NMAD between the two products stops improving significantly.
+The [Nuth and Kääb (2011)](https://doi.org/10.5194/tc-5-271-2011) coregistration approach estimates a horizontal
+translation iteratively by solving a cosine equation between the terrain slope, aspect and the elevation differences.
+The iteration stops if it reaches the maximum number of iteration limit, or if the iterative shift amplitude falls
+below a specified tolerance.
-```{eval-rst}
-.. plot:: code/coregistration_plot_nuth_kaab.py
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for adding a horizontal and vertical shift"
+: code_prompt_hide: "Hide the code for adding a horizontal and vertical shift"
+
+x_shift = 30
+y_shift = 30
+z_shift = 10
+# Affine matrix for 3D transformation
+matrix = np.array(
+ [
+ [1, 0, 0, x_shift],
+ [0, 1, 0, y_shift],
+ [0, 0, 1, z_shift],
+ [0, 0, 0, 1],
+ ]
+)
+# We create misaligned elevation data
+tba_dem_shifted = xdem.coreg.apply_matrix(ref_dem, matrix)
```
-*Caption: Demonstration of the Nuth and Kääb (2011) approach from Svalbard. Note that large improvements are seen, but nonlinear offsets still exist. The NMAD is calculated from the off-glacier surfaces.*
+```{code-cell} ipython3
+# Define a coregistration based on the Nuth and Kääb (2011) method
+nuth_kaab = xdem.coreg.NuthKaab()
+# Fit to data and apply
+aligned_dem = nuth_kaab.fit_and_apply(ref_dem, tba_dem_shifted)
+```
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before NK")
+(tba_dem_shifted - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[0])
+ax[1].set_title("After NK")
+(aligned_dem - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+plt.tight_layout()
+```
-### Limitations
+(dh-minimize)=
+### Minimization of dh
-The Nuth and Kääb (2011) coregistration approach does not take rotation into account.
-Rotational corrections are often needed on for example satellite derived DEMs, so a complementary tool is required for a perfect fit.
-1st or higher degree [Deramping] can be used for small rotational corrections.
-For large rotations, the Nuth and Kääb (2011) approach will not work properly, and [ICP] is recommended instead.
+{class}`xdem.coreg.DhMinimize`
-### Example
+- **Performs:** Horizontal and vertical shifts.
+- **Supports weights:** Planned.
+- **Pros:** Can be used to perform both local and global registration by tuning the minimizer.
+- **Cons:** Sensitive to noise.
-```{code-cell} ipython3
-from xdem import coreg
+The minimization of elevation differences (or dh) coregistration approach estimates a horizontal
+translation by minimizing any statistic on the elevation differences, typically their spread (defaults to the NMAD).
-nuth_kaab = coreg.NuthKaab()
-# Fit the data to a suitable x/y/z offset.
-nuth_kaab.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)
-# Apply the transformation to the data (or any other data)
-aligned_dem = nuth_kaab.apply(tba_dem)
+```{code-cell} ipython3
+# Define a coregistration based on the dh minimization approach
+dh_minimize = xdem.coreg.DhMinimize()
+# Fit to data and apply
+aligned_dem = dh_minimize.fit_and_apply(ref_dem, tba_dem_shifted)
```
-```{eval-rst}
-.. minigallery:: xdem.coreg.NuthKaab
- :add-heading:
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before dh\nminimize")
+(tba_dem_shifted - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[0])
+ax[1].set_title("After dh\nminimize")
+(aligned_dem - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+plt.tight_layout()
```
-## Vertical shift
+(vshift)=
+### Vertical shift
{class}`xdem.coreg.VerticalShift`
-- **Performs:** (Weighted) Vertical shift using the mean, median or anything else
-- **Supports weights** (soon)
-- **Recommended for:** A precursor step to e.g. ICP.
+- **Performs:** Vertical shifting using any custom function (mean, median, percentile).
+- **Supports weights:** Planned.
+- **Pros:** Useful to have as independent step to refine vertical alignment precisely as it is the most sensitive to outliers, by refining inliers and the central estimate function.
+- **Cons**: Always needs to be combined with another approach.
-``VerticalShift`` has very similar functionality to the z-component of `Nuth and Kääb (2011)`_.
-This function is more customizable, for example allowing changing of the vertical shift algorithm (from weighted average to e.g. median).
-It should also be faster, since it is a single function call.
+The vertical shift coregistration is simply a shift based on an estimate of the mean elevation differences with customizable arguments.
-### Limitations
-Only performs vertical corrections, so it should be combined with another approach.
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for adding a vertical shift"
+: code_prompt_hide: "Hide the code for adding a vertical shift"
-### Example
+# Apply a vertical shift of 10 meters
+tba_dem_vshifted = ref_dem + 10
+```
```{code-cell} ipython3
-vshift = coreg.VerticalShift()
-# Note that the transform argument is not needed, since it is a simple vertical correction.
-vshift.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)
-
-# Apply the vertical shift to a DEM
-shifted_dem = vshift.apply(tba_dem)
+# Define a coregistration object based on a vertical shift correction
+vshift = xdem.coreg.VerticalShift(vshift_reduc_func=np.median)
+# Fit and apply
+aligned_dem = vshift.fit_and_apply(ref_dem, tba_dem_vshifted)
+```
-# Use median shift instead
-vshift_median = coreg.VerticalShift(vshift_reduc_func=np.median)
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before vertical\nshift")
+(tba_dem_vshifted - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[0])
+ax[1].set_title("After vertical\nshift")
+(aligned_dem - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+plt.tight_layout()
```
-## ICP
+(icp)=
+### Iterative closest point
{class}`xdem.coreg.ICP`
-- **Performs:** Rigid transform correction (translation + rotation).
-- **Does not support weights**
-- **Recommended for:** Data with low noise and a high relative rotation.
-
-Iterative Closest Point (ICP) coregistration, which is based on [Besl and McKay (1992)](https://doi.org/10.1117/12.57955), works by iteratively moving the data until it fits the reference as well as possible.
-The DEMs are read as point clouds; collections of points with X/Y/Z coordinates, and a nearest neighbour analysis is made between the reference and the data to be aligned.
-After the distances are calculated, a rigid transform is estimated to minimise them.
-The transform is attempted, and then distances calculated again.
-If the distance is lowered, another rigid transform is estimated, and this is continued in a loop.
-The loop stops if it reaches the max iteration limit or if the distances do not improve significantly between iterations.
-The opencv implementation of ICP includes outlier removal, since extreme outliers will heavily interfere with the nearest neighbour distances.
-This may improve results on noisy data significantly, but care should still be taken, as the risk of landing in [local minima](https://en.wikipedia.org/wiki/Maxima_and_minima) increases.
+- **Performs:** Rigid transform transformation (3D translation + 3D rotation).
+- **Does not support weights.**
+- **Pros:** Efficient at estimating rotation and shifts simultaneously.
+- **Cons:** Poor sub-pixel accuracy for horizontal shifts, sensitive to outliers, and runs slowly with large samples.
-### Limitations
+Iterative Closest Point (ICP) coregistration is an iterative point cloud registration method from [Besl and McKay (1992)](https://doi.org/10.1117/12.57955). It aims at iteratively minimizing the distance between closest neighbours by applying sequential rigid transformations. If DEMs are used as inputs, they are converted to point clouds.
+As for Nuth and Kääb (2011), the iteration stops if it reaches the maximum number of iteration limit or if the iterative transformation amplitude falls below a specified tolerance.
-ICP often works poorly on noisy data.
-The outlier removal functionality of the opencv implementation is a step in the right direction, but it still does not compete with other coregistration approaches when the relative rotation is small.
-In cases of high rotation, ICP is the only approach that can account for this properly, but results may need refinement, for example with the [Nuth and Kääb (2011)] approach.
-
-Due to the repeated nearest neighbour calculations, ICP is often the slowest coregistration approach out of the alternatives.
-
-### Example
+ICP is currently based on [OpenCV's implementation](https://docs.opencv.org/4.x/dc/d9b/classcv_1_1ppf__match__3d_1_1ICP.html) (an optional dependency), which includes outlier removal arguments. This may improve results significantly on outlier-prone data, but care should still be taken, as the risk of landing in [local minima](https://en.wikipedia.org/wiki/Maxima_and_minima) increases.
```{code-cell} ipython3
-# Instantiate the object with default parameters
-icp = coreg.ICP()
-# Fit the data to a suitable transformation.
-icp.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for adding a shift and rotation"
+: code_prompt_hide: "Hide the code for adding a shift and rotation"
+
+# Apply a rotation of 0.2 degrees in X, 0.1 in Y and 0 in Z
+e = np.deg2rad([0.2, 0.1, 0])
+# Add X/Y/Z shifts in meters
+shifts = np.array([10, 20, 5])
+# Affine matrix for 3D transformation
+import pytransform3d
+matrix_rot = pytransform3d.rotations.matrix_from_euler(e, i=0, j=1, k=2, extrinsic=True)
+matrix = np.diag(np.ones(4, dtype=float))
+matrix[:3, :3] = matrix_rot
+matrix[:3, 3] = shifts
+
+centroid = [ref_dem.bounds.left + 5000, ref_dem.bounds.top - 2000, np.median(ref_dem)]
+# We create misaligned elevation data
+tba_dem_shifted_rotated = xdem.coreg.apply_matrix(ref_dem, matrix, centroid=centroid)
+```
-# Apply the transformation matrix to the data (or any other data)
-aligned_dem = icp.apply(tba_dem)
+```{code-cell} ipython3
+# Define a coregistration based on ICP
+icp = xdem.coreg.ICP()
+# Fit to data and apply
+aligned_dem = icp.fit_and_apply(ref_dem, tba_dem_shifted_rotated)
```
-```{eval-rst}
-.. minigallery:: xdem.coreg.ICP
- :add-heading:
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before ICP")
+(tba_dem_shifted_rotated - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[0])
+ax[1].set_title("After ICP")
+(aligned_dem - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+plt.tight_layout()
```
-## The CoregPipeline object
+## Building coregistration pipelines
-{class}`xdem.coreg.CoregPipeline`
+### The {class}`~xdem.coreg.CoregPipeline` object
-Often, more than one coregistration approach is necessary to obtain the best results.
-For example, ICP works poorly with large initial biases, so a `CoregPipeline` can be constructed to perform both sequentially:
+Often, more than one coregistration approach is necessary to obtain the best results, and several need to be combined
+sequentially. A {class}`~xdem.coreg.CoregPipeline` can be constructed for this:
```{code-cell} ipython3
-pipeline = coreg.CoregPipeline([coreg.BiasCorr(), coreg.ICP()])
-
-# pipeline.fit(... # etc.
+# We can list sequential coregistration methods to apply
+pipeline = xdem.coreg.CoregPipeline([xdem.coreg.ICP(), xdem.coreg.NuthKaab()])
-# This works identically to the syntax above
-pipeline2 = coreg.BiasCorr() + coreg.ICP()
+# Or sum them, which works identically as the syntax above
+pipeline = xdem.coreg.ICP() + xdem.coreg.NuthKaab()
```
-The `CoregPipeline` object exposes the same interface as the `Coreg` object.
-The results of a pipeline can be used in other programs by exporting the combined transformation matrix using {func}`xdem.coreg.CoregPipeline.to_matrix`.
+The {class}`~xdem.coreg.CoregPipeline` object exposes the same interface as the {class}`~xdem.coreg.Coreg` object.
+The results of a pipeline can be used in other programs by exporting the combined transformation matrix using {func}`~xdem.coreg.Coreg.to_matrix`.
-This class is heavily inspired by the [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn-pipeline-pipeline) and [make_pipeline()](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.make_pipeline.html#sklearn.pipeline.make_pipeline) functionalities in `scikit-learn`.
+```{margin}
+2Here again, this class is heavily inspired by SciKit-Learn's [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn-pipeline-pipeline) and [make_pipeline()](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.make_pipeline.html#sklearn.pipeline.make_pipeline) functionalities.
+```
-```{eval-rst}
-.. minigallery:: xdem.coreg.CoregPipeline
- :add-heading:
+```{code-cell} ipython3
+# Fit to data and apply the pipeline of ICP + Nuth and Kääb
+aligned_dem = pipeline.fit_and_apply(ref_dem, tba_dem_shifted_rotated)
```
-### Suggested pipelines
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before ICP + NK")
+(tba_dem_shifted_rotated - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[0])
+ax[1].set_title("After ICP + NK")
+(aligned_dem - ref_dem).plot(cmap='RdYlBu', vmin=-30, vmax=30, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+plt.tight_layout()
+```
-For sub-pixel accuracy, the [Nuth and Kääb (2011)] approach should almost always be used.
+### Recommended pipelines
+
+To ensure sub-pixel accuracy, the [Nuth and Kääb (2011)](https://doi.org/10.5194/tc-5-271-2011) coregistration should almost always be used as a final step.
The approach does not account for rotations in the dataset, however, so a combination is often necessary.
-For small rotations, a 1st degree deramp could be used:
+For small rotations, a 1st degree deramp can be used in combination:
```{code-cell} ipython3
-coreg.NuthKaab() + coreg.Deramp(poly_order=1)
+pipeline = xdem.coreg.NuthKaab() + xdem.coreg.Deramp(poly_order=1)
```
-For larger rotations, ICP is the only reliable approach (but does not outperform in sub-pixel accuracy):
+For larger rotations, ICP can be used instead:
```{code-cell} ipython3
-coreg.ICP() + coreg.NuthKaab()
+pipeline = xdem.coreg.ICP() + xdem.coreg.NuthKaab()
```
-For large shifts, rotations and high amounts of noise:
+Additionally, ICP tends to fail with large initial vertical differences, so a preliminary vertical shifting can be used:
```{code-cell} ipython3
-coreg.BiasCorr() + coreg.ICP() + coreg.NuthKaab()
+pipeline = xdem.coreg.VerticalShift() + xdem.coreg.ICP() + xdem.coreg.NuthKaab()
```
+
+(coreg-meta)=
+## Coregistration metadata
+
+The metadata surrounding a coregistration method, which can be displayed by {func}`~xdem.coreg.Coreg.info`, is stored in
+the {attr}`~xdem.coreg.Coreg.meta` nested dictionary.
+This metadata is divided into **inputs** and **outputs**. Input metadata corresponds to what arguments are
+used when initializing a {class}`~xdem.coreg.Coreg` object, while output metadata are created during the call to
+{func}`~xdem.coreg.Coreg.fit`. Together, they allow to apply the transformation during the
+{func}`~xdem.coreg.Coreg.apply` step of the coregistration.
+
+```{code-cell} ipython3
+# Example of metadata info after fitting
+my_coreg_pipeline.info()
+```
+
+For both **inputs** and **outputs**, four consistent categories of metadata are defined.
+
+**Note:** Some metadata, such as the parameters `fit_or_bin` and `fit_func` described below, are pre-defined for affine coregistration methods and cannot be modified. They only take user-specified value for {ref}`biascorr`.
+
+**1. Randomization metadata (common to all)**:
+
+- An input `subsample` to define the subsample size of valid data to use in all methods (recommended for performance),
+- An input `random_state` to define the random seed for reproducibility of the subsampling (and potentially other random aspects such as optimizers),
+- An output `subsample_final` that stores the final subsample size used, which can be smaller than requested depending on the amount of valid data intersecting the two elevation datasets.
+
+**2. Fitting and binning metadata (common to nearly all methods)**:
+
+- An input `fit_or_bin` to either fit a parametric model by passing **"fit"**, perform an empirical binning by passing **"bin"**, or to fit a parametric model to the binning with **"bin_and_fit" (only "fit" or "bin_and_fit" possible for affine methods)**,
+- An input `fit_func` to pass any parametric function to fit to the bias **(pre-defined for affine methods)**,
+- An input `fit_optimizer` to pass any optimizer function to perform the fit minimization,
+- An input `bin_sizes` to pass the size or edges of the bins for each variable,
+- An input `bin_statistic` to pass the statistic to compute in each bin,
+- An input `bin_apply_method` to pass the method to apply the binning for correction,
+- An output `fit_params` that stores the optimized parameters for `fit_func`,
+- An output `fit_perr` that stores the error of optimized parameters (only for default `fit_optimizer`),
+- An output `bin_dataframe` that stores the dataframe of binned statistics.
+
+**3. Iteration metadata (common to all iterative methods)**:
+
+- An input `max_iterations` to define the maximum number of iterations at which to stop the method,
+- An input `tolerance` to define the tolerance at which to stop iterations (tolerance unit defined in method description),
+- An output `last_iteration` that stores the last iteration of the method,
+- An output `all_tolerances` that stores the tolerances computed at each iteration.
+
+**4. Affine metadata (common to all affine methods)**:
+
+- An output `matrix` that stores the estimated affine matrix,
+- An output `centroid` that stores the centroid coordinates with which to apply the affine transformation,
+- Outputs `shift_x`, `shift_y` and `shift_z` that store the easting, northing and vertical offsets, respectively.
+
+```{tip}
+In xDEM, you can extract the translations and rotations of an affine matrix using {class}`xdem.coreg.AffineCoreg.to_translations` and
+{class}`xdem.coreg.AffineCoreg.to_rotations`.
+
+To further manipulate affine matrices, see the [documentation of pytransform3d](https://dfki-ric.github.io/pytransform3d/rotations.html).
+```
+
+**5. Specific metadata (only for certain methods)**:
+
+These metadata are only inputs specific to a given method, outlined in the method description.
+
+For instance, for {class}`xdem.coreg.Deramp`, an input `poly_order` to define the polynomial order used for the fit, and
+for {class}`xdem.coreg.DirectionalBias`, an input `angle` to define the angle at which to do the directional correction.
diff --git a/doc/source/dem_class.md b/doc/source/dem_class.md
index 6d2bf07b..3abb1c77 100644
--- a/doc/source/dem_class.md
+++ b/doc/source/dem_class.md
@@ -36,6 +36,14 @@ The complete list of {class}`~geoutils.Raster` attributes and methods can be fou
A {class}`~xdem.DEM` is opened by instantiating with either a {class}`str`, a {class}`pathlib.Path`, a {class}`rasterio.io.DatasetReader` or a
{class}`rasterio.io.MemoryFile`, as for a {class}`~geoutils.Raster`.
+```{code-cell} ipython3
+:tags: [remove-cell]
+
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 400
+pyplot.rcParams['savefig.dpi'] = 400
+```
```{code-cell} ipython3
import xdem
@@ -81,9 +89,12 @@ import geoutils as gu
fn_glacier_outlines = xdem.examples.get_path("longyearbyen_glacier_outlines")
vect_gla = gu.Vector(fn_glacier_outlines)
+# Crop outlines to those intersecting the DEM
+vect_gla = vect_gla.crop(dem)
+
# Plot the DEM and the vector file
dem.plot(cmap="terrain", cbar_title="Elevation (m)")
-vect_gla.plot(dem) # We pass the DEM as reference for the plot CRS/extent
+vect_gla.plot(dem, ec="k", fc="none") # We pass the DEM as reference for the plot CRS
```
## Vertical referencing
@@ -118,7 +129,7 @@ by calling the function corresponding to the attribute name such as {func}`~xdem
```{code-cell} ipython3
# Derive slope using the Zevenberg and Thorne (1987) method
slope = dem.slope(method="ZevenbergThorne")
-slope.plot(cmap="Reds", cbar_title="Slope (degrees)")
+slope.plot(cmap="Reds", cbar_title="Slope (°)")
```
```{note}
@@ -140,7 +151,7 @@ dem_tba = xdem.DEM(filename_tba)
dem_tba_coreg = dem_tba.coregister_3d(dem)
# Plot the elevation change of the DEM due to coregistration
-dh_tba = dem_tba - dem_tba_coreg.reproject(dem_tba)
+dh_tba = dem_tba - dem_tba_coreg.reproject(dem_tba, silent=True)
dh_tba.plot(cmap="Spectral", cbar_title="Elevation change due to coreg (m)")
```
@@ -157,10 +168,17 @@ stable terrain as a proxy.
```{code-cell} ipython3
# Estimate elevation uncertainty assuming both DEMs have similar precision
-sig_dem, rho_sig = dem.estimate_uncertainty(dem_tba_coreg, precision_of_other="same")
+sig_dem, rho_sig = dem.estimate_uncertainty(dem_tba_coreg, precision_of_other="same", random_state=42)
# The error map variability is estimated from slope and curvature by default
-sig_dem.plot(cmap="Purples", cbar_title=r"Error in elevation (1$\sigma$, m)")
+sig_dem.plot(cmap="Purples", cbar_title=r"Random error in elevation (1$\sigma$, m)")
# The spatial correlation function represents how much errors are correlated at a certain distance
-rho_sig(1000) # Correlation at 1 km
+print("Elevation errors at a distance of 1 km are correlated at {:.2f} %.".format(rho_sig(1000) * 100))
+```
+
+```{note}
+We use `random_state` to ensure a fixed randomized output. It is **only necessary if you need your results to be exactly reproductible**.
+
+For more details on quantifying random and structured errors, see the {ref}`uncertainty` page.
+```
diff --git a/doc/source/ecosystem.md b/doc/source/ecosystem.md
new file mode 100644
index 00000000..44a3391e
--- /dev/null
+++ b/doc/source/ecosystem.md
@@ -0,0 +1,37 @@
+(ecosystem)=
+
+# Ecosystem
+
+xDEM is but a single tool among a large landscape of open tools for geospatial elevation analysis! Below is a list of
+other **tools that you might find useful to combine with xDEM**, in particular for retrieving elevation data or to perform complementary analysis.
+
+```{seealso}
+Tools listed below only relate to elevation data. To analyze georeferenced rasters, vectors and point cloud data,
+check out **xDEM's sister-package [GeoUtils](https://geoutils.readthedocs.io/)**.
+```
+## Python
+
+Great Python tools for **pre-processing and retrieving elevation data**:
+- [SlideRule](https://slideruleearth.io/) to pre-process and retrieve high-resolution elevation data in the cloud, including in particular [ICESat-2](https://icesat-2.gsfc.nasa.gov/) and [GEDI](https://gedi.umd.edu/),
+- [pDEMtools](https://pdemtools.readthedocs.io/en/latest/) to pre-process and retrieve [ArcticDEM](https://www.pgc.umn.edu/data/arcticdem/) and [REMA](https://www.pgc.umn.edu/data/rema/) high-resolution DEMs available in polar regions,
+- [icepyx](https://icepyx.readthedocs.io/en/latest/) to retrieve ICESat-2 data.
+
+Complementary Python tools to **analyze elevation data** are for instance:
+- [PDAL](https://pdal.io/en/latest/) for working with dense elevation point clouds,
+- [demcompare](https://demcompare.readthedocs.io/en/stable/) to compare two DEMs together,
+- [RichDEM](https://richdem.readthedocs.io/en/latest/) for in-depth terrain analysis, with a large range of method including many relevant to hydrology.
+
+## Julia
+
+If you are working in Julia, the [Geomorphometry](https://github.com/Deltares/Geomorphometry.jl) package provides a
+wide range of terrain analysis for elevation data.
+
+## R
+
+If you are working in R, the [MultiscaleDTM](https://ailich.github.io/MultiscaleDTM/) package provides modular tools
+for terrain analysis at multiple scales!
+
+## Other community resources
+
+Whether to retrieve data among their wide range of open datasets, or to dive into their other resources, be sure to check out the
+amazing [OpenTopography](https://opentopography.org/) and [OpenAltimetry](https://openaltimetry.earthdatacloud.nasa.gov/data/) efforts!
diff --git a/doc/source/elevation_intricacies.md b/doc/source/elevation_intricacies.md
new file mode 100644
index 00000000..511972f8
--- /dev/null
+++ b/doc/source/elevation_intricacies.md
@@ -0,0 +1,91 @@
+(elevation-intricacies)=
+# Georeferencing intricacies
+
+Georeferenced elevation data comes in different types and relies on different attributes than typical georeferenced
+data, which **can make quantitative analysis more delicate**.
+
+Below, we summarize these aspects to help grasp a general understanding of the intricacies of elevation data.
+
+## Types of elevation data
+
+There are a number of types of elevation data:
+
+1. **Digital elevation models (DEMs)** are a type of raster, i.e. defined on a regular grid with each grid cell representing an elevation. Additionally, DEMs are rasters than are almost always chosen to be single-band, and floating-type to accurately represent elevation in meters,
+2. **Elevation point clouds** are simple point clouds with XYZ coordinates, often with a list of attributes attached to each point,
+3. **Contour or breaklines** are 2D or 3D vector lines representing elevation. **They are usually not used for analysis**, instead for visualisation,
+4. **Elevation triangle irregular networks (TINs)** are a triangle mesh representing the continuous elevation surface. **They are usually not used for analysis**, instead in-memory for visualization or for conversion from an elevation point cloud to a DEM.
+
+```{note}
+xDEM supports the two elevation data types primarily used for quantitative analysis: the {class}`~xdem.DEM` and the elevation point cloud (currently as a {class}`~geopandas.GeoDataFrame` for some operations, more soon!).
+
+See the **{ref}`elevation-objects` features pages** for more details.
+```
+
+```{eval-rst}
+.. plot:: code/intricacies_datatypes.py
+ :width: 90%
+```
+
+
+Additionally, there are a critical differences for elevation point clouds depending on point density:
+- **Sparse elevation point clouds** (e.g., altimetry) are generally stored as small vector-type datasets (e.g., SHP). Due to their sparsity, for subsequent analysis, they are rarely gridded into a DEM, and instead compared with DEMs at the point cloud coordinates by interpolation of the DEM,
+- **Dense elevation point clouds** (e.g., lidar) are large datasets generally stored in specific formats (LAS). Due to their high density, they are often gridded into DEMs by triangular interpolation of the point cloud.
+
+```{note}
+For point–DEM interfacing, xDEM inherit functionalities from [GeoUtils's point–raster interfacing](https://geoutils.readthedocs.io/en/stable/raster_vector_point.html#rasterpoint-operations).
+See for instance {class}`xdem.DEM.interp_points`.
+```
+
+## A third georeferenced dimension
+
+Elevation data is unique among georeferenced data, in the sense that it **adds a third vertical dimension that also requires georeferencing**.
+
+For this purpose, elevation data is related to a vertical [coordinate reference system (CRS)](https://en.wikipedia.org/wiki/Spatial_reference_system). A vertical CRS is a **1D, often gravity-related, coordinate reference system of surface elevation** (or height), used to expand a 2D horizontal CRS to a 3D CRS.
+
+There are two types of models of surface elevation:
+- **Ellipsoids** model the surface of the Earth as a three-dimensional shape created from a two-dimensional ellipse, which are already used by 2D CRS,
+- **Geoids** model the surface of the Earth based on its gravity field (approximately mean sea-level). Since Earth's mass is not uniform, and the direction of gravity slightly changes, the shape of a geoid is irregular,
+
+which are directly associated with two types of 3D CRSs:
+- **Ellipsoidal heights** CRSs, that are simply 2D CRS promoted to 3D by explicitly adding an elevation axis to the ellipsoid used by the 2D CRS,
+- **Geoid heights** CRSs, that are a compound of a 2D CRS and a vertical CRS (2D + 1D), where the vertical CRS of the geoid is added relative to the ellipsoid.
+
+
+Problematically, until the early 2020s, **most elevation data was distributed without a 3D CRS in its metadata**. The vertical reference was generally provided separately, in a user guide or website of the data provider.
+Therefore, it is important to either define your vertical CRSs manually before any analysis, or double-check that all your datasets are on the same vertical reference.
+
+```{note}
+For this reason, xDEM includes {ref}`tools to easily set a vertical CRS`. See for instance {class}`xdem.DEM.set_vcrs`.
+```
+
+## The interpretation of pixel value for DEMs
+
+Among the elevation data types listed above, DEMs are the only gridded dataset. While gridded datasets have become
+ubiquitous for quantitative anaysis, they also suffer from a problem of pixel interpretation.
+
+Pixel interpretation describes how a grid cell value should be interpreted, and has two definitions:
+- **“Area” (the most common)** where the value represents a sampling over the region of the pixel (and typically refers to the upper-left corner coordinate), or
+- **“Point”** where the value relates to a point sample (and typically refers to the center of the pixel).
+
+**This interpretation difference disproportionally affects DEMs** as they are the primary type of gridded data associated with the least-common "Point" interpretation, and often rely on auxiliary point data such as ground-control points (GCPs).
+
+**In different software packages, gridded data are interpreted differently**, resulting in (undesirable) half-pixel shifts during analysis. Additionally, different storage formats have different standards for grid coordinate interpretation, also sometimes resulting in a half-pixel shift (e.g., GeoTIFF versus netCDF).
+
+```{note}
+To perform consistent pixel interpretation of DEMs, xDEM relies on [the raster pixel interpretation of GeoUtils, which mirrors GDAL's GCP behaviour](https://geoutils.readthedocs.io/en/stable/georeferencing.html#pixel-interpretation-only-for-rasters).
+
+This means that, by default, pixel interpretation induces a half-pixel shift during DEM–point interfacing for a “Point” interpretation, but only raises a warning for DEM–DEM operations if interpretations differ.
+This default behaviour can be modified at the package-level by using [GeoUtils’ configuration](https://geoutils.readthedocs.io/en/stable/config.html).
+
+See {class}`xdem.DEM.set_area_or_point` to re-set the pixel interpretation of your DEM.
+```
+
+----------------
+
+:::{admonition} References and more reading
+:class: tip
+
+For more information about **vertical referencing**, see [educational material from the National Geodetic Survey](https://geodesy.noaa.gov/datums/index.shtml) and [NOAA's VDatum tutorials](https://vdatum.noaa.gov/docs/datums.html).
+
+For more information about **pixel interpretation**, see [GIS StackExchange discussions](https://gis.stackexchange.com/questions/122670/is-there-a-standard-for-the-coordinates-of-pixels-in-georeferenced-rasters) and [GeoTIFF standard from the Open Geospatial Consortium](https://docs.ogc.org/is/19-008r4/19-008r4.html#_requirements_class_gtrastertypegeokey).
+:::
diff --git a/doc/source/elevation_point_cloud.md b/doc/source/elevation_point_cloud.md
index 21b73504..636f6232 100644
--- a/doc/source/elevation_point_cloud.md
+++ b/doc/source/elevation_point_cloud.md
@@ -14,4 +14,7 @@ kernelspec:
# The elevation point cloud ({class}`~xdem.EPC`)
-In construction, planned for 2024.
+In construction, planned for 2025.
+
+However, **elevation point clouds are already supported for coregistration and bias correction** by passing a {class}`geopandas.GeoDataFrame`
+associated to an elevation column name argument `z_name` to {func}`~xdem.coreg.Coreg.fit_and_apply`.
diff --git a/doc/source/gapfill.md b/doc/source/gapfill.md
new file mode 100644
index 00000000..46bfba04
--- /dev/null
+++ b/doc/source/gapfill.md
@@ -0,0 +1,271 @@
+---
+file_format: mystnb
+jupytext:
+ formats: md:myst
+ text_representation:
+ extension: .md
+ format_name: myst
+kernelspec:
+ display_name: xdem-env
+ language: python
+ name: xdem
+---
+(interpolation)=
+# Gap-filling
+
+xDEM contains routines to gap-fill elevation data or elevation differences depending on the type of terrain.
+
+```{important}
+Most of the approaches below are application-specific (e.g., glaciers) and might be moved to a separate package
+in future releases.
+```
+
+So far, xDEM has three types of gap-filling methods:
+
+- Inverse-distance weighting interpolation,
+- Local hypsometric interpolation (only relevant for elevation differences and glacier applications),
+- Regional hypsometric interpolation (also for glaciers).
+
+The last two methods are described in [McNabb et al. (2019)](https://doi.org/10.5194/tc-13-895-2019).
+
+```{code-cell} ipython3
+:tags: [remove-cell]
+
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 600
+pyplot.rcParams['savefig.dpi'] = 600
+pyplot.rcParams['font.size'] = 9 # Default 10 is a bit too big for coregistration plots
+```
+
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for opening example data"
+: code_prompt_hide: "Hide the code for opening example data"
+
+from datetime import datetime
+
+import geoutils as gu
+import numpy as np
+import matplotlib.pyplot as plt
+
+import xdem
+
+# Load a reference DEM from 2009
+dem_2009 = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"), datetime=datetime(2009, 8, 1))
+# Load a DEM from 1990
+dem_1990 = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"), datetime=datetime(1990, 8, 1))
+# Load glacier outlines from 1990.
+glaciers_1990 = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
+glaciers_2010 = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines_2010"))
+
+# Make a dictionary of glacier outlines where the key represents the associated date.
+outlines = {
+ datetime(1990, 8, 1): glaciers_1990,
+ datetime(2009, 8, 1): glaciers_2010,
+}
+
+# Cropping DEMs to a smaller extent to visualize the gap-filling better
+bounds = (dem_2009.bounds.left, dem_2009.bounds.bottom,
+ dem_2009.bounds.left + 200 * dem_2009.res[0], dem_2009.bounds.bottom + 150 * dem_2009.res[1])
+dem_2009 = dem_2009.crop(bounds)
+dem_1990 = dem_1990.crop(bounds)
+```
+
+We create a difference of DEMs object {class}`xdem.ddem.dDEM` to experiment on:
+
+```{code-cell} ipython3
+ddem = xdem.dDEM(raster=dem_2009 - dem_1990, start_time=dem_1990.datetime, end_time=dem_2009.datetime)
+
+# The example DEMs are void-free, so let's make some random voids.
+# Introduce a fifth of nans randomly throughout the dDEM.
+mask = np.zeros_like(ddem.data, dtype=bool)
+mask.ravel()[(np.random.choice(ddem.data.size, int(ddem.data.size/5), replace=False))] = True
+ddem.set_mask(mask)
+```
+
+## Inverse-distance weighting interpolation
+
+Inverse-distance weighting (IDW) interpolation of elevation differences is arguably the simplest approach: voids are filled by a weighted-mean of the surrounding pixels values, with weight inversely proportional to their distance to the void pixel.
+
+```{code-cell} ipython3
+ddem_idw = ddem.interpolate(method="idw")
+```
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+ddem_idw = ddem.copy(new_array=ddem_idw)
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before IDW\ngap-filling")
+ddem.plot(cmap='RdYlBu', vmin=-20, vmax=20, ax=ax[0])
+ax[1].set_title("After IDW\ngap-filling")
+ddem_idw.plot(cmap='RdYlBu', vmin=-20, vmax=20, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+```
+
+## Local hypsometric interpolation
+
+This approach assumes that there is a relationship between the elevation and the elevation change in the dDEM, which is often the case for glaciers.
+Elevation change gradients in late 1900s and 2000s on glaciers often have the signature of large thinning in the lower parts, while the upper parts might be less negative, or even positive.
+This relationship is strongly correlated for a specific glacier, and weakly correlated on regional scales.
+With the local (glacier specific) hypsometric approach, elevation change gradients are estimated for each glacier separately.
+This is simply a linear or polynomial model estimated with the dDEM and a reference DEM.
+Then, voids are interpolated by replacing them with what "should be there" at that elevation, according to the model.
+
+```{code-cell} ipython3
+ddem_localhyps = ddem.interpolate(method="local_hypsometric", reference_elevation=dem_2009, mask=glaciers_1990)
+```
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+ddem_localhyps = ddem.copy(new_array=ddem_localhyps)
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before local\nhypsometric\ngap-filling")
+ddem.plot(cmap='RdYlBu', vmin=-20, vmax=20, ax=ax[0])
+ax[1].set_title("After local\nhypsometric\ngap-filling")
+ddem_localhyps.plot(cmap='RdYlBu', vmin=-20, vmax=20, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+```
+
+Where the binning can be visualized as such:
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code for hypsometric binning"
+: code_prompt_hide: "Hide code for hypsometric binning"
+
+dem_2009 = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
+dem_1990 = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
+outlines_1990 = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
+
+ddem = xdem.dDEM(dem_2009 - dem_1990, start_time=np.datetime64("1990-08-01"), end_time=np.datetime64("2009-08-01"))
+
+ddem.data /= 2009 - 1990
+
+scott_1990 = outlines_1990.query("NAME == 'Scott Turnerbreen'")
+mask = scott_1990.create_mask(ddem)
+
+ddem_bins = xdem.volume.hypsometric_binning(ddem[mask], dem_2009[mask])
+stds = xdem.volume.hypsometric_binning(ddem[mask], dem_2009[mask], aggregation_function=np.std)
+
+plt.figure(figsize=(8, 8))
+plt.grid(zorder=0)
+plt.plot(ddem_bins["value"], ddem_bins.index.mid, linestyle="--", zorder=1)
+
+plt.barh(
+ y=ddem_bins.index.mid,
+ width=stds["value"],
+ left=ddem_bins["value"] - stds["value"] / 2,
+ height=(ddem_bins.index.left - ddem_bins.index.right) * 1,
+ zorder=2,
+ edgecolor="black",
+)
+for bin in ddem_bins.index:
+ plt.vlines(ddem_bins.loc[bin, "value"], bin.left, bin.right, color="black", zorder=3)
+
+plt.xlabel("Elevation change (m / a)")
+plt.twiny()
+plt.barh(
+ y=ddem_bins.index.mid,
+ width=ddem_bins["count"] / ddem_bins["count"].sum(),
+ left=0,
+ height=(ddem_bins.index.left - ddem_bins.index.right) * 1,
+ zorder=2,
+ alpha=0.2,
+)
+plt.xlabel("Normalized area distribution (hypsometry)")
+
+plt.ylabel("Elevation (m a.s.l.)")
+
+plt.tight_layout()
+plt.show()
+```
+
+*Caption: Hypsometric elevation change of Scott Turnerbreen on Svalbard from 1990--2009. The width of the bars indicate the standard deviation of the bin. The light blue background bars show the area distribution with elevation.*
+
+## Regional hypsometric interpolation
+
+Similarly to local hypsometric interpolation, the elevation change is assumed to be largely elevation-dependent.
+With the regional approach (often also called "global"), elevation change gradients are estimated for all glaciers in an entire region, instead of estimating one by one.
+
+```{code-cell} ipython3
+ddem.set_mask(mask)
+ddem_reghyps = ddem.interpolate(method="regional_hypsometric", reference_elevation=dem_2009, mask=glaciers_1990)
+```
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show plotting code"
+: code_prompt_hide: "Hide plotting code"
+
+ddem_reghyps = ddem.copy(new_array=ddem_reghyps)
+
+# Plot before and after
+f, ax = plt.subplots(1, 2)
+ax[0].set_title("Before regional\nhypsometric\ngap-filling")
+ddem.plot(cmap='RdYlBu', vmin=-10, vmax=10, ax=ax[0])
+ax[1].set_title("After regional\nhypsometric\ngap-filling")
+ddem_reghyps.plot(cmap='RdYlBu', vmin=-10, vmax=10, ax=ax[1], cbar_title="Elevation differences (m)")
+_ = ax[1].set_yticklabels([])
+```
+
+```{code-cell} ipython3
+:tags: [hide-input]
+:mystnb:
+: code_prompt_show: "Show code for hypsometric binning"
+: code_prompt_hide: "Hide code for hypsometric binning"
+
+mask = outlines_1990.create_mask(ddem)
+
+ddem_bins = xdem.volume.hypsometric_binning(ddem[mask], dem_2009[mask])
+stds = xdem.volume.hypsometric_binning(ddem[mask], dem_2009[mask], aggregation_function=np.std)
+
+plt.figure(figsize=(8, 8))
+plt.grid(zorder=0)
+
+
+plt.plot(ddem_bins["value"], ddem_bins.index.mid, linestyle="--", zorder=1)
+
+plt.barh(
+ y=ddem_bins.index.mid,
+ width=stds["value"],
+ left=ddem_bins["value"] - stds["value"] / 2,
+ height=(ddem_bins.index.left - ddem_bins.index.right) * 1,
+ zorder=2,
+ edgecolor="black",
+)
+for bin in ddem_bins.index:
+ plt.vlines(ddem_bins.loc[bin, "value"], bin.left, bin.right, color="black", zorder=3)
+
+plt.xlabel("Elevation change (m / a)")
+plt.twiny()
+plt.barh(
+ y=ddem_bins.index.mid,
+ width=ddem_bins["count"] / ddem_bins["count"].sum(),
+ left=0,
+ height=(ddem_bins.index.left - ddem_bins.index.right) * 1,
+ zorder=2,
+ alpha=0.2,
+)
+plt.xlabel("Normalized area distribution (hypsometry)")
+plt.ylabel("Elevation (m a.s.l.)")
+
+plt.tight_layout()
+plt.show()
+```
+*Caption: Regional hypsometric elevation change in central Svalbard from 1990--2009. The width of the bars indicate the standard deviation of the bin. The light blue background bars show the area distribution with elevation.*
diff --git a/doc/source/guides.md b/doc/source/guides.md
new file mode 100644
index 00000000..98f3c03a
--- /dev/null
+++ b/doc/source/guides.md
@@ -0,0 +1,15 @@
+(guides)=
+# Guides to elevated analysis
+
+This section is a collection of guides gathering background knowledge related to elevation data to help grasp how to best
+elevate your analysis!
+
+```{toctree}
+:maxdepth: 2
+
+elevation_intricacies
+static_surfaces
+accuracy_precision
+robust_estimators
+spatial_stats
+```
diff --git a/doc/source/how_to_install.md b/doc/source/how_to_install.md
index f55c16f3..d30b344f 100644
--- a/doc/source/how_to_install.md
+++ b/doc/source/how_to_install.md
@@ -8,7 +8,7 @@
mamba install -c conda-forge xdem
```
-```{important}
+```{tip}
Solving dependencies can take a long time with `conda`, `mamba` significantly speeds up the process. Install it with:
conda install mamba -n base -c conda-forge
@@ -16,18 +16,6 @@ Solving dependencies can take a long time with `conda`, `mamba` significantly sp
Once installed, the same commands can be run by simply replacing `conda` by `mamba`. More details available in the [mamba documentation](https://mamba.readthedocs.io/en/latest/).
```
-If running into the `sklearn` error `ImportError: dlopen: cannot load any more object with static TLS`, your system
-needs to update its `glibc` (see details [here](https://github.com/scikit-learn/scikit-learn/issues/14485#issuecomment-822678559)).
-If you have no administrator right on the system, you might be able to circumvent this issue by installing a working
-environment with specific downgraded versions of `scikit-learn` and `numpy`:
-
-```bash
-mamba create -n xdem-env -c conda-forge xdem scikit-learn==0.20.3 numpy==1.19.*
-```
-
-On very old systems, if the above install results in segmentation faults, try setting more specifically
-`numpy==1.19.2=py37h54aff64_0` (works with Debian 8.11, GLIBC 2.19).
-
## Installing with ``pip``
```bash
@@ -45,4 +33,4 @@ git clone https://github.com/GlacioHack/xdem.git
mamba env create -f xdem/dev-environment.yml
```
-After installing, you can check that everything is working by running the tests: `pytest -rA`.
+After installing, you can check that everything is working by running the tests: `pytest`.
diff --git a/doc/source/imgs/accuracy_precision_dem.png b/doc/source/imgs/accuracy_precision_dem.png
new file mode 100644
index 00000000..657fc946
Binary files /dev/null and b/doc/source/imgs/accuracy_precision_dem.png differ
diff --git a/doc/source/imgs/stable_terrain_diagram.png b/doc/source/imgs/stable_terrain_diagram.png
new file mode 100644
index 00000000..6014f1b7
Binary files /dev/null and b/doc/source/imgs/stable_terrain_diagram.png differ
diff --git a/doc/source/index.md b/doc/source/index.md
index c74caf4f..87b7c4f5 100644
--- a/doc/source/index.md
+++ b/doc/source/index.md
@@ -30,6 +30,23 @@ xDEM aims at making the analysis of digital elevation models **easy**, **modular
::::
+```{important}
+:class: margin
+xDEM ``v0.1`` is released, with nearly all features planned 4 years ago 🎉! We are now adding an **Xarray accessor**, and re-structuring the "Uncertainty" features for 2025. Note the version you are
+working on for reproducibility!
+```
+
+xDEM is **tailored to perform quantitative analysis that implicitly understands the intricacies of elevation data**,
+both from a **georeferencing viewpoint** (vertical referencing, nodata values, projection, pixel interpretation) and
+a **statistical viewpoint** (outlier robustness, specificities of 3D alignment and error structure).
+
+It exposes **an intuitive object-based API to foster accessibility**, and strives **to be computationally scalable**
+through Dask.
+
+Additionally, through its sister-package [GeoUtils](https://geoutils.readthedocs.io/en/stable/), xDEM is built on top
+of core geospatial packages (Rasterio, GeoPandas, PyProj) and numerical packages (NumPy, Xarray, SciPy) to provide
+**consistent higher-level functionalities at the interface of DEMs and elevation point cloud objects**.
+
----------------
# Where to start?
@@ -58,24 +75,19 @@ Run a short example of the package functionalities.
:::
:::{grid-item-card} {material-regular}`preview;2em` Features
-:link: vertical-ref
+:link: dem-class
:link-type: ref
Dive into the full documentation.
+++
-[Learn more »](vertical-ref)
+[Learn more »](dem-class)
:::
::::
----------------
-:::{important}
-xDEM is in early stages of development and its features might evolve rapidly. Note the version you are
-working on for reproducibility!
-We are working on making features fully consistent for the first long-term release `v0.1` (planned early 2024).
-:::
```{toctree}
:caption: Getting started
@@ -84,14 +96,7 @@ We are working on making features fully consistent for the first long-term relea
about_xdem
how_to_install
quick_start
-```
-
-```{toctree}
-:caption: Background
-:maxdepth: 2
-
-intro_robuststats
-intro_accuracy_precision
+citation
```
```{toctree}
@@ -103,10 +108,19 @@ vertical_ref
terrain
coregistration
biascorr
-comparison
+gapfill
uncertainty
```
+```{toctree}
+:caption: Resources
+:maxdepth: 2
+
+guides
+cheatsheet
+ecosystem
+```
+
```{toctree}
:caption: Gallery of examples
:maxdepth: 2
@@ -116,10 +130,14 @@ advanced_examples/index.rst
```
```{toctree}
-:caption: API Reference
+:caption: Reference
:maxdepth: 2
-api.rst
+api
+config
+release_notes
+publis
+background
```
# Indices and tables
diff --git a/doc/source/intro_accuracy_precision.md b/doc/source/intro_accuracy_precision.md
deleted file mode 100644
index 4bfd6e30..00000000
--- a/doc/source/intro_accuracy_precision.md
+++ /dev/null
@@ -1,112 +0,0 @@
-(intro)=
-
-# Analysis of accuracy and precision
-
-Digital Elevation Models are numerical, gridded representations of elevation. They are generated from different
-instruments (e.g., optical sensors, radar, lidar), acquired in different conditions (e.g., ground, airborne, satellite)
-, and using different post-processing techniques (e.g., photogrammetry, interferometry).
-
-While some complexities are specific to certain instruments and methods, all DEMs generally possess:
-
-- a [ground sampling distance](https://en.wikipedia.org/wiki/Ground_sample_distance) (GSD), or pixel size, **that does not necessarily represent the underlying spatial resolution of the observations**,
-- a [georeferencing](https://en.wikipedia.org/wiki/Georeferencing) **that can be subject to shifts, tilts or other deformations** due to inherent instrument errors, noise, or associated processing schemes,
-- a large number of [outliers](https://en.wikipedia.org/wiki/Outlier) **that remain difficult to filter** as they can originate from various sources (e.g., photogrammetric blunders, clouds).
-
-These factors lead to difficulties in assessing the accuracy and precision of DEMs, which are necessary to perform
-further analysis.
-
-In xDEM, we provide a framework with state-of-the-art methods published in the scientific literature to make DEM
-calculations consistent, reproducible, and easy.
-
-## Accuracy and precision
-
-[Accuracy and precision](https://en.wikipedia.org/wiki/Accuracy_and_precision) describe random and systematic errors,
-respectively.
-
-*Note: sometimes "accuracy" is also used to describe both types of errors, and "trueness" systematic errors, as defined
-in* [ISO 5725-1](https://www.iso.org/obp/ui/#iso:std:iso:5725:-1:ed-1:v1:en) *. Here, we used accuracy for systematic
-errors as, to our knowledge, it is a more commonly used terminology in remote sensing applications.*
-
-:::{figure} imgs/precision_accuracy.png
-:width: 80%
-
-Source: [antarcticglaciers.org](http://www.antarcticglaciers.org/glacial-geology/dating-glacial-sediments2/precision-and-accuracy-glacial-geology/), accessed 29.06.21.
-:::
-
-For DEMs, we thus have:
-
-- **DEM accuracy** (systematic error) describes how close a DEM is to the true location of measured elevations on the Earth's surface,
-- **DEM precision** (random error) of a DEM describes the typical spread of its error in measurement, independently of a possible bias from the true positioning.
-
-The spatial structure of DEMs complexifies the notion of accuracy and precision, however. Spatially structured
-systematic errors are often related to the gridded nature of DEMs, creating **affine biases** while other, **specific
-biases** exist at the pixel scale. For random errors, a variability in error magnitude or **heteroscedasticity** exists
-across the DEM, while spatially structured patterns of errors are linked to **spatial correlations**.
-
-:::{figure} https://github.com/rhugonnet/dem_error_study/blob/main/figures/fig_2.png?raw=true
-:width: 100%
-
-Source: [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922).
-:::
-
-## Absolute or relative accuracy
-
-The measure of accuracy can be further divided into two aspects:
-
-- the **absolute accuracy** of a DEM describes the average shift to the true positioning. Studies interested in analyzing features of a single DEM in relation to other georeferenced data might give great importance to this potential bias.
-- the **relative accuracy** of a DEM is related to the potential shifts, tilts, and deformations with reference to other elevation data that does not necessarily matches the true positioning. Studies interested in comparing DEMs between themselves might be only interested in this accuracy.
-
-TODO: Add another little schematic!
-
-## Optimizing DEM absolute accuracy
-
-Shifts due to poor absolute accuracy are common in elevation datasets, and can be easily corrected by performing a DEM
-co-registration to precise and accurate, quality-controlled elevation data such as [ICESat](https://icesat.gsfc.nasa.gov/icesat/) and [ICESat-2](https://icesat-2.gsfc.nasa.gov/).
-Quality-controlled DEMs aligned on high-accuracy data also exists, such as TanDEM-X global DEM (see [Rizzoli et al.
-(2017)](https://doi.org/10.1016/j.isprsjprs.2017.08.008)).
-
-Those biases can be corrected using the methods described in {ref}`coregistration`.
-
-```{eval-rst}
-.. minigallery:: xdem.coreg.Coreg
- :add-heading: Examples that use coregistration functions
-```
-
-## Optimizing DEM relative accuracy
-
-As the **absolute accuracy** can be corrected a posteriori using reference elevation datasets, many analyses only focus
-on **relative accuracy**, i.e. the remaining biases between several DEMs co-registered relative one to another.
-By harnessing the denser, nearly continuous sampling of raster DEMs (in opposition to the sparser sampling of
-higher-accuracy point elevation data), one can identify and correct other types of biases:
-
-- Terrain-related biases that can originate from the difference of resolution of DEMs, or instrument processing deformations (e.g., curvature-related biases described in [Gardelle et al. (2012)](https://doi.org/10.3189/2012JoG11J175)).
-- Directional biases that can be linked to instrument noise, such as along-track oscillations observed in many widepsread DEM products such as SRTM, ASTER, SPOT, Pléiades (e.g., [Girod et al. (2017)](https://doi.org/10.3390/rs9070704)).
-
-Those biases can be tackled by iteratively combining co-registration and bias-correction methods described
-in {ref}`coregistration` and {ref}`biascorr`.
-
-TODO: add mini-gallery for bias correction methods
-
-## Quantifying DEM precision
-
-While dealing with **accuracy** is quite straightforward as it consists of minimizing the differences (biases) between
-several datasets, assessing the **precision** of DEMs can be much more complex.
-Measurement errors of a DEM cannot be quantified by a simple difference and require statistical inference.
-
-The **precision** of DEMs has historically been reported by a single metric (e.g., precision of $\pm$ 2 m), but
-recent studies (e.g., [Rolstad et al. (2009)](https://doi.org/10.3189/002214309789470950), [Dehecq et al. (2020)](https://doi.org/10.3389/feart.2020.566802) and [Hugonnet et al. (2021)](https://doi.org/10.1038/s41586-021-03436-z))
-have shown the limitations of such simple metrics and provide more statistically-advanced methods to account for
-potential variabilities in precision and related correlations in space.
-However, the lack of implementations of these methods in a modern programming language makes them hard to reproduce,
-validate, and apply consistently. This is why one of the main goals of xDEM is to simplify state-of-the-art
-statistical measures, to allow accurate DEM uncertainty estimation for everyone.
-
-The tools for quantifying DEM precision are described in {ref}`spatialstats`.
-
-% Functions that are used in several examples create duplicate examples instead of being merged into the list.
-% Circumventing manually by selecting functions used only once in each example for now.
-
-```{eval-rst}
-.. minigallery:: xdem.spatialstats.infer_heteroscedasticity_from_stable xdem.spatialstats.get_variogram_model_func xdem.spatialstats.sample_empirical_variogram
- :add-heading: Examples that use spatial statistics functions
-```
diff --git a/doc/source/publis.md b/doc/source/publis.md
new file mode 100644
index 00000000..10d5b4cc
--- /dev/null
+++ b/doc/source/publis.md
@@ -0,0 +1,48 @@
+(publis)=
+
+# Use in publications
+
+Below, a list of publications making use of the xDEM package (that we are aware of!).
+
+## Articles
+
+### Pre-prints
+
+- Hartl, L., Schmitt, P., Schuster, L., Helfricht, K., Abermann, J., & Maussion, F. (2024). **Recent observations and glacier modeling point towards near complete glacier loss in western Austria (Ötztal and Stubai mountain range) if 1.5 °C is not met**.
+- Liu, Z., Filhol, S., & Treichler, D. (2024). **Retrieving snow depth distribution by downscaling ERA5 Reanalysis with ICESat-2 laser altimetry**. In arXiv [physics.geo-ph]. arXiv.
+- Mattea, E., Berthier, E., Dehecq, A., Bolch, T., Bhattacharya, A., Ghuffar, S., Barandun, M., & Hoelzle, M. (2024). **Five decades of Abramov glacier dynamics reconstructed with multi-sensor optical remote sensing**.
+- Zhu, Y., Liu, S., Wei, J., Wu, K., Bolch, T., Xu, J., Guo, W., Jiang, Z., Xie, F., Yi, Y., Shangguan, D., Yao, X., & Zhang, Z. (2024). **Glacier-level and gridded mass change in the rivers’ sources in the eastern Tibetan Plateau (ETPR) from 1970s to 2000**.
+- Walden, J., Jacquemart, M., Higman, B., Hugonnet, R., Manconi, A., & Farinotti, D. (2024). **A regional analysis of paraglacial landslide activation in southern coastal Alaska**.
+
+### 2024
+
+- Dømgaard, M., Schomacker, A., Isaksson, E., Millan, R., Huiban, F., Dehecq, A., Fleischer, A., Moholdt, G., Andersen, J. K., & Bjørk, A. A. (2024). **Early aerial expedition photos reveal 85 years of glacier growth and stability in East Antarctica**. *Nature Communications*, 15(1), 4466.
+- Piermattei, L., Zemp, M., Sommer, C., Brun, F., Braun, M. H., Andreassen, L. M., Belart, J. M. C., Berthier, E., Bhattacharya, A., Boehm Vock, L., Bolch, T., Dehecq, A., Dussaillant, I., Falaschi, D., Florentine, C., Floricioiu, D., Ginzler, C., Guillet, G., Hugonnet, R., … Yang, R. (2024). **Observing glacier elevation changes from spaceborne optical and radar sensors – an inter-comparison experiment using ASTER and TanDEM-X data**. *The Cryosphere*, 18(7), 3195–3230.
+
+### 2023
+
+- Bernat, M., Belart, J. M. C., Berthier, E., Jóhannesson, T., Hugonnet, R., Dehecq, A., Magnússon, E., & Gunnarsson, A. (2023). **Geodetic mass balance of Mýrdalsjökull ice cap, 1999--2021**. *Jökull*, 73(1), 35–53.
+- Khadka, N., Shrestha, N. ., Basnet, K., Manandhar, R., Sharma, S., & Shrestha, B. (2023). **Glacier Area, Mass and Associated Glacial Lake Change in Kawari basin, Western Nepal**. *Jalawaayu*, 3(1), 63–72.
+- Brun, F., King, O., Réveillet, M., Amory, C., Planchot, A., Berthier, E., Dehecq, A., Bolch, T., Fourteau, K., Brondex, J., Dumont, M., Mayer, C., Leinss, S., Hugonnet, R., & Wagnon, P. (2023). **Everest South Col Glacier did not thin during the period 1984–2017**. *The Cryosphere*, 17(8), 3251–3268.
+- Knuth, F., Shean, D., Bhushan, S., Schwat, E., Alexandrov, O., McNeil, C., Dehecq, A., Florentine, C., & O’Neel, S. (2023). **Historical Structure from Motion (HSfM): Automated processing of historical aerial photographs for long-term topographic change analysis**. *Remote Sensing of Environment*, 285, 113379.
+- Schwat, E., Istanbulluoglu, E., Horner-Devine, A., Anderson, S., Knuth, F., & Shean, D. (2023). **Multi-decadal erosion rates from glacierized watersheds on Mount Baker, Washington, USA, reveal topographic, climatic, and lithologic controls on sediment yields**. *Geomorphology*, 438(108805), 108805.
+
+### 2022
+
+- Farnsworth, W. R., Ingólfsson, Ó., Mannerfelt, E. S., Kalliokoski, M. H., Guðmundsdóttir, E. R., Retelle, M., Allaart, L., Brynjólfsson, S., Furze, M. F. A., Hancock, H. J., Kjær, K. H., Pieńkowski, A. J., & Schomacker, A. (2022). **Vedde Ash constrains Younger Dryas glacier re-advance and rapid glacio-isostatic rebound on Svalbard**. *Quaternary Science Advances*, 5(100041), 100041.
+- Mannerfelt, E. S., Dehecq, A., Hugonnet, R., Hodel, E., Huss, M., Bauder, A., & Farinotti, D. (2022). **Halving of Swiss glacier volume since 1931 observed from terrestrial image photogrammetry**. *The Cryosphere*, 16(8), 3249–3268.
+- Abad, L., Hölbling, D., Dabiri, Z., & Robson, B. A. (2022). **An open-source-based workflow for DEM generation from Sentinel-1 for landslide volume estimation**. *ISPRS - International Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences*, XLVIII-4/W1-2022, 5–11.
+- Hugonnet, R., Brun, F., Berthier, E., Dehecq, A., Mannerfelt, E. S., Eckert, N., & Farinotti, D. (2022). **Uncertainty Analysis of Digital Elevation Models by Spatial Inference From Stable Terrain**. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 15, 6456–6472.
+
+## Theses
+
+### PhD
+
+- Hugonnet, R. (2022). **Global glacier mass change by spatiotemporal analysis of digital elevation models**,
+
+### Master
+
+- Vlieghe, P-L. (2023). **Revealing the Recent Height Changes of the Great Altesch Glacier Using TanDEM-X DEM Series**,
+- Saheed, A. (2023). **Investigation of changes in Briksdalsbreen, western Norway from 1966 - 2020**,
+- Liu, Z. (2022). **Snow Depth Retrieval and Downscaling using Satellite Laser Altimetry, Machine Learning, and Climate Reanalysis: A Case Study in Mainland Norway**,
+- Bernat, M. (2022). **Geodetic mass balance of Mýrdalsjökull ice cap, 1999−2021: DEM processing and climate analysis**,
diff --git a/doc/source/quick_start.md b/doc/source/quick_start.md
index f7e67751..da3058bb 100644
--- a/doc/source/quick_start.md
+++ b/doc/source/quick_start.md
@@ -33,6 +33,16 @@ Below, in a few lines, we load two DEMs and a vector of glacier outlines, crop t
align the DEMs using coregistration, estimate the elevation change, estimate elevation change error using stable
terrain, and finally plot and save the result!
+
+```{code-cell} ipython3
+:tags: [remove-cell]
+
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 600
+pyplot.rcParams['savefig.dpi'] = 600
+```
+
```{code-cell} ipython3
import xdem
import geoutils as gu
@@ -46,6 +56,11 @@ fn_glacier_outlines = xdem.examples.get_path("longyearbyen_glacier_outlines")
print(f"DEM 1: {fn_dem_ref}, \nDEM 2: {fn_dem_tba}, \nOutlines: {fn_glacier_outlines}")
```
+```{tip}
+:class: margin
+Set up your {ref}`verbosity` to manage outputs to the console (or a file) during execution!
+```
+
```{code-cell} ipython3
# Open files by instantiating DEM and Vector
# (DEMs are loaded lazily = only metadata but not array unless required)
@@ -101,7 +116,7 @@ os.remove("dh_error.tif")
## More examples
To dive into more illustrated code, explore our gallery of examples that is composed of:
-- An {ref}`examples-basic` section on simpler routines (terrain attributes, pre-defined coregistration and uncertainty pipelines),
+- A {ref}`examples-basic` section on simpler routines (terrain attributes, pre-defined coregistration and uncertainty pipelines),
- An {ref}`examples-advanced` section using advanced pipelines (for in-depth coregistration and uncertainty analysis).
See also the concatenated list of examples below.
diff --git a/doc/source/release_notes.md b/doc/source/release_notes.md
new file mode 100644
index 00000000..5aab8d04
--- /dev/null
+++ b/doc/source/release_notes.md
@@ -0,0 +1,72 @@
+# Release notes
+
+Below, the release notes for all minor versions and our roadmap to a first major version.
+
+## 0.1.0
+
+xDEM 0.1.0 is the **first minor release** since the creation of the project in 2020. It is the result of years of work
+to consolidate and re-structure features into a mature and stable API to minimize future breaking changes.
+
+**All the core features drafted at the start of the project are now supported**, and there is a **clear roadmap
+towards a first major release 1.0**. This minor release also adds many tests and improves significantly the documentation
+from the early-development state of the package.
+
+The re-structuring created some breaking changes, though minor.
+
+See details below, including **a guide to help migrate code from early-development versions**.
+
+### Features
+
+xDEM now gathers the following core features:
+- **Elevation data objects** core to quantatiative analysis, which are DEMs and elevation point clouds,
+- **Vertical referencing** including automatic 3D CRS fetching,
+- **Terrain analysis** for many attributes,
+- **Coregistration** with the choice of several methods, including modular pipeline building,
+- **Bias corrections** for any variable, also modular and supported by pipelines,
+- **Uncertainty analysis** based on several robust methods.
+
+Recent additions include in particular **point-raster support for coregistration**, and the **expansion of
+`DEM` class methods** to cover all features of the package, with for instance `DEM.coregister_3d()` or `DEM.slope()`.
+
+### Guides and other resources
+
+xDEM integrates **background material on quantitative analysis of elevation data** to help users use the various methods
+of the package. This material includes **several illustrated guide pages**, **a cheatsheet** on how to recognize and correct
+typical elevation errors, and more.
+
+### Future deprecations
+
+We have added warnings throughout the documentation and API related to planned deprecations:
+- **Gap-filling features specific to glacier-applications** will be moved to a separate package,
+- **Uncertainty analysis tools related to variography** will change API to rely on SciKit-GStat variogram objects,
+- The **dDEM** and **DEMCollection** classes will likely be refactored or removed.
+
+Changes related to **gap-filling** and **uncertainty analysis** will have deprecation warnings, while the function
+remain available during a few more releases.
+
+### Migrate from early versions
+
+The following changes **might be required to solve breaking changes**, depending on your early-development version:
+- Rename `.show()` to `.plot()` for all data objects,
+- Rename `.dtypes` to `dtype` for `DEM` objects,
+- Operations `.crop()`, `shift()` and `to_vcrs()` are not done in-place by default anymore, replace by `dem = dem.crop()` or `dem.crop(..., inplace=True)` to mirror the old default behaviour,
+- Rename `.shift()` to `.translate()` for `DEM` objects,
+- Several function arguments are renamed, in particular `dst_xxx` arguments of `.reproject()` are all renamed to `xxx` e.g. `dst_crs` to `crs`, as well as the arguments of `Coreg.fit()` renamed from `xxx_dem` to `xxx_elev` to be generic to any elevation data,
+- All `BiasCorr1D`, `BiasCorr2D` and `BiasCorrND` classes are removed in favor of a single `BiasCorr` class that implicitly understands the number of dimensions from the length of input `bias_vars`,
+- New user warnings are sometimes raised, in particular if some metadata is not properly defined such as `.nodata`. Those should give an indication as how to silence them.
+
+Additionally, **some important yet non-breaking changes**:
+- The sequential use of `Coreg.fit()` and `Coreg.apply()` to the same `tba_elev` is now discouraged and updated everywhere in the documentation, use `Coreg.fit_and_apply()` or `DEM.coregister_3d()` instead,
+- The use of a separate module for terrain attributes such as `xdem.terrain.slope()` is now discouraged, use `DEM.slope()` instead.
+
+## Roadmap to 1.0
+
+Based on recent and ongoing progress, we envision the following roadmap.
+
+**Releases of 0.2, 0.3, 0.4, etc**, for the following planned (ongoing) additions:
+- The **addition of an elevation point cloud `EPC` data object**, inherited from the ongoing `PointCloud` object of GeoUtils alongside many features at the interface of point and raster,
+- The **addition of an Xarray accessor `dem`** mirroring the `DEM` object, to work natively with Xarray objects and add support on out-of-memory Dask operations for most of xDEM's features,
+- The **addition of an GeoPandas accessor `epc`** mirroring the `EPC` object, to work natively with GeoPandas objects,
+- The **re-structuration of uncertainty analysis features** to rely directly on SciKit-GStat's `Variogram` object.
+
+**Release of 1.0** once all these additions are fully implemented, and after feedback from the community.
diff --git a/doc/source/intro_robuststats.md b/doc/source/robust_estimators.md
similarity index 60%
rename from doc/source/intro_robuststats.md
rename to doc/source/robust_estimators.md
index e410c5e1..d0c1dc2e 100644
--- a/doc/source/intro_robuststats.md
+++ b/doc/source/robust_estimators.md
@@ -1,17 +1,21 @@
-(robuststats)=
+(robust-estimators)=
-# The need for robust statistics
+# Need for robust estimators
-Digital elevation models often contain outliers that can be traced back to instrument acquisition or processing artefacts, and which hamper further analysis.
+Elevation data often contain outliers that can be traced back to instrument acquisition or processing artefacts, and which hamper further analysis.
-In order to mitigate their effect, xDEM integrates [robust statistics](https://en.wikipedia.org/wiki/Robust_statistics) at different levels:
-- Robust optimizers for the fitting of parametric models during {ref}`coregistration` and {ref}`biascorr`,
-- Robust measures for the central tendency (e.g., mean) and dispersion (e.g., standard deviation), to evaluate DEM quality and converge during {ref}`coregistration`,
-- Robust measures for estimating spatial autocorrelation for uncertainty analysis in {ref}`spatialstats`.
+In order to mitigate their effect, the analysis of elevation data can integrate [robust statistics](https://en.wikipedia.org/wiki/Robust_statistics) at different levels:
+- **Robust estimators for the central tendency and statistical dispersion** used during {ref}`coregistration`, {ref}`biascorr` and {ref}`uncertainty`,
+- **Robust estimators for estimating spatial autocorrelation** applied to error propagation in {ref}`uncertainty`,
+- **Robust optimizers for the fitting of parametric models** during {ref}`coregistration` and {ref}`biascorr`.
-Yet, there is a downside to robust statistical measures. Those can yield less precise estimates for small samples sizes and,
-in some cases, hide patterns inherent to the data. This is why, when outliers show identifiable patterns, it is better
-to first resort to outlier filtering (see {ref}`filters`) and perform analysis using traditional statistical measures.
+Yet, there is a downside to robust statistical estimators. Those can yield less precise estimates for small samples sizes and,
+in some cases, hide patterns inherent to the data. This is why, when outliers show identifiable patterns, it can be better
+to first resort to outlier filtering and perform analysis using traditional statistical measures.
+
+```{important}
+In xDEM, robust estimators are used everywhere by default.
+```
(robuststats-meanstd)=
@@ -20,7 +24,7 @@ to first resort to outlier filtering (see {ref}`filters`) and perform analysis u
### Central tendency
The [central tendency](https://en.wikipedia.org/wiki/Central_tendency) represents the central value of a sample, and is
-core to the analysis of sample accuracy (see {ref}`intro`). It is most often measured by the [mean](https://en.wikipedia.org/wiki/Mean).
+core to the analysis of sample accuracy (see {ref}`accuracy-precision`). It is most often measured by the [mean](https://en.wikipedia.org/wiki/Mean).
However, the mean is a measure sensitive to outliers. Therefore, in many cases (e.g., when working with unfiltered
DEMs) using the [median](https://en.wikipedia.org/wiki/Median) as measure of central tendency is preferred.
@@ -28,20 +32,23 @@ When working with weighted data, the [weighted median](https://en.wikipedia.org/
to the 50{sup}`th` [weighted percentile](https://en.wikipedia.org/wiki/Percentile#Weighted_percentile) can be
used as a robust measure of central tendency.
-The median is used by default in the alignment routines of {ref}`coregistration` and {ref}`biascorr`.
+The {func}`numpy.median` is used by default in the alignment routines of **{ref}`coregistration` and {ref}`biascorr`**.
+
+```{eval-rst}
+.. plot:: code/robust_mean_std.py
+ :width: 90%
+```
(robuststats-nmad)=
### Dispersion
The [statistical dispersion](https://en.wikipedia.org/wiki/Statistical_dispersion) represents the spread of a sample,
-and is core to the analysis of sample precision (see {ref}`intro`). It is typically measured by the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation).
+and is core to the analysis of sample precision (see {ref}`accuracy-precision`). It is typically measured by the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation).
However, very much like the mean, the standard deviation is a measure sensitive to outliers.
The median equivalent of a standard deviation is the normalized median absolute deviation (NMAD), which corresponds to the [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) scaled by a factor of ~1.4826 to match the dispersion of a
-normal distribution. It has been shown to provide more robust measures of dispersion with outliers when working
-with DEMs (e.g., [Höhle and Höhle (2009)](https://doi.org/10.1016/j.isprsjprs.2009.02.003)).
-It is defined as:
+normal distribution. It is a more robust measure of dispersion with outliers, defined as:
$$
\textrm{NMAD}(x) = 1.4826 \cdot \textrm{median}_{i} \left ( \mid x_{i} - \textrm{median}(x) \mid \right )
@@ -49,10 +56,6 @@ $$
where $x$ is the sample.
-```python
-nmad = xdem.spatialstats.nmad
-```
-
```{note}
The NMAD estimator has a good synergy with {ref}`Dowd's variogram` for spatial autocorrelation, as their median-based measure of dispersion is the same.
```
@@ -61,9 +64,7 @@ The half difference between 84{sup}`th` and 16{sup}`th` percentiles, or the abso
can also be used as a robust dispersion measure equivalent to the standard deviation.
When working with weighted data, the difference between the 84{sup}`th` and 16{sup}`th` [weighted percentile](https://en.wikipedia.org/wiki/Percentile#Weighted_percentile), or the absolute 68{sup}`th` weighted percentile can be used as a robust measure of dispersion.
-```{important}
-The NMAD is used by default for estimating elevation measurement errors in {ref}`spatialstats`.
-```
+The {func}`xdem.spatialstats.nmad` is used by default in **{ref}`coregistration`, {ref}`biascorr` and {ref}`uncertainty`**.
(robuststats-corr)=
@@ -72,7 +73,7 @@ The NMAD is used by default for estimating elevation measurement errors in {ref}
[Variogram](https://en.wikipedia.org/wiki/Variogram) analysis exploits statistical measures equivalent to the covariance,
and is therefore also subject to outliers.
Based on [SciKit-GStat](https://mmaelicke.github.io/scikit-gstat/index.html), xDEM allows to specify robust variogram
-estimators such as Dowd's variogram based on medians ([Dowd (1984)](https://en.wikipedia.org/wiki/Variogram)) defined as:
+estimators such as Dowd's variogram based on medians defined as:
$$
2\gamma (h) = 2.198 \cdot \textrm{median}_{i} \left ( Z_{x_{i}} - Z_{x_{i+h}} \right )
@@ -86,8 +87,11 @@ Dowd's estimator has a good synergy with the {ref}`NMAD` for e
Other estimators can be chosen from [SciKit-GStat's list of estimators](https://scikit-gstat.readthedocs.io/en/latest/reference/estimator.html).
-```{important}
-Dowd's variogram is used by default to estimate spatial auto-correlation of elevation measurement errors in {ref}`spatialstats`.
+Dowd's variogram is used by default to estimate spatial auto-correlation of elevation measurement errors in **{ref}`uncertainty`**.
+
+```{eval-rst}
+.. plot:: code/robust_vario.py
+ :width: 90%
```
(robuststats-regression)=
@@ -98,7 +102,7 @@ Dowd's variogram is used by default to estimate spatial auto-correlation of elev
When performing least-squares linear regression, the traditional [loss functions](https://en.wikipedia.org/wiki/Loss_function) that are used are not robust to outliers.
-A robust soft L1 loss default is used by default when xDEM performs least-squares regression through [scipy.optimize](https://docs.scipy.org/doc/scipy/reference/optimize.html#).
+A robust soft L1 loss default is used by default to perform least-squares regression through [scipy.optimize](https://docs.scipy.org/doc/scipy/reference/optimize.html#) in **{ref}`coregistration` and {ref}`biascorr`**.
### Robust estimators
@@ -108,3 +112,13 @@ The {ref}`coregistration` and {ref}`biascorr` methods encapsulate some of those
- The Random sample consensus estimator [RANSAC](https://en.wikipedia.org/wiki/Random_sample_consensus),
- The [Theil-Sen](https://en.wikipedia.org/wiki/Theil%E2%80%93Sen_estimator) estimator,
- The [Huber loss](https://en.wikipedia.org/wiki/Huber_loss) estimator.
+
+----------------
+
+:::{admonition} References and more reading
+:class: tip
+
+**References:**
+- [Dowd (1984)](https://doi.org/10.1007/978-94-009-3699-7_6), The Variogram and Kriging: Robust and Resistant Estimators,
+- [Höhle and Höhle (2009)](https://doi.org/10.1016/j.isprsjprs.2009.02.003), Accuracy assessment of digital elevation models by means of robust statistical methods.
+:::
diff --git a/doc/source/spatial_stats.md b/doc/source/spatial_stats.md
new file mode 100644
index 00000000..8ba85b24
--- /dev/null
+++ b/doc/source/spatial_stats.md
@@ -0,0 +1,206 @@
+---
+file_format: mystnb
+jupytext:
+ formats: md:myst
+ text_representation:
+ extension: .md
+ format_name: myst
+kernelspec:
+ display_name: xdem-env
+ language: python
+ name: xdem
+---
+(spatial-stats)=
+
+# Spatial statistics for error analysis
+
+Performing error (or uncertainty) analysis of spatial variable, such as elevation data, requires **joint knowledge from
+two scientific fields: spatial statistics and uncertainty quantification.**
+
+Spatial statistics, also referred to as [geostatistics](https://en.wikipedia.org/wiki/Geostatistics) in geoscience,
+is a body of theory for the analysis of spatial variables. It primarily relies on modelling the dependency of
+variables in space (spatial autocorrelation) to better describe their spatial characteristics, and
+utilize this in further quantitative analysis.
+
+[Uncertainty quantification](https://en.wikipedia.org/wiki/Uncertainty_quantification) is the science of characterizing
+uncertainties quantitatively, and includes a wide range of methods including in particular theoretical error propagation.
+In measurement science, such as remote sensing, such uncertainty propagation is tightly linked with the field
+of [metrology](https://en.wikipedia.org/wiki/Metrology).
+
+In the following, we describe the basics assumptions and concepts required to perform a spatial uncertainty analysis of
+elevation data, described in the **feature page {ref}`uncertainty`**.
+
+## Assumptions for inference in spatial statistics
+
+In spatial statistics, the covariance of a variable of interest is generally simplified into a spatial variogram, which
+**describes the covariance only as function of the spatial lag** (spatial distance between two variable values).
+However, to utilize this simplification of the covariance in subsequent analysis, the variable of interest must
+respect [the assumption of second-order stationarity](https://www.aspexit.com/en/fundamental-assumptions-of-the-variogram-second-order-stationarity-intrinsic-stationarity-what-is-this-all-about/).
+That is, verify the three following assumptions:
+
+> 1. The mean of the variable of interest is stationary in space, i.e. constant over sufficiently large areas,
+> 2. The variance of the variable of interest is stationary in space, i.e. constant over sufficiently large areas.
+> 3. The covariance between two observations only depends on the spatial distance between them, i.e. no other factor than this distance plays a role in the spatial correlation of measurement errors.
+
+```{eval-rst}
+.. plot:: code/spatialstats_stationarity_assumption.py
+ :width: 90%
+```
+
+In other words, for a reliable analysis, elevation data should:
+
+> 1. Not contain elevation biases that do not average out over sufficiently large distances (e.g., shifts, tilts), but can contain pseudo-periodic biases (e.g., along-track undulations),
+> 2. Not contain random elevation errors that vary significantly across space.
+> 3. Not contain factors that affect the spatial distribution of elevation errors, except for the distance between observations.
+
+While assumption **1.** is verified after coregistration and bias corrections, other assumptions are generally not
+(e.g., larger errors on steep slope). To address this, we must estimate the variability of our random errors
+(or heteroscedasticity), to then transform our data to achieve second-order stationarity.
+
+```{note}
+If there is no significant spatial variability in random errors in your elevation data (e.g., lidar),
+you can **jump directly to the {ref}`spatialstats-corr` section**.
+```
+
+## Heteroscedasticity
+
+Elevation [heteroscedasticity](https://en.wikipedia.org/wiki/Heteroscedasticity) corresponds to a variability in
+precision of elevation observations, that are linked to terrain or instrument variables.
+
+$$
+\sigma_{dh} = \sigma_{dh}(\textrm{var}_{1},\textrm{var}_{2}, \textrm{...}) \neq \textrm{constant}
+$$
+
+While a single elevation difference (for a pixel or footpring) does not allow to capture random errors, larger samples
+do. [Data binning](https://en.wikipedia.org/wiki/Data_binning), for instance, is a method that allows to estimate the
+statistical spread of a sample per category, and can easily be used with one or more explanatory variables,
+such as slope:
+
+```{eval-rst}
+.. plot:: code/spatialstats_heterosc_slope.py
+ :width: 90%
+```
+
+Then, a model (parametric or not) can be fit to infer the variability of random errors at any data location.
+
+## Standardization
+
+In order to verify the assumptions of spatial statistics and be able to use stable terrain as a reliable proxy in
+further analysis (see **{ref}`static-surfaces` guide page**), [standardization](https://en.wikipedia.org/wiki/Standard_score)
+of the elevation differences by their mean $\mu$ and spread $\sigma$ are required to reach a stationary variance.
+
+```{eval-rst}
+.. plot:: code/spatialstats_standardizing.py
+ :width: 90%
+```
+
+For elevation differences, the mean is already centered on zero but the variance is non-stationary,
+which yields more simply:
+
+$$
+z_{dh} = \frac{dh(\textrm{var}_{1}, \textrm{var}_{2}, \textrm{...})}{\sigma_{dh}(\textrm{var}_{1}, \textrm{var}_{2}, \textrm{...})}
+$$
+
+where $z_{dh}$ is the standardized elevation difference sample.
+
+(spatialstats-corr)=
+
+## Spatial correlation of errors
+
+Spatial correlation of elevation errors correspond to a dependency between measurement errors of spatially
+close pixels in elevation data. Those can be related to the resolution of the data (short-range correlation), or to
+instrument noise and deformations (mid- to long-range correlations).
+
+[Variograms](https://en.wikipedia.org/wiki/Variogram) are functions that describe the spatial correlation of a sample.
+The variogram $2\gamma(h)$ is a function of the distance between two points, referred to as spatial lag $d$.
+The output of a variogram is the correlated variance of the sample.
+
+$$
+2\gamma(d) = \textrm{var}\left(Z(\textrm{s}_{1}) - Z(\textrm{s}_{2})\right)
+$$
+
+where $Z(\textrm{s}_{i})$ is the value taken by the sample at location $\textrm{s}_{i}$, and sample positions
+$\textrm{s}_{1}$ and $\textrm{s}_{2}$ are separated by a distance $d$.
+
+```{eval-rst}
+.. plot:: code/spatialstats_variogram_covariance.py
+ :width: 90%
+```
+
+For elevation differences $dh$, this translates into:
+
+$$
+2\gamma_{dh}(d) = \textrm{var}\left(dh(\textrm{s}_{1}) - dh(\textrm{s}_{2})\right)
+$$
+
+The variogram essentially describes the spatial covariance $C$ in relation to the variance of the entire sample
+$\sigma_{dh}^{2}$:
+
+$$
+\gamma_{dh}(d) = \sigma_{dh}^{2} - C_{dh}(d)
+$$
+
+
+Variograms are estimated empirically by [data binning](https://en.wikipedia.org/wiki/Data_binning) depending on
+the pairwise distance among data points, then modelled by functional forms called variogram models (e.g., spherical,
+gaussian).
+
+As pairwise combinations grow exponentially with data points, variograms are estimated using a random subsample of the
+elevation data (whether gridded or point). Additionally, as elevation data usually contains patterns of correlation
+at both short-range (close to pixel size) and long-range (close to DEM extent), the default pairwise sampling can be
+tuned to perform well at all desired ranges.
+
+```{note}
+In xDEM, variograms use by default a subsample size of 1000 data points (leading to 500,000 pairwise differences estimated),
+for computational efficiency.
+xDEM also uses by default a random sampling method that selects a uniform amount of data points by pairwise log-distance,
+in order to capture potential correlations at all ranges.
+```
+
+For more details on variography, **we refer to [the documentation of SciKit-GStat](https://scikit-gstat.readthedocs.io/en/latest/userguide/userguide.html).**
+
+## Error propagation
+
+Once the heteroscedasticity $\sigma_{dh}(\textrm{var}_{1},\textrm{var}_{2}, \textrm{...})$ and spatial
+correlation of errors $\gamma_{dh}(d)$ are both estimated, those can be used to propagate errors to derivatives
+of the elevation data.
+
+**For simple derivatives such as spatial derivatives** (e.g., mean or sum in an area), exact
+[theoretical propagation](https://en.wikipedia.org/wiki/Propagation_of_uncertainty) relying on the above components
+can be employed.
+
+For instance, to propagate errors to the mean of elevation differences in an area $A$ containing $N$ data points:
+
+$$
+\sigma_{\overline{dh}} = \frac{1}{N^2} \sum_{i=1}^{N} \sum_{j=1}^{N} (1 - \gamma_{z_{dh}}(d_{ij})) \sigma_{dh_{i}} \sigma_{dh_{j}}
+$$
+
+
+where $d_{ij}$ is the distance between data point $i$ and $j$.
+
+```{note}
+xDEM implement several methods to approximate the above equation which scales exponentially with data points, to preserve
+computational efficiency.
+```
+
+**For more complex derivatives such as terrain attributes**, the heteroscedasticity and spatial correlation of errors
+can be used to constrain simulation methods that numerically generate realizations of the structure of errors.
+
+For instance, to propagate errors to terrain slope, one can derive many realizations of random fields based on the error structure estimated
+for the DEM. Then, for each realization, add the random field to the DEM, and derive its slope. Finally,
+the error in slope can be estimated from the spread of all slope realizations.
+
+----------------
+
+:::{admonition} References and more reading
+:class: tip
+
+For random field generation, see for example [GSTools](https://geostat-framework.readthedocs.io/projects/gstools/en/stable/).
+
+**References:**
+
+- [Heuvelink et al. (1989)](https://doi.org/10.1080/02693798908941518), Propagation of errors in spatial modelling with GIS,
+- [Rolstad et al. (2009)](http://dx.doi.org/10.3189/002214309789470950), Spatially integrated geodetic glacier mass balance and its uncertainty based on geostatistical analysis: Application to the western Svartisen ice cap, Norway,
+- [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922), Uncertainty analysis of digital elevation models by spatial inference from stable terrain.
+
+:::
diff --git a/doc/source/sphinxext.py b/doc/source/sphinxext.py
new file mode 100644
index 00000000..8229047e
--- /dev/null
+++ b/doc/source/sphinxext.py
@@ -0,0 +1,13 @@
+"""Functions for documentation configuration only, importable by sphinx"""
+# To reset resolution setting for each sphinx-gallery example
+def reset_mpl(gallery_conf, fname):
+ # To get a good resolution for displayed figures
+ from matplotlib import pyplot
+
+ pyplot.rcParams["figure.dpi"] = 400
+ pyplot.rcParams["savefig.dpi"] = 400
+
+ # Reset logging to default
+ import logging
+
+ logging.basicConfig(force=True)
diff --git a/doc/source/static_surfaces.md b/doc/source/static_surfaces.md
new file mode 100644
index 00000000..248280c3
--- /dev/null
+++ b/doc/source/static_surfaces.md
@@ -0,0 +1,101 @@
+(static-surfaces)=
+
+# Static surfaces as error proxy
+
+Below, a short guide explaining the use of static surfaces as an error proxy for quantitative elevation analysis.
+
+## The great benefactor of elevation analysis
+
+Elevation data benefits from an uncommon asset, which is that **large proportions of planetary surface elevations
+usually remain virtually unchanged through time** (at least, within decadal time scales). Those static surfaces,
+sometimes also referred to as "stable terrain", generally refer to bare-rock, grasslands, and are often isolated by
+excluding dynamic surfaces such as glaciers, snow, forests and cities. If small proportions of static surfaces are
+not masked, they are generally filtered out by robust estimators (see {ref}`robust-estimators`).
+
+:::{figure} imgs/stable_terrain_diagram.png
+:width: 100%
+
+Source: [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922).
+:::
+
+## Use for coregistration and further uncertainty analysis
+
+Elevation data can rarely be compared to simultaneous acquisitions to assess its sources of error. This is
+where **static surfaces come to the rescue, and can act as an error proxy**. By assuming no changes happened on these
+surfaces, and that they have the same error structure as other surfaces, it becomes possible to perform
+coregistration, bias-correction and further uncertainty analysis!
+
+Below, we summarize the basic principles of how using static surfaces allows to perform coregistration and uncertainty analysis, and the related limitations.
+
+### For coregistration and bias-correction (systematic errors)
+
+**Static surfaces $S$ are key to a coregistration or bias correction transformation $C$** for which it is assumed that,
+for two sets of elevation data $h_{1}$ and $h_{2}$, we have:
+
+$$
+(h_{1} - C(h_{2}))_{S} \approx 0
+$$
+
+and aim to find the best transformation $C$ to minimize this problem.
+
+The above relation is not generally true for every pixel or footprint, however, due to random errors that
+exist in all data. Consequently, we can only write:
+
+$$
+\textrm{mean} (h_{1} - C(h_{2}))_{S \gg r^{2}} \approx 0
+$$
+
+where $r$ is the correlation range of random errors, and $S \gg r^{2}$ assumes that static surfaces cover a domain much
+larger than this correlation range. If static surfaces cover too small an area, coregistration will naturally become
+less reliable.
+
+```{note}
+One of the objectives of xDEM is to allow to use knowledge on random errors to refine
+coregistration for limited static surface areas, stay tuned!
+```
+
+### For further uncertainty analysis (random errors)
+
+**Static surfaces are also essential for uncertainty analysis aiming to infer the random errors of elevation
+data** but, in this case, we have to consider the effect of random errors from both sets of elevation data.
+
+We first assume that elevation $h_{2}$ is now largely free of systematic errors after performing coregistration and
+bias corrections $C$. The analysis of elevation differences $dh$ on static surfaces $S$ will represent the mixed random
+errors of the two sets of data, that we can assume are statistically independent (if indeed acquired separately), which yields:
+
+$$
+\sigma_{dh, S} = \sigma_{h_{\textrm{1}} - h_{\textrm{2}}} = \sqrt{\sigma_{h_{\textrm{1}}}^{2} + \sigma_{h_{\textrm{2}}}^{2}}
+$$
+
+where $\sigma$ is the random error at any data point.
+
+If one set of elevation data is known to be of much higher-precision, one can assume that the analysis of differences
+will represent only the precision of the rougher DEM. For instance, $\sigma_{h_{1}} = 3 \sigma_{h_{2}}$ implies that more than
+95% of $\sigma_{dh}$ comes from $\sigma_{h_{1}}$ from the above equation.
+
+More generally:
+
+$$
+\sigma_{dh, S} = \sigma_{h_{\textrm{higher precision}} - h_{\textrm{lower precision}}} \approx \sigma_{h_{\textrm{lower precision}}}
+$$
+
+And the same applies to the spatial correlation of these random errors:
+
+$$
+\rho_{dh, S}(d) = \rho_{h_{\textrm{higher precision}} - h_{\textrm{lower precision}}}(d) \approx \rho_{h_{\textrm{lower precision}}}(d)
+$$
+
+where $\rho(d)$ is the spatial correlation, and $d$ is the spatial lag (distance between data points).
+
+----------------
+
+:::{admonition} References and more reading
+:class: tip
+
+Static surfaces can be used as a **proxy for assessing systematic and random errors**, which directly relates to
+what is commonly referred to as accuracy and precision of elevation data, detailed in the **next guide page on {ref}`accuracy-precision`**.
+
+See the **{ref}`spatial-stats` guide page** for more details on spatial statistics applied to uncertainty quantification.
+
+**References:** [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922), Uncertainty analysis of digital elevation models by spatial inference from stable terrain.
+:::
diff --git a/doc/source/terrain.md b/doc/source/terrain.md
index 3d6d16f7..e4dcfff4 100644
--- a/doc/source/terrain.md
+++ b/doc/source/terrain.md
@@ -1,3 +1,17 @@
+---
+file_format: mystnb
+mystnb:
+ execution_timeout: 150
+jupytext:
+ formats: md:myst
+ text_representation:
+ extension: .md
+ format_name: myst
+kernelspec:
+ display_name: xdem-env
+ language: python
+ name: xdem
+---
(terrain-attributes)=
# Terrain attributes
@@ -9,11 +23,94 @@ and tested for consistency against [gdaldem](https://gdal.org/programs/gdaldem.h
## Quick use
-Terrain attribute methods can either be called directly from a {class}`~xdem.DEM` (e.g., {func}`xdem.DEM.slope`) or
-through the {class}`~xdem.terrain` module which allows array input. If computational performance
-is key, xDEM can rely on [RichDEM](https://richdem.readthedocs.io/) by specifying `use_richdem=True` for speed-up
-of its supported attributes (slope, aspect, curvature).
+Terrain attribute methods can be derived directly from a {class}`~xdem.DEM`, using for instance {func}`xdem.DEM.slope`.
+```{code-cell} ipython3
+:tags: [remove-cell]
+
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 600
+pyplot.rcParams['savefig.dpi'] = 600
+```
+
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the opening of example files."
+: code_prompt_hide: "Hide the opening of example files."
+
+import xdem
+
+# Open a DEM from a filename on disk
+filename_dem = xdem.examples.get_path("longyearbyen_ref_dem")
+dem = xdem.DEM(filename_dem)
+```
+
+```{code-cell} ipython3
+# Slope from DEM method
+slope = dem.slope()
+# Or from terrain module using an array input
+slope = xdem.terrain.slope(dem.data, resolution=dem.res)
+```
+
+:::{admonition} Coming soon
+:class: note
+
+We are working on further optimizing the computational performance of certain terrain attributes using convolution.
+:::
+
+
+## Summary of supported methods
+
+```{list-table}
+ :widths: 1 1 1
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Attribute
+ - Unit (if DEM in meters)
+ - Reference
+ * - {ref}`slope`
+ - Degrees (default) or radians
+ - [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) or [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107)
+ * - {ref}`aspect`
+ - Degrees (default) or radians
+ - [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) or [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107)
+ * - {ref}`hillshade`
+ - Unitless
+ - [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) or [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107)
+ * - {ref}`curv`
+ - Meters{sup}`-1` * 100
+ - [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107)
+ * - {ref}`plancurv`
+ - Meters{sup}`-1` * 100
+ - [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107)
+ * - {ref}`profcurv`
+ - Meters{sup}`-1` * 100
+ - [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107)
+ * - {ref}`tpi`
+ - Meters
+ - [Weiss (2001)](http://www.jennessent.com/downloads/TPI-poster-TNC_18x22.pdf)
+ * - {ref}`tri`
+ - Meters
+ - [Riley et al. (1999)](http://download.osgeo.org/qgis/doc/reference-docs/Terrain_Ruggedness_Index.pdf) or [Wilson et al. (2007)](http://dx.doi.org/10.1080/01490410701295962)
+ * - {ref}`roughness`
+ - Meters
+ - [Dartnell (2000)](https://environment.sfsu.edu/node/11292)
+ * - {ref}`rugosity`
+ - Unitless
+ - [Jenness (2004)]()
+ * - {ref}`fractrough`
+ - Fractal dimension number (1 to 3)
+ - [Taud and Parrot (2005)](https://doi.org/10.4000/geomorphologie.622)
+```
+
+```{note}
+Only grids with **equal pixel size in X and Y** are currently supported. Transform into such a grid with {func}`xdem.DEM.reproject`.
+```
+
+(slope)=
## Slope
{func}`xdem.DEM.slope`
@@ -22,20 +119,63 @@ The slope of a DEM describes the tilt, or gradient, of each pixel in relation to
It is most often described in degrees, where a flat surface is 0° and a vertical cliff is 90°.
No tilt direction is stored in the slope map; a 45° tilt westward is identical to a 45° tilt eastward.
-The slope can be computed either by the method of [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) (default)
-based on a refined gradient formulation on a 3x3 pixel window, or by the method of [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107) based on a plane fit on a 3x3 pixel window.
+The slope $\alpha$ can be computed either by the method of [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) (default)
+based on a refined gradient formulation on a 3x3 pixel window, or by the method of
+[Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107) based on a plane fit on a 3x3 pixel window.
+
+For both methods, $\alpha = arctan(\sqrt{p^{2} + q^{2}})$ where $p$ and $q$ are the gradient components west-to-east and south-to-north, respectively, with for [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918):
+
+$$
+p_{\textrm{Horn}}=\frac{(h_{++} + 2h_{+0} + h_{+-}) - (h_{-+} + 2h_{-0} + h_{--})}{8 \Delta x},\\
+q_{\textrm{Horn}}=\frac{(h_{++} + 2h_{0+} + h_{-+}) - (h_{+-} + 2h_{0-} + h_{--})}{8 \Delta y},
+$$
+
+and for [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107):
+
+$$
+p_{\textrm{ZevTho}} = \frac{h_{+0} - h_{-0}}{2 \Delta x},\\
+q_{\textrm{ZevTho}} = \frac{h_{0+} - h_{0-}}{2 \Delta y},
+$$
+
+where $h_{ij}$ is the elevation at pixel $ij$, where indexes $i$ and $j$ correspond to east-west and north-south directions respectively,
+and take values of either the center ($0$), west or south ($-$), or east or north ($+$):
+
+```{list-table}
+ :widths: 1 1 1 1
+ :header-rows: 1
+ :stub-columns: 1
+
+ * -
+ - West
+ - Center
+ - East
+ * - North
+ - $h_{-+}$
+ - $h_{0+}$
+ - $h_{++}$
+ * - Center
+ - $h_{-0}$
+ - $h_{00}$
+ - $h_{+0}$
+ * - South
+ - $h_{--}$
+ - $h_{0-}$
+ - $h_{+-}$
+```
+
+
+Finally, $\Delta x$
+and $\Delta y$ correspond to the pixel resolution west-east and south-north, respectively.
-The differences between methods are illustrated in the {ref}`sphx_glr_basic_examples_plot_terrain_attributes.py`
+The differences between methods are illustrated in the {ref}`sphx_glr_advanced_examples_plot_slope_methods.py`
example.
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_001.png
-:width: 600
-```
-
-```{eval-rst}
-.. minigallery:: xdem.terrain.slope
+```{code-cell} ipython3
+slope = dem.slope()
+slope.plot(cmap="Reds", cbar_title="Slope (°)")
```
+(aspect)=
## Aspect
{func}`xdem.DEM.aspect`
@@ -44,169 +184,212 @@ The aspect describes the orientation of strongest slope.
It is often reported in degrees, where a slope tilting straight north corresponds to an aspect of 0°, and an eastern
aspect is 90°, south is 180° and west is 270°. By default, a flat slope is given an arbitrary aspect of 180°.
-As the aspect is directly based on the slope, it varies between the method of [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) (default) and that of [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107).
+The aspect $\theta$ is based on the same variables as the slope, and thus varies similarly between the method of
+[Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) (default) and that of
+[Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107):
+
+$$
+\theta = arctan(\frac{p}{q}),
+$$
+
+with $p$ and $q$ defined in the slope section.
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_002.png
-:width: 600
+```{warning}
+A north aspect represents the upper direction of the Y axis in the coordinate reference system of the
+input, {attr}`xdem.DEM.crs`, which might not represent the true north.
```
-```{eval-rst}
-.. minigallery:: xdem.terrain.aspect
- :add-heading:
+```{code-cell} ipython3
+aspect = dem.aspect()
+aspect.plot(cmap="twilight", cbar_title="Aspect (°)")
```
+(hillshade)=
## Hillshade
{func}`xdem.DEM.hillshade`
The hillshade is a slope map, shaded by the aspect of the slope.
-The slope map is a good tool to visualize terrain, but it does not distinguish between a mountain and a valley.
-It may therefore be slightly difficult to interpret in mountainous terrain.
-Hillshades are therefore often preferable for visualizing DEMs.
With a westerly azimuth (a simulated sun coming from the west), all eastern slopes are slightly darker.
-This mode of shading the slopes often generates a map that is much more easily interpreted than the slope map.
+This mode of shading the slopes often generates a map that is much more easily interpreted than the slope.
+
+The hillshade $hs$ is directly based on the slope $\alpha$ and aspect $\theta$, and thus also varies between the method of [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) (default) and that of [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107), and
+is often scaled between 1 and 255:
-As the hillshade is directly based on the slope and aspect, it varies between the method of [Horn (1981)](http://dx.doi.org/10.1109/PROC.1981.11918) (default) and that of [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107).
+$$
+hs = 1 + 254 \left[ sin(alt) cos(\alpha) + cos(alt) sin(\alpha) sin(2\pi - azim - \theta) \right],
+$$
+
+where $alt$ is the shading altitude (90° = from above) and $azim$ is the shading azimuth (0° = north).
Note, however, that the hillshade is not a shadow map; no occlusion is taken into account so it does not represent "true" shading.
It therefore has little analytic purpose, but it still constitutes a great visualization tool.
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_003.png
-:width: 600
-```
-
-```{eval-rst}
-.. minigallery:: xdem.terrain.hillshade
- :add-heading:
+```{code-cell} ipython3
+hillshade = dem.hillshade()
+hillshade.plot(cmap="Greys_r", cbar_title="Hillshade")
```
+(curv)=
## Curvature
{func}`xdem.DEM.curvature`
-The curvature map is the second derivative of elevation, which highlights the convexity or concavity of the terrain.
+The curvature is the second derivative of elevation, which highlights the convexity or concavity of the terrain.
If a surface is convex (like a mountain peak), it will have positive curvature.
If a surface is concave (like a through or a valley bottom), it will have negative curvature.
The curvature values in units of m{sup}`-1` are quite small, so they are by convention multiplied by 100.
-The curvature is based on the method of [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107).
+The curvature $c$ is based on the method of [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107).
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_004.png
-:width: 600
-```
+$$
-```{eval-rst}
-.. minigallery:: xdem.terrain.curvature
- :add-heading:
+c = - 100 \frac{(h_{+0} + h_{-0} + h_{0+} + h_{0-}) - 4 h_{00}}{\Delta x \Delta y}.
+
+$$
+
+```{code-cell} ipython3
+curvature = dem.curvature()
+curvature.plot(cmap="RdGy_r", cbar_title="Curvature (100 / m)", vmin=-1, vmax=1, interpolation="antialiased")
```
+(plancurv)=
## Planform curvature
{func}`xdem.DEM.planform_curvature`
-The planform curvature is the curvature perpendicular to the direction of slope, reported in 100 m{sup}`-1`.
+The planform curvature is the curvature perpendicular to the direction of slope, reported in 100 m{sup}`-1`, also based
+on [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107):
-It is based on the method of [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107).
+$$
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_005.png
-:width: 600
-```
+planc = -2\frac{DH² + EG² -FGH}{G²+H²}
-```{eval-rst}
-.. minigallery:: xdem.terrain.planform_curvature
- :add-heading:
+$$
+
+with:
+
+$$
+
+D &= \frac{h_{0+} + h_{0-} - 2h_{00}} {2\Delta y^{2}}, \\
+E &= \frac{h_{+0} + h_{-0} - 2h_{00}} {2\Delta x^{2}}, \\
+F &= \frac{h_{--} + h_{++} - h_{-+} - h_{+-}} {4 \Delta x \Delta y}, \\
+G &= \frac{h_{0-} - h_{0+}}{2\Delta y}, \\
+H &= \frac{h_{-0} - h_{+0}}{2\Delta x}.
+
+$$
+
+```{code-cell} ipython3
+planform_curvature = dem.planform_curvature()
+planform_curvature.plot(cmap="RdGy_r", cbar_title="Planform curvature (100 / m)", vmin=-1, vmax=1, interpolation="antialiased")
```
+(profcurv)=
## Profile curvature
{func}`xdem.DEM.profile_curvature`
-The profile curvature is the curvature parallel to the direction of slope, reported in 100 m{sup}`-1`..
+The profile curvature is the curvature parallel to the direction of slope, reported in 100 m{sup}`-1`, also based on
+[Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107):
-It is based on the method of [Zevenbergen and Thorne (1987)](http://dx.doi.org/10.1002/esp.3290120107).
+$$
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_006.png
-:width: 600
-```
+profc = 2\frac{DG² + EH² + FGH}{G²+H²}
-```{eval-rst}
-.. minigallery:: xdem.terrain.profile_curvature
- :add-heading:
+$$
+
+based on the equations in the planform curvature section for $D$, $E$, $F$, $G$ and $H$.
+
+```{code-cell} ipython3
+profile_curvature = dem.profile_curvature()
+profile_curvature.plot(cmap="RdGy_r", cbar_title="Profile curvature (100 / m)", vmin=-1, vmax=1, interpolation="antialiased")
```
-## Topographic Position Index
+(tpi)=
+## Topographic position index
{func}`xdem.DEM.topographic_position_index`
-The Topographic Position Index (TPI) is a metric of slope position, based on the method of [Weiss (2001)](http://www.jennessent.com/downloads/TPI-poster-TNC_18x22.pdf) that corresponds to the difference of the elevation of a central
+The topographic position index (TPI) is a metric of slope position, described in [Weiss (2001)](http://www.jennessent.com/downloads/TPI-poster-TNC_18x22.pdf), that corresponds to the difference of the elevation of a central
pixel with the average of that of neighbouring pixels. Its unit is that of the DEM (typically meters) and it can be
computed for any window size (default 3x3 pixels).
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_007.png
-:width: 600
-```
+$$
+tpi = h_{00} - \textrm{mean}_{i\neq 0, j\neq 0} (h_{ij}) .
+$$
-```{eval-rst}
-.. minigallery:: xdem.terrain.topographic_position_index
- :add-heading:
+```{code-cell} ipython3
+tpi = dem.topographic_position_index()
+tpi.plot(cmap="Spectral", cbar_title="Topographic position index (m)", vmin=-5, vmax=5)
```
-## Terrain Ruggedness Index
+(tri)=
+## Terrain ruggedness index
{func}`xdem.DEM.terrain_ruggedness_index`
-The Terrain Ruggedness Index (TRI) is a metric of terrain ruggedness, based on cumulated differences in elevation between
+The terrain ruggedness index (TRI) is a metric of terrain ruggedness, based on cumulated differences in elevation between
a central pixel and its surroundings. Its unit is that of the DEM (typically meters) and it can be computed for any
window size (default 3x3 pixels).
For topography (default), the method of [Riley et al. (1999)](http://download.osgeo.org/qgis/doc/reference-docs/Terrain_Ruggedness_Index.pdf) is generally used, where the TRI is computed by the squareroot of squared differences with
-neighbouring pixels.
+neighbouring pixels:
+
+$$
+tri_{\textrm{Riley}} = \sqrt{\sum_{ij}(h_{00} - h_{ij})^{2}}.
+$$
For bathymetry, the method of [Wilson et al. (2007)](http://dx.doi.org/10.1080/01490410701295962) is generally used,
-where the TRI is defined by the mean absolute difference with neighbouring pixels
+where the TRI is defined by the mean absolute difference with neighbouring pixels:
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_008.png
-:width: 600
-```
+$$
+tri_{\textrm{Wilson}} = \textrm{mean}_{ij} (|h_{00} - h{ij}|) .
+$$
-```{eval-rst}
-.. minigallery:: xdem.terrain.terrain_ruggedness_index
- :add-heading:
+```{code-cell} ipython3
+tri = dem.terrain_ruggedness_index()
+tri.plot(cmap="Purples", cbar_title="Terrain ruggedness index (m)")
```
+(roughness)=
## Roughness
{func}`xdem.DEM.roughness`
-The roughness is a metric of terrain ruggedness, based on the maximum difference in elevation in the surroundings.
-The roughness is based on the method of [Dartnell (2000)](http://dx.doi.org/10.14358/PERS.70.9.1081). Its unit is that of the DEM (typically meters) and it can be computed for any window size (default 3x3 pixels).
+The roughness is a metric of terrain ruggedness, based on the maximum difference in elevation in the surroundings,
+described in [Dartnell (2000)](https://environment.sfsu.edu/node/11292). Its unit is that of the DEM (typically meters) and it can be computed for any window size (default 3x3 pixels).
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_009.png
-:width: 600
-```
+$$
+r_{\textrm{D}} = \textrm{max}_{ij} (h{ij}) - \textrm{min}_{ij} (h{ij}) .
+$$
-```{eval-rst}
-.. minigallery:: xdem.terrain.roughness
- :add-heading:
+```{code-cell} ipython3
+roughness = dem.roughness()
+roughness.plot(cmap="Oranges", cbar_title="Roughness (m)")
```
+(rugosity)=
## Rugosity
{func}`xdem.DEM.rugosity`
-The rugosity is a metric of terrain ruggedness, based on the ratio between planimetric and real surface area. The
-rugosity is based on the method of [Jenness (2004)]().
+The rugosity is a metric of terrain ruggedness, based on the ratio between planimetric and real surface area,
+described in [Jenness (2004)]().
It is unitless, and is only supported for a 3x3 window size.
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_010.png
-:width: 600
-```
+$$
+r_{\textrm{J}} = \frac{\sum_{k \in [1,8]} A(T_{00, k})}{\Delta x \Delta y}.
+$$
-```{eval-rst}
-.. minigallery:: xdem.terrain.rugosity
- :add-heading:
+where $A(T_{00,k})$ is the area of one of the 8 triangles connected from the center of the center pixel $00$ to the
+centers of its 8 neighbouring pixels $k$, cropped to intersect only the center pixel. This surface area is computed in three dimensions, accounting for elevation differences.
+
+```{code-cell} ipython3
+rugosity = dem.rugosity()
+rugosity.plot(cmap="YlOrRd", cbar_title="Rugosity")
```
+(fractrough)=
## Fractal roughness
{func}`xdem.DEM.fractal_roughness`
@@ -217,13 +400,9 @@ The fractal roughness is computed by estimating the fractal dimension in 3D spac
DEM pixels. Its unit is that of a dimension, and is always between 1 (dimension of a line in 3D space) and 3
(dimension of a cube in 3D space). It can only be computed on window sizes larger than 5x5 pixels, and defaults to 13x13.
-```{image} basic_examples/images/sphx_glr_plot_terrain_attributes_011.png
-:width: 600
-```
-
-```{eval-rst}
-.. minigallery:: xdem.terrain.fractal_roughness
- :add-heading:
+```{code-cell} ipython3
+fractal_roughness = dem.fractal_roughness()
+fractal_roughness.plot(cmap="Reds", cbar_title="Fractal roughness (dimensions)")
```
## Generating multiple attributes at once
@@ -234,5 +413,4 @@ Multiple terrain attributes can be calculated from the same gradient using the {
```{eval-rst}
.. minigallery:: xdem.terrain.get_terrain_attribute
- :add-heading:
```
diff --git a/doc/source/uncertainty.md b/doc/source/uncertainty.md
index 0830e1ca..095149b6 100644
--- a/doc/source/uncertainty.md
+++ b/doc/source/uncertainty.md
@@ -1,5 +1,7 @@
---
file_format: mystnb
+mystnb:
+ execution_timeout: 90
jupytext:
formats: md:myst
text_representation:
@@ -12,369 +14,385 @@ kernelspec:
---
(uncertainty)=
-# Uncertainty analysis
-
-To analyze DEMs, xDEM integrates spatial uncertainty analysis tools from the recent DEM literature,
-in particular in [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922) and
-[Rolstad et al. (2009)](https://doi.org/10.3189/002214309789470950). The implementation of these methods relies
-partially on the package [scikit-gstat](https://mmaelicke.github.io/scikit-gstat/index.html) for spatial statistics.
-
-The uncertainty analysis tools can be used to assess the precision of DEMs (see the definition of precision in {ref}`intro`).
-In particular, they help to:
-
-> - account for elevation heteroscedasticity (e.g., varying precision such as with terrain slope or stereo-correlation),
-> - quantify the spatial correlation of errors in DEMs (e.g., native spatial resolution, instrument noise),
-> - estimate robust errors for observations analyzed in space (e.g., average or sum of elevation, or of elevation changes),
-> - propagate errors between spatial ensembles at different scales (e.g., sum of glacier volume changes).
-
-(spatialstats-intro)=
-
-## Spatial statistics for DEM precision estimation
-
-### Assumptions for statistical inference in spatial statistics
-
-
-Spatial statistics, also referred to as [geostatistics](https://en.wikipedia.org/wiki/Geostatistics), are essential
-for the analysis of observations distributed in space. Spatial statistics are valid if the variable of interest
-verifies [the assumption of second-order stationarity](https://www.aspexit.com/en/fundamental-assumptions-of-the-variogram-second-order-stationarity-intrinsic-stationarity-what-is-this-all-about/).
-That is, if the three following assumptions are verified:
-
-> 1. The mean of the variable of interest is stationary in space, i.e. constant over sufficiently large areas,
-> 2. The variance of the variable of interest is stationary in space, i.e. constant over sufficiently large areas.
-> 3. The covariance between two observations only depends on the spatial distance between them, i.e. no other factor than this distance plays a role in the spatial correlation of measurement errors.
+```{code-cell} ipython3
+:tags: [remove-cell]
-```{eval-rst}
-.. plot:: code/spatialstats_stationarity_assumption.py
- :width: 90%
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 600
+pyplot.rcParams['savefig.dpi'] = 600
```
-In other words, for a reliable analysis, the DEM should:
-
-> 1. Not contain systematic biases that do not average out over sufficiently large distances (e.g., shifts, tilts), but can contain pseudo-periodic biases (e.g., along-track undulations),
-> 2. Not contain measurement errors that vary significantly across space.
-> 3. Not contain factors that affect the spatial distribution of measurement errors, except for the distance between observations.
-
-### Quantifying the precision of a single DEM, or of a difference of DEMs
-
-To statistically infer the precision of a DEM, it is compared against independent elevation observations.
-
-Significant measurement errors can originate from both sets of elevation observations, and the analysis of differences will represent the mixed precision of the two.
-As there is no reason for a dependency between the elevation data sets, the analysis of elevation differences yields:
-
-$$
-\sigma_{dh} = \sigma_{h_{\textrm{precision1}} - h_{\textrm{precision2}}} = \sqrt{\sigma_{h_{\textrm{precision1}}}^{2} + \sigma_{h_{\textrm{precision2}}}^{2}}
-$$
-
-If the other elevation data is known to be of higher-precision, one can assume that the analysis of differences will represent only the precision of the rougher DEM.
-
-$$
-\sigma_{dh} = \sigma_{h_{\textrm{higher precision}} - h_{\textrm{lower precision}}} \approx \sigma_{h_{\textrm{lower precision}}}
-$$
-
-### Using stable terrain as a proxy
-
-Stable terrain is the terrain that has supposedly not been subject to any elevation change. It often refers to bare-rock,
-and is generally computed by simply excluding glaciers, snow and forests.
-
-Due to the sparsity of synchronous acquisitions, elevation data cannot be easily compared for simultaneous acquisition
-times. Thus, stable terrain is used a proxy to assess the precision of a DEM on all its terrain,
-including moving terrain that is generally of greater interest for analysis.
+# Uncertainty analysis
-As shown in [Hugonnet et al. (2022)](https://doi.org/10.1109/jstars.2022.3188922), accounting for {ref}`spatialstats-heterosc` is needed to reliably
-use stable terrain as a proxy for other types of terrain.
+xDEM integrates uncertainty analysis tools from the recent literature that **rely on joint methods from two
+scientific fields: spatial statistics and uncertainty quantification**.
-(spatialstats-metrics)=
+While uncertainty analysis technically refers to both systematic and random errors, systematic errors of elevation data
+are corrected using {ref}`coregistration` and {ref}`biascorr`, so we here refer to uncertainty analysis for **quantifying and
+propagating random errors** (including structured errors).
-## Metrics for DEM precision
+In detail, xDEM provides tools to:
-Historically, the precision of DEMs has been reported as a single value indicating the random error at the scale of a
-single pixel, for example $\pm 2$ meters at the 1$\sigma$ [confidence level](https://en.wikipedia.org/wiki/Confidence_interval).
+1. Estimate and model elevation **heteroscedasticity, i.e. variable random errors** (e.g., such as with terrain slope or stereo-correlation),
+2. Estimate and model the **spatial correlation of random errors** (e.g., from native spatial resolution or instrument noise),
+3. Perform **error propagation to elevation derivatives** (e.g., spatial average, or more complex derivatives such as slope and aspect).
-However, there is some limitations to this simple metric:
+:::{admonition} More reading
+:class: tip
-> - the variability of the pixel-wise precision is not reported. The pixel-wise precision can vary depending on terrain- or instrument-related factors, such as the terrain slope. In rare occurrences, part of this variability has been accounted in recent DEM products, such as TanDEM-X global DEM that partitions the precision between flat and steep slopes ([Rizzoli et al. (2017)](https://doi.org/10.1016/j.isprsjprs.2017.08.008)),
-> - the area-wise precision of a DEM is generally not reported. Depending on the inherent resolution of the DEM, and patterns of noise that might plague the observations, the precision of a DEM over a surface area can vary significantly.
+For an introduction on spatial statistics applied to uncertainty quantification for elevation data, we recommend reading
+the **{ref}`spatial-stats` guide page** and, for details on variography, the **documentation of [SciKit-GStat](https://scikit-gstat.readthedocs.io/en/latest/)**.
-### Pixel-wise elevation measurement error
+Additionally, we recommend reading the **{ref}`static-surfaces` guide page** on which uncertainty analysis relies.
+:::
-The pixel-wise measurement error corresponds directly to the dispersion $\sigma_{dh}$ of the sample $dh$.
-To estimate the pixel-wise measurement error for elevation data, two issues arise:
+## Quick use
-> 1. The dispersion $\sigma_{dh}$ cannot be estimated directly on changing terrain,
-> 2. The dispersion $\sigma_{dh}$ can show important non-stationarities.
+The estimation of the spatial structure of random errors of elevation data is conveniently
+wrapped in a single method {func}`~xdem.DEM.estimate_uncertainty`, which estimates, models and returns **a map of
+variable error** matching the DEM, and **a function describing the spatial correlation of these errors**.
-The section {ref}`spatialstats-heterosc` describes how to quantify the measurement error as a function of
-several explanatory variables by using stable terrain as a proxy.
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for opening example data and coregistering it"
+: code_prompt_hide: "Hide the code for opening example data and coregistering it"
-### Spatially-integrated elevation measurement error
+import xdem
+import matplotlib.pyplot as plt
+import geoutils as gu
+import numpy as np
-The [standard error](https://en.wikipedia.org/wiki/Standard_error) of a statistic is the dispersion of the
-distribution of this statistic. For spatially distributed samples, the standard error of the mean corresponds to the
-error of a mean (or sum) of samples in space.
+# Open two DEMs
+ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
+tba_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
-The standard error $\sigma_{\overline{dh}}$ of the mean $\overline{dh}$ of the elevation changes
-samples $dh$ can be written as:
+# Open glacier outlines as vector
+glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
-$$
-\sigma_{\overline{dh}} = \frac{\sigma_{dh}}{\sqrt{N}},
-$$
+# Create a stable ground mask (not glacierized) to mark "inlier data"
+inlier_mask = ~glacier_outlines.create_mask(ref_dem)
+tba_dem_coreg = tba_dem.coregister_3d(ref_dem, inlier_mask=inlier_mask)
+```
-where $\sigma_{dh}$ is the dispersion of the samples, and $N$ is the number of **independent** observations.
+```{code-cell} ipython3
+# Estimate elevation uncertainty assuming both DEMs have similar precision
+sig_dem, rho_sig = tba_dem_coreg.estimate_uncertainty(ref_dem, stable_terrain=inlier_mask, precision_of_other="same")
-To estimate the standard error of the mean for elevation data, two issue arises:
+# The error map variability is estimated from slope and curvature by default
+sig_dem.plot(cmap="Purples", cbar_title=r"Error in elevation (1$\sigma$, m)")
-> 1. The dispersion of elevation differences $\sigma_{dh}$ is not stationary, a necessary assumption for spatial statistics.
-> 2. The number of pixels in the DEM $N$ does not equal the number of independent observations in the DEMs, because of spatial correlations.
+# The spatial correlation function represents how much errors are correlated at a certain distance
+print("Random elevation errors at a distance of 1 km are correlated at {:.2f} %.".format(rho_sig(1000) * 100))
+```
-The sections {ref}`spatialstats-corr` and {ref}`spatialstats-errorpropag` describe how to account for spatial correlations
-and use those to integrate and propagate measurement errors in space.
+Three methods can be considered for this estimation, which are described right below.
+Additionally, the subfunctions used to perform the uncertainty analysis are detailed in **the {ref}`error-struc` section** below.
+
+## Summary of available methods
+
+Methods for modelling the structure of error are based on [spatial statistics](https://en.wikipedia.org/wiki/Spatial_statistics), and methods for
+propagating errors to spatial derivatives analytically rely on [uncertainty propagation](https://en.wikipedia.org/wiki/Propagation_of_uncertainty).
+
+To improve the robustness of the uncertainty analysis, we provide refined frameworks for application to elevation data based on
+[Rolstad et al. (2009)](http://dx.doi.org/10.3189/002214309789470950) and [Hugonnet et al. (2022)](http://dx.doi.org/10.1109/JSTARS.2022.3188922),
+both for modelling the structure of error and to efficiently perform error propagation.
+**These frameworks are generic, simply extending an aspect of the uncertainty analysis to better work on elevation data**,
+and thus generally encompass methods described in other studies on the topic (e.g., [Anderson et al. (2019)](http://dx.doi.org/10.1002/esp.4551)).
+
+The tables below summarize the characteristics of these methods.
+
+### Estimating and modelling the structure of error
+
+Frequently, in spatial statistics, a single correlation range is considered ("basic" method below).
+However, elevation data often contains errors with correlation ranges spanning different orders of magnitude.
+For this, [Rolstad et al. (2009)](http://dx.doi.org/10.3189/002214309789470950) and
+[Hugonnet et al. (2022)](http://dx.doi.org/10.1109/JSTARS.2022.3188922) consider
+potential multiple ranges of spatial correlation (instead of a single one). In addition, [Hugonnet et al. (2022)](http://dx.doi.org/10.1109/JSTARS.2022.3188922)
+considers potential heteroscedasticity or variable errors (instead of homoscedasticity, or constant errors), also common in elevation data.
+
+Because accounting for possible multiple correlation ranges also works if you have a single correlation range in your data,
+and accounting for potential heteroscedasticity also works on homoscedastic data, **there is little to lose by using
+a more advanced framework! (most often, only a bit of additional computation time)**
+
+```{list-table}
+ :widths: 1 1 1 1
+ :header-rows: 1
+ :stub-columns: 1
+ :align: center
+
+ * - Method
+ - Heteroscedasticity (i.e. variable error)
+ - Correlations (single-range)
+ - Correlations (multi-range)
+ * - Basic
+ - ❌
+ - ✅
+ - ❌
+ * - R2009
+ - ❌
+ - ✅
+ - ✅
+ * - H2022 (default)
+ - ✅
+ - ✅
+ - ✅
+```
-## Workflow for DEM precision estimation
+For consistency, all methods default to robust estimators: the normalized median absolute deviation (NMAD) for the
+spread, and Dowd's estimator for the variogram. See the **{ref}`robust-estimators` guide page** for details.
+
+### Propagating errors to spatial derivatives
+
+Exact uncertainty propagation scales exponentially with data (by computing every pairwise combinations,
+for potentially millions of elevation data points or pixels).
+To remedy this, [Rolstad et al. (2009)](http://dx.doi.org/10.3189/002214309789470950) and [Hugonnet et al. (2022)](http://dx.doi.org/10.1109/JSTARS.2022.3188922)
+both provide an approximation of exact uncertainty propagations for spatial derivatives (to avoid long
+computing times). **These approximations are valid in different contexts**, described below.
+
+```{list-table}
+ :widths: 1 1 1 1
+ :header-rows: 1
+ :stub-columns: 1
+ :align: center
+
+ * - Method
+ - Accuracy
+ - Computing time
+ - Validity
+ * - Exact discretized
+ - Exact
+ - Slow on large samples (exponential complexity)
+ - Always
+ * - R2009
+ - Conservative
+ - Instantaneous (numerical integration)
+ - Only for near-circular contiguous areas
+ * - H2022 (default)
+ - Accurate
+ - Fast (linear complexity)
+ - As long as variance is nearly stationary
+```
(spatialstats-heterosc)=
-### Elevation heteroscedasticity
+## Core concept for error proxy
-Elevation data contains significant variability in measurement errors.
+Below, we examplify the different steps of uncertainty analysis of **elevation differences between two datasets on
+static surfaces as an error proxy**.
-xDEM provides tools to **quantify** this variability using explanatory variables, **model** those numerically to
-estimate a function predicting elevation error, and **standardize** data for further analysis.
+In case you want to **convert the uncertainties of elevation differences into that of a "target" elevation dataset**, it can be either assumed that:
+- **The "other" elevation dataset is much more precise**, in which case the uncertainties in elevation differences directly approximate that of the "target" elevation dataset,
+- **The "other" elevation dataset has similar precision**, in which case the uncertainties of elevation differences quadratically combine twice that of the "target" elevation dataset.
-#### Quantify and model heteroscedasticity
+:::{admonition} More reading (reminder)
+:class: tip
-Elevation [heteroscedasticity](https://en.wikipedia.org/wiki/Heteroscedasticity) corresponds to a variability in
-precision of elevation observations, that are linked to terrain or instrument variables.
+To clarify these conversions of error proxy, see the **{ref}`static-surfaces` guide page**.
+For more statistical background on the methods below, see the **{ref}`spatial-stats` guide page**.
+:::
-$$
-\sigma_{dh} = \sigma_{dh}(\textrm{var}_{1},\textrm{var}_{2}, \textrm{...}) \neq \textrm{constant}
-$$
+(error-struc)=
+## Spatial structure of error
-Owing to the large number of samples of elevation data, we can easily estimate this variability by [binning](https://en.wikipedia.org/wiki/Data_binning) the data and estimating the statistical dispersion (see
-{ref}`robuststats-meanstd`) across several explanatory variables using {func}`xdem.spatialstats.nd_binning`.
+Below we detail the steps used to estimate the two components of uncertainty: heteroscedasticity and spatial
+correlation of errors in {func}`~xdem.DEM.estimate_uncertainty`, as these are most easily customized
+by calling their subfunctions independently.
+```{important}
+Some uncertainty functionalities are **being adapted to operate directly in SciKit-GStat** (e.g., fitting a sum of
+variogram models, pairwise subsampling for grid data). This will allow to simplify function inputs and outputs of xDEM,
+for instance by relying on a single, consistent {func}`~skgstat.Variogram` object.
-```{code-cell} ipython3
-:tags: [hide-input, hide-output]
-import geoutils as gu
-import numpy as np
+This will trigger API changes in future package versions.
+```
-import xdem
+### Heteroscedasticity
-# Load data
-dh = gu.Raster(xdem.examples.get_path("longyearbyen_ddem"))
-ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
-glacier_mask = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
-mask = glacier_mask.create_mask(dh)
+The first component of uncertainty is the estimation and modelling of elevation
+[heteroscedasticity](https://en.wikipedia.org/wiki/Heteroscedasticity) (or variability in
+random elevation errors) through {func}`~xdem.spatialstats.infer_heteroscedasticity_from_stable`, which has three steps.
-slope = xdem.terrain.get_terrain_attribute(ref_dem, attribute=["slope"])
+**Step 1: Empirical estimation of heteroscedasticity**
-# Keep only stable terrain data
-dh.load()
-dh.set_mask(mask)
-dh_arr = gu.raster.get_array_and_mask(dh)[0]
-slope_arr = gu.raster.get_array_and_mask(slope)[0]
+The variability in errors is empirically estimated by [data binning](https://en.wikipedia.org/wiki/Data_binning)
+in N-dimensions of the elevation differences on stable terrain, using the function {func}`~xdem.spatialstats.nd_binning`.
+Plotting of 1- and 2D binnings can be facilitated by the functions {func}`~xdem.spatialstats.plot_1d_binning` and
+{func}`~xdem.spatialstats.plot_2d_binning`.
-# Subsample to run the snipped code faster
-indices = gu.raster.subsample_array(dh_arr, subsample=10000, return_indices=True, random_state=42)
-dh_arr = dh_arr[indices]
-slope_arr = slope_arr[indices]
-```
+The most common explanatory variables for elevation heteroscedasticity are the terrain slope and curvature (used as
+default, see {ref}`terrain-attributes`), and other quality metrics passed by the user such as the correlation
+(for [stereo](https://en.wikipedia.org/wiki/Photogrammetry#Stereophotogrammetry) DEMs)
+or the interferometric coherence (for [InSAR](https://en.wikipedia.org/wiki/Interferometric_synthetic-aperture_radar) DEMs).
```{code-cell} ipython3
-# Estimate the measurement error by bin of slope, using the NMAD as robust estimator
-df_ns = xdem.spatialstats.nd_binning(
- dh_arr, list_var=[slope_arr], list_var_names=["slope"], statistics=["count", xdem.spatialstats.nmad]
+# Get elevation differences and stable terrain mask
+dh = ref_dem - tba_dem_coreg
+glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
+stable_terrain = ~glacier_outlines.create_mask(dh)
+
+# Derive slope and curvature
+slope, curv = ref_dem.get_terrain_attribute(attribute=["slope", "curvature"])
+
+# Use only array of stable terrain
+dh_arr = dh[stable_terrain]
+slope_arr = slope[stable_terrain]
+curv_arr = curv[stable_terrain]
+
+# Estimate the variable error by bin of slope and curvature
+df_h = xdem.spatialstats.nd_binning(
+ dh_arr, list_var=[slope_arr, curv_arr], list_var_names=["slope", "curv"], statistics=["count", xdem.spatialstats.nmad], list_var_bins=[np.linspace(0, 60, 10), np.linspace(-10, 10, 10)]
)
-```
-```{eval-rst}
-.. plot:: code/spatialstats_heterosc_slope.py
- :width: 90%
+# Plot 2D binning
+xdem.spatialstats.plot_2d_binning(df_h, "slope", "curv", "nmad", "Slope (degrees)", "Curvature (100 m-1)", "NMAD (m)")
```
-The most common explanatory variables are:
-
-> - the terrain slope and terrain curvature (see {ref}`terrain-attributes`) that can explain a large part of the terrain-related variability in measurement error,
-> - the quality of stereo-correlation that can explain a large part of the measurement error of DEMs generated by stereophotogrammetry,
-> - the interferometric coherence that can explain a large part of the measurement error of DEMs generated by [InSAR](https://en.wikipedia.org/wiki/Interferometric_synthetic-aperture_radar).
+**Step 2: Modelling of the heteroscedasticity**
-Once quantified, elevation heteroscedasticity can be modelled numerically by linear interpolation across several
-variables using {func}`xdem.spatialstats.interp_nd_binning`.
+Once empirically estimated, elevation heteroscedasticity can be modelled either by a function fit, or by
+N-D linear interpolation using {func}`~xdem.spatialstats.interp_nd_binning`, in order to yield a value for any slope
+and curvature:
```{code-cell} ipython3
# Derive a numerical function of the measurement error
-err_dh = xdem.spatialstats.interp_nd_binning(df_ns, list_var_names=["slope"])
+sig_dh_func = xdem.spatialstats.interp_nd_binning(df_h, list_var_names=["slope", "curv"])
```
-#### Standardize elevation differences for further analysis
+**Step 3: Applying the model**
+
+Using the model, we can estimate the random error on all terrain using their slope
+and curvature, and derive a map of random errors in elevation change:
-In order to verify the assumptions of spatial statistics and be able to use stable terrain as a reliable proxy in
-further analysis (see {ref}`spatialstats-intro`), [standardization](https://en.wikipedia.org/wiki/Standard_score)
-of the elevation differences are required to reach a stationary variance.
+```{code-cell} ipython3
+# Apply function to the slope and curvature on all terrain
+sig_dh_arr = sig_dh_func((slope.data, curv.data))
-```{eval-rst}
-.. plot:: code/spatialstats_standardizing.py
- :width: 90%
+# Convert to raster and plot
+sig_dh = dh.copy(new_array=sig_dh_arr)
+sig_dh.plot(cmap="Purples", cbar_title=r"Random error in elevation change (1$\sigma$, m)")
```
-For application to DEM precision estimation, the mean is already centered on zero and the variance is non-stationary,
-which yields:
+### Spatial correlation of errors
-$$
-z_{dh} = \frac{dh(\textrm{var}_{1}, \textrm{var}_{2}, \textrm{...})}{\sigma_{dh}(\textrm{var}_{1}, \textrm{var}_{2}, \textrm{...})}
-$$
+The second component of uncertainty is the estimation and modelling of spatial correlations of random errors through
+{func}`~xdem.spatialstats.infer_spatial_correlation_from_stable`, which has three steps.
-where $z_{dh}$ is the standardized elevation difference sample.
+**Step 1: Standardization**
-Code-wise, standardization is as simple as a division of the elevation differences `dh` using the estimated measurement
-error:
+If heteroscedasticity was considered, elevation differences can be standardized by the variable error to
+reduce its influence on the estimation of spatial correlations. Otherwise, elevation differences are used directly.
```{code-cell} ipython3
# Standardize the data
-z_dh = dh_arr / err_dh(slope_arr)
+z_dh = dh / sig_dh
+# Mask values to keep only stable terrain
+z_dh.set_mask(~stable_terrain)
+# Plot the standardized data on stable terrain
+z_dh.plot(cmap="RdBu", vmin=-3, vmax=3, cbar_title="Standardized elevation changes (unitless)")
```
-To later de-standardize estimations of the dispersion of a given subsample of elevation differences,
-possibly after further analysis of {ref}`spatialstats-corr` and {ref}`spatialstats-errorpropag`,
-one simply needs to apply the opposite operation.
+**Step 2: Empirical estimation of the variogram**
-For a single pixel $\textrm{P}$, the dispersion is directly the elevation measurement error evaluated for the
-explanatory variable of this pixel as, per construction, $\sigma_{z_{dh}} = 1$:
+An empirical variogram can be estimated with {func}`~xdem.spatialstats.sample_empirical_variogram`.
-$$
-\sigma_{dh}(\textrm{P}) = 1 \cdot \sigma_{dh}(\textrm{var}_{1}(\textrm{P}), \textrm{var}_{2}(\textrm{P}), \textrm{...})
-$$
-
-For a mean of pixels $\overline{dh}\vert_{\mathbb{S}}$ in the subsample $\mathbb{S}$, the standard error of the mean
-of the standardized data $\overline{\sigma_{z_{dh}}}\vert_{\mathbb{S}}$ can be de-standardized by multiplying by the
-average measurement error of the pixels in the subsample, evaluated through the explanatory variables of each pixel:
-
-$$
-\sigma_{\overline{dh}}\vert_{\mathbb{S}} = \sigma_{\overline{z_{dh}}}\vert_{\mathbb{S}} \cdot \overline{\sigma_{dh}(\textrm{var}_{1}, \textrm{var}_{2}, \textrm{...})}\vert_{\mathbb{S}}
-$$
-
-Estimating the standard error of the mean of the standardized data $\sigma_{\overline{z_{dh}}}\vert_{\mathbb{S}}$
-requires an analysis of spatial correlation and a spatial integration of this correlation, described in the next sections.
-
-```{eval-rst}
-.. minigallery:: xdem.spatialstats.infer_heteroscedasticity_from_stable xdem.spatialstats.nd_binning
- :add-heading: Examples that deal with elevation heteroscedasticity
- :heading-level: "
+```{code-cell} ipython3
+# Sample empirical variogram
+df_vgm = xdem.spatialstats.sample_empirical_variogram(values=z_dh, subsample=500, n_variograms=5, random_state=42)
```
-(spatialstats-corr)=
-
-### Spatial correlation of elevation measurement errors
-
-Spatial correlation of elevation measurement errors correspond to a dependency between measurement errors of spatially
-close pixels in elevation data. Those can be related to the resolution of the data (short-range correlation), or to
-instrument noise and deformations (mid- to long-range correlations).
+**Step 3: Modelling of the variogram**
-xDEM provides tools to **quantify** these spatial correlation with pairwise sampling optimized for grid data and to
-**model** correlations simultaneously at multiple ranges.
+Once empirically estimated, the variogram can be modelled by a functional form with {func}`~xdem.spatialstats.fit_sum_model_variogram`.
+Plotting of the empirical and modelled variograms is facilitated by {func}`~xdem.spatialstats.plot_variogram`.
-#### Quantify spatial correlations
-
-[Variograms](https://en.wikipedia.org/wiki/Variogram) are functions that describe the spatial correlation of a sample.
-The variogram $2\gamma(h)$ is a function of the distance between two points, referred to as spatial lag $l$
-(usually noted $h$, here avoided to avoid confusion with the elevation and elevation differences).
-The output of a variogram is the correlated variance of the sample.
+```{code-cell} ipython3
+# Fit the sum of a gaussian and spherical model
+func_sum_vgm, params_variogram_model = xdem.spatialstats.fit_sum_model_variogram(
+ list_models=["Gaussian", "Spherical"], empirical_variogram=df_vgm
+)
+# Plot empirical and modelled variogram
+xdem.spatialstats.plot_variogram(df_vgm, [func_sum_vgm], ["Sum of gaussian and spherical"], xscale="log")
+```
-$$
-2\gamma(l) = \textrm{var}\left(Z(\textrm{s}_{1}) - Z(\textrm{s}_{2})\right)
-$$
+## Propagation of errors
-where $Z(\textrm{s}_{i})$ is the value taken by the sample at location $\textrm{s}_{i}$, and sample positions
-$\textrm{s}_{1}$ and $\textrm{s}_{2}$ are separated by a distance $l$.
+The two uncertainty components estimated above allow to propagate elevation errors.
+xDEM provides methods to theoretically propagate errors to spatial derivatives (mean or sum in an area), with efficient
+computing times.
+For more complex derivatives (such as terrain attributes), we recommend to combine the structure of error
+defined above with random field simulation methods available in packages such as [GSTools](https://geostat-framework.readthedocs.io/projects/gstools/en/stable/).
-For elevation differences $dh$, this translates into:
+### Spatial derivatives
-$$
-2\gamma_{dh}(l) = \textrm{var}\left(dh(\textrm{s}_{1}) - dh(\textrm{s}_{2})\right)
-$$
+The propagation of random errors to a spatial derivative is done with
+{func}`~xdem.spatialstats.spatial_error_propagation`, which divides into three steps.
-The variogram essentially describes the spatial covariance $C$ in relation to the variance of the entire sample
-$\sigma_{dh}^{2}$:
+Each step derives a part of the standard error in the area.
+For example, for the error of the mean elevation difference $\sigma_{\overline{dh}}$:
$$
-\gamma_{dh}(l) = \sigma_{dh}^{2} - C_{dh}(l)
+\sigma_{\overline{dh}} = \frac{\overline{\sigma_{dh}}}{\sqrt{N_{eff}}}
$$
-```{eval-rst}
-.. plot:: code/spatialstats_variogram_covariance.py
- :width: 90%
-```
-
-Empirical variograms are variograms estimated directly by [binned](https://en.wikipedia.org/wiki/Data_binning) analysis
-of variance of the data. Historically, empirical variograms were estimated for point data by calculating all possible
-pairwise differences in the samples. This amounts to $N^2$ pairwise calculations for $N$ samples, which is
-not well-suited to grid data that contains many millions of points and would be impossible to comupute. Thus, in order
-to estimate a variogram for large grid data, subsampling is necessary.
-
-Random subsampling of the grid samples used is a solution, but often unsatisfactory as it creates a clustering
-of pairwise samples that unevenly represents lag classes (most pairwise differences are found at mid distances, but too
-few at short distances and long distances).
-
-To remedy this issue, xDEM provides {func}`xdem.spatialstats.sample_empirical_variogram`, an empirical variogram estimation tool
-that encapsulates a pairwise subsampling method described in `skgstat.MetricSpace.RasterEquidistantMetricSpace`.
-This method compares pairwise distances between a center subset and equidistant subsets iteratively across a grid, based on
-[sparse matrices](https://en.wikipedia.org/wiki/Sparse_matrix) routines computing pairwise distances of two separate
-subsets, as in [scipy.cdist](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html)
-(instead of using pairwise distances within the same subset, as implemented in most spatial statistics packages).
-The resulting pairwise differences are evenly distributed across the grid and across lag classes (in 2 dimensions, this
-means that lag classes separated by a factor of $\sqrt{2}$ have an equal number of pairwise differences computed).
-
```{code-cell} ipython3
-# Sample empirical variogram
-df_vgm = xdem.spatialstats.sample_empirical_variogram(values=dh, subsample=10, random_state=42)
+# Get an area of interest where we want to propagate errors
+outline_brom = gu.Vector(glacier_outlines.ds[glacier_outlines.ds["NAME"] == "Brombreen"])
+mask_brom = outline_brom.create_mask(dh)
```
-The variogram is returned as a {class}`~pandas.DataFrame` object.
-
-With all spatial lags sampled evenly, estimating a variogram requires significantly less samples, increasing the
-robustness of the spatial correlation estimation and decreasing computing time!
-
-#### Model spatial correlations
-
-Once an empirical variogram is estimated, fitting a function model allows to simplify later analysis by directly
-providing a function form (e.g., for kriging equations, or uncertainty analysis - see {ref}`spatialstats-errorpropag`),
-which would otherwise have to be numerically modelled.
+**Step 1: Account for variable error**
-Generally, in spatial statistics, a single model is used to describe the correlation in the data.
-In elevation data, however, spatial correlations are observed at different scales, which requires fitting a sum of models at
-multiple ranges (introduced in [Rolstad et al. (2009)](https://doi.org/10.3189/002214309789470950) for glaciology
-applications).
-
-This can be performed through the function {func}`xdem.spatialstats.fit_sum_model_variogram`, which expects as input a
-`pd.Dataframe` variogram.
+We compute the mean of the variable random error in the area $\overline{\sigma_{dh}}$.
```{code-cell} ipython3
-# Fit sum of double-range spherical model
-func_sum_vgm, params_variogram_model = xdem.spatialstats.fit_sum_model_variogram(
- list_models=["Gaussian", "Spherical"], empirical_variogram=df_vgm
-)
+# Calculate the mean random error in the area
+mean_sig = np.nanmean(sig_dh[mask_brom])
```
-```{eval-rst}
-.. minigallery:: xdem.spatialstats.infer_spatial_correlation_from_stable xdem.spatialstats.sample_empirical_variogram
- :add-heading: Examples that deal with spatial correlations
- :heading-level: "
-```
+**Step 2: Account for spatial correlation**
-(spatialstats-errorpropag)=
+We estimate the number of effective samples in the area $N_{eff}$ due to the spatial correlations.
-### Spatially integrated measurement errors
+```{note}
+:class: margin
-After quantifying and modelling spatial correlations, those an effective sample size, and elevation measurement error:
+**We notice a warning below:** The resolution for rasterizing the outline was automatically chosen based on the short correlation range.
+```
```{code-cell} ipython3
+---
+mystnb:
+ output_stderr: show
+---
# Calculate the area-averaged uncertainty with these models
-neff = xdem.spatialstats.number_effective_samples(area=1000, params_variogram_model=params_variogram_model)
+neff = xdem.spatialstats.number_effective_samples(area=outline_brom, params_variogram_model=params_variogram_model)
```
-TODO: Add this section based on Rolstad et al. (2009), Hugonnet et al. (in prep)
+**Step 3: Derive final error**
-### Propagation of correlated errors
+And we can now compute our final random error for the mean elevation change in this area of interest:
-TODO: Add this section based on Krige's relation (Webster & Oliver, 2007), Hugonnet et al. (in prep)
+```{code-cell} ipython3
+# Compute the standard error
+sig_dh_brom = mean_sig / np.sqrt(neff)
+
+# Mean elevation difference
+dh_brom = np.nanmean(dh[mask_brom])
+
+# Plot the result
+dh.plot(cmap="RdYlBu", cbar_title="Elevation differences (m)")
+outline_brom.plot(dh, fc="none", ec="black", lw=2)
+plt.text(
+ outline_brom.ds.centroid.x.values[0],
+ outline_brom.ds.centroid.y.values[0] - 1500,
+ f"{dh_brom:.2f} \n$\\pm$ {sig_dh_brom:.2f} m",
+ color="black",
+ fontweight="bold",
+ va="top",
+ ha="center",
+)
+plt.show()
+```
diff --git a/doc/source/vertical_ref.md b/doc/source/vertical_ref.md
index 76fdba5e..3c49f2de 100644
--- a/doc/source/vertical_ref.md
+++ b/doc/source/vertical_ref.md
@@ -12,15 +12,64 @@ kernelspec:
---
(vertical-ref)=
+```{code-cell} ipython3
+:tags: [remove-cell]
+
+# To get a good resolution for displayed figures
+from matplotlib import pyplot
+pyplot.rcParams['figure.dpi'] = 600
+pyplot.rcParams['savefig.dpi'] = 600
+```
+
# Vertical referencing
-xDEM supports the use of **vertical coordinate reference systems (vertical CRSs)** and vertical transformations for DEMs
+xDEM supports the use of **vertical coordinate reference systems (vertical CRSs) and vertical transformations for elevation data**
by conveniently wrapping PROJ pipelines through [Pyproj](https://pyproj4.github.io/pyproj/stable/) in the {class}`~xdem.DEM` class.
-```{important}
+```{note}
**A {class}`~xdem.DEM` already possesses a {class}`~xdem.DEM.crs` attribute that defines its 2- or 3D CRS**, inherited from
{class}`~geoutils.Raster`. Unfortunately, most DEM products do not yet come with a 3D CRS in their raster metadata, and
vertical CRSs often have to be set by the user. See {ref}`vref-setting` below.
+
+For more reading on referencing for elevation data, see the **{ref}`elevation-intricacies` guide page.**
+```
+
+## Quick use
+
+The parsing, setting and transformation of vertical CRSs revolves around **three methods**, all described in details further below:
+- The **instantiation** of {class}`~xdem.DEM` that implicitly tries to set the vertical CRS from the metadata (or explicitly through the `vcrs` argument),
+- The **setting** method {func}`~xdem.DEM.set_vcrs` to explicitly set the vertical CRS of a {class}`~xdem.DEM`,
+- The **transformation** method {func}`~xdem.DEM.to_vcrs` to explicitly transform the vertical CRS of a {class}`~xdem.DEM`.
+
+```{caution}
+As of now, **[Rasterio](https://rasterio.readthedocs.io/en/stable/) does not support vertical transformations during CRS reprojection** (even if the CRS
+provided contains a vertical axis).
+We therefore advise to perform horizontal transformation and vertical transformation independently using {func}`DEM.reproject` and {func}`DEM.to_vcrs`, respectively.
+```
+
+To pass a vertical CRS argument, xDEM accepts string of the most commonly used (`"EGM96"`, `"EGM08"` and `"Ellipsoid"`),
+any {class}`pyproj.crs.CRS` objects and any PROJ grid name (available at [https://cdn.proj.org/](https://cdn.proj.org/)) which is **automatically downloaded**.
+
+```{code-cell} ipython3
+:tags: [hide-cell]
+:mystnb:
+: code_prompt_show: "Show the code for opening example data"
+: code_prompt_hide: "Hide the code for opening example data"
+
+import xdem
+import matplotlib.pyplot as plt
+
+ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
+```
+
+```{code-cell} ipython3
+# Set current vertical CRS
+ref_dem.set_vcrs("EGM96")
+# Transform to a local reference system from https://cdn.proj.org/
+trans_dem = ref_dem.to_vcrs("no_kv_arcgp-2006-sk.tif")
+
+# Plot the elevation differences of the vertical transformation
+(trans_dem - ref_dem).plot(cmap='RdYlBu', cbar_title="Elevation differences of\n vertical transform (m)")
```
## What is a vertical CRS?
@@ -37,19 +86,6 @@ In xDEM, we merge these into a single vertical CRS attribute {class}`DEM.vcrs` with only a vertical axis is either a {class}`~pyproj.crs.BoundCRS` (when created from a grid) or a {class}`~pyproj.crs.VerticalCRS` (when created in any other manner).
-## Methods to manipulate vertical CRSs
-
-The parsing, setting and transformation of vertical CRSs revolves around **three methods**, all described in details further below:
-- The **instantiation** of {class}`~xdem.DEM` that implicitly tries to set the vertical CRS from the metadata (or explicitly through the `vcrs` argument),
-- The **setting** method {func}`~xdem.DEM.set_vcrs` to explicitly set the vertical CRS of a {class}`~xdem.DEM`,
-- The **transformation** method {func}`~xdem.DEM.to_vcrs` to explicitly transform the vertical CRS of a {class}`~xdem.DEM`.
-
-```{caution}
-As of now, **[Rasterio](https://rasterio.readthedocs.io/en/stable/) does not support vertical transformations during CRS reprojection** (even if the CRS
-provided contains a vertical axis).
-We therefore advise to perform horizontal transformation and vertical transformation independently using {func}`DEM.reproject` and {func}`DEM.to_vcrs`, respectively.
-```
-
(vref-setting)=
## Automated vertical CRS detection
@@ -97,7 +133,7 @@ os.remove("mydem_with3dcrs.tif")
2. **From the {attr}`~xdem.DEM.product` name of the DEM**, if parsed from the filename by {class}`geoutils.SatelliteImage`.
-```{see-also}
+```{seealso}
The {class}`~geoutils.SatelliteImage` parent class that parses the product metadata is described in [a dedicated section of GeoUtils' documentation](https://geoutils.readthedocs.io/en/latest/satimg_class.html).
```
@@ -217,7 +253,7 @@ To transform a {class}`~xdem.DEM` to a different vertical CRS, {func}`~xdem.DEM.
```{note}
If your transformation requires a grid that is not available locally, it will be **downloaded automatically**.
-xDEM uses only the best available (i.e. best accuracy) transformation returned by {class}`pyproj.transformer.TransformerGroup`, considering the area-of-interest as the DEM extent {class}`~xdem.DEM.bounds`.
+xDEM uses only the best available (i.e. best accuracy) transformation returned by {class}`pyproj.transformer.TransformerGroup`.
```
```{code-cell} ipython3
diff --git a/environment.yml b/environment.yml
index 8f5279c5..70613281 100644
--- a/environment.yml
+++ b/environment.yml
@@ -2,7 +2,7 @@ name: xdem
channels:
- conda-forge
dependencies:
- - python>=3.9,<3.12
+ - python>=3.10,<3.13
- geopandas>=0.12.0
- numba=0.*
- numpy=1.*
@@ -13,7 +13,7 @@ dependencies:
- tqdm
- scikit-image=0.*
- scikit-gstat>=1.0.18,<1.1
- - geoutils=0.1.9
+ - geoutils=0.1.10
- pip
# To run CI against latest GeoUtils
diff --git a/examples/advanced/plot_blockwise_coreg.py b/examples/advanced/plot_blockwise_coreg.py
index 873d3716..41e1010a 100644
--- a/examples/advanced/plot_blockwise_coreg.py
+++ b/examples/advanced/plot_blockwise_coreg.py
@@ -25,7 +25,7 @@
import xdem
# %%
-# **Example files**
+# We open example files.
reference_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
dem_to_be_aligned = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
@@ -48,7 +48,7 @@
diff_before = reference_dem - dem_to_be_aligned
-diff_before.plot(cmap="coolwarm_r", vmin=-10, vmax=10)
+diff_before.plot(cmap="RdYlBu", vmin=-10, vmax=10)
plt.show()
# %%
@@ -70,9 +70,7 @@
# Coregistration is performed with the ``.fit()`` method.
# This runs in multiple threads by default, so more CPU cores are preferable here.
-blockwise.fit(reference_dem, dem_to_be_aligned, inlier_mask=inlier_mask)
-
-aligned_dem = blockwise.apply(dem_to_be_aligned)
+aligned_dem = blockwise.fit_and_apply(reference_dem, dem_to_be_aligned, inlier_mask=inlier_mask)
# %%
# The estimated shifts can be visualized by applying the coregistration to a completely flat surface.
@@ -83,7 +81,7 @@
np.zeros_like(dem_to_be_aligned.data), transform=dem_to_be_aligned.transform, crs=dem_to_be_aligned.crs
)[0]
plt.title("Vertical correction")
-plt.imshow(z_correction, cmap="coolwarm_r", vmin=-10, vmax=10, extent=plt_extent)
+plt.imshow(z_correction, cmap="RdYlBu", vmin=-10, vmax=10, extent=plt_extent)
for _, row in blockwise.stats().iterrows():
plt.annotate(round(row["z_off"], 1), (row["center_x"], row["center_y"]), ha="center")
@@ -92,11 +90,12 @@
diff_after = reference_dem - aligned_dem
-diff_after.plot(cmap="coolwarm_r", vmin=-10, vmax=10)
+diff_after.plot(cmap="RdYlBu", vmin=-10, vmax=10)
plt.show()
# %%
-# We can compare the NMAD to validate numerically that there was an improvment:
+# We can compare the NMAD to validate numerically that there was an improvement:
+
-print(f"Error before: {xdem.spatialstats.nmad(diff_before):.2f} m")
-print(f"Error after: {xdem.spatialstats.nmad(diff_after):.2f} m")
+print(f"Error before: {xdem.spatialstats.nmad(diff_before[inlier_mask]):.2f} m")
+print(f"Error after: {xdem.spatialstats.nmad(diff_after[inlier_mask]):.2f} m")
diff --git a/examples/advanced/plot_demcollection.py b/examples/advanced/plot_demcollection.py
index 5d57ca53..c3b3fa94 100644
--- a/examples/advanced/plot_demcollection.py
+++ b/examples/advanced/plot_demcollection.py
@@ -85,8 +85,6 @@
scott_extent = [518600, 523800, 8666600, 8672300]
-plt.figure(figsize=(8, 5))
-
for i in range(2):
plt.subplot(1, 2, i + 1)
@@ -98,8 +96,9 @@
# The 2009 - 2060 DEM is inverted since the reference year is 2009
ddem_2060 = -demcollection.ddems[2].data.squeeze()
- plt.imshow(ddem_2060, cmap="coolwarm_r", vmin=-50, vmax=50, extent=extent)
+ plt.imshow(ddem_2060, cmap="RdYlBu", vmin=-50, vmax=50, extent=extent)
plt.xlim(scott_extent[:2])
plt.ylim(scott_extent[2:])
plt.show()
+plt.tight_layout()
diff --git a/examples/advanced/plot_deramp.py b/examples/advanced/plot_deramp.py
index 20c4f205..5047993a 100644
--- a/examples/advanced/plot_deramp.py
+++ b/examples/advanced/plot_deramp.py
@@ -1,11 +1,11 @@
"""
-Bias correction with deramping
+Bias-correction with deramping
==============================
-(On latest only) Update will follow soon with more consistent bias correction examples.
-In ``xdem``, this approach is implemented through the :class:`xdem.biascorr.Deramp` class.
+Deramping can help correct rotational or doming errors in elevation data.
+In xDEM, this approach is implemented through the :class:`xdem.coreg.Deramp` class.
-For more information about the approach, see :ref:`biascorr-deramp`.
+See also the :ref:`deramp` section in feature pages.
"""
import geoutils as gu
import numpy as np
@@ -13,7 +13,7 @@
import xdem
# %%
-# **Example files**
+# We open example files.
reference_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
dem_to_be_aligned = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
@@ -22,11 +22,10 @@
inlier_mask = ~glacier_outlines.create_mask(reference_dem)
# %%
-# The DEM to be aligned (a 1990 photogrammetry-derived DEM) has some vertical and horizontal biases that we want to avoid.
-# These can be visualized by plotting a change map:
+# We visualize the patterns of error from the elevation differences.
diff_before = reference_dem - dem_to_be_aligned
-diff_before.plot(cmap="coolwarm_r", vmin=-10, vmax=10, cbar_title="Elevation change (m)")
+diff_before.plot(cmap="RdYlBu", vmin=-10, vmax=10, cbar_title="Elevation differences (m)")
# %%
@@ -34,23 +33,22 @@
deramp = xdem.coreg.Deramp(poly_order=2)
-deramp.fit(reference_dem, dem_to_be_aligned, inlier_mask=inlier_mask)
-corrected_dem = deramp.apply(dem_to_be_aligned)
+corrected_dem = deramp.fit_and_apply(reference_dem, dem_to_be_aligned, inlier_mask=inlier_mask)
# %%
# Then, the new difference can be plotted.
diff_after = reference_dem - corrected_dem
-diff_after.plot(cmap="coolwarm_r", vmin=-10, vmax=10, cbar_title="Elevation change (m)")
+diff_after.plot(cmap="RdYlBu", vmin=-10, vmax=10, cbar_title="Elevation differences (m)")
# %%
# We compare the median and NMAD to validate numerically that there was an improvement (see :ref:`robuststats-meanstd`):
inliers_before = diff_before[inlier_mask]
-med_before, nmad_before = np.median(inliers_before), xdem.spatialstats.nmad(inliers_before)
+med_before, nmad_before = np.ma.median(inliers_before), xdem.spatialstats.nmad(inliers_before)
inliers_after = diff_after[inlier_mask]
-med_after, nmad_after = np.median(inliers_after), xdem.spatialstats.nmad(inliers_after)
+med_after, nmad_after = np.ma.median(inliers_after), xdem.spatialstats.nmad(inliers_after)
print(f"Error before: median = {med_before:.2f} - NMAD = {nmad_before:.2f} m")
print(f"Error after: median = {med_after:.2f} - NMAD = {nmad_after:.2f} m")
diff --git a/examples/advanced/plot_heterosc_estimation_modelling.py b/examples/advanced/plot_heterosc_estimation_modelling.py
index a979fbae..df392e00 100644
--- a/examples/advanced/plot_heterosc_estimation_modelling.py
+++ b/examples/advanced/plot_heterosc_estimation_modelling.py
@@ -4,31 +4,30 @@
Digital elevation models have a precision that can vary with terrain and instrument-related variables. This variability
in variance is called `heteroscedasticy `_,
-and rarely accounted for in DEM studies (see :ref:`intro`). Quantifying elevation heteroscedasticity is essential to
+and rarely accounted for in DEM studies (see :ref:`accuracy-precision`). Quantifying elevation heteroscedasticity is essential to
use stable terrain as an error proxy for moving terrain, and standardize data towards a stationary variance, necessary
-to apply spatial statistics (see :ref:`spatialstats`).
+to apply spatial statistics (see :ref:`uncertainty`).
Here, we show an advanced example in which we look for terrain-dependent explanatory variables to explain the
-heteroscedasticity for a DEM difference at Longyearbyen. We use `data binning `_
-and robust statistics in N-dimension with :func:`xdem.spatialstats.nd_binning`, apply a N-dimensional interpolation with
-:func:`xdem.spatialstats.interp_nd_binning`, and scale our interpolant function with a two-step standardization
-:func:`xdem.spatialstats.two_step_standardization` to produce the final elevation error function.
-
-**References**: `Hugonnet et al. (2021) `_, Equation 1, Extended Data Fig.
-3a and `Hugonnet et al. (2022) `_, Figs. 4 and S6–S9. Equations 7 or 8 can
-be used to convert elevation change errors into elevation errors.
+heteroscedasticity for a DEM difference at Longyearbyen. We detail the steps used by
+:func:`~xdem.spatialstats.infer_heteroscedasticity_from_stable` exemplified in :ref:`sphx_glr_basic_examples_plot_infer_heterosc.py`.
+
+We use `data binning `_ and robust statistics in N-dimension with
+:func:`~xdem.spatialstats.nd_binning`, apply a N-dimensional interpolation with
+:func:`~xdem.spatialstats.interp_nd_binning`, and scale our interpolant function with a two-step standardization
+:func:`~xdem.spatialstats.two_step_standardization` to produce the final elevation error function.
+
+**Reference:** `Hugonnet et al. (2022) `_.
"""
import geoutils as gu
# sphinx_gallery_thumbnail_number = 8
-import matplotlib.pyplot as plt
import numpy as np
import xdem
# %%
-# Here, we detail the steps used by ``xdem.spatialstats.infer_heteroscedasticity_from_stable`` exemplified in
-# :ref:`sphx_glr_basic_examples_plot_infer_heterosc.py`. First, we load example files and create a glacier mask.
+# We load example files and create a glacier mask.
ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
dh = xdem.DEM(xdem.examples.get_path("longyearbyen_ddem"))
@@ -100,11 +99,10 @@
# %%
# The relation with the plan curvature remains ambiguous.
# We should better define our bins to avoid sampling bins with too many or too few samples. For this, we can partition
-# the data in quantiles in :func:`xdem.spatialstats.nd_binning`.
-# *Note: we need a higher number of bins to work with quantiles and still resolve the edges of the distribution. As
-# with many dimensions the ND bin size increases exponentially, we avoid binning all variables at the same
-# time and instead bin one at a time.*
-# We define 1000 quantile bins of size 0.001 (equivalent to 0.1% percentile bins) for the profile curvature:
+# the data in quantiles in :func:`xdem.spatialstats.nd_binning`. We define 1000 quantile bins of size
+# 0.001 (equivalent to 0.1% percentile bins) for the profile curvature:
+#
+# .. note:: We need a higher number of bins to work with quantiles and still resolve the edges of the distribution.
df = xdem.spatialstats.nd_binning(
values=dh_arr,
diff --git a/examples/advanced/plot_norm_regional_hypso.py b/examples/advanced/plot_norm_regional_hypso.py
index 6a313c67..e9b7c50e 100644
--- a/examples/advanced/plot_norm_regional_hypso.py
+++ b/examples/advanced/plot_norm_regional_hypso.py
@@ -2,7 +2,9 @@
Normalized regional hypsometric interpolation
=============================================
-There are many ways of interpolating gaps in a dDEM.
+.. caution:: This functionality is specific to glaciers, and might be removed in future package versions.
+
+There are many ways of interpolating gaps in elevation differences.
In the case of glaciers, one very useful fact is that elevation change generally varies with elevation.
This means that if valid pixels exist in a certain elevation bin, their values can be used to fill other pixels in the same approximate elevation.
Filling gaps by elevation is the main basis of "hypsometric interpolation approaches", of which there are many variations of.
@@ -24,7 +26,6 @@
2. Re-scales that signal to fit each glacier once determined.
The consequence is a much more accurate interpolation approach that can be used in a multitude of glacierized settings.
-
"""
import geoutils as gu
@@ -95,8 +96,7 @@
)
-plt.figure(figsize=(8, 5))
-plt.imshow(ddem_filled.data, cmap="coolwarm_r", vmin=-10, vmax=10, extent=plt_extent)
+plt.imshow(ddem_filled.data, cmap="RdYlBu", vmin=-10, vmax=10, extent=plt_extent)
plt.colorbar()
plt.show()
@@ -105,7 +105,7 @@
# We can plot the difference between the actual and the interpolated values, to validate the method.
difference = (ddem_filled - ddem)[random_nans]
-median = np.nanmedian(difference)
+median = np.ma.median(difference)
nmad = xdem.spatialstats.nmad(difference)
plt.title(f"Median: {median:.2f} m, NMAD: {nmad:.2f} m")
diff --git a/examples/advanced/plot_slope_methods.py b/examples/advanced/plot_slope_methods.py
index 5c005c97..4dc697bb 100644
--- a/examples/advanced/plot_slope_methods.py
+++ b/examples/advanced/plot_slope_methods.py
@@ -5,8 +5,9 @@
Terrain slope and aspect can be estimated using different methods.
Here is an example of how to generate the two with each method, and understand their differences.
-For more information, see the :ref:`terrain-attributes` chapter and the
-:ref:`sphx_glr_basic_examples_plot_terrain_attributes.py` example.
+See also the :ref:`terrain-attributes` feature page.
+
+**References:** `Horn (1981) `_, `Zevenbergen and Thorne (1987) `_.
"""
import matplotlib.pyplot as plt
import numpy as np
@@ -14,13 +15,11 @@
import xdem
# %%
-# **Example data**
-
+# We open example data.
dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
def plot_attribute(attribute, cmap, label=None, vlim=None):
- plt.figure(figsize=(8, 5))
if vlim is not None:
if isinstance(vlim, (int, np.integer, float, np.floating)):
@@ -30,7 +29,7 @@ def plot_attribute(attribute, cmap, label=None, vlim=None):
else:
vlims = {}
- attribute.plot(cmap=cmap, cbar_title=label)
+ attribute.plot(cmap=cmap, cbar_title=label, **vlims)
plt.xticks([])
plt.yticks([])
@@ -40,15 +39,15 @@ def plot_attribute(attribute, cmap, label=None, vlim=None):
# %%
-# Slope with method of `Horn (1981) `_ (GDAL default), based on a refined
-# approximation of the gradient (page 18, bottom left, and pages 20-21).
+# Slope with method of Horn (1981) (GDAL default), based on a refined
+# approximation of the gradient (page 18, bottom left, and pages 20-21).
slope_horn = xdem.terrain.slope(dem)
plot_attribute(slope_horn, "Reds", "Slope of Horn (1981) (°)")
# %%
-# Slope with method of `Zevenbergen and Thorne (1987) `_, Equation 13.
+# Slope with method of Zevenbergen and Thorne (1987), Equation 13.
slope_zevenberg = xdem.terrain.slope(dem, method="ZevenbergThorne")
@@ -109,4 +108,4 @@ def plot_attribute(attribute, cmap, label=None, vlim=None):
# differences for areas with nearly flat slopes, owing to the high sensitivity of orientation estimation
# for flat terrain.
-# Note: the default aspect for a 0° slope is 180°, as in GDAL.
+# .. note:: The default aspect for a 0° slope is 180°, as in GDAL.
diff --git a/examples/advanced/plot_standardization.py b/examples/advanced/plot_standardization.py
index f7864749..f436e01c 100644
--- a/examples/advanced/plot_standardization.py
+++ b/examples/advanced/plot_standardization.py
@@ -5,13 +5,13 @@
Digital elevation models have both a precision that can vary with terrain or instrument-related variables, and
a spatial correlation of errors that can be due to effects of resolution, processing or instrument noise.
Accouting for non-stationarities in elevation errors is essential to use stable terrain as a proxy to infer the
-precision on other types of terrain and reliably use spatial statistics (see :ref:`spatialstats`).
+precision on other types of terrain and reliably use spatial statistics (see :ref:`uncertainty`).
Here, we show an example of standardization of the data based on terrain-dependent explanatory variables
(see :ref:`sphx_glr_basic_examples_plot_infer_heterosc.py`) and combine it with an analysis of spatial correlation
(see :ref:`sphx_glr_basic_examples_plot_infer_spatial_correlation.py`) .
-**Reference**: `Hugonnet et al. (2022) `_, Equation 12.
+**Reference**: `Hugonnet et al. (2022) `_.
"""
import geoutils as gu
@@ -105,7 +105,6 @@
z_dh = z_dh / scale_fac_std
print(f"NMAD after scale-correction: {nmad(z_dh.data):.1f}")
-plt.figure(figsize=(8, 5))
plt_extent = [
ref_dem.bounds.left,
ref_dem.bounds.right,
@@ -128,8 +127,8 @@
df_vgm = xdem.spatialstats.sample_empirical_variogram(
values=z_dh.data.squeeze(),
gsd=dh.res[0],
- subsample=1000,
- n_variograms=10,
+ subsample=500,
+ n_variograms=5,
estimator="dowd",
random_state=42,
)
@@ -163,7 +162,6 @@
medals_shp = gu.Vector(glacier_outlines.ds[glacier_outlines.ds["NAME"] == "Medalsbreen"])
medals_mask = medals_shp.create_mask(dh)
-plt.figure(figsize=(8, 5))
ax = plt.gca()
plt_extent = [
ref_dem.bounds.left,
@@ -225,13 +223,9 @@
medals_dh = np.nanmean(dh.data[medals_mask.data])
# Plot the result
-plt.figure(figsize=(8, 5))
-ax = plt.gca()
-plt.imshow(dh.data, cmap="RdYlBu", vmin=-50, vmax=50, extent=plt_extent)
-cbar = plt.colorbar(ax=ax)
-cbar.set_label("Elevation differences (m)")
-svendsen_shp.ds.plot(ax=ax, fc="none", ec="tab:olive", lw=2)
-medals_shp.ds.plot(ax=ax, fc="none", ec="tab:gray", lw=2)
+dh.plot(cmap="RdYlBu", vmin=-50, vmax=50, cbar_title="Elevation differences (m)")
+svendsen_shp.plot(fc="none", ec="tab:olive", lw=2)
+medals_shp.plot(fc="none", ec="tab:gray", lw=2)
plt.plot([], [], color="tab:olive", label="Svendsenbreen glacier")
plt.plot([], [], color="tab:gray", label="Medalsbreen glacier")
ax.text(
diff --git a/examples/advanced/plot_variogram_estimation_modelling.py b/examples/advanced/plot_variogram_estimation_modelling.py
index 43cff5a6..5b7d5f84 100644
--- a/examples/advanced/plot_variogram_estimation_modelling.py
+++ b/examples/advanced/plot_variogram_estimation_modelling.py
@@ -3,20 +3,22 @@
==============================================
Digital elevation models have errors that are often `correlated in space `_.
-While many DEM studies used solely short-range `variogram `_ to
-estimate the correlation of elevation measurement errors (e.g., `Howat et al. (2008) `_
-, `Wang and Kääb (2015) `_), recent studies show that variograms of multiple ranges
+While many DEM studies used solely short-range `variograms `_ to
+estimate the correlation of elevation measurement errors, recent studies show that variograms of multiple ranges
provide larger, more reliable estimates of spatial correlation for DEMs.
Here, we show an example in which we estimate the spatial correlation for a DEM difference at Longyearbyen, and its
-impact on the standard error with averaging area. We first estimate an empirical variogram with
-:func:`xdem.spatialstats.sample_empirical_variogram` based on routines of `scikit-gstat
+impact on the standard error of the mean of elevation differences in an area. We detail the steps used
+by :func:`~xdem.spatialstats.infer_spatial_correlation_from_stable` exemplified in
+# :ref:`sphx_glr_basic_examples_plot_infer_spatial_correlation.py`.
+
+We first estimate an empirical variogram with :func:`~xdem.spatialstats.sample_empirical_variogram` based on routines of `SciKit-GStat
`_. We then fit the empirical variogram with a sum of variogram
-models using :func:`xdem.spatialstats.fit_sum_model_variogram`. Finally, we perform spatial propagation for a range of
-averaging area using :func:`xdem.spatialstats.number_effective_samples`, and empirically validate the improved
-robustness of our results using :func:`xdem.spatialstats.patches_method`, an intensive Monte-Carlo sampling approach.
+models using :func:`~xdem.spatialstats.fit_sum_model_variogram`. Finally, we perform spatial propagation for a range of
+averaging area using :func:`~xdem.spatialstats.number_effective_samples`, and empirically validate the improved
+robustness of our results using :func:`~xdem.spatialstats.patches_method`, an intensive Monte-Carlo sampling approach.
-**Reference:** `Hugonnet et al. (2022) `_, Figure 5 and Equations 13–16.
+**References:** `Rolstad et al. (2009) `_, `Hugonnet et al. (2022) `_.
"""
import geoutils as gu
@@ -49,7 +51,6 @@
# elevation differences. The per-pixel precision is about :math:`\pm` 2.5 meters.
# **Does this mean that every pixel has an independent measurement error of** :math:`\pm` **2.5 meters?**
# Let's plot the elevation differences to visually check the quality of the data.
-plt.figure(figsize=(8, 5))
dh.plot(ax=plt.gca(), cmap="RdYlBu", vmin=-4, vmax=4, cbar_title="Elevation differences (m)")
# %%
@@ -65,7 +66,6 @@
# %%
# We plot the elevation differences after filtering to check that we successively removed glacier signals.
-plt.figure(figsize=(8, 5))
dh.plot(ax=plt.gca(), cmap="RdYlBu", vmin=-4, vmax=4, cbar_title="Elevation differences (m)")
# %%
@@ -73,18 +73,17 @@
# The empirical variogram describes the variance between the elevation differences of pairs of pixels depending on their
# distance. This distance between pairs of pixels if referred to as spatial lag.
#
-# To perform this procedure effectively, we use improved methods that provide efficient pairwise sampling methods for
-# large grid data in `scikit-gstat `_, which are encapsulated
-# conveniently by :func:`xdem.spatialstats.sample_empirical_variogram`:
-# Dowd's variogram is used for robustness in conjunction with the NMAD (see :ref:`robuststats-corr`).
+# To perform this procedure effectively, we use methods that provide efficient pairwise sampling methods for
+# large grid data in `SciKit-GStat `_, which are encapsulated
+# conveniently by :func:`~xdem.spatialstats.sample_empirical_variogram`. # Dowd's variogram is used for
+# robustness in conjunction with the NMAD (see :ref:`robuststats-corr`).
df = xdem.spatialstats.sample_empirical_variogram(
- values=dh, subsample=1000, n_variograms=10, estimator="dowd", random_state=42
+ values=dh, subsample=500, n_variograms=5, estimator="dowd", random_state=42
)
# %%
-# *Note: in this example, we add a* ``random_state`` *argument to yield a reproducible random sampling of pixels within
-# the grid.*
+# .. note:: In this example, we add a ``random_state`` argument to yield a reproducible random sampling of pixels within the grid.
# %%
# We plot the empirical variogram:
@@ -164,7 +163,7 @@
# patches method to run over long processing times, increasing from areas of 5 pixels to areas of 10000 pixels exponentially.
areas_emp = [4000 * 2 ** (i) for i in range(10)]
-df_patches = xdem.spatialstats.patches_method(dh, gsd=dh.res[0], areas=areas_emp)
+df_patches = xdem.spatialstats.patches_method(dh, gsd=dh.res[0], areas=areas_emp, n_patches=200)
fig, ax = plt.subplots()
@@ -185,8 +184,7 @@
plt.show()
# %%
-# *Note: in this example, we add a* ``random_state`` *argument to the patches method to yield a reproducible random
-# sampling, and set* ``n_patches`` *to reduce computing time.*
+# .. note:: In this example, we set ``n_patches`` to a moderate number to reduce computing time.
# %%
# Using a single-range variogram highly underestimates the measurement error integrated over an area, by over a factor
@@ -194,7 +192,7 @@
#
# **But, in this case, the error is still too small. Why?**
# The small size of the sampling area against the very large range of the noise implies that we might not verify the
-# assumption of second-order stationarity (see :ref:`spatialstats`). Longer range correlations might be omitted by
+# assumption of second-order stationarity (see :ref:`uncertainty`). Longer range correlations might be omitted by
# our analysis, due to the limits of the variogram sampling. In other words, a small part of the variance could be
# fully correlated over a large part of the grid: a vertical bias.
#
diff --git a/examples/basic/plot_dem_subtraction.py b/examples/basic/plot_dem_subtraction.py
index ee158338..c9e0e458 100644
--- a/examples/basic/plot_dem_subtraction.py
+++ b/examples/basic/plot_dem_subtraction.py
@@ -7,7 +7,7 @@
xDEM allows to use any operator on :class:`xdem.DEM` objects, such as :func:`+` or :func:`-` as well as most NumPy functions
while respecting nodata values and checking that georeferencing is consistent. This functionality is inherited from `GeoUtils' Raster class `_.
-Before DEMs can be compared, they need to be reprojected to the same grid and have the same 3D CRSs. The :func:`geoutils.Raster.reproject` and :func:`xdem.DEM.to_vcrs` methods are used for this.
+Before DEMs can be compared, they need to be reprojected to the same grid and have the same 3D CRSs. The :func:`~xdem.DEM.reproject` and :func:`~xdem.DEM.to_vcrs` methods are used for this.
"""
import geoutils as gu
@@ -28,30 +28,30 @@
# %%
# In this particular case, the two DEMs are already on the same grid (they have the same bounds, resolution and coordinate system).
-# If they don't, we need to reproject one DEM to fit the other using :func:`geoutils.Raster.reproject`:
+# If they don't, we need to reproject one DEM to fit the other using :func:`xdem.DEM.reproject`:
dem_1990 = dem_1990.reproject(dem_2009)
# %%
# Oops!
-# GeoUtils just warned us that ``dem_1990`` did not need reprojection. We can hide this output with ``.reproject(..., silent=True)``.
-# By default, :func:`xdem.DEM.reproject` uses "bilinear" resampling (assuming resampling is needed).
+# GeoUtils just warned us that ``dem_1990`` did not need reprojection. We can hide this output with ``silent``.
+# By default, :func:`~xdem.DEM.reproject` uses "bilinear" resampling (assuming resampling is needed).
# Other options are detailed at `geoutils.Raster.reproject() `_ and `rasterio.enums.Resampling `_.
#
-# We now compute the difference by simply substracting, passing `stats=True` to :func:`geoutils.Raster.info` to print statistics.
+# We now compute the difference by simply substracting, passing ``stats=True`` to :func:`xdem.DEM.info` to print statistics.
ddem = dem_2009 - dem_1990
ddem.info(stats=True)
# %%
-# It is a new :class:`xdem.DEM` instance, loaded in memory.
+# It is a new :class:`~xdem.DEM` instance, loaded in memory.
# Let's visualize it, with some glacier outlines.
# Load the outlines
glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
glacier_outlines = glacier_outlines.crop(ddem, clip=True)
-ddem.plot(cmap="coolwarm_r", vmin=-20, vmax=20, cbar_title="Elevation change (m)")
+ddem.plot(cmap="RdYlBu", vmin=-20, vmax=20, cbar_title="Elevation differences (m)")
glacier_outlines.plot(ref_crs=ddem, fc="none", ec="k")
# %%
diff --git a/examples/basic/plot_icp_coregistration.py b/examples/basic/plot_icp_coregistration.py
index 14acca2b..568452cc 100644
--- a/examples/basic/plot_icp_coregistration.py
+++ b/examples/basic/plot_icp_coregistration.py
@@ -2,13 +2,13 @@
Iterative closest point coregistration
======================================
-Iterative Closest Point (ICP) is a registration methods accounting for both rotation and translation.
+Iterative closest point (ICP) is a registration method accounting for both rotations and translations.
-It is used primarily to correct rotations, as it performs worse than :ref:`coregistration-nuthkaab` for sub-pixel shifts.
+It is used primarily to correct rotations, as it generally performs worse than :ref:`nuthkaab` for sub-pixel shifts.
Fortunately, xDEM provides the best of two worlds by allowing a combination of the two methods in a pipeline,
demonstrated below!
-**Reference**: `Besl and McKay (1992) `_.
+**References**: `Besl and McKay (1992) `_.
"""
# sphinx_gallery_thumbnail_number = 2
import matplotlib.pyplot as plt
@@ -42,15 +42,15 @@
[0, 0, 0, 1],
]
)
-
+centroid = [dem.bounds.left + dem.width / 2, dem.bounds.bottom + dem.height / 2, np.nanmean(dem)]
# This will apply the matrix along the center of the DEM
-rotated_dem = xdem.coreg.apply_matrix(dem, matrix=rotation_matrix)
+rotated_dem = xdem.coreg.apply_matrix(dem, matrix=rotation_matrix, centroid=centroid)
# %%
# We can plot the difference between the original and rotated DEM.
# It is now artificially tilting from east down to the west.
diff_before = dem - rotated_dem
-diff_before.plot(cmap="coolwarm_r", vmin=-20, vmax=20)
+diff_before.plot(cmap="RdYlBu", vmin=-20, vmax=20, cbar_title="Elevation differences (m)")
plt.show()
# %%
@@ -71,18 +71,16 @@
plt.figure(figsize=(6, 12))
for i, (approach, name) in enumerate(approaches):
- approach.fit(
+ corrected_dem = approach.fit_and_apply(
reference_elev=dem,
to_be_aligned_elev=rotated_dem,
)
- corrected_dem = approach.apply(elev=rotated_dem)
-
diff = dem - corrected_dem
ax = plt.subplot(3, 1, i + 1)
plt.title(name)
- diff.plot(cmap="coolwarm_r", vmin=-20, vmax=20, ax=ax)
+ diff.plot(cmap="RdYlBu", vmin=-20, vmax=20, ax=ax, cbar_title="Elevation differences (m)")
plt.tight_layout()
plt.show()
@@ -91,8 +89,8 @@
# %%
# The results show what we expected:
#
-# * ``ICP`` alone handled the rotational offset, but left a horizontal offset as it is not sub-pixel accurate (in this case, the resolution is 20x20m).
-# * ``NuthKaab`` barely helped at all, since the offset is purely rotational.
-# * ``ICP + NuthKaab`` first handled the rotation, then fit the reference with sub-pixel accuracy.
+# - **ICP** alone handled the rotational offset, but left a horizontal offset as it is not sub-pixel accurate (in this case, the resolution is 20x20m).
+# - **Nuth and Kääb** barely helped at all, since the offset is purely rotational.
+# - **ICP + Nuth and Kääb** first handled the rotation, then fit the reference with sub-pixel accuracy.
#
# The last result is an almost identical raster that was offset but then corrected back to its original position!
diff --git a/examples/basic/plot_infer_heterosc.py b/examples/basic/plot_infer_heterosc.py
index 691d49ec..dd6aa179 100644
--- a/examples/basic/plot_infer_heterosc.py
+++ b/examples/basic/plot_infer_heterosc.py
@@ -7,8 +7,7 @@
using terrain slope and maximum curvature as explanatory variables, with stable terrain as an error proxy for moving
terrain.
-**Reference**: `Hugonnet et al. (2022) `_, Figs. 4 and S6–S9. Equations 7
-or 8 can be used to convert elevation change errors into elevation errors.
+**Reference:** `Hugonnet et al. (2022) `_.
"""
import geoutils as gu
@@ -16,7 +15,7 @@
import xdem
# %%
-# We load a difference of DEMs at Longyearbyen, already coregistered using :ref:`coregistration-nuthkaab` as shown in
+# We load a difference of DEMs at Longyearbyen, already coregistered using :ref:`nuthkaab` as shown in
# the :ref:`sphx_glr_basic_examples_plot_nuth_kaab.py` example. We also load the reference DEM to derive terrain
# attributes and the glacier outlines here corresponding to moving terrain.
dh = xdem.DEM(xdem.examples.get_path("longyearbyen_ddem"))
diff --git a/examples/basic/plot_infer_spatial_correlation.py b/examples/basic/plot_infer_spatial_correlation.py
index 83fc7785..16d69270 100644
--- a/examples/basic/plot_infer_spatial_correlation.py
+++ b/examples/basic/plot_infer_spatial_correlation.py
@@ -6,7 +6,7 @@
rely on a non-stationary spatial statistics framework to estimate and model spatial correlations in elevation error.
We use a sum of variogram forms to model this correlation, with stable terrain as an error proxy for moving terrain.
-**Reference**: `Hugonnet et al. (2022) `_, Figure 5 and Equations 13–16.
+**References:** `Rolstad et al. (2009) `_, `Hugonnet et al. (2022) `_.
"""
import geoutils as gu
@@ -14,7 +14,7 @@
import xdem
# %%
-# We load a difference of DEMs at Longyearbyen, already coregistered using :ref:`coregistration-nuthkaab` as shown in
+# We load a difference of DEMs at Longyearbyen, already coregistered using :ref:`nuthkaab` as shown in
# the :ref:`sphx_glr_basic_examples_plot_nuth_kaab.py` example. We also load the glacier outlines here corresponding to
# moving terrain.
dh = xdem.DEM(xdem.examples.get_path("longyearbyen_ddem"))
@@ -34,7 +34,7 @@
# %%
# The first output corresponds to the dataframe of the empirical variogram, by default estimated using Dowd's estimator
-# and the circular sampling scheme of ``skgstat.RasterEquidistantMetricSpace`` (Fig. S13 of Hugonnet et al. (2022)). The
+# and a circular sampling scheme in SciKit-GStat (following Fig. S13 of Hugonnet et al. (2022)). The
# ``lags`` columns is the upper bound of spatial lag bins (lower bound of first bin being 0), the ``exp`` column is the
# "experimental" variance value of the variogram in that bin, the ``count`` the number of pairwise samples, and
# ``err_exp`` the 1-sigma error of the "experimental" variance, if more than one variogram is estimated with the
diff --git a/examples/basic/plot_logging_configuration.py b/examples/basic/plot_logging_configuration.py
new file mode 100644
index 00000000..5f0a347b
--- /dev/null
+++ b/examples/basic/plot_logging_configuration.py
@@ -0,0 +1,51 @@
+"""
+Configuring verbosity level
+===========================
+
+This example demonstrates how to configure verbosity level, or logging, using a coregistration method.
+Logging can be customized to various severity levels, from ``DEBUG`` for detailed diagnostic output, to ``INFO`` for
+general updates, ``WARNING`` for potential issues, and ``ERROR`` or ``CRITICAL`` for serious problems.
+
+Setting the verbosity to a certain severity level prints all outputs from that level and those above. For instance,
+level ``INFO`` also prints warnings, error and critical messages.
+
+See also :ref:`config`.
+
+.. important:: The verbosity level defaults to ``WARNING``, so no ``INFO`` or ``DEBUG`` is printed.
+"""
+
+import logging
+
+import xdem
+
+# %%
+# We start by configuring the logging level, which can be as simple as specifying we want to print information.
+logging.basicConfig(level=logging.INFO)
+
+# %%
+# We can change the configuration even more by specifying the format, date, and multiple destinations for the output.
+logging.basicConfig(
+ level=logging.INFO, # Change this level to DEBUG or WARNING to see different outputs.
+ format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+ datefmt="%Y-%m-%d %H:%M:%S",
+ handlers=[
+ logging.FileHandler("../xdem_example.log"), # Save logs to a file
+ logging.StreamHandler(), # Also print logs to the console
+ ],
+ force=True, # To re-set from previous logging
+)
+
+# %%
+# We can now load example files and demonstrate the logging through a functionality, such as coregistration.
+reference_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
+dem_to_be_aligned = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
+coreg = xdem.coreg.NuthKaab()
+
+# %%
+# With the ``INFO`` verbosity level defined above, we can follow the iteration with a detailed format, saved to file.
+aligned_dem = coreg.fit_and_apply(reference_dem, dem_to_be_aligned)
+
+# %%
+# With a more severe verbosity level, there is no output.
+logging.basicConfig(level=logging.ERROR, force=True)
+aligned_dem = coreg.fit_and_apply(reference_dem, dem_to_be_aligned)
diff --git a/examples/basic/plot_nuth_kaab.py b/examples/basic/plot_nuth_kaab.py
index e390f7af..ad4df559 100644
--- a/examples/basic/plot_nuth_kaab.py
+++ b/examples/basic/plot_nuth_kaab.py
@@ -2,10 +2,13 @@
Nuth and Kääb coregistration
============================
-Nuth and Kääb (`2011 `_) coregistration allows for horizontal and vertical shifts to be estimated and corrected for.
-In ``xdem``, this approach is implemented through the :class:`xdem.coreg.NuthKaab` class.
+The Nuth and Kääb coregistration corrects horizontal and vertical shifts, and is especially performant for precise
+sub-pixel alignment in areas with varying slope.
+In xDEM, this approach is implemented through the :class:`xdem.coreg.NuthKaab` class.
-For more information about the approach, see :ref:`coregistration-nuthkaab`.
+See also the :ref:`nuthkaab` section in feature pages.
+
+**Reference:** `Nuth and Kääb (2011) `_.
"""
import geoutils as gu
import numpy as np
@@ -13,47 +16,46 @@
import xdem
# %%
-# **Example files**
+# We open example files.
reference_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
dem_to_be_aligned = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
-# Create a stable ground mask (not glacierized) to mark "inlier data"
+# We create a stable ground mask (not glacierized) to mark "inlier data".
inlier_mask = ~glacier_outlines.create_mask(reference_dem)
-
# %%
-# The DEM to be aligned (a 1990 photogrammetry-derived DEM) has some vertical and horizontal biases that we want to avoid.
+# The DEM to be aligned (a 1990 photogrammetry-derived DEM) has some vertical and horizontal biases that we want to reduce.
# These can be visualized by plotting a change map:
diff_before = reference_dem - dem_to_be_aligned
-diff_before.plot(cmap="coolwarm_r", vmin=-10, vmax=10, cbar_title="Elevation change (m)")
-
+diff_before.plot(cmap="RdYlBu", vmin=-10, vmax=10, cbar_title="Elevation change (m)")
# %%
-# Horizontal and vertical shifts can be estimated using :class:`xdem.coreg.NuthKaab`.
-# First, the shifts are estimated, and then they can be applied to the data:
+# Horizontal and vertical shifts can be estimated using :class:`~xdem.coreg.NuthKaab`.
+# The shifts are estimated then applied to the to-be-aligned elevation data:
nuth_kaab = xdem.coreg.NuthKaab()
+aligned_dem = nuth_kaab.fit_and_apply(reference_dem, dem_to_be_aligned, inlier_mask)
-nuth_kaab.fit(reference_dem, dem_to_be_aligned, inlier_mask)
+# %%
+# The shifts are stored in the affine metadata output
-aligned_dem = nuth_kaab.apply(dem_to_be_aligned)
+print([nuth_kaab.meta["outputs"]["affine"][s] for s in ["shift_x", "shift_y", "shift_z"]])
# %%
# Then, the new difference can be plotted to validate that it improved.
diff_after = reference_dem - aligned_dem
-diff_after.plot(cmap="coolwarm_r", vmin=-10, vmax=10, cbar_title="Elevation change (m)")
-
+diff_after.plot(cmap="RdYlBu", vmin=-10, vmax=10, cbar_title="Elevation change (m)")
# %%
# We compare the median and NMAD to validate numerically that there was an improvement (see :ref:`robuststats-meanstd`):
inliers_before = diff_before[inlier_mask]
-med_before, nmad_before = np.median(inliers_before), xdem.spatialstats.nmad(inliers_before)
+med_before, nmad_before = np.ma.median(inliers_before), xdem.spatialstats.nmad(inliers_before)
inliers_after = diff_after[inlier_mask]
-med_after, nmad_after = np.median(inliers_after), xdem.spatialstats.nmad(inliers_after)
+med_after, nmad_after = np.ma.median(inliers_after), xdem.spatialstats.nmad(inliers_after)
print(f"Error before: median = {med_before:.2f} - NMAD = {nmad_before:.2f} m")
print(f"Error after: median = {med_after:.2f} - NMAD = {nmad_after:.2f} m")
diff --git a/examples/basic/plot_spatial_error_propagation.py b/examples/basic/plot_spatial_error_propagation.py
index 1b126832..e5f3153c 100644
--- a/examples/basic/plot_spatial_error_propagation.py
+++ b/examples/basic/plot_spatial_error_propagation.py
@@ -7,8 +7,7 @@
other operation), which is computationally intensive. Here, we rely on published formulations to perform
computationally-efficient spatial propagation for the mean of elevation (or elevation differences) in an area.
-**References**: `Hugonnet et al. (2022) `_, Figure S16, Equations 17–19 and
-`Rolstad et al. (2009) `_, Equation 8.
+**References:** `Rolstad et al. (2009) `_, `Hugonnet et al. (2022) `_.
"""
import geoutils as gu
import matplotlib.pyplot as plt
@@ -32,8 +31,8 @@
)
# %%
-# We use the error map to standardize the elevation differences before variogram estimation, following Equation 12 of
-# Hugonnet et al. (2022), which is more robust as it removes the variance variability due to heteroscedasticity.
+# We use the error map to standardize the elevation differences before variogram estimation, which is more robust
+# as it removes the variance variability due to heteroscedasticity.
zscores = dh / errors
emp_variogram, params_variogram_model, spatial_corr_function = xdem.spatialstats.infer_spatial_correlation_from_stable(
dvalues=zscores, list_models=["Gaussian", "Spherical"], unstable_mask=glacier_outlines, random_state=42
@@ -42,7 +41,7 @@
# %%
# With our estimated heteroscedasticity and spatial correlation, we can now perform the spatial propagation of errors.
# We select two glaciers intersecting this elevation change map in Svalbard. The best estimation of their standard error
-# is done by directly providing the shapefile, which relies on Equation 18 of Hugonnet et al. (2022).
+# is done by directly providing the shapefile (Equation 18, Hugonnet et al., 2022).
areas = [
glacier_outlines.ds[glacier_outlines.ds["NAME"] == "Brombreen"],
glacier_outlines.ds[glacier_outlines.ds["NAME"] == "Medalsbreen"],
@@ -55,8 +54,8 @@
print(f"The error (1-sigma) in mean elevation change for {glacier_name} is {stderr_gla:.2f} meters.")
# %%
-# When passing a numerical area value, we compute an approximation with disk shape from Equation 8 of Rolstad et al.
-# (2009). This approximation is practical to visualize changes in elevation error when averaging over different area
+# When passing a numerical area value, we compute an approximation with disk shape (Equation 8, Rolstad et al., 2009).
+# This approximation is practical to visualize changes in elevation error when averaging over different area
# sizes, but is less accurate to estimate the standard error of a certain area shape.
areas = 10 ** np.linspace(1, 12)
stderrs = xdem.spatialstats.spatial_error_propagation(
diff --git a/examples/basic/plot_terrain_attributes.py b/examples/basic/plot_terrain_attributes.py
index d0b79d3d..69bf2ea1 100644
--- a/examples/basic/plot_terrain_attributes.py
+++ b/examples/basic/plot_terrain_attributes.py
@@ -5,134 +5,25 @@
Terrain attributes generated from a DEM have a multitude of uses for analytic and visual purposes.
Here is an example of how to generate these products.
-For more information, see the :ref:`terrain-attributes` chapter and the
-:ref:`sphx_glr_advanced_examples_plot_slope_methods.py` example.
+For more information, see the :ref:`terrain-attributes` feature page.
+
+**References:** `Horn (1981) `_ (slope, aspect, hillshade),
+`Zevenbergen and Thorne (1987) `_ (curvature),
+`Riley et al. (1999) `_ (terrain
+ruggedness index), `Jenness (2004) `_ (rugosity).
"""
-# sphinx_gallery_thumbnail_number = 12
+# sphinx_gallery_thumbnail_number = 1
import matplotlib.pyplot as plt
import xdem
# %%
-# **Example data**
+# We load the example data.
dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
-
-def plot_attribute(attribute, cmap, label=None, vlim=None):
-
- add_cbar = True if label is not None else False
-
- fig = plt.figure(figsize=(8, 5))
- ax = fig.add_subplot(111)
-
- if vlim is not None:
- if isinstance(vlim, (int, float)):
- vlims = {"vmin": -vlim, "vmax": vlim}
- elif len(vlim) == 2:
- vlims = {"vmin": vlim[0], "vmax": vlim[1]}
- else:
- vlims = {}
-
- attribute.plot(ax=ax, cmap=cmap, add_cbar=add_cbar, cbar_title=label, **vlims)
-
- plt.xticks([])
- plt.yticks([])
- plt.tight_layout()
-
- plt.show()
-
-
-# %%
-# Slope
-# -----
-
-slope = xdem.terrain.slope(dem)
-
-plot_attribute(slope, "Reds", "Slope (°)")
-
-# %%
-# Note that all functions also work with numpy array as inputs, if resolution is specified
-
-slope = xdem.terrain.slope(dem.data, resolution=dem.res)
-
-# %%
-# Aspect
-# ------
-
-aspect = xdem.terrain.aspect(dem)
-
-plot_attribute(aspect, "twilight", "Aspect (°)")
-
-# %%
-# Hillshade
-# ---------
-
-hillshade = xdem.terrain.hillshade(dem, azimuth=315.0, altitude=45.0)
-
-plot_attribute(hillshade, "Greys_r")
-
-# %%
-# Curvature
-# ---------
-
-curvature = xdem.terrain.curvature(dem)
-
-plot_attribute(curvature, "RdGy_r", "Curvature (100 / m)", vlim=1)
-
-# %%
-# Planform curvature
-# ------------------
-
-planform_curvature = xdem.terrain.planform_curvature(dem)
-
-plot_attribute(planform_curvature, "RdGy_r", "Planform curvature (100 / m)", vlim=1)
-
-# %%
-# Profile curvature
-# -----------------
-profile_curvature = xdem.terrain.profile_curvature(dem)
-
-plot_attribute(profile_curvature, "RdGy_r", "Profile curvature (100 / m)", vlim=1)
-
-# %%
-# Topographic Position Index
-# --------------------------
-tpi = xdem.terrain.topographic_position_index(dem)
-
-plot_attribute(tpi, "Spectral", "Topographic Position Index", vlim=5)
-
-# %%
-# Terrain Ruggedness Index
-# ------------------------
-tri = xdem.terrain.terrain_ruggedness_index(dem)
-
-plot_attribute(tri, "Purples", "Terrain Ruggedness Index")
-
-# %%
-# Roughness
-# ---------
-roughness = xdem.terrain.roughness(dem)
-
-plot_attribute(roughness, "Oranges", "Roughness")
-
-# %%
-# Rugosity
-# --------
-rugosity = xdem.terrain.rugosity(dem)
-
-plot_attribute(rugosity, "YlOrRd", "Rugosity")
-
-# %%
-# Fractal roughness
-# -----------------
-fractal_roughness = xdem.terrain.fractal_roughness(dem)
-
-plot_attribute(fractal_roughness, "Reds", "Fractal roughness")
-
# %%
-# Generating multiple attributes at once
-# --------------------------------------
+# We generate multiple terrain attribute at once (more efficient computationally as some depend on each other).
attributes = xdem.terrain.get_terrain_attribute(
dem.data,
diff --git a/pyproject.toml b/pyproject.toml
index 5f8ff586..ceef909f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -13,7 +13,7 @@ version_file = "xdem/_version.py"
fallback_version = "0.0.1"
[tool.black]
-target_version = ['py36']
+target_version = ['py310']
[tool.pytest.ini_options]
addopts = "--doctest-modules -W error::UserWarning"
diff --git a/requirements.txt b/requirements.txt
index 98df6ee8..0d5001f8 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -11,5 +11,5 @@ scipy==1.*
tqdm
scikit-image==0.*
scikit-gstat>=1.0.18,<1.1
-geoutils==0.1.9
+geoutils==0.1.10
pip
diff --git a/setup.cfg b/setup.cfg
index 6817781f..a31d36ff 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,5 +1,5 @@
[metadata]
-author = The GlacioHack Team
+author = xDEM contributors
name = xdem
version = 0.0.20
description = Analysis of digital elevation models (DEMs)
@@ -33,7 +33,7 @@ download_url = https://pypi.org/project/xdem/
packages = find:
zip_safe = False # https://mypy.readthedocs.io/en/stable/installed_packages.html
include_package_data = True
-python_requires = >=3.9,<3.13
+python_requires = >=3.10,<3.13
# Avoid pinning dependencies in requirements.txt (which we don't do anyways, and we rely mostly on Conda)
# (https://caremad.io/posts/2013/07/setup-vs-requirement/, https://github.com/pypa/setuptools/issues/1951)
install_requires = file: requirements.txt
@@ -52,7 +52,6 @@ opt =
opencv
openh264
pytransform3d
- richdem
noisyopt
test =
pytest
@@ -60,6 +59,7 @@ test =
pyyaml
flake8
pylint
+ richdem
doc =
sphinx
sphinx-book-theme
diff --git a/tests/conftest.py b/tests/conftest.py
new file mode 100644
index 00000000..675084e1
--- /dev/null
+++ b/tests/conftest.py
@@ -0,0 +1,142 @@
+from typing import Callable, List, Union
+
+import geoutils as gu
+import numpy as np
+import pytest
+import richdem as rd
+from geoutils.raster import RasterType
+
+from xdem._typing import NDArrayf
+
+
+@pytest.fixture(scope="session") # type: ignore
+def raster_to_rda() -> Callable[[RasterType], rd.rdarray]:
+ def _raster_to_rda(rst: RasterType) -> rd.rdarray:
+ """
+ Convert geoutils.Raster to richDEM rdarray.
+ """
+ arr = rst.data.filled(rst.nodata).squeeze()
+ rda = rd.rdarray(arr, no_data=rst.nodata)
+ rda.geotransform = rst.transform.to_gdal()
+ return rda
+
+ return _raster_to_rda
+
+
+@pytest.fixture(scope="session") # type: ignore
+def get_terrainattr_richdem(raster_to_rda: Callable[[RasterType], rd.rdarray]) -> Callable[[RasterType, str], NDArrayf]:
+ def _get_terrainattr_richdem(rst: RasterType, attribute: str = "slope_radians") -> NDArrayf:
+ """
+ Derive terrain attribute for DEM opened with geoutils.Raster using RichDEM.
+ """
+ rda = raster_to_rda(rst)
+ terrattr = rd.TerrainAttribute(rda, attrib=attribute)
+ terrattr[terrattr == terrattr.no_data] = np.nan
+ return np.array(terrattr)
+
+ return _get_terrainattr_richdem
+
+
+@pytest.fixture(scope="session") # type: ignore
+def get_terrain_attribute_richdem(
+ get_terrainattr_richdem: Callable[[RasterType, str], NDArrayf]
+) -> Callable[[RasterType, Union[str, list[str]], bool, float, float, float], Union[RasterType, list[RasterType]]]:
+ def _get_terrain_attribute_richdem(
+ dem: RasterType,
+ attribute: Union[str, List[str]],
+ degrees: bool = True,
+ hillshade_altitude: float = 45.0,
+ hillshade_azimuth: float = 315.0,
+ hillshade_z_factor: float = 1.0,
+ ) -> Union[RasterType, List[RasterType]]:
+ """
+ Derive one or multiple terrain attributes from a DEM using RichDEM.
+ """
+ if isinstance(attribute, str):
+ attribute = [attribute]
+
+ if not isinstance(dem, gu.Raster):
+ raise ValueError("DEM must be a geoutils.Raster object.")
+
+ terrain_attributes = {}
+
+ # Check which products should be made to optimize the processing
+ make_aspect = any(attr in attribute for attr in ["aspect", "hillshade"])
+ make_slope = any(
+ attr in attribute
+ for attr in [
+ "slope",
+ "hillshade",
+ "planform_curvature",
+ "aspect",
+ "profile_curvature",
+ "maximum_curvature",
+ ]
+ )
+ make_hillshade = "hillshade" in attribute
+ make_curvature = "curvature" in attribute
+ make_planform_curvature = "planform_curvature" in attribute or "maximum_curvature" in attribute
+ make_profile_curvature = "profile_curvature" in attribute or "maximum_curvature" in attribute
+
+ if make_slope:
+ terrain_attributes["slope"] = get_terrainattr_richdem(dem, "slope_radians")
+
+ if make_aspect:
+ # The aspect of RichDEM is returned in degrees, we convert to radians to match the others
+ terrain_attributes["aspect"] = np.deg2rad(get_terrainattr_richdem(dem, "aspect"))
+ # For flat slopes, RichDEM returns a 90° aspect by default, while GDAL return a 180° aspect
+ # We stay consistent with GDAL
+ slope_tmp = get_terrainattr_richdem(dem, "slope_radians")
+ terrain_attributes["aspect"][slope_tmp == 0] = np.pi
+
+ if make_hillshade:
+ # If a different z-factor was given, slopemap with exaggerated gradients.
+ if hillshade_z_factor != 1.0:
+ slopemap = np.arctan(np.tan(terrain_attributes["slope"]) * hillshade_z_factor)
+ else:
+ slopemap = terrain_attributes["slope"]
+
+ azimuth_rad = np.deg2rad(360 - hillshade_azimuth)
+ altitude_rad = np.deg2rad(hillshade_altitude)
+
+ # The operation below yielded the closest hillshade to GDAL (multiplying by 255 did not work)
+ # As 0 is generally no data for this uint8, we add 1 and then 0.5 for the rounding to occur between
+ # 1 and 255
+ terrain_attributes["hillshade"] = np.clip(
+ 1.5
+ + 254
+ * (
+ np.sin(altitude_rad) * np.cos(slopemap)
+ + np.cos(altitude_rad) * np.sin(slopemap) * np.sin(azimuth_rad - terrain_attributes["aspect"])
+ ),
+ 0,
+ 255,
+ ).astype("float32")
+
+ if make_curvature:
+ terrain_attributes["curvature"] = get_terrainattr_richdem(dem, "curvature")
+
+ if make_planform_curvature:
+ terrain_attributes["planform_curvature"] = get_terrainattr_richdem(dem, "planform_curvature")
+
+ if make_profile_curvature:
+ terrain_attributes["profile_curvature"] = get_terrainattr_richdem(dem, "profile_curvature")
+
+ # Convert the unit if wanted.
+ if degrees:
+ for attr in ["slope", "aspect"]:
+ if attr not in terrain_attributes:
+ continue
+ terrain_attributes[attr] = np.rad2deg(terrain_attributes[attr])
+
+ output_attributes = [terrain_attributes[key].reshape(dem.shape) for key in attribute]
+
+ if isinstance(dem, gu.Raster):
+ output_attributes = [
+ gu.Raster.from_array(attr, transform=dem.transform, crs=dem.crs, nodata=-99999)
+ for attr in output_attributes
+ ]
+
+ return output_attributes if len(output_attributes) > 1 else output_attributes[0]
+
+ return _get_terrain_attribute_richdem
diff --git a/tests/test_coreg/test_affine.py b/tests/test_coreg/test_affine.py
index af10213f..9e82d556 100644
--- a/tests/test_coreg/test_affine.py
+++ b/tests/test_coreg/test_affine.py
@@ -96,32 +96,17 @@ class TestAffineCoreg:
# Check all point-raster possibilities supported
# Use the reference DEM for both, it will be artificially misaligned during tests
# Raster-Raster
- fit_args_rst_rst = dict(
- reference_elev=ref,
- to_be_aligned_elev=tba,
- inlier_mask=inlier_mask,
- verbose=True,
- )
+ fit_args_rst_rst = dict(reference_elev=ref, to_be_aligned_elev=tba, inlier_mask=inlier_mask)
# Convert DEMs to points with a bit of subsampling for speed-up
ref_pts = ref.to_pointcloud(data_column_name="z", subsample=50000, random_state=42).ds
tba_pts = ref.to_pointcloud(data_column_name="z", subsample=50000, random_state=42).ds
# Raster-Point
- fit_args_rst_pts = dict(
- reference_elev=ref,
- to_be_aligned_elev=tba_pts,
- inlier_mask=inlier_mask,
- verbose=True,
- )
+ fit_args_rst_pts = dict(reference_elev=ref, to_be_aligned_elev=tba_pts, inlier_mask=inlier_mask)
# Point-Raster
- fit_args_pts_rst = dict(
- reference_elev=ref_pts,
- to_be_aligned_elev=tba,
- inlier_mask=inlier_mask,
- verbose=True,
- )
+ fit_args_pts_rst = dict(reference_elev=ref_pts, to_be_aligned_elev=tba, inlier_mask=inlier_mask)
all_fit_args = [fit_args_rst_rst, fit_args_rst_pts, fit_args_pts_rst]
@@ -182,13 +167,13 @@ def test_from_classmethods(self) -> None:
# Check that the from_translation function works as expected.
x_offset = 5
- coreg_obj2 = AffineCoreg.from_translation(x_off=x_offset)
+ coreg_obj2 = AffineCoreg.from_translations(x_off=x_offset)
transformed_points2 = coreg_obj2.apply(self.points)
assert np.array_equal(self.points.geometry.x.values + x_offset, transformed_points2.geometry.x.values)
# Try to make a Coreg object from a nan translation (should fail).
try:
- AffineCoreg.from_translation(np.nan)
+ AffineCoreg.from_translations(np.nan)
except ValueError as exception:
if "non-finite values" not in str(exception):
raise exception
@@ -284,12 +269,12 @@ def test_coreg_translations__synthetic(self, fit_args, shifts, coreg_method) ->
"coreg_method__shift",
[
(coreg.NuthKaab, (9.202739, 2.735573, -1.97733)),
- (coreg.DhMinimize, (10.0850892, 2.898166, -1.943001)),
+ (coreg.DhMinimize, (10.0850892, 2.898172, -1.943001)),
(coreg.ICP, (8.73833, 1.584255, -1.943957)),
],
) # type: ignore
def test_coreg_translations__example(
- self, coreg_method__shift: tuple[type[AffineCoreg], tuple[float, float, float]], verbose: bool = False
+ self, coreg_method__shift: tuple[type[AffineCoreg], tuple[float, float, float]]
) -> None:
"""
Test that the translation co-registration outputs are always exactly the same on the real example data.
@@ -303,7 +288,7 @@ def test_coreg_translations__example(
coreg_method, expected_shifts = coreg_method__shift
c = coreg_method(subsample=50000)
- c.fit(ref, tba, inlier_mask=inlier_mask, verbose=verbose, random_state=42)
+ c.fit(ref, tba, inlier_mask=inlier_mask, random_state=42)
# Check the output translations match the exact values
shifts = [c.meta["outputs"]["affine"][k] for k in ["shift_x", "shift_y", "shift_z"]] # type: ignore
@@ -367,7 +352,7 @@ def test_coreg_vertical_translation__synthetic(self, fit_args, vshift) -> None:
@pytest.mark.parametrize("coreg_method__vshift", [(coreg.VerticalShift, -2.305015)]) # type: ignore
def test_coreg_vertical_translation__example(
- self, coreg_method__vshift: tuple[type[AffineCoreg], tuple[float, float, float]], verbose: bool = False
+ self, coreg_method__vshift: tuple[type[AffineCoreg], tuple[float, float, float]]
) -> None:
"""
Test that the vertical translation co-registration output is always exactly the same on the real example data.
@@ -382,7 +367,7 @@ def test_coreg_vertical_translation__example(
# Run co-registration
c = coreg_method(subsample=50000)
- c.fit(ref, tba, inlier_mask=inlier_mask, verbose=verbose, random_state=42)
+ c.fit(ref, tba, inlier_mask=inlier_mask, random_state=42)
# Check the output translations match the exact values
vshift = c.meta["outputs"]["affine"]["shift_z"]
@@ -479,9 +464,7 @@ def test_coreg_rigid__synthetic(self, fit_args, shifts_rotations, coreg_method)
[(coreg.ICP, (8.738332, 1.584255, -1.943957, 0.0069004, -0.00703, -0.0119733))],
) # type: ignore
def test_coreg_rigid__example(
- self,
- coreg_method__shifts_rotations: tuple[type[AffineCoreg], tuple[float, float, float]],
- verbose: bool = False,
+ self, coreg_method__shifts_rotations: tuple[type[AffineCoreg], tuple[float, float, float]]
) -> None:
"""
Test that the rigid co-registration outputs is always exactly the same on the real example data.
@@ -496,9 +479,9 @@ def test_coreg_rigid__example(
# Run co-registration
c = coreg_method(subsample=50000)
- c.fit(ref, tba, inlier_mask=inlier_mask, verbose=verbose, random_state=42)
+ c.fit(ref, tba, inlier_mask=inlier_mask, random_state=42)
- # Check the output translations match the exact values
+ # Check the output translations and rotations match the exact values
fit_matrix = c.meta["outputs"]["affine"]["matrix"]
fit_shifts = fit_matrix[:3, 3]
fit_rotations = pytransform3d.rotations.euler_from_matrix(fit_matrix[0:3, 0:3], i=0, j=1, k=2, extrinsic=True)
diff --git a/tests/test_coreg/test_base.py b/tests/test_coreg/test_base.py
index 5632f7eb..1b1401e4 100644
--- a/tests/test_coreg/test_base.py
+++ b/tests/test_coreg/test_base.py
@@ -69,12 +69,7 @@ class TestCoregClass:
ref, tba, outlines = load_examples() # Load example reference, to-be-aligned and mask.
inlier_mask = ~outlines.create_mask(ref)
- fit_params = dict(
- reference_elev=ref,
- to_be_aligned_elev=tba,
- inlier_mask=inlier_mask,
- verbose=False,
- )
+ fit_params = dict(reference_elev=ref, to_be_aligned_elev=tba, inlier_mask=inlier_mask)
# Create some 3D coordinates with Z coordinates being 0 to try the apply functions.
points_arr = np.array([[1, 2, 3, 4], [1, 2, 3, 4], [0, 0, 0, 0]], dtype="float64").T
points = gpd.GeoDataFrame(
@@ -131,7 +126,7 @@ def recursive_typeddict_items(typed_dict: Mapping[str, Any]) -> Iterable[str]:
# Check that info() contains the mapped string for an example
c = coreg.Coreg(meta={"subsample": 10000})
- assert dict_key_to_str["subsample"] in c.info(verbose=False)
+ assert dict_key_to_str["subsample"] in c.info(as_str=True)
@pytest.mark.parametrize("coreg_class", [coreg.VerticalShift, coreg.ICP, coreg.NuthKaab]) # type: ignore
def test_copy(self, coreg_class: Callable[[], Coreg]) -> None:
@@ -629,7 +624,6 @@ class TestCoregPipeline:
inlier_mask=inlier_mask,
transform=ref.transform,
crs=ref.crs,
- verbose=True,
)
# Create some 3D coordinates with Z coordinates being 0 to try the apply functions.
points_arr = np.array([[1, 2, 3, 4], [1, 2, 3, 4], [0, 0, 0, 0]], dtype="float64").T
@@ -860,7 +854,6 @@ class TestBlockwiseCoreg:
inlier_mask=inlier_mask,
transform=ref.transform,
crs=ref.crs,
- verbose=False,
)
# Create some 3D coordinates with Z coordinates being 0 to try the apply functions.
points_arr = np.array([[1, 2, 3, 4], [1, 2, 3, 4], [0, 0, 0, 0]], dtype="float64").T
@@ -955,7 +948,6 @@ def test_blockwise_coreg_large_gaps(self) -> None:
ddem_post = (aligned - self.ref).data.compressed()
ddem_pre = (tba - self.ref).data.compressed()
assert abs(np.nanmedian(ddem_pre)) > abs(np.nanmedian(ddem_post))
- # TODO: Figure out why STD here is larger since PR #530
# assert np.nanstd(ddem_pre) > np.nanstd(ddem_post)
diff --git a/tests/test_coreg/test_biascorr.py b/tests/test_coreg/test_biascorr.py
index e084445d..7be3175a 100644
--- a/tests/test_coreg/test_biascorr.py
+++ b/tests/test_coreg/test_biascorr.py
@@ -45,32 +45,17 @@ class TestBiasCorr:
# Check all possibilities supported by biascorr:
# Raster-Raster
- fit_args_rst_rst = dict(
- reference_elev=ref,
- to_be_aligned_elev=tba,
- inlier_mask=inlier_mask,
- verbose=True,
- )
+ fit_args_rst_rst = dict(reference_elev=ref, to_be_aligned_elev=tba, inlier_mask=inlier_mask)
# Convert DEMs to points with a bit of subsampling for speed-up
tba_pts = tba.to_pointcloud(data_column_name="z", subsample=50000, random_state=42).ds
ref_pts = ref.to_pointcloud(data_column_name="z", subsample=50000, random_state=42).ds
# Raster-Point
- fit_args_rst_pts = dict(
- reference_elev=ref,
- to_be_aligned_elev=tba_pts,
- inlier_mask=inlier_mask,
- verbose=True,
- )
+ fit_args_rst_pts = dict(reference_elev=ref, to_be_aligned_elev=tba_pts, inlier_mask=inlier_mask)
# Point-Raster
- fit_args_pts_rst = dict(
- reference_elev=ref_pts,
- to_be_aligned_elev=tba,
- inlier_mask=inlier_mask,
- verbose=True,
- )
+ fit_args_pts_rst = dict(reference_elev=ref_pts, to_be_aligned_elev=tba, inlier_mask=inlier_mask)
all_fit_args = [fit_args_rst_rst, fit_args_rst_pts, fit_args_pts_rst]
diff --git a/tests/test_ddem.py b/tests/test_ddem.py
index a5ae9f4b..e41b078f 100644
--- a/tests/test_ddem.py
+++ b/tests/test_ddem.py
@@ -44,7 +44,7 @@ def test_filled_data(self) -> None:
assert ddem2.filled_data is None
- ddem2.interpolate(method="linear")
+ ddem2.interpolate(method="idw")
assert ddem2.fill_method is not None
diff --git a/tests/test_demcollection.py b/tests/test_demcollection.py
index 544d195f..19bf7dd3 100644
--- a/tests/test_demcollection.py
+++ b/tests/test_demcollection.py
@@ -74,7 +74,7 @@ def test_init(self) -> None:
if "NaNs found in dDEM" not in str(exception):
raise exception
- # print(cumulative_dh)
+ # logging.info(cumulative_dh)
# raise NotImplementedError
@@ -119,7 +119,7 @@ def test_ddem_interpolation(self) -> None:
assert dems.ddems[0].filled_data is None
# Interpolate the nans
- dems.ddems[0].interpolate(method="linear")
+ dems.ddems[0].interpolate(method="idw")
# Make sure that the filled_data is available again
assert dems.ddems[0].filled_data is not None
diff --git a/tests/test_doc.py b/tests/test_doc.py
index 8adbcd7a..465e4be6 100644
--- a/tests/test_doc.py
+++ b/tests/test_doc.py
@@ -1,4 +1,5 @@
"""Functions to test the documentation."""
+import logging
import os
import platform
import shutil
@@ -34,7 +35,7 @@ def run_code(filename: str) -> None:
exec(infile.read().replace("plt.show()", "plt.close()"))
except Exception as exception:
if isinstance(exception, DeprecationWarning):
- print(exception)
+ logging.warning(exception)
else:
raise RuntimeError(f"Failed on {filename}") from exception
diff --git a/tests/test_spatialstats.py b/tests/test_spatialstats.py
index 45b4db8a..faf38d38 100644
--- a/tests/test_spatialstats.py
+++ b/tests/test_spatialstats.py
@@ -442,7 +442,7 @@ def test_estimate_model_heteroscedasticity_and_infer_from_stable(self) -> None:
dvalues=self.diff, list_var=[self.slope, self.maximum_curv], unstable_mask=self.outlines
)
- df_binning_2, err_fun_2 = xdem.spatialstats.estimate_model_heteroscedasticity(
+ df_binning_2, err_fun_2 = xdem.spatialstats._estimate_model_heteroscedasticity(
dvalues=self.diff[~self.mask],
list_var=[self.slope[~self.mask], self.maximum_curv[~self.mask]],
list_var_names=["var1", "var2"],
@@ -524,7 +524,7 @@ def test_sample_multirange_variogram_default(self) -> None:
# Check that all type of coordinate inputs work
# Only the array and the ground sampling distance
xdem.spatialstats.sample_empirical_variogram(
- values=self.diff.data, gsd=self.diff.res[0], subsample=10, random_state=42, verbose=True
+ values=self.diff.data, gsd=self.diff.res[0], subsample=10, random_state=42
)
# Test multiple runs
@@ -583,7 +583,7 @@ def test_sample_empirical_variogram_speed(self) -> None:
# Shape
shape = values.shape
- keyword_arguments = {"subsample": subsample, "extent": extent, "shape": shape, "verbose": False}
+ keyword_arguments = {"subsample": subsample, "extent": extent, "shape": shape}
runs, samples, ratio_subsample = xdem.spatialstats._choose_cdist_equidistant_sampling_parameters(
**keyword_arguments
)
@@ -720,7 +720,7 @@ def test_choose_cdist_equidistant_sampling_parameters(self, subsample: int, shap
pdist_pairwise_combinations = subsample**2 / 2
# Run the function
- keyword_arguments = {"subsample": subsample, "extent": extent, "shape": shape, "verbose": False}
+ keyword_arguments = {"subsample": subsample, "extent": extent, "shape": shape}
runs, samples, ratio_subsample = xdem.spatialstats._choose_cdist_equidistant_sampling_parameters(
**keyword_arguments
)
@@ -745,7 +745,7 @@ def test_choose_cdist_equidistant_sampling_parameters(self, subsample: int, shap
def test_errors_subsample_parameter(self) -> None:
"""Tests that an error is raised when the subsample argument is too little"""
- keyword_arguments = {"subsample": 3, "extent": (0, 1, 0, 1), "shape": (10, 10), "verbose": False}
+ keyword_arguments = {"subsample": 3, "extent": (0, 1, 0, 1), "shape": (10, 10)}
with pytest.raises(ValueError, match="The number of subsamples needs to be at least 10."):
xdem.spatialstats._choose_cdist_equidistant_sampling_parameters(**keyword_arguments)
@@ -869,7 +869,7 @@ def test_estimate_model_spatial_correlation_and_infer_from_stable(self) -> None:
zscores = diff_on_stable / errors
# Run wrapper estimate and model function
- emp_vgm_1, params_model_vgm_1, _ = xdem.spatialstats.estimate_model_spatial_correlation(
+ emp_vgm_1, params_model_vgm_1, _ = xdem.spatialstats._estimate_model_spatial_correlation(
dvalues=zscores, list_models=["Gau", "Sph"], subsample=10, random_state=42
)
diff --git a/tests/test_terrain.py b/tests/test_terrain.py
index 9097ec4a..027822d3 100644
--- a/tests/test_terrain.py
+++ b/tests/test_terrain.py
@@ -183,7 +183,7 @@ def test_attribute_functions_against_gdaldem(self, attribute: str) -> None:
"attribute",
["slope_Horn", "aspect_Horn", "hillshade_Horn", "curvature", "profile_curvature", "planform_curvature"],
) # type: ignore
- def test_attribute_functions_against_richdem(self, attribute: str) -> None:
+ def test_attribute_functions_against_richdem(self, attribute: str, get_terrain_attribute_richdem) -> None:
"""
Test that all attribute functions give the same results as those of RichDEM within a small tolerance.
@@ -202,12 +202,14 @@ def test_attribute_functions_against_richdem(self, attribute: str) -> None:
# Functions for RichDEM wrapper methods
functions_richdem = {
- "slope_Horn": lambda dem: xdem.terrain.slope(dem, degrees=True, use_richdem=True),
- "aspect_Horn": lambda dem: xdem.terrain.aspect(dem, degrees=True, use_richdem=True),
- "hillshade_Horn": lambda dem: xdem.terrain.hillshade(dem, use_richdem=True),
- "curvature": lambda dem: xdem.terrain.curvature(dem, use_richdem=True),
- "profile_curvature": lambda dem: xdem.terrain.profile_curvature(dem, use_richdem=True),
- "planform_curvature": lambda dem: xdem.terrain.planform_curvature(dem, use_richdem=True),
+ "slope_Horn": lambda dem: get_terrain_attribute_richdem(dem, attribute="slope", degrees=True),
+ "aspect_Horn": lambda dem: get_terrain_attribute_richdem(dem, attribute="aspect", degrees=True),
+ "hillshade_Horn": lambda dem: get_terrain_attribute_richdem(dem, attribute="hillshade"),
+ "curvature": lambda dem: get_terrain_attribute_richdem(dem, attribute="curvature"),
+ "profile_curvature": lambda dem: get_terrain_attribute_richdem(dem, attribute="profile_curvature"),
+ "planform_curvature": lambda dem: get_terrain_attribute_richdem(
+ dem, attribute="planform_curvature", degrees=True
+ ),
}
# Copy the DEM to ensure that the inter-test state is unchanged, and because the mask will be modified.
@@ -355,15 +357,6 @@ def test_get_terrain_attribute_errors(self) -> None:
"""Test the get_terrain_attribute function raises appropriate errors."""
# Below, re.escape() is needed to match expressions that have special characters (e.g., parenthesis, bracket)
- with pytest.raises(
- ValueError,
- match=re.escape("RichDEM can only compute the slope and aspect using the " "default method of Horn (1981)"),
- ):
- xdem.terrain.slope(self.dem, method="ZevenbergThorne", use_richdem=True)
-
- with pytest.raises(ValueError, match="To derive RichDEM attributes, the DEM passed must be a Raster object"):
- xdem.terrain.slope(self.dem.data, resolution=self.dem.res, use_richdem=True)
-
with pytest.raises(
ValueError,
match=re.escape(
diff --git a/xdem/coreg/affine.py b/xdem/coreg/affine.py
index c2288f7f..69090979 100644
--- a/xdem/coreg/affine.py
+++ b/xdem/coreg/affine.py
@@ -2,8 +2,9 @@
from __future__ import annotations
+import logging
import warnings
-from typing import Any, Callable, Iterable, Literal, TypeVar, overload
+from typing import Any, Callable, Iterable, Literal, TypeVar
import xdem.coreg.base
@@ -31,10 +32,12 @@
_bin_or_and_fit_nd,
_get_subsample_mask_pts_rst,
_preprocess_pts_rst_subsample,
+ _reproject_horizontal_shift_samecrs,
)
from xdem.spatialstats import nmad
try:
+ import pytransform3d.rotations
import pytransform3d.transformations
_HAS_P3D = True
@@ -53,67 +56,6 @@
######################################
-@overload
-def _reproject_horizontal_shift_samecrs(
- raster_arr: NDArrayf,
- src_transform: rio.transform.Affine,
- dst_transform: rio.transform.Affine = None,
- *,
- return_interpolator: Literal[False] = False,
- resampling: Literal["nearest", "linear", "cubic", "quintic", "slinear", "pchip", "splinef2d"] = "linear",
-) -> NDArrayf:
- ...
-
-
-@overload
-def _reproject_horizontal_shift_samecrs(
- raster_arr: NDArrayf,
- src_transform: rio.transform.Affine,
- dst_transform: rio.transform.Affine = None,
- *,
- return_interpolator: Literal[True],
- resampling: Literal["nearest", "linear", "cubic", "quintic", "slinear", "pchip", "splinef2d"] = "linear",
-) -> Callable[[tuple[NDArrayf, NDArrayf]], NDArrayf]:
- ...
-
-
-def _reproject_horizontal_shift_samecrs(
- raster_arr: NDArrayf,
- src_transform: rio.transform.Affine,
- dst_transform: rio.transform.Affine = None,
- return_interpolator: bool = False,
- resampling: Literal["nearest", "linear", "cubic", "quintic", "slinear", "pchip", "splinef2d"] = "linear",
-) -> NDArrayf | Callable[[tuple[NDArrayf, NDArrayf]], NDArrayf]:
- """
- Reproject a raster only for a horizontal shift (transform update) in the same CRS.
-
- This function exists independently of Raster.reproject() because Rasterio has unexplained reprojection issues
- that can create non-negligible sub-pixel shifts that should be crucially avoided for coregistration.
- See https://github.com/rasterio/rasterio/issues/2052#issuecomment-2078732477.
-
- Here we use SciPy interpolation instead, modified for nodata propagation in geoutils.interp_points().
- """
-
- # We are reprojecting the raster array relative to itself without changing its pixel interpretation, so we can
- # force any pixel interpretation (area_or_point) without it having any influence on the result, here "Area"
- if not return_interpolator:
- coords_dst = _coords(transform=dst_transform, area_or_point="Area", shape=raster_arr.shape)
- # If we just want the interpolator, we don't need to coordinates of destination points
- else:
- coords_dst = None
-
- output = _interp_points(
- array=raster_arr,
- area_or_point="Area",
- transform=src_transform,
- points=coords_dst,
- method=resampling,
- return_interpolator=return_interpolator,
- )
-
- return output
-
-
def _check_inputs_bin_before_fit(
bin_before_fit: bool,
fit_optimizer: Callable[..., tuple[NDArrayf, Any]],
@@ -158,7 +100,6 @@ def _iterate_method(
constant_inputs: tuple[Any, ...],
tolerance: float,
max_iterations: int,
- verbose: bool = False,
) -> Any:
"""
Function to iterate a method (e.g. ICP, Nuth and Kääb) until it reaches a tolerance or maximum number of iterations.
@@ -170,7 +111,6 @@ def _iterate_method(
:param constant_inputs: Constant inputs to method, should be all positional arguments after first.
:param tolerance: Tolerance to reach for the method statistic (i.e. maximum value for the statistic).
:param max_iterations: Maximum number of iterations for the method.
- :param verbose: Whether to print progress.
:return: Final output of iterated method.
"""
@@ -179,8 +119,8 @@ def _iterate_method(
new_inputs = iterating_input
# Iteratively run the analysis until the maximum iterations or until the error gets low enough
- # If verbose is True, will use progressbar and print additional statements
- pbar = trange(max_iterations, disable=not verbose, desc=" Progress")
+ # If logging level <= INFO, will use progressbar and print additional statements
+ pbar = trange(max_iterations, disable=logging.getLogger().getEffectiveLevel() > logging.INFO, desc=" Progress")
for i in pbar:
# Apply method and get new statistic to compare to tolerance, new inputs for next iterations, and
@@ -189,11 +129,11 @@ def _iterate_method(
# Print final results
# TODO: Allow to pass a string to _iterate_method on how to print/describe exactly the iterating input
- if verbose:
+ if logging.getLogger().getEffectiveLevel() <= logging.DEBUG:
pbar.write(f" Iteration #{i + 1:d} - Offset: {new_inputs}; Magnitude: {new_statistic}")
if i > 1 and new_statistic < tolerance:
- if verbose:
+ if logging.getLogger().getEffectiveLevel() <= logging.INFO:
pbar.write(f" Last offset was below the residual offset threshold of {tolerance} -> stopping")
break
@@ -305,7 +245,6 @@ def _preprocess_pts_rst_subsample_interpolator(
area_or_point: Literal["Area", "Point"] | None,
z_name: str,
aux_vars: None | dict[str, NDArrayf] = None,
- verbose: bool = False,
) -> tuple[Callable[[float, float], NDArrayf], None | dict[str, NDArrayf], int]:
"""
Mirrors coreg.base._preprocess_pts_rst_subsample, but returning an interpolator for efficiency in iterative methods.
@@ -326,7 +265,6 @@ def _preprocess_pts_rst_subsample_interpolator(
transform=transform,
area_or_point=area_or_point,
aux_vars=aux_vars,
- verbose=verbose,
)
# Return interpolator of elevation differences and subsampled auxiliary variables
@@ -498,7 +436,6 @@ def _nuth_kaab_iteration_step(
aspect: NDArrayf,
res: tuple[int, int],
params_fit_bin: InFitOrBinDict,
- verbose: bool = False,
) -> tuple[tuple[float, float, float], float]:
"""
Iteration step of Nuth and Kääb (2011), passed to the iterate_method function.
@@ -511,7 +448,6 @@ def _nuth_kaab_iteration_step(
:param slope_tan: Array of slope tangent.
:param aspect: Array of aspect.
:param res: Resolution of DEM.
- :param verbose: Whether to print statements.
"""
# Calculate the elevation difference with offsets
@@ -566,7 +502,6 @@ def nuth_kaab(
params_random: InRandomDict,
z_name: str,
weights: NDArrayf | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[tuple[float, float, float], int]:
"""
@@ -574,8 +509,7 @@ def nuth_kaab(
:return: Final estimated offset: east, north, vertical (in georeferenced units).
"""
- if verbose:
- print("Running Nuth and Kääb (2011) coregistration")
+ logging.info("Running Nuth and Kääb (2011) coregistration")
# Check that DEM CRS is projected, otherwise slope is not correctly calculated
if not crs.is_projected:
@@ -603,12 +537,10 @@ def nuth_kaab(
aux_vars=aux_vars,
transform=transform,
area_or_point=area_or_point,
- verbose=verbose,
z_name=z_name,
)
- if verbose:
- print(" Iteratively estimating horizontal shift:")
+ logging.info("Iteratively estimating horizontal shift:")
# Initialise east, north and vertical offset variables (these will be incremented up and down)
initial_offset = (0.0, 0.0, 0.0)
# Resolution
@@ -622,7 +554,6 @@ def nuth_kaab(
constant_inputs=constant_inputs,
tolerance=tolerance,
max_iterations=max_iterations,
- verbose=verbose,
)
return final_offsets, subsample_final
@@ -655,7 +586,6 @@ def _dh_minimize_fit_func(
def _dh_minimize_fit(
dh_interpolator: Callable[[float, float], NDArrayf],
params_fit_or_bin: InFitOrBinDict,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[float, float, float]:
"""
@@ -664,7 +594,6 @@ def _dh_minimize_fit(
:param dh_interpolator: Interpolator returning elevation differences at the subsampled points for a certain
horizontal offset (see _preprocess_pts_rst_subsample_interpolator).
:param params_fit_or_bin: Parameters for fitting or binning.
- :param verbose: Whether to print statements.
:return: Optimized offsets (easing, northing, vertical) in georeferenced unit.
"""
@@ -681,7 +610,6 @@ def fit_func(coords_offsets: tuple[float, float]) -> np.floating[Any]:
if params_fit_or_bin["fit_minimizer"] == scipy.optimize.minimize:
if "method" not in kwargs.keys():
kwargs.update({"method": "Nelder-Mead"})
- kwargs.update({"options": {"xatol": 10e-6, "maxiter": 500}})
# This method has trouble when initialized with 0,0, so defaulting to 1,1
# (tip from Simon Gascoin: https://github.com/GlacioHack/xdem/pull/595#issuecomment-2387104719)
init_offsets = (1, 1)
@@ -713,7 +641,6 @@ def dh_minimize(
params_fit_or_bin: InFitOrBinDict,
z_name: str,
weights: NDArrayf | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[tuple[float, float, float], int]:
"""
@@ -723,8 +650,7 @@ def dh_minimize(
:return: Final estimated offset: east, north, vertical (in georeferenced units).
"""
- if verbose:
- print("Running dh minimization coregistration.")
+ logging.info("Running dh minimization coregistration.")
# Perform preprocessing: subsampling and interpolation of inputs and auxiliary vars at same points
dh_interpolator, _, subsample_final = _preprocess_pts_rst_subsample_interpolator(
@@ -734,15 +660,12 @@ def dh_minimize(
inlier_mask=inlier_mask,
transform=transform,
area_or_point=area_or_point,
- verbose=verbose,
z_name=z_name,
)
# Perform fit
# TODO: To match original implementation, need to add back weight support for point data
- final_offsets = _dh_minimize_fit(
- dh_interpolator=dh_interpolator, params_fit_or_bin=params_fit_or_bin, verbose=verbose
- )
+ final_offsets = _dh_minimize_fit(dh_interpolator=dh_interpolator, params_fit_or_bin=params_fit_or_bin)
return final_offsets, subsample_final
@@ -763,15 +686,13 @@ def vertical_shift(
vshift_reduc_func: Callable[[NDArrayf], np.floating[Any]],
z_name: str,
weights: NDArrayf | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[float, int]:
"""
Vertical shift coregistration, for any point-raster or raster-raster input, including subsampling.
"""
- if verbose:
- print("Running vertical shift coregistration")
+ logging.info("Running vertical shift coregistration")
# Pre-process point-raster inputs to the same subsampled points
sub_ref, sub_tba, _ = _preprocess_pts_rst_subsample(
@@ -783,7 +704,6 @@ def vertical_shift(
crs=crs,
area_or_point=area_or_point,
z_name=z_name,
- verbose=verbose,
)
# Get elevation difference
dh = sub_ref - sub_tba
@@ -794,8 +714,7 @@ def vertical_shift(
# TODO: We might need to define the type of bias_func with Callback protocols to get the optional argument,
# TODO: once we have the weights implemented
- if verbose:
- print("Vertical shift estimated")
+ logging.info("Vertical shift estimated")
# Get final subsample size
subsample_final = len(sub_ref)
@@ -849,6 +768,34 @@ def to_matrix(self) -> NDArrayf:
"""Convert the transform to a 4x4 transformation matrix."""
return self._to_matrix_func()
+ def to_translations(self) -> tuple[float, float, float]:
+ """
+ Extract X/Y/Z translations from the affine transformation matrix.
+
+ :return: Easting, northing and vertical translations (in georeferenced unit).
+ """
+
+ matrix = self.to_matrix()
+ shift_x = matrix[0, 3]
+ shift_y = matrix[1, 3]
+ shift_z = matrix[2, 3]
+
+ return shift_x, shift_y, shift_z
+
+ def to_rotations(self) -> tuple[float, float, float]:
+ """
+ Extract X/Y/Z euler rotations (extrinsic convention) from the affine transformation matrix.
+
+ Warning: This function only works for a rigid transformation (rotation and translation).
+
+ :return: Extrinsinc Euler rotations along easting, northing and vertical directions (degrees).
+ """
+
+ matrix = self.to_matrix()
+ rots = pytransform3d.rotations.euler_from_matrix(matrix, i=0, j=1, k=2, extrinsic=True, strict_check=True)
+ rots = np.rad2deg(np.array(rots))
+ return rots[0], rots[1], rots[2]
+
def centroid(self) -> tuple[float, float, float] | None:
"""Get the centroid of the coregistration, if defined."""
meta_centroid = self._meta["outputs"]["affine"].get("centroid")
@@ -870,7 +817,6 @@ def _preprocess_rst_pts_subsample_interpolator(
crs: rio.crs.CRS | None = None,
area_or_point: Literal["Area", "Point"] | None = None,
z_name: str = "z",
- verbose: bool = False,
) -> tuple[Callable[[float, float], NDArrayf], None | dict[str, NDArrayf]]:
"""
Pre-process raster-raster or point-raster datasets into 1D arrays subsampled at the same points
@@ -892,7 +838,6 @@ def _preprocess_rst_pts_subsample_interpolator(
transform=transform,
area_or_point=area_or_point,
aux_vars=aux_vars,
- verbose=verbose,
)
# Return interpolator of elevation differences and subsampled auxiliary variables
@@ -932,7 +877,7 @@ def from_matrix(cls, matrix: NDArrayf) -> AffineCoreg:
return cls(matrix=valid_matrix)
@classmethod
- def from_translation(cls, x_off: float = 0.0, y_off: float = 0.0, z_off: float = 0.0) -> AffineCoreg:
+ def from_translations(cls, x_off: float = 0.0, y_off: float = 0.0, z_off: float = 0.0) -> AffineCoreg:
"""
Instantiate a generic Coreg class from a X/Y/Z translation.
@@ -944,13 +889,39 @@ def from_translation(cls, x_off: float = 0.0, y_off: float = 0.0, z_off: float =
:returns: An instantiated generic Coreg class.
"""
+ # Initialize a diagonal matrix
matrix = np.diag(np.ones(4, dtype=float))
+ # Add the three translations (which are in the last column)
matrix[0, 3] = x_off
matrix[1, 3] = y_off
matrix[2, 3] = z_off
return cls.from_matrix(matrix)
+ @classmethod
+ def from_rotations(cls, x_rot: float = 0.0, y_rot: float = 0.0, z_rot: float = 0.0) -> AffineCoreg:
+ """
+ Instantiate a generic Coreg class from a X/Y/Z rotation.
+
+ :param x_rot: The rotation (degrees) to apply around the X (west-east) direction.
+ :param y_rot: The rotation (degrees) to apply around the Y (south-north) direction.
+ :param z_rot: The rotation (degrees) to apply around the Z (vertical) direction.
+
+ :raises ValueError: If the given rotation contained invalid values.
+
+ :returns: An instantiated generic Coreg class.
+ """
+
+ # Initialize a diagonal matrix
+ matrix = np.diag(np.ones(4, dtype=float))
+ # Convert rotations to radians
+ e = np.deg2rad(np.array([x_rot, y_rot, z_rot]))
+ # Derive 3x3 rotation matrix, and insert in 4x4 affine matrix
+ rot_matrix = pytransform3d.rotations.matrix_from_euler(e, i=0, j=1, k=2, extrinsic=True)
+ matrix[0:3, 0:3] = rot_matrix
+
+ return cls.from_matrix(matrix)
+
def _to_matrix_func(self) -> NDArrayf:
# FOR DEVELOPERS: This function needs to be implemented if the `self._meta['matrix']` keyword is not None.
@@ -970,8 +941,8 @@ class VerticalShift(AffineCoreg):
Estimates the mean vertical offset between two elevation datasets based on a reductor function (median, mean, or
any custom reductor function).
- The estimated vertical shift is stored in the `self.meta` key "shift_z" (in unit of the elevation dataset inputs,
- typically meters).
+ The estimated vertical shift is stored in the `self.meta["outputs"]["affine"]` key "shift_z" (in unit of the
+ elevation dataset inputs, typically meters).
"""
def __init__(
@@ -1000,7 +971,6 @@ def _fit_rst_rst(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
"""Estimate the vertical shift using the vshift_func."""
@@ -1015,7 +985,6 @@ def _fit_rst_rst(
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
**kwargs,
)
@@ -1030,7 +999,6 @@ def _fit_rst_pts(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
"""Estimate the vertical shift using the vshift_func."""
@@ -1049,7 +1017,6 @@ def _fit_rst_pts(
vshift_reduc_func=self._meta["inputs"]["affine"]["vshift_reduc_func"],
z_name=z_name,
weights=weights,
- verbose=verbose,
**kwargs,
)
@@ -1071,9 +1038,10 @@ class ICP(AffineCoreg):
Estimates a rigid transform (rotation + translation) between two elevation datasets.
- The transform is stored in the `self.meta` key "matrix", with rotation centered on the coordinates in the key
- "centroid". The translation parameters are also stored individually in the keys "shift_x", "shift_y" and "shift_z"
- (in georeferenced units for horizontal shifts, and unit of the elevation dataset inputs for the vertical shift).
+ The estimated transform is stored in the `self.meta["outputs"]["affine"]` key "matrix", with rotation centered
+ on the coordinates in the key "centroid". The translation parameters are also stored individually in the
+ keys "shift_x", "shift_y" and "shift_z" (in georeferenced units for horizontal shifts, and unit of the
+ elevation dataset inputs for the vertical shift).
Requires 'opencv'. See opencv doc for more info:
https://docs.opencv.org/master/dc/d9b/classcv_1_1ppf__match__3d_1_1ICP.html
@@ -1118,7 +1086,6 @@ def _fit_rst_rst(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
"""Estimate the rigid transform from tba_dem to ref_dem."""
@@ -1158,7 +1125,6 @@ def _fit_rst_rst(
transform=transform,
crs=crs,
area_or_point=area_or_point,
- verbose=verbose,
z_name="z",
)
@@ -1173,7 +1139,6 @@ def _fit_rst_pts(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
@@ -1250,8 +1215,7 @@ def _fit_rst_pts(
rej = self._meta["inputs"]["specific"]["rejection_scale"]
num_lv = self._meta["inputs"]["specific"]["num_levels"]
icp = cv2.ppf_match_3d_ICP(max_it, tol, rej, num_lv)
- if verbose:
- print("Running ICP...")
+ logging.info("Running ICP...")
try:
# Use points as reference
_, residual, matrix = icp.registerModelToScene(points["raster"], points["point"])
@@ -1269,8 +1233,7 @@ def _fit_rst_pts(
if ref == "raster":
matrix = xdem.coreg.base.invert_matrix(matrix)
- if verbose:
- print("ICP finished")
+ logging.info("ICP finished")
assert residual < 1000, f"ICP coregistration failed: residual={residual}, threshold: 1000"
@@ -1292,9 +1255,9 @@ class NuthKaab(AffineCoreg):
Estimate horizontal and vertical translations by iterative slope/aspect alignment.
- The translation parameters are stored in the `self.meta` keys "shift_x", "shift_y" and "shift_z" (in georeferenced
- units for horizontal shifts, and unit of the elevation dataset inputs for the vertical shift), as well as
- in the "matrix" transform.
+ The translation parameters are stored in the `self.meta["outputs"]["affine"]` keys "shift_x", "shift_y" and
+ "shift_z" (in georeferenced units for horizontal shifts, and unit of the elevation dataset inputs for the
+ vertical shift), as well as in the "matrix" transform.
"""
def __init__(
@@ -1356,7 +1319,6 @@ def _fit_rst_rst(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
"""Estimate the x/y/z offset between two DEMs."""
@@ -1372,7 +1334,6 @@ def _fit_rst_rst(
z_name=z_name,
weights=weights,
bias_vars=bias_vars,
- verbose=verbose,
**kwargs,
)
@@ -1387,7 +1348,6 @@ def _fit_rst_pts(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
"""
@@ -1408,7 +1368,6 @@ def _fit_rst_pts(
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
params_random=params_random,
params_fit_or_bin=params_fit_or_bin,
max_iterations=self._meta["inputs"]["iterative"]["max_iterations"],
@@ -1439,9 +1398,9 @@ class DhMinimize(AffineCoreg):
Estimates vertical and horizontal translations.
- The translation parameters are stored in the `self.meta` keys "shift_x", "shift_y" and "shift_z" (in georeferenced
- units for horizontal shifts, and unit of the elevation dataset inputs for the vertical shift), as well as
- in the "matrix" transform.
+ The translation parameters are stored in the `self.meta["outputs"]["affine"]` keys "shift_x", "shift_y" and
+ "shift_z" (in georeferenced units for horizontal shifts, and unit of the elevation dataset inputs for the
+ vertical shift), as well as in the "matrix" transform.
"""
def __init__(
@@ -1472,7 +1431,6 @@ def _fit_rst_rst(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
@@ -1487,7 +1445,6 @@ def _fit_rst_rst(
z_name=z_name,
weights=weights,
bias_vars=bias_vars,
- verbose=verbose,
**kwargs,
)
@@ -1502,7 +1459,6 @@ def _fit_rst_pts(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
@@ -1519,7 +1475,6 @@ def _fit_rst_pts(
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
params_random=params_random,
params_fit_or_bin=params_fit_or_bin,
**kwargs,
diff --git a/xdem/coreg/base.py b/xdem/coreg/base.py
index 4e64d37d..c30a0fb6 100644
--- a/xdem/coreg/base.py
+++ b/xdem/coreg/base.py
@@ -5,6 +5,7 @@
import concurrent.futures
import copy
import inspect
+import logging
import warnings
from typing import (
Any,
@@ -39,7 +40,7 @@
raster,
subdivide_array,
)
-from geoutils.raster.georeferencing import _bounds, _res
+from geoutils.raster.georeferencing import _bounds, _coords, _res
from geoutils.raster.interpolate import _interp_points
from geoutils.raster.raster import _cast_pixel_interpretation, _shift_transform
from tqdm import tqdm
@@ -72,7 +73,7 @@
# Map each key name to a descriptor string
dict_key_to_str = {
"subsample": "Subsample size requested",
- "random_state": "Random generator for subsampling and (if applic.) optimizer",
+ "random_state": "Random generator",
"subsample_final": "Subsample size drawn from valid values",
"fit_or_bin": "Fit, bin or bin+fit",
"fit_func": "Function to fit",
@@ -91,22 +92,24 @@
"tolerance": "Tolerance to reach (pixel size)",
"last_iteration": "Iteration at which algorithm stopped",
"all_tolerances": "Tolerances at each iteration",
- "terrain_attribute": "Terrain attribute used for TerrainBias",
- "angle": "Angle used for DirectionalBias",
- "poly_order": "Polynomial order used for Deramp",
- "best_poly_order": "Best polynomial order kept for fit",
- "best_nb_sin_freq": "Best number of sinusoid frequencies kept for fit",
+ "terrain_attribute": "Terrain attribute used for correction",
+ "angle": "Angle of directional correction",
+ "poly_order": "Polynomial order",
+ "best_poly_order": "Best polynomial order",
+ "best_nb_sin_freq": "Best number of sinusoid frequencies",
"vshift_reduc_func": "Reduction function used to remove vertical shift",
"centroid": "Centroid found for affine rotation",
"shift_x": "Eastward shift estimated (georeferenced unit)",
"shift_y": "Northward shift estimated (georeferenced unit)",
"shift_z": "Vertical shift estimated (elevation unit)",
"matrix": "Affine transformation matrix estimated",
+ "rejection_scale": "Rejection scale",
+ "num_levels": "Number of levels",
}
#####################################
# Generic functions for preprocessing
-#####################################
+###########################################
def _calculate_ddem_stats(
@@ -544,7 +547,7 @@ def _postprocess_coreg_apply(
###############################################
-def _get_subsample_on_valid_mask(params_random: InRandomDict, valid_mask: NDArrayb, verbose: bool = False) -> NDArrayb:
+def _get_subsample_on_valid_mask(params_random: InRandomDict, valid_mask: NDArrayb) -> NDArrayb:
"""
Get mask of values to subsample on valid mask (works for both 1D or 2D arrays).
@@ -585,12 +588,9 @@ def _get_subsample_on_valid_mask(params_random: InRandomDict, valid_mask: NDArra
# If no subsample is taken, use all valid values
subsample_mask = valid_mask
- if verbose:
- print(
- "Using a subsample of {} among {} valid values.".format(
- np.count_nonzero(subsample_mask), np.count_nonzero(valid_mask)
- )
- )
+ logging.debug(
+ "Using a subsample of %d among %d valid values.", np.count_nonzero(subsample_mask), np.count_nonzero(valid_mask)
+ )
return subsample_mask
@@ -603,7 +603,6 @@ def _get_subsample_mask_pts_rst(
transform: rio.transform.Affine, # Never None thanks to Coreg.fit() pre-process
area_or_point: Literal["Area", "Point"] | None,
aux_vars: None | dict[str, NDArrayf] = None,
- verbose: bool = False,
) -> NDArrayb:
"""
Get subsample mask for raster-raster or point-raster datasets on valid points of all inputs (including
@@ -640,7 +639,7 @@ def _get_subsample_mask_pts_rst(
# (Others are already checked in pre-processing of Coreg.fit())
# Perform subsampling
- sub_mask = _get_subsample_on_valid_mask(params_random=params_random, valid_mask=valid_mask, verbose=verbose)
+ sub_mask = _get_subsample_on_valid_mask(params_random=params_random, valid_mask=valid_mask)
# For one raster and one point cloud
else:
@@ -669,7 +668,7 @@ def _get_subsample_mask_pts_rst(
).astype(bool)
# If there is a subsample, it needs to be done now on the point dataset to reduce later calculations
- sub_mask = _get_subsample_on_valid_mask(params_random=params_random, valid_mask=valid_mask, verbose=verbose)
+ sub_mask = _get_subsample_on_valid_mask(params_random=params_random, valid_mask=valid_mask)
# TODO: Move check to Coreg.fit()?
@@ -748,7 +747,6 @@ def _preprocess_pts_rst_subsample(
area_or_point: Literal["Area", "Point"] | None,
z_name: str,
aux_vars: None | dict[str, NDArrayf] = None,
- verbose: bool = False,
) -> tuple[NDArrayf, NDArrayf, None | dict[str, NDArrayf]]:
"""
Pre-process raster-raster or point-raster datasets into 1D arrays subsampled at the same points
@@ -767,7 +765,6 @@ def _preprocess_pts_rst_subsample(
transform=transform,
area_or_point=area_or_point,
aux_vars=aux_vars,
- verbose=verbose,
)
# Perform subsampling on mask for all inputs
@@ -792,7 +789,6 @@ def _bin_or_and_fit_nd(
values: NDArrayf,
bias_vars: None | dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[None, tuple[NDArrayf, Any]]:
...
@@ -805,7 +801,6 @@ def _bin_or_and_fit_nd(
values: NDArrayf,
bias_vars: None | dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[pd.DataFrame, None]:
...
@@ -818,7 +813,6 @@ def _bin_or_and_fit_nd(
values: NDArrayf,
bias_vars: None | dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[pd.DataFrame, tuple[NDArrayf, Any]]:
...
@@ -830,7 +824,6 @@ def _bin_or_and_fit_nd(
values: NDArrayf,
bias_vars: None | dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[pd.DataFrame | None, tuple[NDArrayf, Any] | None]:
"""
@@ -843,7 +836,6 @@ def _bin_or_and_fit_nd(
:param values: Valid values to bin or fit.
:param bias_vars: Auxiliary variables for certain bias correction classes, as raster or arrays.
:param weights: Array of weights for the coregistration.
- :param verbose: Print progress messages.
"""
if fit_or_bin is None:
@@ -892,13 +884,11 @@ def _bin_or_and_fit_nd(
# Option 1: Run fit and save optimized function parameters
if fit_or_bin == "fit":
-
- # Print if verbose
- if verbose:
- print(
- "Estimating alignment along variables {} by fitting "
- "with function {}.".format(", ".join(list(bias_vars.keys())), params_fit_or_bin["fit_func"].__name__)
- )
+ logging.debug(
+ "Estimating alignment along variables %s by fitting with function %s.",
+ ", ".join(list(bias_vars.keys())),
+ params_fit_or_bin["fit_func"].__name__,
+ )
results = params_fit_or_bin["fit_optimizer"](
f=params_fit_or_bin["fit_func"],
@@ -912,14 +902,11 @@ def _bin_or_and_fit_nd(
# Option 2: Run binning and save dataframe of result
elif fit_or_bin == "bin":
-
- if verbose:
- print(
- "Estimating alignment along variables {} by binning "
- "with statistic {}.".format(
- ", ".join(list(bias_vars.keys())), params_fit_or_bin["bin_statistic"].__name__
- )
- )
+ logging.debug(
+ "Estimating alignment along variables %s by binning with statistic %s.",
+ ", ".join(list(bias_vars.keys())),
+ params_fit_or_bin["bin_statistic"].__name__,
+ )
df = nd_binning(
values=values,
@@ -932,17 +919,12 @@ def _bin_or_and_fit_nd(
# Option 3: Run binning, then fitting, and save both results
else:
-
- # Print if verbose
- if verbose:
- print(
- "Estimating alignment along variables {} by binning with statistic {} and then fitting "
- "with function {}.".format(
- ", ".join(list(bias_vars.keys())),
- params_fit_or_bin["bin_statistic"].__name__,
- params_fit_or_bin["fit_func"].__name__,
- )
- )
+ logging.debug(
+ "Estimating alignment along variables %s by binning with statistic %s and then fitting with function %s.",
+ ", ".join(list(bias_vars.keys())),
+ params_fit_or_bin["bin_statistic"].__name__,
+ params_fit_or_bin["fit_func"].__name__,
+ )
df = nd_binning(
values=values,
@@ -976,9 +958,7 @@ def _bin_or_and_fit_nd(
absolute_sigma=True,
**kwargs,
)
-
- if verbose:
- print(f"{nd}D bias estimated.")
+ logging.debug("%dD bias estimated.", nd)
return df, results
@@ -1075,7 +1055,6 @@ def _iterate_affine_regrid_small_rotations(
matrix: NDArrayf,
centroid: tuple[float, float, float] | None = None,
resampling: Literal["nearest", "linear", "cubic", "quintic"] = "linear",
- verbose: bool = True,
) -> tuple[NDArrayf, rio.transform.Affine]:
"""
Iterative process to find the best reprojection of affine transformation for small rotations.
@@ -1157,12 +1136,12 @@ def _iterate_affine_regrid_small_rotations(
diff_x = x0 - x
diff_y = y0 - y
- if verbose:
- print(
- f"Residual check at iteration number {niter}:"
- f"\n Mean diff x: {np.nanmean(np.abs(diff_x))}"
- f"\n Mean diff y: {np.nanmean(np.abs(diff_y))}"
- )
+ logging.debug(
+ "Residual check at iteration number %d:" "\n Mean diff x: %f" "\n Mean diff y: %f",
+ niter,
+ np.nanmean(np.abs(diff_x)),
+ np.nanmean(np.abs(diff_y)),
+ )
# Get index of points below tolerance in both X/Y for this subsample (all points before convergence update)
# Nodata values are considered having converged
@@ -1170,12 +1149,11 @@ def _iterate_affine_regrid_small_rotations(
subind_diff_y = np.logical_or(np.abs(diff_y) < (tolerance * res_y), ~np.isfinite(diff_y))
subind_converged = np.logical_and(subind_diff_x, subind_diff_y)
- if verbose:
- print(
- f" Points not within tolerance: "
- f"{np.count_nonzero(~subind_diff_x)} for X; "
- f"{np.count_nonzero(~subind_diff_y)} for Y"
- )
+ logging.debug(
+ " Points not within tolerance: %d for X; %d for Y",
+ np.count_nonzero(~subind_diff_x),
+ np.count_nonzero(~subind_diff_y),
+ )
# If all points left are below convergence, update Z one final time and stop here
if all(subind_converged):
@@ -1226,7 +1204,6 @@ def _apply_matrix_rst(
invert: bool = False,
centroid: tuple[float, float, float] | None = None,
resampling: Literal["nearest", "linear", "cubic", "quintic"] = "linear",
- verbose: bool = True,
force_regrid_method: Literal["iterative", "griddata"] | None = None,
) -> tuple[NDArrayf, rio.transform.Affine]:
"""
@@ -1274,7 +1251,7 @@ def _apply_matrix_rst(
rotations = _get_rotations_from_matrix(matrix)
if all(np.abs(rot) < 20 for rot in rotations) and force_regrid_method is None or force_regrid_method == "iterative":
new_dem, transform = _iterate_affine_regrid_small_rotations(
- dem=dem, transform=transform, matrix=matrix, centroid=centroid, resampling=resampling, verbose=verbose
+ dem=dem, transform=transform, matrix=matrix, centroid=centroid, resampling=resampling
)
return new_dem, transform
@@ -1292,12 +1269,74 @@ def _apply_matrix_rst(
return new_dem, transform
+@overload
+def _reproject_horizontal_shift_samecrs(
+ raster_arr: NDArrayf,
+ src_transform: rio.transform.Affine,
+ dst_transform: rio.transform.Affine = None,
+ *,
+ return_interpolator: Literal[False] = False,
+ resampling: Literal["nearest", "linear", "cubic", "quintic", "slinear", "pchip", "splinef2d"] = "linear",
+) -> NDArrayf:
+ ...
+
+
+@overload
+def _reproject_horizontal_shift_samecrs(
+ raster_arr: NDArrayf,
+ src_transform: rio.transform.Affine,
+ dst_transform: rio.transform.Affine = None,
+ *,
+ return_interpolator: Literal[True],
+ resampling: Literal["nearest", "linear", "cubic", "quintic", "slinear", "pchip", "splinef2d"] = "linear",
+) -> Callable[[tuple[NDArrayf, NDArrayf]], NDArrayf]:
+ ...
+
+
+def _reproject_horizontal_shift_samecrs(
+ raster_arr: NDArrayf,
+ src_transform: rio.transform.Affine,
+ dst_transform: rio.transform.Affine = None,
+ return_interpolator: bool = False,
+ resampling: Literal["nearest", "linear", "cubic", "quintic", "slinear", "pchip", "splinef2d"] = "linear",
+) -> NDArrayf | Callable[[tuple[NDArrayf, NDArrayf]], NDArrayf]:
+ """
+ Reproject a raster only for a horizontal shift (transform update) in the same CRS.
+
+ This function exists independently of Raster.reproject() because Rasterio has unexplained reprojection issues
+ that can create non-negligible sub-pixel shifts that should be crucially avoided for coregistration.
+ See https://github.com/rasterio/rasterio/issues/2052#issuecomment-2078732477.
+
+ Here we use SciPy interpolation instead, modified for nodata propagation in geoutils.interp_points().
+ """
+
+ # We are reprojecting the raster array relative to itself without changing its pixel interpretation, so we can
+ # force any pixel interpretation (area_or_point) without it having any influence on the result, here "Area"
+ if not return_interpolator:
+ coords_dst = _coords(transform=dst_transform, area_or_point="Area", shape=raster_arr.shape)
+ # If we just want the interpolator, we don't need to coordinates of destination points
+ else:
+ coords_dst = None
+
+ output = _interp_points(
+ array=raster_arr,
+ area_or_point="Area",
+ transform=src_transform,
+ points=coords_dst,
+ method=resampling,
+ return_interpolator=return_interpolator,
+ )
+
+ return output
+
+
@overload
def apply_matrix(
elev: NDArrayf,
matrix: NDArrayf,
invert: bool = False,
centroid: tuple[float, float, float] | None = None,
+ resample: bool = True,
resampling: Literal["nearest", "linear", "cubic", "quintic"] = "linear",
transform: rio.transform.Affine = None,
z_name: str = "z",
@@ -1312,6 +1351,7 @@ def apply_matrix(
matrix: NDArrayf,
invert: bool = False,
centroid: tuple[float, float, float] | None = None,
+ resample: bool = True,
resampling: Literal["nearest", "linear", "cubic", "quintic"] = "linear",
transform: rio.transform.Affine = None,
z_name: str = "z",
@@ -1325,6 +1365,7 @@ def apply_matrix(
matrix: NDArrayf,
invert: bool = False,
centroid: tuple[float, float, float] | None = None,
+ resample: bool = True,
resampling: Literal["nearest", "linear", "cubic", "quintic"] = "linear",
transform: rio.transform.Affine = None,
z_name: str = "z",
@@ -1355,6 +1396,8 @@ def apply_matrix(
:param invert: Whether to invert the transformation matrix.
:param centroid: The X/Y/Z transformation centroid. Irrelevant for pure translations.
Defaults to the midpoint (Z=0).
+ :param resample: (For translations) If set to True, will resample output on the translated grid to match the input
+ transform. Otherwise, only the transform will be updated and no resampling is done.
:param resampling: Point interpolation method, one of 'nearest', 'linear', 'cubic', or 'quintic'. For more
information, see scipy.ndimage.map_coordinates and scipy.interpolate.interpn. Default is linear.
:param transform: Geotransform of the DEM, only for DEM passed as 2D array.
@@ -1364,15 +1407,18 @@ def apply_matrix(
:return: Affine transformed elevation point cloud or DEM.
"""
+ # Apply matrix to elevation point cloud
if isinstance(elev, gpd.GeoDataFrame):
return _apply_matrix_pts(epc=elev, matrix=matrix, invert=invert, centroid=centroid, z_name=z_name)
+ # Or apply matrix to raster (often requires re-gridding)
else:
+
+ # First, we apply the affine matrix for the array/transform
if isinstance(elev, gu.Raster):
transform = elev.transform
dem = elev.data.filled(np.nan)
else:
dem = elev
-
applied_dem, out_transform = _apply_matrix_rst(
dem=dem,
transform=transform,
@@ -1382,6 +1428,15 @@ def apply_matrix(
resampling=resampling,
**kwargs,
)
+
+ # Then, if resample is True, we reproject the DEM from its out_transform onto the transform
+ if resample:
+ applied_dem = _reproject_horizontal_shift_samecrs(
+ applied_dem, src_transform=out_transform, dst_transform=transform, resampling=resampling
+ )
+ out_transform = transform
+
+ # We return a raster if input was a raster
if isinstance(elev, gu.Raster):
applied_dem = gu.Raster.from_array(applied_dem, out_transform, elev.crs, elev.nodata)
return applied_dem
@@ -1661,14 +1716,14 @@ def meta(self) -> CoregDict:
return self._meta
@overload
- def info(self, verbose: Literal[True] = ...) -> None:
+ def info(self, as_str: Literal[False] = ...) -> None:
...
@overload
- def info(self, verbose: Literal[False]) -> str:
+ def info(self, as_str: Literal[True]) -> str:
...
- def info(self, verbose: bool = True) -> None | str:
+ def info(self, as_str: bool = False) -> None | str:
"""Summarize information about this coregistration."""
# Define max tabulation: longest name + 2 spaces
@@ -1685,19 +1740,39 @@ def recursive_items(dictionary: Mapping[str, Any]) -> Iterable[tuple[str, Any]]:
existing_deep_keys = [k for k, v in recursive_items(self._meta)]
# Formatting function for key values, rounding up digits for numbers and returning function names
- def format_coregdict_values(val: Any) -> str:
+ def format_coregdict_values(val: Any, tab: int) -> str:
+ """
+ Format coregdict values for printing.
+
+ :param val: Input value.
+ :param tab: Tabulation (if value is printed on multiple lines).
+
+ :return: String representing input value.
+ """
- # Function to round to a certain number of digits relative to magnitude, for floating numbers
- def round_to_n(x: float | np.floating[Any], n: int) -> float | np.floating[Any]:
- return round(x, -int(np.floor(np.log10(x))) + (n - 1)) # type: ignore
+ # Function to get decimal to round to a certain number of digits relative to magnitude, for floating numbers
+ def dec_round_to_n(x: float | np.floating[Any], n: int) -> int:
+ return -int(np.floor(np.log10(np.abs(x)))) + (n - 1)
- # Different formatting depending on key value type
+ # Different formatting to string depending on key value type
if isinstance(val, (float, np.floating)):
- return str(round_to_n(val, 3))
+ if np.isfinite(val):
+ str_val = str(round(val, dec_round_to_n(val, 3)))
+ else:
+ str_val = str(val)
+ elif isinstance(val, np.ndarray):
+ min_val = np.min(val)
+ str_val = str(np.round(val, decimals=dec_round_to_n(min_val, 3)))
elif callable(val):
- return val.__name__
+ str_val = val.__name__
else:
- return str(val)
+ str_val = str(val)
+
+ # Add tabulation if string has a return to line
+ if "\n" in str_val:
+ str_val = "\n".ljust(tab).join(str_val.split("\n"))
+
+ return str_val
# Sublevels of metadata to show
sublevels = {
@@ -1727,7 +1802,7 @@ def round_to_n(x: float | np.floating[Any], n: int) -> float | np.floating[Any]:
if len(existing_level_keys) > 0:
inputs_str += [f" {lv}\n"]
inputs_str += [
- f" {dict_key_to_str[k]}:".ljust(tab) + f"{format_coregdict_values(v)}\n"
+ f" {dict_key_to_str[k]}:".ljust(tab) + f"{format_coregdict_values(v, tab)}\n"
for k, v in existing_level_keys
]
@@ -1743,7 +1818,7 @@ def round_to_n(x: float | np.floating[Any], n: int) -> float | np.floating[Any]:
if len(existing_level_keys) > 0:
outputs_str += [f" {lv}\n"]
outputs_str += [
- f" {dict_key_to_str[k]}:".ljust(tab) + f"{format_coregdict_values(v)}\n"
+ f" {dict_key_to_str[k]}:".ljust(tab) + f"{format_coregdict_values(v, tab)}\n"
for k, v in existing_level_keys
]
elif not self._fit_called:
@@ -1756,13 +1831,13 @@ def round_to_n(x: float | np.floating[Any], n: int) -> float | np.floating[Any]:
final_str = header_str + inputs_str + outputs_str
# Return as string or print (default)
- if verbose:
+ if as_str:
+ return "".join(final_str)
+ else:
print("".join(final_str))
return None
- else:
- return "".join(final_str)
- def _get_subsample_on_valid_mask(self, valid_mask: NDArrayb, verbose: bool = False) -> NDArrayb:
+ def _get_subsample_on_valid_mask(self, valid_mask: NDArrayb) -> NDArrayb:
"""
Get mask of values to subsample on valid mask.
@@ -1773,7 +1848,10 @@ def _get_subsample_on_valid_mask(self, valid_mask: NDArrayb, verbose: bool = Fal
params_random = self._meta["inputs"]["random"]
# Derive subsampling mask
- sub_mask = _get_subsample_on_valid_mask(params_random=params_random, valid_mask=valid_mask, verbose=verbose)
+ sub_mask = _get_subsample_on_valid_mask(
+ params_random=params_random,
+ valid_mask=valid_mask,
+ )
# Write final subsample to class
self._meta["outputs"]["random"] = {"subsample_final": int(np.count_nonzero(sub_mask))}
@@ -1791,7 +1869,6 @@ def _preprocess_rst_pts_subsample(
crs: rio.crs.CRS | None = None,
area_or_point: Literal["Area", "Point"] | None = None,
z_name: str = "z",
- verbose: bool = False,
) -> tuple[NDArrayf, NDArrayf, None | dict[str, NDArrayf]]:
"""
Pre-process raster-raster or point-raster datasets into 1D arrays subsampled at the same points
@@ -1813,7 +1890,6 @@ def _preprocess_rst_pts_subsample(
transform=transform,
area_or_point=area_or_point,
aux_vars=aux_vars,
- verbose=verbose,
)
# Perform subsampling on mask for all inputs
@@ -1844,7 +1920,6 @@ def fit(
crs: rio.crs.CRS | None = None,
area_or_point: Literal["Area", "Point"] | None = None,
z_name: str = "z",
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
**kwargs: Any,
) -> CoregType:
@@ -1861,7 +1936,6 @@ def fit(
:param crs: CRS of the reference elevation, only if provided as 2D array.
:param area_or_point: Pixel interpretation of the DEMs, only if provided as 2D arrays.
:param z_name: Column name to use as elevation, only for point elevation data passed as geodataframe.
- :param verbose: Print progress messages.
:param random_state: Random state or seed number to use for calculations (to fix random sampling).
"""
@@ -1910,7 +1984,6 @@ def fit(
"area_or_point": area_or_point,
"z_name": z_name,
"weights": weights,
- "verbose": verbose,
}
# If bias_vars are defined, update dictionary content to array
@@ -2057,7 +2130,6 @@ def fit_and_apply(
z_name: str = "z",
resample: bool = True,
resampling: str | rio.warp.Resampling = "bilinear",
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
fit_kwargs: dict[str, Any] | None = None,
apply_kwargs: dict[str, Any] | None = None,
@@ -2079,7 +2151,6 @@ def fit_and_apply(
z_name: str = "z",
resample: bool = True,
resampling: str | rio.warp.Resampling = "bilinear",
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
fit_kwargs: dict[str, Any] | None = None,
apply_kwargs: dict[str, Any] | None = None,
@@ -2101,7 +2172,6 @@ def fit_and_apply(
z_name: str = "z",
resample: bool = True,
resampling: str | rio.warp.Resampling = "bilinear",
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
fit_kwargs: dict[str, Any] | None = None,
apply_kwargs: dict[str, Any] | None = None,
@@ -2122,7 +2192,6 @@ def fit_and_apply(
z_name: str = "z",
resample: bool = True,
resampling: str | rio.warp.Resampling = "bilinear",
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
fit_kwargs: dict[str, Any] | None = None,
apply_kwargs: dict[str, Any] | None = None,
@@ -2142,7 +2211,6 @@ def fit_and_apply(
:param resample: If set to True, will reproject output Raster on the same grid as input. Otherwise, \
only the transform might be updated and no resampling is done.
:param resampling: Resampling method if resample is used. Defaults to "bilinear".
- :param verbose: Print progress messages.
:param random_state: Random state or seed number to use for calculations (to fix random sampling).
:param fit_kwargs: Keyword arguments to be passed to fit.
:param apply_kwargs: Keyword argument to be passed to apply.
@@ -2164,7 +2232,6 @@ def fit_and_apply(
crs=crs,
area_or_point=area_or_point,
z_name=z_name,
- verbose=verbose,
random_state=random_state,
**fit_kwargs,
)
@@ -2452,9 +2519,9 @@ def _apply_func(self, **kwargs: Any) -> tuple[NDArrayf | gpd.GeoDataFrame, affin
try:
applied_elev = self._apply_pts(**kwargs)
- # If it doesn't exist, use opencv's perspectiveTransform
+ # If it doesn't exist, use apply_matrix()
except NotImplementedCoregApply:
- if self.is_affine: # This only works on it's rigid, however.
+ if self.is_affine:
applied_elev = _apply_matrix_pts(
epc=kwargs["elev"],
@@ -2473,7 +2540,6 @@ def _bin_or_and_fit_nd( # type: ignore
values: NDArrayf,
bias_vars: None | dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs,
) -> None:
"""
@@ -2496,7 +2562,6 @@ def _bin_or_and_fit_nd( # type: ignore
values=values,
bias_vars=bias_vars,
weights=weights,
- verbose=verbose,
**kwargs,
)
@@ -2541,7 +2606,6 @@ def _fit_rst_rst(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
# FOR DEVELOPERS: This function needs to be implemented by subclassing.
@@ -2558,7 +2622,6 @@ def _fit_rst_pts(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
# FOR DEVELOPERS: This function needs to be implemented by subclassing.
@@ -2574,7 +2637,6 @@ def _fit_pts_pts(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
# FOR DEVELOPERS: This function needs to be implemented by subclassing.
@@ -2622,6 +2684,32 @@ def __init__(self, pipeline: list[Coreg]) -> None:
def __repr__(self) -> str:
return f"Pipeline: {self.pipeline}"
+ @overload
+ def info(self, as_str: Literal[False] = ...) -> None:
+ ...
+
+ @overload
+ def info(self, as_str: Literal[True]) -> str:
+ ...
+
+ def info(self, as_str: bool = False) -> None | str:
+ """Summarize information about this coregistration."""
+
+ # Get the pipeline information for each step as a string
+ final_str = []
+ for i, step in enumerate(self.pipeline):
+
+ final_str.append(f"Pipeline step {i}:\n" f"################\n")
+ step_str = step.info(as_str=True)
+ final_str.append(step_str)
+
+ # Return as string or print (default)
+ if as_str:
+ return "".join(final_str)
+ else:
+ print("".join(final_str))
+ return None
+
def copy(self: CoregType) -> CoregType:
"""Return an identical copy of the class."""
new_coreg = self.__new__(type(self))
@@ -2685,7 +2773,6 @@ def fit(
crs: rio.crs.CRS | None = None,
area_or_point: Literal["Area", "Point"] | None = None,
z_name: str = "z",
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
**kwargs: Any,
) -> CoregType:
@@ -2721,8 +2808,7 @@ def fit(
out_transform = transform
for i, coreg in enumerate(self.pipeline):
- if verbose:
- print(f"Running pipeline step: {i + 1} / {len(self.pipeline)}")
+ logging.debug("Running pipeline step: %d / %d", i + 1, len(self.pipeline))
main_args_fit = {
"reference_elev": ref_dem,
@@ -2732,7 +2818,6 @@ def fit(
"crs": crs,
"z_name": z_name,
"weights": weights,
- "verbose": verbose,
"subsample": subsample,
"random_state": random_state,
}
@@ -2958,7 +3043,6 @@ def fit(
crs: rio.crs.CRS | None = None,
area_or_point: Literal["Area", "Point"] | None = None,
z_name: str = "z",
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
**kwargs: Any,
) -> CoregType:
@@ -2995,11 +3079,19 @@ def fit(
area_or_point=area_or_point,
)
+ # Define inlier mask if None, before indexing subdivided array in process function below
+ if inlier_mask is None:
+ mask = np.ones(tba_dem.shape, dtype=bool)
+ else:
+ mask = inlier_mask
+
groups = self.subdivide_array(tba_dem.shape if isinstance(tba_dem, np.ndarray) else ref_dem.shape)
indices = np.unique(groups)
- progress_bar = tqdm(total=indices.size, desc="Processing chunks", disable=(not verbose))
+ progress_bar = tqdm(
+ total=indices.size, desc="Processing chunks", disable=logging.getLogger().getEffectiveLevel() > logging.INFO
+ )
def process(i: int) -> dict[str, Any] | BaseException | None:
"""
@@ -3010,10 +3102,10 @@ def process(i: int) -> dict[str, Any] | BaseException | None:
* If it fails: The associated exception.
* If the block is empty: None
"""
- inlier_mask = groups == i
+ group_mask = groups == i
# Find the corresponding slice of the inlier_mask to subset the data
- rows, cols = np.where(inlier_mask)
+ rows, cols = np.where(group_mask)
arrayslice = np.s_[rows.min() : rows.max() + 1, cols.min() : cols.max() + 1]
# Copy a subset of the two DEMs, the mask, the coreg instance, and make a new subset transform
@@ -3022,7 +3114,7 @@ def process(i: int) -> dict[str, Any] | BaseException | None:
if any(np.all(~np.isfinite(dem)) for dem in (ref_subset, tba_subset)):
return None
- mask_subset = inlier_mask[arrayslice].copy()
+ mask_subset = mask[arrayslice].copy()
west, top = rio.transform.xy(transform, min(rows), min(cols), offset="ul")
transform_subset = rio.transform.from_origin(west, top, transform.a, -transform.e) # type: ignore
procstep = self.procstep.copy()
@@ -3041,7 +3133,6 @@ def process(i: int) -> dict[str, Any] | BaseException | None:
z_name=z_name,
subsample=subsample,
random_state=random_state,
- verbose=verbose,
)
nmad, median = procstep.error(
reference_elev=ref_subset,
diff --git a/xdem/coreg/biascorr.py b/xdem/coreg/biascorr.py
index 83e5eda3..a51042b5 100644
--- a/xdem/coreg/biascorr.py
+++ b/xdem/coreg/biascorr.py
@@ -1,6 +1,7 @@
"""Bias corrections (i.e., non-affine coregistration) classes."""
from __future__ import annotations
+import logging
from typing import Any, Callable, Iterable, Literal, TypeVar
import geopandas as gpd
@@ -23,6 +24,8 @@ class BiasCorr(Coreg):
Variables for bias-correction can include the elevation coordinates (deramping, directional biases), terrain
attributes (terrain corrections), or any other user-input variable (quality metrics, land cover).
+
+ The binning and/or fitting correction parameters are stored in the `self.meta["outputs"]["fitorbin"]`.
"""
def __init__(
@@ -41,7 +44,7 @@ def __init__(
"""
Instantiate an N-dimensional bias correction using binning, fitting or both sequentially.
- All "fit_" arguments apply to "fit" and "bin_and_fit", and "bin_" arguments to "bin" and "bin_and_fit".
+ All fit arguments apply to "fit" and "bin_and_fit", and bin arguments to "bin" and "bin_and_fit".
:param fit_or_bin: Whether to fit or bin, or both. Use "fit" to correct by optimizing a function or
"bin" to correct with a statistic of central tendency in defined bins, or "bin_and_fit" to perform a fit on
@@ -153,7 +156,6 @@ def _fit_rst_rst_and_rst_pts( # type: ignore
z_name: str,
bias_vars: None | dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs,
) -> None:
"""Function for fitting raster-raster and raster-point for bias correction methods."""
@@ -168,7 +170,6 @@ def _fit_rst_rst_and_rst_pts( # type: ignore
area_or_point=area_or_point,
z_name=z_name,
aux_vars=bias_vars,
- verbose=verbose,
)
# Derive difference to get dh
@@ -179,7 +180,6 @@ def _fit_rst_rst_and_rst_pts( # type: ignore
values=diff,
bias_vars=sub_bias_vars,
weights=weights,
- verbose=verbose,
**kwargs,
)
@@ -194,7 +194,6 @@ def _fit_rst_rst(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
"""Called by other classes"""
@@ -209,7 +208,6 @@ def _fit_rst_rst(
z_name=z_name,
weights=weights,
bias_vars=bias_vars,
- verbose=verbose,
**kwargs,
)
@@ -224,7 +222,6 @@ def _fit_rst_pts(
z_name: str,
weights: NDArrayf | None = None,
bias_vars: dict[str, NDArrayf] | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> None:
"""Called by other classes"""
@@ -239,7 +236,6 @@ def _fit_rst_pts(
z_name=z_name,
weights=weights,
bias_vars=bias_vars,
- verbose=verbose,
**kwargs,
)
@@ -298,6 +294,8 @@ def _apply_rst( # type: ignore
class DirectionalBias(BiasCorr):
"""
Bias correction for directional biases, for example along- or across-track of satellite angle.
+
+ The binning and/or fitting correction parameters are stored in the `self.meta["outputs"]["fitorbin"]`.
"""
def __init__(
@@ -344,24 +342,16 @@ def _fit_rst_rst( # type: ignore
z_name: str,
bias_vars: dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs,
) -> None:
- if verbose:
- print("Estimating rotated coordinates.")
+ logging.info("Estimating rotated coordinates.")
x, _ = gu.raster.get_xy_rotated(
raster=gu.Raster.from_array(data=ref_elev, crs=crs, transform=transform, nodata=-9999),
along_track_angle=self._meta["inputs"]["specific"]["angle"],
)
- # Parameters dependent on resolution cannot be derived from the rotated x coordinates, need to be passed below
- if "hop_length" not in kwargs:
- # The hop length will condition jump in function values, need to be larger than average resolution
- average_res = (transform[0] + abs(transform[4])) / 2
- kwargs.update({"hop_length": average_res})
-
super()._fit_rst_rst_and_rst_pts(
ref_elev=ref_elev,
tba_elev=tba_elev,
@@ -372,7 +362,6 @@ def _fit_rst_rst( # type: ignore
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
**kwargs,
)
@@ -387,15 +376,13 @@ def _fit_rst_pts( # type: ignore
z_name: str,
bias_vars: dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs,
) -> None:
# Figure out which data is raster format to get gridded attributes
rast_elev = ref_elev if not isinstance(ref_elev, gpd.GeoDataFrame) else tba_elev
- if verbose:
- print("Estimating rotated coordinates.")
+ logging.info("Estimating rotated coordinates.")
x, _ = gu.raster.get_xy_rotated(
raster=gu.Raster.from_array(data=rast_elev, crs=crs, transform=transform, nodata=-9999),
@@ -418,7 +405,6 @@ def _fit_rst_pts( # type: ignore
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
**kwargs,
)
@@ -447,6 +433,8 @@ class TerrainBias(BiasCorr):
With elevation: often useful for nadir image DEM correction, where the focal length is slightly miscalculated.
With curvature: often useful for a difference of DEMs with different effective resolution.
+ The binning and/or fitting correction parameters are stored in the `self.meta["outputs"]["fitorbin"]`.
+
DISCLAIMER: An elevation correction may introduce error when correcting non-photogrammetric biases, as generally
elevation biases are interlinked with curvature biases.
See Gardelle et al. (2012) (Figure 2), http://dx.doi.org/10.3189/2012jog11j175, for curvature-related biases.
@@ -506,7 +494,6 @@ def _fit_rst_rst( # type: ignore
z_name: str,
bias_vars: dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs,
) -> None:
@@ -537,7 +524,6 @@ def _fit_rst_rst( # type: ignore
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
**kwargs,
)
@@ -552,7 +538,6 @@ def _fit_rst_pts( # type: ignore
z_name: str,
bias_vars: dict[str, NDArrayf] = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs,
) -> None:
@@ -586,7 +571,6 @@ def _fit_rst_pts( # type: ignore
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
**kwargs,
)
@@ -617,7 +601,10 @@ def _apply_rst(
class Deramp(BiasCorr):
"""
Correct for a 2D polynomial along X/Y coordinates, for example from residual camera model deformations
- (dome-like errors).
+ (dome-like errors) or tilts (rotational errors).
+
+ The correction parameters are stored in the `self.meta["outputs"]["fitorbin"]` key "fit_params", that can be passed
+ to the associated function in key "fit_func".
"""
def __init__(
@@ -670,7 +657,6 @@ def _fit_rst_rst( # type: ignore
z_name: str,
bias_vars: dict[str, NDArrayf] | None = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs,
) -> None:
@@ -690,7 +676,6 @@ def _fit_rst_rst( # type: ignore
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
p0=p0,
**kwargs,
)
@@ -706,7 +691,6 @@ def _fit_rst_pts( # type: ignore
z_name: str,
bias_vars: dict[str, NDArrayf] | None = None,
weights: None | NDArrayf = None,
- verbose: bool = False,
**kwargs,
) -> None:
@@ -729,7 +713,6 @@ def _fit_rst_pts( # type: ignore
area_or_point=area_or_point,
z_name=z_name,
weights=weights,
- verbose=verbose,
p0=p0,
**kwargs,
)
diff --git a/xdem/coreg/workflows.py b/xdem/coreg/workflows.py
index 44e66471..8c1b0349 100644
--- a/xdem/coreg/workflows.py
+++ b/xdem/coreg/workflows.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+import logging
+
import geoutils as gu
import matplotlib.pyplot as plt
import numpy as np
@@ -55,14 +57,14 @@ def create_inlier_mask(
# - Sanity check on inputs - #
# Check correct input type of shp_list
if not isinstance(shp_list, (list, tuple)):
- raise ValueError("`shp_list` must be a list/tuple")
+ raise ValueError("Argument `shp_list` must be a list/tuple.")
for el in shp_list:
if not isinstance(el, (str, gu.Vector)):
- raise ValueError("`shp_list` must be a list/tuple of strings or geoutils.Vector instance")
+ raise ValueError("Argument `shp_list` must be a list/tuple of strings or geoutils.Vector instance.")
# Check correct input type of inout
if not isinstance(inout, (list, tuple)):
- raise ValueError("`inout` must be a list/tuple")
+ raise ValueError("Argument `inout` must be a list/tuple.")
if len(shp_list) > 0:
if len(inout) == 0:
@@ -72,18 +74,18 @@ def create_inlier_mask(
# Check that inout contains only 1 and -1
not_valid = [el for el in np.unique(inout) if ((el != 1) & (el != -1))]
if len(not_valid) > 0:
- raise ValueError("`inout` must contain only 1 and -1")
+ raise ValueError("Argument `inout` must contain only 1 and -1.")
else:
- raise ValueError("`inout` must be of same length as shp")
+ raise ValueError("Argument `inout` must be of same length as shp.")
# Check slope_lim type
if not isinstance(slope_lim, (list, tuple)):
- raise ValueError("`slope_lim` must be a list/tuple")
+ raise ValueError("Argument `slope_lim` must be a list/tuple.")
if len(slope_lim) != 2:
- raise ValueError("`slope_lim` must contain 2 elements")
+ raise ValueError("Argument `slope_lim` must contain 2 elements.")
for el in slope_lim:
if (not isinstance(el, (int, float, np.integer, np.floating))) or (el < 0) or (el > 90):
- raise ValueError("`slope_lim` must be a tuple/list of 2 elements in the range [0-90]")
+ raise ValueError("Argument `slope_lim` must be a tuple/list of 2 elements in the range [0-90].")
# Initialize inlier_mask with no masked pixel
inlier_mask = np.ones(src_dem.data.shape, dtype="bool")
@@ -140,7 +142,6 @@ def dem_coregistration(
slope_lim: list[Number] | tuple[Number, Number] = (0.1, 40),
plot: bool = False,
out_fig: str = None,
- verbose: bool = False,
) -> tuple[DEM, Coreg, pd.DataFrame, NDArrayf]:
"""
A one-line function to coregister a selected DEM to a reference DEM.
@@ -168,7 +169,6 @@ def dem_coregistration(
be excluded.
:param plot: Set to True to plot a figure of elevation diff before/after coregistration.
:param out_fig: Path to the output figure. If None will display to screen.
- :param verbose: Set to True to print details on screen during coregistration.
:returns: A tuple containing 1) coregistered DEM as an xdem.DEM instance 2) the coregistration method \
3) DataFrame of coregistration statistics (count of obs, median and NMAD over stable terrain) before and after \
@@ -176,29 +176,28 @@ def dem_coregistration(
"""
# Check inputs
if not isinstance(coreg_method, Coreg):
- raise ValueError("`coreg_method` must be an xdem.coreg instance (e.g. xdem.coreg.NuthKaab())")
+ raise ValueError("Argument `coreg_method` must be an xdem.coreg instance (e.g. xdem.coreg.NuthKaab()).")
if isinstance(ref_dem_path, str):
if not isinstance(src_dem_path, str):
raise ValueError(
- f"`ref_dem_path` is string but `src_dem_path` has type {type(src_dem_path)}."
+ f"Argument `ref_dem_path` is string but `src_dem_path` has type {type(src_dem_path)}."
"Both must have same type."
)
elif isinstance(ref_dem_path, gu.Raster):
if not isinstance(src_dem_path, gu.Raster):
raise ValueError(
- f"`ref_dem_path` is of Raster type but `src_dem_path` has type {type(src_dem_path)}."
+ f"Argument `ref_dem_path` is of Raster type but `src_dem_path` has type {type(src_dem_path)}."
"Both must have same type."
)
else:
- raise ValueError("`ref_dem_path` must be either a string or a Raster")
+ raise ValueError("Argument `ref_dem_path` must be either a string or a Raster.")
if grid not in ["ref", "src"]:
- raise ValueError(f"`grid` must be either 'ref' or 'src' - currently set to {grid}")
+ raise ValueError(f"Argument `grid` must be either 'ref' or 'src' - currently set to {grid}.")
# Load both DEMs
- if verbose:
- print("Loading and reprojecting input data")
+ logging.info("Loading and reprojecting input data")
if isinstance(ref_dem_path, str):
if grid == "ref":
@@ -219,8 +218,7 @@ def dem_coregistration(
src_dem = DEM(src_dem.astype(np.float32))
# Create raster mask
- if verbose:
- print("Creating mask of inlier pixels")
+ logging.info("Creating mask of inlier pixels")
inlier_mask = create_inlier_mask(
src_dem,
@@ -242,7 +240,7 @@ def dem_coregistration(
med_orig, nmad_orig = np.median(inlier_data), nmad(inlier_data)
# Coregister to reference - Note: this will spread NaN
- coreg_method.fit(ref_dem, src_dem, inlier_mask, verbose=verbose)
+ coreg_method.fit(ref_dem, src_dem, inlier_mask)
dem_coreg = coreg_method.apply(src_dem, resample=resample, resampling=resampling)
# Calculate coregistered ddem (might need resampling if resample set to False), needed for stats and plot only
diff --git a/xdem/ddem.py b/xdem/ddem.py
index 589eaec2..487fe903 100644
--- a/xdem/ddem.py
+++ b/xdem/ddem.py
@@ -2,7 +2,7 @@
from __future__ import annotations
import warnings
-from typing import Any
+from typing import Any, Literal
import geoutils as gu
import numpy as np
@@ -163,7 +163,7 @@ def from_array(
def interpolate(
self,
- method: str = "linear",
+ method: Literal["idw", "local_hypsometric", "regional_hypsometric"] = "idw",
reference_elevation: NDArrayf | np.ma.masked_array[Any, np.dtype[np.floating[Any]]] | xdem.DEM = None,
mask: NDArrayf | xdem.DEM | gu.Vector = None,
) -> NDArrayf | None:
@@ -188,8 +188,8 @@ def interpolate(
f" different from 'self' ({self.data.shape})"
)
- if method == "linear":
- self.filled_data = xdem.volume.linear_interpolation(self.data)
+ if method == "idw":
+ self.filled_data = xdem.volume.idw_interpolation(self.data)
elif method == "local_hypsometric":
assert reference_elevation is not None
assert mask is not None
@@ -231,7 +231,7 @@ def interpolate(
diff = abs(np.nanmean(interpolated_ddem - self.data))
assert diff < 0.01, (diff, self.data.mean())
- self.filled_data = xdem.volume.linear_interpolation(interpolated_ddem)
+ self.filled_data = xdem.volume.idw_interpolation(interpolated_ddem)
elif method == "regional_hypsometric":
assert reference_elevation is not None
diff --git a/xdem/dem.py b/xdem/dem.py
index c235b897..7d6533d8 100644
--- a/xdem/dem.py
+++ b/xdem/dem.py
@@ -3,7 +3,7 @@
import pathlib
import warnings
-from typing import Any, Literal, overload
+from typing import Any, Callable, Literal, overload
import geopandas as gpd
import numpy as np
@@ -21,6 +21,7 @@
from xdem.spatialstats import (
infer_heteroscedasticity_from_stable,
infer_spatial_correlation_from_stable,
+ nmad,
)
from xdem.vcrs import (
_build_ccrs_from_crs_and_vcrs,
@@ -110,7 +111,10 @@ def __init__(
# Ensure DEM has only one band: self.bands can be None when data is not loaded through the Raster class
if self.bands is not None and len(self.bands) > 1:
- raise ValueError("DEM rasters should be composed of one band only")
+ raise ValueError(
+ "DEM rasters should be composed of one band only. Either use argument `bands` to specify "
+ "a single band on opening, or use .split_bands() on an opened raster."
+ )
# If the CRS in the raster metadata has a 3rd dimension, could set it as a vertical reference
vcrs_from_crs = _vcrs_from_crs(CRS(self.crs))
@@ -375,69 +379,44 @@ def to_vcrs(
)
@copy_doc(terrain, remove_dem_res_params=True)
- def slope(
- self,
- method: str = "Horn",
- degrees: bool = True,
- use_richdem: bool = False,
- ) -> RasterType:
- return terrain.slope(self, method=method, degrees=degrees, use_richdem=use_richdem)
+ def slope(self, method: str = "Horn", degrees: bool = True) -> RasterType:
+ return terrain.slope(self, method=method, degrees=degrees)
@copy_doc(terrain, remove_dem_res_params=True)
def aspect(
self,
method: str = "Horn",
degrees: bool = True,
- use_richdem: bool = False,
) -> RasterType:
- return terrain.aspect(self, method=method, degrees=degrees, use_richdem=use_richdem)
+ return terrain.aspect(self, method=method, degrees=degrees)
@copy_doc(terrain, remove_dem_res_params=True)
def hillshade(
- self,
- method: str = "Horn",
- azimuth: float = 315.0,
- altitude: float = 45.0,
- z_factor: float = 1.0,
- use_richdem: bool = False,
+ self, method: str = "Horn", azimuth: float = 315.0, altitude: float = 45.0, z_factor: float = 1.0
) -> RasterType:
- return terrain.hillshade(
- self, method=method, azimuth=azimuth, altitude=altitude, z_factor=z_factor, use_richdem=use_richdem
- )
+ return terrain.hillshade(self, method=method, azimuth=azimuth, altitude=altitude, z_factor=z_factor)
@copy_doc(terrain, remove_dem_res_params=True)
- def curvature(
- self,
- use_richdem: bool = False,
- ) -> RasterType:
+ def curvature(self) -> RasterType:
- return terrain.curvature(self, use_richdem=use_richdem)
+ return terrain.curvature(self)
@copy_doc(terrain, remove_dem_res_params=True)
- def planform_curvature(
- self,
- use_richdem: bool = False,
- ) -> RasterType:
+ def planform_curvature(self) -> RasterType:
- return terrain.planform_curvature(self, use_richdem=use_richdem)
+ return terrain.planform_curvature(self)
@copy_doc(terrain, remove_dem_res_params=True)
- def profile_curvature(
- self,
- use_richdem: bool = False,
- ) -> RasterType:
+ def profile_curvature(self) -> RasterType:
- return terrain.profile_curvature(self, use_richdem=use_richdem)
+ return terrain.profile_curvature(self)
@copy_doc(terrain, remove_dem_res_params=True)
- def maximum_curvature(
- self,
- use_richdem: bool = False,
- ) -> RasterType:
+ def maximum_curvature(self) -> RasterType:
- return terrain.maximum_curvature(self, use_richdem=use_richdem)
+ return terrain.maximum_curvature(self)
@copy_doc(terrain, remove_dem_res_params=True)
def topographic_position_index(self, window_size: int = 3) -> RasterType:
@@ -505,54 +484,109 @@ def coregister_3d(
def estimate_uncertainty(
self,
- other_dem: DEM,
+ other_elev: DEM | gpd.GeoDataFrame,
stable_terrain: Mask | NDArrayb = None,
+ approach: Literal["H2022", "R2009", "Basic"] = "H2022",
precision_of_other: Literal["finer"] | Literal["same"] = "finer",
+ spread_estimator: Callable[[NDArrayf], np.floating[Any]] = nmad,
+ variogram_estimator: Literal["matheron", "cressie", "genton", "dowd"] = "dowd",
list_vars: tuple[RasterType | str, ...] = ("slope", "maximum_curvature"),
- list_vario_models: str | tuple[str, ...] = ("spherical", "spherical"),
+ list_vario_models: str | tuple[str, ...] = ("gaussian", "spherical"),
+ z_name: str = "z",
+ random_state: int | np.random.Generator | None = None,
) -> tuple[RasterType, Variogram]:
"""
Estimate uncertainty of DEM.
- Derives a map of per-pixel errors (based on slope and curvature by default) and a function describing the
+ Derives either a map of variable errors (based on slope and curvature by default) and a function describing the
spatial correlation of error (between 0 and 1) with spatial lag (distance between observations).
- Uses stable terrain as an error proxy and assumes a higher or similar-precision DEM is used as reference,
- see Hugonnet et al. (2022).
+ Uses stable terrain as an error proxy and assumes a higher or similar-precision DEM is used as reference.
+
+ See Hugonnet et al. (2022) for methodological details.
- :param other_dem: Other DEM to use for estimation, either of finer or similar precision for reliable estimates.
+ :param other_elev: Other elevation dataset to use for estimation, either of finer or similar precision for
+ reliable estimates.
:param stable_terrain: Mask of stable terrain to use as error proxy.
+ :param approach: Whether to use Hugonnet et al., 2022 (variable errors, multiple ranges of error correlation),
+ or Rolstad et al., 2009 (constant error, multiple ranges of error correlation), or a basic approach
+ (constant error, single range of error correlation). Note that all approaches use robust estimators of
+ variance (NMAD) and variograms (Dowd) by default, despite not being used in Rolstad et al., 2009. These
+ estimators can be tuned separately.
:param precision_of_other: Whether finer precision (3 times more precise = 95% of estimated error will come from
this DEM) or similar precision (for instance another acquisition of the same DEM).
+ :param spread_estimator: Estimator for statistical dispersion (e.g., standard deviation), defaults to the
+ normalized median absolute deviation (NMAD) for robustness.
+ :param variogram_estimator: Estimator for empirical variogram, defaults to Dowd for robustness and consistency
+ with the NMAD estimator for the spread.
+ :param z_name: Column name to use as elevation, only for point elevation data passed as geodataframe.
:param list_vars: Variables to use to predict error variability (= elevation heteroscedasticity). Either rasters
or names of a terrain attributes. Defaults to slope and maximum curvature of the DEM.
:param list_vario_models: Variogram forms to model the spatial correlation of error. A list translates into
- a sum of models.
+ a sum of models. Uses three by default for a method allowing multiple correlation range, otherwise one.
:return: Uncertainty raster, Variogram of uncertainty correlation.
"""
- # Elevation change
- dh = other_dem.reproject(self, silent=True) - self
+ # Summarize approach steps
+ approach_dict = {
+ "H2022": {"heterosc": True, "multi_range": True},
+ "R2009": {"heterosc": False, "multi_range": True},
+ "Basic": {"heterosc": False, "multi_range": False},
+ }
+
+ # Elevation change with the other DEM or elevation point cloud
+ if isinstance(other_elev, DEM):
+ dh = other_elev.reproject(self, silent=True) - self
+ elif isinstance(other_elev, gpd.GeoDataFrame):
+ other_elev = other_elev.to_crs(self.crs)
+ points = (other_elev.geometry.x.values, other_elev.geometry.y.values)
+ dh = other_elev[z_name].values - self.interp_points(points)
+ stable_terrain = stable_terrain
+ else:
+ raise TypeError("Other elevation should be a DEM or elevation point cloud object.")
# If the precision of the other DEM is the same, divide the dh values by sqrt(2)
# See Equation 7 and 8 of Hugonnet et al. (2022)
if precision_of_other == "same":
dh /= np.sqrt(2)
- # Derive terrain attributes of DEM if string are passed in the list of variables
- list_var_rast = []
- for var in list_vars:
- if isinstance(var, str):
- list_var_rast.append(getattr(terrain, var)(self))
- else:
- list_var_rast.append(var)
+ # If approach allows heteroscedasticity, derive a map of errors
+ if approach_dict[approach]["heterosc"]:
+ # Derive terrain attributes of DEM if string are passed in the list of variables
+ list_var_rast = []
+ for var in list_vars:
+ if isinstance(var, str):
+ list_var_rast.append(getattr(terrain, var)(self))
+ else:
+ list_var_rast.append(var)
+
+ # Estimate variable error from these variables
+ sig_dh = infer_heteroscedasticity_from_stable(
+ dvalues=dh, list_var=list_var_rast, spread_statistic=spread_estimator, stable_mask=stable_terrain
+ )[0]
+ # Otherwise, return a constant error raster
+ else:
+ sig_dh = self.copy(new_array=spread_estimator(dh[stable_terrain]) * np.ones(self.shape))
- # Estimate per-pixel uncertainty
- sig_dh = infer_heteroscedasticity_from_stable(dvalues=dh, list_var=list_var_rast, stable_mask=stable_terrain)[0]
+ # If the approach does not allow multiple ranges of spatial correlation
+ if not approach_dict[approach]["multi_range"]:
+ if not isinstance(list_vario_models, str) and len(list_vario_models) > 1:
+ warnings.warn(
+ "Several variogram models passed but this approach uses a single range,"
+ "keeping only the first model.",
+ category=UserWarning,
+ )
+ list_vario_models = list_vario_models[0]
- # Estimate spatial correlation
+ # Otherwise keep all ranges
corr_sig = infer_spatial_correlation_from_stable(
- dvalues=dh, list_models=list(list_vario_models), stable_mask=stable_terrain, errors=sig_dh
+ dvalues=dh,
+ list_models=list(list_vario_models),
+ stable_mask=stable_terrain,
+ errors=sig_dh,
+ estimator=variogram_estimator,
+ random_state=random_state,
)[2]
+
return sig_dh, corr_sig
diff --git a/xdem/demcollection.py b/xdem/demcollection.py
index 0ec72181..52cd1681 100644
--- a/xdem/demcollection.py
+++ b/xdem/demcollection.py
@@ -36,7 +36,9 @@ def __init__(
if timestamps is None:
timestamp_attributes = [dem.datetime for dem in dems]
if any(stamp is None for stamp in timestamp_attributes):
- raise ValueError("'timestamps' not provided and the given DEMs do not all have datetime attributes")
+ raise ValueError(
+ "Argument `timestamps` not provided and the given DEMs do not all have datetime " "attributes"
+ )
timestamps = timestamp_attributes
@@ -183,7 +185,7 @@ def get_dh_series(
:returns: A dataframe of dH values and respective areas with an Interval[Timestamp] index.
"""
if len(self.ddems) == 0:
- raise ValueError("dDEMs have not yet been calculated")
+ raise ValueError("dDEMs have not yet been calculated.")
dh_values = pd.DataFrame(columns=["dh", "area"], dtype=float)
for _, ddem in enumerate(self.ddems):
@@ -249,7 +251,7 @@ def get_cumulative_series(
# Get the dV series (where all indices are: "year to reference_year")
d_series = self.get_dv_series(mask=mask, outlines_filter=outlines_filter, nans_ok=nans_ok)
else:
- raise ValueError("Invalid argument: '{dh=}'. Choices: ['dh', 'dv']")
+ raise ValueError("Invalid argument: '{dh=}'. Choices: ['dh', 'dv'].")
# Simplify the index to just "year" (implicitly still the same as above)
cumulative_dh = pd.Series(dtype=d_series.dtype)
diff --git a/xdem/examples.py b/xdem/examples.py
index bb4d58d1..153c19a3 100644
--- a/xdem/examples.py
+++ b/xdem/examples.py
@@ -63,7 +63,7 @@ def download_longyearbyen_examples(overwrite: bool = False) -> None:
with open(tar_path, "wb") as outfile:
outfile.write(response.read())
else:
- raise ValueError(f"Longyearbyen data fetch gave non-200 response: {response.status_code}")
+ raise ValueError(f"Longyearbyen data fetch gave non-200 response: {response.status_code}.")
# Extract the tarball
with tarfile.open(tar_path) as tar:
diff --git a/xdem/filters.py b/xdem/filters.py
index cdf2a921..9cc8c994 100644
--- a/xdem/filters.py
+++ b/xdem/filters.py
@@ -29,7 +29,7 @@ def gaussian_filter_scipy(array: NDArrayf, sigma: float) -> NDArrayf:
"""
# Check that array dimension is 2 or 3
if np.ndim(array) not in [2, 3]:
- raise ValueError(f"Invalid array shape given: {array.shape}. Expected 2D or 3D array")
+ raise ValueError(f"Invalid array shape given: {array.shape}. Expected 2D or 3D array.")
# In case array does not contain NaNs, use scipy's gaussian filter directly
if np.count_nonzero(np.isnan(array)) == 0:
@@ -71,7 +71,7 @@ def gaussian_filter_cv(array: NDArrayf, sigma: float) -> NDArrayf:
:returns: the filtered array (same shape as input)
"""
if not _has_cv2:
- raise ValueError("Optional dependency needed. Install 'opencv'")
+ raise ValueError("Optional dependency needed. Install 'opencv'.")
# Check that array dimension is 2, or can be squeezed to 2D
orig_shape = array.shape
@@ -81,9 +81,9 @@ def gaussian_filter_cv(array: NDArrayf, sigma: float) -> NDArrayf:
if orig_shape[0] == 1:
array = array.squeeze()
else:
- raise NotImplementedError("Case of array of dimension 3 not implemented")
+ raise NotImplementedError("Case of array of dimension 3 not implemented.")
else:
- raise ValueError(f"Invalid array shape given: {orig_shape}. Expected 2D or 3D array")
+ raise ValueError(f"Invalid array shape given: {orig_shape}. Expected 2D or 3D array.")
# In case array does not contain NaNs, use OpenCV's gaussian filter directly
# With kernel size (0, 0), i.e. set to default, and borderType=BORDER_REFLECT, the output is equivalent to scipy
diff --git a/xdem/fit.py b/xdem/fit.py
index 6dad6552..c309120c 100644
--- a/xdem/fit.py
+++ b/xdem/fit.py
@@ -4,6 +4,7 @@
from __future__ import annotations
import inspect
+import logging
import warnings
from typing import Any, Callable
@@ -135,7 +136,7 @@ def polynomial_2d(xx: tuple[NDArrayf, NDArrayf], *params: NDArrayf) -> NDArrayf:
#######################################################################
-def _choice_best_order(cost: NDArrayf, margin_improvement: float = 20.0, verbose: bool = False) -> int:
+def _choice_best_order(cost: NDArrayf, margin_improvement: float = 20.0) -> int:
"""
Choice of the best order (polynomial, sum of sinusoids) with a margin of improvement. The best cost value does
not necessarily mean the best predictive fit because high-degree polynomials tend to overfit, and sum of sinusoids
@@ -143,7 +144,6 @@ def _choice_best_order(cost: NDArrayf, margin_improvement: float = 20.0, verbose
:param cost: cost function residuals to the polynomial
:param margin_improvement: improvement margin (percentage) below which the lesser degree polynomial is kept
- :param verbose: if text should be printed
:return: degree: degree for the best-fit polynomial
"""
@@ -159,12 +159,11 @@ def _choice_best_order(cost: NDArrayf, margin_improvement: float = 20.0, verbose
# Choose the good-performance cost with lowest degree
ind = next((i for i, j in enumerate(below_margin) if j))
- if verbose:
- print("Order " + str(ind_min + 1) + " has the minimum cost value of " + str(min_cost))
- print(
- "Order " + str(ind + 1) + " is selected as its cost is within a " + str(margin_improvement) + "% margin of"
- " the minimum cost"
- )
+ logging.debug("Order " + str(ind_min + 1) + " has the minimum cost value of " + str(min_cost))
+ logging.debug(
+ "Order " + str(ind + 1) + " is selected as its cost is within a " + str(margin_improvement) + "% margin of"
+ " the minimum cost"
+ )
return ind
@@ -185,7 +184,6 @@ def _wrapper_scipy_leastsquares(
:param p0: Initial guess.
:param x: X vector.
:param y: Y vector.
- :param verbose: Whether to print out statements.
:return:
"""
@@ -328,7 +326,6 @@ def robust_norder_polynomial_fit(
margin_improvement: float = 20.0,
subsample: float | int = 1,
linear_pkg: str = "scipy",
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
**kwargs: Any,
) -> tuple[NDArrayf, int]:
@@ -349,7 +346,6 @@ def robust_norder_polynomial_fit(
If > 1 will be considered the number of pixels to extract.
:param linear_pkg: package to use for Linear estimator, one of 'scipy' and 'sklearn'.
:param random_state: Random seed.
- :param verbose: Whether to print text.
:returns coefs, degree: Polynomial coefficients and degree for the best-fit polynomial
"""
@@ -361,9 +357,9 @@ def robust_norder_polynomial_fit(
# Raise errors for input string parameters
if not isinstance(estimator_name, str) or estimator_name not in ["Linear", "Theil-Sen", "RANSAC", "Huber"]:
- raise ValueError('Attribute estimator must be one of "Linear", "Theil-Sen", "RANSAC" or "Huber".')
+ raise ValueError('Attribute `estimator` must be one of "Linear", "Theil-Sen", "RANSAC" or "Huber".')
if not isinstance(linear_pkg, str) or linear_pkg not in ["sklearn", "scipy"]:
- raise ValueError('Attribute linear_pkg must be one of "scipy" or "sklearn".')
+ raise ValueError('Attribute `linear_pkg` must be one of "scipy" or "sklearn".')
# Extract xdata from iterable
if len(xdata) == 1:
@@ -404,7 +400,7 @@ def robust_norder_polynomial_fit(
else:
# Otherwise, we use sklearn
if not _has_sklearn:
- raise ValueError("Optional dependency needed. Install 'scikit-learn'")
+ raise ValueError("Optional dependency needed. Install 'scikit-learn'.")
# Define the polynomial model to insert in the pipeline
model = PolynomialFeatures(degree=deg)
@@ -418,7 +414,7 @@ def robust_norder_polynomial_fit(
list_coeffs[deg - 1, 0 : coef.size] = coef
# Choose the best polynomial with a margin of improvement on the cost
- final_index = _choice_best_order(cost=list_costs, margin_improvement=margin_improvement, verbose=verbose)
+ final_index = _choice_best_order(cost=list_costs, margin_improvement=margin_improvement)
# The degree of the best polynomial corresponds to the index plus one
return np.trim_zeros(list_coeffs[final_index], "b"), final_index + 1
@@ -447,7 +443,6 @@ def robust_nfreq_sumsin_fit(
subsample: float | int = 1,
hop_length: float | None = None,
random_state: int | np.random.Generator | None = None,
- verbose: bool = False,
**kwargs: Any,
) -> tuple[NDArrayf, int]:
"""
@@ -468,7 +463,6 @@ def robust_nfreq_sumsin_fit(
:param subsample: If <= 1, will be considered a fraction of valid pixels to extract.
If > 1 will be considered the number of pixels to extract.
:param random_state: Random seed.
- :param verbose: If text should be printed.
:param kwargs: Keyword arguments to pass to scipy.optimize.basinhopping
:returns coefs, degree: sinusoid coefficients (amplitude, frequency, phase) x N, Number N of summed sinusoids
@@ -498,9 +492,13 @@ def wrapper_cost_sumofsin(p: NDArrayf, x: NDArrayf, y: NDArrayf) -> float:
return _cost_sumofsin(x, y, cost_func, *p)
# If no significant resolution is provided, assume that it is the mean difference between sampled X values
+ x_res = np.mean(np.diff(np.sort(xdata)))
+
+ # The hop length will condition jump in function values, needs of magnitude slightly lower than the signal
if hop_length is None:
- x_res = np.mean(np.diff(np.sort(xdata)))
- hop_length = x_res
+ hop = float(np.percentile(ydata, 90) - np.percentile(ydata, 10))
+ else:
+ hop = hop_length
# Loop on all frequencies
costs = np.empty(max_nb_frequency)
@@ -508,8 +506,7 @@ def wrapper_cost_sumofsin(p: NDArrayf, x: NDArrayf, y: NDArrayf) -> float:
for nb_freq in np.arange(1, max_nb_frequency + 1):
- if verbose:
- print(f"Fitting with {nb_freq} frequency")
+ logging.info("Fitting with %d frequency", nb_freq)
b = bounds_amp_wave_phase
# If bounds are not provided, define as the largest possible bounds
@@ -522,7 +519,7 @@ def wrapper_cost_sumofsin(p: NDArrayf, x: NDArrayf, y: NDArrayf) -> float:
ub_phase = 2 * np.pi
# For the wavelength: from the resolution and coordinate extent
# (we don't want the lower bound to be zero, to avoid divisions by zero)
- lb_wavelength = hop_length / 5
+ lb_wavelength = x_res / 5
ub_wavelength = xdata.max() - xdata.min()
b = []
@@ -537,18 +534,17 @@ def wrapper_cost_sumofsin(p: NDArrayf, x: NDArrayf, y: NDArrayf) -> float:
# First guess for the mean parameters
p0 = (np.abs((lb + ub) / 2)).squeeze()
- if verbose:
- print("Bounds")
- print(lb)
- print(ub)
+ logging.debug("Bounds")
+ logging.debug(lb)
+ logging.debug(ub)
# Minimize the globalization with a larger number of points
- minimizer_kwargs = dict(args=(xdata, ydata), method="L-BFGS-B", bounds=scipy_bounds)
+ minimizer_kwargs = dict(args=(xdata, ydata), bounds=scipy_bounds)
myresults = scipy.optimize.basinhopping(
wrapper_cost_sumofsin,
p0,
- disp=verbose,
- T=hop_length * 50,
+ disp=logging.getLogger().getEffectiveLevel() < logging.WARNING,
+ T=hop,
minimizer_kwargs=minimizer_kwargs,
seed=random_state,
**kwargs,
@@ -556,9 +552,8 @@ def wrapper_cost_sumofsin(p: NDArrayf, x: NDArrayf, y: NDArrayf) -> float:
myresults = myresults.lowest_optimization_result
myresults_x = np.array([np.round(myres, 5) for myres in myresults.x])
- if verbose:
- print("Final result")
- print(myresults_x)
+ logging.info("Final result")
+ logging.info(myresults_x)
# Write results for this number of frequency
costs[nb_freq - 1] = wrapper_cost_sumofsin(myresults_x, xdata, ydata)
@@ -567,17 +562,15 @@ def wrapper_cost_sumofsin(p: NDArrayf, x: NDArrayf, y: NDArrayf) -> float:
# Replace NaN cost by infinity
costs[np.isnan(costs)] = np.inf
- if verbose:
- print("Costs")
- print(costs)
+ logging.info("Costs")
+ logging.info(costs)
final_index = _choice_best_order(cost=costs)
final_coefs = amp_freq_phase[final_index][~np.isnan(amp_freq_phase[final_index])]
- if verbose:
- print("Selecting best performing number of frequencies:")
- print(final_coefs)
+ logging.info("Selecting best performing number of frequencies:")
+ logging.info(final_coefs)
# If an amplitude coefficient is almost zero, remove the coefs of that frequency and lower the degree
final_degree = final_index + 1
diff --git a/xdem/misc.py b/xdem/misc.py
index a7f38c0b..5d5c96fc 100644
--- a/xdem/misc.py
+++ b/xdem/misc.py
@@ -51,7 +51,7 @@ def generate_random_field(
rng = np.random.default_rng(random_state)
if not _has_cv2:
- raise ValueError("Optional dependency needed. Install 'opencv'")
+ raise ValueError("Optional dependency needed. Install 'opencv'.")
field = cv2.resize(
cv2.GaussianBlur(
@@ -195,7 +195,7 @@ def diff_environment_yml(
"""
if not _has_yaml:
- raise ValueError("Test dependency needed. Install 'pyyaml'")
+ raise ValueError("Test dependency needed. Install 'pyyaml'.")
if not input_dict:
# Load the yml as dictionaries
diff --git a/xdem/spatialstats.py b/xdem/spatialstats.py
index 5cdbcc86..1898fb62 100644
--- a/xdem/spatialstats.py
+++ b/xdem/spatialstats.py
@@ -3,6 +3,7 @@
import inspect
import itertools
+import logging
import math as m
import multiprocessing as mp
import warnings
@@ -542,7 +543,7 @@ def error_fun(*args: tuple[ArrayLike, ...]) -> NDArrayf:
return zscores, error_fun
-def estimate_model_heteroscedasticity(
+def _estimate_model_heteroscedasticity(
dvalues: NDArrayf,
list_var: list[NDArrayf],
list_var_names: list[str],
@@ -829,7 +830,7 @@ def infer_heteroscedasticity_from_stable(
list_var_stable_arr = list_all_arr[1:]
# Estimate and model the heteroscedasticity using only stable terrain
- df, fun = estimate_model_heteroscedasticity(
+ df, fun = _estimate_model_heteroscedasticity(
dvalues=dvalues_stable_arr,
list_var=list_var_stable_arr,
list_var_names=list_var_names,
@@ -1146,15 +1147,17 @@ def _choose_cdist_equidistant_sampling_parameters(**kwargs: Any) -> tuple[int, i
# And the number of total pairwise comparison
total_pairwise_comparison = runs * subsample_per_disk_per_run**2 * nb_rings
- if kwargs["verbose"]:
- print(
- "Equidistant circular sampling will be performed for {} runs (random center points) with pairwise "
- "comparison between {} samples (points) of the central disk and again {} samples times {} independent "
- "rings centered on the same center point. This results in approximately {} pairwise comparisons (duplicate"
- " pairwise points randomly selected will be removed).".format(
- runs, subsample_per_disk_per_run, subsample_per_disk_per_run, nb_rings, total_pairwise_comparison
- )
- )
+ logging.info(
+ "Equidistant circular sampling will be performed for %d runs (random center points) with pairwise "
+ "comparison between %d samples (points) of the central disk and again %d samples times %d independent "
+ "rings centered on the same center point. This results in approximately %d pairwise comparisons (duplicate "
+ "pairwise points randomly selected will be removed).",
+ runs,
+ subsample_per_disk_per_run,
+ subsample_per_disk_per_run,
+ nb_rings,
+ total_pairwise_comparison,
+ )
return runs, subsample_per_disk_per_run, ratio_subsample
@@ -1243,8 +1246,7 @@ def _wrapper_get_empirical_variogram(argdict: dict[str, Any]) -> pd.DataFrame:
:return: Empirical variogram (variance, upper bound of lag bin, counts)
"""
- if argdict["verbose"]:
- print("Working on run " + str(argdict["i"]) + " out of " + str(argdict["imax"]))
+ logging.debug("Working on run " + str(argdict["i"]) + " out of " + str(argdict["imax"]))
argdict.pop("i")
argdict.pop("imax")
@@ -1275,7 +1277,6 @@ def sample_empirical_variogram(
subsample_method: str = "cdist_equidistant",
n_variograms: int = 1,
n_jobs: int = 1,
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
# **kwargs: **EmpiricalVariogramKArgs, # This will work in Python 3.12, fails in the meantime, we'll be able to
# remove some type ignores from this function in the future
@@ -1331,7 +1332,6 @@ def sample_empirical_variogram(
:param subsample_method: Spatial subsampling method
:param n_variograms: Number of independent empirical variogram estimations (to estimate empirical variogram spread)
:param n_jobs: Number of processing cores
- :param verbose: Print statements during processing
:param random_state: Random state or seed number to use for calculations (to fix random sampling during testing)
:return: Empirical variogram (variance, upper bound of lag bin, counts)
@@ -1428,7 +1428,6 @@ def sample_empirical_variogram(
"coords": coords,
"subsample_method": subsample_method,
"subsample": subsample,
- "verbose": verbose,
}
if subsample_method in ["cdist_equidistant", "pdist_ring", "pdist_disk", "pdist_point"]:
# The shape is needed for those three methods
@@ -1456,8 +1455,7 @@ def sample_empirical_variogram(
# Differentiate between 1 core and several cores for multiple runs
# All variogram runs have random sampling inherent to their subfunctions, so we provide the same input arguments
if n_jobs == 1:
- if verbose:
- print("Using 1 core...")
+ logging.info("Using 1 core...")
list_df_run = []
for i in range(n_variograms):
@@ -1473,8 +1471,7 @@ def sample_empirical_variogram(
list_df_run.append(df_run)
else:
- if verbose:
- print("Using " + str(n_jobs) + " cores...")
+ logging.info("Using " + str(n_jobs) + " cores...")
pool = mp.Pool(n_jobs, maxtasksperchild=1)
list_argdict = [
@@ -1775,7 +1772,7 @@ def variogram_sum(h: float, *args: list[float]) -> float:
return variogram_sum_fit, df_params
-def estimate_model_spatial_correlation(
+def _estimate_model_spatial_correlation(
dvalues: NDArrayf | RasterType,
list_models: list[str | Callable[[NDArrayf, float, float], NDArrayf]],
estimator: str = "dowd",
@@ -1785,7 +1782,6 @@ def estimate_model_spatial_correlation(
subsample_method: str = "cdist_equidistant",
n_variograms: int = 1,
n_jobs: int = 1,
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
bounds: list[tuple[float, float]] = None,
p0: list[float] = None,
@@ -1815,7 +1811,6 @@ def estimate_model_spatial_correlation(
:param subsample_method: Spatial subsampling method
:param n_variograms: Number of independent empirical variogram estimations (to estimate empirical variogram spread)
:param n_jobs: Number of processing cores
- :param verbose: Print statements during processing
:param random_state: Random state or seed number to use for calculations (to fix random sampling during testing)
:param bounds: Bounds of range and sill parameters for each model (shape K x 4 = K x range lower, range upper,
sill lower, sill upper).
@@ -1834,7 +1829,6 @@ def estimate_model_spatial_correlation(
subsample_method=subsample_method,
n_variograms=n_variograms,
n_jobs=n_jobs,
- verbose=verbose,
random_state=random_state,
**kwargs,
)
@@ -1861,7 +1855,6 @@ def infer_spatial_correlation_from_stable(
subsample_method: str = "cdist_equidistant",
n_variograms: int = 1,
n_jobs: int = 1,
- verbose: bool = False,
bounds: list[tuple[float, float]] = None,
p0: list[float] = None,
random_state: int | np.random.Generator | None = None,
@@ -1897,7 +1890,6 @@ def infer_spatial_correlation_from_stable(
:param subsample_method: Spatial subsampling method
:param n_variograms: Number of independent empirical variogram estimations (to estimate empirical variogram spread)
:param n_jobs: Number of processing cores
- :param verbose: Print statements during processing
:param bounds: Bounds of range and sill parameters for each model (shape K x 4 = K x range lower, range upper,
sill lower, sill upper).
:param p0: Initial guess of ranges and sills each model (shape K x 2 = K x range first guess, sill first guess).
@@ -1922,7 +1914,7 @@ def infer_spatial_correlation_from_stable(
dvalues_stable_arr /= errors_arr
# Estimate and model spatial correlations
- empirical_variogram, params_variogram_model, spatial_correlation_func = estimate_model_spatial_correlation(
+ empirical_variogram, params_variogram_model, spatial_correlation_func = _estimate_model_spatial_correlation(
dvalues=dvalues_stable_arr,
list_models=list_models,
estimator=estimator,
@@ -1932,7 +1924,6 @@ def infer_spatial_correlation_from_stable(
subsample_method=subsample_method,
n_variograms=n_variograms,
n_jobs=n_jobs,
- verbose=verbose,
random_state=random_state,
bounds=bounds,
p0=p0,
@@ -2646,7 +2637,6 @@ def _patches_convolution(
patch_shape: str = "circular",
method: str = "scipy",
statistic_between_patches: Callable[[NDArrayf], np.floating[Any]] = nmad,
- verbose: bool = False,
return_in_patch_statistics: bool = False,
) -> tuple[float, float, float] | tuple[float, float, float, pd.DataFrame]:
"""
@@ -2659,7 +2649,6 @@ def _patches_convolution(
:param method: Method to perform the convolution, "scipy" or "numba"
:param statistic_between_patches: Statistic to compute between all patches, typically a measure of spread, applied
to the first in-patch statistic, which is typically the mean
- :param verbose: Print statement to console
:param return_in_patch_statistics: Whether to return the dataframe of statistics for all patches and areas
@@ -2678,8 +2667,7 @@ def _patches_convolution(
else:
raise ValueError('Kernel shape should be "square" or "circular".')
- if verbose:
- print("Computing the convolution on the entire array...")
+ logging.info("Computing the convolution on the entire array...")
mean_img, nb_valid_img, nb_pixel_per_kernel = mean_filter_nan(
img=values, kernel_size=kernel_size, kernel_shape=patch_shape, method=method
)
@@ -2692,8 +2680,7 @@ def _patches_convolution(
# kernel size (i.e., the diameter of the circular patch, or side of the square patch) to ensure no dependency
# There are as many combinations for this calculation as the square of the kernel_size
- if verbose:
- print("Computing statistic between patches for all independent combinations...")
+ logging.info("Computing statistic between patches for all independent combinations...")
list_statistic_estimates = []
list_nb_independent_patches = []
for i in range(kernel_size):
@@ -2733,7 +2720,6 @@ def _patches_loop_quadrants(
perc_min_valid: float = 80.0,
statistics_in_patch: Iterable[Callable[[NDArrayf], np.floating[Any]] | str] = (np.nanmean,),
statistic_between_patches: Callable[[NDArrayf], np.floating[Any]] = nmad,
- verbose: bool = False,
random_state: int | np.random.Generator | None = None,
return_in_patch_statistics: bool = False,
) -> tuple[float, float, float] | tuple[float, float, float, pd.DataFrame]:
@@ -2751,7 +2737,6 @@ def _patches_loop_quadrants(
to the first in-patch statistic, which is typically the mean
:param patch_shape: Shape of patch, either "circular" or "square".
:param n_patches: Maximum number of patches to sample
- :param verbose: Print statement to console
:param random_state: Random state or seed number to use for calculations (to fix random sampling during testing)
:param return_in_patch_statistics: Whether to return the dataframe of statistics for all patches and areas
@@ -2801,8 +2786,7 @@ def _patches_loop_quadrants(
for idx_quadrant in list_idx_quadrant:
- if verbose:
- print("Working on a new quadrant")
+ logging.info("Working on a new quadrant")
# Select center coordinates
i = list_quadrant[idx_quadrant][0]
@@ -2828,8 +2812,7 @@ def _patches_loop_quadrants(
u = u + 1
if u > n_patches:
break
- if verbose:
- print("Found valid quadrant " + str(u) + " (maximum: " + str(n_patches) + ")")
+ logging.info("Found valid quadrant " + str(u) + " (maximum: " + str(n_patches) + ")")
df = pd.DataFrame()
df = df.assign(tile=[str(i) + "_" + str(j)])
@@ -2882,7 +2865,6 @@ def patches_method(
vectorized: bool = True,
convolution_method: str = "scipy",
n_patches: int = 1000,
- verbose: bool = False,
*,
return_in_patch_statistics: Literal[False] = False,
random_state: int | np.random.Generator | None = None,
@@ -2904,7 +2886,6 @@ def patches_method(
vectorized: bool = True,
convolution_method: str = "scipy",
n_patches: int = 1000,
- verbose: bool = False,
*,
return_in_patch_statistics: Literal[True],
random_state: int | np.random.Generator | None = None,
@@ -2925,7 +2906,6 @@ def patches_method(
vectorized: bool = True,
convolution_method: str = "scipy",
n_patches: int = 1000,
- verbose: bool = False,
return_in_patch_statistics: bool = False,
random_state: int | np.random.Generator | None = None,
) -> pd.DataFrame | tuple[pd.DataFrame, pd.DataFrame]:
@@ -2964,7 +2944,6 @@ def patches_method(
:param vectorized: Whether to use the vectorized (convolution) method or the for loop in quadrants
:param convolution_method: Convolution method to use, either "scipy" or "numba" (only for vectorized)
:param n_patches: Maximum number of patches to sample (only for non-vectorized)
- :param verbose: Print statement to console
:param return_in_patch_statistics: Whether to return the dataframe of statistics for all patches and areas
:param random_state: Random state or seed number to use for calculations (only for non-vectorized, for testing)
@@ -2998,7 +2977,6 @@ def patches_method(
patch_shape=patch_shape,
method=convolution_method,
statistic_between_patches=statistic_between_patches,
- verbose=verbose,
return_in_patch_statistics=return_in_patch_statistics,
)
@@ -3013,7 +2991,6 @@ def patches_method(
perc_min_valid=perc_min_valid,
statistics_in_patch=statistics_in_patch,
statistic_between_patches=statistic_between_patches,
- verbose=verbose,
return_in_patch_statistics=return_in_patch_statistics,
random_state=random_state,
)
diff --git a/xdem/terrain.py b/xdem/terrain.py
index 836bc6ee..57f5057c 100644
--- a/xdem/terrain.py
+++ b/xdem/terrain.py
@@ -11,13 +11,6 @@
from xdem._typing import MArrayf, NDArrayf
-try:
- import richdem as rd
-
- _has_rd = True
-except ImportError:
- _has_rd = False
-
available_attributes = [
"slope",
"aspect",
@@ -34,34 +27,6 @@
]
-def _raster_to_rda(rst: RasterType) -> rd.rdarray:
- """
- Get georeferenced richDEM array from geoutils.Raster
- :param rst: DEM as raster
- :return: DEM
- """
- arr = rst.data.filled(rst.nodata).squeeze()
- rda = rd.rdarray(arr, no_data=rst.nodata)
- rda.geotransform = rst.transform.to_gdal()
-
- return rda
-
-
-def _get_terrainattr_richdem(rst: RasterType, attribute: str = "slope_radians") -> NDArrayf:
- """
- Derive terrain attribute for DEM opened with rasterio. One of "slope_degrees", "slope_percentage", "aspect",
- "profile_curvature", "planform_curvature", "curvature" and others (see RichDEM documentation).
- :param rst: DEM as raster
- :param attribute: RichDEM terrain attribute
- :return:
- """
- rda = _raster_to_rda(rst)
- terrattr = rd.TerrainAttribute(rda, attrib=attribute)
- terrattr[terrattr == terrattr.no_data] = np.nan
-
- return np.array(terrattr)
-
-
@numba.njit(parallel=True) # type: ignore
def _get_quadric_coefficients(
dem: NDArrayf,
@@ -321,15 +286,15 @@ def get_quadric_coefficients(
if len(dem_arr.shape) != 2:
raise ValueError(
f"Invalid input array shape: {dem.shape}, parsed into {dem_arr.shape}. "
- "Expected 2D array or 3D array of shape (1, row, col)"
+ "Expected 2D array or 3D array of shape (1, row, col)."
)
if any(dim < 3 for dim in dem_arr.shape):
- raise ValueError(f"DEM (shape: {dem.shape}) is too small. Smallest supported shape is (3, 3)")
+ raise ValueError(f"DEM (shape: {dem.shape}) is too small. Smallest supported shape is (3, 3).")
# Resolution is in other tools accepted as a tuple. Here, it must be just one number, so it's best to sanity check.
if isinstance(resolution, Sized):
- raise ValueError("Resolution must be the same for X and Y directions")
+ raise ValueError("Resolution must be the same for X and Y directions.")
allowed_fill_methods = ["median", "mean", "none"]
allowed_edge_methods = ["nearest", "wrap", "none"]
@@ -337,7 +302,7 @@ def get_quadric_coefficients(
[fill_method, edge_method], ["fill", "edge"], (allowed_fill_methods, allowed_edge_methods)
):
if value.lower() not in allowed:
- raise ValueError(f"Invalid {name} method: '{value}'. Choices: {allowed}")
+ raise ValueError(f"Invalid {name} method: '{value}'. Choices: {allowed}.")
# Try to run the numba JIT code. It should never fail at this point, so if it does, it should be reported!
try:
@@ -540,7 +505,7 @@ def get_windowed_indexes(
http://download.osgeo.org/qgis/doc/reference-docs/Terrain_Ruggedness_Index.pdf, for topography and from Wilson
et al. (2007), http://dx.doi.org/10.1080/01490410701295962, for bathymetry.
- Topographic Position Index from Weiss (2001), http://www.jennessent.com/downloads/TPI-poster-TNC_18x22.pdf.
- - Roughness from Dartnell (2000), http://dx.doi.org/10.14358/PERS.70.9.1081.
+ - Roughness from Dartnell (2000), thesis referenced in Wilson et al. (2007) above.
- Fractal roughness from Taud et Parrot (2005), https://doi.org/10.4000/geomorphologie.622.
Nearly all are also referenced in Wilson et al. (2007).
@@ -638,7 +603,6 @@ def get_terrain_attribute(
tri_method: str = "Riley",
fill_method: str = "none",
edge_method: str = "none",
- use_richdem: bool = False,
window_size: int = 3,
) -> NDArrayf:
...
@@ -657,7 +621,6 @@ def get_terrain_attribute(
tri_method: str = "Riley",
fill_method: str = "none",
edge_method: str = "none",
- use_richdem: bool = False,
window_size: int = 3,
) -> list[NDArrayf]:
...
@@ -676,7 +639,6 @@ def get_terrain_attribute(
tri_method: str = "Riley",
fill_method: str = "none",
edge_method: str = "none",
- use_richdem: bool = False,
window_size: int = 3,
) -> list[RasterType]:
...
@@ -695,7 +657,6 @@ def get_terrain_attribute(
tri_method: str = "Riley",
fill_method: str = "none",
edge_method: str = "none",
- use_richdem: bool = False,
window_size: int = 3,
) -> RasterType:
...
@@ -713,7 +674,6 @@ def get_terrain_attribute(
tri_method: str = "Riley",
fill_method: str = "none",
edge_method: str = "none",
- use_richdem: bool = False,
window_size: int = 3,
) -> NDArrayf | list[NDArrayf] | RasterType | list[RasterType]:
"""
@@ -727,13 +687,12 @@ def get_terrain_attribute(
- Terrain Ruggedness Index (topography) from Riley et al. (1999),
http://download.osgeo.org/qgis/doc/reference-docs/Terrain_Ruggedness_Index.pdf.
- Terrain Ruggedness Index (bathymetry) from Wilson et al. (2007), http://dx.doi.org/10.1080/01490410701295962.
- - Roughness from Dartnell (2000), http://dx.doi.org/10.14358/PERS.70.9.1081.
+ - Roughness from Dartnell (2000), thesis referenced in Wilson et al. (2007) above.
- Rugosity from Jenness (2004), https://doi.org/10.2193/0091-7648(2004)032[0829:CLSAFD]2.0.CO;2.
- Fractal roughness from Taud et Parrot (2005), https://doi.org/10.4000/geomorphologie.622.
Aspect and hillshade are derived using the slope, and thus depend on the same method.
More details on the equations in the functions get_quadric_coefficients() and get_windowed_indexes().
- The slope, aspect ("Horn" method), and all curvatures ("ZevenbergThorne" method) can also be derived using RichDEM.
Attributes:
@@ -765,7 +724,6 @@ def get_terrain_attribute(
:param tri_method: Method to calculate the Terrain Ruggedness Index: "Riley" (topography) or "Wilson" (bathymetry).
:param fill_method: See the 'get_quadric_coefficients()' docstring for information.
:param edge_method: See the 'get_quadric_coefficients()' docstring for information.
- :param use_richdem: Whether to use richDEM for slope, aspect and curvature calculations.
:param window_size: The window size for windowed ruggedness and roughness indexes.
:raises ValueError: If the inputs are poorly formatted or are invalid.
@@ -810,33 +768,6 @@ def get_terrain_attribute(
]
attributes_requiring_surface_fit = [attr for attr in attribute if attr in list_requiring_surface_fit]
- if use_richdem:
-
- if not _has_rd:
- raise ValueError("Optional dependency needed. Install 'richdem'")
-
- if ("slope" in attribute or "aspect" in attribute) and slope_method == "ZevenbergThorne":
- raise ValueError("RichDEM can only compute the slope and aspect using the default method of Horn (1981)")
-
- list_requiring_richdem = [
- "slope",
- "aspect",
- "hillshade",
- "curvature",
- "planform_curvature",
- "profile curvature",
- "maximum_curvature",
- ]
- attributes_using_richdem = [attr for attr in attribute if attr in list_requiring_richdem]
- for attr in attributes_using_richdem:
- attributes_requiring_surface_fit.remove(attr)
-
- if not isinstance(dem, gu.Raster):
- # Here, maybe we could pass the geotransform based on the resolution, and add a "default" projection as
- # this is mandated but likely not used by the rdarray format of RichDEM...
- # For now, not supported
- raise ValueError("To derive RichDEM attributes, the DEM passed must be a Raster object")
-
list_requiring_windowed_index = [
"terrain_ruggedness_index",
"topographic_position_index",
@@ -909,59 +840,44 @@ def get_terrain_attribute(
if make_slope:
- if use_richdem:
- terrain_attributes["slope"] = _get_terrainattr_richdem(dem, attribute="slope_radians")
-
- else:
- if slope_method == "Horn":
- # This calculation is based on page 18 (bottom left) and 20-21 of Horn (1981),
- # http://dx.doi.org/10.1109/PROC.1981.11918.
- terrain_attributes["slope"] = np.arctan(
- (terrain_attributes["surface_fit"][9, :, :] ** 2 + terrain_attributes["surface_fit"][10, :, :] ** 2)
- ** 0.5
- )
+ if slope_method == "Horn":
+ # This calculation is based on page 18 (bottom left) and 20-21 of Horn (1981),
+ # http://dx.doi.org/10.1109/PROC.1981.11918.
+ terrain_attributes["slope"] = np.arctan(
+ (terrain_attributes["surface_fit"][9, :, :] ** 2 + terrain_attributes["surface_fit"][10, :, :] ** 2)
+ ** 0.5
+ )
- elif slope_method == "ZevenbergThorne":
- # This calculation is based on Equation 13 of Zevenbergen and Thorne (1987),
- # http://dx.doi.org/10.1002/esp.3290120107.
- # SLOPE = ARCTAN((G²+H²)**(1/2))
- terrain_attributes["slope"] = np.arctan(
- (terrain_attributes["surface_fit"][6, :, :] ** 2 + terrain_attributes["surface_fit"][7, :, :] ** 2)
- ** 0.5
- )
+ elif slope_method == "ZevenbergThorne":
+ # This calculation is based on Equation 13 of Zevenbergen and Thorne (1987),
+ # http://dx.doi.org/10.1002/esp.3290120107.
+ # SLOPE = ARCTAN((G²+H²)**(1/2))
+ terrain_attributes["slope"] = np.arctan(
+ (terrain_attributes["surface_fit"][6, :, :] ** 2 + terrain_attributes["surface_fit"][7, :, :] ** 2)
+ ** 0.5
+ )
if make_aspect:
- if use_richdem:
- # The aspect of RichDEM is returned in degrees, we convert to radians to match the others
- terrain_attributes["aspect"] = np.deg2rad(_get_terrainattr_richdem(dem, attribute="aspect"))
- # For flat slopes, RichDEM returns a 90° aspect by default, while GDAL return a 180° aspect
- # We stay consistent with GDAL
- slope_tmp = _get_terrainattr_richdem(dem, attribute="slope_radians")
- terrain_attributes["aspect"][slope_tmp == 0] = np.pi
+ # ASPECT = ARCTAN(-H/-G) # This did not work
+ # ASPECT = (ARCTAN2(-G, H) + 0.5PI) % 2PI did work.
+ with warnings.catch_warnings():
+ warnings.filterwarnings("ignore", "invalid value encountered in remainder")
+ if slope_method == "Horn":
+ # This uses the estimates from Horn (1981).
+ terrain_attributes["aspect"] = (
+ -np.arctan2(
+ -terrain_attributes["surface_fit"][9, :, :], terrain_attributes["surface_fit"][10, :, :]
+ )
+ - np.pi
+ ) % (2 * np.pi)
- else:
- # ASPECT = ARCTAN(-H/-G) # This did not work
- # ASPECT = (ARCTAN2(-G, H) + 0.5PI) % 2PI did work.
- with warnings.catch_warnings():
- warnings.filterwarnings("ignore", "invalid value encountered in remainder")
- if slope_method == "Horn":
- # This uses the estimates from Horn (1981).
- terrain_attributes["aspect"] = (
- -np.arctan2(
- -terrain_attributes["surface_fit"][9, :, :], terrain_attributes["surface_fit"][10, :, :]
- )
- - np.pi
- ) % (2 * np.pi)
-
- elif slope_method == "ZevenbergThorne":
- # This uses the slope estimate from Zevenbergen and Thorne (1987).
- terrain_attributes["aspect"] = (
- np.arctan2(
- -terrain_attributes["surface_fit"][6, :, :], terrain_attributes["surface_fit"][7, :, :]
- )
- + np.pi / 2
- ) % (2 * np.pi)
+ elif slope_method == "ZevenbergThorne":
+ # This uses the slope estimate from Zevenbergen and Thorne (1987).
+ terrain_attributes["aspect"] = (
+ np.arctan2(-terrain_attributes["surface_fit"][6, :, :], terrain_attributes["surface_fit"][7, :, :])
+ + np.pi / 2
+ ) % (2 * np.pi)
if make_hillshade:
# If a different z-factor was given, slopemap with exaggerated gradients.
@@ -987,77 +903,56 @@ def get_terrain_attribute(
).astype("float32")
if make_curvature:
-
- if use_richdem:
- terrain_attributes["curvature"] = _get_terrainattr_richdem(dem, attribute="curvature")
-
- else:
- # Curvature is the second derivative of the surface fit equation.
- # (URL in get_quadric_coefficients() docstring)
- # Curvature = -2(D + E) * 100
- terrain_attributes["curvature"] = (
- -2 * (terrain_attributes["surface_fit"][3, :, :] + terrain_attributes["surface_fit"][4, :, :]) * 100
- )
+ # Curvature is the second derivative of the surface fit equation.
+ # (URL in get_quadric_coefficients() docstring)
+ # Curvature = -2(D + E) * 100
+ terrain_attributes["curvature"] = (
+ -2 * (terrain_attributes["surface_fit"][3, :, :] + terrain_attributes["surface_fit"][4, :, :]) * 100
+ )
if make_planform_curvature:
-
- if use_richdem:
- terrain_attributes["planform_curvature"] = _get_terrainattr_richdem(dem, attribute="planform_curvature")
-
- else:
- # PLANC = 2(DH² + EG² -FGH)/(G²+H²)
- with warnings.catch_warnings():
- warnings.filterwarnings("ignore", "invalid value encountered in *divide")
- terrain_attributes["planform_curvature"] = (
- -2
- * (
- terrain_attributes["surface_fit"][3, :, :] * terrain_attributes["surface_fit"][7, :, :] ** 2
- + terrain_attributes["surface_fit"][4, :, :] * terrain_attributes["surface_fit"][6, :, :] ** 2
- - terrain_attributes["surface_fit"][5, :, :]
- * terrain_attributes["surface_fit"][6, :, :]
- * terrain_attributes["surface_fit"][7, :, :]
- )
- / (
- terrain_attributes["surface_fit"][6, :, :] ** 2
- + terrain_attributes["surface_fit"][7, :, :] ** 2
- )
- * 100
+ # PLANC = 2(DH² + EG² -FGH)/(G²+H²)
+ with warnings.catch_warnings():
+ warnings.filterwarnings("ignore", "invalid value encountered in *divide")
+ terrain_attributes["planform_curvature"] = (
+ -2
+ * (
+ terrain_attributes["surface_fit"][3, :, :] * terrain_attributes["surface_fit"][7, :, :] ** 2
+ + terrain_attributes["surface_fit"][4, :, :] * terrain_attributes["surface_fit"][6, :, :] ** 2
+ - terrain_attributes["surface_fit"][5, :, :]
+ * terrain_attributes["surface_fit"][6, :, :]
+ * terrain_attributes["surface_fit"][7, :, :]
)
+ / (terrain_attributes["surface_fit"][6, :, :] ** 2 + terrain_attributes["surface_fit"][7, :, :] ** 2)
+ * 100
+ )
- # Completely flat surfaces trigger the warning above. These need to be set to zero
- terrain_attributes["planform_curvature"][
- terrain_attributes["surface_fit"][6, :, :] ** 2 + terrain_attributes["surface_fit"][7, :, :] ** 2 == 0.0
- ] = 0.0
+ # Completely flat surfaces trigger the warning above. These need to be set to zero
+ terrain_attributes["planform_curvature"][
+ terrain_attributes["surface_fit"][6, :, :] ** 2 + terrain_attributes["surface_fit"][7, :, :] ** 2 == 0.0
+ ] = 0.0
if make_profile_curvature:
-
- if use_richdem:
- terrain_attributes["profile_curvature"] = _get_terrainattr_richdem(dem, attribute="profile_curvature")
-
- else:
- # PROFC = -2(DG² + EH² + FGH)/(G²+H²)
- with warnings.catch_warnings():
- warnings.filterwarnings("ignore", "invalid value encountered in *divide")
- terrain_attributes["profile_curvature"] = (
- 2
- * (
- terrain_attributes["surface_fit"][3, :, :] * terrain_attributes["surface_fit"][6, :, :] ** 2
- + terrain_attributes["surface_fit"][4, :, :] * terrain_attributes["surface_fit"][7, :, :] ** 2
- + terrain_attributes["surface_fit"][5, :, :]
- * terrain_attributes["surface_fit"][6, :, :]
- * terrain_attributes["surface_fit"][7, :, :]
- )
- / (
- terrain_attributes["surface_fit"][6, :, :] ** 2
- + terrain_attributes["surface_fit"][7, :, :] ** 2
- )
- * 100
+ # PROFC = -2(DG² + EH² + FGH)/(G²+H²)
+ with warnings.catch_warnings():
+ warnings.filterwarnings("ignore", "invalid value encountered in *divide")
+ terrain_attributes["profile_curvature"] = (
+ 2
+ * (
+ terrain_attributes["surface_fit"][3, :, :] * terrain_attributes["surface_fit"][6, :, :] ** 2
+ + terrain_attributes["surface_fit"][4, :, :] * terrain_attributes["surface_fit"][7, :, :] ** 2
+ + terrain_attributes["surface_fit"][5, :, :]
+ * terrain_attributes["surface_fit"][6, :, :]
+ * terrain_attributes["surface_fit"][7, :, :]
)
+ / (terrain_attributes["surface_fit"][6, :, :] ** 2 + terrain_attributes["surface_fit"][7, :, :] ** 2)
+ * 100
+ )
- # Completely flat surfaces trigger the warning above. These need to be set to zero
- terrain_attributes["profile_curvature"][
- terrain_attributes["surface_fit"][6, :, :] ** 2 + terrain_attributes["surface_fit"][7, :, :] ** 2 == 0.0
- ] = 0.0
+ # Completely flat surfaces trigger the warning above. These need to be set to zero
+ terrain_attributes["profile_curvature"][
+ terrain_attributes["surface_fit"][6, :, :] ** 2 + terrain_attributes["surface_fit"][7, :, :] ** 2 == 0.0
+ ] = 0.0
if make_maximum_curvature:
minc = np.minimum(terrain_attributes["profile_curvature"], terrain_attributes["planform_curvature"])
@@ -1117,7 +1012,6 @@ def slope(
method: str = "Horn",
degrees: bool = True,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> NDArrayf:
...
@@ -1128,7 +1022,6 @@ def slope(
method: str = "Horn",
degrees: bool = True,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> Raster:
...
@@ -1138,7 +1031,6 @@ def slope(
method: str = "Horn",
degrees: bool = True,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> NDArrayf | Raster:
"""
Generate a slope map for a DEM, returned in degrees by default.
@@ -1150,7 +1042,6 @@ def slope(
:param method: Method to calculate slope: "Horn" or "ZevenbergThorne".
:param degrees: Whether to use degrees or radians (False means radians).
:param resolution: The X/Y resolution of the DEM, only if passed as an array.
- :param use_richdem: Whether to use RichDEM to compute the attribute.
:examples:
>>> dem = np.repeat(np.arange(3), 3).reshape(3, 3)
@@ -1165,9 +1056,7 @@ def slope(
:returns: A slope map of the same shape as 'dem' in degrees or radians.
"""
- return get_terrain_attribute(
- dem, attribute="slope", slope_method=method, resolution=resolution, degrees=degrees, use_richdem=use_richdem
- )
+ return get_terrain_attribute(dem, attribute="slope", slope_method=method, resolution=resolution, degrees=degrees)
@overload
@@ -1175,7 +1064,6 @@ def aspect(
dem: NDArrayf | MArrayf,
method: str = "Horn",
degrees: bool = True,
- use_richdem: bool = False,
) -> NDArrayf:
...
@@ -1185,7 +1073,6 @@ def aspect(
dem: RasterType,
method: str = "Horn",
degrees: bool = True,
- use_richdem: bool = False,
) -> RasterType:
...
@@ -1194,7 +1081,6 @@ def aspect(
dem: NDArrayf | MArrayf | RasterType,
method: str = "Horn",
degrees: bool = True,
- use_richdem: bool = False,
) -> NDArrayf | Raster:
"""
Calculate the aspect of each cell in a DEM, returned in degrees by default. The aspect of flat slopes is 180° by
@@ -1210,7 +1096,6 @@ def aspect(
:param dem: The DEM to calculate the aspect from.
:param method: Method to calculate aspect: "Horn" or "ZevenbergThorne".
:param degrees: Whether to use degrees or radians (False means radians).
- :param use_richdem: Whether to use RichDEM to compute the attribute.
:examples:
>>> dem = np.repeat(np.arange(3), 3).reshape(3, 3)
@@ -1228,9 +1113,7 @@ def aspect(
270.0
"""
- return get_terrain_attribute(
- dem, attribute="aspect", slope_method=method, resolution=1.0, degrees=degrees, use_richdem=use_richdem
- )
+ return get_terrain_attribute(dem, attribute="aspect", slope_method=method, resolution=1.0, degrees=degrees)
@overload
@@ -1241,7 +1124,6 @@ def hillshade(
altitude: float = 45.0,
z_factor: float = 1.0,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> NDArrayf:
...
@@ -1254,7 +1136,6 @@ def hillshade(
altitude: float = 45.0,
z_factor: float = 1.0,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> RasterType:
...
@@ -1266,7 +1147,6 @@ def hillshade(
altitude: float = 45.0,
z_factor: float = 1.0,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> NDArrayf | RasterType:
"""
Generate a hillshade from the given DEM. The value 0 is used for nodata, and 1 to 255 for hillshading.
@@ -1279,7 +1159,6 @@ def hillshade(
:param altitude: The shading altitude in degrees (0-90°). 90° is straight from above.
:param z_factor: Vertical exaggeration factor.
:param resolution: The X/Y resolution of the DEM, only if passed as an array.
- :param use_richdem: Whether to use RichDEM to compute the slope and aspect used for the hillshade.
:raises AssertionError: If the given DEM is not a 2D array.
@@ -1295,7 +1174,6 @@ def hillshade(
hillshade_azimuth=azimuth,
hillshade_altitude=altitude,
hillshade_z_factor=z_factor,
- use_richdem=use_richdem,
)
@@ -1303,7 +1181,6 @@ def hillshade(
def curvature(
dem: NDArrayf | MArrayf,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> NDArrayf:
...
@@ -1312,7 +1189,6 @@ def curvature(
def curvature(
dem: RasterType,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> RasterType:
...
@@ -1320,7 +1196,6 @@ def curvature(
def curvature(
dem: NDArrayf | MArrayf | RasterType,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> NDArrayf | RasterType:
"""
Calculate the terrain curvature (second derivative of elevation) in m-1 multiplied by 100.
@@ -1337,7 +1212,6 @@ def curvature(
:param dem: The DEM to calculate the curvature from.
:param resolution: The X/Y resolution of the DEM, only if passed as an array.
- :param use_richdem: Whether to use RichDEM to compute the attribute.
:raises ValueError: If the inputs are poorly formatted.
@@ -1350,15 +1224,11 @@ def curvature(
:returns: The curvature array of the DEM.
"""
- return get_terrain_attribute(dem=dem, attribute="curvature", resolution=resolution, use_richdem=use_richdem)
+ return get_terrain_attribute(dem=dem, attribute="curvature", resolution=resolution)
@overload
-def planform_curvature(
- dem: NDArrayf | MArrayf,
- resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
-) -> NDArrayf:
+def planform_curvature(dem: NDArrayf | MArrayf, resolution: float | tuple[float, float] | None = None) -> NDArrayf:
...
@@ -1366,7 +1236,6 @@ def planform_curvature(
def planform_curvature(
dem: RasterType,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> RasterType:
...
@@ -1374,7 +1243,6 @@ def planform_curvature(
def planform_curvature(
dem: NDArrayf | MArrayf | RasterType,
resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
) -> NDArrayf | RasterType:
"""
Calculate the terrain curvature perpendicular to the direction of the slope in m-1 multiplied by 100.
@@ -1383,7 +1251,6 @@ def planform_curvature(
:param dem: The DEM to calculate the curvature from.
:param resolution: The X/Y resolution of the DEM, only if passed as an array.
- :param use_richdem: Whether to use RichDEM to compute the attribute.
:raises ValueError: If the inputs are poorly formatted.
@@ -1401,33 +1268,21 @@ def planform_curvature(
:returns: The planform curvature array of the DEM.
"""
- return get_terrain_attribute(
- dem=dem, attribute="planform_curvature", resolution=resolution, use_richdem=use_richdem
- )
+ return get_terrain_attribute(dem=dem, attribute="planform_curvature", resolution=resolution)
@overload
-def profile_curvature(
- dem: NDArrayf | MArrayf,
- resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
-) -> NDArrayf:
+def profile_curvature(dem: NDArrayf | MArrayf, resolution: float | tuple[float, float] | None = None) -> NDArrayf:
...
@overload
-def profile_curvature(
- dem: RasterType,
- resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
-) -> RasterType:
+def profile_curvature(dem: RasterType, resolution: float | tuple[float, float] | None = None) -> RasterType:
...
def profile_curvature(
- dem: NDArrayf | MArrayf | RasterType,
- resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
+ dem: NDArrayf | MArrayf | RasterType, resolution: float | tuple[float, float] | None = None
) -> NDArrayf | RasterType:
"""
Calculate the terrain curvature parallel to the direction of the slope in m-1 multiplied by 100.
@@ -1436,7 +1291,6 @@ def profile_curvature(
:param dem: The DEM to calculate the curvature from.
:param resolution: The X/Y resolution of the DEM, only if passed as an array.
- :param use_richdem: Whether to use RichDEM to compute the attribute.
:raises ValueError: If the inputs are poorly formatted.
@@ -1454,31 +1308,21 @@ def profile_curvature(
:returns: The profile curvature array of the DEM.
"""
- return get_terrain_attribute(dem=dem, attribute="profile_curvature", resolution=resolution, use_richdem=use_richdem)
+ return get_terrain_attribute(dem=dem, attribute="profile_curvature", resolution=resolution)
@overload
-def maximum_curvature(
- dem: NDArrayf | MArrayf,
- resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
-) -> NDArrayf:
+def maximum_curvature(dem: NDArrayf | MArrayf, resolution: float | tuple[float, float] | None = None) -> NDArrayf:
...
@overload
-def maximum_curvature(
- dem: RasterType,
- resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
-) -> RasterType:
+def maximum_curvature(dem: RasterType, resolution: float | tuple[float, float] | None = None) -> RasterType:
...
def maximum_curvature(
- dem: NDArrayf | MArrayf | RasterType,
- resolution: float | tuple[float, float] | None = None,
- use_richdem: bool = False,
+ dem: NDArrayf | MArrayf | RasterType, resolution: float | tuple[float, float] | None = None
) -> NDArrayf | RasterType:
"""
Calculate the signed maximum profile or planform curvature parallel to the direction of the slope in m-1
@@ -1488,13 +1332,12 @@ def maximum_curvature(
:param dem: The DEM to calculate the curvature from.
:param resolution: The X/Y resolution of the DEM, only if passed as an array.
- :param use_richdem: Whether to use RichDEM to compute the attribute.
:raises ValueError: If the inputs are poorly formatted.
:returns: The profile curvature array of the DEM.
"""
- return get_terrain_attribute(dem=dem, attribute="maximum_curvature", resolution=resolution, use_richdem=use_richdem)
+ return get_terrain_attribute(dem=dem, attribute="maximum_curvature", resolution=resolution)
@overload
@@ -1600,7 +1443,7 @@ def roughness(dem: NDArrayf | MArrayf | RasterType, window_size: int = 3) -> NDA
Calculates the roughness, the maximum difference between neighbouring pixels, for any window size. Output is in the
unit of the DEM (typically meters).
- Based on: Dartnell (2000), http://dx.doi.org/10.14358/PERS.70.9.1081.
+ Based on: Dartnell (2000), https://environment.sfsu.edu/node/11292.
:param dem: The DEM to calculate the roughness from.
:param window_size: The size of the window for deriving the metric.
diff --git a/xdem/volume.py b/xdem/volume.py
index 1bfa1596..65d09447 100644
--- a/xdem/volume.py
+++ b/xdem/volume.py
@@ -1,6 +1,7 @@
"""Volume change calculation tools (aimed for glaciers)."""
from __future__ import annotations
+import logging
import warnings
from typing import Any, Callable
@@ -77,7 +78,7 @@ def hypsometric_binning(
elif kind == "custom":
zbins = bins # type: ignore
else:
- raise ValueError(f"Invalid bin kind: {kind}. Choices: ['fixed', 'count', 'quantile', 'custom']")
+ raise ValueError(f"Invalid bin kind: {kind}. Choices: ['fixed', 'count', 'quantile', 'custom'].")
# Generate bins and get bin indices from the mean DEM
indices = np.digitize(ref_dem, bins=zbins)
@@ -248,7 +249,9 @@ def calculate_hypsometry_area(
assert not np.any(np.isnan(ref_dem)), "The given reference DEM has NaNs. No NaNs are allowed to calculate area!"
if timeframe not in ["reference", "nonreference", "mean"]:
- raise ValueError(f"Argument 'timeframe={timeframe}' is invalid. Choices: ['reference', 'nonreference', 'mean']")
+ raise ValueError(
+ f"Argument 'timeframe={timeframe}' is invalid. Choices: ['reference', 'nonreference', 'mean']."
+ )
if isinstance(ddem_bins, pd.DataFrame):
ddem_bins = ddem_bins["value"]
@@ -283,7 +286,7 @@ def calculate_hypsometry_area(
return output
-def linear_interpolation(
+def idw_interpolation(
array: NDArrayf | MArrayf,
max_search_distance: int = 10,
extrapolate: bool = False,
@@ -301,7 +304,7 @@ def linear_interpolation(
:returns: A filled array with no NaNs
"""
if not _has_cv2:
- raise ValueError("Optional dependency needed. Install 'opencv'")
+ raise ValueError("Optional dependency needed. Install 'opencv'.")
# Create a mask for where nans exist
nan_mask = get_mask_from_array(array)
@@ -445,7 +448,7 @@ def local_hypsometric_interpolation(
# List of indexes to loop on
geometry_index = np.unique(mask[mask != 0])
- print(f"Found {len(geometry_index):d} geometries")
+ logging.info("Found %d geometries", len(geometry_index))
# Get fraction of valid pixels for each geometry
coverage = np.zeros(len(geometry_index))
@@ -457,7 +460,7 @@ def local_hypsometric_interpolation(
# Filter geometries with too little coverage
valid_geometry_index = geometry_index[coverage >= min_coverage]
- print(f"Found {len(valid_geometry_index):d} geometries with sufficient coverage")
+ logging.info("Found %d geometries with sufficient coverage", len(valid_geometry_index))
idealized_ddem = nodata * np.ones_like(dem)
@@ -529,7 +532,7 @@ def local_hypsometric_interpolation(
ddem_difference[idealized_ddem == nodata] = np.nan
# Spatially interpolate the difference between these two products.
- interpolated_ddem_diff = linear_interpolation(np.where(ddem_mask, np.nan, ddem_difference))
+ interpolated_ddem_diff = idw_interpolation(np.where(ddem_mask, np.nan, ddem_difference))
interpolated_ddem_diff[np.isnan(interpolated_ddem_diff)] = 0
# Correct the idealized dDEM with the difference to the original dDEM.
@@ -552,7 +555,6 @@ def get_regional_hypsometric_signal(
ref_dem: NDArrayf | MArrayf | RasterType,
glacier_index_map: NDArrayf | RasterType,
n_bins: int = 20,
- verbose: bool = False,
min_coverage: float = 0.05,
) -> pd.DataFrame:
"""
@@ -561,7 +563,6 @@ def get_regional_hypsometric_signal(
:param ddem: The dDEM to analyse.
:param ref_dem: A void-free reference DEM.
:param glacier_index_map: An array glacier indices of the same shape as the previous inputs.
- :param verbose: Show progress bar.
n_bins = 20 # TODO: This should be an argument.
:param n_bins: The number of elevation bins to subdivide each glacier in.
@@ -585,7 +586,11 @@ def get_regional_hypsometric_signal(
# Start a counter of glaciers that are actually processed.
count = 0
# Loop over each unique glacier.
- for i in tqdm(np.unique(glacier_index_map), desc="Finding regional signal", disable=(not verbose)):
+ for i in tqdm(
+ np.unique(glacier_index_map),
+ desc="Finding regional signal",
+ disable=logging.getLogger().getEffectiveLevel() > logging.INFO,
+ ):
# If i ==0, it's assumed to be periglacial.
if i == 0:
continue
@@ -651,7 +656,6 @@ def norm_regional_hypsometric_interpolation(
glacier_index_map: NDArrayf | RasterType,
min_coverage: float = 0.1,
regional_signal: pd.DataFrame | None = None,
- verbose: bool = False,
min_elevation_range: float = 0.33,
idealized_ddem: bool = False,
) -> NDArrayf:
@@ -665,7 +669,6 @@ def norm_regional_hypsometric_interpolation(
:param glacier_index_map: An array glacier indices of the same shape as the previous inputs.
:param min_coverage: The minimum fractional coverage of a glacier to interpolate. Defaults to 10%.
:param regional_signal: A regional signal is already estimate. Otherwise one will be estimated.
- :param verbose: Show progress bars.
:param min_elevation_range: The minimum allowed min/max bin range to scale a signal from.\
Default: 1/3 of the elevation range needs to be present.
:param idealized_ddem: Replace observed glacier values with the hypsometric signal. Good for error assessments.
@@ -685,7 +688,7 @@ def norm_regional_hypsometric_interpolation(
# If the regional signal was not given as an argument, find it from the dDEM.
if regional_signal is None:
regional_signal = get_regional_hypsometric_signal(
- ddem=ddem_arr, ref_dem=ref_arr, glacier_index_map=glacier_index_map, verbose=verbose
+ ddem=ddem_arr, ref_dem=ref_arr, glacier_index_map=glacier_index_map
)
# The unique indices are the unique glaciers.
@@ -694,7 +697,9 @@ def norm_regional_hypsometric_interpolation(
# Make a copy of the dDEM which will be filled iteratively.
ddem_filled = ddem_arr.copy()
# Loop over all glaciers and fill the dDEM accordingly.
- for i in tqdm(unique_indices, desc="Interpolating dDEM", disable=(not verbose)):
+ for i in tqdm(
+ unique_indices, desc="Interpolating dDEM", disable=logging.getLogger().getEffectiveLevel() > logging.INFO
+ ):
if i == 0: # i==0 is assumed to mean stable ground.
continue
# Create a mask representing a particular glacier.