Add cascading_seach function (#18)

* towards cascade search * add cascading search, restructure docs
uw-cryo · Nov 1, 2024 · 478116f · 478116f
1 parent c6892d2
commit 478116f
Show file tree

Hide file tree

Showing 18 changed files with 442 additions and 106 deletions.
diff --git a/README.md b/README.md
@@ -2,6 +2,7 @@
 
 [![Actions Status][actions-badge]][actions-link]
 [![Documentation Status][rtd-badge]][rtd-link]
+
 [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/uw-cryo/coincident)
 
 <!-- SPHINX-START -->
@@ -32,44 +33,7 @@ See here for more information:
 
 **This tool is under active development, there are no stable releases yet!**
 
-## Development
-
-Use [pixi](https://pixi.sh) for environment management
-
-```bash
-git clone https://github.com/uw-cryo/coincident.git
-cd coincident
-git checkout -b newfeature
-pixi shell --environment dev # type `exit` to deactivate
-pre-commit install
-
-# Or run pre-configured commands:
-pixi run networktest # or 'test'
-pixi run precommit # also runs automatically upon commits
-pixi run lint
-pixi run docs
-```
-
-## Authentication
-
-Some datasets require authentication to _search_ (Maxar) others only require
-authentication to _download_ data (NASA). `coincident` assumes you have the
-following Environment Variables defined:
-
-```bash
-export EARTHDATA_USERNAME=aaaaa
-export EARTHDATA_PASSWORD=bbbbb
-export MAXAR_API_KEY=ccccc
-export PC_SDK_SUBSCRIPTION_KEY=ddddd
-```
-
-Sign up for credentials at the following webpages:
-
-- [](https://urs.earthdata.nasa.gov)
-- [](https://developers.maxar.com/docs/authentication/guides/api-key)
-- [](https://planetarycomputer.developer.azure-api.net)
-
-### Acknowledgements
+## Acknowledgements
 
 - Python packaging template provided by
   <https://github.com/scientific-python/cookie>

diff --git a/docs/conf.py b/docs/conf.py
@@ -65,7 +65,7 @@
 
 # NOTE: consider adding back in once for distinct sections (user guide, examples, API reference)
 # https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/layout.html#primary-sidebar-left
-html_sidebars = {"**": []}
+# html_sidebars = {"**": []}
 
 myst_enable_extensions = [
     "colon_fence",

diff --git a/docs/examples/cascading_search.ipynb b/docs/examples/cascading_search.ipynb
@@ -0,0 +1,213 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Cascading search\n",
+    "\n",
+    "In the [Quickstart](./quickstart.ipynb) notebook we covered searching datasets one-by-one and gradually reducing the spatial domain of our search based on overlapping footprints. \n",
+    "\n",
+    "`coincident` also provides a  `cascading_search()`[#coincident.search.cascading_search] method as a convenience to perform this same type of search in one go.  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import coincident\n",
+    "import geopandas as gpd\n",
+    "import xyzservices.providers as xyz\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "%matplotlib inline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Identify a primary dataset\n",
+    "\n",
+    "Start by loading a full resolution polygon of a 3DEP LiDAR workunit which has a known start_datetime and end_datatime:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "workunit = \"CO_WestCentral_2019\"\n",
+    "df_wesm = coincident.search.wesm.read_wesm_csv()\n",
+    "gf_lidar = coincident.search.wesm.load_by_fid(\n",
+    "    df_wesm[df_wesm.workunit == workunit].index\n",
+    ")\n",
+    "gf_lidar"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Search secondary datasets\n",
+    "\n",
+    "Provide a list that will be searched in order. The list contains tuples of dataset aliases and the temporal pad in days to search before the primary dataset start and end dates"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "secondary_datasets = [\n",
+    "    (\"maxar\", 14),  # +/- 14 days from lidar\n",
+    "    (\"gedi\", 40),  # +/- 40 days from lidar\n",
+    "    (\"icesat-2\", 60),\n",
+    "]\n",
+    "\n",
+    "dataframes = coincident.search.cascading_search(\n",
+    "    gf_lidar,\n",
+    "    secondary_datasets,\n",
+    "    min_overlap_area=30,  # km^2\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Visualize results\n",
+    "\n",
+    "Below we visualize cropped footprints from each secondary dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gf_maxar = dataframes[0]\n",
+    "print(len(gf_maxar))\n",
+    "m = gf_lidar.explore(\n",
+    "    style_kwds=dict(fill=None, color=\"black\"), tiles=xyz.CartoDB.Positron\n",
+    ")  # basemap\n",
+    "gf_maxar.explore(m=m, column=\"datetime\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gf_gedi = dataframes[1]\n",
+    "print(len(gf_gedi))\n",
+    "m = gf_lidar.explore(\n",
+    "    style_kwds=dict(fill=None, color=\"black\"), tiles=xyz.CartoDB.Positron\n",
+    ")  # basemap\n",
+    "gf_gedi.explore(m=m, column=\"datetime\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gf_is2 = dataframes[2]\n",
+    "print(len(gf_maxar))\n",
+    "m = gf_lidar.explore(\n",
+    "    style_kwds=dict(fill=None, color=\"black\"), tiles=xyz.CartoDB.Positron\n",
+    ")  # basemap\n",
+    "gf_is2.explore(m=m, column=\"datetime\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fig, ax = plt.subplots(figsize=(10, 5))\n",
+    "for df in dataframes:\n",
+    "    if \"constellation\" in df.columns:\n",
+    "        label = df.constellation.iloc[0]\n",
+    "    else:\n",
+    "        label = df.collection.iloc[0]\n",
+    "    plt.scatter(x=df[\"datetime\"], y=df[\"collection\"], s=50, marker=\"d\", label=label)\n",
+    "\n",
+    "# NOTE: probably a more robust way to set aspect depending on date range\n",
+    "ax.set_aspect(6)\n",
+    "plt.axvline(\n",
+    "    gf_lidar.start_datetime.iloc[0],\n",
+    "    color=\"black\",\n",
+    "    linestyle=\"--\",\n",
+    "    linewidth=0.5,\n",
+    "    label=\"LiDAR\",\n",
+    ")\n",
+    "plt.axvline(gf_lidar.end_datetime.iloc[0], color=\"black\", linestyle=\"--\", linewidth=0.5)\n",
+    "plt.title(\"CO_WestCentral_2019 Overlaps\")\n",
+    "plt.legend(loc=\"lower right\")\n",
+    "fig.autofmt_xdate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Save results\n",
+    "\n",
+    "The footprints of the last secondary dataset show where we have *spatial* intersections across all datasets. We save this a single MultiPolygon to use in QGIS or [geojson.io](https://geojson.io)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gf_is2.dissolve()[[\"geometry\"]].to_file(\"/tmp/CO_WestCentral_2019_overlaps.geojson\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Summary \n",
+    "\n",
+    "- The CO_WestCentral_2019 USGS 3DEP LiDAR was acquired between 2019-08-21 and 2019-09-19\n",
+    "- We found 7 Maxar Stereo acquisitions with 14 days of the LiDAR\n",
+    "- We found 14 GEDI acquisitions that overlap the lidar+stereo footprints within 40 days of LiDAR \n",
+    "- We found 7 ICESat-2 acquisitions that overlap combined lidar+stereo+GEDI footprints within 60 days of LiDAR\n",
+    "\n",
+    "The final 'overlap' polygons have at least a 30k^2 area in which at least two of the datasets intersect. Acquisition dates for any given footprint vary from 60 days before the LiDAR was acquired through 60 days afterwards. "
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "dev",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/docs/examples/index.md b/docs/examples/index.md
@@ -0,0 +1,11 @@
+# Examples
+
+This section contains Jupyter Notebooks with narrative workflows using the
+`coincident` library.
+
+```{toctree}
+:maxdepth: 1
+
+quickstart
+cascading_search
+```
diff --git a/docs/quickstart.ipynb → docs/examples/quickstart.ipynb b/docs/quickstart.ipynb → docs/examples/quickstart.ipynb
@@ -14,7 +14,7 @@
     "1. reduce to region with coincident maxar stereo within an acceptable temporal range\n",
     "1. optionally reduce further with additional datasets such as icesat-2 and gedi altimetry\n",
     "\n",
-    "This notebook provides and example starting from USGS 3DEP LiDAR in Colorado, USA"
+    "This notebook provides an example starting from USGS 3DEP LiDAR in Colorado, USA"
    ]
   },
   {

diff --git a/docs/index.md b/docs/index.md
@@ -5,26 +5,9 @@
 ```
 
 ```{toctree}
-:maxdepth: 2
 :hidden:
 
-installation
-introduction
-datasets
-```
-
-```{toctree}
-:maxdepth: 2
-:hidden:
-:caption: Examples
-
-quickstart
-```
-
-```{toctree}
-:maxdepth: 2
-:hidden:
-:caption: API
-
-api
+user_guide/index.md
+examples/index.md
+api.rst
 ```
diff --git a/docs/installation.md b/docs/installation.md
diff --git a/docs/user_guide/contribute.md b/docs/user_guide/contribute.md
@@ -0,0 +1,19 @@
+# Contributing Guide
+
+Contributions are welcome!
+
+We recommend use [pixi](https://pixi.sh) for environment management
+
+```bash
+git clone https://github.com/uw-cryo/coincident.git
+cd coincident
+git checkout -b newfeature
+pixi shell --environment dev # type `exit` to deactivate
+pre-commit install
+
+# Or run pre-configured commands:
+pixi run networktest # or 'test'
+pixi run precommit # also runs automatically upon commits
+pixi run lint
+pixi run docs
+```
diff --git a/docs/datasets.md → docs/user_guide/datasets.md b/docs/datasets.md → docs/user_guide/datasets.md
diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md
@@ -0,0 +1,29 @@
+# User Guide
+
+This section contains basic guidance on installing and using the `coincident`
+library
+
+```{toctree}
+:maxdepth: 1
+:hidden:
+:caption: Getting Started
+
+installation
+introduction
+```
+
+```{toctree}
+:maxdepth: 1
+:hidden:
+:caption: Datasets
+
+datasets
+```
+
+```{toctree}
+:maxdepth: 1
+:hidden:
+:caption: Contributing
+
+contribute
+```