tutorial_discover_3d_tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a33e2249",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<img src=\"images/demcompare_picto.png\" alt=\"Demcompare logo\" width=\"70\" align=\"right\"/>\n",
    "<img src=\"images/bulldozer_logo.png\" alt=\"Bulldozer logo\" width=\"70\" align=\"right\"/>\n",
    "<img src=\"images/picto_transparent.png\" alt=\"CARS logo\" width=\"70\" align=\"right\"/>\n",
    "\n",
    "\n",
    "# Discover CNES 3D\n",
    "# open-source tools\n",
    "## through a realistic scenario\n",
    "\n",
    "<img src=\"images/kahramanmaras_city.jpg\" alt=\"City\" width=\"200\" align=\"center\"/>\n",
    "\n",
    "<font size= “3”> _CNES 3D Team represented by David Youssefi, Dimitri Lallement, Emmanuel Dubois_ </font>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89e7c23d",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Outline\n",
    "\n",
    "* Tutorial preparation\n",
    "* Introduction on real earthquake scenario \n",
    "* Scenario steps\n",
    "* Pre-event and post-event DSM productions with CARS\n",
    "* Pre-event and post-event DTM productions with Bulldozer\n",
    "* DHM differences with Demcompare\n",
    "\n",
    "Slides are available here: [https://cars-cnes.github.io/discover-cnes-3d-tools/](https://cars-cnes.github.io/discover-cnes-3d-tools/)\n",
    "\n",
    "Tutorials sources: [https://github.com/cars-cnes/discover-cnes-3d-tools](https://github.com/cars-cnes/discover-cnes-3d-tools)\n",
    "\n",
    "<img src=\"images/demcompare_picto.png\" alt=\"Demcompare logo\" width=\"70\" align=\"right\"/>\n",
    "<img src=\"images/bulldozer_logo.png\" alt=\"Bulldozer logo\" width=\"70\" align=\"right\"/>\n",
    "<img src=\"images/picto_transparent.png\" alt=\"CARS logo\" width=\"70\" align=\"right\"/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c322b27-835e-4372-8550-d9f293995efb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Tutorial preparation\n",
    "\n",
    "\n",
    "1. Local Virtualenv (Linux) : best way\n",
    "\n",
    "2. Use  CNES jupyter hub with personal temporary account (nothing kept after)\n",
    "\n",
    "Other possibilities: virtualization, docker, ... \n",
    "\n",
    "Please ask presenters if environment is compatible.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5771f9e-36be-4e7b-8d89-b1ec443c7ba9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Virtualenv (Linux) Method\n",
    "\n",
    "Prerequisites: \n",
    "- python>3.8\n",
    "- python3-venv or virtualenv\n",
    "- gcc > 8 for vlfeat\n",
    "- Internet access\n",
    "- QGIS or other geotiff reader to see results\n",
    "\n",
    "Steps:\n",
    "\n",
    "0.  Go in your working directory\n",
    "\n",
    "1. Create a virtualenv\n",
    "\n",
    "```\n",
    "python -m venv venv-tutorial/\n",
    "source venv-tutorial/bin/activate\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9198cdb0-966d-46f5-8353-c8c5223af400",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Virtualenv (Linux) Method\n",
    "\n",
    "2. Install CARS and VLFEAT dependency :\n",
    "\n",
    "VLFEAT install: \n",
    "\n",
    "    git clone https://github.com/CNES/vlfeat.git\n",
    "    cd vlfeat && make MEX=$MATLABROOT/bin/ && cd ..\n",
    "    export CFLAGS=\"-I$PWD/vlfeat\"\n",
    "    export LDFLAGS=\"-L$PWD/vlfeat/bin/glnxa64\"\n",
    "    export LD_LIBRARY_PATH=\"$PWD/vlfeat/bin/glnxa64:$LD_LIBRARY_PATH\"\n",
    " \n",
    "CARS install: \n",
    "    \n",
    "    pip install --upgrade \"pip<=23.0.1\" \"numpy>=1.17.0\" cython\n",
    "    pip install cars\n",
    "\n",
    "CARS Doc: [https://cars.readthedocs.io/](https://cars.readthedocs.io/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ce480cb-d73c-4bb3-9308-46a5b158aba2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Virtualenv (Linux) Method\n",
    "\n",
    "\n",
    "3. Install bulldozer\n",
    "```\n",
    "   pip install bulldozer-dtm\n",
    "```\n",
    "\n",
    "for last version: \n",
    "```\n",
    "    git clone https://github.com/cnes/bulldozer\n",
    "    cd bulldozer\n",
    "    pip install .\n",
    "```\n",
    "\n",
    "Bulldozer Doc: [https://github.com/cnes/bulldozer/](https://github.com/cnes/bulldozer/)\n",
    "\n",
    "4. Install demcompare tool\n",
    "```\n",
    "   pip install demcompare[notebook]\n",
    "```\n",
    "\n",
    "Demcompare Doc: [https://demcompare.readthedocs.io/](https://cars.readthedocs.io/)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c0a255b-a005-4794-a344-35b975467a92",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Virtualenv (Linux) Method\n",
    "\n",
    "\n",
    "5. Get tutorial data from [https://github.com/cars-cnes/discover-cnes-3d-tools/](https://github.com/cars-cnes/discover-cnes-3d-tools/) in **data_turkey** directory : download from github or git clone : \n",
    "```\n",
    "git clone https://github.com/cars-cnes/discover-cnes-3d-tools/\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8db6be41-6d49-4d3b-b6b6-7d7d8514efe5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Virtualenv (Linux) Method\n",
    "\n",
    "\n",
    "8. Jupyter notebook installation\n",
    "\n",
    "Install Jupyter packages in previous virtualenv: \n",
    "    \n",
    "    source ./venv-tutorial/bin/activate # enter the virtualenv if not done.\n",
    "    pip install notebook rise bokeh\n",
    "    # add any tool you may need through pip install\n",
    "    \n",
    "Build Jupyter kernel:\n",
    "\n",
    "    python -m ipykernel install --sys-prefix --name=cars-kernel --display-name=cars-kernel\n",
    "\n",
    "9. Run Local Jupyter environnement:\n",
    "\n",
    "```\n",
    "jupyter notebook\n",
    "```\n",
    "\n",
    "10. Follow the proposed localhost link to the jupyter notebook web interface. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2968059-a31c-48f8-9c25-3e0e88e2057e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## CNES Jupyter Hub Method\n",
    "\n",
    "Prerequisites: \n",
    "- Firefox (not too old)\n",
    "- Internet access\n",
    "\n",
    "Steps:\n",
    " \n",
    "1. Connect a first time to https://jupyterhub.cnes.fr/ (use a personal temporary account, ask presenters)\n",
    "\n",
    "2. Select default configuration in \"Server Options\", click \"Start\" and wait (!)\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78c8b825-e767-470e-ad0f-9043cd40c715",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## CNES Jupyter Hub Method\n",
    "\n",
    "Steps:\n",
    "\n",
    "3. Run a terminal in CNES jupyter lab with following needed but no to bother details:\n",
    "\n",
    "Update kernel to a virtualenv with cars, bulldozer and demcompare (update version if needed)\n",
    "\n",
    "    mkdir -p .local/share/jupyter/kernels\n",
    "    ln -s /softs/projets/cars/kits/cars-0.7.0/jupyter/kernels/cars-0.7.0/  .local/share/jupyter/kernels/\n",
    "\n",
    "Resolve numba cache problem\n",
    "    \n",
    "    echo 'export NUMBA_CACHE_DIR=$TMPDIR' >> ~/.bashrc\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2aa0070-3509-406d-960e-a6bac3101828",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## CNES Jupyter Hub Method\n",
    "\n",
    "Steps: \n",
    "\n",
    "4. Stop the server (\"Shutdown\" or File-> Hub Control Panel -> \"Stop my server\") and wait !\n",
    "\n",
    "5. Reconnect to a server (\"Start my server\") to refresh kernels and configuration and select again default configuration.\n",
    "\n",
    "6. Copy data in your directory in a jupyter terminal\n",
    "\n",
    "Command:\n",
    "\n",
    "    cp -R /softs/projets/cars/tutorial/discover-cnes-3d-tools/ ~/\n",
    "    cd ~/discover-3d-tools/\n",
    "\n",
    "8. In a terminal, you can use cnes 3d tools using already install module\n",
    "\n",
    "Command:\n",
    "    \n",
    "    module load cars/0.7.0\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22cb69a9-b921-4667-ac81-e5cfe111ed5e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Ready ?\n",
    "\n",
    "Ask presenter if you have any problems. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05dd31c2-bea9-4aa8-8b88-16936323d499",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Introduction on real earthquake scenario\n",
    "\n",
    "<img src=\"images/demcompare_picto.png\" alt=\"Demcompare logo\" width=\"70\" align=\"right\"/>\n",
    "<img src=\"images/bulldozer_logo.png\" alt=\"Bulldozer logo\" width=\"70\" align=\"right\"/>\n",
    "<img src=\"images/picto_transparent.png\" alt=\"CARS logo\" width=\"70\" align=\"right\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f0c57a6-edf9-41c6-befb-83bcd64ad223",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Introduction on real earthquake scenario \n",
    "\n",
    "Event: [2023 Turkey/Syria Earthquake](https://en.wikipedia.org/wiki/2023_Turkey%E2%80%93Syria_earthquake)\n",
    "\n",
    "<img src=\"images/2023_Turkey_Syria_earthquake.jpg\" alt=\"collapse\" width=\"300\" align=\"center\"/>\n",
    "\n",
    "\n",
    "Links:\n",
    "    [https://en.wikipedia.org/wiki/2023_Turkey%E2%80%93Syria_earthquake](https://en.wikipedia.org/wiki/2023_Turkey%E2%80%93Syria_earthquake)\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55a0881f-010c-4059-a05b-ffa1cea34ba8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## How can satellite help ? \n",
    "\n",
    "- How to see earthquake impacts ? By analysing and comparing satellite pre-event and post-event. \n",
    "\n",
    "- 2D images not enough to see the real differences automatically, 3D can really improve by seeing elevations differences on the scenes. \n",
    "\n",
    "This tutorial show technical steps from satellite data to difference maps of the earthquake zone.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "903800fd-39fd-48ef-8447-b065a94dbd6a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Turkey scenario scene\n",
    "\n",
    "Scene chosen on Kahramanmaraş_Province in Turkey : download Turkey Pleaides data from https://github.com/cars-cnes/discover-cnes-3d-tools/ in **data_turkey/** directory : \n",
    "- Pre-event pleiades images : 2016/04/23\n",
    "- Post-event pleiades images: 2023/04/15\n",
    "\n",
    "Remarks:\n",
    "- Need at least 2 images to do 3D views (stereo).\n",
    "- Pleaides Panchromatic images (one band) are used for 3D for better quality\n",
    "- RGB scenes are also used to better see the scenes.\n",
    "\n",
    "\n",
    "Links:\n",
    "    [https://en.wikipedia.org/wiki/Kahramanmara%C5%9F_Province](https://en.wikipedia.org/wiki/Kahramanmara%C5%9F_Province)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ad150e3-514b-447d-9666-e5355780742a",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Pre-event Panchro scene 1\n",
    "\n",
    "<img src=\"images/pre_event_img1.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7702ebf-cf4d-451c-b679-be16d5a4c22d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Post-event Panchro scene 1\n",
    "\n",
    "<img src=\"images/post_event_img1.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30aad402-bfbf-49c3-b5a3-3ee0a4138d56",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Pre-event RGB scene 1\n",
    "\n",
    "<img src=\"images/pre_event_color1.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36b1804b-0fff-4897-ad2c-fd116ac325a0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Pre-event RGB scene 2\n",
    "\n",
    "<img src=\"images/pre_event_color2.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f77a4e64-1bcf-4bef-9251-cf17a8be7d50",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Post-event RGB scene 1\n",
    "\n",
    "<img src=\"images/post_event_color1.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceabaab2-feb0-4f4e-9b87-18e9bfd3dd8f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Post-event RGB scene 2\n",
    "\n",
    "<img src=\"images/post_event_color2.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60511249-5246-4556-ba4b-5aa39ec40348",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Reminders:\n",
    "\n",
    "- DSM: Digital Surface Models, represents the earth surface with all objects.\n",
    "- DTM: Digital Terrain Models, bare ground surface without any objects like plants and buildings\n",
    "\n",
    "<img src=\"images/dsm_dtm_dhm_illustration.png\" alt=\"collapse\" width=\"300\" align=\"center\"/>\n",
    "\n",
    "\n",
    "- DHM: Digital Height Model (DSM-DTM), contains only the objects above the DTM.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ac124f2-00b6-49ab-8eda-f8b954b72087",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Scenario steps. \n",
    "\n",
    "<img src=\"images/discover3d_pipeline.svg\" alt=\"scenario data process pipeline\" align=\"center\"/>\n",
    "\n",
    "- From satellite pre and post event stereo data, pre and post event Digital Surface Models are created with CARS. \n",
    "- From Digital Surface Models, all \"above terrain\" is removed with Bulldozer to make Digital Terrain Models.\n",
    "- Bulldozer produces also only \"above terrain\" terrain Digital Height Models (DSM-DTM differences)\n",
    "- From this 3D data, we compare Digital Height Models differences. \n",
    "\n",
    "\n",
    "<img src=\"images/picto_transparent.png\" alt=\"CARS logo\" width=\"70\" align=\"right\"/>\n",
    "<img src=\"images/bulldozer_logo.png\" alt=\"Bulldozer logo\" width=\"70\" align=\"right\"/>\n",
    "<img src=\"images/demcompare_picto.png\" alt=\"Demcompare logo\" width=\"70\" align=\"right\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4a9b959-41a9-499f-b3a4-ea4e854c949d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# DSM productions with CARS\n",
    "\n",
    "1. Tutorial DSM production with CARS : see dedicated slides\n",
    "\n",
    "2. Configure CARS with Turkey Data\n",
    "\n",
    "3. Run pre event production\n",
    "\n",
    "4. Run post event production\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8f40a47-4c55-49d6-a78b-334dd57ddbc9",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## CARS configuration\n",
    "\n",
    " **config_cars_pre_event.json** example in **data_turkey** directory:\n",
    "```\n",
    "{\n",
    "    \"inputs\": {\n",
    "        \"sensors\" : {\n",
    "            \"one\": {\n",
    "                \"image\": \"pre_event_img1.tif\",\n",
    "                \"geomodel\": \"pre_event_img1.geom\",\n",
    "                \"color\": \"pre_event_color1.tif\",\n",
    "                \"no_data\": 0\n",
    "            },\n",
    "            \"two\": {\n",
    "                \"image\": \"pre_event_img2.tif\",\n",
    "                \"geomodel\": \"pre_event_img2.geom\",\n",
    "                \"color\": \"pre_event_color2.tif\",            \n",
    "                \"no_data\": 0\n",
    "\t       }\n",
    "        },\n",
    "        \"pairing\": [[\"one\", \"two\"]],\n",
    "        \"initial_elevation\": \"srtm_dir/N37E036.hgt\"\n",
    "    },\n",
    "    \"output\": {\n",
    "        \"out_dir\": \"outputs_cars_pre_event\",\n",
    "        \"dsm_basename\": \"cars_dsm_pre_event.tif\",\n",
    "        \"clr_basename\": \"cars_clr_pre_event.tif\"\n",
    "    }\n",
    "}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3c258fd-c871-43bf-90c4-51f7b759772b",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## CARS DSM production\n",
    "\n",
    "Steps: \n",
    "\n",
    "    # in virtualenv before !\n",
    "    cd data_turkey\n",
    "    \n",
    "    # run pre event production\n",
    "    cars config_cars_pre_event.json\n",
    "    # run post event production from post event scene 1 and post event scene 2\n",
    "    cars config_cars_post_event.json\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32cc5366-6861-4c96-aeb9-9ce9fb9da54c",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## CARS DSM results pre event\n",
    "\n",
    "<img src=\"images/cars_dsm_color_pre_event.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c37333d4-4852-437c-9656-cdc9ec81d21d",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## CARS DSM results post event\n",
    "\n",
    "<img src=\"images/cars_dsm_color_post_event.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e4c8c71-2dc9-4b74-842e-83bac9dd2ae1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# DTM and DHM productions with Bulldozer\n",
    "\n",
    "\n",
    "1. Tutorial DTM and DHM production with Bulldozer : see dedicated slides\n",
    "\n",
    "2. Configure Bulldozer with CARS DSM Turkey Data\n",
    "\n",
    "3. Run pre event DTM and DHM production\n",
    "\n",
    "4. Run post event DTM and DHM production"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c57a28c-5313-4e4e-bb96-0154a3e80ff0",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Bulldozer configuration\n",
    "\n",
    " **config_bulldozer_pre_event.yaml** example in **data_turkey** directory:\n",
    "```yaml\n",
    "#-------------------------#\n",
    "#       Parameters        #\n",
    "#-------------------------#\n",
    "# [Required] - Input DSM path (expected format: \"<folder_1>/<folder_2>/<file>.<[tif/tiff]>\") \n",
    "dsm_path : \"outputs_cars_pre_event/cars_dsm_pre_event.tif\"\n",
    "# [Required] - Output directory path (if the directory doesn't exist, create it)\n",
    "output_dir : \"outputs_bulldozer_pre_event\"\n",
    "\n",
    "#-------------------------#\n",
    "#         Options         #\n",
    "#-------------------------#\n",
    "# [Optional] - If True, generates the DHM (DSM - DTM) in the output directory \n",
    "generate_dhm : True\n",
    "# [Optional] - If null, the output DTM resolution will be the same as the input DSM (float value in meter)\n",
    "output_resolution : null\n",
    "# [Optional] - If null, bulldozer will use the maximum number of available CPU on your system\n",
    "nb_max_workers : 16\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "995422f9-b6cc-4d8a-b916-eb10d059cd37",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Bulldozer configuration\n",
    "\n",
    " **config_bulldozer_pre_event.yaml** example in **data_turkey** directory:\n",
    "```yaml\n",
    "#-------------------------#\n",
    "# Bulldozer core settings #\n",
    "#-------------------------#\n",
    "# /!\\ Modify those data at your own risk (it is suggested to keep the default values) /!\\\n",
    "\n",
    "# [Optional] - DetectDisturbedAreasAndFill part\n",
    "slope_threshold : 2.0\n",
    "# [Optional] - Number of evaluated axis. Vertical and horizontal if True else vertical, horizontal and diagonals.\n",
    "four_connexity : True\n",
    "\n",
    "# [Optional] - DtmExtraction parameters\n",
    "uniform_filter_size : 3\n",
    "prevent_unhook_iter : 10\n",
    "num_outer_iter : 50\n",
    "num_inner_iter : 10\n",
    "mp_tile_size : 1500\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d3bfdb4-6080-4e8a-b23c-3ee48bad4dd5",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Bulldozer DTM and DHM production\n",
    "\n",
    "Steps: \n",
    "\n",
    "    # in virtualenv before !\n",
    "    cd data_turkey\n",
    "    # you need CARS generated DSM produced before.\n",
    "    \n",
    "    # run pre event production\n",
    "    bulldozer --conf config_bulldozer_pre_event.yaml\n",
    "    # run post event production from post event scene 1 and post event scene 2\n",
    "    bulldozer --conf config_bulldozer_post_event.yaml\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d200f692-b20e-4035-9090-bc8bdf4cc30c",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Bulldozer pre event DTM results \n",
    "\n",
    "<img src=\"images/bulldozer_dtm_pre_event.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9faa5be-6cf9-4d84-a080-11d6762b2640",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Bulldozer post event DTM results \n",
    "\n",
    "<img src=\"images/bulldozer_dtm_post_event.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81771d8f-259f-48fb-b43b-2662befe45ef",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Bulldozer pre event DHM results \n",
    "\n",
    "<img src=\"images/bulldozer_dhm_pre_event.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d715882d-7e70-4355-8f25-546cae55b811",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Bulldozer post event DHM results \n",
    "\n",
    "<img src=\"images/bulldozer_dhm_post_event.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0052177-c98d-4ae8-ab57-3d5fc2d0977b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# DHM differences with Demcompare\n",
    "\n",
    "1. Demcompare tutorial (see dedicated slides)\n",
    "2. Demcompare configuration\n",
    "3. DSM and DHM differences production\n",
    "4. DSM and DHM differences results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9a9ea60-805f-494d-be54-069be5a125a8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## DSM vs DHM differences\n",
    "\n",
    "DSM differences is a raw difference between post event et pre event. \n",
    "\n",
    "Using Digital Height Models allows to: \n",
    "\n",
    "- normalize the height to zero (zero equal = terrain)\n",
    "- remove possible vibrations of the satellite\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7863e42-389b-418f-95e8-e04060fa1dbe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Demcompare configuration\n",
    "\n",
    "**config_demcompare_diff_dhm.json** example in **data_turkey** directory:\n",
    "```\n",
    "{\n",
    "    \"output_dir\": \"./outputs_demcompare_diff_dhm/\",\n",
    "\t\"input_ref\" : {\n",
    "        \"path\": \"outputs_bulldozer_post_event/DHM.tif\",\n",
    "\t\t\"zunit\" : \"m\"\n",
    "\t},\n",
    "\t\"input_sec\" : {\n",
    "        \"path\": \"outputs_bulldozer_pre_event/DHM.tif\",\n",
    "\t\t\"zunit\" : \"m\",\n",
    "        \"geoid_georef\" : \"True\",\t\t\n",
    "\t\t\"nodata\" : 0\n",
    "\t},\n",
    "\t\"coregistration\": {\n",
    "\t\t\"method_name\": \"nuth_kaab_internal\",\n",
    "\t\t\"number_of_iterations\": 6,\n",
    "\t\t\"estimated_initial_shift_x\": 0,\n",
    "\t\t\"estimated_initial_shift_y\": 0\n",
    "\t},\t\n",
    "  \"statistics\": {\n",
    "    \"classification_layers\": {\n",
    "      \"Slope0\": {\n",
    "        \"type\": \"slope\",\n",
    "        \"ranges\": [\n",
    "          0,\n",
    "          5,\n",
    "          10,\n",
    "          25,\n",
    "          45\n",
    "        ]\n",
    "      }\n",
    "    },\n",
    "    \"remove_outliers\": \"True\"\n",
    "  },\n",
    "  \"report\": \"default\"\n",
    "}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea0a9f97-e8d8-416f-b036-48120b0d47d1",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## DHM differences (POST-PRE) results\n",
    "\n",
    "- Blue: new elements \n",
    "- Red: collapsed or removed elements\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a6b37bf-bd4e-42f5-b786-3eb8895f4810",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## DHM differences (POST-PRE) results\n",
    "\n",
    "<img src=\"images/demcompare_dhm_diff.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97f9340b-7768-4ac4-8cd3-259e565df546",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "## DHM differences (POST-PRE) results without coregistration\n",
    "TODO: wrong colormap, skip the slides for now \n",
    "<img src=\"images/demcompare_dhm_diff_without_coreg.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0418d4f5-a30f-4787-a247-acd810c2083b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## DHM differences (POST-PRE) results stacked with pre event DSM\n",
    "\n",
    "<img src=\"images/demcompare_dhm_diff_cars_clr_pre_event.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d86ee33-bc23-47b2-bbef-d9cf134a652f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## DHM differences (POST-PRE) stacked with post event DSM\n",
    "\n",
    "<img src=\"images/demcompare_dhm_diff_cars_clr_post_event.png\" alt=\"collapse\" width=\"550\" align=\"center\"/>\n",
    "\n",
    "\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76284494-9fbb-4ef0-b7c6-9300b6356bb1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "This tutorial has shown a real scenario application using 3D maps difference before and after the event. \n",
    "\n",
    "CNES 3D tools have been demonstrated to execute scenario steps.\n",
    "\n",
    "Satellite can be a real added value: \n",
    "- satellite revisit rate on a scene\n",
    "- less expensive than a local LIDAR campaign\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d1cf8ca-584f-4278-bf16-8002227e00f3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Perspectives:\n",
    "\n",
    "Satellite 3D is beginning to be more and more accessible. \n",
    "\n",
    "Tools: \n",
    "- improving presented tools performance and stability\n",
    "\n",
    "Applications:\n",
    "- Automatic count number of buildings\n",
    "- helping people that labelize the impacted areas\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d8668f6-0fea-4a0c-8a03-50673f57fce2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Questions ? \n",
    "\n",
    "General 3D CNES tools contact: [cars@cnes.fr](cars@cnes.fr) \n",
    "\n",
    "Specific contact: \n",
    "\n",
    "- CARS: david.youssefi@cnes.fr\n",
    "- Bulldozer: dimitri.lallement@cnes.fr\n",
    "- Demcompare: emmanuel.dubois@cnes.fr"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Diaporama",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}