example_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UCalib\n",
    "\n",
    "`UCalib` is an open source software written in Python for automatic image calibration from a set of images that are manually calibrated.\n",
    "\n",
    "### Description\n",
    "The calibration algorithm assumes that the camera position and the intrinsic parameters of the camera remain unchanged. The result of the process is a common position and intrinsic camera parameters for all images, and the orientation of the cameras for each of the images. In addition, planviews can be generated for each image. The development of this software is suitable for Argus-type video monitoring stations. Details about the algorithm and methodology are described in\n",
    "> *Simarro, G.; Calvete, D.; Souto, P. UCalib: Cameras Autocalibration on Coastal Video Monitoring Systems. Remote Sens. 2021, 13, 2795. https://doi.org/10.3390/rs13142795*\n",
    "\n",
    "The automatic calibration process consists of two steps:\n",
    "\n",
    "1. [Basis calibration](#basis-calibration)\n",
    "2. [Automatic image calibration](#automatic-image-calibration)\n",
    "\n",
    "Further UCalib allows to generate planviews for the calibrated images:\n",
    "\n",
    "3. [Planview generation](#planviews)\n",
    " \n",
    "A code to verify the quality of the GCPs used in the manual calibration of the basis images is also provided:\n",
    "\n",
    "4. [Check GCP for basis calibration](#gcp-check)\n",
    "\n",
    "### Requirements and project structure\n",
    "To run the software it is necessary to have Python (3.8) and install the following dependencies:\n",
    "- cv2 (4.2.0)\n",
    "- numpy (1.19.5)\n",
    "- scipy (1.6.0)\n",
    "\n",
    "In parenthesis we indicate the version with which the software has been tested. It is possible that it works with older versions. \n",
    "\n",
    "The structure of the project is the following:\n",
    "* `example.py`\n",
    "* `example_notebook.py`\n",
    "* **`ucalib`**\n",
    "  * `ucalib.py`\n",
    "  * `ulises_ucalib.py`\n",
    "* **`example`**\n",
    "  * **`basis`**\n",
    "    * `basisImage01.png`\n",
    "    * `basisImage01cal0.txt`\n",
    "    * `basisImage01cal.txt`\n",
    "    * `basisImage01cdg.txt`\n",
    "    * `basisImage01cdh.txt`\n",
    "    * . . .\n",
    "  * **`basis_check`**\n",
    "    * `basisImage01.png`\n",
    "    * `basisImage01cdg.txt`\n",
    "    * . . .\n",
    "  * **`images`**\n",
    "    * `image000001.png`\n",
    "    * `image000001cal.txt`\n",
    "    * . . .\n",
    "  * **`planviews`**\n",
    "    * `crxyz_planview.txt`\n",
    "    * `xy_planview.txt`\n",
    "    * `image000001plw.png`\n",
    "    * . . .\n",
    "  * **`TMP`**\n",
    "    * `basisImage01cal0_check.png`\n",
    "    * `basisImage01cal_check.png`\n",
    "    * `image000001cal_check.png`\n",
    "    * `image000001_checkplw.png`\n",
    "    * `image000001plw_check.png`\n",
    "    * . . .\n",
    "\n",
    "The local modules of `UCalib` are located in the **`ucalib`** folder.\n",
    "\n",
    "To run the demo in the folder **`example`** with the basis of images in **`basis`** and the images in **`images`** using a Jupyter Notebook we provide the file `example_notebook.ipynb`. For experienced users, the `example.py` file can be run in a terminal. `UCalib` handles `PNG` (recommended) and `JPEG` image formats.\n",
    "\n",
    "## Basis calibration\n",
    "To manually calibrate the images selected for the basis, placed in the folder **`basis`**,  it is necessary that each image `<basisImage>.png` is supplied with a file containing the Ground Control Points (GCP) and, optionally, a file with the Horizon Points (HP). The structure of each of these files is the following:\n",
    "* `<basisImage>cdg.txt`: For each GCP one line with  (minimum 6)\n",
    ">`pixel-column`, `pixel-row`, `x-coordinate`, `y-coordinate`, `z-coordinate`\n",
    "* `<basisImage>cdh.txt`: For each HP one line with (minimum 3)\n",
    ">`pixel-column`, `pixel-row`\n",
    "\n",
    "Quantities must be separated by at least one blank space between them and the last record should not be continued with a newline (return).\n",
    "\n",
    "To generate `<basisImage>cdg.txt` and `<basisImage>cdh.txt` files the [UClick](https://github.com/Ulises-ICM-UPC/UClick) software is available.\n",
    "\n",
    "### Run basis calibration\n",
    "Import modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "sys.path.insert(0, 'ucalib')\n",
    "import ucalib as ucalib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the main path and the path where the basis is located:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pathFolderMain = 'example'\n",
    "pathFolderBasis = os.path.join(pathFolderMain, 'basis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the value of maximum error allowed for the basis calibration:\n",
    "\n",
    "|  | Parameter | Suggested value | Units |\n",
    "|:--|:--:|:--:|:--:|\n",
    "| Critical reprojection pixel error | `eCritical` | _5._ | _pixel_ |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eCritical = 5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select an intrinsic camera calibration model.\n",
    "\n",
    "| Camara model | `parabolic` | `quartic`| `full` |\n",
    "|:--|:--:|:--:|:--:|\n",
    "| Lens radial distortion | parabolic | parabolic + quartic | parabolic + quartic |\n",
    "| Lens tangential distortion | no | no | yes |\n",
    "| Square pixels | yes | yes | no |\n",
    "| Decentering | no | no | yes |\n",
    "\n",
    "The `parabolic` model is recommended by default, unless the images are highly distorted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calibrationModel = 'parabolic'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case certain variables of the calibration model (see table below) are to be fixed, the values have to be provided in the `givenVariablesDict` dictionary. For a full calibration, the dictionary has to be left empty, i.e. `givenVariablesDict = {}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "givenVariablesDict = {'zc':142.5, 'k1a':-0.0025}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To facilitate the verification that the GCPs have been correctly selected in each image of the basis, images showing the GCPs and HPs (black), the reprojection of GCPs (yellow) and the horizon line (yellow) on the images can be generated. Set parameter `verbosePlot = True`, and to `False` otherwise. Images (`<basisImage>cal0_check.png`) will be placed on a **`TMP`** folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "verbosePlot = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the initial calibration algorithm for each image of the basis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ucalib.CalibrationOfBasisImages(pathFolderBasis, eCritical, calibrationModel, givenVariablesDict, verbosePlot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case that the reprojection error of a GCP is higher than the error `eCritical` for a certain image `<basisImage>`, a message will appear suggesting to re-run the calibration of the basis or to modify the values or to delete points in the file `<basisImage>cdg.txt`. If the calibration error of an image exceeds the error `eCritical` the calibration is given as _failed_. Consider re-run the calibration of the basis or verify the GPCs and HPs.\n",
    "\n",
    "Then, run the algorithm to obtain the position and the optimal intrinsic parameters of the camera:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ucalib.CalibrationOfBasisImagesConstantXYZAndIntrinsic(pathFolderBasis, calibrationModel, givenVariablesDict, verbosePlot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result of the calibration, the calibration file `<basisImage>cal.txt` is generated in the **`basis`** directory for each of the images. This file contains the following parameters:\n",
    "\n",
    "| Magnitudes | Variables | Units |\n",
    "|:--|:--:|:--:|\n",
    "| Camera position coordinates | `xc`, `yc`, `zc` | _m_ |\n",
    "| Camera orientation angles | `ph`, `sg`, `ta` | _rad_ |\n",
    "| Lens radial distortion (parabolic, quartic) | `k1a`, `k2a` | _-_ |\n",
    "| Lens tangential distortion (parabolic, quartic) | `p1a`, `p2a` | _-_ |\n",
    "| Pixel size | `sc`, `sr` | _-_ |\n",
    "| Decentering | `oc`, `or` | _pixel_ |\n",
    "| Image size | `nc`, `nr` | _pixel_ |\n",
    "| Calibration error | `errorT`| _pixel_ |\n",
    "\n",
    "The different calibration files `<basisImage>cal.txt` differ only in the angles of the camera orientation  (`ph`, `sg`, `ta`) and the calibration error (`errorT`). A `<basisImage>cal0.txt` file with the initial calibration parameters for each image of the basis will also have been generated.\n",
    "\n",
    "## Automatic image calibration\n",
    "\n",
    "In this second step, each of the images in the folder **`images`** will be automatically calibrated. Set the folder path where images to calibrate automatically are stored. To facilitate the verification of the calibration of each image, images showing the reprojection of the GCPs and the horizon line can be generated. Set parameter `verbosePlot = True`, and to `False` otherwise. Images(`<images>cal_check.png`) will be placed on a **`TMP`** folder. In the case that the images in the folder **`images`** have already been calibrated, set `overwrite = True` to generate them again and to `False` otherwise. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pathFolderImages = os.path.join(pathFolderMain, 'images')\n",
    "overwrite = False\n",
    "verbosePlot = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the values of the automatic image calibration parameters:\n",
    "\n",
    "|  | Parameter | Suggested value | Units |\n",
    "|:--|:--:|:--:|:--:|\n",
    "| Number of features to identify with ORB | `nORB` | _10000_ | _-_ |\n",
    "| Critical homography error | `fC` | _5._ | _pixel_ |\n",
    "| Critical number of pairs | `KC` | _4_ | _-_ |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nORB, fC, KC = 10000, 5., 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the algorithm to calibrate images automatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ucalib.AutoCalibrationOfImages(pathFolderBasis, pathFolderImages, nORB, fC, KC, overwrite, verbosePlot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the images `<image>.png` in directory **`images`**, a calibration file `<image>cal.txt` with the same characteristics as the one described above will be obtained. The autocalibration process may fail because the homography error is higher than the one set by the parameter `fC`, the number of pairs is lower than the critical value `KC` or `ORB` not being able to identify pairs in the image. In any of these cases it is reported that the calibration of the image  `<image>.png` has _failed_.\n",
    "\n",
    "## Planviews\n",
    "\n",
    "Once the frames have been calibrated, planviews can be generated. The region of the planview is the one delimited by the minimum area rectangle containing the points of the plane specified in the file `xy_planview.txt` in the folder **`planviews`**. The planview image will be oriented so that the nearest corner to the point of the first of the file  `xy_planview.txt` will be placed in the upper left corner of the image. The structure of this file is the following:\n",
    "* `xy_planview.txt`: For each points one line with \n",
    "> `x-coordinate`, `y-coordinate`\n",
    "\n",
    "A minimum number of three not aligned points is required. These points are to be given in the same coordinate system as the GCPs.\n",
    "\n",
    "Set the folder path where the file `xy_planview.txt` is located and the value of `z0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pathFolderPlanviews = os.path.join(pathFolderMain, 'planviews')\n",
    "z0 = 3.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resolution of the planviews is fixed by the pixels-per-meter established in the parameter `ppm`. To help verifying that the points for setting the planview are correctly placed, it is possible to show such points on the frames and on the planviews. Set the parameter `verbosePlot = True`, and to `False` otherwise. The images (`<image>_checkplw.png` and `<image>plw_check.png`) will be placed in a **`TMP`** folder. In the case that planviews have already been generated, set `overwrite = True` to generate them again and to `False` otherwise. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ppm = 2.0\n",
    "overwrite = False\n",
    "verbosePlot = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the algorithm to generate the planviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ucalib.PlanviewsFromImages(pathFolderImages, pathFolderPlanviews, z0, ppm, overwrite, verbosePlot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, for each of the calibrated images `<image>.png` in folder **`images`**, a planview `<image>plw.png` will be placed in the folder **`planviews`**. Note that objects outside the plane at height `z0` will show apparent displacements due to real camera movement. In the same folder, the file `crxyz_planview.txt` will be located, containing the coordinates of the corner of the planviews images:\n",
    "* `crxyz_planview.txt`: For each corner one line with \n",
    ">`pixel-column`, `pixel-row`, `x-coordinate`, `y-coordinate`, `z-coordinate`\n",
    "\n",
    "## GCP check\n",
    "\n",
    "To verify the quality of the GCPs used in the manual calibration of the basis images, a RANSAC (RANdom SAmple Consensus) is performed. Points of the files `<basisImage>cdg.txt` located at the **`basis_check`** folder will be tested. The calibration of the points (minimum 6) is done assuming a _parabolic_ camera model and requires the maximum reprojection pixel error `eCritical` for the GCPs. Set the folder and run the RANSAC algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pathFolderBasisCheck = os.path.join(pathFolderMain, 'basis_check')\n",
    "ucalib.CheckGCPs(pathFolderBasisCheck, eCritical)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each file `<basisImage>cdg.txt`, the GCPs that should be revised or excluded will be reported.\n",
    "\n",
    "## Contact us\n",
    "\n",
    "Are you experiencing problems? Do you want to give us a comment? Do you need to get in touch with us? Please contact us!\n",
    "\n",
    "To do so, we ask you to use the [Issues section](https://github.com/Ulises-ICM-UPC/UCalib/issues) instead of emailing us.\n",
    "\n",
    "## Contributions\n",
    "\n",
    "Contributions to this project are welcome. To do a clean pull request, please follow these [guidelines](https://github.com/MarcDiethelm/contributing/blob/master/README.md).\n",
    "\n",
    "## License\n",
    "\n",
    "UCalib is released under a [GPLv3 license](https://github.com/Ulises-ICM-UPC/UCalib/blob/main/LICENSE). If you use UCalib in an academic work, please cite:\n",
    "\n",
    "    @Article{rs13142795,\n",
    "      AUTHOR = {Simarro, Gonzalo and Calvete, Daniel and Souto, Paola},\n",
    "      TITLE = {UCalib: Cameras Autocalibration on Coastal Video Monitoring Systems},\n",
    "      JOURNAL = {Remote Sensing},\n",
    "      VOLUME = {13},\n",
    "      YEAR = {2021},\n",
    "      NUMBER = {14},\n",
    "      ARTICLE-NUMBER = {2795},\n",
    "      URL = {https://www.mdpi.com/2072-4292/13/14/2795},\n",
    "      ISSN = {2072-4292},\n",
    "      DOI = {10.3390/rs13142795}\n",
    "      }\n",
    "\n",
    "    @Online{ulisesdrone, \n",
    "      author = {Simarro, Gonzalo and Calvete, Daniel},\n",
    "      title = {UCalib},\n",
    "      year = 2021,\n",
    "      url = {https://github.com/Ulises-ICM-UPC/UCalib}\n",
    "      }"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}