02_Visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web-based Image Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "- Understand the capabilities and limitations of **img, canvas, WebGL, WebAssembly,** and **WebGPU** browser technologies\n",
    "- Identify the trade-offs in **server-side** and **client-side** rendering\n",
    "- Become familiar with **open source, web-based scientific image visualization tools** such as: itk-vtk-viewer, itkwidgets, kaibu, imagej.js, matplotlib, ipympl, neuroglancer, and the web-based capabilities of paraview, napari "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/thewtex/modern-insights-from-microscopy-images/blob/master/02_Visualization.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install package dependencies\n",
    "import sys\n",
    "\n",
    "!{sys.executable} -m pip install matplotlib ipycanvas ipympl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Currently required to enable running the ipycanvas window on colab\n",
    "try:\n",
    "    from google.colab import output\n",
    "    if hasattr(output, 'enable_custom_widget_manager'):\n",
    "        output.enable_custom_widget_manager()\n",
    "except ModuleNotFoundError:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web browser rendering technologies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `img` elements\n",
    "\n",
    "The [HTML `img` element embeds](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img) an image into a webpage.\n",
    "\n",
    "*Capabilities:*\n",
    "\n",
    "- ✔ Ubiquitous support\n",
    "- ✔ Supported file formats: APNG, AVIF, GIF, JPEG, PNG, SVG, WebP\n",
    "- ✔ Possible to inline support for base64-encoded images\n",
    "\n",
    "*Limitations:*\n",
    "\n",
    "- ✘ Not well suited for dynamic content\n",
    "- ✘ Does not support multiple pixel component types, anisotropic spacing, 3D images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://interactive-examples.mdn.mozilla.net/media/cc0-images/grapefruit-slice-332-332.jpg\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example\n",
    "import IPython\n",
    "\n",
    "IPython.display.HTML('<img src=\"https://interactive-examples.mdn.mozilla.net/media/cc0-images/grapefruit-slice-332-332.jpg\">')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `canvas` elements\n",
    "\n",
    "The [HTML `canvas` element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/canvas) is capable of drawing 2D graphics and animations.\n",
    "\n",
    "*Capabilities:*\n",
    "\n",
    "- ✔ Ubiquitous support\n",
    "- ✔ Well suited for dynamic content\n",
    "- ✔ Low level API\n",
    "- ✔ Many JS libraries available for high level manipulation\n",
    "\n",
    "*Limitations:*\n",
    "\n",
    "- ✘ The Canvas API focuses on 2D graphics\n",
    "- ✘ Limited performance relative to WebGL, WebGPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f5aa86b5333e4f8782fb45d16e990b14",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(height=1200, layout=Layout(width='100%'), width=1600)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ipycanvas intro example\n",
    "# https://ipycanvas.readthedocs.io/en/latest/index.html\n",
    "from math import pi\n",
    "\n",
    "from ipycanvas import Canvas\n",
    "\n",
    "canvas = Canvas(width=1600, height=1200, layout=dict(width=\"100%\"))\n",
    "\n",
    "canvas.fill_style = \"#8ee05e\"\n",
    "canvas.fill_rect(0, 0, canvas.width, canvas.height)\n",
    "\n",
    "canvas.fill_style = \"#f5f533\"\n",
    "canvas.fill_circle(canvas.width / 2.0, canvas.height / 2.0, 500)\n",
    "\n",
    "canvas.stroke_style = \"black\"\n",
    "canvas.line_width = 30\n",
    "canvas.stroke_circle(canvas.width / 2.0, canvas.height / 2.0, 500)\n",
    "\n",
    "canvas.fill_style = \"black\"\n",
    "canvas.fill_circle(canvas.width / 2.7, canvas.height / 3.0, 100)  # Right eye\n",
    "canvas.stroke_arc(canvas.width / 2.0, canvas.height / 2.0, 400, 0, pi, False)  # Mouth\n",
    "canvas.stroke_arc(\n",
    "    canvas.width - canvas.width / 2.7, canvas.height / 2.7, 100, 0, pi, True\n",
    ")  # Left eye\n",
    "\n",
    "canvas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WebGL\n",
    "\n",
    "Hardware accelerated 2D and 3D graphics that also draws on the `canvas` element.\n",
    "\n",
    "*Capabilities:*\n",
    "\n",
    "- ✔ Widespread support\n",
    "- ✔ Well suited for dynamic content\n",
    "- ✔ Great performance\n",
    "- ✔ Low level API\n",
    "- ✔ JS libraries available for high level manipulation, e.g. vtk.js, three.js\n",
    "- ✔ 3D support\n",
    "\n",
    "*Limitations:*\n",
    "\n",
    "- ✘ Resource limitations based on texture size, number of textures\n",
    "- ✘ WebGL 1.0 API that closely conforms to OpenGL ES 2.0 \n",
    "- ✘ WebGL 2.0 API introduces support for much of the OpenGL ES 3.0\n",
    "- ✘ Legacy technology relative to WebGPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WebGL example\n",
    "\n",
    "[tensorboard-plugin-3d](https://tensorboardplugin3d.readthedocs.io/en/latest/):\n",
    "\n",
    "\n",
    "![Tensorboard-plugin-3d](https://www.kitware.com/main/wp-content/uploads/2022/04/input_adjusted_nozoom-1024x520.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video width=\"840\" height=\"460\" controls=\"\"><source src=\"https://www.kitware.com/main/wp-content/uploads/2022/04/bad_spleen-2.mp4\"></video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('<video width=\"840\" height=\"460\" controls=\"\"><source src=\"https://www.kitware.com/main/wp-content/uploads/2022/04/bad_spleen-2.mp4\"></video>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WebGPU\n",
    "\n",
    "WebGPU is the working name for a future web standard and JavaScript API for accelerated graphics and compute, aiming to provide \"modern 3D graphics and computation capabilities\".\n",
    "\n",
    "*Capabilities:*\n",
    "\n",
    "- ✔ Great performance\n",
    "- ✔ Low level, modern graphics API\n",
    "- ✔ Can be compiled into SPIR-V, a cross-API intermediate representation\n",
    "- ✔ 3D support\n",
    "\n",
    "*Limitations:*\n",
    "\n",
    "- ✘ Still undergoing standardization\n",
    "- ✘ New shader language, *WGSL*\n",
    "- ✘ Not yet broadly available"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Server-side vs client-side rendering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Web-based visualization technologies generally fall into two broad categories:\n",
    "\n",
    "1. **Server-side rendering**: The data is rendered on a server, and the rendered screenshot is transferred over the network to the client.\n",
    "2. **Client-side rendering**: The data is rendered on the client. A server may or may not exist. A server may exist to dynamically generate a multi-scale pyramid, etc.\n",
    "\n",
    "The relative advantages are:\n",
    "\n",
    "#### Server-side rendering\n",
    "\n",
    "- ✔ A large server GPU can render for an underpowered client, e.g. a phone.\n",
    "- ✔ A large server GPU may have more graphics memory for an underpowered client.\n",
    "- ✔ Custom hardware capabilities and software APIs, e.g. raytracing, can be provided to all clients.\n",
    "- ✔ There is a shorter path to migrate traditional desktop applications to the web.\n",
    "- ✔ The raw data does not need to be transferred to the client.\n",
    "\n",
    "#### Client-side rendering\n",
    "\n",
    "- ✔ Performance and responsiveness are generally better -- the rendered image and interactive events do not need to be transferred over the network.\n",
    "- ✔ Infinitely scalable with the number of clients -- the server does not get overloaded.\n",
    "- ✔ Lower maintenance and resource costs -- a server often is not required.\n",
    "- ✔ A graphics card or complicated software graphics context is not required on a server.\n",
    "- ✔ Rendered data does not need to be continually sent over the network when interacting with the image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Open source, web-based scientific image visualization tools "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A survey of open source, web-based scientific image visualization tools. Yes, this is an *incomplete* list! :-)\n",
    "\n",
    "#### Server-side rendering\n",
    "\n",
    "- [ParaViewWeb](https://kitware.github.io/paraviewweb/index.html) - ParaViewWeb is a Web framework which brings the power of ParaView and VTK into the Web.\n",
    "- [napari](https://napari.org/) - a multi-dimensional image viewer for python (limited Jupyter support)\n",
    "- [3D Slicer](https://github.com/Slicer/SlicerJupyter) - 3D Slicer is an open source software platform for medical image informatics, image processing, and three-dimensional visualization (limited Jupyter support)\n",
    "\n",
    "#### Client-side rendering\n",
    "\n",
    "- [itk-vtk-viewer](https://kitware.github.io/itk-vtk-viewer/index.html) - ITK/VTK Viewer is an open-source web application for medical and scientific image, mesh, and point set visualization.\n",
    "- [tensorboard-plugin-3d](https://tensorboardplugin3d.readthedocs.io/en/latest/) - TensorBoardPlugin3D is an open-source TensorBoard plugin that supports visualizing 3D output within the TensorBoard application, JupyterLab, Google Colab, and SageMaker.\n",
    "- [ImageJ.js](https://ij.imjoy.io/) - ImageJ running in your web browser via WebAssembly, powered by ImJoy.\n",
    "- [Kaibu](https://kaibu.org/#/app) - Kaibu is a web application for visualizing and annotating multi-dimensional images, built with OpenLayers and itk-vtk-viewer.\n",
    "- [neuroglancer](https://github.com/google/neuroglancer) - Neuroglancer is a WebGL-based viewer for volumetric data.\n",
    "- [webKnossos](https://webknossos.org/) A fully cloud- and browser-based 3D annotation tool for distributed large-scale data analysis in light- and electron-microscopy based Connectomics\n",
    "#### Both\n",
    "\n",
    "- [matplotlib](https://matplotlib.org/) - the defacto Python visualization library, is capable of both server-side and client-side rendering.\n",
    "- *Teaser :-O* - itk-vtk-viewer / [itkwidgets](https://github.com/InsightSoftwareConsortium/itkwidgets) will soon be capable of multiple server-side rendering backends in addition to client-side rendering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Click me:** [3D cell volume visualized with itk-vtk-viewer](https://kitware.github.io/itk-vtk-viewer/app/?image=https://data.kitware.com/api/v1/file/5f499c5a9014a6d84e9b001d/download/C1-retina.nrrd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matplotlib supports multiple rendering backends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available matplotlib backends: ['tk', 'gtk', 'gtk3', 'gtk4', 'wx', 'qt4', 'qt5', 'qt6', 'qt', 'osx', 'nbagg', 'notebook', 'agg', 'svg', 'pdf', 'ps', 'inline', 'ipympl', 'widget']\n"
     ]
    }
   ],
   "source": [
    "%matplotlib -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the following rendering backends:\n",
    "\n",
    "1. `inline`\n",
    "2. `ipympl`\n",
    "3. `widget`\n",
    "4. `nbagg`\n",
    "5. `notebook`\n",
    "\n",
    "- Restart the Jupyter kernel (required when switching backends)\n",
    "- Enable the backend, and plot\n",
    "- Find the browser technology used client-side with the browser debugging tools\n",
    "- Determine if matplotlib is using server-side or client-side rendering.\n",
    "\n",
    "What differences are observed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Currently required to enable running the some matplotlib backends on colab\n",
    "try:\n",
    "    from google.colab import output\n",
    "    if hasattr(output, 'enable_custom_widget_manager'):\n",
    "        output.enable_custom_widget_manager()\n",
    "except ModuleNotFoundError:\n",
    "    pass\n",
    "\n",
    "# Image demo from https://matplotlib.org/3.1.0/gallery/images_contours_and_fields/image_demo.htm\n",
    "def plot_image():\n",
    "    import numpy as np\n",
    "    import matplotlib.cm as cm\n",
    "    import matplotlib.pyplot as plt\n",
    "    import matplotlib.cbook as cbook\n",
    "    from matplotlib.path import Path\n",
    "    from matplotlib.patches import PathPatch\n",
    "    import matplotlib.pyplot as plt\n",
    "    delta = 0.025\n",
    "    x = y = np.arange(-3.0, 3.0, delta)\n",
    "    X, Y = np.meshgrid(x, y)\n",
    "    Z1 = np.exp(-X**2 - Y**2)\n",
    "    Z2 = np.exp(-(X - 1)**2 - (Y - 1)**2)\n",
    "    Z = (Z1 - Z2) * 2\n",
    "\n",
    "    fig, ax = plt.subplots()\n",
    "    im = ax.imshow(Z, interpolation='bilinear', cmap=cm.RdYlGn,\n",
    "               origin='lower', extent=[-3, 3, -3, 3],\n",
    "               vmax=abs(Z).max(), vmin=-abs(Z).max())\n",
    "\n",
    "    plt.show()\n",
    "    \n",
    "%matplotlib inline\n",
    "plot_image()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}