notebooks/403-action-recognition-webcam/403-action-recognition-webcam.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "# Human Action Recognition with OpenVINO™\n",
    "\n",
    "This notebook demonstrates live human action recognition with OpenVINO, using the [Action Recognition Models](https://docs.openvino.ai/2020.2/usergroup13.html) from [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo), specifically an [Encoder](https://docs.openvino.ai/2020.2/_models_intel_action_recognition_0001_encoder_description_action_recognition_0001_encoder.html) and a [Decoder](https://docs.openvino.ai/2020.2/_models_intel_action_recognition_0001_decoder_description_action_recognition_0001_decoder.html). Both models create a sequence to sequence (`\"seq2seq\"`) \\[1\\] system to identify the human activities for [Kinetics-400 dataset](https://deepmind.com/research/open-source/kinetics). The models use the Video Transformer approach with ResNet34 encoder \\[2\\]. The notebook shows how to create the following pipeline:\n",
    "\n",
    "<img align='center' src=\"https://user-images.githubusercontent.com/10940214/148401661-477aebcd-f2d0-4771-b107-4b37f94d0b1e.jpeg\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "Final part of this notebook shows live inference results from a webcam. Additionally, you can also upload a video file.\n",
    "\n",
    "**NOTE**: To use a webcam, you must run this Jupyter notebook on a computer with a webcam. If you run on a server, the webcam will not work. However, you can still do inference on a video in the final step.\n",
    "\n",
    "---\n",
    "\n",
    "\\[1\\] seq2seq: Deep learning models that take a sequence of items to the input and output. In this case, input: video frames, output: actions sequence. This `\"seq2seq\"` is composed of an encoder and a decoder. The encoder captures `\"context\"` of the inputs to be analyzed by the decoder, and finally gets the human action and confidence.\n",
    "\n",
    "\\[2\\] [Video Transformer](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) and [ResNet34](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet34.html).\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "- [Imports](#Imports)\n",
    "- [The models](#The-models)\n",
    "    - [Download the models](#Download-the-models)\n",
    "    - [Load your labels](#Load-your-labels)\n",
    "    - [Load the models](#Load-the-models)\n",
    "        - [Model Initialization function](#Model-Initialization-function)\n",
    "        - [Initialization for Encoder and Decoder](#Initialization-for-Encoder-and-Decoder)\n",
    "    - [Helper functions](#Helper-functions)\n",
    "    - [AI Functions](#AI-Functions)\n",
    "    - [Main Processing Function](#Main-Processing-Function)\n",
    "    - [Run Action Recognition](#Run-Action-Recognition)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q \"openvino-dev>=2023.1.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Imports\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "import os\n",
    "import time\n",
    "from typing import Tuple, List\n",
    "\n",
    "import cv2\n",
    "import numpy as np\n",
    "from IPython import display\n",
    "import openvino as ov\n",
    "from openvino.runtime.ie_api import CompiledModel\n",
    "\n",
    "# Fetch `notebook_utils` module\n",
    "import urllib.request\n",
    "urllib.request.urlretrieve(\n",
    "    url='https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py',\n",
    "    filename='notebook_utils.py'\n",
    ")\n",
    "import notebook_utils as utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "### Download the models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Use `omz_downloader`, which is a command-line tool from the `openvino-dev` package. It automatically creates a directory structure and downloads the selected model.\n",
    "\n",
    "In this case you can use `\"action-recognition-0001\"` as a model name, and the system automatically downloads the two models `\"action-recognition-0001-encoder\"` and `\"action-recognition-0001-decoder\"`\n",
    "\n",
    "> **NOTE**: If you want to download another model, such as `\"driver-action-recognition-adas-0002\"` (`\"driver-action-recognition-adas-0002-encoder\"` + `\"driver-action-recognition-adas-0002-decoder\"`), replace the name of the model in the code below. Using a model outside the list can require different pre- and post-processing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# A directory where the model will be downloaded.\n",
    "base_model_dir = \"model\"\n",
    "# The name of the model from Open Model Zoo.\n",
    "model_name = \"action-recognition-0001\"\n",
    "# Selected precision (FP32, FP16, FP16-INT8).\n",
    "precision = \"FP16\"\n",
    "model_path_decoder = (\n",
    "    f\"model/intel/{model_name}/{model_name}-decoder/{precision}/{model_name}-decoder.xml\"\n",
    ")\n",
    "model_path_encoder = (\n",
    "    f\"model/intel/{model_name}/{model_name}-encoder/{precision}/{model_name}-encoder.xml\"\n",
    ")\n",
    "if not os.path.exists(model_path_decoder) or not os.path.exists(model_path_encoder):\n",
    "    download_command = f\"omz_downloader \" \\\n",
    "                       f\"--name {model_name} \" \\\n",
    "                       f\"--precision {precision} \" \\\n",
    "                       f\"--output_dir {base_model_dir}\"\n",
    "    ! $download_command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load your labels\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "This tutorial uses [Kinetics-400 dataset](https://deepmind.com/research/open-source/kinetics), and also provides the text file embedded into this notebook. \n",
    "\n",
    "> **NOTE**: If you want to run `\"driver-action-recognition-adas-0002\"` model, replace the `kinetics.txt` file to `driver_actions.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the text from the openvino_notebooks storage\n",
    "vocab_file_path = utils.download_file(\n",
    "    \"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/text/kinetics.txt\",\n",
    "    directory=\"data\"\n",
    ")\n",
    "\n",
    "with vocab_file_path.open(mode='r') as f:\n",
    "    labels = [line.strip() for line in f]\n",
    "\n",
    "print(labels[0:9], np.shape(labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Load the models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Load the two models for this particular architecture, Encoder and Decoder. Downloaded models are located in a fixed structure, indicating a vendor, the name of the model, and a precision.\n",
    "\n",
    " 1. Initialize OpenVINO Runtime.\n",
    " 2. Read the network from `*.bin` and `*.xml` files (weights and architecture).\n",
    " 3. Compile the model for specified device.\n",
    " 4. Get input and output names of nodes.\n",
    "\n",
    "Only a few lines of code are required to run the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select device from dropdown list for running inference using OpenVINO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "core = ov.Core()\n",
    "device = widgets.Dropdown(\n",
    "    options=core.available_devices + [\"AUTO\"],\n",
    "    value='AUTO',\n",
    "    description='Device:',\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Initialization function\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize OpenVINO Runtime.\n",
    "core = ov.Core()\n",
    "\n",
    "\n",
    "def model_init(model_path: str, device: str) -> Tuple:\n",
    "    \"\"\"\n",
    "    Read the network and weights from a file, load the\n",
    "    model on CPU and get input and output names of nodes\n",
    "\n",
    "    :param: \n",
    "            model: model architecture path *.xml\n",
    "            device: inference device\n",
    "    :retuns:\n",
    "            compiled_model: Compiled model \n",
    "            input_key: Input node for model\n",
    "            output_key: Output node for model\n",
    "    \"\"\"\n",
    "\n",
    "    # Read the network and corresponding weights from a file.\n",
    "    model = core.read_model(model=model_path)\n",
    "    # Compile the model for specified device.\n",
    "    compiled_model = core.compile_model(model=model, device_name=device)\n",
    "    # Get input and output names of nodes.\n",
    "    input_keys = compiled_model.input(0)\n",
    "    output_keys = compiled_model.output(0)\n",
    "    return input_keys, output_keys, compiled_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Initialization for Encoder and Decoder\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encoder initialization\n",
    "input_key_en, output_keys_en, compiled_model_en = model_init(model_path_encoder, device.value)\n",
    "# Decoder initialization\n",
    "input_key_de, output_keys_de, compiled_model_de = model_init(model_path_decoder, device.value)\n",
    "\n",
    "# Get input size - Encoder.\n",
    "height_en, width_en = list(input_key_en.shape)[2:]\n",
    "# Get input size - Decoder.\n",
    "frames2decode = list(input_key_de.shape)[0:][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper functions\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Use the following helper functions for preprocessing and postprocessing frames:\n",
    "\n",
    "1. Preprocess the input image before running the Encoder model. (`center_crop` and `adaptative_resize`)\n",
    "2. Decode top-3 probabilities into label names. (`decode_output`)\n",
    "3. Draw the Region of Interest (ROI) over the video. (`rec_frame_display`)\n",
    "4. Prepare the frame for displaying label names over the video. (`display_text_fnc`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def center_crop(frame: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Center crop squared the original frame to standardize the input image to the encoder model\n",
    "\n",
    "    :param frame: input frame\n",
    "    :returns: center-crop-squared frame\n",
    "    \"\"\"\n",
    "    img_h, img_w, _ = frame.shape\n",
    "    min_dim = min(img_h, img_w)\n",
    "    start_x = int((img_w - min_dim) / 2.0)\n",
    "    start_y = int((img_h - min_dim) / 2.0)\n",
    "    roi = [start_y, (start_y + min_dim), start_x, (start_x + min_dim)]\n",
    "    return frame[start_y : (start_y + min_dim), start_x : (start_x + min_dim), ...], roi\n",
    "\n",
    "\n",
    "def adaptive_resize(frame: np.ndarray, size: int) -> np.ndarray:\n",
    "    \"\"\"\n",
    "     The frame going to be resized to have a height of size or a width of size\n",
    "\n",
    "    :param frame: input frame\n",
    "    :param size: input size to encoder model\n",
    "    :returns: resized frame, np.array type\n",
    "    \"\"\"\n",
    "    h, w, _ = frame.shape\n",
    "    scale = size / min(h, w)\n",
    "    w_scaled, h_scaled = int(w * scale), int(h * scale)\n",
    "    if w_scaled == w and h_scaled == h:\n",
    "        return frame\n",
    "    return cv2.resize(frame, (w_scaled, h_scaled))\n",
    "\n",
    "\n",
    "def decode_output(probs: np.ndarray, labels: np.ndarray, top_k: int = 3) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Decodes top probabilities into corresponding label names\n",
    "\n",
    "    :param probs: confidence vector for 400 actions\n",
    "    :param labels: list of actions\n",
    "    :param top_k: The k most probable positions in the list of labels\n",
    "    :returns: decoded_labels: The k most probable actions from the labels list\n",
    "              decoded_top_probs: confidence for the k most probable actions\n",
    "    \"\"\"\n",
    "    top_ind = np.argsort(-1 * probs)[:top_k]\n",
    "    out_label = np.array(labels)[top_ind.astype(int)]\n",
    "    decoded_labels = [out_label[0][0], out_label[0][1], out_label[0][2]]\n",
    "    top_probs = np.array(probs)[0][top_ind.astype(int)]\n",
    "    decoded_top_probs = [top_probs[0][0], top_probs[0][1], top_probs[0][2]]\n",
    "    return decoded_labels, decoded_top_probs\n",
    "\n",
    "\n",
    "def rec_frame_display(frame: np.ndarray, roi) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Draw a rec frame over actual frame\n",
    "\n",
    "    :param frame: input frame\n",
    "    :param roi: Region of interest, image section processed by the Encoder\n",
    "    :returns: frame with drawed shape\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    cv2.line(frame, (roi[2] + 3, roi[0] + 3), (roi[2] + 3, roi[0] + 100), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[2] + 3, roi[0] + 3), (roi[2] + 100, roi[0] + 3), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[3] - 3, roi[1] - 3), (roi[3] - 3, roi[1] - 100), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[3] - 3, roi[1] - 3), (roi[3] - 100, roi[1] - 3), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[3] - 3, roi[0] + 3), (roi[3] - 3, roi[0] + 100), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[3] - 3, roi[0] + 3), (roi[3] - 100, roi[0] + 3), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[2] + 3, roi[1] - 3), (roi[2] + 3, roi[1] - 100), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[2] + 3, roi[1] - 3), (roi[2] + 100, roi[1] - 3), (0, 200, 0), 2)\n",
    "    # Write ROI over actual frame\n",
    "    FONT_STYLE = cv2.FONT_HERSHEY_SIMPLEX\n",
    "    org = (roi[2] + 3, roi[1] - 3)\n",
    "    org2 = (roi[2] + 2, roi[1] - 2)\n",
    "    FONT_SIZE = 0.5\n",
    "    FONT_COLOR = (0, 200, 0)\n",
    "    FONT_COLOR2 = (0, 0, 0)\n",
    "    cv2.putText(frame, \"ROI\", org2, FONT_STYLE, FONT_SIZE, FONT_COLOR2)\n",
    "    cv2.putText(frame, \"ROI\", org, FONT_STYLE, FONT_SIZE, FONT_COLOR)\n",
    "    return frame\n",
    "\n",
    "\n",
    "def display_text_fnc(frame: np.ndarray, display_text: str, index: int):\n",
    "    \"\"\"\n",
    "    Include a text on the analyzed frame\n",
    "\n",
    "    :param frame: input frame\n",
    "    :param display_text: text to add on the frame\n",
    "    :param index: index line dor adding text\n",
    "\n",
    "    \"\"\"\n",
    "    # Configuration for displaying images with text.\n",
    "    FONT_COLOR = (255, 255, 255)\n",
    "    FONT_COLOR2 = (0, 0, 0)\n",
    "    FONT_STYLE = cv2.FONT_HERSHEY_DUPLEX\n",
    "    FONT_SIZE = 0.7\n",
    "    TEXT_VERTICAL_INTERVAL = 25\n",
    "    TEXT_LEFT_MARGIN = 15\n",
    "    # ROI over actual frame\n",
    "    (processed, roi) = center_crop(frame)\n",
    "    # Draw a ROI over actual frame.\n",
    "    frame = rec_frame_display(frame, roi)\n",
    "    # Put a text over actual frame.\n",
    "    text_loc = (TEXT_LEFT_MARGIN, TEXT_VERTICAL_INTERVAL * (index + 1))\n",
    "    text_loc2 = (TEXT_LEFT_MARGIN + 1, TEXT_VERTICAL_INTERVAL * (index + 1) + 1)\n",
    "    cv2.putText(frame, display_text, text_loc2, FONT_STYLE, FONT_SIZE, FONT_COLOR2)\n",
    "    cv2.putText(frame, display_text, text_loc, FONT_STYLE, FONT_SIZE, FONT_COLOR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AI Functions\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "<img align='center' src=\"https://user-images.githubusercontent.com/10940214/148401661-477aebcd-f2d0-4771-b107-4b37f94d0b1e.jpeg\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "Following the pipeline above, you will use the next functions to:\n",
    "\n",
    "1. Preprocess a frame before running the Encoder. (`preprocessing`)\n",
    "2. Encoder Inference per frame. (`encoder`)\n",
    "3. Decoder inference per set of frames. (`decoder`)\n",
    "4. Normalize the Decoder output to get confidence values per action recognition label. (`softmax`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessing(frame: np.ndarray, size: int) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Preparing frame before Encoder.\n",
    "    The image should be scaled to its shortest dimension at \"size\"\n",
    "    and cropped, centered, and squared so that both width and\n",
    "    height have lengths \"size\". The frame must be transposed from\n",
    "    Height-Width-Channels (HWC) to Channels-Height-Width (CHW).\n",
    "\n",
    "    :param frame: input frame\n",
    "    :param size: input size to encoder model\n",
    "    :returns: resized and cropped frame\n",
    "    \"\"\"\n",
    "    # Adaptative resize\n",
    "    preprocessed = adaptive_resize(frame, size)\n",
    "    # Center_crop\n",
    "    (preprocessed, roi) = center_crop(preprocessed)\n",
    "    # Transpose frame HWC -> CHW\n",
    "    preprocessed = preprocessed.transpose((2, 0, 1))[None,]  # HWC -> CHW\n",
    "    return preprocessed, roi\n",
    "\n",
    "\n",
    "def encoder(\n",
    "    preprocessed: np.ndarray,\n",
    "    compiled_model: CompiledModel\n",
    ") -> List:\n",
    "    \"\"\"\n",
    "    Encoder Inference per frame. This function calls the network previously\n",
    "    configured for the encoder model (compiled_model), extracts the data\n",
    "    from the output node, and appends it in an array to be used by the decoder.\n",
    "\n",
    "    :param: preprocessed: preprocessing frame\n",
    "    :param: compiled_model: Encoder model network\n",
    "    :returns: encoder_output: embedding layer that is appended with each arriving frame\n",
    "    \"\"\"\n",
    "    output_key_en = compiled_model.output(0)\n",
    "    \n",
    "    # Get results on action-recognition-0001-encoder model\n",
    "    infer_result_encoder = compiled_model([preprocessed])[output_key_en]\n",
    "    return infer_result_encoder\n",
    "\n",
    "\n",
    "def decoder(encoder_output: List, compiled_model_de: CompiledModel) -> List:\n",
    "    \"\"\"\n",
    "    Decoder inference per set of frames. This function concatenates the embedding layer\n",
    "    froms the encoder output, transpose the array to match with the decoder input size.\n",
    "    Calls the network previously configured for the decoder model (compiled_model_de), extracts\n",
    "    the logits and normalize those to get confidence values along specified axis.\n",
    "    Decodes top probabilities into corresponding label names\n",
    "\n",
    "    :param: encoder_output: embedding layer for 16 frames\n",
    "    :param: compiled_model_de: Decoder model network\n",
    "    :returns: decoded_labels: The k most probable actions from the labels list\n",
    "              decoded_top_probs: confidence for the k most probable actions\n",
    "    \"\"\"\n",
    "    # Concatenate sample_duration frames in just one array\n",
    "    decoder_input = np.concatenate(encoder_output, axis=0)\n",
    "    # Organize input shape vector to the Decoder (shape: [1x16x512]]\n",
    "    decoder_input = decoder_input.transpose((2, 0, 1, 3))\n",
    "    decoder_input = np.squeeze(decoder_input, axis=3)\n",
    "    output_key_de = compiled_model_de.output(0)\n",
    "    # Get results on action-recognition-0001-decoder model\n",
    "    result_de = compiled_model_de([decoder_input])[output_key_de]\n",
    "    # Normalize logits to get confidence values along specified axis\n",
    "    probs = softmax(result_de - np.max(result_de))\n",
    "    # Decodes top probabilities into corresponding label names\n",
    "    decoded_labels, decoded_top_probs = decode_output(probs, labels, top_k=3)\n",
    "    return decoded_labels, decoded_top_probs\n",
    "\n",
    "\n",
    "def softmax(x: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Normalizes logits to get confidence values along specified axis\n",
    "    x: np.array, axis=None\n",
    "    \"\"\"\n",
    "    exp = np.exp(x)\n",
    "    return exp / np.sum(exp, axis=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "### Main Processing Function\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Running action recognition function will run in different operations, either a webcam or a video file. See the list of procedures below:\n",
    "\n",
    "1. Create a video player to play with target fps (`utils.VideoPlayer`).\n",
    "2. Prepare a set of frames to be encoded-decoded.\n",
    "3. Run AI functions\n",
    "4. Visualize the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def run_action_recognition(\n",
    "    source: str = \"0\",\n",
    "    flip: bool = True,\n",
    "    use_popup: bool = False,\n",
    "    compiled_model_en: CompiledModel = compiled_model_en,\n",
    "    compiled_model_de: CompiledModel = compiled_model_de,\n",
    "    skip_first_frames: int = 0,\n",
    "):\n",
    "    \"\"\"\n",
    "    Use the \"source\" webcam or video file to run the complete pipeline for action-recognition problem\n",
    "    1. Create a video player to play with target fps\n",
    "    2. Prepare a set of frames to be encoded-decoded\n",
    "    3. Preprocess frame before Encoder\n",
    "    4. Encoder Inference per frame\n",
    "    5. Decoder inference per set of frames\n",
    "    6. Visualize the results\n",
    "\n",
    "    :param: source: webcam \"0\" or video path\n",
    "    :param: flip: to be used by VideoPlayer function for flipping capture image\n",
    "    :param: use_popup: False for showing encoded frames over this notebook, True for creating a popup window.\n",
    "    :param: skip_first_frames: Number of frames to skip at the beginning of the video.\n",
    "    :returns: display video over the notebook or in a popup window\n",
    "\n",
    "    \"\"\"\n",
    "    size = height_en  # Endoder input size - From Cell 5_9\n",
    "    sample_duration = frames2decode  # Decoder input size - From Cell 5_7\n",
    "    # Select frames per second of your source.\n",
    "    fps = 30\n",
    "    player = None\n",
    "    try:\n",
    "        # Create a video player.\n",
    "        player = utils.VideoPlayer(source, flip=flip, fps=fps, skip_first_frames=skip_first_frames)\n",
    "        # Start capturing.\n",
    "        player.start()\n",
    "        if use_popup:\n",
    "            title = \"Press ESC to Exit\"\n",
    "            cv2.namedWindow(title, cv2.WINDOW_GUI_NORMAL | cv2.WINDOW_AUTOSIZE)\n",
    "\n",
    "        processing_times = collections.deque()\n",
    "        processing_time = 0\n",
    "        encoder_output = []\n",
    "        decoded_labels = [0, 0, 0]\n",
    "        decoded_top_probs = [0, 0, 0]\n",
    "        counter = 0\n",
    "        # Create a text template to show inference results over video.\n",
    "        text_inference_template = \"Infer Time:{Time:.1f}ms,{fps:.1f}FPS\"\n",
    "        text_template = \"{label},{conf:.2f}%\"\n",
    "\n",
    "        while True:\n",
    "            counter = counter + 1\n",
    "\n",
    "            # Read a frame from the video stream.\n",
    "            frame = player.next()\n",
    "            if frame is None:\n",
    "                print(\"Source ended\")\n",
    "                break\n",
    "\n",
    "            scale = 1280 / max(frame.shape)\n",
    "\n",
    "            # Adaptative resize for visualization.\n",
    "            if scale < 1:\n",
    "                frame = cv2.resize(frame, None, fx=scale, fy=scale, interpolation=cv2.INTER_AREA)\n",
    "\n",
    "            # Select one frame every two for processing through the encoder.\n",
    "            # After 16 frames are processed, the decoder will find the action,\n",
    "            # and the label will be printed over the frames.\n",
    "\n",
    "            if counter % 2 == 0:\n",
    "                # Preprocess frame before Encoder.\n",
    "                (preprocessed, _) = preprocessing(frame, size)\n",
    "\n",
    "                # Measure processing time.\n",
    "                start_time = time.time()\n",
    "\n",
    "                # Encoder Inference per frame\n",
    "                encoder_output.append(encoder(preprocessed, compiled_model_en))\n",
    "\n",
    "                # Decoder inference per set of frames\n",
    "                # Wait for sample duration to work with decoder model.\n",
    "                if len(encoder_output) == sample_duration:\n",
    "                    decoded_labels, decoded_top_probs = decoder(encoder_output, compiled_model_de)\n",
    "                    encoder_output = []\n",
    "\n",
    "                # Inference has finished. Display the results.\n",
    "                stop_time = time.time()\n",
    "\n",
    "                # Calculate processing time.\n",
    "                processing_times.append(stop_time - start_time)\n",
    "\n",
    "                # Use processing times from last 200 frames.\n",
    "                if len(processing_times) > 200:\n",
    "                    processing_times.popleft()\n",
    "\n",
    "                # Mean processing time [ms]\n",
    "                processing_time = np.mean(processing_times) * 1000\n",
    "                fps = 1000 / processing_time\n",
    "\n",
    "            # Visualize the results.\n",
    "            for i in range(0, 3):\n",
    "                display_text = text_template.format(\n",
    "                    label=decoded_labels[i],\n",
    "                    conf=decoded_top_probs[i] * 100,\n",
    "                )\n",
    "                display_text_fnc(frame, display_text, i)\n",
    "\n",
    "            display_text = text_inference_template.format(Time=processing_time, fps=fps)\n",
    "            display_text_fnc(frame, display_text, 3)\n",
    "\n",
    "            # Use this workaround if you experience flickering.\n",
    "            if use_popup:\n",
    "                cv2.imshow(title, frame)\n",
    "                key = cv2.waitKey(1)\n",
    "                # escape = 27\n",
    "                if key == 27:\n",
    "                    break\n",
    "            else:\n",
    "                # Encode numpy array to jpg.\n",
    "                _, encoded_img = cv2.imencode(\".jpg\", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])\n",
    "                # Create an IPython image.\n",
    "                i = display.Image(data=encoded_img)\n",
    "                # Display the image in this notebook.\n",
    "                display.clear_output(wait=True)\n",
    "                display.display(i)\n",
    "\n",
    "    # ctrl-c\n",
    "    except KeyboardInterrupt:\n",
    "        print(\"Interrupted\")\n",
    "    # Any different error\n",
    "    except RuntimeError as e:\n",
    "        print(e)\n",
    "    finally:\n",
    "        if player is not None:\n",
    "            # Stop capturing.\n",
    "            player.stop()\n",
    "        if use_popup:\n",
    "            cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Action Recognition\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Find out how the model works in a video file. [Any format supported](https://docs.opencv.org/4.5.1/dd/d43/tutorial_py_video_display.html) by OpenCV will work. You can press the stop button anytime while the video file is running, and it will activate the webcam for the next step.\n",
    "\n",
    "> **NOTE**: Sometimes, the video can be cut off if there are corrupted frames. In that case, you can convert it. If you experience any problems with your video, use the [HandBrake](https://handbrake.fr/) and select the MPEG format.\n",
    "\n",
    "if you want to use a web camera as an input source for the demo, please change the value of `USE_WEBCAM` variable to True and specify `cam_id` (the default value is 0, which can be different in multi-camera systems)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "additional_options = {\"skip_first_frames\": 600, \"flip\": False} if not USE_WEBCAM else {\"flip\": True}\n": "additional_options = {\"skip_first_frames\": 1800, \"flip\": False} if not USE_WEBCAM else {\"flip\": True}\n"
    }
   },
   "outputs": [],
   "source": [
    "USE_WEBCAM = False\n",
    "\n",
    "cam_id = 0\n",
    "video_file = \"https://archive.org/serve/ISSVideoResourceLifeOnStation720p/ISS%20Video%20Resource_LifeOnStation_720p.mp4\"\n",
    "\n",
    "source = cam_id if USE_WEBCAM else video_file\n",
    "additional_options = {\"skip_first_frames\": 600, \"flip\": False} if not USE_WEBCAM else {\"flip\": True}\n",
    "run_action_recognition(source=source, use_popup=False, **additional_options)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}