Project_2_ML.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Project-2-ML.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rf1noKBg5BRo"
      },
      "source": [
        "# Machine Learning Project 2 : Multi-object detection and tracking"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4c_V0grXn43Q"
      },
      "source": [
        "## Installation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_eUQs1yfPbaO"
      },
      "source": [
        "First, copy the needed files from github :\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V5QwW5-EKXVZ"
      },
      "source": [
        "!git clone https://github.com/clempetit/ML-project-2\n",
        "!mv /content/ML-project-2/* /content/\n",
        "!rm -r ML-project-2/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkBidGIEswqL"
      },
      "source": [
        "Then some additional packages need to be installed :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2L2JTS1mBn-u"
      },
      "source": [
        "%cd /content/tensorflow_models/research\n",
        "\n",
        "!protoc object_detection/protos/*.proto --python_out=.\n",
        "\n",
        "!cp object_detection/packages/tf2/setup.py .\n",
        "!pip install .\n",
        "!pip install opencv-contrib-python -U"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a4V48rAKasqc"
      },
      "source": [
        "Test Installation (still in tensorflow_models/research) :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L9X6jSttP1fG"
      },
      "source": [
        "!python object_detection/builders/model_builder_tf2_test.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TR4ziRYCRISI"
      },
      "source": [
        "Test GPU support :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "84yj6uSlRKsH"
      },
      "source": [
        "!python -c \"import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Re8iSiFda6po"
      },
      "source": [
        "We also define some useful paths to make the following commands lighter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0LZikyMY5QWS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "736b7ed0-d534-480a-d031-2370e34a5d01"
      },
      "source": [
        "%cd /content\n",
        "TRAINING_PATH = 'training'\n",
        "IMAGE_PATH = 'images'\n",
        "TRAINED_MODEL_PATH = TRAINING_PATH+'/trained-model'\n",
        "PRETRAINED_MODEL_PATH = TRAINING_PATH+'/pre-trained-models'\n",
        "TF_RECORD_PATH = TRAINING_PATH+'/TFRecords'\n",
        "LABEL_MAP_PATH = TF_RECORD_PATH + '/label_map.pbtxt'\n",
        "RESULT_PATH = 'results/'"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mhXVt3JNnwNo"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mt-8zCquvbtt"
      },
      "source": [
        "IMPORTANT : We already provided the backup of a training (see inside the `training/trained-model` folder). If you choose to redo the training, please first delete all the files inside this `training/trained-model` folder. Otherwise, you can skip this section, and go directly to the \"Demo\" section."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z6JpafLQ8W23"
      },
      "source": [
        "Create the TFRecords, containing images and detection data for the training :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J0FX7RrRH2Z2"
      },
      "source": [
        "!python create_tfrecord.py {IMAGE_PATH + '/train'} {TF_RECORD_PATH + '/train.record'} {TF_RECORD_PATH + '/label_map.pbtxt'}\n",
        "!python create_tfrecord.py {IMAGE_PATH + '/test'} {TF_RECORD_PATH + '/test.record'} {TF_RECORD_PATH + '/label_map.pbtxt'}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "op9NiqeXtCpf"
      },
      "source": [
        "Modify the model config for our case :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ZMK_3FGaC0z"
      },
      "source": [
        "!python create_config.py {PRETRAINED_MODEL_PATH + \"/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8\"} {LABEL_MAP_PATH} {TF_RECORD_PATH} {TRAINED_MODEL_PATH} -b 8"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b9i8_A-rvqGN"
      },
      "source": [
        "Train the model (expected to take around 30 minutes to complete with batch size set to 8 in the pipeline config, and with 5000 steps) :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PScZbn7__tPC"
      },
      "source": [
        "!python tensorflow_models/research/object_detection/model_main_tf2.py --model_dir={TRAINED_MODEL_PATH} --pipeline_config_path={TRAINED_MODEL_PATH + '/pipeline.config'} --num_train_steps=5000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-wCPaR5-nz-x"
      },
      "source": [
        "## Demo"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eGJcoR8MAP0k"
      },
      "source": [
        "Run the demo with our SSD model and MOSSE tracking. Here the detection is performed once every 10 frames by default (frequency can be changed with -dt flag) for faster computation. Other frames rely on tracking."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bC4aIz6V_qhY"
      },
      "source": [
        "!rm -r results; mkdir results\n",
        "!python demo_ssd_mosse.py People.mp4 {TRAINED_MODEL_PATH} -o {RESULT_PATH}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZHQHR8FgbxS"
      },
      "source": [
        "Save the results in a zip file (you can then download this zip, which is located in the /content/ directory)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "98vhLEK-TvfL"
      },
      "source": [
        "!zip -r results.zip {RESULT_PATH}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39xqS4TwjpTw"
      },
      "source": [
        "Run the demo with our SSD model and DeepSORT tracking. This tracker allows to accurately keep track of specific objects but detection needs to be performed at every frame, so therefore computation is slower than the first demo as here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oKhQufFfifaH"
      },
      "source": [
        "!rm -r results; mkdir results\n",
        "!python demo_ssd_deepsort.py People.mp4 {TRAINED_MODEL_PATH} deep_sort/mars-small128.pb -o {RESULT_PATH}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7z2mwvTOijyn"
      },
      "source": [
        "!zip -r results.zip {RESULT_PATH}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8IRu0XhZr5Ev"
      },
      "source": [
        "In addition, you can also have a look at a YOLO v3 demo (with KCF tracking), that we had previously written in order to have a first glimpse at object detection, and that we used later to make comparisons with our own SSD model. In order to run this script, you will need first to download the weights [yolov3-spp.weights](https://pjreddie.com/media/files/yolov3-spp.weights) from pjreddie.com (the file weighs about 200MB, which was too big to put inside the github repo) and put the file inside the `yolov3-files` folder (to upload a file, click on the three dots visible when you hover above the directory, and then select \"upload\". The file weighs around 200MB, therefore the upload may take long, sorry for the inconvenience) :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1TOYTY6wsC7I"
      },
      "source": [
        "!rm -r results; mkdir results\n",
        "!python demo_yolo_kcf.py People.mp4 -o {RESULT_PATH}"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}