Machine Learning Project 2 : Multi-object detection and tracking

Please note that the repo contains all the data that we used (lots of .jpg files and a small .mp4 video), and weighs around 800MB. Hence the cloning may take a few minutes to complete.

Overview of files

• In the top-level directory, you will find the different scripts that we wrote for this project :
  • create_tfrecord.py: A program that generates TFRecords from MOT Challenge data.
  • create_config.py: A program that modifies a pre-trained model's default pipeline.config, in order to adapt it to our own model.
  • demo_ssd_mosse.py: A demo program that runs SSD detection with MOSSE tracking on a given input video.
  • demo_ssd_deepsort.py: A demo program that runs SSD detection with DeepSORT tracking on a given input video.
  • demo_yolo_kcf.py: A demo program that runs YOLOv3 detection with KCF tracking on a given input video.
• The tensorflow-models package contains the code of the TensorFlow Object Detection API.
• The deep_sort package contains the official implementation of DeepSORT tracking from Nicolas Wolkje and modified by pythonlessons for Tensorflow V2 compatibility, and the use of their trained DeepSORT model.
• The images folder contains train and test videos from MOT Challenge. For each video the frames are located in the subfolder img1 and the detections file det.txt in the subfolder det.
• The training folder contains the data we need for training the model.
  • The folder pre-trained-models contains the pre-trained MobileNet SSD model. Others pre-trained models can be downloaded from the Tensorflow Model Zoo.
  • The tf-records folder contains our single-entry label map, and is intended to contain the TFRecords generated by create_tfrecord.py.
  • The trained-model folder contains the pipeline.config created by create_config.py, along with the checkpoint files. Initially, we included a checkpoint-6 from a training that we did ourselves on Google Colab, but you are free to empty this whole folder and redo the entire training by following the Training section.
• The yolov3_files folder contains the files needed to run demo_yolo_kcf.py, except for the yolov3 weights that can be downloaded at this link.
• People.mp4 can be used for the demo.
• Project-2-ML.ipynb is a notebook you can run on Google Colab. All our work can be simply reproduced with this notebook.

Our .py scripts are documented and indications about the arguments are present in the respective files.

Installation :

If you are not running on macOS, some steps of the following configuration may fail. If you encounter problems or simply want to avoid all the configuration on your computer, you can simply run our Google Colab notebook available at this link, and go through the steps. We also provided a copy of this notebook in this repo, as "Project_2_ML.ipynb". The notebook clones and uses a github repo that is an exact copy of this one, but that we set to public (cloning private repos on Colab is quite tricky).

Furthermore, Google Colaboratory allows to use powerful CPUs and GPUs, which makes the training and demos faster. If you run the notebook on Google Colab, in the menu, go in Edit > Notebook settings, and select GPU in the "Hardware accelerator" field.

Note that in order to use GPUs from your own device when running the code in local, some extra setup is required for GPU support, which we did not describe here.

You will need to run the protocol buffer (protobuf) compiler protoc to generate some python files from the Tensorflow Object Detection API.

brew install protobuf

Create a new virtual environment with anaconda.

conda create -n ML_Project2_MOT pip python=3.8
conda activate ML_Project2_MOT

Install Tensorflow :

pip install tensorflow

Install everything else that is required by the object-detection API. Please start by changing your directory to tensorflow_models/research :

cd tensorflow_models/research
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
pip install .
pip install opencv-contrib-python -U

You can test the installation with the following command (still from the tensorflow-models/research directory).

python object_detection/builders/model_builder_tf2_test.py

After a little while, it should output something like the following :

----------------------------------------------------------------------
Ran 20 tests in 68.510s

OK (skipped=1)

WARNING : cv2.MultiTracker_create() will be used for tracking, and is present only in the package opencv-contrib-python (not in opencv-python, that may be installed as well by default). If an error occurs while calling MultiTracker_create(), uninstall opencv-python and opencv-contrib-python (if it was already installed), and reinstall opencv-contrib-python only.

pip uninstall opencv-python
pip uninstall opencv-contrib-python
pip install opencv-contrib-python

Training

We already included in the repo a trained model (see ckpt-6 in the /training/trained-model directory). If you want to run the demo directly, you can skip this section and go to the Demo section.

If you want to run the training, please start by deleting every file under training/trained-model.

First, we need to convert the train and test data provided by MOT challenge into TFRecords. Please change your directory to the top-level directory (where our scripts are located).

python create_tfrecord.py images/train/ training/TFRecords/train.record training/TFRecords/label_map.pbtxt
python create_tfrecord.py images/train/ training/TFRecords/test.record training/TFRecords/label_map.pbtxt

Then, we need to create a customized pipeline.config file for our specific training. We will take the default config of the pre-trained model and modify it adequately. The batch size is 4 by default, you can set it to a higher value (e.g. 8 or 16) with flag -b if you are using Google Colab with GPUs.

python create_config.py training/pre-trained-models/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8 training/TFRecords/label_map.pbtxt training/TFRecords training/trained-model

Now we are ready to run the training. This script is provided by the Tensorflow object detection API. Depending on the number of steps that you specify (in the following it is set to 5000), and the batch size (specified in the pipeline configuration above), the training may take some time to complete, especially if you are using only CPU. The program will first print some warnings that can be ignored, and after a little while, it will start to print the progression, at every 100 training steps.

python tensorflow_models/research/object_detection/model_main_tf2.py --model_dir training/trained-model/ --pipeline_config_path training/trained-model/pipeline.config --num_train_steps 5000

Note that reducing the number of steps would of course make the model less accurate, and conversely, too high a value could result in overfitting. After several tries, we estimated that 5000 steps was a good bet.

Demo

As for the training part, you need to be located in the top-level directory (where our scripts are located). In order to test our trained detection model, with either MOSSE or DeepSORT tracking, you can simply run one of the two following scripts. The scripts take as input either a .mp4 file or a folder containing .jpg files. You have the choice to save the results with the -o flag and/or to visualize them in real time with the -d flag (please note that the latter does not work on Google Colab). Some other options are documented inside the scripts.

SSD detection with MOSSE tracking (this script takes in addition, as argument, the trained-model directory) :

# save the results in an empty results/ directory :
rm -r results; mkdir results
python demo_ssd_mosse.py People.mp4 training/trained-model/ -o results/

# display the images in real time. Does not work on Google Colab !
python demo_ssd_mosse.py People.mp4 training/trained-model/ -d

SSD detection with DeepSORT tracking (this scrit takes in addition, as argument, the path to a DeepSORT model that we downloaded from the pythonlessons repo) :

# save the results in an empty results/ directory :
rm -r results; mkdir results
python demo_ssd_deepsort.py People.mp4 training/trained-model/ deep_sort/mars-small128.pb -o results/

# display the images in real time. Does not work on Google Colab !
python demo_ssd_deepsort.py People.mp4 training/trained-model/ deep_sort/mars-small128.pb -d

Note that with this version, some "proper tracking" is performed (keeping record of each track's id, i.e. "which person goes where"), and it may take a few frames for a track to be considered as confirmed. This is why no boxes will appear in the two first frames.

Finally, you can also have a look at a YOLO v3 demo (with KCF tracking), that we had previously written in order to have a first glimpse at object detection, and that we used later to make comparisons with our own SSD model. In order to run this script, you will need first to download the weights yolov3-spp.weights from pjreddie.com (the file weighs about 200MB, which was too big to put inside this repo) and put the file inside the folder yolov3-files :

# save the results in an empty results/ directory :
rm -r results; mkdir results
python demo_yolo_kcf.py People.mp4 -o results/

# display the images in real time. Does not work on Google Colab !
python demo_yolo_kcf.py People.mp4 -d