This repo contains the official implementation of the CVPR 2022 paper:
MTTR_preview.mp4
3/3/2022
We are excited to announce that our paper was accepted for publication at CVPR 2022! 🥳🥳🥳
The paper can be accessed here.
8/12/2021
We listened to your requests and now release interactive demonstrations of MTTR on Google Colab and Hugging Face Spaces! 🚀 🤗
We currently recommend using the Colab version of the demonstration as it is a lot faster (GPU accelerated) and has more options. The Spaces demo on the other hand has a nicer interface but is currently much slower since it runs on CPU.
Enjoy! :)
First, clone this repo to your local machine using:
git clone https://github.com/mttr2021/MTTR.git
The code was tested on a Conda environment installed on Ubuntu 18.04. Install Conda and then create an environment as follows:
conda create -n mttr python=3.9.7 pip -y
conda activate mttr
- Pytorch 1.10:
conda install pytorch==1.10.0 torchvision==0.11.1 -c pytorch -c conda-forge
Note that you might have to change the cudatoolkit version above according to your system's CUDA version.
- Hugging Face transformers 4.11.3:
pip install transformers==4.11.3
- COCO API (for mAP calculations):
pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
- Additional required packages:
pip install h5py wandb opencv-python protobuf av einops ruamel.yaml timm joblib
conda install -c conda-forge pandas matplotlib cython scipy cupy
Follow the instructions here to download the dataset. Then, extract and organize the files inside your cloned repo directory as follows (note that only the necessary files are shown):
MTTR/
└── a2d_sentences/
├── Release/
│ ├── videoset.csv (videos metadata file)
│ └── CLIPS320/
│ └── *.mp4 (video files)
└── text_annotations/
├── a2d_annotation.txt (actual text annotations)
├── a2d_missed_videos.txt
└── a2d_annotation_with_instances/
└── */ (video folders)
└── *.h5 (annotations files)
Follow the instructions here to download the dataset. Then, extract and organize the files inside your cloned repo directory as follows (note that only the necessary files are shown):
MTTR/
└── jhmdb_sentences/
├── Rename_Images/ (frame images)
│ └── */ (action dirs)
├── puppet_mask/ (mask annotations)
│ └── */ (action dirs)
└── jhmdb_annotation.txt (text annotations)
Download the dataset from the competition's website here.
Note that you may be required to sign up to the competition in order to get access to the dataset. This registration process is free and short.
Then, extract and organize the files inside your cloned repo directory as follows (note that only the necessary files are shown):
MTTR/
└── refer_youtube_vos/
├── train/
│ ├── JPEGImages/
│ │ └── */ (video folders)
│ │ └── *.jpg (frame image files)
│ └── Annotations/
│ └── */ (video folders)
│ └── *.png (mask annotation files)
├── valid/
│ └── JPEGImages/
│ └── */ (video folders)
│ └── *.jpg (frame image files)
└── meta_expressions/
├── train/
│ └── meta_expressions.json (text annotations)
└── valid/
└── meta_expressions.json (text annotations)
The following table lists the parameters which can be configured directly from the command line.
The rest of the running/model parameters for each dataset can be configured in configs/DATASET_NAME.yaml.
Note that in order to run the code the path of the relevant .yaml config file needs to be supplied using the -c parameter.
| Command | Description |
|---|---|
| -c | path to dataset configuration file |
| -rm | running mode (train/eval) |
| -ws | window size |
| -bs | training batch size per GPU |
| -ebs | eval batch size per GPU (if not provided, training batch size is used) |
| -ng | number of GPUs to run on |
The following commands can be used to reproduce the main results of our paper using the supplied checkpoint files.
The commands were tested on RTX 3090 24GB GPUs, but it may be possible to run some of them using GPUs with less
memory by decreasing the batch-size -bs parameter.
| Window Size | Command | Checkpoint File | mAP Result |
|---|---|---|---|
| 10 | python main.py -rm eval -c configs/a2d_sentences.yaml -ws 10 -bs 3 -ckpt CHECKPOINT_PATH -ng 2 |
Link | 46.1 |
| 8 | python main.py -rm eval -c configs/a2d_sentences.yaml -ws 8 -bs 3 -ckpt CHECKPOINT_PATH -ng 2 |
Link | 44.7 |
The following commands evaluate our A2D-Sentences-pretrained model on JHMDB-Sentences without additional training.
For this purpose, as explained in our paper, we uniformly sample three frames from each video. To ensure proper
reproduction of our results on other machines we include the metadata of the sampled frames under
datasets/jhmdb_sentences/jhmdb_sentences_samples_metadata.json.
This file is automatically loaded during the evaluation process with the commands below.
To avoid using this file and force sampling different frames, change the seed and generate_new_samples_metadata
parameters under MTTR/configs/jhmdb_sentences.yaml.
| Window Size | Command | Checkpoint File | mAP Result |
|---|---|---|---|
| 10 | python main.py -rm eval -c configs/jhmdb_sentences.yaml -ws 10 -bs 3 -ckpt CHECKPOINT_PATH -ng 2 |
Link | 39.2 |
| 8 | python main.py -rm eval -c configs/jhmdb_sentences.yaml -ws 8 -bs 3 -ckpt CHECKPOINT_PATH -ng 2 |
Link | 36.6 |
The following command evaluates our model on the public validation subset of Refer-YouTube-VOS dataset.
Since annotations are not publicly available for this subset, our code generates a zip file with the predicted masks
under MTTR/runs/[RUN_DATE_TIME]/validation_outputs/submission_epoch_0.zip. This zip needs to be uploaded to the
competition server for evaluation. For your convenience we supply this zip file here as well.
| Window Size | Command | Checkpoint File | Output Zip | J&F Result |
|---|---|---|---|---|
| 12 | python main.py -rm eval -c configs/refer_youtube_vos.yaml -ws 12 -bs 1 -ckpt CHECKPOINT_PATH -ng 8 |
Link | Link | 55.32 |
First, download the Kinetics-400 pretrained weights of Video Swin Transformer from this link. Note that these weights were originally published in video swin's original repo here.
Place the downloaded file inside your cloned repo directory as
MTTR/pretrained_swin_transformer/swin_tiny_patch244_window877_kinetics400_1k.pth.
Next, the following commands can be used to train MTTR as described in our paper.
Note that it may be possible to run some of these commands on GPUs with less memory than the ones mentioned below
by decreasing the batch-size -bs or window-size -ws parameters. However, changing these parameters may also affect
the final performance of the model.
- The command for the following configuration was tested on 2 A6000 48GB GPUs:
| Window Size | Command |
|---|---|
| 10 | python main.py -rm train -c configs/a2d_sentences.yaml -ws 10 -bs 3 -ng 2 |
- The command for the following configuration was tested on 3 RTX 3090 24GB GPUs:
| Window Size | Command |
|---|---|
| 8 | python main.py -rm train -c configs/a2d_sentences.yaml -ws 8 -bs 2 -ng 3 |
- The command for the following configuration was tested on 4 A6000 48GB GPUs:
| Window Size | Command |
|---|---|
| 12 | python main.py -rm train -c configs/refer_youtube_vos.yaml -ws 12 -bs 1 -ng 4 |
- The command for the following configuration was tested on 8 RTX 3090 24GB GPUs.
| Window Size | Command |
|---|---|
| 8 | python main.py -rm train -c configs/refer_youtube_vos.yaml -ws 8 -bs 1 -ng 8 |
Note that this last configuration was not mentioned in our paper. However, it is more memory efficient than the original configuration (window size 12) while producing a model which is almost as good (J&F of 54.56 in our experiments).
As explained in our paper JHMDB-Sentences is used exclusively for evaluation, so training is not supported at this time for this dataset.
Please consider citing our work in your publications if it helped you or if it is relevant to your research:
@inproceedings{botach2021end,
title={End-to-End Referring Video Object Segmentation with Multimodal Transformers},
author={Botach, Adam and Zheltonozhskii, Evgenii and Baskin, Chaim},
booktitle={Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
year={2022}
}