This is the source code accompanying the paper VOS: Learning What You Don’t Know by Virtual Outlier Synthesis by Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li
The codebase is heavily based on ProbDet and Detectron2.
Checkout our
- CVPR'22 work STUD on object detection in video datasets.
- NeurIPS'22 work SIREN on OOD detection for detection transformers.
- ICLR'23 work NPOS on non-parametric outlier synthesis.
- NeurIPS'23 work DREAM-OOD on outlier generation in the pixel space (by diffusion models) if you are interested!
02/05/2023---we have uploaded the code here for reproducing the figure 1 of our paper.
05/08/2022---We have updated the openreview with the new results of using a nonlinear MLP for binary classification instead of the logistic regression, which is slightly better. Please check the code and models here.
pip install -r requirements.txt
In addition, install detectron2 following here.
PASCAL VOC
Download the processed VOC 2007 and 2012 dataset from here.
The VOC dataset folder should have the following structure:
└── VOC_DATASET_ROOT
|
├── JPEGImages
├── voc0712_train_all.json
└── val_coco_format.json
COCO
Download COCO2017 dataset from the official website.
Download the OOD dataset (json file) when the in-distribution dataset is Pascal VOC from here.
Download the OOD dataset (json file) when the in-distribution dataset is BDD-100k from here.
Put the two processed OOD json files to ./anntoations
The COCO dataset folder should have the following structure:
└── COCO_DATASET_ROOT
|
├── annotations
├── xxx (the original json files)
├── instances_val2017_ood_wrt_bdd_rm_overlap.json
└── instances_val2017_ood_rm_overlap.json
├── train2017
└── val2017
BDD-100k
Donwload the BDD-100k images from the official website.
Download the processed BDD-100k json files from here and here.
The BDD dataset folder should have the following structure:
└── BDD_DATASET_ROOT
|
├── images
├── val_bdd_converted.json
└── train_bdd_converted.json
OpenImages
Download our OpenImages validation splits here. We created a tarball that contains the out-of-distribution data splits used in our paper for hyperparameter tuning. Do not modify or rename the internal folders as those paths are hard coded in the dataset reader. The OpenImages dataset is created in a similar way following this paper.
The OpenImages dataset folder should have the following structure:
└── OEPNIMAGES_DATASET_ROOT
|
├── coco_classes
└── ood_classes_rm_overlap
Visualization of the OOD datasets
The OOD images with respect to different in-distribution datasets can be downloaded from ID-VOC-OOD-COCO, ID-VOC-OOD-openimages, ID-BDD-OOD-COCO, ID-BDD-OOD-openimages.
Firstly, enter the detection folder by running
cd detection
Before training, modify the dataset address by changing "dataset-dir" according to your local dataset address.
Vanilla Faster-RCNN with VOC as the in-distribution dataset
python train_net.py
--dataset-dir path/to/dataset/dir
--num-gpus 8
--config-file VOC-Detection/faster-rcnn/vanilla.yaml
--random-seed 0
--resume
Vanilla Faster-RCNN with BDD as the in-distribution dataset
python train_net.py
--dataset-dir path/to/dataset/dir
--num-gpus 8
--config-file BDD-Detection/faster-rcnn/vanilla.yaml
--random-seed 0
--resume
VOS on ResNet
python train_net_gmm.py
--dataset-dir path/to/dataset/dir
--num-gpus 8
--config-file VOC-Detection/faster-rcnn/vos.yaml
--random-seed 0
--resume
VOS on RegNet
Before training using the RegNet as the backbone, download the pretrained RegNet backbone from here.
python train_net_gmm.py
--dataset-dir path/to/dataset/dir
--num-gpus 8
--config-file VOC-Detection/faster-rcnn/regnetx.yaml
--random-seed 0
--resume
Before training on VOS, change "VOS.STARTING_ITER" and "VOS.SAMPLE_NUMBER" in the config file to the desired numbers in paper.
Evaluation with the in-distribution dataset to be VOC
Firstly run on the in-distribution dataset:
python apply_net.py
--dataset-dir path/to/dataset/dir
--test-dataset voc_custom_val
--config-file VOC-Detection/faster-rcnn/vos.yaml
--inference-config Inference/standard_nms.yaml
--random-seed 0
--image-corruption-level 0
--visualize 0
Then run on the OOD dataset:
python apply_net.py
--dataset-dir path/to/dataset/dir
--test-dataset coco_ood_val
--config-file VOC-Detection/faster-rcnn/vos.yaml
--inference-config Inference/standard_nms.yaml
--random-seed 0
--image-corruption-level 0
--visualize 0
Obtain the metrics using:
python voc_coco_plot.py
--name vos
--thres xxx
--energy 1
--seed 0
Here the threshold is determined according to ProbDet. It will be displayed in the screen as you finish evaluating on the in-distribution dataset.
Evaluation with the in-distribution dataset to be BDD
Firstly run on the in-distribution dataset:
python apply_net.py
--dataset-dir path/to/dataset/dir
--test-dataset bdd_custom_val
--config-file BDD-Detection/faster-rcnn/vos.yaml
--inference-config Inference/standard_nms.yaml
--random-seed 0
--image-corruption-level 0
--visualize 0
Then run on the OOD dataset:
python apply_net.py
--dataset-dir path/to/dataset/dir
--test-dataset coco_ood_val_bdd
--config-file BDD-Detection/faster-rcnn/vos.yaml
--inference-config Inference/standard_nms.yaml
--random-seed 0
--image-corruption-level 0
--visualize 0
Obtain the metrics using:
python bdd_coco_plot.py
--name vos
--thres xxx
--energy 1
--seed 0
Pretrained models
The pretrained models for Pascal-VOC can be downloaded from vanilla and VOS-ResNet and VOS-RegNet.
The pretrained models for BDD-100k can be downloaded from vanilla and VOS-ResNet and VOS-RegNet.
Train on WideResNet
cd classification/CIFAR/ &
python train_virtual.py
--start_epoch 40
--sample_number 1000
--sample_from 10000
--select 1
--loss_weight 0.1
where "start_epoch" denotes the starting epoch of the uncertainty regularization branch.
"sample_number" denotes the size of the in-distribution queue.
"sample_from" and "select" are used to approximate the likelihood threshold during virtual outlier synthesis.
"loss_weight" denotes the weight of the regularization loss.
Please see Section 3 and Section 4.1 in the paper for details.
Train on DenseNet
cd classification/CIFAR/ &
python train_virtual_dense.py
--start_epoch 40
--sample_number 1000
--sample_from 10000
--select 1
--loss_weight 0.1
Evaluation on different classifiers
cd classification/CIFAR/ &
python test.py
--model_name xx
--method_name xx
--score energy
--num_to_avg 10
where "model_name" denotes the model architectures. ("res" denotes the WideResNet and "dense" denotes the DenseNet.)
"method_name" denotes the checkpoint name you are loading.
Pretrained models
We provide the pretrained models using WideResNet and DenseNet with the in-distribution dataset to be CIFAR-10.
If you found any part of this code is useful in your research, please consider citing our paper:
@article{du2022vos,
title={VOS: Learning What You Don’t Know by Virtual Outlier Synthesis},
author={Du, Xuefeng and Wang, Zhaoning and Cai, Mu and Li, Yixuan},
journal={Proceedings of the International Conference on Learning Representations},
year={2022}
}
