MNIST-baselines

This repository contains various baseline models for comparison on the polluted MNIST dataset

Requirements

• NumPy
• scikit-learn
• PyTorch

Usage

Create a model

1. Prepare Model_Name.json in config/
2. Prepare Model_Name.py in models/
3. Prepare Trainer_Name.py in trainers/ (optional)

Train a model

python3 main.py --method Model_Name

Test a model

python3 main.py --method Model_Name --test

Trained models

Available at https://drive.google.com/open?id=1jXe2kJJLkhGH8T30juGbyUgsaDQZXISu

Polluted MNIST

Feature Extraction

A visualization of the first 15 principle components:

Model	Accuracy / %	Model	Accuracy / %
Plain DNN	90.72	DNN + Dropout	90.74
DNN + Batch Normalization	92.39	DNN + PCA (0.99 variance)	21.40
DNN + PCA (0.95 variance)	21.34	DNN + PCA (0.90 variance)	20.55
DNN + ICA	11.93	DNN + NMF	11.30
DNN + VAE	95.24

Conventional Classification Approaches

Model	Accuracy / %	Model	Accuracy / %
Empty Model	11.04	Naive Bayes	19.14
Logistic Regression	24.37	Decision Tree (gini)	51.24
Decision Tree (entropy)	50.91	KNN (3 neighbours)	78.82
KNN (5 neighbours)	78.46	KNN (10 neighbours)	76.40
SGD	22.30	SVM (sigmoid kernel)	10.84
SVM (rbf kernel)	85.94	SVM (polynomial kernel)	87.11

The training curves of the above DNN models:

Deep Convolutional Neural Networks

Model	Accuracy / %	Model	Accuracy / %
LeNet-5	98.35	MobileNetV2	99.63
VGG-19	99.64	ShuffleNetG3	99.65
ResNet-101	99.71	GoogLeNet	99.82
PreAct ResNet-152	99.68	PNASNet	99.75
DenseNet-161	99.75	ResNeXt-29-8x64d	99.71
DPN-92	99.76	SENet-18	99.69
MobileNet	99.44	CapsNet	98.84

Error Analysis

Team Members

• Xiangyu Chen
• Zelin Ye
• Yiheng Zhang