Training hyperparameters for ResNet and SEResNet families

[bryanyzhu]

Hi, thank you for the great repo. According to #78 , I tried your weights of ResNet and SE-ResNet on ImageNet and ImageNetC. Both these models achieve good performance, especially on ImageNetC.

Take seresnet50 as an example,

ImageNet top1: 80.3
ImageNet-C mce: 60.4

However, when I train seresnet50 on my own, I couldn't achieve such good performance. I use the commands on your website as below,

./distributed_train.sh 8 ~/data/imagenet -b 64 --model seresnet50 --sched cosine --epochs 90 --lr 0.2 --amp --remode pixel --reprob 0.6 --aug-splits 3 --aa rand-m9-mstd0.5-inc1 --resplit --split-bn --jsd --dist-bn reduce -j 8

and I achieve the following performance,

ImageNet top1: 78.5
ImageNet-C mce: 67.5

Due to time constraint, I only train the model 90 epochs (actually 100 because of 10 cooldown epochs). So the performance drop of 1.8 on ImageNet may be within the range. But the performance drop on ImageNet-C is considerable. Same thing happen to ResNet and ResNetXt as well (big drop on ImageNetC). I'm not sure if this is purely due to my short training or my hyperparameters are not set correctly. Please advise if my hyperparameters are different from yours, or any suggestions are appreciated. Thank you very much.