ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

[amaarora]

Paper - https://arxiv.org/abs/2103.10697

Hey @rwightman - planning to add ConViT to TIMM with pretrained weights. Believe this should be slightly easier as the official code is available in PyTorch (they use TIMM too). Based on initial triage, I think I should be able to wrap this up in the next day or two.

Just want to double check before I start in case you already have plans to get ConViT in TIMM and have code ready locally waiting to be merged. Let me know :)

[rwightman]

@amaarora no, I don't have that one in progress, I do have LeVit on the go though. Switched gears to get EffV2 out.

One issue with this one, and the reason I didn't touch it, it has a non-commercial license. So I can't include it here.

[amaarora]

Thanks @rwightman - working on getting a training run kicked off for Visformer today.

Also, ConViT now under Apache license, so hope to get a PR in for that too!

[rwightman]

@amaarora awesome! was the convit license your push or were they already waiting?

I think all/most of the training elements for the vit-like models are here, but the deit codebase from facebook has the hparams all setup so I'm not sure what's quicker to get up and running with. The deit training loop uses a lot of timm components.

[amaarora]

Re ConViT, the researchers had a pretty fast turnaround - see here.

I think all/most of the training elements for the vit-like models are here, but the deit codebase from facebook has the hparams all setup so I'm not sure what's quicker to get up and running with. The deit training loop uses a lot of timm components.

Yes, I also saw the codebase in diet - it uses timm but has updated specific parts of code. Maybe I'll just use deit to get the training going - looks like an easier option.

[amaarora]

PS: Really happy to see TIMM being part of research :)

[amaarora]

@rwightman Looks like there are pretrained weights already for Visformer here. I am going to first run evaluation on these weights using TIMM validation script to check performance on ImageNet-1k.

[amaarora]

Hey @rwightman - you should be able to find pre-trained weights for visformer_small here - https://www.kaggle.com/aroraaman/visformer-small-chkpt.

Please note I didn;t train the weights myself but edited the keys to match TIMM from here.

You should now be able to do:

import timm 
m = timm.create_model('visformer_small')
m.load_state_dict(torch.load(<path _to_model>))

Hope this helps.