BEiT: BERT Pre-Training of Image Transformers - 88.4% Top1 (+3% compared to ViT-L/16)

[AlexeyAB]

@rwightman Hi,

Is there a plan to add (BEiT: BERT Pre-Training of Image Transformers)? https://github.com/microsoft/unilm/tree/master/beit

• BEiT (ViT-L/16) - 88.4% Top1 384x384 (ImageNet-21k pre-training)
• orig (ViT-L/16) - 85.1% Top1 384x384 (ImageNet-21k pre-training)

It seems that only minimal additions and links to weights are needed:

• beit_base_patch16_224_8k_vocab is the same as vit_base_patch16_224 in the timm: https://github.com/microsoft/unilm/blob/2312c6fb384689d4ffdbc65360bcce7148cfbeed/beit/modeling_pretrain.py#L142

• beit_large_patch16_224_8k_vocab is the same as vit_large_patch16_224 in the timm: https://github.com/microsoft/unilm/blob/2312c6fb384689d4ffdbc65360bcce7148cfbeed/beit/modeling_pretrain.py#L157

[rwightman]

@AlexeyAB yeah, I'd like to get that in at some point, have quite a few other things on my plate right now so not quite sure when I'l squeek it in, but soon(ish)