Pretrained SSL Models in Timm Package

[bryanwong17]

Hi, I am just curious are there pretrained SSL models available in the timm package? Based on what I know, timm.create_model only supports models that are trained in a supervised setting on ImageNet1k and ImageNet22k, but there is nothing trained in SSL ways (SimCLR, Barlow Twins, MoCoV3, MAE, etc).

If not available, what is the easiest way to initialize the model from timm package and load the pretrained SSL weights? Thank you!

[rwightman]

There's quite a few, no simclr, barlow though

>>> timm.list_pretrained('*.*mae*')
['convnextv2_atto.fcmae', 'convnextv2_atto.fcmae_ft_in1k', 'convnextv2_base.fcmae', 'convnextv2_base.fcmae_ft_in1k', 'convnextv2_base.fcmae_ft_in22k_in1k', 'convnextv2_base.fcmae_ft_in22k_in1k_384', 'convnextv2_femto.fcmae', 'convnextv2_femto.fcmae_ft_in1k', 'convnextv2_huge.fcmae', 'convnextv2_huge.fcmae_ft_in1k', 'convnextv2_huge.fcmae_ft_in22k_in1k_384', 'convnextv2_huge.fcmae_ft_in22k_in1k_512', 'convnextv2_large.fcmae', 'convnextv2_large.fcmae_ft_in1k', 'convnextv2_large.fcmae_ft_in22k_in1k', 'convnextv2_large.fcmae_ft_in22k_in1k_384', 'convnextv2_nano.fcmae', 'convnextv2_nano.fcmae_ft_in1k', 'convnextv2_nano.fcmae_ft_in22k_in1k', 'convnextv2_nano.fcmae_ft_in22k_in1k_384', 'convnextv2_pico.fcmae', 'convnextv2_pico.fcmae_ft_in1k', 'convnextv2_tiny.fcmae', 'convnextv2_tiny.fcmae_ft_in1k', 'convnextv2_tiny.fcmae_ft_in22k_in1k', 'convnextv2_tiny.fcmae_ft_in22k_in1k_384', 'vit_base_patch16_224.mae', 'vit_huge_patch14_224.mae', 'vit_large_patch16_224.mae']

>>> timm.list_pretrained('*.*wsl*')
['resnet18.fb_swsl_ig1b_ft_in1k', 'resnet50.fb_swsl_ig1b_ft_in1k', 'resnext50_32x4d.fb_swsl_ig1b_ft_in1k', 'resnext101_32x4d.fb_swsl_ig1b_ft_in1k', 'resnext101_32x8d.fb_swsl_ig1b_ft_in1k', 'resnext101_32x8d.fb_wsl_ig1b_ft_in1k', 'resnext101_32x16d.fb_swsl_ig1b_ft_in1k', 'resnext101_32x16d.fb_wsl_ig1b_ft_in1k', 'resnext101_32x32d.fb_wsl_ig1b_ft_in1k']

>>> timm.list_pretrained('*.*ssl*')
['resnet18.fb_ssl_yfcc100m_ft_in1k', 'resnet50.fb_ssl_yfcc100m_ft_in1k', 'resnext50_32x4d.fb_ssl_yfcc100m_ft_in1k', 'resnext101_32x4d.fb_ssl_yfcc100m_ft_in1k', 'resnext101_32x8d.fb_ssl_yfcc100m_ft_in1k', 'resnext101_32x16d.fb_ssl_yfcc100m_ft_in1k']

>>> timm.list_pretrained('*beit*')
['beit_base_patch16_224.in22k_ft_in22k', 'beit_base_patch16_224.in22k_ft_in22k_in1k', 'beit_base_patch16_384.in22k_ft_in22k_in1k', 'beit_large_patch16_224.in22k_ft_in22k', 'beit_large_patch16_224.in22k_ft_in22k_in1k', 'beit_large_patch16_384.in22k_ft_in22k_in1k', 'beit_large_patch16_512.in22k_ft_in22k_in1k', 'beitv2_base_patch16_224.in1k_ft_in1k', 'beitv2_base_patch16_224.in1k_ft_in22k', 'beitv2_base_patch16_224.in1k_ft_in22k_in1k', 'beitv2_large_patch16_224.in1k_ft_in1k', 'beitv2_large_patch16_224.in1k_ft_in22k', 'beitv2_large_patch16_224.in1k_ft_in22k_in1k']

>>> timm.list_pretrained('*dino*')
['resmlp_12_224.fb_dino', 'resmlp_24_224.fb_dino', 'vit_base_patch8_224.dino', 'vit_base_patch14_dinov2.lvd142m', 'vit_base_patch14_reg4_dinov2.lvd142m', 'vit_base_patch16_224.dino', 'vit_giant_patch14_dinov2.lvd142m', 'vit_giant_patch14_reg4_dinov2.lvd142m', 'vit_large_patch14_dinov2.lvd142m', 'vit_large_patch14_reg4_dinov2.lvd142m', 'vit_small_patch8_224.dino', 'vit_small_patch14_dinov2.lvd142m', 'vit_small_patch14_reg4_dinov2.lvd142m', 'vit_small_patch16_224.dino']

I thought I had moco weights somewhere but guess not... maybe they were a restrictive license?

[rwightman]

Also some SWAG

>>> timm.list_pretrained('*.*swag*')
['regnety_160.swag_ft_in1k', 'regnety_160.swag_lc_in1k', 'regnety_320.swag_ft_in1k', 'regnety_320.swag_lc_in1k', 'regnety_1280.swag_ft_in1k', 'regnety_1280.swag_lc_in1k']

[bryanwong17]

Thank you for the answers!