Pretraining models

[opencomvis]

Hi! Thanks for this great work.
are there pretraining code scripts and the corresponding hyperparameters used for pretrained models?
some pretrained models have some hints such as input size, augmentation, and dataset trained on.
but what I wanted is the codes for pretraining and reproduce these model weights, and possibly train also on my own data.

[TorbenSDJohansen]

Most of the models you are able to access here are not trained using timm. The best way to find the hyperparameters is to look at the model file, which then contains information such as URL to the original paper that introduced the model. Note, however, that it is not uncommon for other (more recent) papers to use the timm codebase, which makes adaptation very simple. Further, you should be able to find good training hyperparameters for a relatively large number of models if you browse, e.g., the discussion page or look at @rwightman's gists.

[rwightman]

@opencomvis moving to discussion, as Torben mentions you can find quite a few different recipes (hparams) in past issues and discussions here... there are templates for good results, several recipe templates were provided in ResNet Strikes Back (https://arxiv.org/abs/2110.00476) and there are discussions here which translate them to specific args.

Unfortunately every model, pretrain vs finetune, dataset combo varies so it's not possible to provide hparams that work best in all situations, you always need to do some amount of searching so having templates is generally more useful.