Attention modules and pretrained networks

[eovallemagallanes]

Hi,

I have create a model such as:

net = timm.models.resnet.resnet18(block_args=block_args, num_classes=num_classes, pretrained=True)

If in block_args an attention module is included, the respective submodules are pretrained also?

This applies to different resnet variants, e.g., resnet50, 101.?

Thanks a lot.

[rwightman]

@eovallemagallanes if you pass different args that essentially change the model architecture there won't be pretrained weights for that... only defined model configs that have urls set for their weights can be used with pretrained flag.

ECA attention as an example, the resnet class lets you specify attention modules https://github.com/rwightman/pytorch-image-models/blob/f7d210d759beb00a3d0834a3ce2d93f6e17f3d38/timm/models/resnet.py#L1243

You can use anythign in https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/create_attn.py although the ResNet setup is designed for channel attention like modules such as SE / ECA / etc that aren't too large

Byob/ByoaNet are more flexible network architectures that allow more variety of configs and have a residual block design to use larger attention layers like Halo / Bottleneck, etc. https://github.com/rwightman/pytorch-image-models/blob/f7d210d759beb00a3d0834a3ce2d93f6e17f3d38/timm/models/byobnet.py#L1171