Add AIMv2 to Transformers #35550
Add AIMv2 to Transformers #35550
Conversation
|
@qubvel @Rocketknight1 Could you help review this PR ? Let me know if you have any suggestions. Thank you in advance for your time and assistance. |
There was a problem hiding this comment.
Hi @AlanPonnachan! Thanks for working on the model 🤗
Please see other model implementations in the repo to follow code style and patterns (e.g. how the attention module should be implemented).
It's better to reuse existing blocks rather than defining new ones. You can also utilize modular converter for inheritance.
Please see similar PRs:
Thanks!
| @@ -0,0 +1,225 @@ | |||
| # coding=utf-8 | |||
There was a problem hiding this comment.
Please refactor it to follow mllama model conversion format. It should be a KEY_MAPPING dict instead of create_rename_keys
| return image | ||
|
|
||
|
|
||
| @torch.no_grad() |
There was a problem hiding this comment.
Do we need no_grad here? The one is used in the code below
| ) | ||
| query, key, value = qkv.unbind(0) | ||
|
|
||
| context_layer = F.scaled_dot_product_attention(query, key, value, attn_mask=mask) |
There was a problem hiding this comment.
Please see other models on how the attention module should be structured.
What does this PR do?
Fixes #35351
This PR adds AIMv2 support in Transformers. AIMv2 showed better performance than SigLIP.
TODO
Who can review?
@qubvel
@Rocketknight1