Add Molmo (7B-D, 7B-O, 70B) #33962
Add Molmo (7B-D, 7B-O, 70B) #33962
Conversation
There was a problem hiding this comment.
Wow looks super nice! Will finish #33859 asap to let you continue!
upstream merge of Arthur's modular PR
|
Still seeing some duplicate imports in the modeling code: from ...modeling_outputs import (
BaseModelOutputWithPast,
CausalLMOutputWithPast,
)
from ...modeling_rope_utils import ROPE_INIT_FUNCTIONS
from ...modeling_utils import PreTrainedModel
from ...utils import (
add_start_docstrings,
add_start_docstrings_to_model_forward,
is_flash_attn_2_available,
is_flash_attn_greater_or_equal_2_10,
logging,
replace_return_docstrings,
)
from .configuration_molmo import MolmoConfig
if is_flash_attn_2_available():
from ...modeling_flash_attention_utils import _flash_attention_forward
from ...modeling_outputs import BaseModelOutput, BaseModelOutputWithPooling, ModelOutput
from ...utils import (
ModelOutput,
is_flash_attn_2_available,
torch_int,
)
from .configuration_molmo import MOLMOConfig, MOLMOVisionConfigOne quick&dirty solution would be to do a pass on the imports once the transformer in modular has finished, so that imports from various modules get merged and normalized to the most likely - but there's also some capitalized (wrong) model names that remain as well, strangely, like class MolmoVisionTransformer(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
embed_dim = config.hidden_size
self.embeddings = MolmoVisionEmbeddings(config)
self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
self.encoder = MOLMOEncoder(config) # wut
self.post_layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps, bias=True)getting there however! |
|
Do you need a review? 🤗 |
|
Maybe a bit pre-mature but when using the script to convert the model to hf I got missmatch issues here: |
|
Hi @molbap, I am using the vision_backbone from this model. The processor creates varying number of crops of different images, so the processed 'images' tensor has dimension (batch_size, num_crops, num_patch, n_pixels) with varying num_crops, even with images of same resolution (after resizing). So anything with batch_size>1 does not work. Is there a way to fix the number of crops? Passing self.max_crops value doesn't seem to help. |
|
@moha23 , that's surprising - I resumed working on this after new year's break, but what you described should not happen as we pad with |
Have not used this branch, but the latest stable version. Thanks! Will check what changes need to be made. |
|
The cohere interface had changed in the attentions refactor, I included it cherry-picking from #35359 and now seems to run fine :) @yonigozlan can you take a look at the fast image processor? It's not inline with #35069 but it's torch/torchvision and indeed faster. @ArthurZucker model should be ready now, ping so it goes back up in your radar! |
molbap
requested review from
Rocketknight1,
SunMarc,
MekkCyber,
muellerzr,
qubvel and
stevhliu
as code owners
| The Molmo model was proposed in [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models | ||
| ]([https://arxiv.org/abs/2409.17146]) by Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, Aniruddha Kembhavi. |
There was a problem hiding this comment.
| The Molmo model was proposed in [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models | |
| ]([https://arxiv.org/abs/2409.17146]) by Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, Aniruddha Kembhavi. | |
| The Molmo model was proposed in [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models]([https://arxiv.org/abs/2409.17146]) by Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, Aniruddha Kembhavi. |
| *Today's most advanced multimodal models remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed models into open ones. As a result, the community is still missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key innovation is a novel, highly detailed image caption dataset collected entirely from human annotators using speech-based descriptions. To enable a wide array of user interactions, we also introduce a diverse dataset mixture for fine-tuning that includes in-the-wild Q&A and innovative 2D pointing data. The success of our approach relies on careful choices for the model architecture details, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets, all of which will be released. The best-in-class 72B model within the Molmo family not only outperforms others in the class of open weight and data models but also compares favorably against proprietary systems like GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human evaluation. | ||
| * |
There was a problem hiding this comment.
| *Today's most advanced multimodal models remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed models into open ones. As a result, the community is still missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key innovation is a novel, highly detailed image caption dataset collected entirely from human annotators using speech-based descriptions. To enable a wide array of user interactions, we also introduce a diverse dataset mixture for fine-tuning that includes in-the-wild Q&A and innovative 2D pointing data. The success of our approach relies on careful choices for the model architecture details, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets, all of which will be released. The best-in-class 72B model within the Molmo family not only outperforms others in the class of open weight and data models but also compares favorably against proprietary systems like GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human evaluation. | |
| * | |
| *Today's most advanced multimodal models remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed models into open ones. As a result, the community is still missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key innovation is a novel, highly detailed image caption dataset collected entirely from human annotators using speech-based descriptions. To enable a wide array of user interactions, we also introduce a diverse dataset mixture for fine-tuning that includes in-the-wild Q&A and innovative 2D pointing data. The success of our approach relies on careful choices for the model architecture details, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets, all of which will be released. The best-in-class 72B model within the Molmo family not only outperforms others in the class of open weight and data models but also compares favorably against proprietary systems like GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human evaluation.* |
|
|
||
| Tips: | ||
|
|
||
| - We advise users to use `padding_side="left"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = "left"` before generating. |
There was a problem hiding this comment.
| - We advise users to use `padding_side="left"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = "left"` before generating. | |
| - We recommend calling `processor.tokenizer.padding_side = "left"` for batched generation because it leads to more accurate results. |
| @@ -334,6 +335,7 @@ def from_pretrained(cls, pretrained_model_name_or_path, **kwargs): | |||
| elif type(config) in PROCESSOR_MAPPING: | |||
| return PROCESSOR_MAPPING[type(config)].from_pretrained(pretrained_model_name_or_path, **kwargs) | |||
|
|
|||
| print("BUT WHY", processor_class) | |||
There was a problem hiding this comment.
lol, some debugging struggles scars left
|
|
||
|
|
||
| # swiglu activation | ||
| class MolmoSwiGLU(nn.Module): |
There was a problem hiding this comment.
we can also just put this in the text MLP 😉
and let's not use single letter variables
| residual = hidden_states | ||
| hidden_states = self.input_layernorm(hidden_states) | ||
|
|
||
| # Self Attention | ||
| hidden_states, self_attn_weights = self.self_attn( | ||
| hidden_states=hidden_states, | ||
| attention_mask=attention_mask, | ||
| position_ids=position_ids, | ||
| past_key_value=past_key_value, | ||
| output_attentions=output_attentions, | ||
| use_cache=use_cache, | ||
| cache_position=cache_position, | ||
| position_embeddings=position_embeddings, | ||
| ) | ||
| hidden_states = residual + hidden_states | ||
|
|
||
| # Fully Connected | ||
| residual = hidden_states | ||
| hidden_states = self.post_attention_layernorm(hidden_states) | ||
| hidden_states = self.mlp(hidden_states) | ||
| hidden_states = residual + hidden_states |
There was a problem hiding this comment.
looks standard! What's the diff ?
| return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim) | ||
|
|
||
|
|
||
| def pooling_eager_attention_forward( |
There was a problem hiding this comment.
this one is exactly the same as bamba, gemma etc, might need to import xxxx as pooling_eager_attention_forward !
There was a problem hiding this comment.
ah right, I moved the q/kv states to the forward to yes we can uniformize this!
| self.embed_dim = config.hidden_size | ||
| self.num_heads = config.num_attention_heads | ||
| self.head_dim = config.head_dim | ||
| self.attention_dropout = config.attention_dropout | ||
| self.scaling = self.head_dim**0.5 | ||
| self.is_causal = True | ||
|
|
There was a problem hiding this comment.
don't think we need all this in the forward!
| def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor: | ||
| """ | ||
| This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, | ||
| num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) | ||
| """ | ||
| batch, num_key_value_heads, slen, head_dim = hidden_states.shape | ||
| if n_rep == 1: | ||
| return hidden_states | ||
| hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim) | ||
| return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim) |
| def _merge_input_ids_with_image_features(self, image_features, inputs_embeds, input_ids, attention_mask, labels): | ||
| pass |
There was a problem hiding this comment.
I don't remember, why is this explicit if it is not changed?
There was a problem hiding this comment.
ah, that's weird indeed 🤔 maybe I wanted to override it so it does nothing? will remove
| logger = logging.get_logger(__name__) | ||
|
|
||
|
|
||
| def get_resize_output_image_size( |
There was a problem hiding this comment.
copied from? (since this file is not modular generated!)
What does this PR do?
As mentioned in issue #33710 , this is a draft to add support for Molmo natively in
transformers.It is also using the new
modularframework introduced in #33248 .Molmo has several existing variants:
The last three models share the same modeling, and thus will be covered by this PR.
Relative to the modular framework:
Choose a base model that's as close as possible from the one you're porting.
In my case, I'm using Llava as a reference. The differences I identify at a glance are the 2d pooling,
Figure out the differences.
Some differences will be a complete modification of the original module, in that case, all have to be redefined.
Some differences will be very tiny. For instance, some layers might be the same, but initialized with a different configuration key.
For instance, the position embeddings are slightly different.
Preserving inheritance across model components renames.
For instance, the code above will trigger
Because the supported pattern is currently searching for a caps-based model name. However, using
modularis very promising and makes for a much smaller modeling file to review.I'll write down hurdles encountered here for future reference so that adding multimodal models to
transformersends up being a breeze.