OVModelForVisionCausalLM

[eaidova]

What does this PR do?

Enables conversion and inference for multimodality llm like llava, llava-next, falcon-vl, pixtral, internvl
Example of usage:

from PIL import Image
import requests
from optimum.intel.openvino import OVModelForVisualCausalLM
from transformers import AutoProcessor

model_id = "llava-hf/llava-v1.6-mistral-7b-hf"
model = OVModelForVisualCausalLM.from_pretrained(model_id)
image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
processor = AutoProcessor.from_pretrained(model_id)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What are these?"},
            {"type": "image"},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(images=raw_image, text=prompt, return_tensors="pt")

output = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print(processor.decode(output[0], skip_special_tokens=True))

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[eaidova]

@echarlaix could you please take a look? Thanks!

P.S. I'm still working on extending models coverage, but I think it makes sense to start looking on general API now

[AlexKoff88]

Do we really export to the ONNX? The docstrings should be revised accordingly.

[eaidova]

I accidentally copied that from https://github.com/huggingface/optimum-intel/blob/main/optimum/intel/openvino/modeling_base.py#L534
I see the same doc string in other model classes for openvino (seq2seq, stable diffusion support e.t.c.), @echarlaix maybe it is time to revise them in all models too?

[echarlaix]

Looking good, thanks a lot @eaidova

[echarlaix]

why is this needed ?

[eaidova]

this is tranable parameter filled by random values during model weights init, we can not capture this during model export as it is not a part of submodels inference forward pass, that is why for transferring it from pt to ov I need to use config for that. If you have better suggestion, I can try to implement this

[echarlaix]

Huge work, thanks @eaidova, left couple of minor comments

[echarlaix]

P.S. I'm still working on extending models coverage, but I think it makes sense to start looking on general API now

Very nice, let's merge this PR and we can always extend support in following PRs, does it work for you @eaidova ?

[eaidova]

P.S. I'm still working on extending models coverage, but I think it makes sense to start looking on general API now

Very nice, let's merge this PR and we can always extend support in following PRs, does it work for you @eaidova ?

@echarlaix , yes, I agree, could you please merge?