bugfix Idefics3 processor - handle gracefully cases with text and no images

[mfarre]

What does this PR do?

Fixing Idefics3 processor to work with batches that do not include images

Who can review?

@andimarafioti

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[andimarafioti]

LGTM!

[andimarafioti]

Let's wait for @yonigozlan to be sure!

[yonigozlan]

Yes that looks much better! Just suggested a check to avoid hallucinations like this:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    }
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}


# Generate
generated_ids = model.generate(**inputs, max_new_tokens=50)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)

['User:What do we see in this image?\nAssistant: The image depicts a scene from a historical or fictional setting, likely from the medieval period, given the attire and architecture. The central focus is on a large, ornate gate, which appears to be the entrance to a castle or a fortified structure.']

also in Idefics3ProcessorTest could we add a test for text only inference and one to check that an error is raised if we have an image in the conversation, but no images are passed to the processor?

[mfarre]

thanks @andimarafioti
thanks @yonigozlan if you give me your LGTM I will merge the changes: I followed your suggestions adding some tests and adding your code proposal.

Yes that looks much better! Just suggested a check to avoid hallucinations like this:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    }
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}


# Generate
generated_ids = model.generate(**inputs, max_new_tokens=50)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)

['User:What do we see in this image?\nAssistant: The image depicts a scene from a historical or fictional setting, likely from the medieval period, given the attire and architecture. The central focus is on a large, ornate gate, which appears to be the entrance to a castle or a fortified structure.']

also in Idefics3ProcessorTest could we add a test for text only inference and one to check that an error is raised if we have an image in the conversation, but no images are passed to the processor?

[yonigozlan]

Looks great thanks @mfarre for fixing this! LGTM for me, but let's maybe wait for @ArthurZucker 's review before merging :)