Implement Vision Language Model chapter

[burtenshaw]

We need to implement the section on VLMs. It should be based on existing content from the huggingface ecosystem, adapted for SmolVLM, adapted to the course structure, and offer exercises.

Material

• Added new Fine-Tuning a smol VLM (SmolVLM) with the Hugging Face Ecosystem (TRL) using SFT on a Consumer GPU recipe 🧑‍🍳️ cookbook#246
• https://huggingface.co/learn/cookbook/fine_tuning_vlm_trl
• https://github.com/merveenoyan/smol-vision/blob/main/Smol_VLM_FT.ipynb
• https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
• https://huggingface.co/blog/smolvlm
• https://huggingface.co/blog/vlms

Steps to do

• add prose that explains vlm in simplest possible terms
• add references to all pages
• add exercise notebook on finetuning SmolVLM

[duydl]

Hi, could the issue be assigned?

[burtenshaw]

Hi, could the issue be assigned?

Of course, I expect others will contribute too, but open a draft PR with the outline and LGTM.

[jungnerd]

Hi there 👋🏻
I'd like to contribute to this issue by providing an exercise notebook demonstrating how to fine-tune VLM with TRL?
The notebook will be based on How to Fine-Tune Multimodal Models or VLMs with Hugging Face TRL but using a SmolVLM instead.

[burtenshaw]

That's great @jungnerd . Thanks.

We're already working on the text for the vlm chapter here: #59 . There's a first draft of the notebook from @duydl .

Why don't Open a PR on to #59 with an exercise notebook?

[duydl]

@jungnerd I had been working on sft for vlm and just finished it. There is still the dpo approach notebook, could be based on this blog Hugging Face Blog: Preference Optimization for VLMs, and probably lots of fixes and optimizations in completed the markdown and notebooks for vlm.

[jungnerd]

@duydl Am I understanding correctly that you’re suggesting creating a notebook to fine-tune SmolVLM using DPO? If so, that sounds great! I think it would be really exciting to work on a fine-tuning notebook with DPO!