Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update SGLang example to use Qwen2-VL #1030

Merged
merged 6 commits into from
Jan 10, 2025
Merged

Update SGLang example to use Qwen2-VL #1030

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4000da4
Update SGLang example to use Qwen2-VL
advay-modal Jan 2, 2025
356ba89
ruff
advay-modal Jan 2, 2025
8348d64
Update 06_gpu_and_ml/llm-serving/sgl_vlm.py
advay-modal Jan 3, 2025
78516d6
Update 06_gpu_and_ml/llm-serving/sgl_vlm.py
advay-modal Jan 3, 2025
39f78a6
PR changes
advay-modal Jan 3, 2025
e5dea24
minor text fixes, l40s, cleaner file handling
charlesfrye Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions 06_gpu_and_ml/llm-serving/sgl_vlm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,13 @@

MINUTES = 60 # seconds

# We use a [LLaVA-NeXT](https://huggingface.co/docs/transformers/en/model_doc/llava_next)
# model built on top of Meta's LLaMA 3 8B.
# We use a [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)
advay-modal marked this conversation as resolved.
Show resolved Hide resolved
# made by Alibaba
advay-modal marked this conversation as resolved.
Show resolved Hide resolved

MODEL_PATH = "lmms-lab/llama3-llava-next-8b"
MODEL_REVISION = "e7e6a9fd5fd75d44b32987cba51c123338edbede"
TOKENIZER_PATH = "lmms-lab/llama3-llava-next-8b-tokenizer"
MODEL_CHAT_TEMPLATE = "llama-3-instruct"
MODEL_PATH = "Qwen/Qwen2-VL-7B-Instruct"
MODEL_REVISION = "a7a06a1cc11b4514ce9edcde0e3ca1d16e5ff2fc"
TOKENIZER_PATH = "Qwen/Qwen2-VL-7B-Instruct"
MODEL_CHAT_TEMPLATE = "qwen2-vl"

# We download it from the Hugging Face Hub using the Python function below.

Expand All @@ -73,13 +73,15 @@ def download_model_to_image():
vlm_image = (
modal.Image.debian_slim(python_version="3.11")
.pip_install( # add sglang and some Python dependencies
"sglang[all]==0.1.17",
"transformers==4.40.2",
"transformers==4.47.1",
"numpy<2",
"fastapi[standard]==0.115.4",
"pydantic==2.9.2",
"starlette==0.41.2",
)
.run_commands(
"pip install 'sglang[all]==0.4.1' --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/",
advay-modal marked this conversation as resolved.
Show resolved Hide resolved
) # as per sglang website: https://sgl-project.github.io/start/install.html
.run_function( # download the model by running a Python function
download_model_to_image
)
Expand Down