How can I run VLLM serving without an internet connection?

[ZPerling]

I cloned the model repository on Hugging Face to my local machine and used the --download-dir parameter to specify the directory. However, when running VLLM, it still tries to connect to Hugging Face, which doesn't work without an internet connection. Even after setting export HF_HUB_OFFLINE=1, offline mode doesn't seem to be working. Is it possible to run VLLM offline and if so, how can I achieve this?

[MarkoMFilip]

We're facing the same issue, any feedback here would be appreciated. Running vLLM online is currently not possible for us...

[Martyprod]

same issue, i'm offline 99% of the time and VLLM is not working offline :(.

[Moran232]

can you show the command you run vlllm?

[Martyprod]

I'm running it in Stability matrix, so can it run totally offline ? I can install it locally if the issue is related to stability matrix itself !
?

[HYZ17]

Using absolute path to load the model, rather than the name on huggingface, works for me. For example,

import os 
os.environ["CUDA_VISIBLE_DEVICES"] = "5"
from vllm import LLM

prompts = ["Hello, my name is", "The capital of France is"]  # Sample prompts.
#llm = LLM(model="/home/.cache/huggingface/hub/models--huggyllama--llama-13b/snapshots/bf57045473f207bb1de1ed035ace226f4d9f9bba",)
outputs = llm.generate(prompts)  # Generate texts from the prompts.
print(outputs)

[llmwesee]

It works but the generated responses are wrong most of the time.

[smahdavi4]

Setting the hugging face transformers module to offline mode (via a global environment variable) worked for me: TRANSFORMERS_OFFLINE=1

[juud79]

same...

[bzr1]

this is not working

[yw0nam]

It works in vllm 0.5.5 with docker. Thanks.

[nitsan-airis]

hey, 0.6.2 version here,

docker run --gpus all -p 8005:8005 -v /abs/path/to/cached/model:/app/model_data --env "TRANSFORMERS_OFFLINE=1" vllm/vllm-openai:v0.6.2 --model meta-llama/Llama-3.1-8B --port 8005 --download_dir /app/model_data

did not work. Specifically got
huggingface_hub.errors.OfflineModeIsEnabled: Cannot reach https://huggingface.co/meta-llama/Llama-3.1-8B/resolve/main/config.json: offline mode is enabled. To disable it, please unset the HF_HUB_OFFLINE environment variable.