Does vllm support CPU?

[gokul427]

Can we use vllm only on CPU without GPU machine?

[mspronesti]

I think the short answer is no, as vLLM's engine relies on custom kernels written in CUDA.

[BrightXiaoHan]

You can try ctranslate2 or llama.cpp.

[VpkPrasanna]

obviously i know this library has been designed on top of cuda kernels for better GPU utilization , but the inference will pretty slow right on CPU? is there any workaround to load these quantized models on CPU to server faster ?
#2326

[hughesadam87]

This was not clear to me either - any way to highlight this in bold somewhere on main docs? Sorry if I overlooked.

I am trying to do some local testing - that's my use case