Expected speed up over HuggingFace

[deepakdalakoti]

Hi,

As per the vllm homepage the results show a huge speed up over huggingface API. However, my benchmarking results show only a modest improvement over Huggingface (~15% ). I am using Llama-2-7b-hf for testing. I used the code and prompt described here for testing. Is anyone seeing similar results? Are there some settings which should improve the results significantly? I am using the following libraries

CUDA: 11.8
vllm: 0.1.3

[deelipvenkat]

same here , not seeing much improvement in latency , although batch inferencing is much faster with vllm compared huggingface.