New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vllm能够直接调用gguf格式的权重吗 #262

Open

liguoyu666 opened this issue Sep 29, 2024 · 1 comment

liguoyu666 commented Sep 29, 2024

我使用04-Qwen2-7B-Instruct vLLM 部署调用.md里的调用方法，把模型权重换为gguf格式，报错，是要自己重新写调用方法吗

Contributor

KMnO4-zx commented Oct 4, 2024

应该是可以的，可以看下vllm的官方文档

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment