e5-mistral-7b-instruct embedding and inference in the same VLLM instance, is it possible? #9703
micuentadecasa
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Is it possible to run an instance of e5-mistral-7b-instruct with VLLM and use it at the same time for embedding and inference?
I don't' want to instantiate two instances that will exhaust my GPU, so it will be great if the same instance can be used for both tasks.
Beta Was this translation helpful? Give feedback.
All reactions