Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mistral 7B with vLLM, Ray Serve on Trn/Inf #650

Open
askulkarni2 opened this issue Sep 13, 2024 · 0 comments
Open

Mistral 7B with vLLM, Ray Serve on Trn/Inf #650

askulkarni2 opened this issue Sep 13, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request gen-ai pattern Distributed Training and Inference Patterns for Various Generative AI Large Language Models (LLMs)

Comments

@askulkarni2
Copy link
Collaborator

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

A pattern that demonstrates running the Mistral 7B model with the cheapest Neuron instances.

Describe the solution you would like

Similar to the llama3-8B-Instruct pattern but for Mistral instead

@askulkarni2 askulkarni2 added enhancement New feature or request gen-ai pattern Distributed Training and Inference Patterns for Various Generative AI Large Language Models (LLMs) labels Sep 13, 2024
@askulkarni2 askulkarni2 self-assigned this Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request gen-ai pattern Distributed Training and Inference Patterns for Various Generative AI Large Language Models (LLMs)
Projects
None yet
Development

No branches or pull requests

1 participant