update vllm deployment and add a local testing script

[omrishiv]

Going to go through the files as they sit in the PR to explain:

Dockerfile:

• Move the stuff that doesn't change to the top of the dockerfile and condense the run command. Fewer layers, and if you have to iterate on the file, this will speed up your build/pushes. More on that in a minute.

ray-service-vllm.yaml

• Since the head node isn't doing much, at least for this example, drop the CPU count
• Added the observability variables to enable the dashboards
• Put the HF secret on the worker. Since the head node deploys the file to the worker, which is what runs it, the worker needs the ENVVAR.

serve.py

• A simple serving test script. Change the port to whatever port you port-forward

vllm_serve.py

• not sure if you need to make the change to how to get the envvar, but more importantly:
• Drop the model max_model_len to fit the KV cache. You'll get an error if you don't.

Some other general tips I've found along the way:

• Add karpenter.sh/do-not-disrupt: true annotations to your head/worker nodes so karpenter doesn't scale them back each time you need to restart your pod for changing the script
• Create a configmap for your python model and mount that on the head node where you put it in your Dockerfile. This is much faster than having to constantly update the code, build the image and push the image.

[omrishiv]

Take a look at serve.py and update the gradio script. I had to make a change there on your request.get. It should be a POST with a json dictionary that uses a different key

[vara-bonthu]

This blueprint also deploys Kube Prometheus stack. Can we update these to localhost services with the port?

[omrishiv]

I don't know what those are for your deployment. I'll let @ratnopamc add them

[vara-bonthu]

we should be able to send all the vLLM config for each model