Set Different GPU Memory Utilization for Each GPU

[MohmedMonsef]

I have two GPUs with varying workloads, so one might have more memory than the other. Is there a way to configure vllm to use different amounts of memory for each GPU individually?

I am using

• pipeline_parallel_size: 1
• tensor_parallel_size: 2
  as my two GPUs are located on one node.

And according to my understanding, the same gpu-memory-utilization setting is applied to all GPUs.

[DarkLight1337]

No, this is not possible. It would complicate things too much to support this.