performance of using tensor parallelism vs using one GPU

[butyuhao]

Using tensor parallelism vs using one GPU, which is faster? Should I use tp when the model can fit into one GPU?

[phucdoitoan]

I observed that TP is slightly faster (?). I test with llama-2-13b-chat-hf.
For offline inference, only 1 prompt: on 1 GPU, vllm has about 40 tokens/s. on 2 GPU it has 45 toknes/s.
For offline inference, 5 prompts, it is 165 token/s for 1 GPU, and 270 tokens/s for 2 GPU.

But, when TP on 2 GPUs, the model takes up an absurd amount of memory, 36.5 GB each, even though dtype is bfloat16 ...

[ywen666]

When my model can fit into one GPU, it seems # GPUs * throughput is always higher than using TP=# GPUs. So if # requests is always greater than # GPUs, I think using separable process on each GPU gives me the best performance. Please correct me if I am wrong.