Reproduce the given benchmark results

[yeliang2258]

Hello，I would like to know which model and test tool are used for the performance data provided on the homepage?

Is the model available on https://huggingface.co/?

[yeliang2258]

lmsys/vicuna-13b-v1.3, young-geng/koala, openlm-research/open_llama_13b, or others ?

[yeliang2258]

@zhuohan123 Please help me solve this problem, thank you

[zhuohan123]

Hi we use the official LLaMA model (e.g., huggyllama/llama-13b). However, all these llama models you listed should perform exactly the same.

[zhuohan123]

We use this script to get the benchmark results: https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_serving.py

[yeliang2258]

https://github.com/vllm-project/vllm/blob/main/benchmarks/launch_tgi_server.sh
In addition, I would like to ask, in this script, what is the TOKENS setting?