llama2-70b

[ Back to MLPerf inference benchmarks index ]

MLPerf inference benchmark

Text summarization with Llama2-70b

Notes

Llama2-70b has two variants - llama2-70b-99 and llama2-70b-99.9 where the 99 and 99.9 specify the required accuracy constraint with respect to the reference fp32 model. Llama2-70b applies only to datacenter category and includes both Offline and Server scenarios.

Please check MLPerf inference GitHub for more details.

Run using the MLCommons CM framework

From Feb 2024, we suggest you to use this GUI to configure MLPerf inference benchmark, generate CM commands to run it across different implementations, models, data sets, software and hardware, and prepare your submissions.

A few ready-to-use CM commands

Install MLCommons CM automation framework with automation recipes for MLPerf as described here.

The following guides explain how to run different implementations of this benchmark via CM:

• MLCommons Reference implementation in Python

Questions? Suggestions?

Check the MLCommons Task Force on Automation and Reproducibility and get in touch via public Discord server.