Skip to content

Commit

Permalink
review: make instruction simple
Browse files Browse the repository at this point in the history
  • Loading branch information
deeplearningfromscratch committed Apr 12, 2024
1 parent 8de5357 commit ec16187
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 22 deletions.
1 change: 0 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# end-to-end quantized model evaluation
.PHONY: qgpt-j
qgpt-j:
-bash scripts/build_qgpt-j_env.sh
-bash scripts/eval_qgpt-j.sh

# end-to-end evaluation
Expand Down
34 changes: 13 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,27 +34,19 @@ TBA
| ROUGEL | 30.0462 (100.19%) | [29.9881](https://github.com/mlcommons/inference/blob/e39003a9c4c89a2215db0ca57ad7a57b16f9a785/tools/submission/submission_checker.py#L1126C92-L1126C99) |
| GEN_LEN | 3971863 (98.88%) | [4016878](https://github.com/mlcommons/inference/blob/e39003a9c4c89a2215db0ca57ad7a57b16f9a785/tools/submission/submission_checker.py#L1126C120-L1126C127) |

- To reproduce the result, you can run one of the following commands:

1.
```
make qgpt-j
```
2.
```
. scripts/build_qgpt-j_env.sh
. scripts/eval_qgpt-j.sh
```
3.
```
export SCENARIO=Offline # SCENARIO is one of [Offline, SingleStream, Server]
export N_COUNT=10833 # N_COUNT is a number between [1, 13368]
export CALIBRATE=false # CALIBRATE is one of [false, true]
export N_CALIB=100 # N_CALIB is a number between [1, 1000]
make qgpt-j
```
To reproduce the above accuracy result, please run as follows:

```
. scripts/build_qgpt-j_env.sh # You can skip this step if you have already set up the environment.
make qgpt-j
```

For other evaluations with different settings, you can also change the following environment variables. Each configuration is defined as follows:

* `SCENARIO` is a MLPerf scenario to evaluate accuracy. It can be one of Offline, SingleStream, or Server.
* `N_COUNT`: the number of data to evaluate the accuracy. ([1, 13368])
* `CALIBRATE`: the calibration is involved in the evaluation step if true. (default: false)
* `N_CALIB`: the number of data to calibrate the quantized model. ([1, 1000])

### Wi8Ai8KVi8 quantized LLaMA2-70b (qLLaMA2-70b)

Expand Down

0 comments on commit ec16187

Please sign in to comment.