Query about GSM8K evaluation

[HCY123902]

In Table 9 of the paper, your evaluation on GSM8K seems to use the 5 shot setting. May I know which evaluation library did you use? Is it lm-evaluation-harness or other existing GitHub implementations

[yumeng5]

Hi @HCY123902

Yes it's the default lm-evaluation-harness evaluation framework.

Best,
Yu

[HCY123902]

Thank you for your reply. I am trying to reproduce the results on GSM8K, and this is what I observed using lm-evaluation-harness v0.4.5.

Method	Setting	Result
LLAMA-3-8B-Instruct	5-shot + strict-match	75.74
+ DPO v0.1	5-shot + strict-match	71.27
+ SimPO v0.1	5-shot + strict-match	71.95

These results seem significantly higher than what is reported on Table 9 of the paper. Therefore, may I know your lm-evaluation-harness version and the exact settings used to reproduce the GSM8K results

[xiamengzhou]

Hi @HCY123902, apologies for the delayed response!

For testing GSM8k, we used the following command and evaluated with the git version b281b0921b636bc36ad05c0b0b0763bd6dd43463:

cd lm-evaluation-harness $1=princeton-nlp/Llama-3-Instruct-8B-SimPO python3 main.py --model=hf --model_args="pretrained=$1,dtype=bfloat16" --tasks=gsm8k --num_fewshot=5 --batch_size=4 --output_path=${OUTPUT_PATH} --no_cache

Let me know if you have any further questions!

[yujiaw98]

Hi @xiamengzhou @yumeng5,

Regarding this issue, I have a quick question: Between the two evaluation metrics for GSM8K, strict-match and flexible-extract, which one would you prefer to highlight as the primary evaluation metric in your paper?

Best,
Yujia