MMLU-FR

[vince62s]

Hello,
Obviously you are working with the huggingface format.
In the OpenNMT-py repo we have tools to convert the original Llama / Mistral (ie consolidated.00.pth) model into the openNMT-py format, and we have the script to run the MMLU eval. We also translated the MMLU dataset into French to get a similar score.

My all point is that you can run MMLU with some huggingface formated models (like the Eleuther tools) but to run it with our MMLU-FR dataset it would require some small adaptation.

Just wondering if your finetuned vigo stuff (llama or mistral) perfoms better then plain models on MMLU-FR

Just in case you're interested to test, contact me on our repo. Cheers.