About the update of environment.yml

[lucasliunju]

Hi,

I try to follow the configuration of environment.yml and cannot run the code successfully. I noticed that the simpo code has some update and I would like to ask whether we need to update the environment.yml

Thank you very much in advance.

[yumeng5]

Hi @lucasliunju

The environment file should be good for Mistral and Llama3 models. If you are running Gemma2 models, you probably need to upgrade the versions of torch and trl to their latest version. We're working on compiling an upgraded environment file that ensures compatibility for all the models and will provide an update soon.

Best,
Yu

[lucasliunju]

Hi @yumeng5 ,

Thank you very much for you reply! I can run the code now. By the way, I try to use alpaca-eval to evaluate the performance of SimPO and I find the openai API is very expensive for students. So I would like to ask whether you have some suggestions on that.

Thank you very much!

[xiamengzhou]

@lucasliunju

We've updated the environment info! For AlpacaEval, we find it necessary to use close source models for judging (i.e., GPT, Claude), as there are significant inconsistencies in preferences between open-source models like llama3 and GPT 😢.