🗺️ Implementation DiscoPOP Loss

[fanconic]

What does this PR do?

This PR implements the DiscoPOP loss. The DiscoPOP loss, also described in the paper as "Log Ratio Modulated Loss" (LRML), was discovered through an LLM discovery process to find optimization functions for offline preference optimization. The DiscoPOP loss outperformed (or performed competitively to) traditional offline preference optimization functions, such as DPO and Hinge loss, on different evaluation tasks (IMDb positive text generation, TL;DR text summarization, and Alpaca Eval 2.0).

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Additional Notes

About tests, I have added tests to the best of my knowledge to make sure that this feature works. While test failures occurred in unrelated parts of the code, the ones I implemented passed successfully. Please let me know if something needs to be fixed.

[qgallouedec]

Suggested change

	### DiscoPOP loss
	The [DiscoPOP](https://huggingface.co/papers/2406.08414) paper uses LLMs to discover more efficient offline preference optimization losses. In the paper the proposed DiscoPOP loss (which is a log-ratio modulated loss) outperformed other optimization losses on different tasks (IMDb positive text generation, Reddit TLDR summarization, and Alpaca Eval 2.0). To use this discovered loss, set the `loss_type` value to `discopop` in the [`DPOConfig`]. Additionally, you can change the `discopop_tau` value to change the shape of the DiscoPOP loss. However, the authors recommed the default value `discopop_tau=0.05`.

Can you make the final remark about discopop_tau in the table instead?

[qgallouedec]

It's sorted by date

Suggested change

	- `"apo_zero"`: APO-zero loss from the [APO](https://huggingface.co/papers/2408.06266) paper.
	- `"apo_down"`: APO-down loss from the [APO](https://huggingface.co/papers/2408.06266) paper.
	- `"discopop"`: DiscoPOP (a.k.a Log-Ratio Modulated Loss, LRML) loss from the [DiscoPOP](https://huggingface.co/papers/2406.08414) paper.
	- `"discopop"`: DiscoPOP (a.k.a Log-Ratio Modulated Loss, LRML) loss from the [DiscoPOP](https://huggingface.co/papers/2406.08414) paper.
	- `"apo_zero"`: APO-zero loss from the [APO](https://huggingface.co/papers/2408.06266) paper.
	- `"apo_down"`: APO-down loss from the [APO](https://huggingface.co/papers/2408.06266) paper.

[qgallouedec]

Why not "lrm" instead by the way?

[fanconic]

I don't really have a clear answer to this. While lrml was the name proposed by the LLM during discovery, the authors agreed to name the best performing one DiscoPOP, which seemed to us like a catchy abbreviation for Discovered Preference Optimization

[qgallouedec]

I don't have strong opinion on this, but LRML might be more informative than DiscoPOP. I mean, we could have a lot of different Discovered Preference Optimization losses.

[qgallouedec]

please remove this file

[fanconic]

Sorry, this slipped in there when adding changed files. Good catch

[qgallouedec]

Thanks @fanconic! Do you have reference results to share?

[qgallouedec]

Thanks for contributing @fanconic 👊

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.