fix: remove depregated arguments from DPOtrainer

[sorydi3]

December 2024 Student Submission

Module Completed

• Module 1: Instruction Tuning
• Module 2: Preference Alignment
• Module 3: Parameter-efficient Fine-tuning
• Module 4: Evaluation
• Module 5: Vision-language Models
• Module 6: Synthetic Datasets
• Module 7: Inference
• Module 8: Deployment

Changes Made

Describe what you've done in this PR:

1. What concepts did you learn?
2. What changes or additions did you make?

I've removed some deprecated arguments from DPOtrainer class that was causing some errors to the notebook, I discovered that in the latest version of TRL (0.13.0), these parameters have been moved to the DPOConfig class.

To resolve the issue, the parameters causing errors need to be moved to the DPOConfig definition.

config = DPOConfig( ... beta=0.1, # DPO-specific temperature parameter max_prompt_length=1024, # Maximum length of the input prompt in tokens max_length=1536 # Maximum combined length of prompt + response in tokens )

https://github.com/huggingface/trl/blob/4c71daf461d86226ee36c30531d481c33c3e618e/trl/trainer/dpo_trainer.py#L149

5. Any challenges you faced?

Notebooks Added/Modified

List any notebooks you've added or modified:

• Added new example in module_name/student_examples/my_example.ipynb
• Modified existing notebook with additional examples
• Added documentation or comments

2_preference_alignment/notebooks/dpo_finetuning_example.ipynb

Checklist

• I have read the module materials
• My code runs without errors
• I have pushed models and datasets to the huggingface hub
• My PR is based on the december-2024 branch

Questions or Discussion Points

Add any questions you have or points you'd like to discuss:
1.
2.

Additional Notes

Any other information that might be helpful for reviewers:

Here is the issue. Issue

[Knight7561]

Please review #124, and check if that solves the issue.