Support user-defined prompt processing strategies for dpo

[nopperl]

Description

Support user-defined prompt pre-processing strategies for DPO training similar to instruction finetuning (in src/axolotl/prompt_strategies/user_defined.py). Specifically, it supports defining different names and a format string for the prompt, chosen and reject fields. Example config:

datasets:
  - ds_type: json
    data_files:
      - test_dpo.jsonl
    split: train
    type: user_defined.default
    field_system: sys
    field_prompt: question
    field_chosen: correct
    field_rejected: wrong
    prompt_format: "<|im_start|>system\n{system}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\nThe answer in JSON format is:"
    chosen_format: "{chosen}<|im_end|>"
    rejected_format: "{rejected}<|im_end|>"

How has this been tested?

Tested it on a custom dataset, might make sense to add test cases.

Types of changes

[winglian]

this is great so far. For consistency, It would be great if the structure matched the SFT format:
https://github.com/OpenAccess-AI-Collective/axolotl/blob/main/README.md?plain=1#L388-L400

datasets:
  - path: repo
    type:
      system_prompt: ""
      field_system: system
      field_instruction: input
      field_output: output
      format: "[INST] {instruction} [/INST]"
      no_input_format: "[INST] {instruction} [/INST]"

[nopperl]

Ah, I missed that. I have adapted it, now it expects the following config:

datasets:
  - path: repo
    split: train
    type:
      field_system: sys
      field_prompt: question
      field_chosen: correct
      field_rejected: wrong
      prompt_format: "<|im_start|>system\n{system}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\nThe answer in JSON format is:"
      chosen_format: "{chosen}<|im_end|>"
      rejected_format: "{rejected}<|im_end|>"

[nopperl]

@winglian I've fixed the build errors, I think this can be merged now.

[winglian]

@nopperl I rebased this and updated the pydantic models for validation.

[winglian]

lgtm, @NanoCode012 ?