[SUBMISSION] December 2024 - Module 2: Preference Alignment

.github/workflows/gitleaks.yml

@@ -0,0 +1,16 @@
+name: gitleaks
+on:
+  pull_request:
+  push:
+  workflow_dispatch:
+jobs:
+  scan:
+    name: gitleaks
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: gitleaks/gitleaks-action@v2
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

1_instruction_tuning/supervised_fine_tuning.md

@@ -16,7 +16,7 @@ For example, if you're developing a customer service application, you might want
 
 ## The Fine-Tuning Process
 
-The supervised fine-tuning process involves training model weights on a task-specific dataset. 
+The supervised fine-tuning process involves adjusting a model's weights on a task-specific dataset. 
 
 First, you'll need to prepare or select a dataset that represents your target task. This dataset should include diverse examples that cover the range of scenarios your model will encounter. The quality of this data is important - each example should demonstrate the kind of output you want your model to produce. Next comes the actual fine-tuning phase, where you'll use frameworks like Hugging Face's `transformers` and `trl` to train the model on your dataset. 
 

2_preference_alignment/notebooks/dpo_finetuning_example.ipynb

@@ -60,7 +60,7 @@
     "import os\n",
     "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
     "from datasets import load_dataset\n",
-    "from trl import DPOTrainer, DPOConfig\n"
+    "from trl import DPOTrainer, DPOConfig"
    ]
   },
   {
@@ -122,9 +122,7 @@
     "device = (\n",
     "    \"cuda\"\n",
     "    if torch.cuda.is_available()\n",
-    "    else \"mps\"\n",
-    "    if torch.backends.mps.is_available()\n",
-    "    else \"cpu\"\n",
+    "    else \"mps\" if torch.backends.mps.is_available() else \"cpu\"\n",
     ")\n",
     "\n",
     "# Model to fine-tune\n",
@@ -269,13 +267,20 @@
     "    # Use bfloat16 precision for faster training\n",
     "    bf16=True,\n",
     "    # Disable wandb/tensorboard logging\n",
-    "    report_to=None,\n",
+    "    report_to=\"none\",\n",
     "    # Keep all columns in dataset even if not used\n",
     "    remove_unused_columns=False,\n",
     "    # Enable MPS (Metal Performance Shaders) for Mac devices\n",
     "    use_mps_device=device == \"mps\",\n",
     "    # Model ID for HuggingFace Hub uploads\n",
     "    hub_model_id=finetune_name,\n",
+    "    # DPO-specific temperature parameter that controls the strength of the preference model\n",
+    "    # Lower values (like 0.1) make the model more conservative in following preferences\n",
+    "    beta=0.1,\n",
+    "    # Maximum length of the input prompt in tokens\n",
+    "    max_prompt_length=1024,\n",
+    "    # Maximum combined length of prompt + response in tokens\n",
+    "    max_length=1536,\n",
     ")"
    ]
   },
@@ -296,11 +301,11 @@
     "    processing_class=tokenizer,\n",
     "    # DPO-specific temperature parameter that controls the strength of the preference model\n",
     "    # Lower values (like 0.1) make the model more conservative in following preferences\n",
-    "    #beta=0.1,\n",
+    "    # beta=0.1,\n",
     "    # Maximum length of the input prompt in tokens\n",
-    "    #max_prompt_length=1024,\n",
+    "    # max_prompt_length=1024,\n",
     "    # Maximum combined length of prompt + response in tokens\n",
-    "    #max_length=1536,\n",
+    "    # max_length=1536,\n",
     ")"
    ]
   },
@@ -357,7 +362,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.7"
+   "version": "3.11.10"
   },
   "widgets": {
    "application/vnd.jupyter.widget-state+json": {

2_preference_alignment/notebooks/orpo_finetuning_example.ipynb

@@ -348,7 +348,7 @@
     "    # Gradual learning rate warmup\n",
     "    warmup_steps=10,\n",
     "    # Disable external logging\n",
-    "    report_to=None,\n",
+    "    report_to=\"none\",\n",
     "    # Where to save model/checkpoints\n",
     "    output_dir=\"./results/\",\n",
     "    # Enable MPS (Metal Performance Shaders) if available\n",