part_6/04_preference-tuning-with-dpo/create-preference-data-ollama.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b1910a06-e8a3-40ac-8201-ff70615b1ba4",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Generating A Preference Dataset With Llama 3.2 And Ollama"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a128651b-f326-4232-a994-42f38b7ed520",
   "metadata": {},
   "source": [
    "- Preference finetuning is a process to align an instruction-finetuned LLM with human preferences\n",
    "- There are multiple ways to create a dataset for preference finetuning an LLM\n",
    "  1. We use the instruction-finetuned LLM to generate multiple responses and have humans rank them based on their preference and/or given preference criteria\n",
    "  2. We use the instruction-finetuned LLM to generate multiple responses and have LLMs rank them based on given preference criteria\n",
    "  3. We use an LLM to generate preferred and dispreferred responses given certain preference criteria\n",
    "- In this notebook, we consider approach 3\n",
    "- This notebook uses Llama 3.2-Instruct model through ollama to generate preference labels for an instruction dataset\n",
    "- The expected format of the instruction dataset is as follows:\n",
    "\n",
    "\n",
    "### Input\n",
    "\n",
    "```json\n",
    "[\n",
    "    {\n",
    "        \"instruction\": \"What is the state capital of California?\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"The state capital of California is Sacramento.\",\n",
    "    },\n",
    "    {\n",
    "        \"instruction\": \"Provide a synonym for 'fast'.\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"A synonym for 'fast' is 'quick'.\",\n",
    "    },\n",
    "    {\n",
    "        \"instruction\": \"What is the capital of Greece?\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"The capital of Greece is Athens.\",\n",
    "\n",
    "    },\n",
    "...\n",
    "]\n",
    "```\n",
    "\n",
    "The output dataset will look as follows, where more polite responses are preferred (`'chosen'`), and more impolite responses are dispreferred (`'rejected'`):\n",
    "\n",
    "```json\n",
    "[\n",
    "    {\n",
    "        \"instruction\": \"What is the state capital of California?\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"The state capital of California is Sacramento.\",\n",
    "        \"rejected\": \"Look, the state capital of California is obviously Sacramento.\",\n",
    "        \"chosen\": \"The state capital of California is Sacramento.\"\n",
    "    },\n",
    "    {\n",
    "        \"instruction\": \"Provide a synonym for 'fast'.\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"A synonym for 'fast' is 'quick'.\",\n",
    "        \"chosen\": \"A suitable alternative to 'fast' would be 'quick'.\",\n",
    "        \"rejected\": \"A synonym for 'fast' is 'quick'.\"\n",
    "    },\n",
    "    {\n",
    "        \"instruction\": \"What is the capital of Greece?\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"The capital of Greece is Athens.\",\n",
    "        \"chosen\": \"I'd be happy to help! The capital of Greece is indeed Athens.\",\n",
    "        \"rejected\": \"The capital of Greece is Athens.\"\n",
    "    },\n",
    "...\n",
    "]\n",
    "```\n",
    "\n",
    "### Output\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "- The code doesn't require a GPU and runs on a laptop given enough RAM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63610acc-db94-437f-8d38-e99dca0299cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\"tqdm\",    # Progress bar\n",
    "        ]\n",
    "\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bcdcb34-ac75-4f4f-9505-3ce0666c42d5",
   "metadata": {},
   "source": [
    "## Installing Ollama and Downloading Llama 3.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a092280-5462-4709-a3fe-8669a4a8a0a6",
   "metadata": {},
   "source": [
    "- Ollama is an application to run LLMs efficiently\n",
    "- It is a wrapper around [llama.cpp](https://github.com/ggerganov/llama.cpp), which implements LLMs in pure C/C++ to maximize efficiency\n",
    "- Note that it is a tool for using LLMs to generate text (inference), not training or finetuning LLMs\n",
    "- Prior to running the code below, install ollama by visiting [https://ollama.com](https://ollama.com) and following the instructions (for instance, clicking on the \"Download\" button and downloading the ollama application for your operating system)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9558a522-650d-401a-84fc-9fd7b1f39da7",
   "metadata": {},
   "source": [
    "- For macOS and Windows users, click on the ollama application you downloaded; if it prompts you to install the command line usage, say \"yes\"\n",
    "- Linux users can use the installation command provided on the ollama website\n",
    "\n",
    "- In general, before we can use ollama from the command line, we have to either start the ollama application or run `ollama serve` in a separate terminal\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_6/images/ollama-serve.webp?1\">\n",
    "\n",
    "\n",
    "- With the ollama application or `ollama serve` running, in a different terminal, on the command line, execute the following command to try out the Llama 3.2 model \n",
    "\n",
    "```bash\n",
    "\n",
    "ollama run llama3.2\n",
    "```\n",
    "\n",
    "\n",
    "The output looks like as follows:\n",
    "\n",
    "```\n",
    "$ ollama run llama3.2\n",
    "pulling manifest\n",
    "pulling aa81b541aae6... 100% ▕████████████████▏ 39 GB\n",
    "pulling 8cf247399e57... 100% ▕████████████████▏ 1.7 KB\n",
    "pulling f1cd752815fc... 100% ▕████████████████▏ 12 KB\n",
    "pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B\n",
    "pulling 3c1c2d3df5b3... 100% ▕████████████████▏ 486 B\n",
    "verifying sha256 digest\n",
    "writing manifest\n",
    "removing any unused layers\n",
    "success\n",
    "```\n",
    "\n",
    "- Note that `llama3.2` refers to the instruction finetuned Llama 3.2 3B-Instruct model\n",
    "\n",
    "- Alternatively, you can also use the smaller, more resource-effiicent Llama 3.1 model, by replacing `llama3.2` with `llama3.1`\n",
    "\n",
    "- After the download has been completed, you will see a command line prompt that allows you to chat with the model\n",
    "\n",
    "- Try a prompt like \"What do llamas eat?\", which should return an output similar to the following:\n",
    "\n",
    "```\n",
    ">>> What do llamas eat?\n",
    "Llamas are ruminant animals, which means they have a four-chambered \n",
    "stomach and eat plants that are high in fiber. In the wild, llamas \n",
    "typically feed on:\n",
    "1. Grasses: They love to graze on various types of grasses, including tall \n",
    "grasses, wheat, oats, and barley.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b5addcb-fc7d-455d-bee9-6cc7a0d684c7",
   "metadata": {},
   "source": [
    "- You can end this session using the input `/bye`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda155ee-cf36-44d3-b634-20ba8e1ca38a",
   "metadata": {},
   "source": [
    "## Using Ollama's REST API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89343a84-0ddc-42fc-bf50-298a342b93c0",
   "metadata": {},
   "source": [
    "- Now, an alternative way to interact with the model is via its REST API in Python via the following function\n",
    "- Before you run the next cells in this notebook, make sure that ollama is still running, as described above, via\n",
    "  - `ollama serve` in a terminal\n",
    "  - the ollama application\n",
    "- Next, run the following code cell to query the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16642a48-1cab-40d2-af08-ab8c2fbf5876",
   "metadata": {},
   "source": [
    "- First, let's try the API with a simple example to make sure it works as intended:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "65b0ba76-1fb1-4306-a7c2-8f3bb637ccdb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Llamas are herbivores, which means they primarily eat plants and plant-based foods. Their diet typically consists of:\n",
      "\n",
      "1. Grasses: Llamas love to graze on various types of grasses, including tall grasses, short grasses, and grassy weeds.\n",
      "2. Hay: High-quality hay, such as timothy hay or alfalfa hay, is a staple in a llama's diet. It provides essential nutrients like fiber, protein, and vitamins.\n",
      "3. Grains: Llamas may also be fed grains like oats, barley, or corn, but these should not make up more than 10% of their diet.\n",
      "4. Fruits and vegetables: Fresh fruits and vegetables, such as apples, carrots, and sweet potatoes, can be given to llamas as treats or added to their hay.\n",
      "5. Browse: Llamas may also eat browse, which includes leaves, twigs, and other vegetation from trees and shrubs.\n",
      "\n",
      "It's essential to note that llamas have a unique digestive system, with a four-chambered stomach, which allows them to break down and extract nutrients from plant material more efficiently than many other animals. However, this also means they can be prone to certain health issues if their diet is not balanced or if they eat too much of the wrong foods.\n",
      "\n",
      "A good rule of thumb for llama owners is to provide a high-quality hay-based diet with limited amounts of grains and treats, and to ensure access to fresh water at all times.\n"
     ]
    }
   ],
   "source": [
    "import urllib.request\n",
    "import json\n",
    "\n",
    "\n",
    "def query_model(prompt, model=\"llama3.2\", url=\"http://localhost:11434/api/chat\"):\n",
    "    # Create the data payload as a dictionary\n",
    "    data = {\n",
    "        \"model\": model,\n",
    "        \"messages\": [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": prompt\n",
    "            }\n",
    "        ],\n",
    "        \"options\": {\n",
    "            \"seed\": 123,\n",
    "            \"temperature\": 0,\n",
    "        }\n",
    "    }\n",
    "\n",
    "    # Convert the dictionary to a JSON formatted string and encode it to bytes\n",
    "    payload = json.dumps(data).encode(\"utf-8\")\n",
    "\n",
    "    # Create a request object, setting the method to POST and adding necessary headers\n",
    "    request = urllib.request.Request(url, data=payload, method=\"POST\")\n",
    "    request.add_header(\"Content-Type\", \"application/json\")\n",
    "\n",
    "    # Send the request and capture the response\n",
    "    response_data = \"\"\n",
    "    with urllib.request.urlopen(request) as response:\n",
    "        # Read and decode the response\n",
    "        while True:\n",
    "            line = response.readline().decode(\"utf-8\")\n",
    "            if not line:\n",
    "                break\n",
    "            response_json = json.loads(line)\n",
    "            response_data += response_json[\"message\"][\"content\"]\n",
    "\n",
    "    return response_data\n",
    "\n",
    "\n",
    "result = query_model(\"What do Llamas eat?\")\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "162a4739-6f03-4092-a5c2-f57a0b6a4c4d",
   "metadata": {},
   "source": [
    "## Load JSON Entries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca011a8b-20c5-4101-979e-9b5fccf62f8a",
   "metadata": {},
   "source": [
    "- Now, let's get to the data generation part\n",
    "- Here, for a hands-on example, we use the `instruction-data.json` file that we originally used to instruction-finetune the model in part 6:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b2d393a-aa92-4190-9d44-44326a6f699b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "json_file = Path(\"..\", \"01_main-code\", \"instruction-data.json\")\n",
    "\n",
    "with open(json_file, \"r\") as file:\n",
    "    json_data = json.load(file)\n",
    "\n",
    "print(\"Number of entries:\", len(json_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6c9751b-59b7-43fe-acc7-14e8daf2fa66",
   "metadata": {},
   "source": [
    "- The structure of this file is as follows, where we have the given response in the test dataset (`'output'`) that we trained the model to generate via instruction finetuning based on the `'input'` and `'instruction'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7222fdc0-5684-4f2b-b741-3e341851359e",
   "metadata": {},
   "outputs": [],
   "source": [
    "json_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcf0331b-6024-4bba-89a9-a088b14a1046",
   "metadata": {},
   "source": [
    "- Below is a small utility function that formats the instruction and input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43263cd3-e5fb-4ab5-871e-3ad6e7d21a8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_input(entry):\n",
    "    instruction_text = (\n",
    "        f\"Below is an instruction that describes a task. Write a response that \"\n",
    "        f\"appropriately completes the request.\"\n",
    "        f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
    "    )\n",
    "\n",
    "    input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
    "    instruction_text + input_text\n",
    "\n",
    "    return instruction_text + input_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a55283-7d51-4136-ba60-f799d49f4098",
   "metadata": {},
   "source": [
    "- Now, let's try the ollama API to generate a `'chosen'` and `'rejected'` response for preference tuning a model\n",
    "- Here, to for illustration purposes, we create answers that are more or less polite\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "735cc089-d127-480a-b39d-0782581f0c41",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "\n",
    "for entry in json_data[:5]:\n",
    "    \n",
    "    politeness = random.choice([\"polite\", \"impolite\"])    \n",
    "    prompt = (\n",
    "        f\"Given the input `{format_input(entry)}` \"\n",
    "        f\"and correct output `{entry['output']}`, \"\n",
    "        f\"slightly rewrite the output to be more {politeness}.\"\n",
    "        \"Keep the modification minimal.\"\n",
    "        \"Only return return the generated response and nothing else.\"\n",
    "    )\n",
    "    print(\"\\nDataset response:\")\n",
    "    print(\">>\", entry['output'])\n",
    "    print(f\"\\n{politeness} response:\")\n",
    "    print(\">>\", query_model(prompt))    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142dfaa7-429f-4eb0-b74d-ff327f79547a",
   "metadata": {},
   "source": [
    "- If we find that the generated responses above look reasonable, we can go to the next step and apply the prompt to the whole dataset\n",
    "- Here, we add a `'chosen'` key for the preferred response and a `'rejected'` response for the dispreferred response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3349dbbc-963f-4af3-9790-12dbfdca63c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from tqdm import tqdm\n",
    "\n",
    "def generate_model_responses(json_data):\n",
    "\n",
    "    for i, entry in enumerate(tqdm(json_data, desc=\"Writing entries\")):\n",
    "        politeness = random.choice([\"polite\", \"impolite\"])    \n",
    "        prompt = (\n",
    "            f\"Given the input `{format_input(entry)}` \"\n",
    "            f\"and correct output `{entry['output']}`, \"\n",
    "            f\"slightly rewrite the output to be more {politeness}.\"\n",
    "            \"Keep the modification minimal.\"\n",
    "            \"Only return return the generated response and nothing else.\"\n",
    "        )\n",
    "        response = query_model(prompt)\n",
    "        \n",
    "        if politeness == \"polite\":\n",
    "            json_data[i][\"chosen\"] = response\n",
    "            json_data[i][\"rejected\"] = entry[\"output\"]\n",
    "        else:\n",
    "            json_data[i][\"rejected\"] = response\n",
    "            json_data[i][\"chosen\"] = entry[\"output\"]    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b071ce84-1866-427f-a272-b46700f364b2",
   "metadata": {},
   "source": [
    "- Let's now apply this evaluation to the whole dataset and compute the average score of each model (this takes about 1 minute per model on an M3 MacBook Air laptop)\n",
    "- Note that ollama is not fully deterministic across operating systems (as of this writing) so the numbers you are getting might slightly differ from the ones shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f700d4b-19e5-4404-afa7-b0f093024232",
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_model_responses(json_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "838d9747-0f7d-46fe-aab5-9ee6b765d021",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"instruction-data-with-preference.json\", \"w\") as file:\n",
    "    json.dump(json_data, file, indent=4)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.12 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}