LongContext_Finetuning_RepetitionTask.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2zfTC6fR91Ft"
   },
   "source": [
    "# Long Context Fine-tuning for Repetition Task\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/togethercomputer/together-cookbook/blob/main/LongContext_Finetuning_RepetitionTask.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KttUqkJ291Fu"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "*This cookbook is part of a technical deep dive blogpost on long context finetuning that you can read [here](https://www.together.ai/blog/long-context-fine-tuning-a-technical-deep-dive).*\n",
    "\n",
    "If you ask an LLM to repeat a sequence back to you, surely this should be easy, right? The answer is not straight forward and might be surprising to many!\n",
    "\n",
    "A lot of the capabilities that we know and trust our LLMs to have, fall short at longer contexts!\n",
    "\n",
    "To solve this repetition task a LLM should be able to use a simple induction head - that just copies a specific part of the input back out.\n",
    "\n",
    "However, for this task at longer contexts non-finetuned models fail quite miserably!\n",
    "\n",
    "In this notebook we will:\n",
    "1. Use a previously created dataset of long input sequences (upto 128k tokens)\n",
    "2. We will setup the repitition task, where we ask the model to repeat the last `k` words of the sequences created in Step 1.\n",
    "3. Demonstrate how even the best LLMs fail at this simple repitition task.\n",
    "4. We will fine-tune the model on ~1975 examples of this long-context task and show a radical improvement.\n",
    "\n",
    "<img src=\"images/repetition_task.png\" width=\"750\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UOHPEo9a91Fv"
   },
   "source": [
    "## Install Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1zkR46Gaf5gu",
    "outputId": "b1172ecb-6c9e-46bf-d61b-034cc2ba79f6"
   },
   "outputs": [],
   "source": [
    "!pip install -q together==1.3.4 python-Levenshtein==0.26.1 tqdm numpy orjson datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "8XBLhZKDf_K4"
   },
   "outputs": [],
   "source": [
    "from together import Together\n",
    "from tqdm.auto import tqdm\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "import orjson\n",
    "import json\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IN0u-3iQnGlN"
   },
   "outputs": [],
   "source": [
    "# Initialize the Together client and setup LLM calling function\n",
    "\n",
    "TOGETHER_API_KEY = os.getenv(\"TOGETHER_API_KEY\")\n",
    "WANDB_API_KEY = os.getenv(\"WANDB_API_KEY\") # If you'd like to view fine-tuning results on W&B\n",
    "\n",
    "client = Together(api_key = TOGETHER_API_KEY)\n",
    "\n",
    "def llm_call(query, model=\"meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\"):\n",
    "    response = client.chat.completions.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "          {\"role\": \"system\", \"content\": \"You are a helpful chatbot.\"},\n",
    "          {\"role\": \"user\", \"content\": query},\n",
    "        ],\n",
    "        temperature=1.0,\n",
    "        seed=42,\n",
    "    )\n",
    "    result = response.choices[0].message.content\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Preperation\n",
    "\n",
    "In order to create a long context dataset for our repetition task we extracted 2000 samples of varying length as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "ds_iterator = load_dataset(\n",
    "    \"HuggingFaceFW/fineweb-edu\",\n",
    "    \"sample-10BT\",\n",
    ")['train']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract 2000 examples of documents with 64k to 128k tokens\n",
    "\n",
    "long_documents_128k = []\n",
    "for sample in tqdm(ds_iterator.filter(lambda x: x['token_count'] > 64000 and x['token_count'] < 128000)):\n",
    "    # From 64k tokens to 128k tokens\n",
    "    if (len(long_documents_128k) < 2000):\n",
    "        document = sample['text']\n",
    "        long_documents_128k.append(document)\n",
    "    else:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract 2000 examples of documents with 24k to 32k tokens\n",
    "\n",
    "long_documents_32k = []\n",
    "for sample in tqdm(ds_iterator.filter(lambda x: x['token_count'] > 24000 and x['token_count'] < 32000)):\n",
    "    # From 24k tokens to 32k tokens\n",
    "    if (len(long_documents_32k) < 2000):\n",
    "        document = sample['text']\n",
    "        long_documents_32k.append(document)\n",
    "    else:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter for high quality samples \n",
    "\n",
    "long_documents = []\n",
    "for sample in tqdm(ds_iterator['train']):\n",
    "    # From 64k tokens to 128k tokens\n",
    "    if (len(sample['raw_content']) > 230000 and \n",
    "        len(sample['raw_content']) < 430000):\n",
    "        \n",
    "        signals = json.loads(sample[\"quality_signals\"])\n",
    "        try:\n",
    "            wiki_score = signals['rps_doc_ml_wikiref_score'][0][-1]\n",
    "        except:\n",
    "            wiki_score = 0\n",
    "            \n",
    "        if (wiki_score > 0.5 and\n",
    "            len(long_documents) < 2000):\n",
    "            \n",
    "            document = x['raw_content']\n",
    "            long_documents.append(document)\n",
    "\n",
    "            if len(long_documents) % 10 == 0:\n",
    "                print(len(long_documents))\n",
    "\n",
    "    if len(long_documents) >= 2000:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write out dataset\n",
    "\n",
    "Path(\"long_documents.json\").write_bytes(orjson.dumps({\n",
    "    \"32k\": long_documents_32k,\n",
    "    \"128k\": long_documents_128k\n",
    "}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q533v1BQq6wd"
   },
   "source": [
    "## Repetition Task Definition\n",
    "\n",
    "We used the code above to previously curate a dataset of long sequences by processing the [FineWeb](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1) and [RedPajama datasets](https://www.together.ai/blog/redpajama-data-v2) and retrieving 2000 English documents of 32k and 128k context length each.\n",
    "\n",
    "For each of these documents we want to setup a task prompt that we can pass into an LLM.\n",
    "\n",
    "This was done as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "long_documents = orjson.loads(Path(\"long_documents.json\").read_bytes())\n",
    "long_documents_32k = long_documents[\"32k\"]\n",
    "\n",
    "task_items = []\n",
    "\n",
    "for document in long_documents_32k:\n",
    "    n = np.random.randint(1, 100)\n",
    "    prompt = f\"Return last {n} words from this text: \\n\\n\"\n",
    "    target = \" \".join(document.split()[-n:])\n",
    "\n",
    "    task_items.append({\n",
    "        \"prompt\": prompt + document,\n",
    "        \"completion\": target\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the task prompts outlined above we need the LLM to, given an input sequence of arbitrary length, repeat the last `k` words of the sequence back to us. Where K is an random number beteween 1 and 100.\n",
    "\n",
    "We also extract the correct last `k` words directly from the document and store this to use for comparision with ground truth later.\n",
    "\n",
    "We have provided a JSON file from which you can load the task prompts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wo4cffP_q4fp"
   },
   "outputs": [],
   "source": [
    "# Load the task items from provided JSON file\n",
    "\n",
    "task_items = orjson.loads(Path(\"task_items.json\").read_bytes())\n",
    "task_items = task_items['task1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "opyRMVIXq4fp",
    "outputId": "a948dc16-3165-4ed8-f423-a3782f2a8e94"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2000"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verify that we have all 2000 task items\n",
    "\n",
    "len(task_items)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fdhWVg0Fnh6h"
   },
   "outputs": [],
   "source": [
    "# Select one task item for demonstration\n",
    "\n",
    "item = task1_items[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "Bz5EMsf2n1m2",
    "outputId": "41f24c9d-bd40-4dcc-c589-88a867a72ee7"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'Return last 67 words from this text: \\n\\n- freely available\\nToxins 2010, 2(4), 461-493; doi:10.3390/to'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What does a task item look like?\n",
    "\n",
    "item['prompt'][:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "id": "gdDZWh3En3DQ",
    "outputId": "1dc2fd96-16f7-49cc-975e-a341195ac15f"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'S.P.A.; Marmejo, J.; Giusti, W.; Deetz, K. Oligonucleotides with fluorescent dyes at opposite ends provide a quenched probe system useful for detecting PCR products and nucleic acid hybridization. PCR Met. Appl. 1995, 4, 357–362. [Google Scholar] © 2010 by the authors; licensee Molecular Diversity Preservation International, Basel, Switzerland This article is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Correct the prompt for the task item - ground truth\n",
    "\n",
    "item['completion']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LRQzcMrboCYE"
   },
   "outputs": [],
   "source": [
    "# How does a LLM model perform on this task item?\n",
    "\n",
    "query = item['prompt']\n",
    "\n",
    "result = llm_call(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "id": "dBjMj9Zro71E",
    "outputId": "ad80208c-575c-4a3c-9dc4-5eacb5b2e895"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'Here\\'s the last 67 words from this text in a more readable format:\\n\\n\"Detection of Ochratoxin A (OTA) Producers in Contaminated Commodities using PCR-Based Techniques. Real-time PCR (RT-PCR) can detect and quantify fungus DNA, providing new tools for fungal detection and quantification. RT-PCR can be performed using different chemistries, such as SYBR® Green I dye and TaqMan®. Both systems have proven useful in monitoring and quantifying OTA fungal producers in many food commodities.\"'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LVsyNJ_791Fz"
   },
   "source": [
    "As we can see from the single example above, our LLM is not great at this task.\n",
    "\n",
    "Ideally the LLM should be able to use an induction head to repeat a previously seen sequence back out. An induction head is a key component in transformer models that specializes in pattern recognition and prediction. Like a pattern-matching expert, it identifies repeated sequences in text and uses previous occurrences to predict what comes next. For example, if a phrase appeared before and was followed by specific text, the induction head remembers this pattern and applies it to similar future situations. This capability is fundamental to how transformers process language, enabling them to learn from repetition and make informed predictions based on previously seen patterns. Think of it as the model's memory mechanism for recognizing and utilizing recurring patterns in text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q-wHkFm691Fz"
   },
   "source": [
    "## Use Levenshtein Distance to Evaluate\n",
    "\n",
    "For this repetition task we need an exact comparision between the correct sequence of words to the LLM output sequence of words.\n",
    "\n",
    "Since this is an exact matching task we will use Levenshtein Distance.\n",
    "\n",
    "Levenshtein Distance measures how different two strings are by counting the minimum number of single-character changes (including inserting a character, deleting a character, or replacing a character) needed to turn one string into another.\n",
    "\n",
    "For example the levenshtein distance between `kitten` and `sitting` is 3 since we need 3 operations to for from one to the other.\n",
    "\n",
    "```python\n",
    "kitten → sitten  (replace 'k' with 's')\n",
    "sitten → sittin  (replace 'e' with 'i')\n",
    "sittin → sitting (insert 'g')\n",
    "\n",
    "Total Levenshtein Distance = 3 operations\n",
    "```\n",
    "\n",
    "Think of it like measuring the \"editing effort\" needed to transform one word into another. The lower the number, the more similar the strings are. A distance of 0 means the strings are identical, while larger numbers indicate more differences.\n",
    "\n",
    "For our purpose we will use `ratio = 1 - (leven_distance / (len1 + len2))` to obtain a score between `0` and `1`.\n",
    "\n",
    "- `0` implies that the two strings are very different\n",
    "- `1` implies that that two strings are identical\n",
    "\n",
    "For our repetition task higher is better!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XGBzTkQYoL2w"
   },
   "outputs": [],
   "source": [
    "from Levenshtein import ratio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "l3fANAajoMKy",
    "outputId": "4a3bc762-a7ec-4474-abc4-5dc40fcf5665"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3618290258449304"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratio(item['completion'], result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q-KH9EBr91Fz"
   },
   "source": [
    "Next we will loop over the first 25 task items and see how well our Llama 3.1 70B model performs at this task!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 66,
     "referenced_widgets": [
      "3239a272659640419bad96a90504fbcf",
      "beadd3e9dba143b2af21e1590791d801",
      "6c996f26cafa49d085c7f46875944ba7",
      "340af7569d624ffeb3eee41fcd75ea94",
      "ca8842e9c91042f997afba83f9f1d0a0",
      "5b52d2c761c04cb6aa7dfe5b0e0794e1",
      "403e211f1c744cc8b6dc73610bf7546e",
      "d4b89785530041b9b6d20962bdd6d366",
      "5796ef8086d3418ab2cd2106d7be48a9",
      "5297a0cf82ad4f90b75de65b1a223e21",
      "896c6e14f6d74e2d8a78c9e203eb3e01"
     ]
    },
    "id": "KVfjpuP7pNKU",
    "outputId": "a8159310-efb6-4a65-bda5-6dc194701230"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3239a272659640419bad96a90504fbcf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.377094996064535 103.44\n"
     ]
    }
   ],
   "source": [
    "scores = []\n",
    "length_differences = []\n",
    "for item in tqdm(task_items[:25]):\n",
    "    query = item['prompt']\n",
    "    result = llm_call(query)\n",
    "    score = ratio(item['completion'], result)\n",
    "    length_differences.append(abs(len(item['completion'].split()) - len(result.split())))\n",
    "    scores.append(score)\n",
    "print(np.mean(scores), np.mean(length_differences))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9UHPP8dGpcQ5",
    "outputId": "a6c30c79-41a1-4756-8db5-c9d04e186c72"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.3004739336492891,\n",
       " 0.3609022556390977,\n",
       " 0.4263494967978042,\n",
       " 0.34236804564907275,\n",
       " 0.41393034825870645,\n",
       " 0.35359116022099446,\n",
       " 0.8858057630736392,\n",
       " 0.36530442035029187,\n",
       " 0.2229924898902369,\n",
       " 0.39370078740157477,\n",
       " 0.36111111111111116,\n",
       " 0.33757961783439494,\n",
       " 0.3677758318739054,\n",
       " 0.8792569659442724,\n",
       " 0.5407725321888412,\n",
       " 0.1308455926324602,\n",
       " 0.318349299926308,\n",
       " 0.21875,\n",
       " 0.33444816053511706,\n",
       " 0.2104413347685683,\n",
       " 0.24250681198910085,\n",
       " 0.6222222222222222,\n",
       " 0.3307692307692308,\n",
       " 0.3536842105263158,\n",
       " 0.11344327836081958]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lpZtTqOGqo4T"
   },
   "source": [
    "As we can see above, Llama3.1 70B performs suboptimally at this repetition task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L9nAO1RJuMNr"
   },
   "source": [
    "## Fine-tune on Repetition Task\n",
    "\n",
    "Below we will fine-tune a smaller Llama 3.1 8B model on 1975 examples of this repetition task and see if we can get it to outperform on this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rzAD62btuBWO",
    "outputId": "438a689a-705e-4217-bef8-343bdfd4c349"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38838055"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate a file excluding the first 25 task items to train on. The first 25 task items will be used for evaluation.\n",
    "Path(\"task_train.jsonl\").write_text(\"\\n\".join([orjson.dumps(item).decode(\"utf-8\") for item in task_items[25:]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EhSC9_qZurk7"
   },
   "outputs": [],
   "source": [
    "response = client.files.upload(file=\"task_train.jsonl\", check=True)\n",
    "task_file_id = response.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_JrnYLnju6hR"
   },
   "outputs": [],
   "source": [
    "response = client.fine_tuning.create(\n",
    "  training_file = task_file_id,\n",
    "  model = 'meta-llama/Meta-Llama-3.1-8B-32k-Instruct-Reference',\n",
    "  n_epochs = 1,\n",
    "  n_checkpoints = 1,\n",
    "  batch_size = \"max\",\n",
    "  learning_rate = 7e-5,\n",
    "  suffix = 'long-context-finetune',\n",
    "  wandb_api_key = WANDB_API_KEY,\n",
    "  lora=True,\n",
    ")\n",
    "\n",
    "task_fine_tuning_job_id = response.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ydgVml5W91F0"
   },
   "source": [
    "Once the Model is finetuned we can assess how well it performs.\n",
    "\n",
    "## Deploy Model and Run Evals\n",
    "\n",
    "Before we can run the evaluations we need to deploy our finetuned model as a Dedicated Endpoint(DE). After fine-tuning completes, access your model through the Together AI dashboard. Go to Models, select your fine-tuned model, and select Deploy. Choose from the available hardware options - we'll use a single A100-80GB GPU for this example.\n",
    "\n",
    "<img src=\"images/deploy_CFT.png\" height=\"650\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "referenced_widgets": [
      "d43a6b6e19c545f783795c59862f834d"
     ]
    },
    "id": "8cf4df2b-814a-442b-98f8-2eba8aeeebcd",
    "outputId": "21fd1112-bdba-40f7-fa5c-87680980e960"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d43a6b6e19c545f783795c59862f834d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8149842379839035 15.08\n"
     ]
    }
   ],
   "source": [
    "scores = []\n",
    "length_differences = []\n",
    "\n",
    "for item in tqdm(task_items[:25]):\n",
    "    query = item['prompt']\n",
    "\n",
    "    # We have deployed the finetuned model here to evaluate it\n",
    "    result = llm_call(query, model=\"thepowerfuldeez/Meta-Llama-3.1-8B-32k-Instruct-Reference-long-context-finetune-ce1f61d6-afb7623b\")\n",
    "\n",
    "    score = ratio(item['completion'], result)\n",
    "    length_differences.append(abs(len(item['completion'].split()) - len(result.split())))\n",
    "    scores.append(score)\n",
    "\n",
    "print(np.mean(scores), np.mean(length_differences))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jCgoUf6avdup"
   },
   "source": [
    "We can see that even for the smaller 8B model the **Score improves from `0.37` to `0.81`** after finetuning when compared to the 70B untuned model.\n",
    "\n",
    "We can also see that after finetuning the model more often outputs the correct number of words, with the length difference decreasing from `103.44` before finetuning to `15.08` afterwards.\n",
    "\n",
    "To learn more about our fine-tuning API read the docs [here](https://docs.together.ai/reference/finetune)!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "3239a272659640419bad96a90504fbcf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_beadd3e9dba143b2af21e1590791d801",
       "IPY_MODEL_6c996f26cafa49d085c7f46875944ba7",
       "IPY_MODEL_340af7569d624ffeb3eee41fcd75ea94"
      ],
      "layout": "IPY_MODEL_ca8842e9c91042f997afba83f9f1d0a0"
     }
    },
    "340af7569d624ffeb3eee41fcd75ea94": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5297a0cf82ad4f90b75de65b1a223e21",
      "placeholder": "​",
      "style": "IPY_MODEL_896c6e14f6d74e2d8a78c9e203eb3e01",
      "value": " 25/25 [04:29&lt;00:00, 15.70s/it]"
     }
    },
    "403e211f1c744cc8b6dc73610bf7546e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5297a0cf82ad4f90b75de65b1a223e21": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5796ef8086d3418ab2cd2106d7be48a9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "5b52d2c761c04cb6aa7dfe5b0e0794e1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c996f26cafa49d085c7f46875944ba7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d4b89785530041b9b6d20962bdd6d366",
      "max": 25,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5796ef8086d3418ab2cd2106d7be48a9",
      "value": 25
     }
    },
    "896c6e14f6d74e2d8a78c9e203eb3e01": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "beadd3e9dba143b2af21e1590791d801": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5b52d2c761c04cb6aa7dfe5b0e0794e1",
      "placeholder": "​",
      "style": "IPY_MODEL_403e211f1c744cc8b6dc73610bf7546e",
      "value": "100%"
     }
    },
    "ca8842e9c91042f997afba83f9f1d0a0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d4b89785530041b9b6d20962bdd6d366": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}