generate_finetune_datasets.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9fd6de0a-8c4a-4a09-8c47-2af452e95cd4",
   "metadata": {},
   "source": [
    "# Query/Context Dataset Generation\n",
    "***\n",
    "\n",
    "This notebook walks students through the process of generating datasets of query/context pairs which can be used for two primary purposes:\n",
    "- Fine-tuning an embedding model\n",
    "- Serve as ground truth for retrieval evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1e7a0b9a-0f63-465f-b238-286433923925",
   "metadata": {},
   "outputs": [],
   "source": [
    "from retrieval_evaluation import QueryContextGenerator\n",
    "from llama_index.finetuning import EmbeddingQAFinetuneDataset\n",
    "from prompt_templates import qa_generation_prompt\n",
    "from preprocessing import FileIO\n",
    "from rich import print\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "env = load_dotenv('.env', override=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d9c94f26-2b9b-4cb4-9be8-8e8c21d0d785",
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiate generate with openai_key, model_id default is 'gpt-3.5-turbo-0613'\n",
    "generator = QueryContextGenerator(openai_key=os.environ['OPENAI_API_KEY'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97c56caf-f0d4-48dc-883f-00955af5ec6f",
   "metadata": {},
   "source": [
    "### Load raw data\n",
    "Load raw data from parquet file.  Raw data should be in the same format as the dataset (corpus) created in [Notebook 1](https://github.com/americanthinker/vectorsearch-applications/blob/main/1-Data_Preprocessing_Week1_COLAB.ipynb). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bc3833d7-9f28-439d-893c-f76731a599b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of data: (26448, 12)\n",
      "Memory Usage: 2.42+ MB\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "26448"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path = '../impact_theory_minilm_256.parquet'\n",
    "data = FileIO().load_parquet(data_path)\n",
    "len(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c8787b-3c0c-4999-942f-9f8362eb945b",
   "metadata": {},
   "source": [
    "### Data Length Analysis\n",
    "Conduct an analysis of the length of the content chunks.  Can use either raw words or tokens to assess length.  The main point here is to get a sense of the mean length of content chunks in the data and to set the `total_chars` param in the `clean_validate_data` method with an appropriate value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "656fdd2d-9057-4f6c-a767-a8070cec3cda",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>26448.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>991.729053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>126.344870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>944.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1005.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1060.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1974.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  0\n",
       "count  26448.000000\n",
       "mean     991.729053\n",
       "std      126.344870\n",
       "min        4.000000\n",
       "25%      944.000000\n",
       "50%     1005.000000\n",
       "75%     1060.000000\n",
       "max     1974.000000"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#in this example the mean content length is @ 1,000\n",
    "lengths = [len(d['content']) for d in data]\n",
    "df = pd.DataFrame(lengths)\n",
    "df.describe() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "202dbfca-0a73-4756-b7e7-58f5ef6f62e0",
   "metadata": {},
   "source": [
    "### Split Data\n",
    "\n",
    "The `train_val_split` function will clean and validate the raw data as a first step and then split into user defined train/val splits.  \n",
    "- Cleaning simply strips the keys from the data that are not needed for the query/content generation process\n",
    "- Validation consists of ensuring that only content chunks of length > `total_chars` are passed to the LLM (this step prevents the LLM from asking questions from sparse context)\n",
    "\n",
    "Users define the number of training samples and validation samples to generate.  Number of questions per content chunk can also be set to more than 1, however a note of caution:\n",
    "- Setting `num_questions_per_chunk` > 1 saves time (and money) by asking more than one question per content chunk, however, the dataset will be less diverse.  There is also the potential for the model to generate lower quality questions if the content chunk isn't large enough or meaningful enough to generate more than one question from the content.\n",
    "- Retrieval evaluation results from fine-tuning an embedding model with 200-300 training samples showed an uptick of 5-10% points.  Upper bound on retrieval improvement as a funtion of training sample size is yet to be determined (have fun pushing the boundaries! 👊)\n",
    "- A validation data set is not required for seeing improvement from fine tuning.  The addition of a validation dataset, however, allows a user to test the results of fine tuning on an unseen dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e4380c60-c0e9-4f35-b576-59adb173bec3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length Training Data: 10\n",
      "Length Validation Data: 5\n"
     ]
    }
   ],
   "source": [
    "#split data into train/val sets\n",
    "#in this example we are creating a training set of n=10, val set of n=5, and asking the LLM to only ask 1 question per chunk. \n",
    "train, val = generator.train_val_split(data, 10, 5, 1, total_chars=950)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac2ac9e4-3f6f-4903-88f7-7c397ddf66c6",
   "metadata": {},
   "source": [
    "### Generate QA pairs\n",
    "\n",
    "To generate query/context pairs we need to pass in our cleaned data splits, a question asking generation prompt, and the number of questions per chunk (needs to be same value passed into the `train_val_split` function.\n",
    "The `qa_generation_prompt` is already preconfigured and supplies the LLM with additional context about the Impact Theory show to ensure high quality questions are asked given the additional context.   \n",
    "Print out the prompt to see what is being asked of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "548ee55b-daa5-4ae3-ae8d-c9d67ac8c859",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "Impact Theory episode summary and episode guest are below:\n",
       "\n",
       "---------------------\n",
       "Summary: <span style=\"font-weight: bold\">{</span>summary<span style=\"font-weight: bold\">}</span>\n",
       "---------------------\n",
       "Guest: <span style=\"font-weight: bold\">{</span>guest<span style=\"font-weight: bold\">}</span>\n",
       "---------------------\n",
       "Given the Summary and Guest of the episode as context use the following randomly selected transcript section \\  \n",
       "of the episode and not prior knowledge, generate questions that can be answered by the transcript section: \n",
       "\n",
       "---------------------\n",
       "<span style=\"font-weight: bold\">{</span>transcript<span style=\"font-weight: bold\">}</span>\n",
       "---------------------\n",
       "\n",
       "Your task is to create <span style=\"font-weight: bold\">{</span>num_questions_per_chunk<span style=\"font-weight: bold\">}</span> questions that can only be answered given the previous context and\n",
       "transcript details. The question should randomly start with How, Why, or What.   \n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "Impact Theory episode summary and episode guest are below:\n",
       "\n",
       "---------------------\n",
       "Summary: \u001b[1m{\u001b[0msummary\u001b[1m}\u001b[0m\n",
       "---------------------\n",
       "Guest: \u001b[1m{\u001b[0mguest\u001b[1m}\u001b[0m\n",
       "---------------------\n",
       "Given the Summary and Guest of the episode as context use the following randomly selected transcript section \\  \n",
       "of the episode and not prior knowledge, generate questions that can be answered by the transcript section: \n",
       "\n",
       "---------------------\n",
       "\u001b[1m{\u001b[0mtranscript\u001b[1m}\u001b[0m\n",
       "---------------------\n",
       "\n",
       "Your task is to create \u001b[1m{\u001b[0mnum_questions_per_chunk\u001b[1m}\u001b[0m questions that can only be answered given the previous context and\n",
       "transcript details. The question should randomly start with How, Why, or What.   \n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(qa_generation_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eb27ebe-15ac-4807-a579-6474dce6d548",
   "metadata": {},
   "source": [
    "The output from this function is a llama_index class `EmbeddingQAFinetuneDataset`, which is a simple wrapper for a series of three dictionaries (`corpus`, `queries`, and `relevant_docs`).  The llama_index class is not absolutely necessary, but it is helpful in making transitions smoother when using the llama_index `SentenceTransformersFinetuneEngine` class for fine-tuning.  It takes roughly 80 seconds to generate 100 query/context pairs so a sample size of 300 takes about 4 minutes (much faster than if you were to do this manually!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "72a3fd61-44ea-4642-ba5c-b7acbf800e88",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:08<00:00,  1.25it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:03<00:00,  1.27it/s]\n"
     ]
    }
   ],
   "source": [
    "training_set = generator.generate_qa_embedding_pairs(train, qa_generation_prompt, 1)\n",
    "val_set = generator.generate_qa_embedding_pairs(val, qa_generation_prompt, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1e6130c7-d4b6-4d75-b63e-acc85ffed3f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 5)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#EmbeddingQAFinetuneDataset has no len, so check length of queries instead\n",
    "len(training_set.queries), len(val_set.queries)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7be6432-def2-4dcf-8801-1c56d5f09a86",
   "metadata": {},
   "source": [
    "### Dataset Analysis\n",
    "\n",
    "Always a good idea to check the quality of the pairs generated.  Most pairs will be high quality but some will not be, this is a chance for human intervention to adjust the questions manually to ensure the quality remains high. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cda268d4-93b2-4ed4-9700-6819cb7a9af4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_qa_pairs(data: EmbeddingQAFinetuneDataset, print_results: bool=True):\n",
    "    pairs = []\n",
    "    for k, v in data.queries.items():\n",
    "        doc_id = data.relevant_docs[k][0]\n",
    "        context = data.corpus[doc_id]\n",
    "        pairs.append((v, context))\n",
    "    if print_results:\n",
    "        for tup in pairs:\n",
    "            print(f'Question: {tup[0]}\\nContext: {tup[1]}\\n\\n')\n",
    "    return pairs    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6141be90-7c11-40f7-b635-c82f80ea394b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: How does Chase Jarvis describe his initial reaction to meditation and why did he feel that way?\n",
      "Context: Tim Bilyeu Because I'm just taking it from a neurological perspective. Dr. Tim Jackson There you go. Tim Bilyeu Purely from a neurological perspective because I know what's happening. You're tapping into the parasympathetic nervous system, you're calming down the sympathetic. It's just, it's biochemistry, right? Dr. Tim Jackson If you believe in biochemistry, then give it a shot. I'm a huge advocate. I try not to sell it too hard because anytime someone's trying to sell you something, it feels inauthentic, but it's just given me a lot of joy. Dr. Tim Jackson Yeah. No, I think that it's like you ... Actually, you embraced it pretty quickly. Like Tim, I did not. I really felt that it was super soft, and I never thought of it as taking a step off my edge. It just felt like probably because my self-narrative ... Growing up, I was not good at sports. I did not feel overly tough, and so toughening up was lesson number one for me as an entrepreneur.\n",
      "\n",
      "\n",
      "Question: How does Tom Bilyeu suggest leaders scale their influence and ensure alignment with the company's culture as the team grows?\n",
      "Context: You start with the people closest to you because you've been able to write the culture down. You know which way they have to act. You hold them accountable. You ask them to hold you accountable, right? You're not trying to be above this. You're trying to be in it, trying to show people what it means. So, now you've got your core group. Then they're going to have their core group. And then that core group is going to have their core group. And now that influence goes down. So, you definitely have to set the example. But when you scale, you have got to get people to understand what the culture is. Get them to lead by example. Get them to intoxicate people with certainty. Get them to show people what it means to have a goal and to go after it and to live up to the standards of that culture. And now, when you elevate people and you're lifting them up and you're not trying to beat them down, then you get something really, really incredible. Then people will follow you because it's a game of elevation. It's a game of skill acquisition.\n",
      "\n",
      "\n",
      "Question: How does food impact genetic expression according to the study of nutrigenomics?\n",
      "Context: And every single bite of food that I was eating, and this is something that I've been studying now for over 10 years, I absolutely love this, is something called nutrigenomics. And it's a study looking at how every single molecule of food that you eat impacts your genetic expression. Food is that powerful. And several other epigenetic factors, like sleep, which might be the biggest epigenetic influence, you get to choose what kind of copies are being made of you. Your genes are basically blueprint to print out certain copies. And there's upwards of like 4,000 different variations that one gene can do. And you get to have a big role, a big part in how those genes are getting expressed. So understanding that when we eat a particular food, it isn't just, I'm just eating this thing. It's setting off a cascade of events. And it's incredibly empowering, but also can be really sobering. And again, I don't want to get people into the neurotic state, because I've been there.\n",
      "\n",
      "\n",
      "Question: How does Tom Bilyeu respond to a question about finding one's audience and the value of doing so?\n",
      "Context: So there literally isn't an arena where even if you're a solo writer, being able to write persuasive arguments is still monetizable. So I would say that's universal, applies everywhere. All right, next question from Aaron Elias. What is your take on energy upgrades and waves sweeping the earth? What is your take on ascension? I have no idea. I've never heard those words put together in that fashion before. I fear I'm gonna fail you entirely. I have absolutely nothing but ignorance on this topic. Forgive me. All right, next up, Carolina Wilk. Tom, what do you think about Dr. John Demartini's teachings? Two in a row, totally ignorant. I have no idea who Dr. John Demartini is. Forgive me. Next question, Jessica Terzia. How do you find your audience once you have a passion? I want to speak, but I can't help everyone at once. I was told I need to find my audience. Did you do this and did you find value in this? All right, let's deconstruct this a bit.\n",
      "\n",
      "\n",
      "Question: How does Meagan Good approach pushing herself beyond her limits and constantly striving for personal growth?\n",
      "Context: But I start to feel that when my mind and my body is screaming, if I push just a little bit further, and then when I push further, I realize that I can push just a little bit further, and I just keep going as far as I can. And once I know that I can get to this level, the next opportunity I have, I start pushing past that next level. And I don't know. I guess it's I don't want things to be the same. I want to continue to go to the next level. I don't ever want to get comfortable and say this is good, this is enough. I want to be satisfied and thankful with this is good and this is enough. But I don't want to get comfortable in the sense of like I know that I can access more. And again, I want to access everything that there is for me to get until there's nothing left, you know? Dude, I love that so much. Like that is the driving force in my life is to be at the same time, to be like I love my life, I'm way grateful. But it's me versus me.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pairs = show_qa_pairs(val_set, print_results=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62170665-3b89-49ef-a266-ab24a86ab43c",
   "metadata": {},
   "source": [
    "### Save to Disk  \n",
    "Save to disk using your own filepaths, below is an example using the length of the sets as part of the filepath."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "192d9bb8-8535-4d1e-9229-f93cffb368e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# training_set.save_json('./data/training_data_10.json')\n",
    "# val_set.save_json('./data/validation_data_5.json')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}