walkthrough.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3373f2c7-da8e-49eb-8300-25ae4a00fdaa",
   "metadata": {},
   "source": [
    "# Welcome to Lumigator Foxfooding from [Mozilla.ai](https://www.mozilla.ai/)! 🐊 🦊"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ba77ea-aae4-4c55-ad18-1b665836532c",
   "metadata": {},
   "source": [
    "\n",
    "## Agenda\n",
    "\n",
    "+ Working with Jupyter Notebooks\n",
    "+ Lumigator Platform 🐊 and Machine Learning Overview  \n",
    "+ Understanding Machine Learning Workflows \n",
    "+ Thunderbird Ground Truth Dataset Walkthrough\n",
    "+ Selection of models \n",
    "    + 1 encoder/decoder (BART) \n",
    "    + 2 decoder (Mistral and GPT-4o)\n",
    " to evaluate against GT\n",
    "+ Run evaluation experiment\n",
    "+ Discuss results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e41dab1",
   "metadata": {},
   "source": [
    "## Jupyter Walkthrough\n",
    "\n",
    "[Jupyter Notebooks](https://jupyter-notebook.readthedocs.io/en/stable/) are an executable code/text environment for (usually) Python code. Our Jupyter environment is in JupyterHub. To work with Jupyter, click \"run cell\" to run the code and see results below the cell you're currently running. Cells are executed sequentially. \n",
    "\n",
    "In some cells, you will see cases where there are variables that you'll need to pre-populate before running the cell. They look like this. The code will not work unless you replace it!  \n",
    "\n",
    "```python\n",
    "# suggestion: \"lumigator_enthusiasts\"\n",
    "team_name = TEAM_NAME_HERE\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccf10cbb-f6a0-4a24-9d6d-30fac1f7d996",
   "metadata": {},
   "source": [
    "# Running cells \n",
    "To run a cell, press the \"play\" icon in the top bar (you can also hit Shift+Enter to run and proceed to the following cell).\n",
    "\n",
    "\n",
    "<img src=\"assets/running.png\" alt=\"drawing\" width=\"400\"/>\n",
    "\n",
    "Your files are located on the left-hand side. They'll be saved for the duration of our session, but if you'd like to keep them, make sure to download them. \n",
    "\n",
    "<img src=\"assets/files.png\" alt=\"drawing\" width=\"400\"/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cb7563e-3ab3-486f-b45a-7bd0c47fbcce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets' try running some code!\n",
    "# You can see the output below!\n",
    "print(\"Welcome to Lumigator!🐊\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ecaa846",
   "metadata": {},
   "source": [
    "## Machine learning glossary\n",
    "\n",
    "Some terms you'll hear us using throughout the session: \n",
    "\n",
    "+ **Machine learning** - The process of creating a model that learns from data\n",
    "+ **Dataset** - Data used to train models and evaluate their performance\n",
    "+ **LLM** - Large language model, [a text-based model that performs next-word predictions](https://www.nvidia.com/en-us/glossary/large-language-models/) \n",
    "+ **Tokens** - Words broken up into pieces to be used in an LLM \n",
    "+ **Inference** - The process of getting a prediction from a large language model \n",
    "+ **Embeddings** - Numerical representations of text generated by modeling \n",
    "+ **Encoder-decoder models** - A neural network architecture comprised of two neural networks, an encoder that takes the input vectors from our data and creates an embedding of a fixed length, and a decoder, also a neural network, which takes the embeddings encoded as input and generates a static set of outputs such as translated text or a text summary\n",
    "+ **Decoder-only models** - Given a fixed input prompt, uses its representation to generate a sequence of words one at a time, with each word being conditioned on the ones generated previously\n",
    "+ **Task** - Machine learning tasks to fit a specific model type, including translation, summarization, completion, etc. \n",
    "+ **Ground truth** - Information that has been evaluated to be true by humans (or LLMs, in some cases) to be correct, that we can use as a point of comparison for our model. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "709af016-5f3f-4552-8a8a-e731e60eb89d",
   "metadata": {},
   "source": [
    "The process of **machine learning** is the process of creating a mathematical model that tries to approximate the world. A **machine learning model** is a set of instructions for generating a given\n",
    "output from data. The instructions are learned from the features of the input data itself."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545e65e6-4e05-4aac-9bd5-c8521fbcb9e2",
   "metadata": {},
   "source": [
    "Within the universe of modeling approaches, there are **supervised** and **unsupervised** approaches, as well as reinforcement learning. \n",
    "Language modeling of the kind we do with LLMs falls in the realm of neural network approaches. \n",
    "\n",
    "<img src=\"assets/ml_family.png\" alt=\"drawing\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f67ff2a",
   "metadata": {},
   "source": [
    "## How do LLMs work? \n",
    "\n",
    "There are many different kinds of LLMs and many different kinds of architectures. For our evaluations, we use two different kinds:\n",
    "\n",
    "+ **Encoder/Decoder** - BART model, Converts input data into a fixed-size representation (similar to encoder models). These models are trained first to generate text into numerical representations, then to output text based on those numerical representations. They're good for synthesis as opposed to generation. \n",
    "+ **Decoder-only** - most GPT-family models like Mistral, GPT, and others we'll be working with.  These models are pre-trained with text data in an autoregressive manner, for next-token prediction given previous tokens.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8139990",
   "metadata": {},
   "source": [
    "## LLM Workflows\n",
    "\n",
    "1. Pre-train the model itself and generate a model artifact\n",
    "2. Generate ground truth for our business use-case\n",
    "3. Pick several models we'd like to use to evaluate\n",
    "4. Run an evaluation loop consisting of looking at the ground truth in comparison to model results\n",
    "5. Analyze our evaluations.\n",
    "\n",
    "<img src=\"assets/llm_steps.png\" alt=\"drawing\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c44d8a1",
   "metadata": {},
   "source": [
    "Lumigator on a technical level is a **Python-based FastAPI web app** with services that run **jobs and deployments on a Ray cluster** which can be run either locally or in the cloud, depending on your computer specs. Results are stored in **Postgres**. Larger models loaded from HuggingFace require GPUs.  \n",
    "\n",
    "<img src=\"assets/platform.png\" alt=\"drawing\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed38e49",
   "metadata": {},
   "source": [
    "What is Ray? [A distributed runtime for Python programs](https://github.com/ray-project/ray) that includes a Core library with primitives (Tasks, Actors, and objects) and a suite of ML libraries (Tune, Serve) that allow to build components of the machine learning model workflow. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "838d2fd4",
   "metadata": {},
   "source": [
    "## Nota bene: Machine learning is alchemy\n",
    "\n",
    "When we think of traditional software application workflows, we think of an example such as adding a button. We can clearly test that we've added a blue button to our application, and that it works correctly. Machine learning is not like this! It involves a lot of experimentation, tweaking of hyperparameters and prompts and trying different models. Expect for the process to be imperfect, with many iterative loops. Luckily, Lumigator helps take away the uncertainty of at least model selection :)\n",
    "\n",
    "> There’s a self-congratulatory feeling in the air. We say things like “machine learning is the new electricity”. I’d like to offer an alternative metaphor: machine learning has become alchemy. - [Ben Recht and Ali Rahimi](https://archives.argmin.net/2017/12/05/kitchen-sinks/)\n",
    "\n",
    "Ultimately, the final conclusion of whether a model is good is if humans think it's good. \n",
    "\n",
    "With that in mind, let's dive into setting up experiments with Lumigator to test our models!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bbdf25d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We have a library of utility functions that will help us connect to the Lumigator API\n",
    "# Let's take a second to walk through them \n",
    "\n",
    "import lumigator_demo as ld\n",
    "import eval_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8237ee62-33d1-480d-864f-03f70537c668",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing packages we need to work with data \n",
    "# python standard libraries\n",
    "import time\n",
    "import json\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "\n",
    "# Random string generator\n",
    "import random\n",
    "import string\n",
    "import shortuuid\n",
    "\n",
    "# third-party libraries\n",
    "from datasets import load_dataset\n",
    "from IPython.display import clear_output\n",
    "\n",
    "# wrap columns for inspection\n",
    "pd.set_option('display.max_colwidth', 0)\n",
    "# stylesheet for visibility\n",
    "plt.style.use(\"fast\")\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a852789a",
   "metadata": {},
   "source": [
    "# Understanding the Lumigator App and API \n",
    "\n",
    " The app itself consists of an API, which you can access and test out methods in the [OpenAPI spec](https://swagger.io/specification/), at the platform URL, under docs. \n",
    "\n",
    "<img src=\"assets/openapi.png\" alt=\"drawing\" width=\"200\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c6f9648",
   "metadata": {},
   "source": [
    "Large language models today are consumed in one of several ways:\n",
    "\n",
    "+ As **API endpoints** for proprietary models hosted by OpenAI, Anthropic, or major cloud providers\n",
    "+ As **model artifacts** downloaded from HuggingFace’s Model Hub and/or trained/fine-tuned using HuggingFace libraries and hosted on local storage\n",
    "+ As model artifacts available in a format optimized for **local inference**, typically GGUF, and accessed via applications like llama.cpp or ollama\n",
    "+ As ONNX, a format which optimizes sharing between backend ML frameworks\n",
    "\n",
    "We use API endpoints and local storage in Lumigator. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "429bca64-a6c2-45ee-9c3a-43e1db8905f3",
   "metadata": {},
   "source": [
    "\n",
    "We currently have 5 key endpoints on the platform. \n",
    "\n",
    "+ `Health` - Status of the application, running status of jobs and deployments. \n",
    "+ `Datasets` - Data that we add to the platform for evaluation. We can upload, delete, and save different data in the platform. - We'll use this to save our ground truth and experiment data\n",
    "+ `Experiments` - Our actual evaluation experiments. We can list all previous evaluations, create new ones, and get their results.\n",
    "+ `Groundtruth` - Running Ray-serve deployments with locally-hosted models\n",
    "+ `Completions` - Access to external APIs such as Mistral and OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3961f5ff",
   "metadata": {},
   "source": [
    "## Model Task: Summarization\n",
    "\n",
    "The task we'll be working with is **summarization**, aka we want to generate a summary of our text. \n",
    "\n",
    "In our business case, which is to create summaries of conversation threads, much as you might see in Slack or an email chain, the models need to be able to extract key information from those threads while still being able to accept a large context window to capture the entire conversation history. \n",
    "\n",
    "We identified that it is far more valuable to conduct **abstractive** summaries, or summaries that identify important sections in the text and generate highlights,  rather than **extractive** ones, which pick a subset of sentences and add them together for our use cases since the final interface will be natural language. We want the summary results not to need to be interpreted from often incoherent text snippets produced by extractive summaries. \n",
    "\n",
    "For more on summarization as a use-case, [see our blog post here.](https://blog.mozilla.ai/on-model-selection-for-text-summarization/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "819e0905",
   "metadata": {},
   "source": [
    "## Ground Truth for Models\n",
    "\n",
    "The term ground truth comes from geology and geospatial sciences, where actual information was collected on the ground to validate data acquired through remote sensing, such as satellite imagery or aerial photography. Since then, the concept has been adopted in other fields, particularly in machine learning and artificial intelligence, to refer to the accurate, real-world data used for training and testing models. \n",
    "\n",
    "The **best ground truth is human-generated** but building it is a very expensive task.\n",
    "One recent trend is to rely on large language models but (as you will see later) they have their own pitfalls.\n",
    "An intermediate approach uses different LLMs to provide ground truth \"candidates\" which are then subject to human pairwise evaluation.\n",
    "For the sake of explanation, we will generate our ground truth by performing inference against existing models that are trained for summarization.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31fe6236-e93b-4328-b78c-2eda264b32b8",
   "metadata": {},
   "source": [
    "## Our Input data\n",
    "\n",
    "Let's take a look at the [data we'll be using first from the Thunderbird public mailing list.](https://thunderbird.topicbox.com/groups/addons/T18e84db141355abd-M4cca8e3f9e4fee9ae14b9dbb/self-hosted-version-of-extension-is-incorrectly-appearing-in-atn)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59e1493b-ecbd-4b2c-a13a-58365cc15023",
   "metadata": {},
   "source": [
    "## Generating Data for Ground Truth Evaluation\n",
    "\n",
    "In order to generate a ground truth summary for our data, we first need an input dataset. In this case we use threads from the [Thunderbird public mailing list](https://thunderbird.topicbox.com/latest).\n",
    "\n",
    "Our selection criteria: \n",
    "\n",
    "+ Collect 100 recent and \"complete\" email threads for evaluation\n",
    "+ Clean them of email formatting such as `>`\n",
    "+ BART, the baseline model we're using, accepts up to a 1024-token-long context window as input. This means that we have to have input email threads that are ~ approximately 1000 words, so keeping on the conservative side for smaller models. \n",
    "\n",
    "Once we've collected them, we'd like to take a look at the data before we generate summaries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36323a9e-aec4-402f-81c0-164f4f890723",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-17T17:59:45.160392Z",
     "start_time": "2024-07-17T17:59:45.157516Z"
    }
   },
   "outputs": [],
   "source": [
    "# show information about the Thunderbird dataset\n",
    "dataset_id = \"db7ff8c2-a255-4d75-915d-77ba73affc53\"\n",
    "r = ld.dataset_info(dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5ab1a6c-30bc-421d-908d-20268cdefb98",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-17T18:00:10.371832Z",
     "start_time": "2024-07-17T18:00:10.354800Z"
    }
   },
   "outputs": [],
   "source": [
    "# download the dataset into a pandas dataframe\n",
    "df = ld.dataset_download(dataset_id)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfb735b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'd like to make sure our data is clean for LLM input\n",
    "# This is often not necessary since most LLMs are trained on internet-formatted data\n",
    "# But we'll be careful here\n",
    "\n",
    "import re\n",
    "from string import punctuation\n",
    "\n",
    "def preprocess_text(text:str):\n",
    "    text = text.lower()  # Lowercase text\n",
    "    text = re.sub(f\"[{re.escape(punctuation)}]\", \"\", text)  # Remove punctuation\n",
    "    text = \" \".join(text.split())  # Remove extra spaces, tabs, and new lines\n",
    "    text = re.sub(r\"\\b[0-9]+\\b\\s*\", \"\", text)\n",
    "    return text\n",
    "\n",
    "df[\"examples\"].map(preprocess_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "157ddccc-33ab-4204-967a-d266b8ff051e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-17T18:00:25.102809Z",
     "start_time": "2024-07-17T18:00:25.098826Z"
    }
   },
   "outputs": [],
   "source": [
    "# Examine a single sample \n",
    "# we define the data with examples\n",
    "df['examples'].iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c57a4fa9-ce82-43ec-a2b1-0fc984b23cea",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-17T18:00:28.965534Z",
     "start_time": "2024-07-17T18:00:28.962726Z"
    }
   },
   "outputs": [],
   "source": [
    "# Add a function to do some simple character counts for model input\n",
    "df['char_count'] = df['examples'].str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1336138",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect our data\n",
    "df.head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb9694e3-d4d3-46b6-9023-4744f1ab73e3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-17T18:00:31.627488Z",
     "start_time": "2024-07-17T18:00:31.620802Z"
    }
   },
   "outputs": [],
   "source": [
    "# Show statistics about characters count\n",
    "df['char_count'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a28c016-5aae-47f2-a18e-e470cae8ef4e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-07-17T18:00:33.594470Z",
     "start_time": "2024-07-17T18:00:33.477793Z"
    }
   },
   "outputs": [],
   "source": [
    "# Generate plot of character counts\n",
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "ax.hist(df['char_count'], bins=30)\n",
    "ax.set_xlabel('Character Count')\n",
    "ax.set_ylabel('Frequency')\n",
    "\n",
    "stats = df['char_count'].describe().apply(lambda x: f\"{x:.0f}\")\n",
    "\n",
    "# Add text boxes for statistics\n",
    "plt.text(1.05, 0.95, stats.to_string(), \n",
    "         transform=ax.transAxes, verticalalignment='top')\n",
    "\n",
    "# Adjust layout\n",
    "plt.tight_layout()\n",
    "fig.subplots_adjust(right=0.75)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffe26e9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform Ground Truth Generation with Mistral \n",
    "\n",
    "mistral_responses = []\n",
    "\n",
    "for sample in df['examples'][0:10]:\n",
    "    res = ld.get_mistral_ground_truth(sample)\n",
    "    print(f\"Mistral Summary:\", res)\n",
    "    mistral_responses.append((sample, res['text']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44d6274a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create a result set we can look at\n",
    "mistral_results_df = pd.DataFrame(mistral_responses, columns=['examples', 'mistral_response'])\n",
    "mistral_results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9534ac3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's take a look at all available deployments\n",
    "ld.get_deployments()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f8a00b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform Ground Truth Generation with BART\n",
    "\n",
    "deployment_id = ld.get_summarizer_deployment_id()\n",
    "\n",
    "bart_responses = []\n",
    "\n",
    "for prompt in df['examples'][0:10]:\n",
    "    response = ld.get_bart_ground_truth(deployment_id, prompt)\n",
    "    response_dict = json.loads(response.text)\n",
    "    results = response_dict.get('deployment_response', {}).get('result', 'No result found')\n",
    "    print(\"BART:\", results)\n",
    "    bart_responses.append((prompt, results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b234fab9",
   "metadata": {},
   "outputs": [],
   "source": [
    "bart_results_df = pd.DataFrame(bart_responses, columns=['examples', 'bart_response'])\n",
    "bart_results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a1483a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine results and examine multiple versions of ground truth\n",
    "merged_df = pd.merge(bart_results_df, mistral_results_df, on='examples', how='outer')\n",
    "merged_df.to_csv('ground_truth.csv', index=False)\n",
    "merged_df "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61a9104b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now that we have the data, let's save it to the cluster so we can use it later on\n",
    "bart_results_df = bart_results_df.rename(columns={\"bart_response\": \"ground_truth\"})\n",
    "bart_results_df.to_csv('bart_ground_truth.csv', index=False)\n",
    "ld.dataset_upload(\"bart_ground_truth.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dbd7536",
   "metadata": {},
   "outputs": [],
   "source": [
    "mistral_results_df = mistral_results_df.rename(columns={\"mistral_response\": \"ground_truth\"})\n",
    "mistral_results_df.to_csv('mistral_ground_truth.csv', index=False)\n",
    "ld.dataset_upload(\"mistral_ground_truth.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9e5de01",
   "metadata": {},
   "outputs": [],
   "source": [
    "# And let's check that data loaded\n",
    "ld.get_datasets()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f18c813-36f8-4439-880d-441499e80493",
   "metadata": {},
   "source": [
    "# Experiments\n",
    "Let's start by creating a team name for our experiments to organize our data, pick a team name below and run the cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67092b57-e4ff-4fd7-a46b-f6ab9c56c2a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create an ID for our experiments \n",
    "alphabet = string.ascii_lowercase + string.digits\n",
    "su = shortuuid.ShortUUID(alphabet=alphabet)\n",
    "\n",
    "def shortuuid_random():\n",
    "    return su.random(length=8)\n",
    "\n",
    "short_guid = shortuuid_random()\n",
    "team_name = f\"gator_{short_guid}\"\n",
    "team_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e914253-e217-42f8-85a2-59f32daf6779",
   "metadata": {},
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd66191-c874-4303-bd26-ef93388a124f",
   "metadata": {},
   "source": [
    "After generating the ground truth (either manually or with the aid of some models) and uploading the dataset to lumigator, we are ready to start evaluating models on it.\n",
    "\n",
    "Note that when you uploaded your datasets you were returned some information that included a `dataset_id`. This is a unique identifier to your own dataset that you can reuse across different experiment. Please add your dataset identifier in the cell below to use it from now on.\n",
    "\n",
    "Note that we have also provided a few pre-generated datasets below, in the same format as the one you just generated. If you want to try one of them you can just remove the `YOUR_DATASET_ID` line and uncomment (by removing the trailing `#` character) the one with the dataset you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45148cb9-3d69-4f4d-b679-2eb23e377051",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_id = YOUR_DATASET_ID\n",
    "# dataset_id = \"fd454e33-e0c1-4c3b-a5f3-151a2be8beaa\" # Mistral GT - 10 samples\n",
    "# dataset_id = \"daaa63ae-84fc-4557-a301-97d71b4ca7fe\" # Bart GT - 10 samples\n",
    "# dataset_id = \"1bc65c24-5f28-4ede-9578-f56e4cbdeb5f\" # Mistral-API GT - 100 samples\n",
    "# dataset_id = \"6bb9378a-012f-486b-afac-01b56f00456e\" # Bart GT - 100 samples\n",
    "# dataset_id = \"a36061aa-b18a-4abc-a7de-d652670ed971\" # Mistral-llamafile GT - 100 samples\n",
    "\n",
    "# now look for the dataset on lumigator\n",
    "r = ld.dataset_info(dataset_id)\n",
    "dataset_name = json.loads(r.text)['filename']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2876b7f8-e645-4791-aa27-351e3296e2de",
   "metadata": {},
   "source": [
    "## Model Selection\n",
    "\n",
    "What you see below are different lists of models we have already tested for the summarization task.\n",
    "The `models` variable at the end provides you with a selection, but you can choose any combination of them.\n",
    "\n",
    "Note that different model types are specified with different prefixes:\n",
    "\n",
    "- `hf://` is used for HuggingFace models which are downloaded and ran as Ray jobs\n",
    "- `mistral://` is used for models which are accessed through the Mistral API\n",
    "- `oai://` is used for models which are accessed through an OpenAI-compatible API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37fcdf76-c9ad-4459-b0c3-49098286ba85",
   "metadata": {},
   "outputs": [],
   "source": [
    "enc_dec_models = [\n",
    "    'hf://facebook/bart-large-cnn',\n",
    "    'hf://mikeadimech/longformer-qmsum-meeting-summarization', \n",
    "    'hf://mrm8488/t5-base-finetuned-summarize-news',\n",
    "    'hf://Falconsai/text_summarization',\n",
    "]\n",
    "\n",
    "dec_models = [\n",
    "    'mistral://open-mistral-7b',\n",
    "]\n",
    "\n",
    "gpts = [\n",
    "    \"oai://gpt-4o-mini\",\n",
    "    \"oai://gpt-4-turbo\",\n",
    "    \"oai://gpt-3.5-turbo-0125\"  \n",
    "]\n",
    "\n",
    "models = [\n",
    "    enc_dec_models[0],\n",
    "    dec_models[0],\n",
    "    gpts[0]\n",
    "]\n",
    "\n",
    "models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74658955-4a41-4be1-b29d-8efa734bed9d",
   "metadata": {},
   "source": [
    "## Run Evaluations\n",
    "\n",
    "The following cell will start the actual model evaluations.\n",
    "Once you run it, new jobs will be submitted to ray (one for each model) and the outcomes of these submissions will be printed.\n",
    "Each evaluation job will first use the provided model to summarize each of the emails in the input dataset. After that, it will calculate a few metrics to evaluate how close the predicted summaries are to the ground truth provided in the dataset.\n",
    "\n",
    "Each job starts with a `created` status. While the job runs, you will be able to track its status by running the cell in the section **Track evaluation jobs**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b73f6a1-7be9-43a7-8304-4cb6a408318e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set this value to limit the evaluation to the first max_samples items (0=all)\n",
    "max_samples = 0\n",
    "\n",
    "responses = []\n",
    "for model in models:\n",
    "    descr = f\"Testing {model} summarization model on {dataset_name}\"\n",
    "    responses.append(ld.experiments_submit(model, team_name, descr, dataset_id, max_samples))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f787190-4581-4c26-beca-0260303a68bd",
   "metadata": {},
   "source": [
    "### Track evaluation jobs\n",
    "\n",
    "Run the following to track your evaluation jobs.\n",
    "\n",
    "- *NOTE*: you won't be able to run other cells while this one is running. However, you can interrupt it whenever you want by clicking on the \"stop\" button above and run it at a later time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2858bcea-6fcb-4441-b56e-5d89502f10e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "job_ids = [ld.get_resource_id(r) for r in responses]\n",
    "\n",
    "wip = ld.show_experiment_statuses(job_ids)\n",
    "while wip == True:\n",
    "    time.sleep(5)\n",
    "    clear_output()\n",
    "    wip=ld.show_experiment_statuses(job_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32d8cbf3-e44f-4f1e-b520-3f6fac147fca",
   "metadata": {},
   "source": [
    "## Show evaluation results\n",
    "\n",
    "Once all evaluations are completed, their results will be stored on our platform and available for download. \n",
    "You can download them individually with the command\n",
    "\n",
    "```python\n",
    "ld.experiments_result_download(job_id)\n",
    "```\n",
    "\n",
    "The following cell iterates on all your job ids, downloads results from each, and builds a table comparing different metrics for each model.\n",
    "The metrics we use to evaluate are ROUGE, METEOR, and BERT score. They all measure similarity between predicted summaries and those provided with the ground truth, but each of them focuses on different aspects. The image below shows their main characteristics and the tradeoffs between their flexibility and their computational cost.\n",
    "\n",
    "<img src=\"assets/metrics.png\" alt=\"drawing\" width=\"900\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b8709b8-59ba-4a9c-81bd-59b2fc612f99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# after the jobs complete, gather evaluation results\n",
    "eval_results = []\n",
    "for job_id in job_ids:\n",
    "    eval_results.append(ld.experiments_result_download(job_id))\n",
    "\n",
    "# convert results into a pandas dataframe\n",
    "ld.eval_results_to_table(eval_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b450ce",
   "metadata": {},
   "source": [
    "## Analysis of Evaluation Results\n",
    "\n",
    "The tablel above is just a summary of all the evaluation results.\n",
    "The `eval_results` object contains way more details from which you'll be able to get a few more insights in the following cells.\n",
    "\n",
    "### Direct access to all data\n",
    "\n",
    "The following cell shows you the kind of information that's available in each of the `eval_results` elements. This information is nested at different depth levels. You can access each using the `get_nested_value` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eadbdaec-a1e9-4731-bfdd-b471e6a0fbcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# eval_results is a list holding information for each of the models you defined before\n",
    "# for each element, you can access different metrics, time performance, and predictions\n",
    "eval_results[0].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ec4d37e-4750-404e-b40c-3778b8196d64",
   "metadata": {},
   "outputs": [],
   "source": [
    "# see how much time it took for a model to summarize all the input samples\n",
    "ld.get_nested_value(eval_results[0], \"summarization_time\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13a73261-e5a4-46a6-9b7c-1bc5e62dc1ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# see all the bertscore data\n",
    "ld.get_nested_value(eval_results[0], \"bertscore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f6e8285-f407-4a5f-acda-3400554c3d2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# see mean bert precision\n",
    "ld.get_nested_value(eval_results[0], \"bertscore/precision_mean\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeab09c5-b8b6-4551-9699-c0e48b841f48",
   "metadata": {},
   "source": [
    "### Show information from previous runs\n",
    "As long as you have a list of job_ids, you can load their evaluation data again and inspect their results.\n",
    "The `eval_data` variable contains a few sets of job ids from previous evaluations, that you can use to get more insights about how different models perform on this dataset.\n",
    "\n",
    "In the following, feel free to choose which job list you want to work, either your original `job_ids` or something from `eval_data` (hint: if you write `eval_data.` and then hit Tab you'll be suggested a few different lists)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddea7929",
   "metadata": {},
   "outputs": [],
   "source": [
    "ld.runs_to_eval_table(eval_data.mistral_10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99e09f95-6ea5-4f83-b13c-4ad4f45ff483",
   "metadata": {},
   "outputs": [],
   "source": [
    "ld.runs_to_eval_table(eval_data.bart_10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81235c96-cf4a-4e55-805d-0cef6daa669e",
   "metadata": {},
   "source": [
    "### See the samples with the best and worst values for a given model and metric\n",
    "\n",
    "Sometimes an individual average score is hard to interpret and to get some sense of it one wants to look into the data. With the following you can get more insights from the best and worst predictions for a given model and metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2c2e494-6036-41e9-81b2-0854e91dbba4",
   "metadata": {},
   "outputs": [],
   "source": [
    "ld.show_best_worst(eval_data.bart_10, \"hf://facebook/bart-large-cnn\", \"bertscore/f1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd7da512-e927-43b6-8450-89e20a55f5f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "ld.show_best_worst(eval_data.mistral_10, \"mistral://open-mistral-7b\", \"bertscore/f1\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}