demos/ragas_rag_eval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![ragas_image](images/ragas.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RAG Evaluations Demonstration: RAGAS\n",
    "\n",
    "\"Ragas is a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. RAG denotes a class of LLM applications that use external data to augment the LLM’s context. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Evaluation Dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Repo card metadata block was not found. Setting CardData to empty.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['question', 'ground_truth', 'answer', 'contexts'],\n",
       "    num_rows: 20\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download amnesty_qa dataset\n",
    "from datasets import load_dataset, Dataset\n",
    "\n",
    "amnesty_qa = load_dataset(\"explodinggradients/amnesty_qa\", \"english_v2\", trust_remote_code=True)\n",
    "eval_data = amnesty_qa['eval']\n",
    "eval_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following dataset from Exploding Gradients is representative of how we would need to gather data to run evaluations of our RAG Pipeline.\n",
    "\n",
    "The following data fields are needed:\n",
    "- question/input: The initial prompt provided to the llm (prior to RAG)\n",
    "- answer/output: The generated response from the llm (after RAG)\n",
    "- context: The context picked up by the retrieval stage of RAG\n",
    "- ground truth: A foundational answer to the question that the llm output can be compared to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'answer': 'Amnesty International made several recommendations to the Special '\n",
      "           'Rapporteur on Human Rights Defenders. These recommendations '\n",
      "           'include:\\n'\n",
      "           '\\n'\n",
      "           '1. Urging states to fully implement the UN Declaration on Human '\n",
      "           'Rights Defenders and ensure that national laws and policies are in '\n",
      "           'line with international human rights standards.\\n'\n",
      "           '\\n'\n",
      "           '2. Calling on states to create a safe and enabling environment for '\n",
      "           'human rights defenders, including by adopting legislation that '\n",
      "           'protects defenders from threats, attacks, and reprisals.\\n'\n",
      "           '\\n'\n",
      "           '3. Encouraging states to establish effective mechanisms for the '\n",
      "           'protection of human rights defenders, such as national human '\n",
      "           'rights institutions and specialized units within law enforcement '\n",
      "           'agencies.\\n'\n",
      "           '\\n'\n",
      "           '4. Urging states to investigate and hold accountable those '\n",
      "           'responsible for attacks and threats against human rights '\n",
      "           'defenders, ensuring that they are brought to justice in fair '\n",
      "           'trials.\\n'\n",
      "           '\\n'\n",
      "           '5. Calling on states to repeal or amend laws that are used to '\n",
      "           'criminalize or restrict the work of human rights defenders, '\n",
      "           'including laws on defamation, sedition, and counter-terrorism.\\n'\n",
      "           '\\n'\n",
      "           '6. Encouraging states to provide adequate resources and support to '\n",
      "           'human rights defenders, including funding for their work, access '\n",
      "           'to legal assistance, and psychosocial support.\\n'\n",
      "           '\\n'\n",
      "           '7. Urging states to consult and engage with human rights defenders '\n",
      "           'in the development and implementation of laws, policies, and '\n",
      "           'programs that affect their rights and work.\\n'\n",
      "           '\\n'\n",
      "           '8. Calling on states to ensure that human rights defenders are '\n",
      "           'able to access and share information freely, including through the '\n",
      "           'internet and other digital platforms.\\n'\n",
      "           '\\n'\n",
      "           '9. Encouraging states to protect the rights of specific groups of '\n",
      "           'human rights defenders who face particular risks and challenges, '\n",
      "           'such as women defenders, indigenous defenders, and defenders '\n",
      "           'working on issues related to sexual orientation and gender '\n",
      "           'identity.\\n'\n",
      "           '\\n'\n",
      "           '10. Urging states to cooperate with the Special Rapporteur and '\n",
      "           'other UN mechanisms in the promotion and protection of human '\n",
      "           'rights defenders, including by responding to communications and '\n",
      "           'requests for visits.',\n",
      " 'contexts': ['In recent years, Amnesty International has focused on issues '\n",
      "              'such as the increasing threats faced by human rights defenders, '\n",
      "              'particularly in authoritarian regimes, and the need for '\n",
      "              'stronger international mechanisms to support and protect their '\n",
      "              \"work. The organization's recommendations to the Special \"\n",
      "              'Rapporteur often include calls for greater accountability for '\n",
      "              'perpetrators of violence and harassment against defenders, as '\n",
      "              'well as measures to ensure their safety and freedom to carry '\n",
      "              'out their important work.',\n",
      "              'Amnesty International recommends that the Special Rapporteur on '\n",
      "              'the Situation of Human Rights Defenders embeds a focus on child '\n",
      "              'and young HRDs in the future work of the mandate. \\n'\n",
      "              'Amnesty International recommends raising awareness about the '\n",
      "              'differences between \"child\" and \"youth\" HRDs and the different '\n",
      "              'context, needs, barriers, and challenges they face as '\n",
      "              'defenders. \\n'\n",
      "              'Amnesty International recommends incorporating age '\n",
      "              'disaggregated data on the lived experiences of child and young '\n",
      "              'HRDs in all future reports or interventions by the Special '\n",
      "              'Rapporteur. \\n'\n",
      "              'Amnesty International recommends creating online and '\n",
      "              'face-to-face safe spaces for child and young HRDs and their '\n",
      "              'child and youth-led organizations to actively engage with the '\n",
      "              'Special and other human rights mechanisms.',\n",
      "              'Amnesty International is known for its advocacy work on behalf '\n",
      "              'of human rights defenders around the world. The organization '\n",
      "              'often provides detailed reports and recommendations to the '\n",
      "              'Special Rapporteur on Human Rights Defenders, highlighting '\n",
      "              'specific cases of abuse and calling for action to protect those '\n",
      "              'at risk.'],\n",
      " 'ground_truth': 'The recommendations made by Amnesty International to the '\n",
      "                 'Special Rapporteur on Human Rights Defenders include '\n",
      "                 'embedding a focus on child and young HRDs in future work, '\n",
      "                 'raising awareness about the differences and challenges they '\n",
      "                 'face, incorporating age disaggregated data in reports, and '\n",
      "                 'creating safe spaces for engagement.',\n",
      " 'question': 'What are the recommendations made by Amnesty International to '\n",
      "             'the Special Rapporteur on Human Rights Defenders?'}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "# sample the data\n",
    "\n",
    "pprint(eval_data[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RAGAS performs analysis directly on dataset objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = Dataset.from_dict(eval_data[2:7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to use local LLMs with RAGAS is to use langchain. RAGAS has a built in wrapper that automatically accepts langchain models. In this case, we'll use an Ollama server running a local instance of llama3 as our evaluation model (LLM-as-a-judge).\n",
    "\n",
    "**Note: It's likely possible to use models from HuggingFace and skirt the need to use Ollama by writing a custom class using the RAGAS BaseModel class, but I had lots of difficulty getting this to work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.llms import Ollama\n",
    "\n",
    "# create LLM using ollama\n",
    "llama3 = Ollama(model=\"llama3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.embeddings import OllamaEmbeddings\n",
    "\n",
    "# Create an embeddings object that \n",
    "embeddings = OllamaEmbeddings(model='llama3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "001e008777cb419680e5835b6700d52e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# import the metrics from RAGAS\n",
    "from ragas.metrics import (\n",
    "    answer_relevancy, \n",
    "    context_precision,\n",
    "    context_recall\n",
    ")\n",
    "# import the evaluate function from RAGAS\n",
    "from ragas import evaluate\n",
    "\n",
    "# run the evaluation\n",
    "results = evaluate(\n",
    "    dataset=dataset,\n",
    "    llm=llama3,\n",
    "    embeddings=embeddings,\n",
    "    metrics=[\n",
    "        answer_relevancy, \n",
    "        context_precision,\n",
    "        context_recall\n",
    "    ],\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>answer</th>\n",
       "      <th>contexts</th>\n",
       "      <th>answer_relevancy</th>\n",
       "      <th>context_precision</th>\n",
       "      <th>context_recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Which private companies in the Americas are the largest GHG emitters according to the Carbon Maj...</td>\n",
       "      <td>The largest private companies in the Americas that are the largest GHG emitters according to the...</td>\n",
       "      <td>According to the Carbon Majors database, the largest private companies in the Americas that are ...</td>\n",
       "      <td>[The issue of greenhouse gas emissions has become a major concern for environmentalists and poli...</td>\n",
       "      <td>0.890880</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What action did Amnesty International urge its supporters to take in response to the killing of ...</td>\n",
       "      <td>Amnesty International urged its supporters to send appeals for the defenders' freedom to Nigeria...</td>\n",
       "      <td>Amnesty International urged its supporters to write letters to the Nigerian government, calling ...</td>\n",
       "      <td>[In the case of the Ogoni 9, Amnesty International called on its supporters to take action by si...</td>\n",
       "      <td>0.919549</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What are the recommendations made by Amnesty International to the Special Rapporteur on Human Ri...</td>\n",
       "      <td>The recommendations made by Amnesty International to the Special Rapporteur on Human Rights Defe...</td>\n",
       "      <td>Amnesty International made several recommendations to the Special Rapporteur on Human Rights Def...</td>\n",
       "      <td>[In recent years, Amnesty International has focused on issues such as the increasing threats fac...</td>\n",
       "      <td>0.905215</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Who are the target audience of the two books created by Amnesty International on child rights?</td>\n",
       "      <td>The target audience of the two books created by Amnesty International on child rights are childr...</td>\n",
       "      <td>The target audience of the two books created by Amnesty International on child rights are likely...</td>\n",
       "      <td>[In addition to children, parents, teachers, and caregivers are also key target audiences for Am...</td>\n",
       "      <td>0.936374</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Which right guarantees access to comprehensive information about past human rights violations, i...</td>\n",
       "      <td>The right that guarantees access to comprehensive information about past human rights violations...</td>\n",
       "      <td>The right that guarantees access to comprehensive information about past human rights violations...</td>\n",
       "      <td>[The right to truth is a fundamental human right that seeks to uncover the full extent of past h...</td>\n",
       "      <td>0.876583</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                              question  \\\n",
       "0  Which private companies in the Americas are the largest GHG emitters according to the Carbon Maj...   \n",
       "1  What action did Amnesty International urge its supporters to take in response to the killing of ...   \n",
       "2  What are the recommendations made by Amnesty International to the Special Rapporteur on Human Ri...   \n",
       "3       Who are the target audience of the two books created by Amnesty International on child rights?   \n",
       "4  Which right guarantees access to comprehensive information about past human rights violations, i...   \n",
       "\n",
       "                                                                                          ground_truth  \\\n",
       "0  The largest private companies in the Americas that are the largest GHG emitters according to the...   \n",
       "1  Amnesty International urged its supporters to send appeals for the defenders' freedom to Nigeria...   \n",
       "2  The recommendations made by Amnesty International to the Special Rapporteur on Human Rights Defe...   \n",
       "3  The target audience of the two books created by Amnesty International on child rights are childr...   \n",
       "4  The right that guarantees access to comprehensive information about past human rights violations...   \n",
       "\n",
       "                                                                                                answer  \\\n",
       "0  According to the Carbon Majors database, the largest private companies in the Americas that are ...   \n",
       "1  Amnesty International urged its supporters to write letters to the Nigerian government, calling ...   \n",
       "2  Amnesty International made several recommendations to the Special Rapporteur on Human Rights Def...   \n",
       "3  The target audience of the two books created by Amnesty International on child rights are likely...   \n",
       "4  The right that guarantees access to comprehensive information about past human rights violations...   \n",
       "\n",
       "                                                                                              contexts  \\\n",
       "0  [The issue of greenhouse gas emissions has become a major concern for environmentalists and poli...   \n",
       "1  [In the case of the Ogoni 9, Amnesty International called on its supporters to take action by si...   \n",
       "2  [In recent years, Amnesty International has focused on issues such as the increasing threats fac...   \n",
       "3  [In addition to children, parents, teachers, and caregivers are also key target audiences for Am...   \n",
       "4  [The right to truth is a fundamental human right that seeks to uncover the full extent of past h...   \n",
       "\n",
       "   answer_relevancy  context_precision  context_recall  \n",
       "0          0.890880                1.0        0.666667  \n",
       "1          0.919549                1.0        1.000000  \n",
       "2          0.905215                1.0        1.000000  \n",
       "3          0.936374                1.0        1.000000  \n",
       "4          0.876583                1.0        1.000000  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 100)\n",
    "results_df = results.to_pandas()\n",
    "results_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look closer at how one of these metrics works:\n",
    "\n",
    "Answer Relevancy assesses how pertinent the generated answer is to the given prompt. A lower score is assigned to answers that are incomplete or contain redundant information and higher scores indicate better relevancy. This metric is computed using the question, the context and the answer.\n",
    "\n",
    "The Answer Relevancy is defined as the mean cosine similartiy of the original question to a number of artifical questions, which are generated (reverse engineered) based on the answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![answer_relvancy_eq](images/answer_relevancy_eq.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that in order to have a good answer relevancy score, you need to not only have accurate retrieval context, synthesis, and generation, but you also need to have a strong evaluator (in regards to both generation and embeddings). If your evaluator is unable to generate good questions based off the answers, then the cosine similarity may suffer and lower the overall score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RAGAS Pros\n",
    "- Ease of use (the simplest/most lightweight of the three frameworks)\n",
    "- Test set generation tools built in\n",
    "\n",
    "\n",
    "#### RAGAS Cons\n",
    "- No custom metrics (LLM-as-a-judge only)\n",
    "- No reasoning supplied by scores (lots of inherit trust)\n",
    "- Customizing your evaluator LLM is a little more difficult\n",
    "- Only supports RAG metrics (not well suited for overall evaluations)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}