7_LLM_RAG_Evaluation_231122.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "af775680",
      "metadata": {
        "id": "af775680"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/drive/17NP--ME3ycAd7DhVep2tECaKiJJEw3Q8?usp=sharing\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# RAG/LLM Evaluation using llama-index\n",
        "\n",
        "RAG/LLM 응답 평가 코드 실습\n",
        "\n",
        "231122\n",
        "\n",
        "- [ref1](https://betterprogramming.pub/llamaindex-how-to-evaluate-your-rag-retrieval-augmented-generation-applications-2c83490f489)\n",
        "- [ref2](https://gpt-index.readthedocs.io/en/stable/examples/evaluation/relevancy_eval.html)\n",
        "- [ref3](https://gpt-index.readthedocs.io/en/stable/examples/evaluation/faithfulness_eval.html)\n",
        "- [ref4](https://gpt-index.readthedocs.io/en/stable/examples/evaluation/correctness_eval.html)\n",
        "- [ref5](https://github.com/run-llama/llama_index/tree/main/docs/examples/evaluation)\n",
        "- [ref6](https://blog.llamaindex.ai/building-and-evaluating-a-qa-system-with-llamaindex-3f02e9d87ce1)"
      ],
      "metadata": {
        "id": "-RD1WAuxvPtq"
      },
      "id": "-RD1WAuxvPtq"
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 0) OpenAI API키 설정\n",
        "\n",
        "- [API key 확인](https://platform.openai.com/api-keys)\n",
        "- [API key 발급받기](https://m.blog.naver.com/mynameistk/223062993136)\n",
        "\n",
        "본인 OPENAI_API_KEY를 아래에 입력하세요(예시: OPENAI_API_KEY: 'sk-xxxxxxxx')"
      ],
      "metadata": {
        "id": "QzF0AEk4vLJG"
      },
      "id": "QzF0AEk4vLJG"
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "OPENAI_API_KEY='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'\n",
        "os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY"
      ],
      "metadata": {
        "id": "nJM6OIubvN1m"
      },
      "id": "nJM6OIubvN1m",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "f3f797ad",
      "metadata": {
        "id": "f3f797ad"
      },
      "source": [
        "# 0) Setting\n",
        "\n",
        "This notebook walks through the process of generating a list of questions that could be asked about your data. This is useful for setting up an evaluation pipeline using the `FaithfulnessEvaluator` and `RelevancyEvaluator` evaluation tools."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6a2cfee1",
      "metadata": {
        "id": "6a2cfee1"
      },
      "source": [
        "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "848aa824",
      "metadata": {
        "id": "848aa824",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "243981dc-2358-428a-b41b-eb3bceec7ac2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting llama-index==0.9.4\n",
            "  Downloading llama_index-0.9.4-py3-none-any.whl (890 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/890.2 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m204.8/890.2 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━\u001b[0m \u001b[32m706.6/890.2 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m890.2/890.2 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (2.0.23)\n",
            "Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (3.8.6)\n",
            "Collecting aiostream<0.6.0,>=0.5.2 (from llama-index==0.9.4)\n",
            "  Downloading aiostream-0.5.2-py3-none-any.whl (39 kB)\n",
            "Collecting beautifulsoup4<5.0.0,>=4.12.2 (from llama-index==0.9.4)\n",
            "  Downloading beautifulsoup4-4.12.2-py3-none-any.whl (142 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.0/143.0 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting dataclasses-json<0.6.0,>=0.5.7 (from llama-index==0.9.4)\n",
            "  Downloading dataclasses_json-0.5.14-py3-none-any.whl (26 kB)\n",
            "Collecting deprecated>=1.2.9.3 (from llama-index==0.9.4)\n",
            "  Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (2023.6.0)\n",
            "Collecting httpx (from llama-index==0.9.4)\n",
            "  Downloading httpx-0.25.1-py3-none-any.whl (75 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.0/75.0 kB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (1.5.8)\n",
            "Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (3.8.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (1.23.5)\n",
            "Collecting openai>=1.1.0 (from llama-index==0.9.4)\n",
            "  Downloading openai-1.3.4-py3-none-any.whl (220 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m220.5/220.5 kB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas[jinja2] in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (1.5.3)\n",
            "Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (8.2.3)\n",
            "Collecting tiktoken>=0.3.3 (from llama-index==0.9.4)\n",
            "  Downloading tiktoken-0.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m27.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-index==0.9.4) (4.5.0)\n",
            "Collecting typing-inspect>=0.8.0 (from llama-index==0.9.4)\n",
            "  Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
            "Collecting urllib3<2 (from llama-index==0.9.4)\n",
            "  Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.8/143.8 kB\u001b[0m \u001b[31m21.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index==0.9.4) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index==0.9.4) (3.3.2)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index==0.9.4) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index==0.9.4) (4.0.3)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index==0.9.4) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index==0.9.4) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index==0.9.4) (1.3.1)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4<5.0.0,>=4.12.2->llama-index==0.9.4) (2.5)\n",
            "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.6.0,>=0.5.7->llama-index==0.9.4)\n",
            "  Downloading marshmallow-3.20.1-py3-none-any.whl (49 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated>=1.2.9.3->llama-index==0.9.4) (1.14.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->llama-index==0.9.4) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->llama-index==0.9.4) (1.3.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->llama-index==0.9.4) (2023.6.3)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->llama-index==0.9.4) (4.66.1)\n",
            "Requirement already satisfied: anyio<4,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai>=1.1.0->llama-index==0.9.4) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai>=1.1.0->llama-index==0.9.4) (1.7.0)\n",
            "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai>=1.1.0->llama-index==0.9.4) (1.10.13)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx->llama-index==0.9.4) (2023.7.22)\n",
            "Collecting httpcore (from httpx->llama-index==0.9.4)\n",
            "  Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m11.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->llama-index==0.9.4) (3.4)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->llama-index==0.9.4) (1.3.0)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index==0.9.4) (3.0.1)\n",
            "Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from tiktoken>=0.3.3->llama-index==0.9.4) (2.31.0)\n",
            "Collecting mypy-extensions>=0.3.0 (from typing-inspect>=0.8.0->llama-index==0.9.4)\n",
            "  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
            "\u001b[33mWARNING: pandas 1.5.3 does not provide the extra 'jinja2'\u001b[0m\u001b[33m\n",
            "\u001b[0mRequirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas[jinja2]->llama-index==0.9.4) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas[jinja2]->llama-index==0.9.4) (2023.3.post1)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai>=1.1.0->llama-index==0.9.4) (1.1.3)\n",
            "Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.10/dist-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.6.0,>=0.5.7->llama-index==0.9.4) (23.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas[jinja2]->llama-index==0.9.4) (1.16.0)\n",
            "Collecting h11<0.15,>=0.13 (from httpcore->httpx->llama-index==0.9.4)\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: urllib3, mypy-extensions, marshmallow, h11, deprecated, beautifulsoup4, aiostream, typing-inspect, httpcore, tiktoken, httpx, dataclasses-json, openai, llama-index\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 2.0.7\n",
            "    Uninstalling urllib3-2.0.7:\n",
            "      Successfully uninstalled urllib3-2.0.7\n",
            "  Attempting uninstall: beautifulsoup4\n",
            "    Found existing installation: beautifulsoup4 4.11.2\n",
            "    Uninstalling beautifulsoup4-4.11.2:\n",
            "      Successfully uninstalled beautifulsoup4-4.11.2\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "llmx 0.0.15a0 requires cohere, which is not installed.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed aiostream-0.5.2 beautifulsoup4-4.12.2 dataclasses-json-0.5.14 deprecated-1.2.14 h11-0.14.0 httpcore-1.0.2 httpx-0.25.1 llama-index-0.9.4 marshmallow-3.20.1 mypy-extensions-1.0.0 openai-1.3.4 tiktoken-0.5.1 typing-inspect-0.9.0 urllib3-1.26.18\n"
          ]
        }
      ],
      "source": [
        "!pip install llama-index==0.9.4"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "qaSQAMUhtG2L"
      },
      "id": "qaSQAMUhtG2L"
    },
    {
      "cell_type": "code",
      "source": [
        "# attach to the same event-loop\n",
        "import nest_asyncio\n",
        "\n",
        "nest_asyncio.apply()"
      ],
      "metadata": {
        "id": "qDuFqsn547hB"
      },
      "id": "qDuFqsn547hB",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!mkdir -p './test_wiki_data'\n",
        "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/evaluation/test_wiki_data/nyc_text.txt' -O './test_wiki_data/nyc_text.txt'"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NF7px2QB2ygh",
        "outputId": "41daa39f-3776-456e-e05a-0cfa72b72033"
      },
      "id": "NF7px2QB2ygh",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-11-21 13:03:25--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/evaluation/test_wiki_data/nyc_text.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 130289 (127K) [text/plain]\n",
            "Saving to: ‘./test_wiki_data/nyc_text.txt’\n",
            "\n",
            "\r          ./test_wi   0%[                    ]       0  --.-KB/s               \r./test_wiki_data/ny 100%[===================>] 127.24K  --.-KB/s    in 0.01s   \n",
            "\n",
            "2023-11-21 13:03:25 (10.9 MB/s) - ‘./test_wiki_data/nyc_text.txt’ saved [130289/130289]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1) Relevancy Evaluator\n",
        "- [ref](https://github.com/run-llama/llama_index/blob/main/docs/examples/evaluation/relevancy_eval.ipynb)\n",
        "\n",
        "This notebook uses the RelevancyEvaluator to measure if the `response + source` nodes match the `query`.\n",
        "This is useful for measuring if the query was actually answered by the response.\n",
        "\n",
        "<img src=\"https://miro.medium.com/v2/resize:fit:1100/format:webp/1*2dXP4Hr1dBYzPqBcjfRFVg.png\" width=\"500\">"
      ],
      "metadata": {
        "id": "wKqhWtX-8yVl"
      },
      "id": "wKqhWtX-8yVl"
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index import (\n",
        "    TreeIndex,\n",
        "    VectorStoreIndex,\n",
        "    SimpleDirectoryReader,\n",
        "    LLMPredictor,\n",
        "    ServiceContext,\n",
        "    Response,\n",
        ")\n",
        "from llama_index.llms import OpenAI\n",
        "from llama_index.evaluation import RelevancyEvaluator\n",
        "import pandas as pd\n",
        "\n",
        "pd.set_option(\"display.max_colwidth\", 0)"
      ],
      "metadata": {
        "id": "iQGuINVf8KGS"
      },
      "id": "iQGuINVf8KGS",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# gpt-3 (davinci)\n",
        "gpt3 = OpenAI(temperature=0, model=\"text-davinci-003\")\n",
        "service_context_gpt3 = ServiceContext.from_defaults(llm=gpt3)\n",
        "\n",
        "# gpt-4\n",
        "gpt4 = OpenAI(temperature=0, model=\"gpt-4\")\n",
        "service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZhAvMcJi8KAO",
        "outputId": "fbffa336-e449-47bf-82dd-579b279dca69"
      },
      "id": "ZhAvMcJi8KAO",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /tmp/llama_index...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "evaluator = RelevancyEvaluator(service_context=service_context_gpt3)\n",
        "evaluator_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)"
      ],
      "metadata": {
        "id": "xHsNVVvg8J5p"
      },
      "id": "xHsNVVvg8J5p",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "documents = SimpleDirectoryReader(\"./test_wiki_data\").load_data()"
      ],
      "metadata": {
        "id": "wOQ_kXv48Jx_"
      },
      "id": "wOQ_kXv48Jx_",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create vector index\n",
        "vector_index = VectorStoreIndex.from_documents(\n",
        "    documents, service_context=ServiceContext.from_defaults(chunk_size=512)\n",
        ")"
      ],
      "metadata": {
        "id": "6v0PADBZ8Prq"
      },
      "id": "6v0PADBZ8Prq",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# define jupyter display function\n",
        "def display_eval_df(query: str, response: Response, eval_result: str) -> None:\n",
        "    eval_df = pd.DataFrame(\n",
        "        {\n",
        "            \"Query\": query,\n",
        "            \"Response\": str(response),\n",
        "            \"Source\": response.source_nodes[0].node.text[:1000] + \"...\",\n",
        "            \"Evaluation Result\": \"Pass\" if eval_result.passing else \"Fail\",\n",
        "        },\n",
        "        index=[0],\n",
        "    )\n",
        "    eval_df = eval_df.style.set_properties(\n",
        "        **{\n",
        "            \"inline-size\": \"600px\",\n",
        "            \"overflow-wrap\": \"break-word\",\n",
        "        },\n",
        "        subset=[\"Response\", \"Source\"]\n",
        "    )\n",
        "    display(eval_df)"
      ],
      "metadata": {
        "id": "T9DxkSrB8Piq"
      },
      "id": "T9DxkSrB8Piq",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "query_str = (\n",
        "    \"What battles took place in New York City in the American Revolution?\"\n",
        ")\n",
        "query_engine = vector_index.as_query_engine()\n",
        "response_vector = query_engine.query(query_str)\n",
        "eval_result = evaluator_gpt4.evaluate_response(\n",
        "    query=query_str, response=response_vector\n",
        ")\n",
        "\n",
        "display_eval_df(query_str, response_vector, eval_result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i9HoHIQR8SAH",
        "outputId": "a7f982b4-aed3-45d1-cb18-1eb99d60ee56"
      },
      "id": "i9HoHIQR8SAH",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7b97b24eeec0>"
            ],
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_bf14f_row0_col1, #T_bf14f_row0_col2 {\n",
              "  inline-size: 600px;\n",
              "  overflow-wrap: break-word;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_bf14f\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_bf14f_level0_col0\" class=\"col_heading level0 col0\" >Query</th>\n",
              "      <th id=\"T_bf14f_level0_col1\" class=\"col_heading level0 col1\" >Response</th>\n",
              "      <th id=\"T_bf14f_level0_col2\" class=\"col_heading level0 col2\" >Source</th>\n",
              "      <th id=\"T_bf14f_level0_col3\" class=\"col_heading level0 col3\" >Evaluation Result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_bf14f_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_bf14f_row0_col0\" class=\"data row0 col0\" >What battles took place in New York City in the American Revolution?</td>\n",
              "      <td id=\"T_bf14f_row0_col1\" class=\"data row0 col1\" >The Battle of Long Island was the largest battle of the American Revolutionary War that took place in New York City.</td>\n",
              "      <td id=\"T_bf14f_row0_col2\" class=\"data row0 col2\" >=== American Revolution ===\n",
              "\n",
              "The Stamp Act Congress met in New York in October 1765, as the Sons of Liberty organization emerged in the city and skirmished over the next ten years with British troops stationed there. The Battle of Long Island, the largest battle of the American Revolutionary War, was fought in August 1776 within the modern-day borough of Brooklyn. After the battle, in which the Americans were defeated, the British made the city their military and political base of operations in North America. The city was a haven for Loyalist refugees and escaped slaves who joined the British lines for freedom newly promised by the Crown for all fighters. As many as 10,000 escaped slaves crowded into the city during the British occupation. When the British forces evacuated at the close of the war in 1783, they transported 3,000 freedmen for resettlement in Nova Scotia. They resettled other freedmen in England and the Caribbean.\n",
              "The only attempt at a peaceful solution to the war took pl...</td>\n",
              "      <td id=\"T_bf14f_row0_col3\" class=\"data row0 col3\" >Pass</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "query_str = \"What are the airports in New York City?\"\n",
        "query_engine = vector_index.as_query_engine()\n",
        "response_vector = query_engine.query(query_str)\n",
        "eval_result = evaluator_gpt4.evaluate_response(\n",
        "    query=query_str, response=response_vector\n",
        ")\n",
        "display_eval_df(query_str, response_vector, eval_result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0P6lJNyh8R2V",
        "outputId": "6b7e12e3-7bf6-400a-e809-1876952fc568"
      },
      "id": "0P6lJNyh8R2V",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7b97b28dae30>"
            ],
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_699ba_row0_col1, #T_699ba_row0_col2 {\n",
              "  inline-size: 600px;\n",
              "  overflow-wrap: break-word;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_699ba\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_699ba_level0_col0\" class=\"col_heading level0 col0\" >Query</th>\n",
              "      <th id=\"T_699ba_level0_col1\" class=\"col_heading level0 col1\" >Response</th>\n",
              "      <th id=\"T_699ba_level0_col2\" class=\"col_heading level0 col2\" >Source</th>\n",
              "      <th id=\"T_699ba_level0_col3\" class=\"col_heading level0 col3\" >Evaluation Result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_699ba_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_699ba_row0_col0\" class=\"data row0 col0\" >What are the airports in New York City?</td>\n",
              "      <td id=\"T_699ba_row0_col1\" class=\"data row0 col1\" >The airports in New York City include John F. Kennedy International Airport, Newark Liberty International Airport, LaGuardia Airport, Stewart International Airport, Long Island MacArthur Airport, Trenton–Mercer Airport, and Westchester County Airport.</td>\n",
              "      <td id=\"T_699ba_row0_col2\" class=\"data row0 col2\" >along the Northeast Corridor, and long-distance train service to other North American cities.The Staten Island Railway rapid transit system solely serves Staten Island, operating 24 hours a day. The Port Authority Trans-Hudson (PATH train) links Midtown and Lower Manhattan to northeastern New Jersey, primarily Hoboken, Jersey City, and Newark. Like the New York City Subway, the PATH operates 24 hours a day; meaning three of the six rapid transit systems in the world which operate on 24-hour schedules are wholly or partly in New York (the others are a portion of the Chicago \"L\", the PATCO Speedline serving Philadelphia, and the Copenhagen Metro).\n",
              "Multibillion-dollar heavy rail transit projects under construction in New York City include the Second Avenue Subway, and the East Side Access project.\n",
              "\n",
              "\n",
              "==== Buses ====\n",
              "\n",
              "New York City's public bus fleet runs 24/7 and is the largest in North America. The Port Authority Bus Terminal, the main intercity bus terminal of the city, serves 7,000 buse...</td>\n",
              "      <td id=\"T_699ba_row0_col3\" class=\"data row0 col3\" >Pass</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "query_str = \"Who is the mayor of New York City?\"\n",
        "query_engine = vector_index.as_query_engine()\n",
        "response_vector = query_engine.query(query_str)\n",
        "eval_result = evaluator_gpt4.evaluate_response(\n",
        "    query=query_str, response=response_vector\n",
        ")\n",
        "display_eval_df(query_str, response_vector, eval_result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zxc02-_d8Rn-",
        "outputId": "96391dfb-fbbf-4ca9-d281-c0483f9ee016"
      },
      "id": "zxc02-_d8Rn-",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7b97b24ef8e0>"
            ],
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_e5de3_row0_col1, #T_e5de3_row0_col2 {\n",
              "  inline-size: 600px;\n",
              "  overflow-wrap: break-word;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_e5de3\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_e5de3_level0_col0\" class=\"col_heading level0 col0\" >Query</th>\n",
              "      <th id=\"T_e5de3_level0_col1\" class=\"col_heading level0 col1\" >Response</th>\n",
              "      <th id=\"T_e5de3_level0_col2\" class=\"col_heading level0 col2\" >Source</th>\n",
              "      <th id=\"T_e5de3_level0_col3\" class=\"col_heading level0 col3\" >Evaluation Result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_e5de3_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_e5de3_row0_col0\" class=\"data row0 col0\" >Who is the mayor of New York City?</td>\n",
              "      <td id=\"T_e5de3_row0_col1\" class=\"data row0 col1\" >Eric Adams is the mayor of New York City.</td>\n",
              "      <td id=\"T_e5de3_row0_col2\" class=\"data row0 col2\" >=== Politics ===\n",
              "The present mayor is Eric Adams. He was elected in 2021 with 67% of the vote, and assumed office on January 1, 2022.\n",
              "The Democratic Party holds the majority of public offices. As of April 2016, 69% of registered voters in the city are Democrats and 10% are Republicans. New York City has not been carried by a Republican  presidential election since President Calvin Coolidge won the five boroughs in 1924. A Republican candidate for statewide office has not won all five boroughs of the city since it was incorporated in 1898. In 2012, Democrat Barack Obama became the first presidential candidate of any party to receive more than 80% of the overall vote in New York City, sweeping all five boroughs. Party platforms center on affordable housing, education, and economic development, and labor politics are of importance in the city. Thirteen out of 27 U.S. congressional districts in the state of New York include portions of New York City.New York is one of the most important so...</td>\n",
              "      <td id=\"T_e5de3_row0_col3\" class=\"data row0 col3\" >Pass</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Evaluate Source Nodes\n",
        "Evaluate the set of returned sources, and determine which sources actually contain the answer to a given query."
      ],
      "metadata": {
        "id": "Hd11IErQ8eDh"
      },
      "id": "Hd11IErQ8eDh"
    },
    {
      "cell_type": "code",
      "source": [
        "from typing import List\n",
        "\n",
        "\n",
        "# define jupyter display function\n",
        "def display_eval_sources(\n",
        "    query: str, response: Response, eval_result: List[str]\n",
        ") -> None:\n",
        "    sources = [s.node.get_text() for s in response.source_nodes]\n",
        "    eval_df = pd.DataFrame(\n",
        "        {\n",
        "            \"Source\": sources,\n",
        "            \"Eval Result\": eval_result,\n",
        "        },\n",
        "    )\n",
        "    eval_df.style.set_caption(query)\n",
        "    eval_df = eval_df.style.set_properties(\n",
        "        **{\n",
        "            \"inline-size\": \"600px\",\n",
        "            \"overflow-wrap\": \"break-word\",\n",
        "        },\n",
        "        subset=[\"Source\"]\n",
        "    )\n",
        "\n",
        "    display(eval_df)"
      ],
      "metadata": {
        "id": "SYn0UODb8dUl"
      },
      "id": "SYn0UODb8dUl",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# NOTE: you can set response_mode=\"no_text\" to get just the sources\n",
        "query_str = \"What are the airports in New York City?\"\n",
        "query_engine = vector_index.as_query_engine(\n",
        "    similarity_top_k=3, response_mode=\"no_text\"\n",
        ")\n",
        "response_vector = query_engine.query(query_str)\n",
        "eval_source_result_full = [\n",
        "    evaluator_gpt4.evaluate(\n",
        "        query=query_str,\n",
        "        response=response_vector.response,\n",
        "        contexts=[source_node.get_content()],\n",
        "    )\n",
        "    for source_node in response_vector.source_nodes\n",
        "]\n",
        "eval_source_result = [\n",
        "    \"Pass\" if result.passing else \"Fail\" for result in eval_source_result_full\n",
        "]"
      ],
      "metadata": {
        "id": "FSd23k1d8dPF"
      },
      "id": "FSd23k1d8dPF",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "display_eval_sources(query_str, response_vector, eval_source_result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iBQ6SPrV8dJU",
        "outputId": "9dc51a6e-b0d1-478b-bfdc-9a8d1efe960f"
      },
      "id": "iBQ6SPrV8dJU",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7b97b22f6590>"
            ],
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_d0dbe_row0_col0, #T_d0dbe_row1_col0, #T_d0dbe_row2_col0 {\n",
              "  inline-size: 600px;\n",
              "  overflow-wrap: break-word;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_d0dbe\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_d0dbe_level0_col0\" class=\"col_heading level0 col0\" >Source</th>\n",
              "      <th id=\"T_d0dbe_level0_col1\" class=\"col_heading level0 col1\" >Eval Result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_d0dbe_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_d0dbe_row0_col0\" class=\"data row0 col0\" >along the Northeast Corridor, and long-distance train service to other North American cities.The Staten Island Railway rapid transit system solely serves Staten Island, operating 24 hours a day. The Port Authority Trans-Hudson (PATH train) links Midtown and Lower Manhattan to northeastern New Jersey, primarily Hoboken, Jersey City, and Newark. Like the New York City Subway, the PATH operates 24 hours a day; meaning three of the six rapid transit systems in the world which operate on 24-hour schedules are wholly or partly in New York (the others are a portion of the Chicago \"L\", the PATCO Speedline serving Philadelphia, and the Copenhagen Metro).\n",
              "Multibillion-dollar heavy rail transit projects under construction in New York City include the Second Avenue Subway, and the East Side Access project.\n",
              "\n",
              "\n",
              "==== Buses ====\n",
              "\n",
              "New York City's public bus fleet runs 24/7 and is the largest in North America. The Port Authority Bus Terminal, the main intercity bus terminal of the city, serves 7,000 buses and 200,000 commuters daily, making it the busiest bus station in the world.\n",
              "\n",
              "\n",
              "=== Air ===\n",
              "\n",
              "New York's airspace is the busiest in the United States and one of the world's busiest air transportation corridors. The three busiest airports in the New York metropolitan area include John F. Kennedy International Airport, Newark Liberty International Airport, and LaGuardia Airport; 130.5 million travelers used these three airports in 2016. JFK and Newark Liberty were the busiest and fourth busiest U.S. gateways for international air passengers, respectively, in 2012; as of 2011, JFK was the busiest airport for international passengers in North America.Plans have advanced to expand passenger volume at a fourth airport, Stewart International Airport near Newburgh, New York, by the Port Authority of New York and New Jersey. Plans were announced in July 2015 to entirely rebuild LaGuardia Airport in a multibillion-dollar project to replace its aging facilities. Other commercial airports in or serving the New York metropolitan area include Long Island MacArthur Airport, Trenton–Mercer Airport and Westchester County Airport. The primary general aviation airport serving the area is Teterboro Airport.</td>\n",
              "      <td id=\"T_d0dbe_row0_col1\" class=\"data row0 col1\" >Fail</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_d0dbe_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_d0dbe_row1_col0\" class=\"data row1 col0\" >See or edit raw graph data.\n",
              "\n",
              "\n",
              "=== Parks ===\n",
              "\n",
              "The city of New York has a complex park system, with various lands operated by the National Park Service, the New York State Office of Parks, Recreation and Historic Preservation, and the New York City Department of Parks and Recreation. In its 2018 ParkScore ranking, the Trust for Public Land reported that the park system in New York City was the ninth-best park system among the fifty most populous U.S. cities. ParkScore ranks urban park systems by a formula that analyzes median park size, park acres as percent of city area, the percent of city residents within a half-mile of a park, spending of park services per resident, and the number of playgrounds per 10,000 residents. In 2021, the New York City Council banned the use of synthetic pesticides by city agencies and instead required organic lawn management. The effort was started by teacher Paula Rogovin's kindergarten class at P.S. 290.\n",
              "\n",
              "\n",
              "==== National parks ====\n",
              "\n",
              "Gateway National Recreation Area contains over 26,000 acres (110 km2), most of it in New York City. In Brooklyn and Queens, the park contains over 9,000 acres (36 km2) of salt marsh, wetlands, islands, and water, including most of Jamaica Bay and the Jamaica Bay Wildlife Refuge. Also in Queens, the park includes a significant portion of the western Rockaway Peninsula, most notably Jacob Riis Park and Fort Tilden. In Staten Island, it includes Fort Wadsworth, with historic pre-Civil War era Battery Weed and Fort Tompkins, and Great Kills Park, with beaches, trails, and a marina.\n",
              "The Statue of Liberty National Monument and Ellis Island Immigration Museum are managed by the National Park Service and are in both New York and New Jersey. They are joined in the harbor by Governors Island National Monument. Historic sites under federal management on Manhattan Island include Stonewall National Monument; Castle Clinton National Monument; Federal Hall National Memorial; Theodore Roosevelt Birthplace National Historic Site; General Grant National Memorial (Grant's Tomb); African Burial Ground National Monument; and Hamilton Grange National Memorial. Hundreds of properties are listed on the National Register of Historic Places or as a National Historic Landmark.</td>\n",
              "      <td id=\"T_d0dbe_row1_col1\" class=\"data row1 col1\" >Fail</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_d0dbe_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_d0dbe_row2_col0\" class=\"data row2 col0\" >New York has witnessed a growing combined volume of international and domestic tourists, reflecting over 60 million visitors to the city per year, the world's busiest tourist destination. Approximately 12 million visitors to New York City have been from outside the United States, with the highest numbers from the United Kingdom, Canada, Brazil, and China. Multiple sources have called New York the most photographed city in the world.I Love New York (stylized I ❤ NY) is both a logo and a song that are the basis of an advertising campaign and have been used since 1977 to promote tourism in New York City, and later to promote New York State as well. The trademarked logo, owned by New York State Empire State Development, appears in souvenir shops and brochures throughout the city and state, some licensed, many not. The song is the state song of New York.\n",
              "The majority of the most high-profile tourist destinations to the city are situated in Manhattan. These include Times Square; Broadway theater productions; the Empire State Building; the Statue of Liberty; Ellis Island; the United Nations headquarters; the World Trade Center (including the National September 11 Memorial & Museum and One World Trade Center); the art museums along Museum Mile; green spaces such as Central Park, Washington Square Park, the High Line, and the medieval gardens of The Cloisters; the Stonewall Inn; Rockefeller Center; ethnic enclaves including the Manhattan Chinatown, Koreatown, Curry Hill, Harlem, Spanish Harlem, Little Italy, and Little Australia; luxury shopping along Fifth and Madison Avenues; and events such as the Halloween Parade in Greenwich Village; the Brooklyn Bridge (shared with Brooklyn); the Macy's Thanksgiving Day Parade; the lighting of the Rockefeller Center Christmas Tree; the St. Patrick's Day Parade; seasonal activities such as ice skating in Central Park in the wintertime; the Tribeca Film Festival; and free performances in Central Park at SummerStage.Points of interest have also developed in the city outside Manhattan and have made the outer boroughs tourist destinations in their own right.</td>\n",
              "      <td id=\"T_d0dbe_row2_col1\" class=\"data row2 col1\" >Fail</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# NOTE: you can set response_mode=\"no_text\" to get just the sources\n",
        "query_str = \"Who is the mayor of New York City?\"\n",
        "query_engine = vector_index.as_query_engine(\n",
        "    similarity_top_k=3, response_mode=\"no_text\"\n",
        ")\n",
        "eval_source_result_full = [\n",
        "    evaluator_gpt4.evaluate(\n",
        "        query=query_str,\n",
        "        response=response_vector.response,\n",
        "        contexts=[source_node.get_content()],\n",
        "    )\n",
        "    for source_node in response_vector.source_nodes\n",
        "]\n",
        "eval_source_result = [\n",
        "    \"Pass\" if result.passing else \"Fail\" for result in eval_source_result_full\n",
        "]\n",
        "\n",
        "display_eval_sources(query_str, response_vector, eval_source_result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kOedeb_G8dDR",
        "outputId": "54fbaee0-3f3e-412c-a2c5-e73d8cb99f6a"
      },
      "id": "kOedeb_G8dDR",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7b979ad93c70>"
            ],
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_05c61_row0_col0, #T_05c61_row1_col0, #T_05c61_row2_col0 {\n",
              "  inline-size: 600px;\n",
              "  overflow-wrap: break-word;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_05c61\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_05c61_level0_col0\" class=\"col_heading level0 col0\" >Source</th>\n",
              "      <th id=\"T_05c61_level0_col1\" class=\"col_heading level0 col1\" >Eval Result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_05c61_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_05c61_row0_col0\" class=\"data row0 col0\" >along the Northeast Corridor, and long-distance train service to other North American cities.The Staten Island Railway rapid transit system solely serves Staten Island, operating 24 hours a day. The Port Authority Trans-Hudson (PATH train) links Midtown and Lower Manhattan to northeastern New Jersey, primarily Hoboken, Jersey City, and Newark. Like the New York City Subway, the PATH operates 24 hours a day; meaning three of the six rapid transit systems in the world which operate on 24-hour schedules are wholly or partly in New York (the others are a portion of the Chicago \"L\", the PATCO Speedline serving Philadelphia, and the Copenhagen Metro).\n",
              "Multibillion-dollar heavy rail transit projects under construction in New York City include the Second Avenue Subway, and the East Side Access project.\n",
              "\n",
              "\n",
              "==== Buses ====\n",
              "\n",
              "New York City's public bus fleet runs 24/7 and is the largest in North America. The Port Authority Bus Terminal, the main intercity bus terminal of the city, serves 7,000 buses and 200,000 commuters daily, making it the busiest bus station in the world.\n",
              "\n",
              "\n",
              "=== Air ===\n",
              "\n",
              "New York's airspace is the busiest in the United States and one of the world's busiest air transportation corridors. The three busiest airports in the New York metropolitan area include John F. Kennedy International Airport, Newark Liberty International Airport, and LaGuardia Airport; 130.5 million travelers used these three airports in 2016. JFK and Newark Liberty were the busiest and fourth busiest U.S. gateways for international air passengers, respectively, in 2012; as of 2011, JFK was the busiest airport for international passengers in North America.Plans have advanced to expand passenger volume at a fourth airport, Stewart International Airport near Newburgh, New York, by the Port Authority of New York and New Jersey. Plans were announced in July 2015 to entirely rebuild LaGuardia Airport in a multibillion-dollar project to replace its aging facilities. Other commercial airports in or serving the New York metropolitan area include Long Island MacArthur Airport, Trenton–Mercer Airport and Westchester County Airport. The primary general aviation airport serving the area is Teterboro Airport.</td>\n",
              "      <td id=\"T_05c61_row0_col1\" class=\"data row0 col1\" >Fail</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_05c61_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_05c61_row1_col0\" class=\"data row1 col0\" >See or edit raw graph data.\n",
              "\n",
              "\n",
              "=== Parks ===\n",
              "\n",
              "The city of New York has a complex park system, with various lands operated by the National Park Service, the New York State Office of Parks, Recreation and Historic Preservation, and the New York City Department of Parks and Recreation. In its 2018 ParkScore ranking, the Trust for Public Land reported that the park system in New York City was the ninth-best park system among the fifty most populous U.S. cities. ParkScore ranks urban park systems by a formula that analyzes median park size, park acres as percent of city area, the percent of city residents within a half-mile of a park, spending of park services per resident, and the number of playgrounds per 10,000 residents. In 2021, the New York City Council banned the use of synthetic pesticides by city agencies and instead required organic lawn management. The effort was started by teacher Paula Rogovin's kindergarten class at P.S. 290.\n",
              "\n",
              "\n",
              "==== National parks ====\n",
              "\n",
              "Gateway National Recreation Area contains over 26,000 acres (110 km2), most of it in New York City. In Brooklyn and Queens, the park contains over 9,000 acres (36 km2) of salt marsh, wetlands, islands, and water, including most of Jamaica Bay and the Jamaica Bay Wildlife Refuge. Also in Queens, the park includes a significant portion of the western Rockaway Peninsula, most notably Jacob Riis Park and Fort Tilden. In Staten Island, it includes Fort Wadsworth, with historic pre-Civil War era Battery Weed and Fort Tompkins, and Great Kills Park, with beaches, trails, and a marina.\n",
              "The Statue of Liberty National Monument and Ellis Island Immigration Museum are managed by the National Park Service and are in both New York and New Jersey. They are joined in the harbor by Governors Island National Monument. Historic sites under federal management on Manhattan Island include Stonewall National Monument; Castle Clinton National Monument; Federal Hall National Memorial; Theodore Roosevelt Birthplace National Historic Site; General Grant National Memorial (Grant's Tomb); African Burial Ground National Monument; and Hamilton Grange National Memorial. Hundreds of properties are listed on the National Register of Historic Places or as a National Historic Landmark.</td>\n",
              "      <td id=\"T_05c61_row1_col1\" class=\"data row1 col1\" >Fail</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_05c61_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_05c61_row2_col0\" class=\"data row2 col0\" >New York has witnessed a growing combined volume of international and domestic tourists, reflecting over 60 million visitors to the city per year, the world's busiest tourist destination. Approximately 12 million visitors to New York City have been from outside the United States, with the highest numbers from the United Kingdom, Canada, Brazil, and China. Multiple sources have called New York the most photographed city in the world.I Love New York (stylized I ❤ NY) is both a logo and a song that are the basis of an advertising campaign and have been used since 1977 to promote tourism in New York City, and later to promote New York State as well. The trademarked logo, owned by New York State Empire State Development, appears in souvenir shops and brochures throughout the city and state, some licensed, many not. The song is the state song of New York.\n",
              "The majority of the most high-profile tourist destinations to the city are situated in Manhattan. These include Times Square; Broadway theater productions; the Empire State Building; the Statue of Liberty; Ellis Island; the United Nations headquarters; the World Trade Center (including the National September 11 Memorial & Museum and One World Trade Center); the art museums along Museum Mile; green spaces such as Central Park, Washington Square Park, the High Line, and the medieval gardens of The Cloisters; the Stonewall Inn; Rockefeller Center; ethnic enclaves including the Manhattan Chinatown, Koreatown, Curry Hill, Harlem, Spanish Harlem, Little Italy, and Little Australia; luxury shopping along Fifth and Madison Avenues; and events such as the Halloween Parade in Greenwich Village; the Brooklyn Bridge (shared with Brooklyn); the Macy's Thanksgiving Day Parade; the lighting of the Rockefeller Center Christmas Tree; the St. Patrick's Day Parade; seasonal activities such as ice skating in Central Park in the wintertime; the Tribeca Film Festival; and free performances in Central Park at SummerStage.Points of interest have also developed in the city outside Manhattan and have made the outer boroughs tourist destinations in their own right.</td>\n",
              "      <td id=\"T_05c61_row2_col1\" class=\"data row2 col1\" >Fail</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2) Faithfulness Evaluator\n",
        "- [ref](https://github.com/run-llama/llama_index/blob/main/docs/examples/evaluation/faithfulness_eval.ipynb)\n",
        "\n",
        "This notebook uses the FaithfulnessEvaluator module to measure if the `response` from a query engine matches any `source` nodes.\n",
        "This is useful for measuring if the response was hallucinated.\n",
        "The data is extracted from the [New York City wikipedia page](https://en.wikipedia.org/wiki/New_York_City).\n",
        "\n",
        "<img src=\"https://miro.medium.com/v2/resize:fit:1100/format:webp/1*6H0F9yAXuIMxoIReDN9eKw.png\" width=\"500\">"
      ],
      "metadata": {
        "id": "mfQzXiNz8_vt"
      },
      "id": "mfQzXiNz8_vt"
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.evaluation import FaithfulnessEvaluator\n",
        "\n",
        "# gpt-4\n",
        "gpt4 = OpenAI(temperature=0, model=\"gpt-4\")\n",
        "service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)\n",
        "\n",
        "evaluator_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)"
      ],
      "metadata": {
        "id": "LOMLZu4z8_T4"
      },
      "id": "LOMLZu4z8_T4",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "documents = SimpleDirectoryReader(\"./test_wiki_data/\").load_data()"
      ],
      "metadata": {
        "id": "wnsUS79P8_Mi"
      },
      "id": "wnsUS79P8_Mi",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create vector index\n",
        "service_context = ServiceContext.from_defaults(chunk_size=512)\n",
        "vector_index = VectorStoreIndex.from_documents(\n",
        "    documents, service_context=service_context\n",
        ")"
      ],
      "metadata": {
        "id": "RCW0zLFr8c84"
      },
      "id": "RCW0zLFr8c84",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# define jupyter display function\n",
        "def display_eval_df(response: Response, eval_result: str) -> None:\n",
        "    if response.source_nodes == []:\n",
        "        print(\"no response!\")\n",
        "        return\n",
        "    eval_df = pd.DataFrame(\n",
        "        {\n",
        "            \"Response\": str(response),\n",
        "            \"Source\": response.source_nodes[0].node.text[:1000] + \"...\",\n",
        "            \"Evaluation Result\": \"Pass\" if eval_result.passing else \"Fail\",\n",
        "        },\n",
        "        index=[0],\n",
        "    )\n",
        "    eval_df = eval_df.style.set_properties(\n",
        "        **{\n",
        "            \"inline-size\": \"600px\",\n",
        "            \"overflow-wrap\": \"break-word\",\n",
        "        },\n",
        "        subset=[\"Response\", \"Source\"]\n",
        "    )\n",
        "    display(eval_df)"
      ],
      "metadata": {
        "id": "rccnmT2j9Pki"
      },
      "id": "rccnmT2j9Pki",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "To run evaluations you can call the .evaluate_response() function on the Response object return from the query to run the evaluations. Lets evaluate the outputs of the vector_index."
      ],
      "metadata": {
        "id": "9LEnQKXY9Szg"
      },
      "id": "9LEnQKXY9Szg"
    },
    {
      "cell_type": "code",
      "source": [
        "query_engine = vector_index.as_query_engine()\n",
        "response_vector = query_engine.query(\"How did New York City get its name?\")\n",
        "eval_result = evaluator_gpt4.evaluate_response(response=response_vector)\n",
        "\n",
        "display_eval_df(response_vector, eval_result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-RoXtLNK8c1c",
        "outputId": "72af6660-5197-4708-bed5-76855757fe0e"
      },
      "id": "-RoXtLNK8c1c",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7b97b22bb910>"
            ],
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_0a81f_row0_col0, #T_0a81f_row0_col1 {\n",
              "  inline-size: 600px;\n",
              "  overflow-wrap: break-word;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_0a81f\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_0a81f_level0_col0\" class=\"col_heading level0 col0\" >Response</th>\n",
              "      <th id=\"T_0a81f_level0_col1\" class=\"col_heading level0 col1\" >Source</th>\n",
              "      <th id=\"T_0a81f_level0_col2\" class=\"col_heading level0 col2\" >Evaluation Result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_0a81f_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_0a81f_row0_col0\" class=\"data row0 col0\" >New York City got its name when it came under British control in 1664. It was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York.</td>\n",
              "      <td id=\"T_0a81f_row0_col1\" class=\"data row0 col1\" >The city came under British control in 1664 and was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city was regained by the Dutch in July 1673 and was renamed New Orange for one year and three months; the city has been continuously named New York since November 1674. New York City was the capital of the United States from 1785 until 1790, and has been the largest U.S. city since 1790. The Statue of Liberty greeted millions of immigrants as they came to the U.S. by ship in the late 19th and early 20th centuries, and is a symbol of the U.S. and its ideals of liberty and peace. In the 21st century, New York City has emerged as a global node of creativity, entrepreneurship, and as a symbol of freedom and cultural diversity. The New York Times has won the most Pulitzer Prizes for journalism and remains the U.S. media's \"newspaper of record\". In 2019, New York City was voted the greatest city in the world in a survey of over 30,000 p...</td>\n",
              "      <td id=\"T_0a81f_row0_col2\" class=\"data row0 col2\" >Pass</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Benchmark on Generated Question\n",
        "Now lets generate a few more questions so that we have more to evaluate with and run a small benchmark.\n",
        "\n",
        "<img src=\"https://miro.medium.com/v2/resize:fit:1400/format:webp/1*JRRi91-cPqV_31ui4cWKvg.png\" width=\"500\">"
      ],
      "metadata": {
        "id": "0YC_0qX59Y2c"
      },
      "id": "0YC_0qX59Y2c"
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.evaluation import DatasetGenerator\n",
        "\n",
        "documents = SimpleDirectoryReader(\"./test_wiki_data/\").load_data()\n",
        "\n",
        "question_generator = DatasetGenerator.from_documents(documents)\n",
        "eval_questions = question_generator.generate_questions_from_nodes(20)\n",
        "\n",
        "eval_questions"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-AnJQ92W9VVT",
        "outputId": "f9512ab3-7eeb-46ed-8d43-3ffbd15958b0"
      },
      "id": "-AnJQ92W9VVT",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:llama_index.service_context:chunk_size_limit is deprecated, please specify chunk_size instead\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['What is the population of New York City as of 2020?',\n",
              " 'Which city is the second-largest in the United States after New York City?',\n",
              " 'How many boroughs make up New York City?',\n",
              " 'What is the largest metropolitan area in the U.S. by both population and urban area?',\n",
              " 'What is the gross metropolitan product of the New York metropolitan area?',\n",
              " 'How many languages are spoken in New York City, making it the most linguistically diverse city in the world?',\n",
              " 'Who founded the trading post on Manhattan Island that would later become New York City?',\n",
              " 'When did New York City become the capital of the United States?',\n",
              " 'What is the most visited tourist attraction in New York City?',\n",
              " 'What is the largest single-operator rapid transit system in the world?',\n",
              " 'What percentage of households in New York enslaved Africans by 1730?',\n",
              " 'When was Columbia University founded in Lower Manhattan?',\n",
              " 'Which battle was the largest battle of the American Revolutionary War and where was it fought?',\n",
              " 'What event led to the establishment of the freedom of the press in North America?',\n",
              " 'When did New York City become the national capital of the U.S.?',\n",
              " 'What major event occurred in 1898 that led to the formation of the City of New York?',\n",
              " 'What was the significance of the Stonewall riots in New York City?',\n",
              " 'How did the Great Depression impact New York City?',\n",
              " 'What major event caused economic damage and loss of life in New York City in 2001?',\n",
              " 'What natural disaster heavily affected New York City in 2012?']"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import asyncio\n",
        "\n",
        "\n",
        "def evaluate_query_engine(query_engine, questions):\n",
        "    c = [query_engine.aquery(q) for q in questions]\n",
        "    results = asyncio.run(asyncio.gather(*c))\n",
        "    print(\"finished query\")\n",
        "\n",
        "    total_correct = 0\n",
        "    for r in results:\n",
        "        # evaluate with gpt 4\n",
        "        eval_result = (\n",
        "            1 if evaluator_gpt4.evaluate_response(response=r).passing else 0\n",
        "        )\n",
        "        total_correct += eval_result\n",
        "\n",
        "    return total_correct, len(results)"
      ],
      "metadata": {
        "id": "6FP_i7Eo9U-J"
      },
      "id": "6FP_i7Eo9U-J",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "vector_query_engine = vector_index.as_query_engine()\n",
        "correct, total = evaluate_query_engine(vector_query_engine, eval_questions)\n",
        "\n",
        "print(f\"score: {correct}/{total}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "talQQV98-klD",
        "outputId": "4ea5c295-c322-402a-ea43-fda23ac3a0e0"
      },
      "id": "talQQV98-klD",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "finished query\n",
            "score: 18/20\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3) Guideline Evaluator\n",
        "- [ref](https://github.com/run-llama/llama_index/blob/main/docs/examples/evaluation/guideline_eval.ipynb)\n",
        "\n",
        "This notebook shows how to use `GuidelineEvaluator` to evaluate a question answer system given user specified guidelines.\n",
        "\n",
        "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
      ],
      "metadata": {
        "id": "MAmMnYit_ICi"
      },
      "id": "MAmMnYit_ICi"
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.evaluation import GuidelineEvaluator\n",
        "from llama_index import ServiceContext\n",
        "from llama_index.llms import OpenAI\n",
        "\n",
        "# Needed for running async functions in Jupyter Notebook\n",
        "import nest_asyncio\n",
        "\n",
        "nest_asyncio.apply()"
      ],
      "metadata": {
        "id": "Co_u1Z3M_Qjx"
      },
      "id": "Co_u1Z3M_Qjx",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "GUIDELINES = [\n",
        "    \"The response should fully answer the query.\",\n",
        "    \"The response should avoid being vague or ambiguous.\",\n",
        "    \"The response should be specific and use statistics or numbers when possible.\"]"
      ],
      "metadata": {
        "id": "BuawoZzd_QaX"
      },
      "id": "BuawoZzd_QaX",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "service_context = ServiceContext.from_defaults(llm=OpenAI(model=\"gpt-4\"))\n",
        "\n",
        "evaluators = [\n",
        "    GuidelineEvaluator(service_context=service_context, guidelines=guideline)\n",
        "    for guideline in GUIDELINES\n",
        "]"
      ],
      "metadata": {
        "id": "IBAZzakb-3QG"
      },
      "id": "IBAZzakb-3QG",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sample_data = {\n",
        "    \"query\": \"Tell me about global warming.\",\n",
        "    \"contexts\": [\n",
        "        (\n",
        "            \"Global warming refers to the long-term increase in Earth's\"\n",
        "            \" average surface temperature due to human activities such as the\"\n",
        "            \" burning of fossil fuels and deforestation.\"\n",
        "        ),\n",
        "        (\n",
        "            \"It is a major environmental issue with consequences such as\"\n",
        "            \" rising sea levels, extreme weather events, and disruptions to\"\n",
        "            \" ecosystems.\"\n",
        "        ),\n",
        "        (\n",
        "            \"Efforts to combat global warming include reducing carbon\"\n",
        "            \" emissions, transitioning to renewable energy sources, and\"\n",
        "            \" promoting sustainable practices.\"\n",
        "        ),\n",
        "    ],\n",
        "    \"response\": (\n",
        "        \"Global warming is a critical environmental issue caused by human\"\n",
        "        \" activities that lead to a rise in Earth's temperature. It has\"\n",
        "        \" various adverse effects on the planet.\"\n",
        "    ),\n",
        "}"
      ],
      "metadata": {
        "id": "dEg8_d6Q_ZH_"
      },
      "id": "dEg8_d6Q_ZH_",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for guideline, evaluator in zip(GUIDELINES, evaluators):\n",
        "    eval_result = evaluator.evaluate(\n",
        "        query=sample_data[\"query\"],\n",
        "        contexts=sample_data[\"contexts\"],\n",
        "        response=sample_data[\"response\"],\n",
        "    )\n",
        "    print(\"=====\")\n",
        "    print(f\"Guideline: {guideline}\")\n",
        "    print(f\"Pass: {eval_result.passing}\")\n",
        "    print(f\"Feedback: {eval_result.feedback}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1tTgL9jU_bsd",
        "outputId": "e098e393-8a05-4f29-b9ac-0bdeb308a64c"
      },
      "id": "1tTgL9jU_bsd",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "=====\n",
            "Guideline: The response should fully answer the query.\n",
            "Pass: False\n",
            "Feedback: The response is partially correct but it does not fully answer the query. It should include more details about the causes, effects, and potential solutions to global warming. It could also mention the scientific consensus on the issue and the role of greenhouse gases in global warming.\n",
            "=====\n",
            "Guideline: The response should avoid being vague or ambiguous.\n",
            "Pass: False\n",
            "Feedback: The response is too vague and does not provide specific details about global warming. It should include more information about the causes, effects, and potential solutions to global warming.\n",
            "=====\n",
            "Guideline: The response should be specific and use statistics or numbers when possible.\n",
            "Pass: False\n",
            "Feedback: The response is too general and lacks specific details. It does not include any statistics or numbers related to global warming, such as the rate at which the Earth's temperature is rising, the amount of greenhouse gases being emitted, or the projected impacts of global warming. Please include more specific information and data to improve the response.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 4) Correctness Evaluator\n",
        "- [ref](https://github.com/run-llama/llama_index/blob/main/docs/examples/evaluation/correctness_eval.ipynb)\n",
        "\n",
        "This notebook uses the `CorrectnessEvaluator` to evaluate the relevance and correctness of a generated answer against a reference answer.\n",
        "\n",
        "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
      ],
      "metadata": {
        "id": "eE784DtIAKH7"
      },
      "id": "eE784DtIAKH7"
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.evaluation import CorrectnessEvaluator\n",
        "from llama_index.llms import OpenAI\n",
        "from llama_index import ServiceContext"
      ],
      "metadata": {
        "id": "J30e9iet_bmV"
      },
      "id": "J30e9iet_bmV",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "service_context = ServiceContext.from_defaults(llm=OpenAI(\"gpt-4\"))\n",
        "evaluator = CorrectnessEvaluator(service_context=service_context)"
      ],
      "metadata": {
        "id": "ZC3ZdJtA_bfa"
      },
      "id": "ZC3ZdJtA_bfa",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"Can you explain the theory of relativity proposed by Albert Einstein in detail?\"\n",
        "\n",
        "reference = \"\"\"\n",
        "Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).\n",
        "\n",
        "General relativity, published in 1915, extended these ideas to include the effects of gravity. According to general relativity, gravity is not a force between masses, as described by Newton's theory of gravity, but rather the result of the warping of space and time by mass and energy. Massive objects, such as planets and stars, cause a curvature in spacetime, and smaller objects follow curved paths in response to this curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet, causing it to create a depression that other objects (representing smaller masses) naturally move towards.\n",
        "\n",
        "In essence, general relativity provided a new understanding of gravity, explaining phenomena like the bending of light by gravity (gravitational lensing) and the precession of the orbit of Mercury. It has been confirmed through numerous experiments and observations and has become a fundamental theory in modern physics.\n",
        "\"\"\"\n",
        "\n",
        "response = \"\"\"\n",
        "Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).\n",
        "\n",
        "However, general relativity, published in 1915, extended these ideas to include the effects of magnetism. According to general relativity, gravity is not a force between masses but rather the result of the warping of space and time by magnetic fields generated by massive objects. Massive objects, such as planets and stars, create magnetic fields that cause a curvature in spacetime, and smaller objects follow curved paths in response to this magnetic curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet with magnets underneath, causing it to create a depression that other objects (representing smaller masses) naturally move towards due to magnetic attraction.\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "ms-9ugSMAVwS"
      },
      "id": "ms-9ugSMAVwS",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "result = evaluator.evaluate(\n",
        "    query=query,\n",
        "    response=response,\n",
        "    reference=reference,\n",
        ")\n",
        "print('score: %.1f'%result.score)\n",
        "print('feedback: %s'%result.feedback)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FezAhXz3AVpN",
        "outputId": "31772e84-2e2e-4527-945c-d46e53c89720"
      },
      "id": "FezAhXz3AVpN",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "score: 2.0\n",
            "feedback: The generated answer is relevant to the user query as it attempts to explain the theory of relativity. However, it contains significant mistakes. General relativity is about the effects of gravity, not magnetism. The warping of space and time is caused by mass and energy, not magnetic fields. The analogy used in the generated answer is also incorrect as it introduces magnets, which are not part of the original concept.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Finish"
      ],
      "metadata": {
        "id": "klwZ-bVSFdo9"
      },
      "id": "klwZ-bVSFdo9"
    },
    {
      "cell_type": "code",
      "source": [
        "print('complete')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3e0JgaTTPEXo",
        "outputId": "3168285c-a1d9-47d3-de04-622351ab7322"
      },
      "id": "3e0JgaTTPEXo",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "complete\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "g8zW6SKvVfWx"
      },
      "id": "g8zW6SKvVfWx",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3"
    },
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "wKqhWtX-8yVl",
        "mfQzXiNz8_vt",
        "MAmMnYit_ICi",
        "eE784DtIAKH7"
      ],
      "machine_shape": "hm",
      "gpuType": "V100"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 5
}