part_5/01_main-code/part_5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bfabadb8-5935-45ff-b39c-db7a29012129",
   "metadata": {
    "id": "bfabadb8-5935-45ff-b39c-db7a29012129"
   },
   "source": [
    "# Part 5: Finetuning for Text Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90",
    "outputId": "9495f150-9d79-4910-d6e7-6c0d9aae4a41"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matplotlib version: 3.9.2\n",
      "numpy version: 1.26.4\n",
      "tiktoken version: 0.7.0\n",
      "torch version: 2.4.0\n",
      "tensorflow version: 2.18.0\n",
      "pandas version: 2.2.3\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\"matplotlib\",\n",
    "        \"numpy\",\n",
    "        \"tiktoken\",\n",
    "        \"torch\",\n",
    "        \"tensorflow\", # For OpenAI's pretrained weights\n",
    "        \"pandas\"      # Dataset loading\n",
    "       ]\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a445828a-ff10-4efa-9f60-a2e2aed4c87d",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/part-overview.webp\" width=820px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "946c3e56-b04b-4b0f-b35f-b485ce5b28df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility to prevent certain cells from being executed twice\n",
    "\n",
    "from IPython.core.magic import register_line_cell_magic\n",
    "\n",
    "executed_cells = set()\n",
    "\n",
    "@register_line_cell_magic\n",
    "def run_once(line, cell):\n",
    "    if line not in executed_cells:\n",
    "        get_ipython().run_cell(cell)\n",
    "        executed_cells.add(line)\n",
    "    else:\n",
    "        print(f\"Cell '{line}' has already been executed.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c",
   "metadata": {
    "id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c"
   },
   "source": [
    "## 5.1 Categories of finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac45579d-d485-47dc-829e-43be7f4db57b",
   "metadata": {},
   "source": [
    "- The most common ways to finetune language models are instruction-finetuning and classification finetuning\n",
    "- Instruction-finetuning, depicted below, is the topic of the next part"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c29ef42-46d9-43d4-8bb4-94974e1665e4",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/instructions.webp\" width=700px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7f60321-95b8-46a9-97bf-1d07fda2c3dd",
   "metadata": {},
   "source": [
    "- Classification finetuning, the topic of this part, is a procedure you may already be familiar with if you have a background in machine learning -- it's similar to training a convolutional network to classify handwritten digits, for example\n",
    "- In classification finetuning, we have a specific number of class labels (for example, \"spam\" and \"not spam\") that the model can output\n",
    "- A classification finetuned model can only predict classes it has seen during training (for example, \"spam\" or \"not spam\"), whereas an instruction-finetuned model can usually perform many tasks\n",
    "- We can think of a classification-finetuned model as a very specialized model; in practice, it is much easier to create a specialized model than a generalist model that performs well on many different tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b37a0c4-0bb1-4061-b1fe-eaa4416d52c3",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/spam-non-spam.webp\" width=700px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf",
   "metadata": {
    "id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf"
   },
   "source": [
    "## 5.2 Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f628975-d2e8-4f7f-ab38-92bb868b7067",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/overview-1.webp\" width=700px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fbd459f-63fa-4d8c-8499-e23103156c7d",
   "metadata": {
    "id": "9fbd459f-63fa-4d8c-8499-e23103156c7d"
   },
   "source": [
    "- We use a dataset consisting of spam and non-spam text messages to finetune the LLM to classify them\n",
    "- First, we download and unzip the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
    "outputId": "424e4423-f623-443c-ab9e-656f9e867559"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sms_spam_collection/SMSSpamCollection.tsv already exists. Skipping download and extraction.\n"
     ]
    }
   ],
   "source": [
    "import urllib.request\n",
    "import zipfile\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
    "zip_path = \"sms_spam_collection.zip\"\n",
    "extracted_path = \"sms_spam_collection\"\n",
    "data_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\n",
    "\n",
    "def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\n",
    "    if data_file_path.exists():\n",
    "        print(f\"{data_file_path} already exists. Skipping download and extraction.\")\n",
    "        return\n",
    "\n",
    "    # Downloading the file\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        with open(zip_path, \"wb\") as out_file:\n",
    "            out_file.write(response.read())\n",
    "\n",
    "    # Unzipping the file\n",
    "    with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n",
    "        zip_ref.extractall(extracted_path)\n",
    "\n",
    "    # Add .tsv file extension\n",
    "    original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\n",
    "    os.rename(original_file_path, data_file_path)\n",
    "    print(f\"File downloaded and saved as {data_file_path}\")\n",
    "\n",
    "download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1",
   "metadata": {
    "id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1"
   },
   "source": [
    "- The dataset is saved as a tab-separated text file, which we can load into a pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
    "outputId": "a16c5cde-d341-4887-a93f-baa9bec542ab"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>spam</td>\n",
       "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5568</th>\n",
       "      <td>ham</td>\n",
       "      <td>Will ü b going to esplanade fr home?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5569</th>\n",
       "      <td>ham</td>\n",
       "      <td>Pity, * was in mood for that. So...any other s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5570</th>\n",
       "      <td>ham</td>\n",
       "      <td>The guy did some bitching but I acted like i'd...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5571</th>\n",
       "      <td>ham</td>\n",
       "      <td>Rofl. Its true to its name</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5572 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Label                                               Text\n",
       "0      ham  Go until jurong point, crazy.. Available only ...\n",
       "1      ham                      Ok lar... Joking wif u oni...\n",
       "2     spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3      ham  U dun say so early hor... U c already then say...\n",
       "4      ham  Nah I don't think he goes to usf, he lives aro...\n",
       "...    ...                                                ...\n",
       "5567  spam  This is the 2nd time we have tried 2 contact u...\n",
       "5568   ham               Will ü b going to esplanade fr home?\n",
       "5569   ham  Pity, * was in mood for that. So...any other s...\n",
       "5570   ham  The guy did some bitching but I acted like i'd...\n",
       "5571   ham                         Rofl. Its true to its name\n",
       "\n",
       "[5572 rows x 2 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(data_file_path, sep=\"\\t\", header=None, names=[\"Label\", \"Text\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7b6e631-4f0b-4aab-82b9-8898e6663109",
   "metadata": {
    "id": "e7b6e631-4f0b-4aab-82b9-8898e6663109"
   },
   "source": [
    "- When we check the class distribution, we see that the data contains \"ham\" (i.e., \"not spam\") much more frequently than \"spam\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "495a5280-9d7c-41d4-9719-64ab99056d4c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "495a5280-9d7c-41d4-9719-64ab99056d4c",
    "outputId": "761e0482-43ba-4f46-f4b7-6774dae51b38"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label\n",
      "ham     4825\n",
      "spam     747\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df[\"Label\"].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f773f054-0bdc-4aad-bbf6-397621bf63db",
   "metadata": {
    "id": "f773f054-0bdc-4aad-bbf6-397621bf63db"
   },
   "source": [
    "- For simplicity, and because we prefer a small dataset for educational purposes anyway (it will make it possible to finetune the LLM faster), we subsample (undersample) the dataset so that it contains 747 instances from each class\n",
    "- (Next to undersampling, there are several other ways to deal with class balances, but they are out of the scope of a book on LLMs; you can find examples and more information in the [`imbalanced-learn` user guide](https://imbalanced-learn.org/stable/user_guide.html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7be4a0a2-9704-4a96-b38f-240339818688",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7be4a0a2-9704-4a96-b38f-240339818688",
    "outputId": "396dc415-cb71-4a88-e85d-d88201c6d73f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label\n",
      "ham     747\n",
      "spam    747\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "%%run_once balance_df\n",
    "\n",
    "\n",
    "def create_balanced_dataset(df):\n",
    "    \n",
    "    # Count the instances of \"spam\"\n",
    "    num_spam = df[df[\"Label\"] == \"spam\"].shape[0]\n",
    "    \n",
    "    # Randomly sample \"ham\" instances to match the number of \"spam\" instances\n",
    "    ham_subset = df[df[\"Label\"] == \"ham\"].sample(num_spam, random_state=123)\n",
    "    \n",
    "    # Combine ham \"subset\" with \"spam\"\n",
    "    balanced_df = pd.concat([ham_subset, df[df[\"Label\"] == \"spam\"]])\n",
    "\n",
    "    return balanced_df\n",
    "\n",
    "\n",
    "balanced_df = create_balanced_dataset(df)\n",
    "print(balanced_df[\"Label\"].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3fd2f5a-06d8-4d30-a2e3-230b86c559d6",
   "metadata": {
    "id": "d3fd2f5a-06d8-4d30-a2e3-230b86c559d6"
   },
   "source": [
    "- Next, we change the string class labels \"ham\" and \"spam\" into integer class labels 0 and 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c1b10c3d-5d57-42d0-8de8-cf80a06f5ffd",
   "metadata": {
    "id": "c1b10c3d-5d57-42d0-8de8-cf80a06f5ffd"
   },
   "outputs": [],
   "source": [
    "%%run_once label_mapping\n",
    "balanced_df[\"Label\"] = balanced_df[\"Label\"].map({\"ham\": 0, \"spam\": 1})    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e6f7f062-ef4e-4020-8275-71990cab4414",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4307</th>\n",
       "      <td>0</td>\n",
       "      <td>Awww dat is sweet! We can think of something t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4138</th>\n",
       "      <td>0</td>\n",
       "      <td>Just got to  &amp;lt;#&amp;gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4831</th>\n",
       "      <td>0</td>\n",
       "      <td>The word \"Checkmate\" in chess comes from the P...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4461</th>\n",
       "      <td>0</td>\n",
       "      <td>This is wishing you a great day. Moji told me ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5440</th>\n",
       "      <td>0</td>\n",
       "      <td>Thank you. do you generally date the brothas?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5537</th>\n",
       "      <td>1</td>\n",
       "      <td>Want explicit SEX in 30 secs? Ring 02073162414...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5540</th>\n",
       "      <td>1</td>\n",
       "      <td>ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5547</th>\n",
       "      <td>1</td>\n",
       "      <td>Had your contract mobile 11 Mnths? Latest Moto...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5566</th>\n",
       "      <td>1</td>\n",
       "      <td>REMINDER FROM O2: To get 2.50 pounds free call...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>1</td>\n",
       "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1494 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Label                                               Text\n",
       "4307      0  Awww dat is sweet! We can think of something t...\n",
       "4138      0                             Just got to  &lt;#&gt;\n",
       "4831      0  The word \"Checkmate\" in chess comes from the P...\n",
       "4461      0  This is wishing you a great day. Moji told me ...\n",
       "5440      0      Thank you. do you generally date the brothas?\n",
       "...     ...                                                ...\n",
       "5537      1  Want explicit SEX in 30 secs? Ring 02073162414...\n",
       "5540      1  ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...\n",
       "5547      1  Had your contract mobile 11 Mnths? Latest Moto...\n",
       "5566      1  REMINDER FROM O2: To get 2.50 pounds free call...\n",
       "5567      1  This is the 2nd time we have tried 2 contact u...\n",
       "\n",
       "[1494 rows x 2 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "balanced_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5715e685-35b4-4b45-a86c-8a8694de9d6f",
   "metadata": {
    "id": "5715e685-35b4-4b45-a86c-8a8694de9d6f"
   },
   "source": [
    "- Let's now define a function that randomly divides the dataset into training, validation, and test subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "uQl0Psdmx15D",
   "metadata": {
    "id": "uQl0Psdmx15D"
   },
   "outputs": [],
   "source": [
    "def random_split(df, train_frac, validation_frac):\n",
    "    # Shuffle the entire DataFrame\n",
    "    df = df.sample(frac=1, random_state=123).reset_index(drop=True)\n",
    "\n",
    "    # Calculate split indices\n",
    "    train_end = int(len(df) * train_frac)\n",
    "    validation_end = train_end + int(len(df) * validation_frac)\n",
    "\n",
    "    # Split the DataFrame\n",
    "    train_df = df[:train_end]\n",
    "    validation_df = df[train_end:validation_end]\n",
    "    test_df = df[validation_end:]\n",
    "\n",
    "    return train_df, validation_df, test_df\n",
    "\n",
    "train_df, validation_df, test_df = random_split(balanced_df, 0.7, 0.1)\n",
    "# Test size is implied to be 0.2 as the remainder\n",
    "\n",
    "train_df.to_csv(\"train.csv\", index=None)\n",
    "validation_df.to_csv(\"validation.csv\", index=None)\n",
    "test_df.to_csv(\"test.csv\", index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8d7a0c5-1d5f-458a-b685-3f49520b0094",
   "metadata": {},
   "source": [
    "## 5.3 Creating data loaders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7126108a-75e7-4862-b0fb-cbf59a18bb6c",
   "metadata": {
    "id": "7126108a-75e7-4862-b0fb-cbf59a18bb6c"
   },
   "source": [
    "- Note that the text messages have different lengths; if we want to combine multiple training examples in a batch, we have to either\n",
    "  1. truncate all messages to the length of the shortest message in the dataset or batch\n",
    "  2. pad all messages to the length of the longest message in the dataset or batch\n",
    "\n",
    "- We choose option 2 and pad all messages to the longest message in the dataset\n",
    "- For that, we use `<|endoftext|>` as a padding token, as discussed in part 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0829f33f-1428-4f22-9886-7fee633b3666",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/pad-input-sequences.webp?123\" width=700px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
    "outputId": "b5b48439-32c8-4b37-cca2-c9dc8fa86563"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50256]\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "print(tokenizer.encode(\"<|endoftext|>\", allowed_special={\"<|endoftext|>\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f582ff-68bf-450e-bd87-5fb61afe431c",
   "metadata": {
    "id": "04f582ff-68bf-450e-bd87-5fb61afe431c"
   },
   "source": [
    "- The `SpamDataset` class below identifies the longest sequence in the training dataset and adds the padding token to the others to match that sequence length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d7791b52-af18-4ac4-afa9-b921068e383e",
   "metadata": {
    "id": "d7791b52-af18-4ac4-afa9-b921068e383e"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "\n",
    "class SpamDataset(Dataset):\n",
    "    def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):\n",
    "        self.data = pd.read_csv(csv_file)\n",
    "\n",
    "        # Pre-tokenize texts\n",
    "        self.encoded_texts = [\n",
    "            tokenizer.encode(text) for text in self.data[\"Text\"]\n",
    "        ]\n",
    "\n",
    "        if max_length is None:\n",
    "            self.max_length = self._longest_encoded_length()\n",
    "        else:\n",
    "            self.max_length = max_length\n",
    "            # Truncate sequences if they are longer than max_length\n",
    "            self.encoded_texts = [\n",
    "                encoded_text[:self.max_length]\n",
    "                for encoded_text in self.encoded_texts\n",
    "            ]\n",
    "\n",
    "        # Pad sequences to the longest sequence\n",
    "        self.encoded_texts = [\n",
    "            encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))\n",
    "            for encoded_text in self.encoded_texts\n",
    "        ]\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        encoded = self.encoded_texts[index]\n",
    "        label = self.data.iloc[index][\"Label\"]\n",
    "        return (\n",
    "            torch.tensor(encoded, dtype=torch.long),\n",
    "            torch.tensor(label, dtype=torch.long)\n",
    "        )\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def _longest_encoded_length(self):\n",
    "        max_length = 0\n",
    "        for encoded_text in self.encoded_texts:\n",
    "            encoded_length = len(encoded_text)\n",
    "            if encoded_length > max_length:\n",
    "                max_length = encoded_length\n",
    "        return max_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "uzj85f8ou82h",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uzj85f8ou82h",
    "outputId": "d08f1cf0-c24d-445f-a3f8-793532c3716f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "120\n"
     ]
    }
   ],
   "source": [
    "train_dataset = SpamDataset(\n",
    "    csv_file=\"train.csv\",\n",
    "    max_length=None,\n",
    "    tokenizer=tokenizer\n",
    ")\n",
    "\n",
    "print(train_dataset.max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15bdd932-97eb-4b88-9cf9-d766ea4c3a60",
   "metadata": {},
   "source": [
    "- We also pad the validation and test set to the longest training sequence\n",
    "- Note that validation and test set samples that are longer than the longest training example are being truncated via `encoded_text[:self.max_length]` in the `SpamDataset` code\n",
    "- This behavior is entirely optional, and it would also work well if we set `max_length=None` in both the validation and test set cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e",
   "metadata": {
    "id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e"
   },
   "outputs": [],
   "source": [
    "val_dataset = SpamDataset(\n",
    "    csv_file=\"validation.csv\",\n",
    "    max_length=train_dataset.max_length,\n",
    "    tokenizer=tokenizer\n",
    ")\n",
    "test_dataset = SpamDataset(\n",
    "    csv_file=\"test.csv\",\n",
    "    max_length=train_dataset.max_length,\n",
    "    tokenizer=tokenizer\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20170d89-85a0-4844-9887-832f5d23432a",
   "metadata": {},
   "source": [
    "- Next, we use the dataset to instantiate the data loaders, which is similar to creating the data loaders in previous parts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64bcc349-205f-48f8-9655-95ff21f5e72f",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/batch.webp\" width=700px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
    "outputId": "3266c410-4fdb-4a8c-a142-7f707e2525ab"
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "num_workers = 0\n",
    "batch_size = 8\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = DataLoader(\n",
    "    dataset=train_dataset,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=True,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=True,\n",
    ")\n",
    "\n",
    "val_loader = DataLoader(\n",
    "    dataset=val_dataset,\n",
    "    batch_size=batch_size,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=False,\n",
    ")\n",
    "\n",
    "test_loader = DataLoader(\n",
    "    dataset=test_dataset,\n",
    "    batch_size=batch_size,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7335db-e0bb-4e27-80c5-eea11e593a57",
   "metadata": {},
   "source": [
    "- As a verification step, we iterate through the data loaders and ensure that the batches contain 8 training examples each, where each training example consists of 120 tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4dee6882-4c3a-4964-af15-fa31f86ad047",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loader:\n",
      "Input batch dimensions: torch.Size([8, 120])\n",
      "Label batch dimensions torch.Size([8])\n"
     ]
    }
   ],
   "source": [
    "print(\"Train loader:\")\n",
    "for input_batch, target_batch in train_loader:\n",
    "    pass\n",
    "\n",
    "print(\"Input batch dimensions:\", input_batch.shape)\n",
    "print(\"Label batch dimensions\", target_batch.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cdd7947-7039-49bf-8a5e-c0a2f4281ca1",
   "metadata": {},
   "source": [
    "- Lastly, let's print the total number of batches in each dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "IZfw-TYD2zTj",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IZfw-TYD2zTj",
    "outputId": "6934bbf2-9797-4fbe-d26b-1a246e18c2fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "130 training batches\n",
      "19 validation batches\n",
      "38 test batches\n"
     ]
    }
   ],
   "source": [
    "print(f\"{len(train_loader)} training batches\")\n",
    "print(f\"{len(val_loader)} validation batches\")\n",
    "print(f\"{len(test_loader)} test batches\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c",
   "metadata": {
    "id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c"
   },
   "source": [
    "## 5.4 Initializing a model with pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e1af8b-8bd1-4b44-8b8b-dc031496e208",
   "metadata": {},
   "source": [
    "- In this section, we initialize the pretrained model we worked with in the previous part\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/overview-2.webp\" width=700px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2992d779-f9fb-4812-a117-553eb790a5a9",
   "metadata": {
    "id": "2992d779-f9fb-4812-a117-553eb790a5a9"
   },
   "outputs": [],
   "source": [
    "CHOOSE_MODEL = \"gpt2-medium (355M)\"\n",
    "INPUT_PROMPT = \"Every effort moves\"\n",
    "\n",
    "BASE_CONFIG = {\n",
    "    \"vocab_size\": 50257,     # Vocabulary size\n",
    "    \"context_length\": 1024,  # Context length\n",
    "    \"drop_rate\": 0.0,        # Dropout rate\n",
    "    \"qkv_bias\": True         # Query-key-value bias\n",
    "}\n",
    "\n",
    "model_configs = {\n",
    "    \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
    "    \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
    "    \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
    "    \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
    "}\n",
    "\n",
    "BASE_CONFIG.update(model_configs[CHOOSE_MODEL])\n",
    "\n",
    "assert train_dataset.max_length <= BASE_CONFIG[\"context_length\"], (\n",
    "    f\"Dataset length {train_dataset.max_length} exceeds model's context \"\n",
    "    f\"length {BASE_CONFIG['context_length']}. Reinitialize data sets with \"\n",
    "    f\"`max_length={BASE_CONFIG['context_length']}`\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "022a649a-44f5-466c-8a8e-326c063384f5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "022a649a-44f5-466c-8a8e-326c063384f5",
    "outputId": "7091e401-8442-4f47-a1d9-ecb42a1ef930"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File already exists and is up-to-date: ../../gpt2/355M/checkpoint\n",
      "File already exists and is up-to-date: ../../gpt2/355M/encoder.json\n",
      "File already exists and is up-to-date: ../../gpt2/355M/hparams.json\n",
      "File already exists and is up-to-date: ../../gpt2/355M/model.ckpt.data-00000-of-00001\n",
      "File already exists and is up-to-date: ../../gpt2/355M/model.ckpt.index\n",
      "File already exists and is up-to-date: ../../gpt2/355M/model.ckpt.meta\n",
      "File already exists and is up-to-date: ../../gpt2/355M/vocab.bpe\n"
     ]
    }
   ],
   "source": [
    "from gpt_download import download_and_load_gpt2\n",
    "from previous_parts import GPTModel, load_weights_into_gpt\n",
    "\n",
    "model_size = CHOOSE_MODEL.split(\" \")[-1].lstrip(\"(\").rstrip(\")\")\n",
    "settings, params = download_and_load_gpt2(model_size=model_size, models_dir=\"../../gpt2\")\n",
    "\n",
    "model = GPTModel(BASE_CONFIG)\n",
    "load_weights_into_gpt(model, params)\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8e056c-abe0-415f-b34d-df686204259e",
   "metadata": {},
   "source": [
    "- To ensure that the model was loaded correctly, let's double-check that it generates coherent text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d8ac25ff-74b1-4149-8dc5-4c429d464330",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Every effort moves you forward, but you must be careful. You must not let your guard down\n"
     ]
    }
   ],
   "source": [
    "from previous_parts import (\n",
    "    generate_text_simple,\n",
    "    text_to_token_ids,\n",
    "    token_ids_to_text\n",
    ")\n",
    "\n",
    "\n",
    "text_1 = \"Every effort moves you\"\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(text_1, tokenizer),\n",
    "    max_new_tokens=15,\n",
    "    context_size=BASE_CONFIG[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69162550-6a02-4ece-8db1-06c71d61946f",
   "metadata": {},
   "source": [
    "- Before we finetune the model as a classifier, let's see if the model can perhaps already classify spam messages via prompting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "94224aa9-c95a-4f8a-a420-76d01e3a800c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is the following text 'spam'? Answer with 'yes' or 'no': 'You are a winner you have been specially selected to receive $1000 cash or a $2000 award.'\n",
      "\n",
      "'You are a winner you have been specially selected to receive $1000 cash or a $2000 award.'\n"
     ]
    }
   ],
   "source": [
    "text_2 = (\n",
    "    \"Is the following text 'spam'? Answer with 'yes' or 'no':\"\n",
    "    \" 'You are a winner you have been specially\"\n",
    "    \" selected to receive $1000 cash or a $2000 award.'\"\n",
    ")\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(text_2, tokenizer),\n",
    "    max_new_tokens=23,\n",
    "    context_size=BASE_CONFIG[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ce39ed0-2c77-410d-8392-dd15d4b22016",
   "metadata": {},
   "source": [
    "- As we can see, the model is not very good at following instructions\n",
    "- This is expected, since it has only been pretrained and not instruction-finetuned (instruction finetuning will be covered in the next part)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522",
   "metadata": {
    "id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522"
   },
   "source": [
    "## 5.5 Adding a classification head"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6e9d66f-76b2-40fc-9ec5-3f972a8db9c0",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/lm-head.webp\" width=700px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "217bac05-78df-4412-bd80-612f8061c01d",
   "metadata": {},
   "source": [
    "- In this section, we are modifying the pretrained LLM to make it ready for classification finetuning\n",
    "- Let's take a look at the model architecture first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b23aff91-6bd0-48da-88f6-353657e6c981",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1d8f7a01-b7c0-48d4-b1e7-8c12cc7ad932",
    "outputId": "b6a5b9b5-a92f-498f-d7cb-b58dd99e4497"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPTModel(\n",
      "  (tok_emb): Embedding(50257, 1024)\n",
      "  (pos_emb): Embedding(1024, 1024)\n",
      "  (drop_emb): Dropout(p=0.0, inplace=False)\n",
      "  (trf_blocks): Sequential(\n",
      "    (0): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (1): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (2): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (3): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (4): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (5): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (6): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (7): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (8): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (9): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (10): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (11): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (12): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (13): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (14): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (15): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (16): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (17): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (18): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (19): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (20): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (21): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (22): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (23): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_key): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (W_value): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "  )\n",
      "  (final_norm): LayerNorm()\n",
      "  (out_head): Linear(in_features=1024, out_features=50257, bias=False)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f640a76-dd00-4769-9bc8-1aed0cec330d",
   "metadata": {},
   "source": [
    "- Above, we can see the architecture we implemented in part 4 neatly laid out\n",
    "- The goal is to replace and finetune the output layer\n",
    "- To achieve this, we first freeze the model, meaning that we make all layers non-trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fkMWFl-0etea",
   "metadata": {
    "id": "fkMWFl-0etea"
   },
   "outputs": [],
   "source": [
    "for param in model.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72155f83-87d9-476a-a978-a15aa2d44147",
   "metadata": {},
   "source": [
    "- Then, we replace the output layer (`model.out_head`), which originally maps the layer inputs to 50,257 dimensions (the size of the vocabulary)\n",
    "- Since we finetune the model for binary classification (predicting 2 classes, \"spam\" and \"not spam\"), we can replace the output layer as shown below, which will be trainable by default\n",
    "- Note that we use `BASE_CONFIG[\"emb_dim\"]` (which is equal to 768 in the `\"gpt2-small (124M)\"` model) to keep the code below more general"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7e759fa0-0f69-41be-b576-17e5f20e04cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "num_classes = 2\n",
    "model.out_head = torch.nn.Linear(in_features=BASE_CONFIG[\"emb_dim\"], out_features=num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30be5475-ae77-4f97-8f3e-dec462b1339f",
   "metadata": {},
   "source": [
    "- Technically, it's sufficient to only train the output layer\n",
    "- However, as I found in [Finetuning Large Language Models](https://magazine.sebastianraschka.com/p/finetuning-large-language-models), experiments show that finetuning additional layers can noticeably improve the performance\n",
    "- So, we are also making the last transformer block and the final `LayerNorm` module connecting the last transformer block to the output layer trainable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0be7c1eb-c46c-4065-8525-eea1b8c66d10",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/trainable.webp\" width=600px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7",
   "metadata": {
    "id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7"
   },
   "outputs": [],
   "source": [
    "for param in model.trf_blocks[-1].parameters():\n",
    "    param.requires_grad = True\n",
    "\n",
    "for param in model.final_norm.parameters():\n",
    "    param.requires_grad = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f012b899-8284-4d3a-97c0-8a48eb33ba2e",
   "metadata": {},
   "source": [
    "- We can still use this model similar to before in previous parts\n",
    "- For example, let's feed it some text input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
    "outputId": "27e041b1-d731-48a1-cf60-f22d4565304e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs: tensor([[5211,  345,  423,  640]])\n",
      "Inputs dimensions: torch.Size([1, 4])\n"
     ]
    }
   ],
   "source": [
    "inputs = tokenizer.encode(\"Do you have time\")\n",
    "inputs = torch.tensor(inputs).unsqueeze(0)\n",
    "print(\"Inputs:\", inputs)\n",
    "print(\"Inputs dimensions:\", inputs.shape) # shape: (batch_size, num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbf8481-772d-467b-851c-a62b86d0cb1b",
   "metadata": {},
   "source": [
    "- What's different compared to previous parts is that it now has two output dimensions instead of 50,257"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
    "outputId": "9cae7448-253d-4776-973e-0af190b06354"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outputs:\n",
      " tensor([[[-3.7743,  0.5238],\n",
      "         [-7.7890,  1.5769],\n",
      "         [-4.9057,  1.3229],\n",
      "         [-3.6375,  1.0749]]])\n",
      "Outputs dimensions: torch.Size([1, 4, 2])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    outputs = model(inputs)\n",
    "\n",
    "print(\"Outputs:\\n\", outputs)\n",
    "print(\"Outputs dimensions:\", outputs.shape) # shape: (batch_size, num_tokens, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75430a01-ef9c-426a-aca0-664689c4f461",
   "metadata": {},
   "source": [
    "- As discussed in previous parts, for each input token, there's one output vector\n",
    "- Since we fed the model a text sample with 4 input tokens, the output consists of 4 2-dimensional output vectors above"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df9144f-6817-4be4-8d4b-5d4dadfe4a9b",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/input-and-output.webp\" width=600px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3bb8616-c791-4f5c-bac0-5302f663e46a",
   "metadata": {},
   "source": [
    "- In part 3, we discussed the attention mechanism, which connects each input token to each other input token\n",
    "- In part 3, we then also introduced the causal attention mask that is used in GPT-like models; this causal mask lets a current token only attend to the current and previous token positions\n",
    "- Based on this causal attention mechanism, the 4th (last) token contains the most information among all tokens because it's the only token that includes information about all other tokens\n",
    "- Hence, we are particularly interested in this last token, which we will finetune for the spam classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
    "outputId": "e79eb155-fa1f-46ed-ff8c-d828c3a3fabd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last output token: tensor([[-3.6375,  1.0749]])\n"
     ]
    }
   ],
   "source": [
    "print(\"Last output token:\", outputs[:, -1, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8df08ae0-e664-4670-b7c5-8a2280d9b41b",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/attention-mask.webp\" width=400px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32aa4aef-e1e9-491b-9adf-5aa973e59b8c",
   "metadata": {},
   "source": [
    "## 5.6 Calculating the classification loss and accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "669e1fd1-ace8-44b4-b438-185ed0ba8b33",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/overview-3.webp?1\" width=600px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a7df4ee-0a34-4a4d-896d-affbbf81e0b3",
   "metadata": {},
   "source": [
    "- Before explaining the loss calculation, let's have a brief look at how the model outputs are turned into class labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "557996dd-4c6b-49c4-ab83-f60ef7e1d69e",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/class-argmax.webp\" width=800px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c77faab1-3461-4118-866a-6171f2b89aa0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last output token: tensor([[-3.6375,  1.0749]])\n"
     ]
    }
   ],
   "source": [
    "print(\"Last output token:\", outputs[:, -1, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edd71fa-628a-4d00-b81d-6d8bcb2c341d",
   "metadata": {},
   "source": [
    "- Similar to part 5, we convert the outputs (logits) into probability scores via the `softmax` function and then obtain the index position of the largest probability value via the `argmax` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "b81efa92-9be1-4b9e-8790-ce1fc7b17f01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class label: 1\n"
     ]
    }
   ],
   "source": [
    "probas = torch.softmax(outputs[:, -1, :], dim=-1)\n",
    "label = torch.argmax(probas)\n",
    "print(\"Class label:\", label.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "414a6f02-307e-4147-a416-14d115bf8179",
   "metadata": {},
   "source": [
    "- Note that the softmax function is optional here, as explained in part 5, because the largest outputs correspond to the largest probability scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "f9f9ad66-4969-4501-8239-3ccdb37e71a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class label: 1\n"
     ]
    }
   ],
   "source": [
    "logits = outputs[:, -1, :]\n",
    "label = torch.argmax(logits)\n",
    "print(\"Class label:\", label.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb20d3a-cbba-4ab1-8584-d94e16589505",
   "metadata": {},
   "source": [
    "- We can apply this concept to calculate the so-called classification accuracy, which computes the percentage of correct predictions in a given dataset\n",
    "- To calculate the classification accuracy, we can apply the preceding `argmax`-based prediction code to all examples in a dataset and calculate the fraction of correct predictions as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3ecf9572-aed0-4a21-9c3b-7f9f2aec5f23",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_accuracy_loader(data_loader, model, device, num_batches=None):\n",
    "    model.eval()\n",
    "    correct_predictions, num_examples = 0, 0\n",
    "\n",
    "    if num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "\n",
    "            with torch.no_grad():\n",
    "                logits = model(input_batch)[:, -1, :]  # Logits of last output token\n",
    "            predicted_labels = torch.argmax(logits, dim=-1)\n",
    "\n",
    "            num_examples += predicted_labels.shape[0]\n",
    "            correct_predictions += (predicted_labels == target_batch).sum().item()\n",
    "        else:\n",
    "            break\n",
    "    return correct_predictions / num_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7165fe46-a284-410b-957f-7524877d1a1a",
   "metadata": {},
   "source": [
    "- Let's apply the function to calculate the classification accuracies for the different datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "390e5255-8427-488c-adef-e1c10ab4fb26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on mps device.\n",
      "Training accuracy: 46.25%\n",
      "Validation accuracy: 45.00%\n",
      "Test accuracy: 48.75%\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Note:\n",
    "# Uncommenting the following lines will allow the code to run on Apple Silicon chips, if applicable,\n",
    "# which is approximately 2x faster than on an Apple CPU (as measured on an M3 MacBook Air).\n",
    "# As of this writing, in PyTorch 2.4, the results obtained via CPU and MPS were identical.\n",
    "# However, in earlier versions of PyTorch, you may observe different results when using MPS.\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "   device = torch.device(\"cuda\")\n",
    "elif torch.backends.mps.is_available():\n",
    "   device = torch.device(\"mps\")\n",
    "else:\n",
    "   device = torch.device(\"cpu\")\n",
    "print(f\"Running on {device} device.\")\n",
    "\n",
    "model.to(device) # no assignment model = model.to(device) necessary for nn.Module classes\n",
    "\n",
    "torch.manual_seed(123) # For reproducibility due to the shuffling in the training data loader\n",
    "\n",
    "train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=10)\n",
    "val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=10)\n",
    "test_accuracy = calc_accuracy_loader(test_loader, model, device, num_batches=10)\n",
    "\n",
    "print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
    "print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30345e2a-afed-4d22-9486-f4010f90a871",
   "metadata": {},
   "source": [
    "- As we can see, the prediction accuracies are not very good, since we haven't finetuned the model, yet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f4a9d15-8fc7-48a2-8734-d92a2f265328",
   "metadata": {},
   "source": [
    "- Before we can start finetuning (/training), we first have to define the loss function we want to optimize during training\n",
    "- The goal is to maximize the spam classification accuracy of the model; however, classification accuracy is not a differentiable function\n",
    "- Hence, instead, we minimize the cross-entropy loss as a proxy for maximizing the classification accuracy (you can learn more about this topic in lecture 8 of my freely available [Introduction to Deep Learning](https://sebastianraschka.com/blog/2021/dl-course.html#l08-multinomial-logistic-regression--softmax-regression) class)\n",
    "\n",
    "- The `calc_loss_batch` function is the same here as in part 5, except that we are only interested in optimizing the last token `model(input_batch)[:, -1, :]` instead of all tokens `model(input_batch)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2f1e9547-806c-41a9-8aba-3b2822baabe4",
   "metadata": {
    "id": "2f1e9547-806c-41a9-8aba-3b2822baabe4"
   },
   "outputs": [],
   "source": [
    "def calc_loss_batch(input_batch, target_batch, model, device):\n",
    "    input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "    logits = model(input_batch)[:, -1, :]  # Logits of last output token\n",
    "    loss = torch.nn.functional.cross_entropy(logits, target_batch)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a013aab9-f854-4866-ad55-5b8350adb50a",
   "metadata": {},
   "source": [
    "The `calc_loss_loader` is exactly the same as in part 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b7b83e10-5720-45e7-ac5e-369417ca846b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as in part 4\n",
    "def calc_loss_loader(data_loader, model, device, num_batches=None):\n",
    "    total_loss = 0.\n",
    "    if len(data_loader) == 0:\n",
    "        return float(\"nan\")\n",
    "    elif num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        # Reduce the number of batches to match the total number of batches in the data loader\n",
    "        # if num_batches exceeds the number of batches in the data loader\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            total_loss += loss.item()\n",
    "        else:\n",
    "            break\n",
    "    return total_loss / num_batches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56826ecd-6e74-40e6-b772-d3541e585067",
   "metadata": {},
   "source": [
    "- Using the `calc_closs_loader`, we compute the initial training, validation, and test set losses before we start training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
    "outputId": "49df8648-9e38-4314-854d-9faacd1b2e89"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.894\n",
      "Validation loss: 0.919\n",
      "Test loss: 0.857\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet\n",
    "    train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)\n",
    "    val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)\n",
    "    test_loss = calc_loss_loader(test_loader, model, device, num_batches=5)\n",
    "\n",
    "print(f\"Training loss: {train_loss:.3f}\")\n",
    "print(f\"Validation loss: {val_loss:.3f}\")\n",
    "print(f\"Test loss: {test_loss:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e04b980b-e583-4f62-84a0-4edafaf99d5d",
   "metadata": {},
   "source": [
    "- In the next section, we train the model to improve the loss values and consequently the classification accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "456ae0fd-6261-42b4-ab6a-d24289953083",
   "metadata": {
    "id": "456ae0fd-6261-42b4-ab6a-d24289953083"
   },
   "source": [
    "## 5.7 Finetuning the model on supervised data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a9b099b-0829-4f72-8a2b-4363e3497026",
   "metadata": {},
   "source": [
    "- In this section, we define and use the training function to improve the classification accuracy of the model\n",
    "- The `train_classifier_simple` function below is practically the same as the `train_model_simple` function we used for pretraining the model in part 5\n",
    "- The only two differences are that we now \n",
    "  1. track the number of training examples seen (`examples_seen`) instead of the number of tokens seen\n",
    "  2. calculate the accuracy after each epoch instead of printing a sample text after each epoch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979b6222-1dc2-4530-9d01-b6b04fe3de12",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/training-loop.webp?1\" width=600px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "Csbr60to50FL",
   "metadata": {
    "id": "Csbr60to50FL"
   },
   "outputs": [],
   "source": [
    "# Overall the same as `train_model_simple` in part 4\n",
    "def train_classifier_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n",
    "                            eval_freq, eval_iter):\n",
    "    # Initialize lists to track losses and examples seen\n",
    "    train_losses, val_losses, train_accs, val_accs = [], [], [], []\n",
    "    examples_seen, global_step = 0, -1\n",
    "\n",
    "    # Main training loop\n",
    "    for epoch in range(num_epochs):\n",
    "        model.train()  # Set model to training mode\n",
    "\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward() # Calculate loss gradients\n",
    "            optimizer.step() # Update model weights using loss gradients\n",
    "            examples_seen += input_batch.shape[0] # New: track examples instead of tokens\n",
    "            global_step += 1\n",
    "\n",
    "            # Optional evaluation step\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader, device, eval_iter)\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
    "\n",
    "        # Calculate accuracy after each epoch\n",
    "        train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "        print(f\"Training accuracy: {train_accuracy*100:.2f}% | \", end=\"\")\n",
    "        print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "        train_accs.append(train_accuracy)\n",
    "        val_accs.append(val_accuracy)\n",
    "\n",
    "    return train_losses, val_losses, train_accs, val_accs, examples_seen"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9624cb30-3e3a-45be-b006-c00475b58ae8",
   "metadata": {},
   "source": [
    "- The `evaluate_model` function used in the `train_classifier_simple` is the same as the one we used in part 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "bcc7bc04-6aa6-4516-a147-460e2f466eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as part 4\n",
    "def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "    model.train()\n",
    "    return train_loss, val_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e807bfe9-364d-46b2-9e25-3b000c3ef6f9",
   "metadata": {},
   "source": [
    "- The training takes about 3 minutes on a M2 MacBook Air laptop computer and less than half a minute on a V100 or A100 GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "X7kU3aAj7vTJ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "X7kU3aAj7vTJ",
    "outputId": "504a033e-2bf8-41b5-a037-468309845513"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Step 000000): Train loss 0.799, Val loss 0.853\n",
      "Ep 1 (Step 000050): Train loss 0.538, Val loss 0.552\n",
      "Ep 1 (Step 000100): Train loss 0.368, Val loss 0.434\n",
      "Training accuracy: 82.50% | Validation accuracy: 80.00%\n",
      "Ep 2 (Step 000150): Train loss 0.551, Val loss 0.391\n",
      "Ep 2 (Step 000200): Train loss 0.376, Val loss 0.375\n",
      "Ep 2 (Step 000250): Train loss 0.406, Val loss 0.378\n",
      "Training accuracy: 80.00% | Validation accuracy: 85.00%\n",
      "Ep 3 (Step 000300): Train loss 0.286, Val loss 0.384\n",
      "Ep 3 (Step 000350): Train loss 0.351, Val loss 0.379\n",
      "Training accuracy: 82.50% | Validation accuracy: 85.00%\n",
      "Ep 4 (Step 000400): Train loss 0.231, Val loss 0.366\n",
      "Ep 4 (Step 000450): Train loss 0.308, Val loss 0.352\n",
      "Ep 4 (Step 000500): Train loss 0.361, Val loss 0.384\n",
      "Training accuracy: 92.50% | Validation accuracy: 85.00%\n",
      "Ep 5 (Step 000550): Train loss 0.343, Val loss 0.340\n",
      "Ep 5 (Step 000600): Train loss 0.439, Val loss 0.332\n",
      "Training accuracy: 85.00% | Validation accuracy: 90.00%\n",
      "Training completed in 1.79 minutes.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)\n",
    "\n",
    "num_epochs = 5\n",
    "train_losses, val_losses, train_accs, val_accs, examples_seen = train_classifier_simple(\n",
    "    model, train_loader, val_loader, optimizer, device,\n",
    "    num_epochs=num_epochs, eval_freq=50, eval_iter=5,\n",
    ")\n",
    "\n",
    "end_time = time.time()\n",
    "execution_time_minutes = (end_time - start_time) / 60\n",
    "print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1261bf90-3ce7-4591-895a-044a05538f30",
   "metadata": {},
   "source": [
    "- Similar to part 4, we use matplotlib to plot the loss function for the training and validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "cURgnDqdCeka",
   "metadata": {
    "id": "cURgnDqdCeka"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_values(epochs_seen, examples_seen, train_values, val_values, label=\"loss\"):\n",
    "    fig, ax1 = plt.subplots(figsize=(5, 3))\n",
    "\n",
    "    # Plot training and validation loss against epochs\n",
    "    ax1.plot(epochs_seen, train_values, label=f\"Training {label}\")\n",
    "    ax1.plot(epochs_seen, val_values, linestyle=\"-.\", label=f\"Validation {label}\")\n",
    "    ax1.set_xlabel(\"Epochs\")\n",
    "    ax1.set_ylabel(label.capitalize())\n",
    "    ax1.legend()\n",
    "\n",
    "    # Create a second x-axis for examples seen\n",
    "    ax2 = ax1.twiny()  # Create a second x-axis that shares the same y-axis\n",
    "    ax2.plot(examples_seen, train_values, alpha=0)  # Invisible plot for aligning ticks\n",
    "    ax2.set_xlabel(\"Examples seen\")\n",
    "\n",
    "    fig.tight_layout()  # Adjust layout to make room\n",
    "    plt.savefig(f\"{label}-plot.pdf\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "OIqRt466DiGk",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 307
    },
    "id": "OIqRt466DiGk",
    "outputId": "b16987cf-0001-4652-ddaf-02f7cffc34db"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjPUlEQVR4nO3dd3gUVdvA4d9uyqYX0gMhoQYCJNRECE0NAgoKovAhXRQLiIoo8ipFfBULKiKICgKiUpT2otIjvdcQWmgJCZAKpPfd+f7YsBAIJXUX8tzXNVd2Z87MPDOEPDNnzpyjUhRFQQghhBAmSW3sAIQQQghxZ5KohRBCCBMmiVoIIYQwYZKohRBCCBMmiVoIIYQwYZKohRBCCBMmiVoIIYQwYZKohRBCCBMmiVoIIYQwYZKohRD3pXPnzrz11lvGDkOIakcStRBVZOjQoahUqtumbt26GTs0IYQJMzd2AEJUJ926dWP+/PnF5mk0GiNFI4R4EMgdtRBVSKPR4OnpWWxydnYGYMuWLVhaWrJ9+3ZD+S+++AJ3d3cSExMBWLduHe3bt8fJyQkXFxd69OjBuXPnDOVjYmJQqVT88ccfdOjQAWtra9q0acPp06fZv38/rVu3xs7Oju7du5OcnGxYb+jQofTq1YuPPvoINzc3HBwcePXVV8nPz7/jseTl5TF27Fhq1qyJra0tISEhbNmyxbD8woUL9OzZE2dnZ2xtbWnSpAlr1qy54/a+//57GjRogJWVFR4eHjz33HOGZTqdjqlTp1KnTh2sra0JCgpi2bJlxdY/duwY3bt3x87ODg8PDwYNGkRKSopheefOnRk9ejTvvfceNWrUwNPTk8mTJ98xHiFMhSRqIUzE9WfAgwYNIi0tjcOHDzNhwgTmzp2Lh4cHAFlZWYwZM4YDBw4QHh6OWq2md+/e6HS6YtuaNGkSH374IYcOHcLc3JwXXniB9957j2+//Zbt27dz9uxZJk6cWGyd8PBwTp48yZYtW1i8eDErVqzgo48+umO8o0aNYvfu3SxZsoSjR4/y/PPP061bN86cOQPAyJEjycvLY9u2bURGRvL5559jZ2dX4rYOHDjA6NGjmTJlClFRUaxbt46OHTsalk+dOpWFCxfyww8/cPz4cd5++20GDhzI1q1bAUhNTeWxxx6jRYsWHDhwgHXr1pGYmEjfvn2L7eeXX37B1taWvXv38sUXXzBlyhQ2btx4n/9CQhiJIoSoEkOGDFHMzMwUW1vbYtMnn3xiKJOXl6c0b95c6du3rxIQEKC8/PLLd91mcnKyAiiRkZGKoihKdHS0Aihz5841lFm8eLECKOHh4YZ5U6dOVfz9/YvFVqNGDSUrK8swb/bs2YqdnZ2i1WoVRVGUTp06KW+++aaiKIpy4cIFxczMTLl06VKxeB5//HFl/PjxiqIoSrNmzZTJkyff17lZvny54uDgoKSnp9+2LDc3V7GxsVF27dpVbP7w4cOV/v37K4qiKB9//LHyxBNPFFseFxenAEpUVJQh/vbt2xcr06ZNG2XcuHH3FaMQxiLPqIWoQo8++iizZ88uNq9GjRqGz5aWlvz+++8EBgbi6+vLN998U6zsmTNnmDhxInv37iUlJcVwJx0bG0vTpk0N5QIDAw2fr9+NN2vWrNi8pKSkYtsOCgrCxsbG8L1t27ZkZmYSFxeHr69vsbKRkZFotVoaNmxYbH5eXh4uLi4AjB49mtdee40NGzYQFhZGnz59isV1sy5duuDr60vdunXp1q0b3bp1o3fv3tjY2HD27Fmys7Pp0qVLsXXy8/Np0aIFABEREWzevLnEO/Zz584Z4rx1/15eXredByFMjSRqIaqQra0t9evXv2uZXbt2AXD16lWuXr2Kra2tYVnPnj3x9fVlzpw5eHt7o9PpaNq06W3Pki0sLAyfVSpVifNurS4vjczMTMzMzDh48CBmZmbFll1Pli+99BJdu3bln3/+YcOGDUydOpWvvvqKN95447bt2dvbc+jQIbZs2cKGDRuYOHEikydPZv/+/WRmZgLwzz//ULNmzWLrXW+Il5mZSc+ePfn8889v27aXl5fh883nAMp/HoSoCpKohTAh586d4+2332bOnDksXbqUIUOGsGnTJtRqNVeuXCEqKoo5c+bQoUMHAHbs2FFh+46IiCAnJwdra2sA9uzZg52dHT4+PreVbdGiBVqtlqSkJEMsJfHx8eHVV1/l1VdfZfz48cyZM6fERA1gbm5OWFgYYWFhTJo0CScnJ/7991+6dOmCRqMhNjaWTp06lbhuy5YtWb58OX5+fpiby5818XCR32ghqlBeXh4JCQnF5pmbm+Pq6opWq2XgwIF07dqVYcOG0a1bN5o1a8ZXX33Fu+++i7OzMy4uLvz00094eXkRGxvL+++/X2Gx5efnM3z4cD788ENiYmKYNGkSo0aNQq2+vc1pw4YNGTBgAIMHD+arr76iRYsWJCcnEx4eTmBgIE899RRvvfUW3bt3p2HDhly7do3NmzfTuHHjEvf9999/c/78eTp27IizszNr1qxBp9Ph7++Pvb09Y8eO5e2330an09G+fXvS0tLYuXMnDg4ODBkyhJEjRzJnzhz69+9vaNV99uxZlixZwty5c2+76xfiQSKJWogqtG7dumJVsQD+/v6cOnWKTz75hAsXLvD3338D+irbn376if79+/PEE08QFBTEkiVLGD16NE2bNsXf358ZM2bQuXPnCont8ccfp0GDBnTs2JG8vDz69+9/19eX5s+fz3//+1/eeecdLl26hKurK4888gg9evQAQKvVMnLkSC5evIiDgwPdunW77Zn7dU5OTqxYsYLJkyeTm5tLgwYNWLx4MU2aNAHg448/xs3NjalTp3L+/HmcnJxo2bIl//nPfwDw9vZm586djBs3jieeeIK8vDx8fX3p1q1biRcaQjxIVIqiKMYOQghhXEOHDiU1NZVVq1YZOxQhxC3kUlMIIYQwYZKohRBCCBMmVd9CCCGECZM7aiGEEMKESaIWQgghTJgkaiGEEMKESaIuh1mzZuHn54eVlRUhISHs27fP2CFVmm3bttGzZ0+8vb1RqVS3vcajKAoTJ07Ey8sLa2trwsLCDKMoXXf16lUGDBiAg4MDTk5ODB8+3NA95HVHjx6lQ4cOWFlZ4ePjwxdffFHZh1Yhpk6dSps2bbC3t8fd3Z1evXoRFRVVrExubi4jR47ExcUFOzs7+vTpYxi+8rrY2FieeuopbGxscHd3591336WwsLBYmS1bttCyZUs0Gg3169dnwYIFlX14FWL27NkEBgbi4OCAg4MDbdu2Ze3atYbl1f38lOSzzz5DpVLx1ltvGebJeYLJkyejUqmKTY0aNTIsf+jOkVGHBHmALVmyRLG0tFTmzZunHD9+XHn55ZcVJycnJTEx0dihVYo1a9YoH3zwgbJixQoFUFauXFls+WeffaY4Ojoqq1atUiIiIpSnn35aqVOnjpKTk2Mo061bNyUoKEjZs2ePsn37dqV+/fqG0Y8URVHS0tIUDw8PZcCAAcqxY8eUxYsXK9bW1sqPP/5YVYdZZl27dlXmz5+vHDt2TDly5Ijy5JNPKrVr11YyMzMNZV599VXFx8dHCQ8PVw4cOKA88sgjSrt27QzLCwsLlaZNmyphYWHK4cOHlTVr1iiurq6G0agURVHOnz+v2NjYKGPGjFFOnDihfPfdd4qZmZmybt26Kj3esli9erXyzz//KKdPn1aioqKU//znP4qFhYVy7NgxRVHk/Nxq3759ip+fnxIYGGgYtUxR5DwpiqJMmjRJadKkiRIfH2+YkpOTDcsftnMkibqMgoODlZEjRxq+a7VaxdvbW5k6daoRo6oatyZqnU6neHp6Kl9++aVhXmpqqqLRaJTFixcriqIoJ06cUABl//79hjJr165VVCqVYajE77//XnF2dlby8vIMZcaNG1dsOMYHRVJSkgIoW7duVRRFfz4sLCyUP//801Dm5MmTCqDs3r1bURT9xZBarVYSEhIMZWbPnq04ODgYzsl7772nNGnSpNi++vXrp3Tt2rWyD6lSODs7K3PnzpXzc4uMjAylQYMGysaNG4sNLyrnSW/SpElKUFBQicsexnMkVd9lkJ+fz8GDBwkLCzPMU6vVhIWFsXv3biNGZhzR0dEkJCQUOx+Ojo6EhIQYzsfu3btxcnKidevWhjJhYWGo1Wr27t1rKNOxY0csLS0NZbp27UpUVBTXrl2roqOpGGlpacCNISwPHjxIQUFBsXPUqFEjateuXewcNWvWzDAsJeiPPz09nePHjxvK3LyN62UetN87rVbLkiVLyMrKom3btnJ+bjFy5Eieeuqp245FztMNZ86cwdvbm7p16zJgwABiY2OBh/McSaIug5SUFLRabbF/ZNCP8XvrgAvVwfVjvtv5SEhIwN3dvdhyc3NzatSoUaxMSdu4eR8PAp1Ox1tvvUVoaKhhjOiEhAQsLS1xcnIqVvbWc3Sv479TmfT0dHJycirjcCpUZGQkdnZ2aDQaXn31VVauXElAQICcn5ssWbKEQ4cOMXXq1NuWyXnSCwkJYcGCBaxbt47Zs2cTHR1Nhw4dyMjIeCjPkQzKIUQFGzlyJMeOHavQISgfFv7+/hw5coS0tDSWLVvGkCFD2Lp1q7HDMhlxcXG8+eabbNy4ESsrK2OHY7K6d+9u+BwYGEhISAi+vr788ccfhmFaHyZyR10Grq6umJmZ3daKMDExEU9PTyNFZTzXj/lu58PT05OkpKRiywsLC7l69WqxMiVt4+Z9mLpRo0bx999/s3nzZmrVqmWY7+npSX5+PqmpqcXK33qO7nX8dyrj4ODwQPyBsrS0pH79+rRq1YqpU6cSFBTEt99+K+enyMGDB0lKSqJly5aYm5tjbm7O1q1bmTFjBubm5nh4eMh5KoGTkxMNGzbk7NmzD+XvkiTqMrC0tKRVq1aEh4cb5ul0OsLDw2nbtq0RIzOOOnXq4OnpWex8pKens3fvXsP5aNu2LampqRw8eNBQ5t9//0Wn0xESEmIos23bNgoKCgxlNm7ciL+/P87OzlV0NGWjKAqjRo1i5cqV/Pvvv9SpU6fY8latWmFhYVHsHEVFRREbG1vsHEVGRha7oNm4cSMODg4EBAQYyty8jetlHtTfO51OR15enpyfIo8//jiRkZEcOXLEMLVu3ZoBAwYYPst5ul1mZibnzp3Dy8vr4fxdqvLmaw+JJUuWKBqNRlmwYIFy4sQJZcSIEYqTk1OxVoQPk4yMDOXw4cPK4cOHFUD5+uuvlcOHDysXLlxQFEX/epaTk5Pyv//9Tzl69KjyzDPPlPh6VosWLZS9e/cqO3bsUBo0aFDs9azU1FTFw8NDGTRokHLs2DFlyZIlio2NzQPxetZrr72mODo6Klu2bCn2ykh2drahzKuvvqrUrl1b+ffff5UDBw4obdu2Vdq2bWtYfv2VkSeeeEI5cuSIsm7dOsXNza3EV0beffdd5eTJk8qsWbMemNdq3n//fWXr1q1KdHS0cvToUeX9999XVCqVsmHDBkVR5Pzcyc2tvhVFzpOiKMo777yjbNmyRYmOjlZ27typhIWFKa6urkpSUpKiKA/fOZJEXQ7fffedUrt2bcXS0lIJDg5W9uzZY+yQKs3mzZsV4LZpyJAhiqLoX9GaMGGC4uHhoWg0GuXxxx9XoqKiim3jypUrSv/+/RU7OzvFwcFBGTZsmJKRkVGsTEREhNK+fXtFo9EoNWvWVD777LOqOsRyKencAMr8+fMNZXJycpTXX39dcXZ2VmxsbJTevXsr8fHxxbYTExOjdO/eXbG2tlZcXV2Vd955RykoKChWZvPmzUrz5s0VS0tLpW7dusX2YcpefPFFxdfXV7G0tFTc3NyUxx9/3JCkFUXOz53cmqjlPOlfk/Ly8lIsLS2VmjVrKv369VPOnj1rWP6wnSMZPUsIIYQwYfKMWgghhDBhkqiFEEIIEyaJWgghhDBhkqiFEEIIEyaJWgghhDBhkqiFEEIIEyaJuhzy8vKYPHkyeXl5xg7FpMl5ujc5R/cm5+je5Bzd24N4juQ96nJIT0/H0dGRtLQ0HBwcjB2OyZLzdG9yju5NztG9yTm6twfxHMkdtRBCCGHCJFELIYQQJqzajUddWFjI4cOH8fDwQK0u33VKRkYGAJcuXSI9Pb0iwnsoyXm6NzlH9ybn6N7kHN2bqZwjnU5HYmIiLVq0wNz87qm42j2j3r9/P8HBwcYOQwghhGDfvn20adPmrmWq3R21h4cHoD85Xl5eRo5GCCFEdRQfH09wcLAhJ91NtUvU16u7vby8qFWrlpGjEUIIUZ3dzyNYaUwmhBBCmDBJ1EIIIYQJk0QthBBCmLBq94xaCCHuRqvVUlBQYOwwxAPOwsICMzOzCtmWJGohhAAURSEhIYHU1FRjhyIeEk5OTnh6eqJSqcq1HUnU5VGYD6fXgq07+LY1djRCiHK4nqTd3d2xsbEp9x9XUX0pikJ2djZJSUkA5X4VWBJ1eez4GrZMhfph4Lvc2NEIIcpIq9UakrSLi4uxwxEPAWtrawCSkpJwd3cvVzW4NCYrj8C++p9nwyHtonFjEUKU2fVn0jY2NkaORDxMrv8+lbfNgyTq8qhRF/w6AAocWWTsaIQQ5STV3aIiVdTvkyTq8moxUP/z8K+g0xk3FiGEEA8dSdTl1fhp0DhAaizEbDd2NEIIUW5+fn5Mnz79vstv2bIFlUpV6S3mFyxYgJOTU6XuwxRJoi4vSxto9pz+8+FfjRuLEKJaUalUd50mT55cpu3u37+fESNG3Hf5du3aER8fj6OjY5n2J+5OWn1XhBaD4MA8OLEanrwG1s7GjkgIUQ3Ex8cbPi9dupSJEycSFRVlmGdnZ2f4rCgKWq32nmMfA7i5uZUqDktLSzw9PUu1jrh/ckddEbxbgEdT0OZB5DJjRyOEqCY8PT0Nk6OjIyqVyvD91KlT2Nvbs3btWlq1aoVGo2HHjh2cO3eOZ555Bg8PD+zs7GjTpg2bNm0qtt1bq75VKhVz586ld+/e2NjY0KBBA1avXm1YfmvV9/Uq6vXr19O4cWPs7Ozo1q1bsQuLwsJCRo8ejZOTEy4uLowbN44hQ4bQq1evUp2D2bNnU69ePSwtLfH39+fXX2/UbCqKwuTJk6lduzYajQZvb29Gjx5tWP7999/ToEEDrKys8PDw4LnnnivVvquKJOqKoFLp76oBDi00bixCiAqhKArZ+YVGmRRFqbDjeP/99/nss884efIkgYGBZGZm8uSTTxIeHs7hw4fp1q0bPXv2JDY29q7b+eijj+jbty9Hjx7lySefZMCAAVy9evWO5bOzs5k2bRq//vor27ZtIzY2lrFjxxqWf/755/z+++/Mnz+fnTt3kp6ezqpVq0p1bCtXruTNN9/knXfe4dixY7zyyisMGzaMzZs3A7B8+XK++eYbfvzxR86cOcOqVato1qwZAAcOHGD06NFMmTKFqKgo1q1bR8eOHUu1/6oiVd8VJbAvbJwACUchPgK8gowdkRCiHHIKtARMXG+UfZ+Y0hUby4r58zxlyhS6dOli+F6jRg2Cgm78ffr4449ZuXIlq1evZtSoUXfcztChQ+nfvz8An376KTNmzGDfvn1069atxPIFBQX88MMP1KtXD4BRo0YxZcoUw/LvvvuO8ePH07t3bwBmzpzJmjVrSnVs06ZNY+jQobz++usAjBkzhj179jBt2jQeffRRYmNj8fT0JCwsDAsLC2rXrk1wcDAAsbGx2Nra0qNHD+zt7fH19aVFixal2n9VMfod9axZs/Dz88PKyoqQkBD27dt31/LTp0/H398fa2trfHx8ePvtt8nNza2iaO/CpgY06qH/fEgalQkhTEPr1q2Lfc/MzGTs2LE0btwYJycn7OzsOHny5D3vqAMDAw2fbW1tcXBwMHSRWRIbGxtDkgZ9N5rXy6elpZGYmGhImgBmZma0atWqVMd28uRJQkNDi80LDQ3l5MmTADz//PPk5ORQt25dXn75ZVauXElhYSEAXbp0wdfXl7p16zJo0CB+//13srOzS7X/qmLUO+qlS5cyZswYfvjhB0JCQpg+fTpdu3YlKioKd3f328ovWrSI999/n3nz5tGuXTtOnz7N0KFDUalUfP3110Y4glu0HATHV8DJ1dD9c1BXzMgpQoiqZ21hxokpXY2274pia2tb7PvYsWPZuHEj06ZNo379+lhbW/Pcc8+Rn59/1+1YWFgU+65SqdDdpe+IkspXZJX+/fDx8SEqKopNmzaxceNGXn/9db788ku2bt2Kvb09hw4dYsuWLWzYsIGJEycyefJk9u/fb3KvgBn1jvrrr7/m5ZdfZtiwYQQEBPDDDz9gY2PDvHnzSiy/a9cuQkNDeeGFF/Dz8+OJJ56gf//+97wLr0w63U2/eHU6Q4/p8PoeSdJCPOBUKhU2luZGmSqzh7SdO3cydOhQevfuTbNmzfD09CQmJqbS9lcSR0dHPDw82L9/v2GeVqvl0KFDpdpO48aN2blzZ7F5O3fuJCAgwPDd2tqanj17MmPGDLZs2cLu3buJjIwEwNzcnLCwML744guOHj1KTEwM//77bzmOrHIY7Y46Pz+fgwcPMn78eMM8tVpNWFgYu3fvLnGddu3a8dtvv7Fv3z6Cg4M5f/48a9asYdCgQXfcT15eHnl5eYbvGRkZFRK/Tqfw/ZazLNoby7LX2uHtZA1qNbQeViHbF0KIytCgQQNWrFhBz549UalUTJgw4a53xpXljTfeYOrUqdSvX59GjRrx3Xffce3atVJdpLz77rv07duXFi1aEBYWxl9//cWKFSsMrdgXLFiAVqslJCQEGxsbfvvtN6ytrfH19eXvv//m/PnzdOzYEWdnZ9asWYNOp8Pf37+yDrnMjHZHnZKSglarxcPDo9h8Dw8PEhISSlznhRdeYMqUKbRv3x4LCwvq1atH586d+c9//nPH/UydOhVHR0fDdPOVVnmo1Sp2nE3hclouv+yOKbmQTlsh+xJCiIry9ddf4+zsTLt27ejZsyddu3alZcuWVR7HuHHj6N+/P4MHD6Zt27bY2dnRtWtXrKys7nsbvXr14ttvv2XatGk0adKEH3/8kfnz59O5c2dAPx70nDlzCA0NJTAwkE2bNvHXX3/h4uKCk5MTK1as4LHHHqNx48b88MMPLF68mCZNmlTSEZeDYiSXLl1SAGXXrl3F5r/77rtKcHBwiets3rxZ8fDwUObMmaMcPXpUWbFiheLj46NMmTLljvvJzc1V0tLSDNOJEycUQImLiyv3MWw8nqD4jvtbaTppnZKZW3BjwZmNijInTFHCPy73PoQQlS8nJ0c5ceKEkpOTY+xQqi2tVqs0bNhQ+fDDD40dSoW52+9VXFzcfecio1V9u7q6YmZmRmJiYrH5iYmJd+zhZsKECQwaNIiXXnoJgGbNmpGVlcWIESP44IMPUKtvryDQaDRoNBrD9/T09Ao7hscauVPH1ZbolCz+PBDH0NA6+gW5aXBxH2QmwKMf6N+zFkIIYXDhwgU2bNhAp06dyMvLY+bMmURHR/PCCy8YOzSTY7Sqb0tLS1q1akV4eLhhnk6nIzw8nLZt25a4TnZ29m3J+Ppg3EoVtyYEffX3i+31yXnezhi01xuW+T8FT/wXhm+SJC2EECVQq9UsWLCANm3aEBoaSmRkJJs2baJx48bGDs3kGPX1rDFjxjBkyBBat25NcHAw06dPJysri2HD9A2yBg8eTM2aNZk6dSoAPXv25Ouvv6ZFixaEhIRw9uxZJkyYQM+ePQ0Ju6r1aVmTrzZEEXs1m40nEunW1BMsrKDdG0aJRwghHgQ+Pj63tdgWJTNqou7Xrx/JyclMnDiRhIQEmjdvzrp16wwNzGJjY4vdQX/44YeoVCo+/PBDLl26hJubGz179uSTTz4x1iFgY2nOgJDazNp8jp93nNcn6lspitxZCyGEKBOVYow6YyO6ePEiPj4+xMXFUatWrQrZZmJ6Lu0//5cCrcKqkaE093HSLzi7CXZMB/8noe3rFbIvIUTFy83NJTo6mjp16pSq1bEQd3O336vS5CKjdyH6MPBwsKJnkDcAP++IvrHg2gWI2a4fqKN6XQ8JIYSoIJKoK8jwokZlayLjuZSao5/ZtA+YW0HySbh00IjRCSGEeFBJoq4gTbwdaVfPBa1O4ZddMfqZ1k4Q8Iz+82EZqEMIIUTpSaKuQC910N9VL94bS2aefoQWwzjVkcshP8tIkQkhhHhQSaKuQJ0bulPXzZaMvEL+2B+nn+nXHpzrQH4GnPifcQMUQogSdO7cmbfeesvw3c/Pj+nTp991HZVKxapVq8q974razt1MnjyZ5s2bV+o+KpMk6gqkVqt4sah3svm7ovUdoKhU0GKgvoCMUy2EqEA9e/akW7duJS7bvn07KpWKo0ePlnq7+/fvZ8SIEeUNr5g7Jcv4+Hi6d+9eoft62EiirmB9WtbCycaCuKs5bDxRNLhI8xdApYbYXZBy1rgBCiEeGsOHD2fjxo1cvHjxtmXz58+ndevWBAYGlnq7bm5u2NjYVESI9+Tp6Vmsm2dxO0nUFcza0oyBIb4AzN1e9KqWgzfU76L/LI3KhBAVpEePHri5ubFgwYJi8zMzM/nzzz8ZPnw4V65coX///tSsWRMbGxuaNWvG4sWL77rdW6u+z5w5Q8eOHbGysiIgIICNGzfets64ceNo2LAhNjY21K1blwkTJlBQUADoh5v86KOPiIiIQKVSoVKpDDHfWvUdGRnJY489hrW1NS4uLowYMYLMzEzD8qFDh9KrVy+mTZuGl5cXLi4ujBw50rCv+6HT6ZgyZQq1atVCo9EYOtu6Lj8/n1GjRuHl5YWVlRW+vr6GHjIVRWHy5MnUrl0bjUaDt7c3o0ePvu99l4Uk6kowuK0vFmYqDly4xuHYa/qZLYsalUUsBm2h8YITQpROflbpp5v/j2sL9fMKcu5vu6Vgbm7O4MGDWbBgQbHxDv7880+0Wi39+/cnNzeXVq1a8c8//3Ds2DFGjBjBoEGD2Ldv333tQ6fT8eyzz2JpacnevXv54YcfGDdu3G3l7O3tWbBgASdOnODbb79lzpw5fPPNN4C+F8p33nmHJk2aEB8fT3x8PP369bttG1lZWXTt2hVnZ2f279/Pn3/+yaZNmxg1alSxcps3b+bcuXNs3ryZX375hQULFtx2sXI33377LV999RXTpk3j6NGjdO3alaeffpozZ84AMGPGDFavXs0ff/xBVFQUv//+O35+fgAsX76cb775hh9//JEzZ86watUqmjVrdt/7LgujdiH6sHJ3sOLpoJosP3SRn3dEM/MFZ2jYDWzdIDMRzmyARk8aO0whxP341Lv06zy/AJr01n8+9Rf8ORR828Owf26Umd4Msq/cvu7ktFLt6sUXX+TLL79k69athnGY58+fT58+fXB0dMTR0ZGxY8cayr/xxhusX7+eP/74g+Dg4Htuf9OmTZw6dYr169fj7a0/F59++ultz5U//PBDw2c/Pz/Gjh3LkiVLeO+997C2tsbOzg5zc/M7jo4IsGjRInJzc1m4cCG2trYAzJw5k549e/L5558bupd2dnZm5syZmJmZ0ahRI5566inCw8N5+eWX7+ucTZs2jXHjxvF///d/AHz++eds3ryZ6dOnM2vWLGJjY2nQoAHt27dHpVLh6+trWDc2NhZPT0/CwsKwsLCgdu3a93Uey0PuqCvJ9Q5Q1h5L4OK1bDCzgMCiK0ip/hZCVJBGjRrRrl075s2bB8DZs2fZvn07w4cPB0Cr1fLxxx/TrFkzatSogZ2dHevXryc2Nva+tn/y5El8fHwMSRoocYTDpUuXEhoaiqenJ3Z2dnz44Yf3vY+b9xUUFGRI0gChoaHodDqioqIM85o0aVJsICYvLy+SkpLuax/p6elcvnyZ0NDQYvNDQ0M5efIkoK9eP3LkCP7+/owePZoNGzYYyj3//PPk5ORQt25dXn75ZVauXElhYeXWksoddSUJ8HYgtL4LO89e4ZddMXzwVAC0HAwH5oGtqwzUIcSD4j+XS7+O2U2Noxr11G9Ddct90VuR5YvrJsOHD+eNN95g1qxZzJ8/n3r16tGpUycAvvzyS7799lumT59Os2bNsLW15a233iI/P7/C9r97924GDBjARx99RNeuXXF0dGTJkiV89dVXFbaPm1lYWBT7rlKp0Ol0Fbb9li1bEh0dzdq1a9m0aRN9+/YlLCyMZcuW4ePjQ1RUFJs2bWLjxo28/vrrhhqNW+OqKHJHXYleal8XgCX74sjILQA3f3j3LDz9nSRpIR4Ulraln8xuugcyM9fPs7C+v+2WQd++fVGr1SxatIiFCxfy4osvoir6G7Nz506eeeYZBg4cSFBQEHXr1uX06dP3ve3GjRsTFxdHfHy8Yd6ePXuKldm1axe+vr588MEHtG7dmgYNGnDhwoXih2tpiVarvee+IiIiyMq68ax+586dqNVq/P397zvmu3FwcMDb2/u2ITZ37txJQEBAsXL9+vVjzpw5LF26lOXLl3P16lUArK2t6dmzJzNmzGDLli3s3r2byMiKu/C6lSTqStSpoRv1rneAcqDo9Yky/kcUQog7sbOzo1+/fowfP574+HiGDh1qWNagQQM2btzIrl27OHnyJK+88gqJiYn3ve2wsDAaNmzIkCFDiIiIYPv27XzwwQfFyjRo0IDY2FiWLFnCuXPnmDFjBitXrixWxs/Pj+joaI4cOUJKSgp5eXm37WvAgAFYWVkxZMgQjh07xubNm3njjTcYNGiQ4fl0RXj33Xf5/PPPWbp0KVFRUbz//vscOXKEN998E4Cvv/6axYsXc+rUKU6fPs2ff/6Jp6cnTk5OLFiwgJ9//pljx45x/vx5fvvtN6ytrYs9x65okqgrkVqtYnjRXfX8ndEUam+qmomPgJQzRopMCPGwGT58ONeuXaNr167Fnid/+OGHtGzZkq5du9K5c2c8PT3p1avXfW9XrVazcuVKcnJyCA4O5qWXXuKTTz4pVubpp5/m7bffZtSoUTRv3pxdu3YxYcKEYmX69OlDt27dePTRR3FzcyvxFTEbGxvWr1/P1atXadOmDc899xyPP/44M2fOLN3JuIfRo0czZswY3nnnHZo1a8a6detYvXo1DRo0APQt2L/44gtat25NmzZtiImJYc2aNajVapycnJgzZw6hoaEEBgayadMm/vrrL1xcXCo0xpvJeNSVLLdAS9up4VzLLuD7AS15spkXbP4Utn4OzQdAr+8rPQYhxN3JeNSiMsh41A8IKwszBj1yvQOU8/qZ9R4DM0tAnlMLIYS4O0nUVWBgW18szdQcik3lUOw18AmBd6Kg1yxjhyaEEMLESaKuAu72VjzdXP/M6Ocd0foW3zY1jByVEEKIB4Ek6ipi6AAlMp64q9k3FiSfhrRLRopKCCGEqZNEXUUaeznQvr4rOgV+2RWjn7npI5jVBvZIgzIhhBAlk0RdhYZ30N9VL9lf1AGKT4h+QcRiKKy4XoKEEGVTkb1bCVFRv0/ShWgV6tTAjfrudpxNymTp/jheahcGdp6QmQCn10LAM8YOUYhqydLSErVazeXLl3Fzc8PS0tLQs5cQpaUoCvn5+SQnJ6NWq7G0tCzX9iRRVyF9Byh1GL8ikvk7Yxjazg/z5i/Ajq/h0K+SqIUwErVaTZ06dYiPj+fy5TL07S1ECWxsbKhduzZqdfkqryVRV7HeLWry5fooLqXmsP54Ik+1GKhP1OfC9Y3KHGsaO0QhqiVLS0tq165NYWHhPfukFuJezMzMMDc3r5CaGUnUVczKwoyBj/gyI/wMc3ec56nXQ/Xj1F7YAUcWQad3jR2iENWWSqXCwsKi0kZBEqIspDGZEQx6RN8ByuHYVA5euAYtBuoXHPkNpDGLEEKIm0iiNgI3ew29WlzvAOW8/tm0xgGuxejvrIUQQogikqiN5PqoWuuOJRCXCTTto19w6FfjBSWEEMLkSKI2En9Pezo00HeAsmBXDLQcpF9wcjXkpBozNCGEECZEErURvdRBf1e9dH8c6TWagXsTKMyFyD+NHJkQQghTIYnaiDo2cKWBux2ZeYX8ceDijbvqw1L9LYQQQk8StRGpVCrDYB3zd8ZQ2OR5aP0i9PjGyJEJIYQwFZKojaxXi5q42FpyKTWHddH5+iRds5WxwxJCCGEiJFEb2fUOUADmbI9GURQjRySEEMKUSKI2AQMf8cXSXE1EXCqHYq/B5cOw+g04sdrYoQkhhDAySdQmwM1eQ+/m+j6+526Phqi1cGgh7J9r5MiEEEIYmyRqE3F9rOr1xxO45NcHgvpD5/eNHJUQQghjk0RtIhp62NOxoRs6BeYeK4TeP4BvO2OHJYQQwsgkUZuQl4pe1fpjfxxpOQVGjkYIIYQpkERtQjo0cKWhhx1Z+VqW7o+FhGOw5l2I3mbs0IQQQhiJJGoTolKpeKlosI4FO2PQHVwA+36C/T8bNzAhhBBGI4naxDzd3BtXO0sup+Wy3a67fuapfyDrinEDE0IIYRRGT9SzZs3Cz88PKysrQkJC2Ldv313Lp6amMnLkSLy8vNBoNDRs2JA1a9ZUUbSVz8rCjEGP+AHwdaQGxSsIdAVwdKlxAxNCCGEURk3US5cuZcyYMUyaNIlDhw4RFBRE165dSUpKKrF8fn4+Xbp0ISYmhmXLlhEVFcWcOXOoWbNmFUdeuQY+UlvfAcrFNGJ9i8apPvwrSK9lQghR7Rg1UX/99de8/PLLDBs2jICAAH744QdsbGyYN29eieXnzZvH1atXWbVqFaGhofj5+dGpUyeCgoKqOPLK5WKnoU9L/cXH9ITmYG4FSSfg8iHjBiaEEKLKGS1R5+fnc/DgQcLCwm4Eo1YTFhbG7t27S1xn9erVtG3blpEjR+Lh4UHTpk359NNP0Wq1VRV2lXkxVP+q1qqoTDLrPamfeUiGvxRCiOrGaIk6JSUFrVaLh4dHsfkeHh4kJCSUuM758+dZtmwZWq2WNWvWMGHCBL766iv++9//3nE/eXl5pKenG6aMjIwKPY7K0sDDnk4N3VAUWKZ7VD/z2HLIzzZuYEIIIaqU0RuTlYZOp8Pd3Z2ffvqJVq1a0a9fPz744AN++OGHO64zdepUHB0dDVNAQEAVRlw+LxV1K/pllCtaJz/IS4cT/zNuUEIIIaqU0RK1q6srZmZmJCYmFpufmJiIp6dniet4eXnRsGFDzMzMDPMaN25MQkIC+fn5Ja4zfvx40tLSDNOJEycq7iAqWfv6rvh72JOVr3DQuaj6+7BUfwshRHVitERtaWlJq1atCA8PN8zT6XSEh4fTtm3bEtcJDQ3l7Nmz6HQ6w7zTp0/j5eWFpaVlietoNBocHBwMk729fcUeSCVSqVSGwTr+e6k5ikoNF3bClXNGjkwIIURVMWrV95gxY5gzZw6//PILJ0+e5LXXXiMrK4thw4YBMHjwYMaPH28o/9prr3H16lXefPNNTp8+zT///MOnn37KyJEjjXUIle6Z5t642mk4mm5HknuofqbcVQshRLVhXpaV4uLiUKlU1KpVC4B9+/axaNEiAgICGDFixH1vp1+/fiQnJzNx4kQSEhJo3rw569atMzQwi42NRa2+cS3h4+PD+vXrefvttwkMDKRmzZq8+eabjBs3riyH8UDQmJsxuK0vX288zbzsDoxnOySdMnZYQgghqohKUUrfi0aHDh0YMWIEgwYNIiEhAX9/f5o0acKZM2d44403mDhxYmXEWiEuXryIj48PcXFxhgsNU3clM492n/2LrjCfFc+70qxVqLFDEkIIUQ6lyUVlqvo+duwYwcHBAPzxxx80bdqUXbt28fvvv7NgwYKybFLchYudhmdb1qIAc2ae0Bg7HCGEEFWoTIm6oKAAjUafMDZt2sTTTz8NQKNGjYiPj6+46ITB8PZ+AGw4kciFK1mQkwq56UaNSQghROUrU6Ju0qQJP/zwA9u3b2fjxo1069YNgMuXL+Pi4lKhAQq9+u72POqv7wDl7LLJ8JU/HCi5q1UhhBAPjzIl6s8//5wff/yRzp07079/f0Nf26tXrzZUiYuK91IH/VjVWy8Bhblwcb9xAxJCCFHpytTqu3PnzqSkpJCeno6zs7Nh/ogRI7Cxsamw4ERx7eq50MjTnuUJITQPbc+zPZ42dkhCCCEqWZnuqHNycsjLyzMk6QsXLjB9+nSioqJwd3ev0ADFDSqVipc61CULa76ItKNAJ8NeCiHEw65MifqZZ55h4cKFAKSmphISEsJXX31Fr169mD17doUGKIrrGeSFq52GhPRc1kTGQ9w+uHzE2GFVuR1nUnhm1k72x1w1dihCCFGpypSoDx06RIcOHQBYtmwZHh4eXLhwgYULFzJjxowKDVAUpzE3Y0hbXwD+3bwB5dfesPAZiD9q5MiqTlp2AW//cYSIuFRGLz5Mem6BsUMSQohKU6ZEnZ2dbegze8OGDTz77LOo1WoeeeQRLly4UKEBitsNeMQXjbma8EQ7Mh0aQG6qPlknHDN2aFVi6tqTJGfkARCflsvUNSeNHJEQQlSeMiXq+vXrs2rVKuLi4li/fj1PPPEEAElJSTg4OFRogOJ2NWwt6dOqFpnY8IHdZKjZCnKuwsKnIfHBGR2sLHafu8KS/XEAvNvVH4DF++LYdjrZmGEJIUSlKVOinjhxImPHjsXPz4/g4GDDaFcbNmygRYsWFRqgKNmLofpRtVafymJ/h5/BuwVkX4Ffej60fYHnFmj5z8pIAAaE1Gbko/UNjwHeX36UDKkCF0I8hMqUqJ977jliY2M5cOAA69evN8x//PHH+eabbyosOHFn9d3t6NfaB4C3/xdNZt9l4BUE2Sn6ZJ182sgRVrzv/j1DdEoWHg4axnVvBMC47o2oXcOGy2m5fLrm4bxAEUJUb2Ue5tLT05MWLVpw+fJlLl68CEBwcDCNGjWqsODE3X3YozE1nay5eC2H/4ZfhkGrwLMZZCXBLz0g5YyxQ6wwJ+PT+XHreQA+eropDlYWANhYmvN5n0AAFu+LZceZFKPFKIQQlaFMiVqn0zFlyhQcHR3x9fXF19cXJycnPv74Y3Q6XUXHKO7A3sqCac/re4Vbsj+Of2MLYPBq8GgKmYmwoAdcOWfkKMtPq1N4f/lRCnUK3Zp40q2pZ7Hlbeu5MLioCnzc8qNk5hUaI0whhKgUZUrUH3zwATNnzuSzzz7j8OHDHD58mE8//ZTvvvuOCRMmVHSM4i7a1nNheHv98+pxyyO5ptjB4P+BewBkJjwUyfqXXTFEXEzD3sqcj55pUmKZcd0a4VPDmkupOdIKXAjxUClTov7ll1+YO3cur732GoGBgQQGBvL6668zZ84cGebSCN7t6k99dzuSM/L4cNUxFBsX/Z21WyPIuKx/Zp12ydhhlsnFa9lM2xAFwPjujfFwsCqxnK3mRhX473tj2XlWqsCFEA+HMiXqq1evlvgsulGjRly9Kj1FVTUrCzO+7huEmVrFP5HxrI64DHZuMOQvcPWHWm3A7sHr2lVRFD5cdYzsfC3BfjX4vzY+dy3frp4rgx7RV4G/t0yqwIUQD4cyJeqgoCBmzpx52/yZM2cSGBhY7qBE6QXWcmLUo/UBmPi/4ySm5+qT87C10OdnMLMwcoSltzriMluikrE0UzO1TzPUatU913m/eyNqOeurwD9bK1XgQogHX5kS9RdffMG8efMICAhg+PDhDB8+nICAABYsWMC0adMqOkZxn0Y9Vp9mNR1JyyngvWVHURQFbF3ArGiQNJ0O/v0EUuOMG+h9uJqVz0d/6TtveeOx+tRzs7uv9Ww15nxRVAX+255YdkkVuBDiAVemRN2pUydOnz5N7969SU1NJTU1lWeffZbjx4/z66+/VnSM4j5ZmKn5um8QluZqtp5OZtG+2OIFtkyFbV/ouxstzDNOkPfpv/+c4GpWPv4e9rzSqV6p1m1X35UBIbUBeG/5UbKkClwI8QBTKYpSYWMlRkRE0LJlS7RabUVtssJdvHgRHx8f4uLiqFWrlrHDqRRzt5/nv/+cxMbSjLVvdsDXxVa/IO2SvpvRTuMgsK9xg7yL7WeSGfTzPlQqWP5aO1rWdr73SrfIzCuk6zfbuJSaw6BHfPm4V9NKiFQIIcqmNLmozB2eCNP1YmgdQurUIDtfyzt/RKC9Pm61Y014bZdJJ+mc/BvdhA5p61emJA1gpzHni+f0VeC/7rnArnNSBS6EeDBJon4IqdUqpj0fhK2lGQcuXGPu9vM3FpprbnxOj4fF/SEjoeqDvINvNp0m7moO3o5WjC0adKOsQuu78kJRFfg4qQIXQjygJFE/pHxq2DCxZwAAX204zamE9NsLrXoVotbo37POTKriCG937FKa4aLiv72bYqcxL/c2x3dvRE0na+Ku5vDFOukLXAjx4CnVX8Jnn332rstTU1PLE4uoYH1b+7DheCLhp5IYszSCVSNDsTS/6dqsx3RY8BSknNYn6yF/69+/NoJCrY5xy4+iU6BHoBePNfKokO3aW1nweZ9ABv68l192X6BbUy/a1nOpkG0LIURVKNUdtaOj410nX19fBg8eXFmxilJSqVRM7dMMZxsLTsSnMyP8lkE6atTRd4pi7w3Jp/QNzbKuGCXWn3dEc/xyOo7WFkzqWXI3oWXVvoEr/YOvtwKPIDtfqsCFEA+OCm31/SCoDq2+b7UmMp7Xfz+EWgXLSmpFfeUczH9S3ze4R1N98rapUWXxXbiSxRPfbCOvUMcXzwXSt/XdeyAri4zcArp+s43LabkMbefH5Kcr9mJACCFKQ1p9i2KebObFM8290Skw9o8IcvJveX3OpR4M/RvsPCDxmP7OOrtquoJVFIX/rIwkr1BHu3ouPN+qci6e7K0s+KyoI5QFu2LYc944NQdCCFFakqiriSlPN8XDQcP5lCw+L6lRlWsD/Z20rTskRMKvvSDnWqXHtfzQJXaevYLGXM2nvZuhUt27m9Cy6tjQzdBf+HvLjkoVuBDigSCJuppwtLHgi+f0Y1cv2BVT8uhSbv5F1d6uEB8Bv/aGnNRKiyklM4///qPvJvStsIb4udpW2r6u++Cpxng7WhF7NZsv1kVV+v6EEKK8JFFXI50auhm61nz3zwjScwtuL+TeqChZu8Dlw/Dbs5CbVinxTPnrBKnZBQR4OfBShzqVso9b2VtZMPWmKvC9UgUuhDBxkqirmf882RhfFxsup+Xy0eoTJRfyCNCPZ21dAy4dhLXvV3gcm08lsTriMmoVfN4nEAuzqvtV7NTQjX5FDdbeW3709mf2QghhQiRRVzO2GnO+ej5I34/2oYusP36HXsk8m8Lg/4FvKHSZUqExZOYV8kFRN6HD29ehWS3HCt3+/figR2O8HK24cCWbL9dLFbgQwnRJoq6GWvvV4JWO+hGp/rMikpTMO4yk5RUIQ/8p3gmKTlfu/U9bH8XltFxqOVvzdpeG5d5eWThYWTD12WYAzN8Vzb7oqmnlLoQQpSWJupp6u0sDGnnacyUrn/+siOSOr9Pf3Ar70K/6V7fys8q838Ox1/hldwwAn/Zuho1l+bsJLavO/u70bV0LRYH3lpXw2poQQpgASdTVlMbcjK/6BmFhpmLDiURWHLp09xWyr8KGDyBmOxz+vUz7zC/UMX5FJIoCz7aoSceGxumu9GYfPBWAp4MVMVeymbZBqsCFEKZHEnU11sTbkbfC9FXPk1cf51Jqzp0L29SAgSug/dsQ/HKZ9vfTtnOcSsighq0lH/YIKNM2Kpqj9Y0q8Hk7ozkQI1XgQoiS/bE/jl/3XKjy/UqiruZe6ViXFrWdyMgr5L1lEeh0d+lRtlZrCJt8ozq8MB8K7pLcb3IuOZMZ/54FYGKPAGrYWpYz8orzaCN3nmulrwJ/d9lRcgukClwIcUOhVseUv07w3vKjTF59nOOXK+eV1TuRRF3NmZup+bpvc6ws1Ow8e4WFRc+P76kwD/4YDDPbwJbP4Gr0HYvqdArjV0SSX6ijY0M3nmnuXTHBV6AJPQLwcNAQnZLFNGkFLoQokpZTwIu/HGDeTv3fuDceq09jT4cqjUEStaCOqy3/ebIxAJ+tO8X55Mx7r5QcBbG7IS0OtkyFGc1hXjc4uOC23syWHohjX/RVrC3M+KRX00rtJrSsbq4C/3lnNAcvSBW4ENXdueRMes/aybbTyVhbmPH9gJa8FdYQtbpq/4ZJohYADAzxpX19V3ILdIz5I4JC7T1ew/IKhDEnoPdPUPdRQKVP3H+9CdMawp9D4fR6kq5l8OmakwC880RDfGrYVPqxlNVjjTzo07KoCvxPqQIXojrbejqZXrN2cj4lC29HK5a91pYnm3kZJRZJ1AIAtVrFF88FYm9lzpG4VH7Yeu7eK1naQlA/GLxKn7S7TAG3xqDNg+MrYVFfrGY24+3CefTyTGZYO7/KPoxym1hUBX4+JYuvN542djhCiCqmKApzt59n2Px9ZOQW0srXmf+Nak8T76rvmOk6SdTCwNvJmo+KxmmevukMxy6VosGEgzeEvgmv74ZXtsEjr5OnccFBe40XzdcxPfVNzBb2ABMf/tzR5kYV+Jzt5zl4ofJHEBNCmIa8Qi3vLTvKf/85iU6Bvq1rsejlENzsNUaNSxK1KKZ3i5p0a+JJoU7hnT8iyCssZfWvSgVeQaR3nsKjuu8Zlv8uJ2uEgZkG3BrdaDGuKPq77nJ0nlJZHmvkwbMtaxa1Ao+QKnAhqoHkjDwGzNnLnwcvolbpG5h+3icQjbmZsUMzjUQ9a9Ys/Pz8sLKyIiQkhH379t3XekuWLEGlUtGrV6/KDbAaUalUfNK7Ka52lkQlZpS5+veLdae4nKEl2jmUOq/9AWNPQ6dxNwrE7tE/x57RErSmNy70pB5NcLfXcD45i2+kClyIh9rxy2k8M3MHBy5cw97KnPnDghnevo7JNHw1eqJeunQpY8aMYdKkSRw6dIigoCC6du1KUlLSXdeLiYlh7NixdOjQoYoirT5c7DR82ltf/fvTtvPsL2UnIPtjrvLbnlgAPn22GVYWZmDtBPYeNwrlpYNzHagfBmZF3YgqCuycoW9RbmSONhaGczBn+3kOxZpOFXhiei6rIy4TEZdq7FCEeOCtjYznudm7uZyWS11XW1aNDKWTCfSaeDOVcsdOnqtGSEgIbdq0YebMmQDodDp8fHx44403eP/9kodX1Gq1dOzYkRdffJHt27eTmprKqlWr7mt/Fy9exMfHh7i4OGrVqlVRh/FQGvtnBMsOXqR2DRvWvtkBW829++XOK9Ty5LfbOZecRb/WPnz+XOCdCysK5GeCxl7//fIR+KmT/rN3CwjqD037gK1r+Q+mjN5eeoSVhy9Rz82Wf0Z30F90VLG07AJ2n7/CrnMp7Dybwrlk/eMCCzMVPw1qzaON3Ks8JiEedDqdwrfhZ/g2/AwAHRq4MrN/SxxtLKpk/6XJRUa9o87Pz+fgwYOEhYUZ5qnVasLCwti9e/cd15syZQru7u4MHz78nvvIy8sjPT3dMGVkZFRI7NXBxJ4B1HSyJvZqNp8UvWJ1L7M2n+NcchaudhrDu9l3pFLdSNLXv/s/CWpzuHwY1r4HX/nD4v5w4n/6Tlaq2KSeAbjZaziXnMU3m6qmCjwnX8u208lMXXuSnt/toPnHG3j1t4Ms3H2Bc8lZ+mYAjlYUaBVe+e0gO86kVElcQjwssvMLGbnokCFJD29fh/lD21RZki4t4w1dBKSkpKDVavHw8Cg238PDg1OnTpW4zo4dO/j55585cuTIfe1j6tSpfPTRR+UNtVpysLLgy+cCeWHuXhbtjaVLgAeP+t/57u10Ygazt+i7CZ38dEDpf+m9gqD/YshKgWPLIWKxPmFHrdFPVo7Q5Fn9nbZPcPGRvSqJk40ln/ZuxssLDzBn23m6NfGkRW3nCt1HgVZHRFwqO89eYee5FA7HXqNAW7yiq767HaH1XGhX35VH6rhgozHj9d8PsfFEIi8t3M/CF0MIrlOjQuMS4mF0KTWHl385wIn4dCzMVHzSqxl92/gYO6y7MmqiLq2MjAwGDRrEnDlzcHW9v+rQ8ePHM2bMGMP3S5cuERBgGgNCPAja1XdlaDs/FuyKYdyyo2x4uyNONrf3063TKby//CgFWoWwxu48VZ6OAWxdIeQV/ZR0Co4ugaN/QPolODhfPznXgcB+0LgHeDYrxxHeW5cAD3o192bVkcu8u+wof7/RvlxV4DqdwsmEdHad1Vdn74u+StYtQ2x6O1rRrr4rofVdaFfPFQ8Hq9u2M/OFFoxYeJCtp5MZNn8fv70UUuEXEUI8TA7EXOXV3w6SkpmPq50lPwxsRWs/07/ANWqidnV1xczMjMTExGLzExMT8fT0vK38uXPniImJoWfPnoZ5Op2+By1zc3OioqKoV69esXU0Gg0azY134NLT0yvyEKqFcd0ase1MMueTs5j4v+PM6N/itjK/7b3AodhUbC3NmPJMBXYT6t5IPxDIYxMgZgdELNFXg1+Lhq2fgYX1jUSdchbWjNXfmXe5qRZFpwN1+Z7yTOrZhB1nr3A2KZNvw88wrluj+15XURRirmSz61wKu85eYff5K1zNyi9WxtnGgnb1XGlX34XQeq74utjc8xxqzM34cVArhs3fz+7zVxgybx+LXn6EpjWN1zGDEKbqjwNxfLAykgKtQoCXA3OGtKamk7Wxw7ovRk3UlpaWtGrVivDwcMMrVjqdjvDwcEaNGnVb+UaNGhEZGVls3ocffkhGRgbffvstPj6mXX3xoLK2NOPrvs3pM3sXqyMu80QTD3oE3hhY43JqDl+s07fUHte9Ed6V8cuvNoO6nfTTU9Pg5N9weq2+0dl1yafg/GbITS2+7ux2UJgDNepCjXr6ny719J+daoP5vUfycra15JPeTXnl14P8uPUcXZt40tzH6Y7lE9Nzixp/XWHX2RQup+UWW25raUZwnRqE1nelXT1XGnnal6n/YCsLM+YOac2Qefs4cOEag37ey5IRbfH3tL/3yveiLdD/NCt6hJF9FRKPg5kl1A65Ue7yEUABCxv9hdP1n+bW5b5AEqK8CrU6pq49xc879INqPNnMk2nPB2Fj+eBUKBs90jFjxjBkyBBat25NcHAw06dPJysri2HDhgEwePBgatasydSpU7GysqJp06bF1ndycgK4bb6oWM19nBjZuR4z/j3Lh6uOEexXA3cHKxRFYeL/jpGZV0jL2k4MDPGt/GCud10a1K/4fO/m8PRMfZK4TlsIV86CrgCuxcC5f4uvozIDJ5/bE3jNlre1Nu/axJNnmnvzvyOXeffPCP66qQr8Ti2zDSGbqWlR26koMbsQ5OOEhVnFJDFbSzPmDW3FwJ/3c/RiGq/P2cTCZ5yo6VZDX7tw3e5Z+gFT8rP0re3zs0r4nAUFRT+1+fDMLGgxUL/+pYPw+3P6bb6y7cZ2l78EV86UHJy5dfHkff1zq6HQvL++TPpl2DEdbN2g07s31j27CfIybr8AsLAGS3uwcZELgcpWmAdx+/QXwM0H6P9/AMQfhT3fQ4tB4Bdq3BjvIi2ngDcWH2bb6WQA3gprwOjHGlT5oBrlZfRE3a9fP5KTk5k4cSIJCQk0b96cdevWGRqYxcbGopb/jCZh1GMNCD+VxPHL6by/IpKfh7RmTWQCm04mYWGm4rM+gcb9D+BYC1oOKj5PbQZvH4Mr5+DqObh6vujzef1UkK1P4Ndi4Fz4jfX6/AzNntN/vnhQ/5zcJ4TJPZ9m59kUziRlMmHVMWrYWbL73BWOXUrFTNGiIR8NBdRUFdDUw4o2Pta08rIhwF2DhmzwdgGbomdiiScgZrv+rt6/u36eouir7wtz9X8kS/yZr68hKMgxJFeHvgtZ+GI3+s/ZS/3EddRcMZPcWqFYvbTmxjFtmwY5pRwV7Oae46xr6HuXc/YrXsbOQ38eC7L1MRXeVHtQmKOfbt3v9eMFyIiHfT+Co0/xRL15Klw6cOfY1Bbg4AUONfVd2DrUhHqPQr3H9MsVRT/J34/7pyiQdFKfmM9thgs79f+uADau0PZ1/ecjv+sbexbmmmyiPp+cyUsLD3A+OQtrCzO+7htEdyMNqlFeRk/UAKNGjSqxqhtgy5Ytd113wYIFFR+QKJGluZpv+jWnx3c7+PdUEj9tO8+c7frqpNc616ehRwVUt1Y0lQrsPfXTrX9QFAUyEm5J4Of0Y2u7NrxRLm4v7PsJMhNxbvYc/+3VjNd+28/oyD7Yq7J5iwI0lgWoVbd0SZBaNN38tGbQKn0yub7dte9Box43EpdKpR8qVFfK3trys3CyseS34cF88f1BYjI9OH3ZjCapOTeewzV/QZ/sLW3B0q7oZ0nTzctu+jet1QpG7r1938P+Kf5dp7txIXE9ed/60+2mZ/y27tBhLFjeMrKaVxCYa0rYRtEFiq4AUmP103VmljcS9bUYmNlaX1Myav+NMlFr9bUFDjX1k527/oKuuspIhPNb9LVN57dAZkLx5bbuULczeNzUCDewrz5JN+l9Y15yFPz1lv73rEmv4q9eVrFtp5MZuegQGbmFeDtaMWdIa6MOqlFeJpGoxYOjoYc9Y59oyKdrTjF1rf4Vunputox8tN491jRBKlXRHZkX+LW/c7marSD0LfDQD1jSraknbwbb43M0+c7rmGnA3EqfaIr9vKn1tks9/R+6mq2Kr9vpff1d4K3r3bwtM40+sV1PrFZO+k3aaXjn1Vfp+2MQMVey8Zuzh6WvtNW3Gu/6SdnOU2mp1TcS/f1w8oHHJ9w+v8fXd15HWwCZiZB2Sf82QPpl/c+bL8bSL+sveG696Nn6uf61v+tUZmDvpb8rd6xZ/A79+md7z4crmcfugZN/6e+ak44XX2ZuBb6h+gvKuo/qf+9vbdhYs9Xtv7dHFkHsLv209j1o/LQ+aft1qLJaDUVRmLczhk/+OYFOgVa+zvwwsJXRB9UoL6P3TFbVpGey8tPqFPr/tId9RV2L/vlqW9o8AK84VKiCXP1z2ZsT5/VEamZp9OrWy6k59P1xNxev5VDf3Y4lIx7B1e7B/mNVajqtvsYkLx3cb+p85++3IeGYPpFnxINyH4OudHwXHvtQ/zkjQd/VrUs9aHPvTpeMTqeF+AjwaHqj4eSa9/SPGwBQ6ceXr/uoPjn7PAIWt78OeE9pl/SPiI4s0rcLuc7RR9/3QfP++tqNSpJXqGXCqmP8ceAioB/56uNeTU1iUI2SlCYXSaIWZRJ3NZuRiw4R1tiD0Y83MHY4ogRxV7N5/ofdJKTn0sjTniUjHinxHfhqTafV35lfvyMvdodeNC8jHp6cBq31DVyJ2QELntI3Ohx96Ma2fump76zHqbY+OTn5FP0s+m7nXiWd9Nxm1iOQfBKG/A11isZGOL8VIv/UJ+Y6ncHWpeL2pyhwcb/+OfaxFfoLpetqt6uUqvGUzDxe/fUgBy5cQ62CD54K4MVQP5MZVKMkkqjvQhK1qE7OJ2fS98c9pGTmEVjLkd9eCsHByjS7STRZOq1+un43mnwaDv0CGgfofNOIcF/Ug+y7dOdqptE3eHSqXZTEa0PDJ4q3zC+r3DSI3q5vBJZwDIatvVGrs/xl/XP5p6ZB0P+VeRfHLqUxZ/t5Np1IxNnWkjquttR1taWOqy1+rrbUdbWjprM1Zjc3KC3IgVP/6JP2uc1AUbqxsNFXjbd/W99XQjkcv5zGiIUHuZSag72VOTNfaGlyg2qURBL1XUiiFtXN6cQM/u+nPVzNyqeVrzMLXwy+rwFWRCklR0FqHKTF6n+mxkJanP5zRjyGJHWzHtNv3Klf2A0rRkDtR6DPnBtlLh7Ujz7nUPNGlbS2AC4euNE6+9LB4lX4r+640RFQ1hWwcrjxPnwp6HQKm6OSmLP9PHvO3/uNAUszNT41rKnjakddt6Ik7mJLXTdb3JUUVEeXFq8af+lffSNF0F8MlbIdwNrIeMb8EUFOgZa6rrbMHdKaum52pT1MoyhNLpL/rUI85Bp62PPr8GD6/7SHgxeuMfyX/cwfGoy1pWk+u7uXs0kZHLxwjWea1zTKaGZ35Oavn0pSmF9UtV6UuNOKErnXTaPLXYvRJ/nMOsXXXdwPsooaLtp56Bu3pZyF/FsGGHKpf+M5883PgstQrZ1boGXFoUv8vOO8oU8AM7WKHoFeDG7ri1YHMSlZnE/JIjolk5iUbKKvZJFfqONccpZ+nVvG8bGxNMPPJZg6rp0J9Y6mZd4+srV1qJudr38k8/fbkHJG37DQt91d49PpFGb8e4bpm/Tv73ds6MZ3/VvgaP1w1hbJHbUQ1cSRuFQGzt1LZl4hHRq4MndIa5NtaFOSK5l5fLPpNIv3xaHVKXRo4Mqcwa1NK1mXR06q/q5cbQa1WuvnaQvghw6QeuHG+8zXWdco6q2vKDk71S53CFcy8/h1zwV+3X2BK0Xd3NprzOkfUpuh7fzu2uugTqdwOS2H6JSsm5K4/nPctRy0ujunGjdr2MwI7JRMVgT+hKZ+R/xcbajjoGBj41CscWZ2fiFj/4xgTaT+NbLh7eswvnsjzCuoA6GqIlXfdyGJWlRn+2OuMvjnfeQUaAlr7MHsgS0rrIe0ypJXqGXBzhhm/nuWjDz9q1YWZioKtAqPNXLnh4GtsDQ37WMoN0XRd+GaFqtv8OZYEzyDKuztgnPJmczdHs2KQxfJK9SPn1DTyZphoX70a+ODfTnbNeQX6oi7lk1MUfI+X5TAo1OyiC/qXteDq3Qz289CbReUohGYJ5n/QjfzQ+yy68L5ms/gWLMBqw5ffqBGvroTSdR3IYlaVHc7z6YwbMF+8gt1PNXMi2//r7lJ3o0oisLaYwlMXXuSuKs5ADSt6cCEpwLQKgrD5u8nr1BH1yYezHzB9C84TI2iKOyNvsqcbecJP5VkmB9Yy5GXOtTlyaaeVfJ7kZ1fyIUr2URfT+LJWcRcySI6OZMVhaPwU98YtGmvrhFrtcFka9wZ3qUl/nVqg7WzfrKwMU6r+jKSRH0XkqiFgM1RSYxYeIACrcKzLWoy7fkgk+r/OCIulf/+c4L9MdcA8HDQ8F7XRvRuUdMQ57bTybz0ywHytTp6BHrx7f+1KN7iWJSoQKtjTWQ8c7dHE3kpDdDnt8cbefByhzoE16lhMq81paWlc+3wKqxPLMEtaTdqdHcubKbRJ+ynZ0DDrvp58RH6IXI9mt7oWx703fdq7PTlLe2MkuClMZkQ4q4e9Xdn5gstef33Q6w4fAmNhZpPezcz+h/oy6k5fLk+ipWHLwFgZaHmlY71eKVT3dtGO+rY0I3ZA1vy6m8H+ftoPJbmaqY9Z1oXHKYkI7eAJfvimL8z2jCam8ZcTZ9WtRjevg71TLC1tKOjA46dB0Pnwfr32iOW6AcJybl203RV3/ucNk/f/enNLccTImH3TKjfpXii/rmLfjAaALX5jbvyu00+IfrX6oxAErUQ1VTXJp5M79ecN5ccZvG+ODTmZkzqGWCUZJ2VV8iPW8/x0/bz5Bbo75r6tKzFu1398XS8cy9Zjzf24Lv+LRi56DArDl1CY67mk17NJFnf5FJqDgt2RrN4XxyZRc/4Xe0sGfSIHwMfqY3Lg9JjnYM3dBhz+3xF0ff9fj1x3zxojFsjaPeGvkX8ddpCfWcr2gJ9ctcV6lvVZ92lS2DQD9QjiVoIUdV6BnmTV6hj7J8RLNgVg8ZczfvdG1VZstbqFJYfusi09VEkZeQBEOxXgwk9AmhW6/4GUejW1Itv+im8VXTBYWmmZvLTTYxeO2BskRf1HZT8ExlvaHFd392Ol9rXoVcLE3u1rTxUKn01tsbu9kRaq/WNFvTXmZnDO/pxCijI0Sf37Ku33KWXMDkar9GaJGohqrnnWtUir1DLByuP8eO282gszBjTpeG9VyynXedS+O/fJzkRr+9isnYNG/7zZCO6NvEsdZJ9OsibgkIdY5dF8MvuC1iaq/nPk42rXbLW6RT+PaXvoGRv9I0OStrVc+HlDnXp1NBNahtudn18cwdvY0dyV5KohRAMCPElr0DHlL9PMCP8DFYWal7vXP/eK5ZBdEoWn645ycYT+ta89lbmjH6sAYPb+Zbrve4+rWqRr9UxfkUkc7ZHozE3Y2zXO3RA8pDJLdCy/NBFft4RzfmiDkrM1Sp6BnkzvH0dmtZ8cId4FJKohRBFXmxfh7xCHZ+vO8UX66LQmJsxvH2de694n9KyC/g2/AwLd8dQqFMwU6sYEFKbNx9vUGHPSfsH1ya/UMek1ceZufksGnM1bzzEg8akZObx6+4L/LrnAlevd1BiZc4LwbUZGuqHl+OdOygRDw5J1EIIg9c61yO3QMu34Wf4+O8TWFmoGRDiW65tFmh1/LbnAt+GnyE1uwCAR/3d+OCpxtR3r7gRlK4b0s6P/EIdn6w5yVcbT2NpruaVTg/geOl3cTYpk593nGf5oUvk39RByYvt69CvjQ920pf7Q0X+NYUQxbwV1oDcQi0/bj3PByuPoTE347lWpe9zQFEUwk8m8emak5xP0VfH+nvY88FTjelYyaMbvdyxLvlaHV+uj2Lq2lNYmqsZFlpxtQPGEnslmyl/n2DTyRudgATVcuTljnXp1qRqOigRVU8StRCiGJVKxfvdGpFXoGPBrhjeWxaBxlxNz6D7b3Bz4nI6n6w5wc6zVwD960BjuvjTt3WtKksmIx+tT16Blhn/nuWjv05gaV7+2gFj0ekUFu6O4fN1UeQUaFGpIKyxBy93qEsbP+dq12iuupFELYS4jUqlYlLPAPIKtSzeF8dbS49gYaamW1PPu66XlJHLV+tP88fBOBQFLM3VDG9fh9c71yt3f9Fl8XaXhuRpdYbaAQszNX1bP1h9Q0enZDFu2VH2xehbcYfUqcEnvZtR3930OigRlUMStRCiRCqVftCDvAIdKw5f4o3Fh/hpUGsebeR+W9ncAi1zt5/n+y3nyM7Xj4vcI9CLcd0a4VPDpqpDN7i1dmDc8qNozNU807ym0WK6X1qdwvyd0UzbEEVugQ4bSzPGd2/EgBBfecWqmpFELYS4I7VaxRfPBZKn1fHP0Xhe+e0g84e2IbS+K6B/Dr064jKfrz1l6JYyyMeJiT0a08q3hjFDN7heO5Cv1bFobyxj/ojAwkzNk828jB3aHZ1NyuS9ZREcik0FoH19V6Y+28yoFz3CeCRRCyHuytxMzfR+zckv1LHxRCIv/XKAX14MxkwNH/99kiNxqQB4O1oxrnsjegZ6m9wdn0ql4r/PNCW/UMeygxcZvfgwlmZqwgI8jB1aMYVaHXO2R/PNptPkF+qw05jzwVON+b82PvIcuhqT0bOEEPclr1DLiIUH2Xo6GUtzteG1IBtLM17vXI+XOtQ1+W4ptTqFMX8c4X9HLmNppuanwa3o7H97Vb4xRCVk8N6yCCIu6ke06tTQjanPNsPbSd6FfhiVJhdJW34hxH3RmJvx46BWtK3rQn6hDpUK+rX2Ycu7nRn1WAOTT9IAZmoVXz0fRPemnuRrdbzy60F2nU0xakwFWh3fhZ+hx3fbibiYhr2VOV8+F8iCYW0kSQtAqr6FEKVgZWHGz0Nbs/zQJVrVdibA28HYIZWauZmab/+vBQW/H2TTySSGF1XlB9ep+mfqJy6n8+6yCI5f1vd3HtbYnU96N8PD4c4jhonqR+6ohRClYmNpzqBHfB/IJH2dpbmaWQNa0qmhGzkFWobN38eh2GtVtv/8Qh1fbzzN0zN3cPxyOk42Fkzv15w5g1tLkha3kUQthKiWrlflt6vnQla+liHz9hFZ9Hy4MkVeTOPpmTuYEX6GQp1CtyaebHi7I71a1JQGY6JEkqiFENWWlYUZc4e0po2fMxm5hQyat5eTRcNuVrTcAi1frDtFr+93ciohgxq2lsx8oQWzB7bE3V7uosWdSaIWQlRrNpbmzBvahuY+TqRmFzBw7l7OJmVU6D4Ox16jx3c7+H7LObQ6hR6BXmx8uyM9Ar3lLlrckyRqIUS1Z29lwS8vBtO0pgNXsvJ5Yc5eoosGEimP3AItn/xzgj6zd3E2KRNXOw0/DGzFzBdaVtjQnuLhJ4laCCEAR2sLfn0xhEae9iRl5PHCnD3EXc0u8/YOxFzlyW+3M2d7NDoFereoyca3O96zv3QhbiWJWgghijjbWvLbSyHUd7cjPi2X/nP2cCk1p1TbyM4v5KO/jvP8j7s5n5KFh4OGn4e05pt+zXG2taykyMXDTBK1EELcxNVOw6KXQqjjasvFazkMmLOHxPTc+1p397krdJu+nfk7Y1AUeL5VLTa83YnHG5tWV6XiwSKJWgghbuHuYMWil0PwqWFNzJVsXpizh5TMvDuWz8wrZMKqY/Sfs4fYq9l4OVqxYFgbvnw+CEfrqh/eUzxcJFELIUQJvBytWfTSI3g7WnEuOYuBc/dyLSv/tnI7zqTQ9Ztt/LrnAgD9g2uz4e2OJtOHuHjwSaIWQog78Klhw6KXH8HdXsOphAwG/ryXtJwCANJzC3h/+VEG/ryXS6k51HK25veXQpj6bDPsreQuWlQcSdRCCHEXfq62LHr5EVztLDl+OZ3B8/axNjKert9sY8n+OAAGt/Vl/VsdDeN0C1GRJFELIcQ91He347eXQnC2sSAiLpXXfj9EfFouvi42LBnxCFOeaYqtRsY4EpVDErUQQtyHRp4O/Do8BAcrc1QqeDG0Dmvf7MAjdV2MHZp4yMkloBBC3KemNR3Z9E4nsvO0+LnaGjscUU1IohZCiFJwt7cCe2NHIaoTqfoWQgghTJgkaiGEEMKESaIWQgghTJgkaiGEEMKESaIWQgghTFi1a/Wt0+kAiI+PN3IkQgghqqvrOeh6TrqbapeoExMTAQgODjZyJEIIIaq7xMREateufdcyKkVRlCqKxyQUFhZy+PBhPDw8UKvLV/OfkZFBQEAAJ06cwN5eXqy8FzlfpSfnrHTkfJWOnK/SqcjzpdPpSExMpEWLFpib3/2eudol6oqUnp6Oo6MjaWlpODg4GDsckyfnq/TknJWOnK/SkfNVOsY6X9KYTAghhDBhkqiFEEIIEyaJuhw0Gg2TJk1Co9EYO5QHgpyv0pNzVjpyvkpHzlfpGOt8yTNqIYQQwoTJHbUQQghhwiRRCyGEECZMErUQQghhwiRRl8OsWbPw8/PDysqKkJAQ9u3bZ+yQTNa2bdvo2bMn3t7eqFQqVq1aZeyQTNbUqVNp06YN9vb2uLu706tXL6KioowdlsmaPXs2gYGBODg44ODgQNu2bVm7dq2xw3pgfPbZZ6hUKt566y1jh2KyJk+ejEqlKjY1atSoyvYvibqMli5dypgxY5g0aRKHDh0iKCiIrl27kpSUZOzQTFJWVhZBQUHMmjXL2KGYvK1btzJy5Ej27NnDxo0bKSgo4IknniArK8vYoZmkWrVq8dlnn3Hw4EEOHDjAY489xjPPPMPx48eNHZrJ279/Pz/++COBgYHGDsXkNWnShPj4eMO0Y8eOqtu5IsokODhYGTlypOG7VqtVvL29lalTpxoxqgcDoKxcudLYYTwwkpKSFEDZunWrsUN5YDg7Oytz5841dhgmLSMjQ2nQoIGyceNGpVOnTsqbb75p7JBM1qRJk5SgoCCj7V/uqMsgPz+fgwcPEhYWZpinVqsJCwtj9+7dRoxMPIzS0tIAqFGjhpEjMX1arZYlS5aQlZVF27ZtjR2OSRs5ciRPPfVUsb9j4s7OnDmDt7c3devWZcCAAcTGxlbZvqvd6FkVISUlBa1Wi4eHR7H5Hh4enDp1ykhRiYeRTqfjrbfeIjQ0lKZNmxo7HJMVGRlJ27Ztyc3Nxc7OjpUrVxIQEGDssEzWkiVLOHToEPv37zd2KA+EkJAQFixYgL+/P/Hx8Xz00Ud06NCBY8eOVclgJpKohTBhI0eO5NixY1X7POwB5O/vz5EjR0hLS2PZsmUMGTKErVu3SrIuQVxcHG+++SYbN27EysrK2OE8ELp37274HBgYSEhICL6+vvzxxx8MHz680vcviboMXF1dMTMzM4xtfV1iYiKenp5Giko8bEaNGsXff//Ntm3bqFWrlrHDMWmWlpbUr18fgFatWrF//36+/fZbfvzxRyNHZnoOHjxIUlISLVu2NMzTarVs27aNmTNnkpeXh5mZmREjNH1OTk40bNiQs2fPVsn+5Bl1GVhaWtKqVSvCw8MN83Q6HeHh4fJcTJSboiiMGjWKlStX8u+//1KnTh1jh/TA0el05OXlGTsMk/T4448TGRnJkSNHDFPr1q0ZMGAAR44ckSR9HzIzMzl37hxeXl5Vsj+5oy6jMWPGMGTIEFq3bk1wcDDTp08nKyuLYcOGGTs0k5SZmVns6jM6OpojR45Qo0YNateubcTITM/IkSNZtGgR//vf/7C3tychIQEAR0dHrK2tjRyd6Rk/fjzdu3endu3aZGRksGjRIrZs2cL69euNHZpJsre3v629g62tLS4uLtIO4g7Gjh1Lz5498fX15fLly0yaNAkzMzP69+9fJfuXRF1G/fr1Izk5mYkTJ5KQkEDz5s1Zt27dbQ3MhN6BAwd49NFHDd/HjBkDwJAhQ1iwYIGRojJNs2fPBqBz587F5s+fP5+hQ4dWfUAmLikpicGDBxMfH4+joyOBgYGsX7+eLl26GDs08ZC4ePEi/fv358qVK7i5udG+fXv27NmDm5tblexfRs8SQgghTJg8oxZCCCFMmCRqIYQQwoRJohZCCCFMmCRqIYQQwoRJohZCCCFMmCRqIYQQwoRJohZCCCFMmCRqIYQQwoRJohZCVBqVSsWqVauMHYYQDzRJ1EI8pIYOHYpKpbpt6tatm7FDE0KUgvT1LcRDrFu3bsyfP7/YPI1GY6RohBBlIXfUQjzENBoNnp6exSZnZ2dAXy09e/ZsunfvjrW1NXXr1mXZsmXF1o+MjOSxxx7D2toaFxcXRowYQWZmZrEy8+bNo0mTJmg0Gry8vBg1alSx5SkpKfTu3RsbGxsaNGjA6tWrDcuuXbvGgAEDcHNzw9ramgYNGtx2YSFEdSeJWohqbMKECfTp04eIiAgGDBjA//3f/3Hy5EkAsrKy6Nq1K87Ozuzfv58///yTTZs2FUvEs2fPZuTIkYwYMYLIyEhWr15N/fr1i+3jo48+om/fvhw9epQnn3ySAQMGcPXqVcP+T5w4wdq1azl58iSzZ8/G1dW16k6AEA8CRQjxUBoyZIhiZmam2NraFps++eQTRVEUBVBeffXVYuuEhIQor732mqIoivLTTz8pzs7OSmZmpmH5P//8o6jVaiUhIUFRFEXx9vZWPvjggzvGACgffvih4XtmZqYCKGvXrlUURVF69uypDBs2rGIOWIiHlDyjFuIh9uijjxrGt76uRo0ahs9t27Yttqxt27YcOXIEgJMnTxIUFIStra1heWhoKDqdjqioKFQqFZcvX+bxxx+/awyBgYGGz7a2tjg4OJCUlATAa6+9Rp8+fTh06BBPPPEEvXr1ol27dmU6ViEeVpKohXiI2dra3lYVXVGsra3vq5yFhUWx7yqVCp1OB0D37t25cOECa9asYePGjTz++OOMHDmSadOmVXi8Qjyo5Bm1ENXYnj17bvveuHFjABo3bkxERARZWVmG5Tt37kStVuPv74+9vT1+fn6Eh4eXKwY3NzeGDBnCb7/9xvTp0/npp5/KtT0hHjZyRy3EQywvL4+EhIRi88zNzQ0Ntv78809at25N+/bt+f3339m3bx8///wzAAMGDGDSpEkMGTKEyZMnk5yczBtvvMGgQYPw8PAAYPLkybz66qu4u7vTvXt3MjIy2LlzJ2+88cZ9xTdx4kRatWpFkyZNyMvL4++//zZcKAgh9CRRC/EQW7duHV5eXsXm+fv7c+rUKUDfInvJkiW8/vrreHl5sXjxYgICAgCwsbFh/fr1vPnmm7Rp0wYbGxv69OnD119/bdjWkCFDyM3N5ZtvvmHs2LG4urry3HPP3Xd8lpaWjB8/npiYGKytrenQoQNLliypgCMX4uGhUhRFMXYQQoiqp1KpWLlyJb169TJ2KEKIu5Bn1EIIIYQJk0QthBBCmDB5Ri1ENSVPvYR4MMgdtRBCCGHCJFELIYQQJkwStRBCCGHCJFELIYQQJkwStRBCCGHCJFELIYQQJkwStRBCCGHCJFELIYQQJkwStRBCCGHC/h9TeKfTy62uPgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n",
    "examples_seen_tensor = torch.linspace(0, examples_seen, len(train_losses))\n",
    "\n",
    "plot_values(epochs_tensor, examples_seen_tensor, train_losses, val_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbd28174-1836-44ba-b6c0-7e0be774fadc",
   "metadata": {},
   "source": [
    "- Above, based on the downward slope, we see that the model learns well\n",
    "- Furthermore, the fact that the training and validation loss are very close indicates that the model does not tend to overfit the training data\n",
    "- Similarly, we can plot the accuracy below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "yz8BIsaF0TUo",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 307
    },
    "id": "yz8BIsaF0TUo",
    "outputId": "3a7ed967-1f2a-4c6d-f4a3-0cc8cc9d6c5f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfIAAAEiCAYAAAACr1D/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABxp0lEQVR4nO3deXwM9//A8dfm2NynRC4RV9wRhMRdRQVtilKKEkf50lCKXx1FHG3Taqt6aquOtupsRVtUSxytmxChInWHSEQcISHX7vz+2Fq2CRKS7Ib38/HYRzMzn5l5zyT13pnPpVIURUEIIYQQ5ZKZsQMQQgghxMOTRC6EEEKUY5LIhRBCiHJMErkQQghRjkkiF0IIIcoxSeRCCCFEOSaJXAghhCjHJJELIYQQ5ZgkciGEEKIck0QuhCg1bdu2ZcyYMcYOQ4jHmiRyIUzYwIEDUalUBT6dOnUydmhCCBNhYewAhBD316lTJxYtWmSwzsrKykjRCCFMjTyRC2HirKys8PT0NPi4uLgAsHXrVtRqNX/99Ze+/OzZs6lYsSIXL14EYMOGDbRq1QpnZ2cqVKjAc889x8mTJ/Xlz5w5g0qlYuXKlbRu3RobGxuaNm3KP//8w759+2jSpAn29vZ07tyZS5cu6fcbOHAg3bp1Y8aMGbi7u+Po6Mjw4cPJzc2957Xk5OQwfvx4fHx8sLOzIyQkhK1bt+q3nz17lrCwMFxcXLCzs6NevXqsX7/+nsf74osv8Pf3x9raGg8PD3r27KnfptVqiYqKomrVqtjY2BAYGMiPP/5osP+RI0fo3Lkz9vb2eHh40L9/f9LT0/Xb27Zty2uvvcYbb7yBq6srnp6eTJ8+/Z7xCGEMksiFKMdu10H379+fjIwMDh48yNSpU/nmm2/w8PAAICsri7Fjx7J//35iYmIwMzOje/fuaLVag2NFRkYyZcoUDhw4gIWFBX379uWNN97g448/5q+//uLEiRNMmzbNYJ+YmBgSEhLYunUry5YtY/Xq1cyYMeOe8Y4cOZJdu3axfPly4uPjefHFF+nUqRPHjx8HICIigpycHP78808OHz7Me++9h729faHH2r9/P6+99hozZ84kMTGRDRs20KZNG/32qKgovvvuO7788kv+/vtvXn/9dV5++WW2bdsGwLVr12jXrh2NGjVi//79bNiwgYsXL9KrVy+D83z77bfY2dmxZ88eZs+ezcyZM9m4cWMRf0NClAFFCGGywsPDFXNzc8XOzs7g8/bbb+vL5OTkKA0bNlR69eql1K1bVxk6dOh9j3np0iUFUA4fPqwoiqKcPn1aAZRvvvlGX2bZsmUKoMTExOjXRUVFKbVq1TKIzdXVVcnKytKvmzdvnmJvb69oNBpFURTlqaeeUkaPHq0oiqKcPXtWMTc3V5KTkw3iad++vTJp0iRFURQlICBAmT59epHuzU8//aQ4Ojoq169fL7AtOztbsbW1VXbu3GmwfsiQIUqfPn0URVGUWbNmKR07djTYfu7cOQVQEhMT9fG3atXKoEzTpk2VCRMmFClGIcqC1JELYeKefvpp5s2bZ7DO1dVV/7NareaHH36gQYMG+Pn58dFHHxmUPX78ONOmTWPPnj2kp6frn8STkpKoX7++vlyDBg30P99+mg8ICDBYl5aWZnDswMBAbG1t9cvNmzcnMzOTc+fO4efnZ1D28OHDaDQaatasabA+JyeHChUqAPDaa68xYsQI/vjjDzp06ECPHj0M4rrbM888g5+fH9WqVaNTp0506tSJ7t27Y2try4kTJ7h58ybPPPOMwT65ubk0atQIgEOHDrFly5ZCn/hPnjypj/O/5/fy8ipwH4QwJknkQpg4Ozs7atSocd8yO3fuBODKlStcuXIFOzs7/bawsDD8/PyYP38+3t7eaLVa6tevX6Au29LSUv+zSqUqdN1/X8cXR2ZmJubm5sTGxmJubm6w7XYyfeWVVwgNDWXdunX88ccfREVF8eGHHzJq1KgCx3NwcODAgQNs3bqVP/74g2nTpjF9+nT27dtHZmYmAOvWrcPHx8dgv9sNBTMzMwkLC+O9994rcGwvLy/9z3ffA3j0+yBESZNELkQ5d/LkSV5//XXmz5/PihUrCA8PZ9OmTZiZmXH58mUSExOZP38+rVu3BmD79u0ldu5Dhw5x69YtbGxsANi9ezf29vb4+voWKNuoUSM0Gg1paWn6WArj6+vL8OHDGT58OJMmTWL+/PmFJnIACwsLOnToQIcOHYiMjMTZ2ZnNmzfzzDPPYGVlRVJSEk899VSh+zZu3JiffvqJKlWqYGEh/xSK8kv+eoUwcTk5OaSmphqss7CwwM3NDY1Gw8svv0xoaCiDBg2iU6dOBAQE8OGHH/J///d/uLi4UKFCBb7++mu8vLxISkpi4sSJJRZbbm4uQ4YMYcqUKZw5c4bIyEhGjhyJmVnBdrQ1a9akX79+DBgwgA8//JBGjRpx6dIlYmJiaNCgAc8++yxjxoyhc+fO1KxZk6tXr7Jlyxbq1KlT6LnXrl3LqVOnaNOmDS4uLqxfvx6tVkutWrVwcHBg/PjxvP7662i1Wlq1akVGRgY7duzA0dGR8PBwIiIimD9/Pn369NG3Sj9x4gTLly/nm2++KfDWQAhTJYlcCBO3YcMGg1e9ALVq1eLYsWO8/fbbnD17lrVr1wK6V8Jff/01ffr0oWPHjgQGBrJ8+XJee+016tevT61atfjkk09o27ZticTWvn17/P39adOmDTk5OfTp0+e+3bMWLVrEW2+9xbhx40hOTsbNzY1mzZrx3HPPAaDRaIiIiOD8+fM4OjrSqVOnAnX+tzk7O7N69WqmT59OdnY2/v7+LFu2jHr16gEwa9Ys3N3diYqK4tSpUzg7O9O4cWMmT54MgLe3Nzt27GDChAl07NiRnJwc/Pz86NSpU6FfRIQwVSpFURRjByGEKH8GDhzItWvXWLNmjbFDEeKJJl87hRBCiHJMErkQQghRjsmrdSGEEKIckydyIYQQohyTRC6EEEKUY5LIhRBCiHJMErmRff7551SpUgVra2tCQkLYu3evsUMqcX/++SdhYWF4e3ujUqkKdFdSFIVp06bh5eWFjY0NHTp00M+GdduVK1fo168fjo6OODs7M2TIEP0wnLfFx8fTunVrrK2t8fX1Zfbs2aV9aY8sKiqKpk2b4uDgQMWKFenWrRuJiYkGZbKzs4mIiKBChQrY29vTo0cP/RSltyUlJfHss89ia2tLxYoV+b//+z/y8/MNymzdupXGjRtjZWVFjRo1WLx4cWlf3iObN28eDRo0wNHREUdHR5o3b85vv/2m3/4k35v/evfdd1GpVIwZM0a/7km+P9OnT0elUhl8ateurd/+WN0bo07Z8oRbvny5olarlYULFyp///23MnToUMXZ2Vm5ePGisUMrUevXr1fefPNNZfXq1QqgREdHG2x/9913FScnJ2XNmjXKoUOHlOeff16pWrWqcuvWLX2ZTp06KYGBgcru3buVv/76S6lRo4Z+FitFUZSMjAzFw8ND6devn3LkyBFl2bJlio2NjfLVV1+V1WU+lNDQUGXRokXKkSNHlLi4OKVLly5K5cqVlczMTH2Z4cOHK76+vkpMTIyyf/9+pVmzZkqLFi302/Pz85X69esrHTp0UA4ePKisX79ecXNz088opiiKcurUKcXW1lYZO3ascvToUeXTTz9VzM3NlQ0bNpTp9RbXL7/8oqxbt075559/lMTERGXy5MmKpaWlcuTIEUVRnux7c7e9e/cqVapUURo0aKCfbU5Rnuz7ExkZqdSrV09JSUnRfy5duqTf/jjdG0nkRhQcHKxERETolzUajeLt7a1ERUUZMarS9d9ErtVqFU9PT+X999/Xr7t27ZpiZWWlLFu2TFEURTl69KgCKPv27dOX+e233xSVSqWfEvOLL75QXFxclJycHH2ZCRMmGEy7WR6kpaUpgLJt2zZFUXT3wtLSUlm1apW+TEJCggIou3btUhRF90XJzMxMSU1N1ZeZN2+e4ujoqL8fb7zxhlKvXj2Dc/Xu3VsJDQ0t7UsqcS4uLso333wj9+ZfN27cUPz9/ZWNGzcaTBv7pN+fyMhIJTAwsNBtj9u9kVfrRpKbm0tsbCwdOnTQrzMzM6NDhw7s2rXLiJGVrdOnT5OammpwH5ycnAgJCdHfh127duHs7EyTJk30ZTp06ICZmRl79uzRl2nTpg1qtVpfJjQ0lMTERK5evVpGV/PoMjIygDvTlMbGxpKXl2dwf2rXrk3lypUN7k9AQIB+6lHQXfv169f5+++/9WXuPsbtMuXpb02j0bB8+XKysrJo3ry53Jt/RURE8Oyzzxa4Brk/uil8vb29qVatGv369SMpKQl4/O6NJHIjSU9PR6PRGPyRgG7O5/9OkPE4u32t97sPqampVKxY0WC7hYUFrq6uBmUKO8bd5zB1Wq2WMWPG0LJlS/084ampqajVapydnQ3K/vf+POja71Xm+vXr3Lp1qzQup8QcPnwYe3t7rKysGD58ONHR0dStW1fuDbB8+XIOHDhAVFRUgW1P+v0JCQlh8eLFbNiwgXnz5nH69Glat27NjRs3Hrt7I5OmCGEiIiIiOHLkSIlOM/o4qFWrFnFxcWRkZPDjjz8SHh7Otm3bjB2W0Z07d47Ro0ezceNGrK2tjR2OyencubP+5wYNGhASEoKfnx8rV67UT7v7uJAnciNxc3PD3Ny8QCvJixcv4unpaaSoyt7ta73fffD09CQtLc1ge35+PleuXDEoU9gx7j6HKRs5ciRr165ly5YtVKpUSb/e09OT3Nxcrl27ZlD+v/fnQdd+rzKOjo4m/4+aWq2mRo0aBAUFERUVRWBgIB9//PETf29iY2NJS0ujcePGWFhYYGFhwbZt2/jkk0+wsLDAw8Pjib4//+Xs7EzNmjU5ceLEY/e3I4ncSNRqNUFBQcTExOjXabVaYmJiaN68uREjK1tVq1bF09PT4D5cv36dPXv26O9D8+bNuXbtGrGxsfoymzdvRqvVEhISoi/z559/kpeXpy+zceNGatWqhYuLSxldTfEpisLIkSOJjo5m8+bNVK1a1WB7UFAQlpaWBvcnMTGRpKQkg/tz+PBhgy87GzduxNHRkbp16+rL3H2M22XK49+aVqslJyfnib837du35/Dhw8TFxek/TZo0oV+/fvqfn+T781+ZmZmcPHkSLy+vx+9vp0yb1gkDy5cvV6ysrJTFixcrR48eVYYNG6Y4OzsbtJJ8HNy4cUM5ePCgcvDgQQVQ5syZoxw8eFA5e/asoii67mfOzs7Kzz//rMTHxytdu3YttPtZo0aNlD179ijbt29X/P39DbqfXbt2TfHw8FD69++vHDlyRFm+fLlia2tr8t3PRowYoTg5OSlbt2416CZz8+ZNfZnhw4crlStXVjZv3qzs379fad68udK8eXP99tvdZDp27KjExcUpGzZsUNzd3QvtJvN///d/SkJCgvL555+Xiy5EEydOVLZt26acPn1aiY+PVyZOnKioVCrljz/+UBTlyb43hbm71bqiPNn3Z9y4ccrWrVuV06dPKzt27FA6dOiguLm5KWlpaYqiPF73RhK5kX366adK5cqVFbVarQQHByu7d+82dkglbsuWLQpQ4BMeHq4oiq4L2tSpUxUPDw/FyspKad++vZKYmGhwjMuXLyt9+vRR7O3tFUdHR2XQoEHKjRs3DMocOnRIadWqlWJlZaX4+Pgo7777blld4kMr7L4AyqJFi/Rlbt26pbz66quKi4uLYmtrq3Tv3l1JSUkxOM6ZM2eUzp07KzY2Noqbm5sybtw4JS8vz6DMli1blIYNGypqtVqpVq2awTlM1eDBgxU/Pz9FrVYr7u7uSvv27fVJXFGe7HtTmP8m8if5/vTu3Vvx8vJS1Gq14uPjo/Tu3Vs5ceKEfvvjdG9k9jMhhBCiHJM6ciGEEKIck0QuhBBClGOSyIUQQohyTBK5EEIIUY5JIhdCCCHKMUnkQgghRDkmidzIcnJymD59Ojk5OcYOxSTJ/bk/uT/3Jvfm/uT+3F95uj/Sj9zIrl+/jpOTExkZGTg6Oho7HJMj9+f+5P7cm9yb+5P7c3/l6f7IE7kQQghRjkkiF0IIIcoxmY/8IeXn53Pw4EE8PDwwM3v470M3btwAIDk5mevXr5dUeI8NuT/3J/fn3uTe3J/cn/szhfuj1Wq5ePEijRo1wsLi3ula6sgf0r59+wgODjZ2GEIIIR5ze/fupWnTpvfcLk/kD8nDwwPQ3WAvLy8jRyOEEOJxk5KSQnBwsD7f3Isk8od0+3W6l5cXlSpVMnI0QgghHlcPqr6Vxm5CCCFEOWb0RP75559TpUoVrK2tCQkJYe/evfcsm5eXx8yZM6levTrW1tYEBgayYcMGgzJRUVE0bdoUBwcHKlasSLdu3UhMTDQo07ZtW1QqlcFn+PDhpXJ9QgghRGkyaiJfsWIFY8eOJTIykgMHDhAYGEhoaChpaWmFlp8yZQpfffUVn376KUePHmX48OF0796dgwcP6sts27aNiIgIdu/ezcaNG8nLy6Njx45kZWUZHGvo0KGkpKToP7Nnzy7VaxVCCCFKg1FbrYeEhNC0aVM+++wzQNfU3tfXl1GjRjFx4sQC5b29vXnzzTeJiIjQr+vRowc2NjYsWbKk0HNcunSJihUrsm3bNtq0aQPonsgbNmzI3LlzHzr28+fP4+vry7lz5+5bR67RaMjLy3vo8whhiiwtLTE3Nzd2GEI81oqaZ4zW2C03N5fY2FgmTZqkX2dmZkaHDh3YtWtXofvk5ORgbW1tsM7Gxobt27ff8zwZGRkAuLq6Gqz/4YcfWLJkCZ6enoSFhTF16lRsbW0f9nIKUBSF1NRUrl27VmLHFMKUODs74+npiUqlMnYo4h7OXbmJi50aeytp1/w4M9pvNz09HY1GU6BZvYeHB8eOHSt0n9DQUObMmUObNm2oXr06MTExrF69Go1GU2h5rVbLmDFjaNmyJfXr19ev79u3L35+fnh7exMfH8+ECRNITExk9erV94w3JyfHYPD824MF3MvtJF6xYkVsbW3lHzvx2FAUhZs3b+qrwKT7pWnaeTKdAQv24mqn5qcRLfB1LbkHFWFaytXXtI8//pihQ4dSu3ZtVCoV1atXZ9CgQSxcuLDQ8hERERw5cqTAE/uwYcP0PwcEBODl5UX79u05efIk1atXL/RYUVFRzJgxo0hxajQafRKvUKFCEa9OiPLDxsYGgLS0NCpWrCiv2U3M9ew8/m9VPPlahbQbOby8YA8/Dm+Bu4OVsUMTpcBojd3c3NwwNzfn4sWLBusvXryIp6dnofu4u7uzZs0asrKyOHv2LMeOHcPe3p5q1aoVKDty5EjWrl3Lli1bHtjPOyQkBIATJ07cs8ykSZPIyMjQf44ePXrPsrfrxEvyVb0Qpub237e0ATE9s349SvK1W/i62uDrasPZyzcJX7iX69nyu3ocGS2Rq9VqgoKCiImJ0a/TarXExMTQvHnz++5rbW2Nj48P+fn5/PTTT3Tt2lW/TVEURo4cSXR0NJs3b6Zq1aoPjCUuLg64/ytCKysrHB0d9R8HB4cHHldep4vHmfx9m6Y//k5lVex5VCqY06sh3w8Owc1ezdGU6wz9dj/ZeYVXRYryy6jdz8aOHcv8+fP59ttvSUhIYMSIEWRlZTFo0CAABgwYYNAYbs+ePaxevZpTp07x119/0alTJ7RaLW+88Ya+TEREBEuWLGHp0qU4ODiQmppKamoqt27dAuDkyZPMmjWL2NhYzpw5wy+//MKAAQNo06YNDRo0KNsbIIQQJSg9M4dJqw8DMKxNNZpWcaWKmx2LBwVjb2XBntNXeG3ZQfI1WiNHKkqSURN57969+eCDD5g2bRoNGzYkLi6ODRs26BvAJSUlkZKSoi+fnZ3NlClTqFu3Lt27d8fHx4ft27fj7OysLzNv3jwyMjJo27YtXl5e+s+KFSsA3ZuATZs20bFjR2rXrs24cePo0aMHv/76a5le+5OiSpUqxermt3XrVlQqlbT2F6KYFEXhzejDXM7KpZaHA2OfqanfVt/HifkDmqA2N+OPoxeZsuYIMl/W40NmP3tI9+vfl52dzenTp6latWqB7nKm6kGvSSMjI5k+fXqxj3vp0iXs7OyK3F4gNzeXK1eu4OHhIa9uTVx5/Dt/nP0Ue55xqw5haa5iTURL6nk7FSiz4Ugqr/4Qi1aBV9tW541OtY0QqSgqk+9HLkzL3W8+VqxYwbRp0wyGtrW3t9f/rCgKGo3mvvPj3ubu7l6sONRq9T0bOz7ucnNzUavVxg5DlEPJ124x/Ze/ARjToWahSRygU31P3ukewMTVh/li60lc7dS80rpgY2FRvhh9rHVhGjw9PfUfJycnVCqVfvnYsWM4ODjw22+/ERQUhJWVFdu3b+fkyZN07doVDw8P7O3tadq0KZs2bTI47n9fratUKr755hu6d++Ora0t/v7+/PLLL/rt/321vnjxYpydnfn999+pU6cO9vb2dOrUyeCLR35+Pq+99hrOzs5UqFCBCRMmEB4eTrdu3e55vZcvX6ZPnz74+Phga2tLQEAAy5YtMyij1WqZPXs2NWrUwMrKisqVK/P222/rt58/f54+ffrg6uqKnZ0dTZo0Yc+ePQAMHDiwwPnHjBlD27Zt9ctt27Zl5MiRjBkzBjc3N0JDQwGYM2cOAQEB2NnZ4evry6uvvkpmZqbBsXbs2EHbtm2xtbXFxcWF0NBQrl69ynfffUeFChUMxjwA6NatG/3797/n/RDll1ar8H+rDnEjJ59GlZ35X5v7J+aXgivzf6G1AHhrXQKrD5wvizBFKZJEXgYUReFmbr5RPiVZczJx4kTeffddEhISaNCgAZmZmXTp0oWYmBgOHjxIp06dCAsLIykp6b7HmTFjBr169SI+Pp4uXbrQr18/rly5cs/yN2/e5IMPPuD777/nzz//JCkpifHjx+u3v/fee/zwww8sWrSIHTt2cP36ddasWXPfGLKzswkKCmLdunUcOXKEYcOG0b9/f4NJeyZNmsS7777L1KlTOXr0KEuXLtW338jMzOSpp54iOTmZX375hUOHDvHGG2+g1RavEdG3336LWq1mx44dfPnll4BuhMNPPvmEv//+m2+//ZbNmzcbNOiMi4ujffv21K1bl127drF9+3bCwsLQaDS8+OKLaDQagy9HaWlprFu3jsGDBxcrNlE+fLfrDDtPXsbG0pw5vRpiYf7gf9ZfbVudIa10PXr+78d4Nh+7+IA9hCmTV+tl4FaehrrTfjfKuY/ODMVWXTK/5pkzZ/LMM8/ol11dXQkMDNQvz5o1i+joaH755RdGjhx5z+MMHDiQPn36APDOO+/wySefsHfvXjp16lRo+by8PL788kv9YD0jR45k5syZ+u2ffvopkyZNonv37gB89tlnrF+//r7X4uPjY/BlYNSoUfz++++sXLmS4OBgbty4wccff8xnn31GeHg4ANWrV6dVq1YALF26lEuXLrFv3z798L81atS47zkL4+/vX2DCnjFjxuh/rlKlCm+99RbDhw/niy++AGD27Nk0adJEvwxQr149/c99+/Zl0aJFvPjiiwAsWbKEypUrG7wNEI+HE2mZRP2mGwlzcpfaVHWzK9J+KpWKN7vU4UpWLtEHk3n1hwMsGRJCkyquD95ZmBx5IhdF1qRJE4PlzMxMxo8fT506dXB2dsbe3p6EhIQHPpHf3c3Pzs4OR0fHe854B7qBR+4ecc/Ly0tfPiMjg4sXLxIcHKzfbm5uTlBQ0H1j0Gg0zJo1i4CAAFxdXbG3t+f333/Xx56QkEBOTg7t27cvdP+4uDgaNWpUYAz/4ioszk2bNtG+fXt8fHxwcHCgf//+XL58mZs3b+rPfa+4QDez3x9//EFycjKgq54YOHCgNB58zORrtIxbGUdOvpbW/m683MyvWPubmamY3bMBT9dyJztPy+DF+0hMvf/Q08I0yRN5GbCxNOfozFCjnbuk2NkZftsfP348Gzdu5IMPPqBGjRrY2NjQs2dPcnNz73scS0tLg2WVSnXfV9KFlX/UKoP333+fjz/+mLlz5+rro8eMGaOP/fYQpPfyoO1mZmYFYixsBLT/3tMzZ87w3HPPMWLECN5++21cXV3Zvn07Q4YMITc3F1tb2weeu1GjRgQGBvLdd9/RsWNH/v77b9atW3fffUT588XWkxw6n4GjtQWzezZ4qC9qluZmfNEviJcX7CH27FUGLNQN5Srjspcv8kReBlQqFbZqC6N8SvMpbMeOHQwcOJDu3bsTEBCAp6cnZ86cKbXzFcbJyQkPDw/27dunX6fRaDhw4MB999uxYwddu3bl5ZdfJjAwkGrVqvHPP//ot/v7+2NjY2Mw8uDdGjRoQFxc3D3r9t3d3Q0a5MGdEQTvJzY2Fq1Wy4cffkizZs2oWbMmFy5cKHDue8V12yuvvMLixYtZtGgRHTp0wNfX94HnFuXH4fMZfBJzHIBZ3erj5XT/L3f3Y6M2Z2F4U2p5OHDxeg4DFu4lPTPnwTsKkyGJXDw0f39/Vq9eTVxcHIcOHaJv377FbuxVEkaNGkVUVBQ///wziYmJjB49mqtXr973S4y/vz8bN25k586dJCQk8L///c9g3H9ra2smTJjAG2+8wXfffcfJkyfZvXs3CxYsAKBPnz54enrSrVs3duzYwalTp/jpp5/0U/C2a9eO/fv3891333H8+HEiIyM5cuTIA6+lRo0a5OXl8emnn3Lq1Cm+//57fSO42yZNmsS+fft49dVXiY+P59ixY8ybN4/09HR9mb59+3L+/Hnmz58vjdweM9l5Gl5fGUe+VqFLgCfPB3o/8jGdbC35dnAwPs42nE7PYuCivdyQcdnLDUnk4qHNmTMHFxcXWrRoQVhYGKGhoTRu3LjM45gwYQJ9+vRhwIABNG/eHHt7e0JDQ+87SMmUKVNo3LgxoaGhtG3bVp+U7zZ16lTGjRvHtGnTqFOnDr1799bXzavVav744w8qVqxIly5dCAgI4N1339XPAhYaGsrUqVN54403aNq0KTdu3GDAgAEPvJbAwEDmzJnDe++9R/369fnhhx+IiooyKFOzZk3++OMPDh06RHBwMM2bN+fnn3826Nfv5OREjx49sLe3v283PFH+fPB7IifSMnGzt+KtbgEl9tbN08ma74cEU8FOzZHk6wz7LlbGZS8nZGS3h/S4jez2ONFqtdSpU4devXoxa9YsY4djNO3bt6devXp88sknpXJ8+Tsve7tPXabP/N0oCiwIb0L7Oh4lfo7D5zPoM383mTn5dKrnyef9GmNuJg0ljaGoI7vJE7ko986ePcv8+fP5559/OHz4MCNGjOD06dP07dvX2KEZxdWrV4mOjmbr1q1EREQYOxxRQm5k5zF+1SEUBV5q6lsqSRwgoJITX/cPQm1uxoa/U2Vc9nJAErko98zMzFi8eDFNmzalZcuWHD58mE2bNlGnTh1jh2YUjRo1YuDAgbz33nvUqlXL2OGIEvLW2gTOX71FJRcbpjxXt1TP1aKGGx+/1BAzFSzbm8SHf/zz4J2E0Uj3M1Hu+fr6smPHDmOHYTLKuueAKH2bjl5kxf5zqFTw4YuB2FuV/j/dnQO8eKtbAJOjD/PZlhO42qkZ/O9ocMK0yBO5EEKYsMuZOUxcHQ/A0NbVCKlWoczO3TekMuM76qZDnbn2KGsOJpfZuUXRSSIXQggTpSgKU9YcIT0zl5oe9gZzjJeViKdrMKhlFQDGrzrElsR7j8IojEMSuRBCmKif4y7w25FULMxUzOnVEOsSHKmxqFQqFVOfrUu3ht7kaxVGLIkl9uzVMo9D3JskciGEMEEpGbeY+rNuEKHR7f2p71P4HONlwcxMxfsvBtL2rnHZ/7ko47KbCknkQghhYnRzjMdzIzufQF9nRrSt/uCdSpluXPbGNKrsTMatPAYs2Mv5qzeNHZZAErkQQpicJXvOsv1EOtaWZszpFVikOcbLgq3agkUDm+Jf0Z7U69kMWLCXyzIuu9GZxl+HeGy0bdu2wHzac+fOve8+KpWKNWvWPPK5S+o4QhjTqUuZvLM+AYBJnetQ3d3eyBEZcrZV890Q3bjsp9KzGLR4H5k5+cYO64kmiVwAEBYWRqdOnQrd9tdff6FSqYiPjy/2cfft28ewYcMeNTwD06dPp2HDhgXWp6Sk0Llz5xI9lxBlKV+jZezKQ2TnaWlVw43+xZxjvKx4Odnw3ZBgXO3UxJ/P4H/f7ycnX8ZlNxZJ5AKAIUOGsHHjRs6fP19g26JFi2jSpAkNGjQo9nHd3d2xtS2buY09PT2xsrIqk3OZkgfN/y7Kjy+3nSTu3DUc/p1j3MyExziv7m7P4kFNsVObs+PEZV5fEYdGK0O5GoMkcgHAc889h7u7O4sXLzZYn5mZyapVqxgyZAiXL1+mT58++Pj4YGtrS0BAAMuWLbvvcf/7av348eO0adMGa2tr6taty8aNGwvsM2HCBGrWrImtrS3VqlVj6tSp5OXpplRcvHgxM2bM4NChQ6hUKlQqlT7m/75aP3z4MO3atcPGxoYKFSowbNgwMjMz9dsHDhxIt27d+OCDD/Dy8qJChQpEREToz1WYkydP0rVrVzw8PLC3t6dp06Zs2rTJoExOTg4TJkzA19cXKysratSooZ/+FODvv//mueeew9HREQcHB1q3bs3JkyeBglUTAN26dWPgwIEG93TWrFkMGDAAR0dH/RuP+92323799VeaNm2KtbU1bm5udO/eHYCZM2dSv379AtfbsGFDpk6des/7IUrOkeQM5m7SzTE+4/l6eDs//BzjZaVBJWe+6t8EtbkZ6w+nMvVnGZfdGIyeyD///HOqVKmCtbU1ISEh7N27955l8/LymDlzJtWrV8fa2prAwEA2bNhQ7GNmZ2cTERFBhQoVsLe3p0ePHgZzUZea3KzifzR31T1p8nXr8m4V7bjFYGFhwYABA1i8eLHB/4irVq1Co9HQp08fsrOzCQoKYt26dRw5coRhw4bRv3//+/7O7qbVannhhRdQq9Xs2bOHL7/8kgkTJhQo5+DgwOLFizl69Cgff/wx8+fP56OPPgKgd+/ejBs3jnr16pGSkkJKSgq9e/cucIysrCxCQ0NxcXFh3759rFq1ik2bNjFy5EiDclu2bOHkyZNs2bKFb7/9lsWLFxf4MnO3zMxMunTpQkxMDAcPHqRTp06EhYWRlJSkLzNgwACWLVvGJ598QkJCAl999RX29rp6zuTkZNq0aYOVlRWbN28mNjaWwYMHk59fvDrGDz74gMDAQA4ePKhPtPe7bwDr1q2je/fudOnShYMHDxITE0NwcDAAgwcPJiEhgX379unLHzx4kPj4eAYNGlSs2ETxZedpGLfyEPlahU71POneyMfYIRVZK383PurdEJUKlu5J4qN/v4yIMqQY0fLlyxW1Wq0sXLhQ+fvvv5WhQ4cqzs7OysWLFwst/8Ybbyje3t7KunXrlJMnTypffPGFYm1trRw4cKBYxxw+fLji6+urxMTEKPv371eaNWumtGjRolixnzt3TgGUc+fOFdh269Yt5ejRo8qtW7cMN0Q6Fv9zZPWd/Y+s1q1b2MXwuO9VLXzfYkpISFAAZcuWLfp1rVu3Vl5++eV77vPss88q48aN0y8/9dRTyujRo/XLfn5+ykcffaQoiqL8/vvvioWFhZKcnKzf/ttvvymAEh0dfc9zvP/++0pQUJB+OTIyUgkMDCxQ7u7jfP3114qLi4uSmZmp375u3TrFzMxMSU1NVRRFUcLDwxU/Pz8lPz9fX+bFF19Uevfufc9YClOvXj3l008/VRRFURITExVA2bhxY6FlJ02apFStWlXJzc0tdPt/75+iKErXrl2V8PBw/bKfn5/SrVu3B8b13/vWvHlzpV+/fvcs37lzZ2XEiBH65VGjRilt27a9Z/l7/p2LYntn3VHFb8JaJWjWH0r6jWxjh/NQvt91RvGbsFbxm7BWWbT9lLHDeSzcL8/czahP5HPmzGHo0KEMGjSIunXr8uWXX2Jra8vChQsLLf/9998zefJkunTpQrVq1RgxYgRdunThww8/LPIxMzIyWLBgAXPmzKFdu3YEBQWxaNEidu7cye7du8vkuk1V7dq1adGihf5enThxgr/++oshQ4YAoNFomDVrFgEBAbi6umJvb8/vv/9u8DR6PwkJCfj6+uLt7a1f17x58wLlVqxYQcuWLfH09MTe3p4pU6YU+Rx3nyswMBA7Ozv9upYtW6LVaklMTNSvq1evHubmd0bL8vLyIi3t3kNQZmZmMn78eOrUqYOzszP29vYkJCTo44uLi8Pc3Jynnnqq0P3j4uJo3bo1lpaWxbqe/2rSpEmBdQ+6b3FxcbRv3/6exxw6dCjLli0jOzub3Nxcli5dyuDBgx8pTvFge09f4eu/TgEQ9UIDKtiXz3YeLzfz0w8hO/3Xo/wcJ+OylxWjzX6Wm5tLbGwskyZN0q8zMzOjQ4cO7Nq1q9B9cnJysLa2NlhnY2PD9u3bi3zM2NhY8vLy6NChg75M7dq1qVy5Mrt27aJZs2Yldo0FTL5Q/H3M7/qfunaY7hiq/3z/GnP40eK6y5AhQxg1ahSff/45ixYtonr16vqk9P777/Pxxx8zd+5cAgICsLOzY8yYMSXa2GrXrl3069ePGTNmEBoaipOTE8uXLzf4slaS/ptQVSoVWq32nuXHjx/Pxo0b+eCDD6hRowY2Njb07NlTfw9sbO5fr/mg7WZmZgXqGAurs7/7CwoU7b496NxhYWFYWVkRHR2NWq0mLy+Pnj173ncf8Wgyc/IZtyoORYFeTSrxTN3SmWO8rIxqV4PLmTl8u+ss41YewtlWzVM13Y0d1mPPaE/k6enpaDQaPDwM/3A9PDxITU0tdJ/Q0FDmzJnD8ePH0Wq1bNy4kdWrV5OSklLkY6ampqJWq3F2di7yeUH3JeL69ev6z40bDzE8odqu+B/zu75rmVvo1lnaFO24D6FXr16YmZmxdOlSvvvuOwYPHoxKpWs5u2PHDrp27crLL79MYGAg1apV459/ij5PcZ06dTh37pz+9wUUeAuyc+dO/Pz8ePPNN2nSpAn+/v6cPXvW8HLVajSa+3d1qVOnDocOHSIr605bgR07dmBmZvZIc3Tv2LGDgQMH0r17dwICAvD09DSYNjQgIACtVsu2bdsK3b9Bgwb89ddf92xQ5+7ubnB/NBoNR44ceWBcRblvDRo0ICYm5p7HsLCwIDw8nEWLFrFo0SJeeumlByZ/8WjeXneUc1du4eNsw9RSnmO8LKhUKiLD6vF8oG5c9uHfx3Iw6QkZlz0/B35/ExJ+LfNTG72xW3F8/PHH+Pv7U7t2bdRqNSNHjmTQoEGYmZX+ZURFReHk5KT/1K1b/v+nK4y9vT29e/dm0qRJpKSkGLSW9vf3Z+PGjezcuZOEhAT+97//FauRYIcOHahZsybh4eEcOnSIv/76izfffNOgjL+/P0lJSSxfvpyTJ0/yySefEB0dbVCmSpUqnD59mri4ONLT08nJKTiyVL9+/bC2tiY8PJwjR46wZcsWRo0aRf/+/Qt80SsOf39/Vq9eTVxcHIcOHaJv374GT/BVqlQhPDycwYMHs2bNGk6fPs3WrVtZuXIlACNHjuT69eu89NJL7N+/n+PHj/P999/rX/e3a9eOdevWsW7dOo4dO8aIESO4du1akeJ60H2LjIxk2bJlREZGkpCQwOHDh3nvvfcMyrzyyits3ryZDRs2yGv1Urb52EWW7f13jvFegThYP1p1i6kwM1PxwYuBtKnpzq08DYMW7+P4kzAu+74FsOsz+GUUZGeU6amNlsjd3NwwNzcvkAguXryIp6dnofu4u7uzZs0asrKyOHv2LMeOHcPe3p5q1aoV+Zienp7k5uYW+MfxfucFmDRpEhkZGfrP0aNHi3vJ5caQIUO4evUqoaGhBvXZU6ZMoXHjxoSGhtK2bVs8PT3p1q1bkY9rZmZGdHQ0t27dIjg4mFdeeYW3337boMzzzz/P66+/zsiRI2nYsCE7d+4s0P2pR48edOrUiaeffhp3d/dCu8DZ2try+++/c+XKFZo2bUrPnj1p3749n332WfFuxn/MmTMHFxcXWrRoQVhYGKGhoTRu3NigzLx58+jZsyevvvoqtWvXZujQofo3AxUqVGDz5s1kZmby1FNPERQUxPz58/Wv+AcPHkx4eDgDBgzgqaeeolq1ajz99NMPjKso961t27asWrWKX375hYYNG9KuXbsCPQ78/f1p0aIFtWvXJiQk5FFulbiPq1m5TPhJVyU2uGVVmpXhHONlQW1hxpcvN6ahrzPXbuYxYOFekq/devCO5VnTV6DGM9D1C7Au4wluyqbtXeGCg4OVkSNH6pc1Go3i4+OjREVFFWn/3NxcpXr16sqkSZOKfMxr164plpaWyo8//qgvc+zYMQVQdu3aVeTYH6rVuhAmTqvVKtWrV1c+/PDDB5aVv/OHo9VqlVeXxCp+E9Yq7T/cqtzKzX/wTuXUlcwcpf2HWxW/CWuVpz/YolzOzDF2SCXnVoaibIlSlPzCe6CUhKK2WjdaYzeAsWPHEh4eTpMmTQgODmbu3LlkZWXp+60OGDAAHx8foqKiANizZw/Jyck0bNiQ5ORkpk+fjlar5Y033ijyMZ2cnBgyZAhjx47F1dUVR0dHRo0aRfPmzUu3oZsQJu7SpUssX76c1NRU6Ttein45dIF1h1OwMFPxkZHmGC8rLnZqvhscTM95Ozl1KYtBi/aydGgz7KyMmnoeXXIs/DgErp7W1Y13iDRqOEa9m7179+bSpUtMmzaN1NRUGjZsyIYNG/R1mElJSQb139nZ2UyZMoVTp05hb29Ply5d+P777w0arj3omAAfffQRZmZm9OjRg5ycHEJDQ/niiy/K7LqFMEUVK1bEzc2Nr7/+GhcXF2OH81hKzchm6hpd48VR7fwJqGS8OcbLirezDd8NCeHFL3dy6HwGw5fE8k14E6wsyuEXGK0Wdn0KMTNBmw9OlaFm4XNUlCWVosh4eg/j/Pnz+Pr6cu7cOSpVqmSwLTs7m9OnT1O1atUC3eWEeFzI33nxKIrCgIV7+et4OoGVnPhxRAssTWR60rIQd+4afefv5mauhucaePHxS40wN+Gx5Au4cRGi/wentuiW63aFsE/AxrnUTnm/PHO3J+evSAghjGjJniT+Op6OlYUZH/Zq+EQlcYCGvs581T8IS3MVa+NTmP7L3+VnXPYTm+DLlrokbmEDYR/Di9+WahIvjifrL0kIIYzgTHoW76zTzTE+sXNtalQ0rTnGy0prf3fm9NKNy/797rN8HGPi47Ln58IfU2BJD8i6BBXrwbCtEDQQVKbzNqGctzgwbfcbIUyI8k7+votGo1UYuzKOW3kamlerQHjzKsYOyajCAr25djOXqT//zdxNx6lgp6a/Kd6Tyyfhx8GQEqdbbjoUOs4qOCCXCZBEXgrUajVmZmZcuHABd3d31Gq1fnQ0Ico7RVHIzc3l0qVLmJmZoVarjR2SSfvqz5McSLqGg5UFH/QKNOk5xstK/+ZVSM/M5eOY40z75W+cbdWEBXo/eMeycmg5rBsHuZlg7QxdP4c6zxk7qnuSRF4KzMzMqFq1KikpKVy48BDjqwtRDtja2lK5cuUyGVmxvDp64TofbdQNYxz5fD18ysEc42VlTAd/rmTl8v3us4xdGYeTjSVtTGFc9vwc2P6RLon7tYQXvganezc0MwWSyEuJWq2mcuXK5OfnP3BccCHKG3NzcywsLORN033k5GsYuzKOPI1Cx7oe9GhcfuYYLwsqlYrpz9fj6s1c1sanMHxJLEuHNqOhr7NxA7Owgp4L4dg6aD0OzEy/m5wk8lKkUqmwtLR85CkrhRDlz0cbj3Ms9QYV7NS880KAfOkphLmZijm9GpJxK4+/jqczaNFeVg1vTo2KDmUXhFYLuz/X/dxilO6/HvV0n3JC3okJIUQJ23fmCl/9eRKAqBcCcCunc4yXBd247EEEVnLi6s08BizYy4WyHJf9xEZdy/RN0+FS0WdzNCWSyIUQogRl5eQzbuUhFAV6BlWiY717T8YkdOysLFg0KJhq7nZcyMhmwMK9XM3KLZuT+3eEwL7Q5X1w8y+bc5YwSeRCCFGC3lmfQNKVm/g42zAt7PGc7rg0uNqp+X5ICF5O1pxIy2TQ4n1k5eSX/Inyc2Hb+3Dr33nSVSroPg+aDDapvuHFIYlcCCFKyJbENH7YkwTA+z0b4PiYzDFeVnycbfhucDDOtpbEnbvG8CWx5OaX4HgFV07BwlDY8pZu3vDyMrLcA0giF0KIEnDtZi4TfowHYFDLKrSo4WbkiMonfw8HFg5sio2lOX8dT2fcqkNotSWQcONXwpdt4MIBXd/wBr3L7RP4f0kiF0KIEjD1579Ju5FDdXc7JnSqbexwyrXGlV34sn8QFmYqfj10gRm/PsK47Dk3IHo4rB4KuTegcgsYvh3qhJVs0EYkiVwIIR7RL4cu8OuhC/ruVI/zHONl5ama7nzYKxCVCr7ddZZPN58o/kEuHISvnoJDy0BlBm0nQfiv4Oxb8gEbkfQjF0KIR3Dx+p05xkc+XYNAYw9o8hjp2tCHq1m5TP/1KHM2/oOrnZqXm/k9eEetFvbMg42RoM0Dx0rQYz74tSj9oI2g2E/kVapUYebMmSQlJZVGPEIIUW4oisIbP8aTcSuPAB8nRrarYeyQHjsDW1bltX/v69Sfj7AuPuX+O2RegqW94PfJuiRe+zkY/tdjm8ThIRL5mDFjWL16NdWqVeOZZ55h+fLl5OTklEZsQghh0pbtPce2fy6htjDjo96BT9wc42Xl9Wdq0i+kMooCY1YcZPvx9MILntwM81roBnmxsIZn50DvJWDrWrYBl7GHSuRxcXHs3buXOnXqMGrUKLy8vBg5ciQHDhwojRiFEMLknL2cxVvrjgLwRmitsh1W9AmjUqmY2bU+XQI8ydMo/O/7/cSfv2ZYSFFg9zzISgP3OjB0CzQd8ti0TL+fh/762LhxYz755BMuXLhAZGQk33zzDU2bNqVhw4YsXLjw4VsYCiGEidNoFcatPMTNXA0hVV0Z3LKqsUN67Jmbqfiod0Na1qhAVq6GgYv2cfJS5p0CKpVuutEWr8HQzeDx5AzG89CJPC8vj5UrV/L8888zbtw4mjRpwjfffEOPHj2YPHky/fr1K8k4hRDCZMz/6xT7z17F3sqCD16UOcbLipWFOV/1b0KAjxNXsnL57usPyfp5/J0C9hWh4yxQ2xovSCModqv1AwcOsGjRIpYtW4aZmRkDBgzgo48+onbtO/0mu3fvTtOmTUs0UCGEMAUJKdeZ84duco1pYXXxdX2ykoax2VtZsHhQU8Z+sYppWR9hflAhs3p77Ot3NnZoRlPsRN60aVOeeeYZ5s2bR7du3QqdorNq1aq89NJLJRKgEEKYipx8Da+viCNXo6VDHQ9eDKpk7JCeSBXsrXh76Ass+OxPbubk8dc2O76vmY+t+snsUV3sqz516hR+fvfvx2dnZ8eiRYseOighhDBFH2/SzTHuaqcmSuYYL1uKAnu+hBrPgFsNKrnY0nbYh7z45S4yzt1gxJIDfBPe5InsOVDsK05LS2PPnj0F1u/Zs4f9+/cXO4DPP/+cKlWqYG1tTUhICHv37r1v+blz51KrVi1sbGzw9fXl9ddfJzs7W7+9SpUqqFSqAp+IiAh9mbZt2xbYPnz48GLHLoR4csSevcqX23RzjL/TPQB3B5ljvMzc7hu+YSL8OEg3gxlQ899x2a0tzdj2zyXGl9S47OVMsRN5REQE586dK7A+OTnZIFkWxYoVKxg7diyRkZEcOHCAwMBAQkNDSUtLK7T80qVLmThxIpGRkSQkJLBgwQJWrFjB5MmT9WX27dtHSkqK/rNx40YAXnzxRYNjDR061KDc7NmzixW7EOLJcTM3n3Er49Aq8EIjHzrVlznGy8yprfBlSzj+B5hbQeMBYH6nSjfIz4V5L+vGZf857gIz1x594npNFTuRHz16lMaNGxdY36hRI44ePVqsY82ZM4ehQ4cyaNAg6taty5dffomtrS0LFy4stPzOnTtp2bIlffv2pUqVKnTs2JE+ffoYPMW7u7vj6emp/6xdu5bq1avz1FNPGRzL1tbWoJyjo2OxYhdCPDmi1h/jzOWbeDlZE/l8PWOH82TQ5MGm6fBdN8i8CG61YNgWCB5aoG/407Uq8sGLgQAs3nmGz7c8xLjs5VixE7mVlRUXL14ssD4lJQULi6JXuefm5hIbG0uHDh3uBGNmRocOHdi1a1eh+7Ro0YLY2Fh94j516hTr16+nS5cu9zzHkiVLGDx4cIG6rB9++AE3Nzfq16/PpEmTuHnzZpFjF0I8Obb9c4nvd58F4P2egTjZyBzjpe7qGVjYCbZ/BCgQNBCGbQWPe3+J6tbIh2nP6fqOf/DHPyzd8+QMI17sxm4dO3Zk0qRJ/Pzzzzg5OQFw7do1Jk+ezDPPPFPk46Snp6PRaPDw8DBY7+HhwbFjxwrdp2/fvqSnp9OqVSsURSE/P5/hw4cbvFq/25o1a7h27RoDBw4scBw/Pz+8vb2Jj49nwoQJJCYmsnr16nvGm5OTYzAU7Y0bN4p4pUKI8irjZh5v/HgIgIEtqtDKX+YYL3WHf4S1r0POdbB2grBPoF63Iu06uFVVrmTl8tmWE0xZcxgXW0s6B3iVbrwmoNiJ/IMPPqBNmzb4+fnRqFEjAOLi4vDw8OD7778v8QDvtnXrVt555x2++OILQkJCOHHiBKNHj2bWrFlMnTq1QPkFCxbQuXNnvL29DdYPGzZM/3NAQABeXl60b9+ekydPUr169ULPHRUVxYwZM0r2goQQJm3aL0e4eD2Ham4yx3ipy82C9W9A3BLdsm8z3YxlzpWLdZhxHWtyOSuXZXuTGL08DicbS1rUeLy/gBX71bqPjw/x8fHMnj2bunXrEhQUxMcff8zhw4fx9S36HK9ubm6Ym5sXeE1/8eJFPD0Lb0gydepU+vfvzyuvvEJAQADdu3fnnXfeISoqCq1Wa1D27NmzbNq0iVdeeeWBsYSEhABw4sS961UmTZpERkaG/lPc9gBCiPJlXXwKP8f9O8d474bYqGWO8VKTEq+bNzxuCaCCNm/AwHXFTuKgG5f9rW716VTPk1yNlqHf7efw+YySj9mEPFTveTs7O4On2oehVqsJCgoiJiaGbt26AaDVaomJiWHkyJGF7nPz5k3MzAy/e5ib6/7n+m8rxUWLFlGxYkWeffbZB8YSFxcHgJfXvV/BWFlZYWV1p7vJ9evXH3hcIUT5lHY9mylrDgMQ0bY6DWWO8dIVtxQuHwcHb3jha6ja+pEOZ26mYu5LDRm0aB+7Tl1m4KK9rBrenGru9iUUsGl56GFwjh49SlJSErm5uQbrn3/++SIfY+zYsYSHh9OkSROCg4OZO3cuWVlZDBo0CIABAwbg4+NDVFQUAGFhYcyZM4dGjRrpX61PnTqVsLAwfUIH3ReCRYsWER4eXqAB3smTJ1m6dCldunShQoUKxMfH8/rrr9OmTRsaNGjwsLdDCPGYUBSFiasPc/VmHvW8HRnZzt/YIT3+OkwHM3NoPa7Ephy1tjTn6wFB9Jm/myPJ1+m/YC8/jWiBp5N1iRzflDzUyG7du3fn8OHDqFQq/ZPw7VbhGo2myMfq3bs3ly5dYtq0aaSmptKwYUM2bNigbwCXlJRk8AQ+ZcoUVCoVU6ZMITk5GXd3d8LCwnj77bcNjrtp0yaSkpIYPHhwgXOq1Wo2bdqk/9Lg6+tLjx49mDJlSnFvhRDiMbRi3zk2H0v7d47xhqgtnryRwkrdqW1w4Ft4Yb4ugVtaQ+jbD96vmBysLVk8KJgXv9zF6fQsBizcw8r/NcfZVl3i5zImlVLMnvO3n36/+eYbqlatyt69e7l8+TLjxo3jgw8+oHXrR3slUl6cP38eX19fzp07R6VKMt6yEI+DpMs36fzxn2TlanizSx2Gtqlm7JAeP7euwdwAXav0zrMh5H+lfspzV27SY95O0m7kEOTnwpIhIeWizUNR80yxv2ru2rWLmTNn4ubmhpmZGWZmZrRq1YqoqChee+21RwpaCCGMRaNVGL/qEFm5GoKrujK4lcwxXipsnKHL+9A4HBq9XCan9HW15fshIThaWxB79iqv/hBLnkb74B3LiWInco1Gg4ODA6BreX7hwgUA/Pz8SExMLNnohBCijCzYfoq9Z65gpzbnwxcDMZc5xkvOkZ/g9J93lgNfguc/AbVdmYVQy/POuOxbEi/xxo/xj8247MVO5PXr1+fQId0ACSEhIcyePZsdO3Ywc+ZMqlWT11BCiPInMfUGH/wuc4yXuNws+Hkk/DgYfhoKWZeNGk6TKq580a8x5mYqog8m89a6hMdiXPZiJ/IpU6bo+2zPnDmT06dP07p1a9avX88nn3xS4gEKIURpys3XMnalbo7x9rUr0qtJ0cfDEPdxu2/4we8BFTTurxupzcja1fbg/Z66HkoLd5zmi60njRzRoyt2q/XQ0FD9zzVq1ODYsWNcuXIFFxcXmZtXCFHufLr5OH9fuI6LrSVRPWSO8UemKLD3a/hjCmhywcHr377hbYwdmd4LjStxJSuXt9Yl8P7viVSwU/NScPEHnzEVxXoiz8vLw8LCgiNHjhisd3V1lT9+IUS5cyDpqn6mrLe7B1DR4fHrY1ymsi7Dsj7w2xu6JF6zMwzfYVJJ/LZXWldjRFvdkNyTow+z4UiKkSN6eMVK5JaWllSuXLlYfcWFEMIU3crVMG7lIbQKdGvoTZcnYHKNUnX6T9284f/8BuZqXdeyPsvAroKxI7unN0Jr0buJL1oFXlsWx86T6cYO6aEUu478zTffZPLkyVy5cqU04hFCiDLx7m8JnE7PwtPRmhnP1zd2OOWXJh82vwXfPg83UsCtJgzdrOsfbuJvalUqFW93r0/Huh7karQM+y6WI8nlb1z2YteRf/bZZ5w4cQJvb2/8/PywszPsPnDgwIESC04IIUrDX8cv8e2uf+cYf7EBTrYyx/hDuXoWfnoFzu/VLTceAJ3eLdNuZY/KwtyMT/o0InzhXvacvvLvuOwtqOpWjq6huDvcnuBECCHKo4xbefzfqngABjT3o7W/u5EjKseO/6FL4laOEDYX6vcwdkQPxdrSnPnhTXjpq90cTblO/wV7+GlECzwcy0ebiWIn8sjIyNKIQwghysSMX/4m9Xo2Vd3smNhZ5hh/JE1f0b1ObzwAXKoYO5pH4mhtybeDg+n55U7OXr5J+MK9rBjWvFy8rZHZAIQQT4zfDqew+mAyZir4sFcgtuqHngDyyZR6BJb2hpwbumWVCtpPK/dJ/DZ3Byu+HxyCu4MVx1JvMOTbfdzKNf3G3cVO5GZmZpibm9/zI4QQpijtRjaTo3VzjI9oW53GlV2MHFE5o9XAqnD4ZwPEzDJ2NKWmcgVbvhscjIO1BfvPXmXk0gMmPy57sb+ORkdHGyzn5eVx8OBBvv32W2bMmFFigQkhRElRFIXJ/84xXsfLkdHtaxo7pPLHzBy6fg47PoGnJhg7mlJVx8uRBeFN6b9gDzHH0pjwUzwf9AzEzETH3y92Iu/atWuBdT179qRevXqsWLGCIUOGlEhgQghRUlbtP8+mhDTU5mZ81DtQ5hgvqjPb4UYqBPTULVdupvs8AYKruvJ538b8b0ksqw8kU8FOzeQudUxy8LMS+2tu1qwZMTExJXU4IYQoEeeu3GTGr38DMK5jTWp7Oho5onJAkw+b34bFz+kmPbn0ZM5s2aGuB+/10I3LPv+v03z15ykjR1S4EmnpcevWLT755BN8fHxK4nBCCFEitHfNMd60iguvtJYZGh/oWpJuprJzu3XLAT3AqZJxYzKinkGVuJqVy9vrE3j3t2O42qrp1dS0JtYpdiL/7+QoiqJw48YNbG1tWbJkSYkGJ4QQj2LhjtPsOX0FW7U5H77YUOYYf5CjP8MvoyA7Q9c3/LmP7rxWf4INbVON9Kwcvtp2iomr43GytSS0nqexw9IrdiL/6KOPDBK5mZkZ7u7uhISE4OIirUCFEKbh+MUbzP5d90p4yrN1qVxB5hi/p9yb8PskiF2sW/ZpAj2+AdeqRg3LlEzsVJurWbms3H+eUcsO8t3gYJpVM41x5IudyAcOHFgKYQghRMnJ02h5fWUcufla2tZyp0+wab0KNSkXj8KPg+FSAqCCVq/D05PB3PQHQilLKpWKd7oHcPVmHhuPXmTot/tZ/r9m1PM2/hzrxW7stmjRIlatWlVg/apVq/j2229LJCghhHgUn24+wZHk6zjbWjK7RwOTbGlsdIoC+76B+U/rkri9BwxYAx0iJYnfg4W5GZ/2aURwVVdu5OQTvnAfZy9nGTus4ifyqKgo3NzcCqyvWLEi77zzTokEJYQQDyvu3DX9HONvdatPxXIyXnaZunkFVrwM68ZBfjb4d4QRO6FaW2NHZvKsLc35JrwJdbwcSc/Mof+CvaRdzzZqTMVO5ElJSVStWrDexM/Pj6SkpBIJSgghHsatXA1jV8ah0So8H+jNcw28jR2SabpwEI6tBTNLCI2CvivBruADmiicblz2plR2tSXpyk0GLNxLxq08o8VT7EResWJF4uPjC6w/dOgQFSoUv+L/888/p0qVKlhbWxMSEsLevXvvW37u3LnUqlULGxsbfH19ef3118nOvvNtaPr06ahUKoNP7dqGEyNkZ2cTERFBhQoVsLe3p0ePHly8eLHYsQshTMt7G45x6lIWHo5WzOxaz9jhmK4a7aHjW/DKJmj+qsnPG26KKjpY8/2QYNzsdeOyD/12P9l5xhmXvdiJvE+fPrz22mts2bIFjUaDRqNh8+bNjB49mpdeeqlYx1qxYgVjx44lMjKSAwcOEBgYSGhoKGlpaYWWX7p0KRMnTiQyMpKEhAQWLFjAihUrmDx5skG5evXqkZKSov9s377dYPvrr7/Or7/+yqpVq9i2bRsXLlzghRdeKN6NEEKYlJ0n0lm88wwAs3sG4myrNm5ApuTaOd1kJ1fP3lnXYhR4NzRaSI8Dvwp2fDu4KQ5WFuw9c4WRSw+Sb4xx2ZViysnJUXr16qWoVCrF0tJSsbS0VMzNzZVBgwYpOTk5xTpWcHCwEhERoV/WaDSKt7e3EhUVVWj5iIgIpV27dgbrxo4dq7Rs2VK/HBkZqQQGBt7znNeuXVMsLS2VVatW6dclJCQogLJr164ix37u3DkFUM6dO1fkfYQQpSPjVq7S/J1Nit+Etcqb0fHGDsf0fN9DUSIdFeW77saO5LG0+2S64v/mesVvwlpl3Mo4RavVlshxi5pnit39TK1Ws2LFCt566y3i4uKwsbEhICAAPz+/Yh0nNzeX2NhYJk2apF9nZmZGhw4d2LVrV6H7tGjRgiVLlrB3716Cg4M5deoU69evp3///gbljh8/jre3N9bW1jRv3pyoqCgqV64MQGxsLHl5eXTo0EFfvnbt2lSuXJldu3bRrFnh4wjn5OSQk5OjX75x40axrleYkCunda11cx/Q2vSZGWD9b9eSIz/B6b90jYJqd9Gtu3ERtkYV//xt/g+c/h0F8Z/fIfE38GsBDXrp1uVmwe9vFv+4zV4F938nAzmzAw6vAs8AaHrX/AdrX9e1Vi6Oxv3BJ0j3c8oh2L8IXKtBy9fulPlj6p2pLYuqXneo9pTu58snYeenYF9R1/Xptm2z4fqFBx7qyKnLRGRlYW9vQaca/YEA3YbbvyNLW+h0V2PcXZ9D+vHixXuv39Gzc8Ds35ebsd/q6p+L416/o8L+/orj7t/Rsx/Ar2N0/xUlLqRaBT7r04jhS2L5MfY8FezUTOpSp8zO/9BDtPr7++Pv7//QJ05PT0ej0eDh4WGw3sPDg2PHjhW6T9++fUlPT6dVq1YoikJ+fj7Dhw83eLUeEhLC4sWLqVWrFikpKcyYMYPWrVtz5MgRHBwcSE1NRa1W4+zsXOC8qamp94w3KipKZnd7HBz+UfcPWm4Rkk7biXf+IU3aA7GLdA2CbifynOu6dcXV9JU7iTzl0L/HUO4kifychztuvW53Enl6ou4YtZ8zTBKxi0Ep5qu/Kq3uJPKrZ3XHrdzcMJHHr4DMYrYzca99J5FnXtQdt0INw0T+9xpI+/uBh2oBtLAA8oG0BkCYbsPt35G1s2Ei/2cDnP6zePHe63f07Id3ipzepku6xXGv31Fhf3/FcffvyKWKrmuZKDUd63nybo8GvPFjPF/9eYqO9TwJ8iubQdKKnch79OhBcHAwEyYYTmM3e/Zs9u3bV2gf85KydetW3nnnHb744gtCQkI4ceIEo0ePZtasWUydOhWAzp0768s3aNCAkJAQ/Pz8WLly5SPNzDZp0iTGjh2rX05OTqZu3boPfzGibOVkwm9vQNwPumXfEKje/v77qO3u/OzfEWwrgF/zO+tsXKHt5IL7PYh9xTs/+7XUHcMr8M46S5uHO65LlTs/ezfSHcPtP1+2204q/hN5xbv+zt1r6Y7737G3W7z24Dcc/1WpyZ2fnSrpjmvralim6RDISr/nIbJy8/l+11lu5mpoWtWF1jXcC/8dWVgZ7hjYF/xaFS/ee/6O7mooVrcruNUq3nHv9Tsq7O+vOJ7g8dGNpVcTX65m5WJvbVFmSRwofh25m5ubEh9fsA4qPj5eqVixYpGPk5OTo5ibmyvR0dEG6wcMGKA8//zzhe7TqlUrZfz48Qbrvv/+e8XGxkbRaDT3PFeTJk2UiRMnKoqiKDExMQqgXL161aBM5cqVlTlz5hQ5fqkjL2eWvqSrI5zurCib31aU/DxjRyQekVarVV75dp/iN2GtEvrRNiU7L9/YIQlRooqaZ4rdaj0zMxO1umBrUEtLS65fv17k46jVaoKCggymPtVqtcTExNC8efNC97l58yZmZoYhm5ubA7rJW+4V78mTJ/Hy8gIgKCgIS0tLg/MmJiaSlJR0z/OKx0DbSeBSFcJ//Xf4yRKZ+E8Y0Y+x59l49CKW5io+6t0QKwtzY4ckhFEUO5EHBASwYsWKAuuXL19e7FfNY8eOZf78+Xz77bckJCQwYsQIsrKyGDRoEAADBgwwaAwXFhbGvHnzWL58OadPn2bjxo1MnTqVsLAwfUIfP34827Zt48yZM+zcuZPu3btjbm5Onz59AHBycmLIkCGMHTuWLVu2EBsby6BBg2jevPk9G7qJcigrHRLW3ln2agAj9+vqe0W5d/7qTWb+ehSAsc/Uoo6XzDEunlzFfiyZOnUqL7zwAidPnqRdu3YAxMTEsHTpUn788cdiHat3795cunSJadOmkZqaSsOGDdmwYYO+AVxSUpLBE/iUKVNQqVRMmTKF5ORk3N3dCQsL4+2339aXOX/+PH369OHy5cu4u7vTqlUrdu/ejbu7u77MRx99hJmZGT169CAnJ4fQ0FC++OKL4t4KYaquX4Cvn4abl2HIH+DTWLdensIfC1qtwv+tiudGTj5Bfi4MayNzjIsnm0q51zvp+1i3bh3vvPOOvvtZYGAgkZGRuLq6Ur9+/dKI0+ScP38eX19fzp07R6VK0qjEpCgKrBwAlxLhxUXgISN8PU4Wbj/NzLVHsbE057fRraniZvfgnYQoh4qaZx7qEeXZZ5/l2WefBeD69essW7aM8ePHExsbi0ZjnCHqxBPu6hldFyMbZ91wk10/040jrZY5qB8nJ9Ju8N4GXffUN5+tI0lcCB6ijvy2P//8k/DwcLy9vfnwww9p164du3fvLsnYhCiaIz/Bl63h19F3ulZZO0kSf8zkabSMXXmInHwtbWq60y+ksrFDEsIkFOuJPDU1lcWLF7NgwQKuX79Or169yMnJYc2aNdKnWpS93Cxd3/CDS3TLN1IgNxOsHIwblygVn285Qfz5DJxsZI5xIe5W5CfysLAwatWqRXx8PHPnzuXChQt8+umnpRmbEPeWEg9fPfVvElfphj0duF6S+GMq/vw1Pt2sm2N8Vrf6eDrJHONC3FbkJ/LffvuN1157jREjRjzS0KxCPBJFgT1fwcapoMkFBy94YT5UbW3syEQpyc7TMHblITRahecaePF8oMwxLsTdivxEvn37dm7cuEFQUBAhISF89tlnpKffe+hEIUpc1mVY9hJsmKBL4rW6wPAdksQfc+//nsiJtEwqOlgxq+uT0StGiOIociJv1qwZ8+fPJyUlhf/9738sX74cb29vtFotGzdulNnAROk6tQ3mtdBNdmFuBV0+gJeWgl0xx58W5crOk+ks2H4agPd6NMDFTuYYF+K/it1q3c7OjsGDB7N9+3YOHz7MuHHjePfdd6lYsSLPP/98acQonmSaPIiZCd91hcxU3YQUQzdD8FBdNzPx2LqRncf/rYoHoE9wZZ6uXfEBewjxZHro7mcAtWrVYvbs2Zw/f55ly5aVVExC3LFpOvz1IaBA43AYtgU85fXqk2Dmr0dJvnaLyq62THm27OZ2FqK8KZExK83NzenWrRvdunUricMJcUeLUbrX6e2mQL3uxo5GlJE//k5lVex5VCr4sFcgdlYyvK4Q9/JIT+RClLjcLDi0/M6ygye8ukeS+BPkcmYOk6MPAzCsTTWaVnF9wB5CPNnka64wHXnZML8dXDoGFtZQr5tuvUx28sRQFIXJ0YdJz8yllocDY5+paeyQhDB58i+kMB2W1rouZdkZYCut0Z9E0QeT+f1v3Rzjc3oHyhzjQhSBJHJhXFmXIS8LnP8dN/vpybp6cVt5nfqkuXDtFpE//w3AmA41qeftZOSIhCgfpI5cGM/pP+HLlropR/NzdevMLSWJP4G0WoX/+/EQN3LyaVTZmf/JHONCFJkkclH2NPmw+S349nndRCc5mZB50dhRCSP6btcZdpy4jI2lOXN6NcTCXP5pEqKo5NW6KFtXz8JPr8D5vbrlRi9D59mglnmln1Qn0jKJ+k03x/jkLrWpKnOMC1EskshF2fk7Gn4ZDTkZYOUIz30EAT2NHZUwonyNlnGrdHOMt/Z34+VmfsYOSYhyRxK5KH25N2HDRDjwrW65UlPo8Q24VDFqWML45m09yaFz13C0tmB2T5ljXIiHIYlclK7UI/DjYEhPBFTQeiy0naRr1CaeaEeSM/g45jigm2Pcy8nGyBEJUT5JIhelQ1Fg3zfw+5ugyQF7T3jhK6jW1tiRCROQnafh9RVx5GsVugR4yhzjQjwCSeSidMQugvXjdT/7d4Ru88DOzbgxCZPx4R+JHE/LxM3eire6BcgrdSEegSRyUToC+8CB76BBbwgZLlOOCr3dpy7zjX6O8QBcZY5xIR6J0Ttrfv7551SpUgVra2tCQkLYu3fvfcvPnTuXWrVqYWNjg6+vL6+//jrZ2dn67VFRUTRt2hQHBwcqVqxIt27dSExMNDhG27ZtUalUBp/hw4eXyvU9MTT5usSt1eiWLW3glRhoNkKSuNC7kZ3H+FWHUBR4qakv7et4GDskIco9oybyFStWMHbsWCIjIzlw4ACBgYGEhoaSlpZWaPmlS5cyceJEIiMjSUhIYMGCBaxYsYLJkyfry2zbto2IiAh2797Nxo0bycvLo2PHjmRlZRkca+jQoaSkpOg/s2fPLtVrfawpCizrDb+Mgu1z7qw3k3GyhaG31iZw/uotKrnYMOW5usYOR4jHglFfrc+ZM4ehQ4cyaNAgAL788kvWrVvHwoULmThxYoHyO3fupGXLlvTt2xeAKlWq0KdPH/bs2aMvs2HDBoN9Fi9eTMWKFYmNjaVNmzb69ba2tnh6epbGZT15VCqo3wPO7QWXqsaORpiomISLrNh/TjfH+IuB2Msc40KUCKM9kefm5hIbG0uHDh3uBGNmRocOHdi1a1eh+7Ro0YLY2Fj96/dTp06xfv16unTpcs/zZGRkAODqajh+9w8//ICbmxv169dn0qRJ3Lx5877x5uTkcP36df3nxo0bRbrOB1mxL4mXv9nDT7HnycrJL5Fjloncm5B27M5yYB8YFSsDvIgCcvI1/HY4hQk/6eYYH9q6GiHVZHY7IUqK0b4Sp6eno9Fo8PAwrCPz8PDg2LFjhe7Tt29f0tPTadWqFYqikJ+fz/Dhww1erd9Nq9UyZswYWrZsSf369Q2O4+fnh7e3N/Hx8UyYMIHExERWr159z3ijoqKYMWPGQ1zp/f0Um8zeM1fYfiKdKWuOEFrPgxcaV6JlDTfMzUy0bvniUV3f8OwMGLFDN8mJSgX2FY0dmTARiqKw/+xVVh9IZl38Ba5n676k1vSwlznGhShh5erd1tatW3nnnXf44osvCAkJ4cSJE4wePZpZs2YxderUAuUjIiI4cuQI27dvN1g/bNgw/c8BAQF4eXnRvn17Tp48SfXq1Qs996RJkxg7dqx+OTk5mbp1H72O74MXA4k+mEz0wfOcuXyTNXEXWBN3gYoOVnRt6E33RpWo6+34yOcpEYoC+xfo+obnZ4O9B1w7K7OVCb3T6VlEHzhPdFwy567c0q/3crKma0MfhrSqirWltJ0QoiQZLZG7ublhbm7OxYuGs15dvHjxnnXXU6dOpX///rzyyiuALglnZWUxbNgw3nzzTczM7tQUjBw5krVr1/Lnn39SqVKl+8YSEhICwIkTJ+6ZyK2srLCystIvX79+/cEXWQSVK9gyuoM/r7WvwYGka0QfPM/a+BTSbuQw/6/TzP/rNLU9HejeyIdujXzwcLQukfMW280rusZsx9bqlms8o+sbbu9unHiEybialcva+AusPpjMwaRr+vV2anM6B3jxQiMfQqpVMN03TEKUc0ZL5Gq1mqCgIGJiYujWrRugexUeExPDyJEjC93n5s2bBskawNxc9+1eURT9f0eNGkV0dDRbt26latUHN76Ki4sDwMvL6yGv5tGpVCqC/FwI8nNh2nP12JKYRvSBZDYfS+NY6g2ifjvGexuO0bKGG90b+RBazxO7smosdGYHrB4K15PBzBKemQEhI8DM6L0XhZHk5GvYnJDG6oPJbE1MI0+j+//PTAWt/d15obEPHet6YqOWp28hSptRX62PHTuW8PBwmjRpQnBwMHPnziUrK0vfin3AgAH4+PgQFRUFQFhYGHPmzKFRo0b6V+tTp04lLCxMn9AjIiJYunQpP//8Mw4ODqSmpgLg5OSEjY0NJ0+eZOnSpXTp0oUKFSoQHx/P66+/Tps2bWjQoIFxbsR/qC3MCK3nSWg9T67dzGVtfArRB5OJPXuVv46n89fxdGzVRwit58kLjX1oUb2U6tM1+fDn+/DnbFC04Fodei4E74Ylfy5h8hRFIfbsVVYfTGbtoTv13gD1vB3p3siH5xt6U9HBSG+NhHhCGTWR9+7dm0uXLjFt2jRSU1Np2LAhGzZs0DeAS0pKMngCnzJlCiqViilTppCcnIy7uzthYWG8/fbb+jLz5s0DdIO+3G3RokUMHDgQtVrNpk2b9F8afH196dGjB1OmTCn9C34IzrZqXm7mx8vN/Dh7Oevf+vRkzl6+qf/Zw9GKrg196N7IhzpeJVSfnnEefhoKSTt1y4F9ocv7YGVfMscX5caZ9CxWH0xmzcFkkq7c6d3h6WhN10bevNCoErU8HYwYoRBPNpVy+520KJbz58/j6+vLuXPnHlgHX9IUReFA0jVWH9DVp2fcytNvq+3pwAuNfeja8BHq0xN+hZ9HQvY1UDvAc3OgQa+SCV6UC1ezcll7OIXVB84XqPfuVN+LFxr70EzqvYUoVUXNM5LIH5IxE/ndcvI1bDl2ieiD59l8zLCusmUNN15orKtPt1UX8eXLqW3w3fO6n70bQ88F4FqtlKIXpkT3t5TG6gPJbPlPvXcrf3deaORDx3oeRf9bEkI8kqLmGfk/spyzsjCnU31POtW/f316p3qedC9KfXrVNlCzE7jXgqengIVMaPE4073d0fX3/u/bnbpejrzQ2IfnA72paKzeEkKIB5In8odkKk/k93Im/U59+t31mh6OVnRr6EP3xj7U9nTU9Q2PXwG1nwWrf+s5tRoZJ/0xd/ZyFqsPJLMmTtfe4jYPRyu6NfKRem8hTIC8Wi9lpp7Ib7vfE1cdL0fet1tC/fPLdUOsdv/SiJGK0nbtZi6/xqcQfeA8B+6q97ZV697qvNCoEs2rS723EKZCXq0L4Hb/dFeC/FyZFlaXLccusfrAebYkppGQcp0Zqpp8r7Zk9VlHbA+cp2P9YtSnC5N3vzYUUu8txONB/u99glhZmNOprjudKlzkao8OrD2cQvQBZ1omfcLlVCdYeQi7NUcIlaezcu1+vRrqeDnyQiMfujaUem8hHheSyJ8kGcm6EdouHMRl2Fb6N6tF/2Z+nElvaFCfvvpAMqsPJEs/4XLm9jgDaw4mc+a/9d53t4sQQjxWpI78IZWXOnK9hLXwy0i4dRXU9tBjAdTqZFDkfiN36Vswy8hdJuW/PRVus1Wb06meJy80ljcrQpRX0titlJWbRJ53C/6YAvu+0S17N9Il8QqFTw5zm4ylbbpy87UGY/HnarTAI4wdIIQwSdLYTUDaMd284Wl/65ZbjIJ204rUN9zKQjdzVecArwKzW2375xLb/rkko3yVodv13rdnx7t2s4RH8xNClFvyRP6QTPqJXFEgdjFsmAT5t8DOXde1rEaHRz70aX3/9PMG8017Olrr+h839qGmh9Snl5Qk/Zj65w3qvSs66Pp7l+j4+kIIkyKv1kuZySbyW1fh19Fw9GfdcvV20P0rsK9YoqdRFIX9Z3X909fFy0xYJSnjZh5rD18g+kAy+++q97ax/Le/d2nOeCeEMBmSyEuZSSbypN3w0yuQcQ7MLKB9JDQfWerzhmfn/TtG93/q083NVLT6t85W6tPvLzdfy9bENKIPJhOTULDeu8znoBdCGJ3UkT9prpyGxc+CNh9cquomO/EJKpNTW1veqU+/crs+/UAycefu1KfbW1n8O3qYrj7dTJ4mURSFg+euEX0gmbXxF7gq9d5CiIcgT+QPySSfyDdMgpuX4dkP74ybbkSnLmWy5mAy0XHJBvXpXk7/1qc38sH/CaxPP3flzlzyp9Oz9OsrOljRtaE33RtVoq631HsL8aSTV+ulzCQSeeJv4FEfnH11yyY62cmd+nRdi+sbd9Wn1/dxpHujSjwf6I27g5URoyxdGTfzWHc4heiD59l3pmC9d/dGPrSsIfXeQog7JJGXMqMn8t3zYMNEqNwcwteCefmoJcnO07D53zmvtyamka+9U5/e2t+NFxpXomNdD6wtTe8LSXHl5mvZ9o9ubPu7671VKmhZ/U5/b6n3FkIURurIH3c1O8GWKF09uKI1djRFZm1pTpcAL7rcVZ/+04FkDp27xtbES2xN1NWnd66vmz+9WdXyVZ+uKApx564RfTCZXw8VrPfu3khX7+3pJPXeQoiSIU/kD6nMn8gVBS4cMGzAlnkJ7N1L/9xl4OTt+vSDyZy/eqc+3dvJmq7loD79dr33moPJnLqr3tvdwYpuUu8thHgI8mq9lJVpIr917d++4Wugf7Sub/hjSqvV1affHsHs7vr0AB8nff90N3vj16dn3Mpj/eEUog8ks/fMFf16G0tzQut50L1xJVpWr4CFeel2/xNCPJ7k1frjImnPv33Dk3R9w6+chvsPk16umZmpCK7qSnBVVyLD6hGTkEb0wfNsTbzE4eQMDidn8Pb6BNr8W5/+TBnXp9+u944+eJ5NCWnk5t+p925RvQLdG1WiU31P7KXeWwhRRuRfG1Ol1cD2Obp6cEUDLlWgx0KoVDZ9w02BtaU5zzbw4tkGXlzOzGFtfAqrD+rq07ckXmJL4iUcrCzoHOBJ90aVCKnqWir16YqicOh8BtEHzvPLf+q9a3k40L2xbn5vLyebEj+3EEI8iLxaf0il+mr9+gVYPQzO/KVbDngRnp0D1lLHCrr69OgDuvr05Gt36tN9nG3o2tCbFxr7UKPio9enn7tyU19vf3e9t5v9v/XejX2o6+WISlV+GuMJIcqPouYZo1feff7551SpUgVra2tCQkLYu3fvfcvPnTuXWrVqYWNjg6+vL6+//jrZ2dnFOmZ2djYRERFUqFABe3t7evTowcWLF0v82h5K4m8wr6UuiVvaQbd58MJ8SeJ3qe5uz/jQWvz1xtOsGNaMl5r64mBtQfK1W3yx9SQd5vzJ859tZ9GO06Rn5hTr2Bm38li+N4leX+2i9ewtfLjxH06lZ2FtaUbXht4sHtSU3ZPaMeW5utTzdpIkLoQwOqM+ka9YsYIBAwbw5ZdfEhISwty5c1m1ahWJiYlUrFhwko+lS5cyePBgFi5cSIsWLfjnn38YOHAgL730EnPmzCnyMUeMGMG6detYvHgxTk5OjBw5EjMzM3bs2FHk2Ev8iTwvGzZFwp4vdcueDaDnInCr8ejHfgJk52mISUhj9YHzbPvnkkH/9Kdq6uZP71Cn8Pr0PI2WbYmXiD6YzMaEi1LvLYQwCeWi1XpISAhNmzbls88+A0Cr1eLr68uoUaOYOHFigfIjR44kISGBmJgY/bpx48axZ88etm/fXqRjZmRk4O7uztKlS+nZsycAx44do06dOuzatYtmzZoVKfYSTeSX/tHNG37xsG65+UhoPw0sjN8yuzy6nJnDr4cuEH0wmUPnM/TrHaws6BLgRffGPgRXceVwcoa+v/flrFx9uZoe9nRvVIlujaTeWwhhPCbfaj03N5fY2FgmTZqkX2dmZkaHDh3YtWtXofu0aNGCJUuWsHfvXoKDgzl16hTr16+nf//+RT5mbGwseXl5dOhwZ27u2rVrU7ly5fsm8pycHHJy7rymvXHjxsNfvMGBM2FhKNy6ArZuulfpNTuWzLGfUBXsrRjYsioDW1blRNqd/unJ126xYv85Vuw/h53anKxcjX4fN/vb45z7UM9b6r2FEOWH0RJ5eno6Go0GDw8Pg/UeHh4cO3as0H369u1Leno6rVq1QlEU8vPzGT58OJMnTy7yMVNTU1Gr1Tg7Oxcok5qaes94o6KimDFjRnEv88Gs7OHpyZDwK7zwNTh4lvw5nmA1Kurq08c+U5O9Z64QfSCZ9YdTuJGTj7WlGR3r6ub3blXDTfp7CyHKpXJV6bd161beeecdvvjiC0JCQjhx4gSjR49m1qxZTJ06tVTPPWnSJMaOHatfTk5Opm7duiVz8KavQJMhpT5v+JPMzExFs2oVaFatAjO61uNoynX8K9rjYG1p7NCEEOKRGC2Ru7m5YW5uXqC1+MWLF/H0LPypdOrUqfTv359XXnkFgICAALKyshg2bBhvvvlmkY7p6elJbm4u165dM3gqv995AaysrLCyulNnff369WJd732pVLqPKBPWluY0ruxi7DCEEKJEGO0RUK1WExQUZNBwTavVEhMTQ/PmzQvd5+bNm5j956nV3FzXCllRlCIdMygoCEtLS4MyiYmJJCUl3fO8QgghhKky6qv1sWPHEh4eTpMmTQgODmbu3LlkZWUxaNAgAAYMGICPjw9RUVEAhIWFMWfOHBo1aqR/tT516lTCwsL0Cf1Bx3RycmLIkCGMHTsWV1dXHB0dGTVqFM2bNy9yi3UhhBDCVBg1kffu3ZtLly4xbdo0UlNTadiwIRs2bNA3VktKSjJ4Ap8yZQoqlYopU6aQnJyMu7s7YWFhvP3220U+JsBHH32EmZkZPXr0ICcnh9DQUL744ouyu3AhhBCihMgQrQ+pzKcxFUII8UQpN0O0CiGEEOLhlavuZ6ZEq9UN45mSkmLkSIQQQjyObueX2/nmXiSRP6TbXdyCg4ONHIkQQojH2cWLF6lcufI9t0sd+UPKz8/n4MGDeHh4FOgSVxw3btygbt26HD16FAeHR59683Em96po5D4VndyropH7VHQlea+0Wi0XL16kUaNGWFjc+7lbErmRXb9+HScnJzIyMnB0lKlK70fuVdHIfSo6uVdFI/ep6Ixxr6SxmxBCCFGOSSIXQgghyjFJ5EZmZWVFZGSkwTjuonByr4pG7lPRyb0qGrlPRWeMeyV15EIIIUQ5Jk/kQgghRDkmiVwIIYQoxySRCyGEEOWYJHIj+/zzz6lSpQrW1taEhISwd+9eY4dkcv7880/CwsLw9vZGpVKxZs0aY4dkkqKiomjatCkODg5UrFiRbt26kZiYaOywTNK8efNo0KABjo6OODo60rx5c3777Tdjh2Xy3n33XVQqFWPGjDF2KCZn+vTpqFQqg0/t2rXL5NySyI1oxYoVjB07lsjISA4cOEBgYCChoaGkpaUZOzSTkpWVRWBgIJ9//rmxQzFp27ZtIyIigt27d7Nx40by8vLo2LEjWVlZxg7N5FSqVIl3332X2NhY9u/fT7t27ejatSt///23sUMzWfv27eOrr76iQYMGxg7FZNWrV4+UlBT9Z/v27WVzYkUYTXBwsBIREaFf1mg0ire3txIVFWXEqEwboERHRxs7jHIhLS1NAZRt27YZO5RywcXFRfnmm2+MHYZJunHjhuLv769s3LhReeqpp5TRo0cbOySTExkZqQQGBhrl3PJEbiS5ubnExsbSoUMH/TozMzM6dOjArl27jBiZeFxkZGQA4OrqauRITJtGo2H58uVkZWXRvHlzY4djkiIiInj22WcN/r0SBR0/fhxvb2+qVatGv379SEpKKpPzyuxnRpKeno5Go8HDw8NgvYeHB8eOHTNSVOJxodVqGTNmDC1btqR+/frGDsckHT58mObNm5OdnY29vT3R0dHUrVvX2GGZnOXLl3PgwAH27dtn7FBMWkhICIsXL6ZWrVqkpKQwY8YMWrduzZEjR0p9ohlJ5EI8hiIiIjhy5EjZ1dGVQ7Vq1SIuLo6MjAx+/PFHwsPD2bZtmyTzu5w7d47Ro0ezceNGrK2tjR2OSevcubP+5wYNGhASEoKfnx8rV65kyJAhpXpuSeRG4ubmhrm5uX5e89suXryIp6enkaISj4ORI0eydu1a/vzzTypVqmTscEyWWq2mRo0aAAQFBbFv3z4+/vhjvvrqKyNHZjpiY2NJS0ujcePG+nUajYY///yTzz77jJycHMzNzY0YoelydnamZs2anDhxotTPJXXkRqJWqwkKCiImJka/TqvVEhMTI/V04qEoisLIkSOJjo5m8+bNVK1a1dghlStarZacnBxjh2FS2rdvz+HDh4mLi9N/mjRpQr9+/YiLi5Mkfh+ZmZmcPHkSLy+vUj+XPJEb0dixYwkPD6dJkyYEBwczd+5csrKyGDRokLFDMymZmZkG32pPnz5NXFwcrq6uVK5c2YiRmZaIiAiWLl3Kzz//jIODA6mpqQA4OTlhY2Nj5OhMy6RJk+jcuTOVK1fmxo0bLF26lK1bt/L7778bOzST4uDgUKCNhZ2dHRUqVJC2F/8xfvx4wsLC8PPz48KFC0RGRmJubk6fPn1K/dySyI2od+/eXLp0iWnTppGamkrDhg3ZsGFDgQZwT7r9+/fz9NNP65fHjh0LQHh4OIsXLzZSVKZn3rx5ALRt29Zg/aJFixg4cGDZB2TC0tLSGDBgACkpKTg5OdGgQQN+//13nnnmGWOHJsqp8+fP06dPHy5fvoy7uzutWrVi9+7duLu7l/q5ZfYzIYQQohyTOnIhhBCiHJNELoQQQpRjksiFEEKIckwSuRBCCFGOSSIXQgghyjFJ5EIIIUQ5JolcCCGEKMckkQshhBDlmCRyIYRJUalUrFmzxthhCFFuSCIXQugNHDgQlUpV4NOpUydjhyaEuAcZa10IYaBTp04sWrTIYJ2VlZWRohFCPIg8kQshDFhZWeHp6WnwcXFxAXSvvefNm0fnzp2xsbGhWrVq/Pjjjwb7Hz58mHbt2mFjY0OFChUYNmwYmZmZBmUWLlxIvXr1sLKywsvLi5EjRxpsT09Pp3v37tja2uLv788vv/yi33b16lX69euHu7s7NjY2+Pv7F/jiIcSTRBK5EKJYpk6dSo8ePTh06BD9+vXjpZdeIiEhAYCsrCxCQ0NxcXFh3759rFq1ik2bNhkk6nnz5hEREcGwYcM4fPgwv/zyCzVq1DA4x4wZM+jVqxfx8fF06dKFfv36ceXKFf35jx49ym+//UZCQgLz5s3Dzc2t7G6AEKZGEUKIf4WHhyvm5uaKnZ2dweftt99WFEVRAGX48OEG+4SEhCgjRoxQFEVRvv76a8XFxUXJzMzUb1+3bp1iZmampKamKoqiKN7e3sqbb755zxgAZcqUKfrlzMxMBVB+++03RVEUJSwsTBk0aFDJXLAQjwGpIxdCGHj66af1c5vf5urqqv+5efPmBtuaN29OXFwcAAkJCQQGBmJnZ6ff3rJlS7RaLYmJiahUKi5cuED79u3vG0ODBg30P9vZ2eHo6EhaWhoAI0aMoEePHhw4cICOHTvSrVs3WrRo8VDXKsTjQBK5EMKAnZ1dgVfdJcXGxqZI5SwtLQ2WVSoVWq0WgM6dO3P27FnWr1/Pxo0bad++PREREXzwwQclHq8Q5YHUkQshimX37t0FluvUqQNAnTp1OHToEFlZWfrtO3bswMzMjFq1auHg4ECVKlWIiYl5pBjc3d0JDw9nyZIlzJ07l6+//vqRjidEeSZP5EIIAzk5OaSmphqss7Cw0DcoW7VqFU2aNKFVq1b88MMP7N27lwULFgDQr18/IiMjCQ8PZ/r06Vy6dIlRo0bRv39/PDw8AJg+fTrDhw+nYsWKdO7cmRs3brBjxw5GjRpVpPimTZtGUFAQ9erVIycnh7Vr1+q/SAjxJJJELoQwsGHDBry8vAzW1apVi2PHjgG6FuXLly/n1VdfxcvLi2XLllG3bl0AbG1t+f333xk9ejRNmzbF1taWHj16MGfOHP2xwsPDyc7O5qOPPmL8+PG4ubnRs2fPIsenVquZNGkSZ86cwcbGhtatW7N8+fISuHIhyieVoiiKsYMQQpQPKpWK6OhounXrZuxQhBD/kjpyIYQQohyTRC6EEEKUY1JHLoQoMqmJE8L0yBO5EEIIUY5JIhdCCCHKMUnkQgghRDkmiVwIIYQoxySRCyGEEOWYJHIhhBCiHJNELoQQQpRjksiFEEKIckwSuRBCCFGO/T/JWPzgLBtYKQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_accs))\n",
    "examples_seen_tensor = torch.linspace(0, examples_seen, len(train_accs))\n",
    "\n",
    "plot_values(epochs_tensor, examples_seen_tensor, train_accs, val_accs, label=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90aba699-21bc-42de-a69c-99f370bb0363",
   "metadata": {},
   "source": [
    "- Based on the accuracy plot above, we can see that the model achieves a relatively high training and validation accuracy after epochs 4 and 5\n",
    "- However, we have to keep in mind that we specified `eval_iter=5` in the training function earlier, which means that we only estimated the training and validation set performances\n",
    "- We can compute the training, validation, and test set performances over the complete dataset as follows below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "UHWaJFrjY0zW",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UHWaJFrjY0zW",
    "outputId": "e111e6e6-b147-4159-eb9d-19d4e809ed34"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 86.63%\n",
      "Validation accuracy: 89.26%\n",
      "Test accuracy: 85.67%\n"
     ]
    }
   ],
   "source": [
    "train_accuracy = calc_accuracy_loader(train_loader, model, device)\n",
    "val_accuracy = calc_accuracy_loader(val_loader, model, device)\n",
    "test_accuracy = calc_accuracy_loader(test_loader, model, device)\n",
    "\n",
    "print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
    "print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6882649f-dc7b-401f-84d2-024ff79c74a1",
   "metadata": {},
   "source": [
    "- We can see that the training and validation set performances are practically identical\n",
    "- However, based on the slightly lower test set performance, we can see that the model overfits the training data to a very small degree, as well as the validation data that has been used for tweaking some of the hyperparameters, such as the learning rate\n",
    "- This is normal, however, and this gap could potentially be further reduced by increasing the model's dropout rate (`drop_rate`) or the `weight_decay` in the optimizer setting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74d9ad7-3ec1-450e-8c9f-4fc46d3d5bb0",
   "metadata": {},
   "source": [
    "## 5.8 Using the LLM as a spam classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72ebcfa2-479e-408b-9cf0-7421f6144855",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Sangwan70/Building-an-LLM-From-Scratch/refs/heads/main/part_5/images/overview-4.webp\" width=600px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd5408e6-83e4-4e5a-8503-c2fba6073f31",
   "metadata": {},
   "source": [
    "- Finally, let's use the finetuned GPT model in action\n",
    "- The `classify_review` function below implements the data preprocessing steps similar to the `SpamDataset` we implemented earlier\n",
    "- Then, the function returns the predicted integer class label from the model and returns the corresponding class name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "aHdn6xvL-IW5",
   "metadata": {
    "id": "aHdn6xvL-IW5"
   },
   "outputs": [],
   "source": [
    "def classify_review(text, model, tokenizer, device, max_length=None, pad_token_id=50256):\n",
    "    model.eval()\n",
    "\n",
    "    # Prepare inputs to the model\n",
    "    input_ids = tokenizer.encode(text)\n",
    "    supported_context_length = model.pos_emb.weight.shape[0]\n",
    "    # Note: In the book, this was originally written as pos_emb.weight.shape[1] by mistake\n",
    "    # It didn't break the code but would have caused unnecessary truncation (to 768 instead of 1024)\n",
    "\n",
    "    # Truncate sequences if they too long\n",
    "    input_ids = input_ids[:min(max_length, supported_context_length)]\n",
    "\n",
    "    # Pad sequences to the longest sequence\n",
    "    input_ids += [pad_token_id] * (max_length - len(input_ids))\n",
    "    input_tensor = torch.tensor(input_ids, device=device).unsqueeze(0) # add batch dimension\n",
    "\n",
    "    # Model inference\n",
    "    with torch.no_grad():\n",
    "        logits = model(input_tensor)[:, -1, :]  # Logits of the last output token\n",
    "    predicted_label = torch.argmax(logits, dim=-1).item()\n",
    "\n",
    "    # Return the classified result\n",
    "    return \"spam\" if predicted_label == 1 else \"not spam\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f29682d8-a899-4d9b-b973-f8d5ec68172c",
   "metadata": {},
   "source": [
    "- Let's try it out on a few examples below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "apU_pf51AWSV",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "apU_pf51AWSV",
    "outputId": "d0fde0a5-e7a3-4dbe-d9c5-0567dbab7e62"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not spam\n"
     ]
    }
   ],
   "source": [
    "text_1 = (\n",
    "    \"You are a winner you have been specially\"\n",
    "    \" selected to receive $1000 cash or a $2000 award.\"\n",
    ")\n",
    "\n",
    "print(classify_review(\n",
    "    text_1, model, tokenizer, device, max_length=train_dataset.max_length\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "1g5VTOo_Ajs5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1g5VTOo_Ajs5",
    "outputId": "659b08eb-b6a9-4a8a-9af7-d94c757e93c2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not spam\n"
     ]
    }
   ],
   "source": [
    "text_2 = (\n",
    "    \"Hey, just wanted to check if we're still on\"\n",
    "    \" for dinner tonight? Let me know!\"\n",
    ")\n",
    "\n",
    "print(classify_review(\n",
    "    text_2, model, tokenizer, device, max_length=train_dataset.max_length\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf736e39-0d47-40c1-8d18-1f716cf7a81e",
   "metadata": {},
   "source": [
    "- Finally, let's save the model in case we want to reuse the model later without having to train it again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "mYnX-gI1CfQY",
   "metadata": {
    "id": "mYnX-gI1CfQY"
   },
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), \"../../gpt2/review_classifier.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba78cf7c-6b80-4f71-a50e-3ccc73839af6",
   "metadata": {},
   "source": [
    "- Then, in a new session, we could load the model as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "cc4e68a5-d492-493b-87ef-45c475f353f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_state_dict = torch.load(\"../../gpt2/review_classifier.pth\", map_location=device, weights_only=True)\n",
    "model.load_state_dict(model_state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b70ac71-234f-4eeb-b33d-c62726d50cd4",
   "metadata": {
    "id": "5b70ac71-234f-4eeb-b33d-c62726d50cd4"
   },
   "source": [
    "## Summary and takeaways"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dafdc910-d616-47ab-aa85-f90c6e7ed80e",
   "metadata": {},
   "source": [
    "- See the [./gpt_class_finetune.py](./gpt_class_finetune.py) script, a self-contained script for classification finetuning\n",
    "- You can find the Example solutions in [./additional_examples.ipynb](./additional_examples.ipynb)\n",
    "- In addition, interested readers can find an introduction to parameter-efficient training with low-rank adaptation (LoRA) in [appendix C](../../appendix-C/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfae1c0d-9d8a-4aa6-aecf-ed7373723bf6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "V100",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.12 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}