t81_558_class_04_2_multi_class.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 4: Training for Tabular Data**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "* Part 4.1: Encoding a Feature Vector for Keras Deep Learning [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_1_feature_encode.ipynb)\n",
    "* **Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC** [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_2_multi_class.ipynb)\n",
    "* Part 4.3: Keras Regression for Deep Neural Networks with RMSE [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_3_regression.ipynb)\n",
    "* Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_4_backprop.ipynb)\n",
    "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_5_rmse_logloss.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC\n",
    "\n",
    "* **Binary Classification** - Classification between two possibilities (positive and negative).  Common in medical testing, does the person have the disease (positive) or not (negative).\n",
    "* **Classification** - Classification between more than 2.  The iris dataset (3-way classification).\n",
    "* **Regression** - Numeric prediction.  How many MPG does a car get? (covered in next video)\n",
    "\n",
    "In this class session we will look at some visualizations for all three.\n",
    "\n",
    "\n",
    "It is important to evaluate the level of error in the results produced by a neural network.  In this part we will look at how to evaluate error for both classification and regression neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Binary classification is used to create a model that classifies between only two classes.  These two classes are often called \"positive\" and \"negative\".  Consider the following program that uses the [wcbreast_wdbc dataset](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/datasets_wcbc.ipynb) to classify if a breast tumor is cancerous (malignant) or not (benign).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>mean_radius</th>\n",
       "      <th>mean_texture</th>\n",
       "      <th>mean_perimeter</th>\n",
       "      <th>mean_area</th>\n",
       "      <th>mean_smoothness</th>\n",
       "      <th>mean_compactness</th>\n",
       "      <th>mean_concavity</th>\n",
       "      <th>mean_concave_points</th>\n",
       "      <th>...</th>\n",
       "      <th>worst_radius</th>\n",
       "      <th>worst_texture</th>\n",
       "      <th>worst_perimeter</th>\n",
       "      <th>worst_area</th>\n",
       "      <th>worst_smoothness</th>\n",
       "      <th>worst_compactness</th>\n",
       "      <th>worst_concavity</th>\n",
       "      <th>worst_concave_points</th>\n",
       "      <th>worst_symmetry</th>\n",
       "      <th>worst_fractal_dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  mean_radius  mean_texture  mean_perimeter  mean_area  \\\n",
       "0    842302         M        17.99         10.38          122.80     1001.0   \n",
       "1    842517         M        20.57         17.77          132.90     1326.0   \n",
       "2  84300903         M        19.69         21.25          130.00     1203.0   \n",
       "3  84348301         M        11.42         20.38           77.58      386.1   \n",
       "4  84358402         M        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   mean_smoothness  mean_compactness  mean_concavity  mean_concave_points  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "   ...  worst_radius  worst_texture  worst_perimeter  worst_area  \\\n",
       "0  ...         25.38          17.33           184.60      2019.0   \n",
       "1  ...         24.99          23.41           158.80      1956.0   \n",
       "2  ...         23.57          25.53           152.50      1709.0   \n",
       "3  ...         14.91          26.50            98.87       567.7   \n",
       "4  ...         22.54          16.67           152.20      1575.0   \n",
       "\n",
       "   worst_smoothness  worst_compactness  worst_concavity  worst_concave_points  \\\n",
       "0            0.1622             0.6656           0.7119                0.2654   \n",
       "1            0.1238             0.1866           0.2416                0.1860   \n",
       "2            0.1444             0.4245           0.4504                0.2430   \n",
       "3            0.2098             0.8663           0.6869                0.2575   \n",
       "4            0.1374             0.2050           0.4000                0.1625   \n",
       "\n",
       "   worst_symmetry  worst_fractal_dimension  \n",
       "0          0.4601                  0.11890  \n",
       "1          0.2750                  0.08902  \n",
       "2          0.3613                  0.08758  \n",
       "3          0.6638                  0.17300  \n",
       "4          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "display(df[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curves\n",
    "\n",
    "ROC curves can be a bit confusing.  However, they are very common.  It is important to know how to read them.  Even their name is confusing.  Do not worry about their name, it comes from electrical engineering (EE).\n",
    "\n",
    "Binary classification is common in medical testing.  Often you want to diagnose if someone has a disease.  This can lead to two types of errors, know as false positives and false negatives:\n",
    "\n",
    "* **False Positive** - Your test (neural network) indicated that the patient had the disease; however, the patient did not have the disease.\n",
    "* **False Negative** - Your test (neural network) indicated that the patient did not have the disease; however, the patient did have the disease.\n",
    "* **True Positive** - Your test (neural network) correctly identified that the patient had the disease.\n",
    "* **True Negative** - Your test (neural network) correctly identified that the patient did not have the disease.\n",
    "\n",
    "Types of errors:\n",
    "\n",
    "![Type of Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_4_errors.png \"Type of Error\")\n",
    "\n",
    "Neural networks classify in terms of probability of it being positive. However, at what probability do you give a positive result?  Is the cutoff 50%? 90%?  Where you set this cutoff is called the threshold.  Anything above the cutoff is positive, anything below is negative.  Setting this cutoff allows the model to be more sensitive or specific:\n",
    "\n",
    "More info on Sensitivity vs Specificity: [Khan Academy](https://www.youtube.com/watch?v=Z5TtopYX1Gc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3wU1frH8c9Jp4ReJRSRmgSIoSlFkN5EekKRojT1qnjBXn56vXavIiJFQBQJJpGm0kFEpROQFkB6CYj0DiFlfn+cJSAmkITdPVue9+u1ryyb3ZlvJsuTs2dmnlGWZSGEEML5fEwHEEIIbyUFWAghDJECLIQQhkgBFkIIQ6QACyGEIVKAhRDCECnATqCUekUplaiU2qyU2qiUqm/HZa+0fa2glOp1w+N1lFKj7LUee1FKpdm2wSal1AalVINsvOaCM7Jlst43lFIjTKz7Vhz8fpqnlCpku/+0Umq7UipGKdVRKfWivdYjND/TATydUup+oAMQaVlWslKqGBBgr+VblnWtgFUAegHTbI8nAAn2Wo8dXbYsKwJAKdUaeBdoYjaS+3DC+6ndDf98AmhrWdY+279/sNd6hCYjYMcrDZywLCsZwLKsE5ZlHVFK1VZK/aKUWq+UWqiUKg2glFqmlHpfKbVWKbVTKdXY9niY7bGNtpFPZdvj10aH7wGNbd9/VinVVCk1Rynlo5Taf21UY3vNbqVUSaVUcaXUDKXUOtutoVO3DBQATtsy5VdK/WQbFW9RSj1885Ozeo5t9L9dKTXBNjJcpJTKY/teJaXUkhtG3PfYHn/O9jNvVkq9ecM6XlFK/aGUWgJUdcZGyKGs3k/7b3jfrFVKVQLI6nds25aTbdtxs1Kqq+3x/UqpYkqpcUBF4Afb+6m/Umq07TkllVKzbNt0U3Y+xYgsWJYlNwfegPzARmAnMAY92vMHVgLFbc+JAr603V8G/M92vx2wxHb/M6C37X4AkMd2/4Lta1Ngzg3rzfg38CkwwHa//g3LnAY0st0vB2x3wvZIs22PHcBZoLbtcT+ggO1+MWA3oG76GTN9Dnr0nwpE2L4XD/Sx3V8DdLbdDwLyAq2AL2yv9QHmAA8AtYEttucUsC1/hOn30O3eT7bH9wOv2O73veF3n+nvGHgfGHnDcgvfsJximdzvD4y23Y8Dhtnu+wIFTW8Xd73JFISDWZZ1QSlVG2gMPIh+8/4XCAcWK6VAv4n/vOFlM21f16OLC8Aq4BWlVAgw07KsXTmIEQe8DkwGom3/BmgBhNoyABRQSgVblnU+B8vOqRunIO4HpiilwtHF8B2l1ANAOlAGKAkcveG1WT0HYJ9lWRtt99cDFZRSwUAZy7JmAViWdcW23lboIvy77fn5gcpAMDDLsqxLtue53EfuzN5PN8zNfnvD109s9zP9Hdsej75huadzEKMZushjWVYa+g+pyAUpwE5ge5MuA5YppbYATwKJlmXdn8VLkm1f07D9jizLmqaUWgO0BxYqpQZalrU0mxFWAZWUUsWBTug/AKBHf/dblnU5pz+TPViWtco2h1kcPdovjh4Rpyil9qNHrDfqfYvnJN/wvDQgD7pgZ0YB71qWNf5vDyo1DHD55iiZvJ/6XfvWjU+zfc30d6x0RXb5n9XTyRywgymlql6br7WJALYDxW0jQJRS/kqpsNsspyKw17KsUeidITVvesp59AjuHyz9WXEW8DH6I+hJ27cWAf+6YR0R2f7B7EApVQ09+j8JFASO2Qrrg0D5TF6SnedksCzrHJCklOpkW1+gUiovsBB4VCmV3/Z4GaVUCeBXoLNSKo9tlPiQfX5S+8ni/XTAdj/qhq+rbPez+h3f/HjhHMT4CXjc9jpfpVSBHLxW3EAKsOPlB75WSm1TSm0GQtHTAd2A95VSm9BzerfbkREFbFVKbQSqAVNu+v5mINW2U+TZTF4fB/Th+vQDwNNAHdtOmG3A0Bz+bLmRx7ajcKMtSz/biC7GliUBPdLdkclrs/Ocmz0CPG3b9iuBUpZlLULPja6yjSCnA8GWZW2wZdoIzAB+u5Mf1EEyez+9YfteoO1T0jPAtfdAVr/j/wKFlVJbbe/BB3OQ4RngQdu2Ww/ccvAgsnZtJ4cQwo3ZpmPqWJZ1wnQWkX0yAhZCCENkBCyEEIbICFgIIQyRAiyEEIZIARZCCEOkAAshhCFSgIUQwpAcnYpcrFgxq0KFCg6KInLijz/+AKBqVVds2OVcsi2uk23hmtavX3/CsqziNz+eowJcoUIFEhJcscWs92natCkAy5YtM5rDFci2uE62hWtSSh3I7HGZghBCCEOkAAshhCFSgIUQwhDpByyEyJCSkkJSUhJXrlwxHcUtBQUFERISgr+/f7aeLwVYCJEhKSmJ4OBgKlSowA1X0RDZYFkWJ0+eJCkpibvvvjtbr5EpCCFEhitXrlC0aFEpvrmglKJo0aI5+vQgBVgI8TdSfHMvp9tOCrAQwqUopRg+fHjGvz/66CPeeOMNu6/nnXfe+du/GzS43UVp7E8KsBDCpQQGBjJz5kxOnHDsxT1uLsArV6506PoyIwVYCOFS/Pz8GDx4MJ988sk/vnf8+HG6du1K3bp1qVu3LitWrMh4vGXLlkRGRjJkyBDKly+fUcA7depE7dq1CQsL44svvgDgxRdf5PLly0RERNC7d28A8ufPD0BUVBTz5s3LWGf//v2ZMWMGaWlpPPfcc9StW5eaNWsyfvzfLqqdK1KAhRAu58knnyQmJoazZ8/+7fFnnnmGZ599lnXr1jFjxgwGDhwIwJtvvkmzZs3YsGEDnTt35uDBgxmv+fLLL1m/fj0JCQmMGjWKkydP8t5775EnTx42btxITEzM39YRHR1NXJy+du3Vq1f56aefaNeuHZMmTaJgwYKsW7eOdevWMWHCBPbt23dHP6cchiaEyNSwBcPYeHSjXZcZUSqCkW1G3vZ5BQoUoG/fvowaNYo8efJkPL5kyRK2bduW8e9z585x/vx5li9fzqxZswBo06YNhQsXznjOqFGjMr536NAhdu3aRdGiRbNcd9u2bXn66adJTk5mwYIFPPDAA+TJk4dFixaxefNmpk+fDsDZs2fZtWtXtg85y4wUYCGESxo2bBiRkZEMGDAg47H09HRWrVr1t6IM+hjczCxbtowlS5awatUq8ubNS9OmTW97mFhQUBBNmzZl4cKFxMXF0bNnz4x1fPbZZ7Ru3foOf7LrpAALITKVnZGqIxUpUoQePXowadIkHn30UQBatWrF6NGjee655wDYuHEjERERNGrUiPj4eF544QUWLVrE6dOnAT1KLVy4MHnz5mXHjh2sXr06Y/n+/v6kpKRketZadHQ0EydOJCEhga+++gqA1q1bM3bsWJo1a4a/vz87d+6kTJky5MuXL9c/o8wBCyFc1vDhw/92NMSoUaNISEigZs2ahIaGMm7cOAD+7//+j0WLFhEZGcn8+fMpXbo0wcHBtGnThtTUVGrWrMlrr73Gfffdl7GswYMHU7NmzYydcDdq1aoVv/76Ky1atCAgIACAgQMHEhoaSmRkJOHh4QwZMoTU1NQ7+vlydFn6OnXqWNIP2DVI39frZFtcd6fbYvv27VSvXt1+gZwkOTkZX19f/Pz8WLVqFY8//jgbN9p3/jq7MtuGSqn1lmXVufm5MgUhhHB7Bw8epEePHqSnpxMQEMCECRNMR8oWKcBCCLdXuXJlfv/9d9MxckzmgIUQwhApwEIIYYgUYCGEMEQKsBBCGCIFWAjhUnx9fYmIiCA8PJzu3btz6dKlHC9j4MCBGacsu0LbyaxIARZCuJRrTXK2bt1KQEBAxskWOTFx4kRCQ0MB12g7mRUpwEIIl9W4cWN2794NwMcff0x4eDjh4eGMHKlPk7548SLt27enVq1ahIeHZ3Qxa9q0KQkJCS7TdjIrchywEMIlpaamMn/+fNq0acP69euZPHkya9aswbIs6tevT5MmTdi7dy933XUXc+fOBfhH+8r33nuP0aNHZ3pW3LW2k+3atctoOzl27Ni/tZ1MTk6mYcOGtGrV6o66nmVFCrAQInPDhoG9T+eNiICRt27yc23ECnoE/NhjjzF27Fg6d+6c0fimS5cu/Pbbb7Rp04YRI0bwwgsv0KFDBxo3bpztKM5sO5kVKcBCCJdybQ74Rln1rKlSpQrr169n3rx5vPTSS7Rq1YrXX389W+txZtvJrEgBFkJk7jYjVWd64IEH6N+/Py+++CKWZTFr1iy++eYbjhw5QpEiRejTpw/58+fPaB15I1doO5kVKcBCCJcXGRlJ//79qVevHqAPM7v33ntZuHAhzz33HD4+Pvj7+zN27Nh/vPZa28nIyMh/XH6oVatW9O3bl44dO/6t7eT+/fuJjIzEsiyKFy/O7NmzHfJzSTtKNyUtGK+TbXGdt7ajdCU5aUcph6EJIYQhUoCFEMIQKcBCCGGIFGAhxN/kZL+Q+LucbjspwEKIDEFBQZw8eVKKcC5YlsXJkycJCgrK9mvkMDQhRIaQkBCSkpI4fvy46ShuKSgoiJCQkGw/XwqwECKDv7+/Q065FZmTKQghhDBECrAQQhgiBVgIIQyRAiyEEIZIARZCCEOkAAshhCFSgIUQwhApwEIIYYgUYCGEMEQKsBBCGCIFWAghDJECLIQQhkgBFkIIQ6QAu5nTl0+zYPcC9p/Zz55Te4jdGsuBMwekf6u3O3EC5syBfftg716YMQOOHDGdStyGtKN0E0nnknhy3pP88McP+oGzoFD0nNETgOrFqjO+w3gal29sMKVwut27YcgQWLr0+mNKQbdu+n7t2vDFFxAZaSafuCUZAbs4y7KYtGESYWPCWLxnMa80foWf+v5Eo7KNaFyuMQmDEvis7WdcSb1Ck6+a8PT8p7l49aLp2MLR0tLgk0+gZk1Yvx7eegt++QUaN4ZGjWD1avjf//QouF49ePVVSE42nVrcREbALiwtPY3eM3sTlxhHk/JNmNRxEvcUuQeA//j8B4Dad9Wm9l216R/Rn5d/epnP1n7Gwj0LWdZvGaWDS5uMLxzlyhV4+GFYtAg6dIBx46BMGf09H9uYqn59fRswAJ59Ft5+G+bNg59+gsKFzWUXfyMjYBdlWRZD5gwhLjGOt5u9zdJ+SzOKb2byB+RnVNtRLO27lMPnDtN6amtOXz7txMTCKVJToVcvXXzHjoUffrhefDNTuDB89RXMng2JibpgX7rktLji1qQAu6gXl7zIpN8n8WrjV3m58cv4qOz9qh68+0G+j/6eP07+QYdvO8h0hCexLBg6FGbNgk8/1feVyt5rH34Ypk3TUxPdusHVq47NKrJFCrAL+mTVJ3yw8gMer/M4/3nwPzl+ffOKzfm267esTlpN9++6k26lOyClcLpXX4VJk+C11+Dpp3P++q5d9XTF/Pnw6KO6oAujpAC7mN///J3nlzxP52qd+aztZ6jsjnBu0qV6Fz5v9znzd8/n09Wf2jmlcLqff4Z33oGBA+HNN3O/nEGD4D//gZgYmDrVfvlErkgBdiHJqcn0nd2X4nmLM7HjRHx9fO9oeUNqD6Fj1Y689NNL7Dixw04phdOdO6d3plWurKcecvlHOcPLL0PDhvDUU5CUZJ+MIlekALuQN395k63HtjLhoQkUyVPkjpenlGJ8h/HkC8hHv9n9SE1PtUNK4XQjRsChQ3pnWt68d748X1+9rJQUPaKWqQhjpAC7iDVJa3h/xfs8GvEo7au0t9tyS+UvxZh2Y1h7eC0frPjAbssVTrJgAUyYoItwgwb2W26lSvDBB7BwIUycaL/lihyRAuwC0tLTGPTjIMoEl+Hj1h/bfflR4VH0COvBG8veYM+pPXZfvnCQy5dh8GAIDb2zed+sPP44NG8Ow4fDX3/Zf/nitqQAu4Apm6aw5dgWPm79MQWDCjpkHSNbj8Tf159Xlr7ikOULB/jsMz318PnnEBRk/+X7+OhlX7qkz6QTTicF2LBLKZd47efXqF+mPl2rd3XYekoHl2b4/cOJS4xj3eF1DluPsJOTJ/VRD+3bQ9OmjltP1ap6lD1+POza5bj1iExJATZs1JpRHD5/mA9afpDrQ86y67kGz1E8b3GeX/K8dE9zde+8A+fPw3vvOX5d//d/EBioj44QTiUF2KATl07w7vJ3eajKQzxQ/gGHry84MJjXm7zOsv3LmL97vsPXJ3Jp/34YPRr69YPwcMevr2RJvZNv+nRYs8bx6xMZpAAb9Pavb3Ph6gXea+GEUY7N4NqDqVSkEi8seYG09DSnrVfkwGuv6fnZ/+T8LMhcGz5cF+Lnn5fD0pxICrAhf134i7EJY+lbqy+hxUOdtt4A3wD+++B/2XpsK7N2zHLaekU27dqlz1J76ikICXHeeoODdeH/9Vd9E04hBdiQz9Z+xtW0q7zY8EWnr7tbaDcqFanEBys+kLlgV/O//0FAAPz7385f96OPQokS8P77zl+3l5ICbMD55PN8vu5zOlXrRNViVZ2+fl8fX0bcP4J1R9axbP8yp69fZOHoUX2GWr9+UKqU89efJ49u8jN/Pmze7Pz1eyEpwAZM2DCBM1fO8ELDF4xl6BfRj5L5SvL+ChntuIxRo3SbyBEjzGV44gnIl0+fJSccTgqwk11Nu8onqz+hSfkm1A+pbyxHkF8Qz9R/hoV7FrLx6EZjOYTNuXMwZoxuGVm5srkchQvr44JjY+HAAXM5vIQUYCf7dsu3JJ1LMjr6vebxuo8THBAsPSJcwRdfwNmz8IL59wXPPqs7rn1s/9Pixd9JAXYiy7L4aNVH1ChRgzaV2piOQ6GgQgyprS97dOCMjHaMSUmBkSOhWTOoU8d0GihbFnr31k16Tp0yncajSQF2ol8P/MrWY1sZdt8wh5/1ll1P1X8KgPHrxxtO4sW+/x4OH4Zhw0wnue7ZZ3WPiK+/Np3Eo0kBdqKxCWMpFFSI6PBo01EylCtYjg5VOjDp90kkp8ply40YOxbKl4d27Uwnua5WLbj/fp0tXS5p5ShSgJ3k6IWjzNg+gwERA8jrb4em2nb0RJ0nOHbxGDO3zzQdxfvs2AFLl8KQIbpRuit54gl9YsjSpaaTeCwpwE4yccNEUtNTGVpnqOko/9DynpbcU/gexiSMMR3F+4wbB/7+8NhjppP8U7duUKyYPjpDOIQUYCdITU/li/Vf0KJiC6oUrWI6zj/4KB+G1hnK8oPL2fLXFtNxvMfFi/rEi27d9BloriYoSJ8d98MPcu04B5EC7ARzd87l0LlDPFHnCdNRsjQgYgCBvoGMTRhrOor3iI3Vh549/rjpJFkbMkTPAU+YYDqJR5IC7ARjEsZQJrgMD1V9yHSULBXNW5So8Ci+2fwN55LPmY7jHcaM0e0mGzUynSRrFStC27b6OOWUFNNpPI4UYAc7cOYAi/csZmDkQPx8/EzHuaWhtYdy4eoF4hPjTUfxfL//Dhs26BGmixySmKWhQ3WfinnzTCfxOFKAHezrTV9jYdE/or/pKLd1X8h9VC1alckbJ5uO4vkmT9Zdz3r1Mp3k9tq21b2CJ8v7wt6kADtQupXOVxu/otndzahQqILpOLellGJAxABWHlrJHyf+MB3HcyUn656/nTpBkSKm09yenx888gjMnQvHjplO41GkADvQrwd+Zd+ZfQyIGGA6SrY9UusRfJQPX2+SM6Ac5scf9Sm+A9znfcGAAZCaClOnmk7iUaQAO9DkjZMpEFiALtW7mI6SbXcF30WbSm2YsmmKXLLIUSZPhjJloGVL00myLzQU6tXT2aWJv91IAXaQ88nnmb5tOlFhUS535tvtDIgYwOHzh1m8d7HpKJ7nyBFYsAD69nW9M99uZ8AA2LoV1q83ncRjSAF2kPjEeC6lXHKr6YdrHqryEEXyFJGdcY7wzTf6uNr+/U0nybnoaH1yhuyMsxspwA4yeeNkqhatyn0h95mOkmOBfoH0qdGH2Ttmc+qytCO0G8vSZ741agRVXO+MyNsqVAi6dIFp0+DKFdNpPIIUYAfYe3ovKw6toF+tfi7TdjKn+kf052raVb5L/M50FM+RkKCb7/TrZzpJ7vXvD2fO6CMixB2TAuwAMZtjAOhds7fhJLkXUSqC0OKhTN0ie73tZupUCAzUvR/cVbNm+oKhcjSEXUgBtjPLsojZEsMD5R+gXMFypuPkmlKK3jV6s/zgcvaf2W86jvtLTdW9Hzp00B/l3ZWvL/Tsqc+Kk6tl3DEpwHa2/s/1/HHyD/rU6GM6yh3rVUOfpTVtyzTDSTzAkiX6JIY+7v++oE8fffXm6dNNJ3F7UoDtLGZzDAG+AXQLdeOPmTYVClWgUblGTN08FUuO/bwzMTF65Nu2rekkd+7ee6FaNf0ziTsiBdiOUtNTiU2MpX3l9hTOU9h0HLvoXaM3209sl0vX34mLF2HWLOjeXc8Buzul9EU7f/0VDh40ncatSQG2o6X7lnL0wlH61PSAj5k23UO74+/jT8wWGe3k2vff6yLsCdMP11xrIjRNpqfuhBRgO4rZEkPBwIK0q+xCF1e8Q0XzFqVt5bZM2zJNTk3OrZgYfal3V+77m1MVK0KDBvpoCJmeyjUpwHZyKeUSM7fPpFtoN4L8gkzHsas+Nfrw54U/WbZ/meko7uf4cVi4UI8YfTzsv1ufPpCYCJs3m07itjzsHWHOvF3zuHD1QsaRA56kQ5UO5A/IT+zWWNNR3M+MGZCW5h59f3OqWzd9WFpcnOkkbksKsJ3Ebo2lZL6SNCnfxHQUu8vjn4dO1ToxY/sMrqZdNR3HvcTG6k5iNWqYTmJ/xYtDixb6Z5RpiFyRAmwH55LPMWfnHHqE9cDXx806XGVTdFg0p6+cZvEe6ZCWbYcP6yMFoqNd/7JDuRUdDfv2wdq1ppO4JSnAdvD9ju9JTksmOjzadBSHaXlPSwoHFSY2UaYhsi0+Xo8Mo6JMJ3GcTp30pZVi5X2RG1KA7SA2MZbyBctzf8j9pqM4TIBvAF2rd2X2jtlcTrlsOo57iI2FyEj37HyWXYUKQbt2eh44TY6SySkpwHfo5KWTLNqziKiwKLftfJZd0eHRXLh6gXm75Oq4t7V3r/5YHu25n4oyREfDn3/C8uWmk7gdKcB3aOb2maSmp3r09MM1TSs0pWS+kjINkR3Xjgzo0cNsDmfo0AHy5pVpiFyQAnyHYhNjqVK0ChGlIkxHcThfH1+6h3Znzs45nE8+bzqOa4uN1ScqlC9vOonj5csHHTvq5jwpKabTuBUpwHfg6IWjLNu/zCumH67pWaMnV1Kv8P0f35uO4rq2b9cnJ3jD9MM10dFw4gT89JPpJG5FCvAdmL5tOulWOlFhHryX+yb3hdxHSIEQ4hPjTUdxXXFx+rAzd268nlOtW0OBAvrID5FtUoDvQHxiPGHFwwgrEWY6itP4KB96hPZgwe4FnLlyxnQc12NZugg1aQKlS5tO4zxBQfDww7rr21U5WSe7pADn0uFzh1l+cLlXjX6viQqPIiU9he93yDTEP2zdqqcgPPnY36xERenrxS2Wk3WySwpwLk3fNh0Lix5hXrCX+yZ176pLhUIViEuUHgD/EB+vm+506WI6ifO1bKmPC5ZpiGyTApxLcYlx1CpZi6rFqpqO4nRKKXqE9mDx3sWcvHTSdBzXYVl6/rdZMyhRwnQa5wsIgM6dYfZsuWx9NkkBzoWDZw+yKmmVV45+r+kR1oPU9FRm75htOorr2LQJdu3yjmN/s9KjB5w7B4sWmU7iFqQA58J3id8BeHUBjiwdyT2F75FpiBvFxen2jN44/XBN8+ZQpIi0qMwmKcC5EL8tnsjSkVQqUsl0FGOUUvQI68HSfUs5fvG46TjmXTv6oUULKFrUdBpz/P31H6AffoDL0jPkdqQA59C+0/tYe3gtPUK9d/R7TVRYFGlWGjO3zzQdxbz163X/B2+efrgmKgouXID5800ncXlSgHPo2gkI3jz9cE3NkjWpUrSKTEOA/sjt7693Qnm7pk11s3aZhrgtKcA5FL8tnnpl6nF34btNRzFOKUVUWBS/HPiFoxeOmo5jzrXph1atoHBh02nM8/ODrl1hzhx9NWiRJSnAObD71G42/LnBK0++yEpUWBTpVjozts0wHcWcNWvg4EHvPPkiK1FRcOkSzJ1rOolLkwKcA9emH7qHdjecxHWElQgjtHgo8du8+OD7+Hh9DGzHjqaTuI7GjaFUKTkp4zakAOdAXGIcDco2oGzBsqajuJSosCh+O/AbR84fMR3F+dLTdZFp2xYKFjSdxnX4+upmRHPnwnlpXZoVKcDZtOPEDjb/tVmOfshEj7AeWFgZx0d7lZUr9cU35eiHf4qK0mfE/fij6SQuSwpwNsUnxqNQdAv1ohaD2VStWDVqlqzpndMQ8fG6E9hDD5lO4noaNIAyZWQa4hakAGdTXGIcjco1okyBMqajuKQeoT1YeWglh84eMh3FedLS4Lvv9EUpg4NNp3E9Pj7Qvbs+HvjsWdNpXJIU4GzYemwr245vk6MfbiEqXG8br2rU/ttvcPSoHP1wK1FRuj/w99K6NDNSgLMhbmscPspHph9uoVKRStQuXdu7TsqIi9PXQ2vf3nQS11W/vr4unpyUkSkpwLdhWRaxibE0u7sZJfOXNB3HpUWHR7PuyDr2nNpjOorjpaToi1B27KiLsMicUnoUvGgRnJTWpTeTAnwbG/7cwO5Tu4kO86ILLObStdOzvWIUvHSpvgilN114M7eioyE1FWZKz5CbSQG+jditsfj7+NO5upzjfzvlCpajYdmGxG6NNR3F8WJj9XG/rVubTuL6IiKgShW9zcTfSAG+hXQrnbjEOFpXak2RPEVMx3EL0eHRbDm2hcRjiaajOE5ysh7NdekCgYGm07g+pfQo+Oef4c8/TadxKVKAb2HVoVUcOndIph9yoFtoN3yUj2dPQyxYoK/6INMP2RcVpZsWTZ9uOolLkQJ8C7FbYwnyC6JjVTnHP7tK5S/FgxUeJC4xDsuyTMdxjNhYKFZMX/tNZE9oKNSsKdMQN5ECnIXU9FTit8XTvnJ7ggPlIPuciAqLYufJnWw8utF0FPu7eFFf7aFbN912UWRfdLQ+dfvAAdNJXIYU4Cws27+MYxePER0uHzNzqkv1Lvj5+DFtyzTTUezvxx91m0WZfsd5G+4AABjNSURBVMi5ayesyCg4gxTgLMRsiSE4IJj2leUg+5wqmrcobSq1ITYxlnQr3XQc+4qJgZAQ3W5R5EzFivrEjGke+Ic5l6QAZ+JyymVmbJtB19Cu5PHPYzqOW+pdozdJ55L49cCvpqPYz4kTegdcz566z4HIud69YfNm2LrVdBKXIO+iTMzdNZfzV8/Tu0Zv01HcVseqHcnnn4+YzTGmo9jPd9/pEwp6y/si13r00L2CZRQMSAHOVMyWmIy9+SJ38vrnpXP1zkzfPp3k1GTTcexj2rTre/NF7pQsCS1a6G2Z7mHTU7kgBfgmpy+fZt6ueUSHRePr42s6jlvrXaM3Z66cYf5uD7g8+YEDsHy5Hv0qZTqNe+vdW2/PVatMJzFOCvBNZmyfwdW0q/SuKR8z71SLii0oka8EMVs8YBri2kfmnj3N5vAEnTpBnjx6h6aXkwJ8k5gtMVQpWoXapWubjuL2/Hz8iAqL4sc/fuTsFTduyG1Zulg0aAB33206jfsLDoaHH9ZXykhJMZ3GKCnAN0g6l8Qv+3+hV3gvlHzMtIteNXqRnJbMzO1u3AlryxZITIRevUwn8Ry9eun2lAsXmk5ilBTgG8RsjsHCkukHO6pfpj6VilTim83fmI6Se1Om6LPe5MKb9tO6tT6d+xs3fl/YgRRgG8uy+HrT1zQs25BKRSqZjuMxlFL0rdmXn/f/zIEzbngKamoqTJ2qr3pRvLjpNJ4jIEDPp3//PZw+bTqNMVKAbRKOJLD9xHb61uprOorHeaTWIwDuOQpetAj++gv69TOdxPP07atbe3rxVZOlANtM2TSFQN/AjKs6CPupUKgCTco3YcqmKe7XIW3KFChSRF/5WNhX7dr6uOopU0wnMUYKMHA17Srfbv2WTtU6USiokOk4HqlfrX7sOrWL1UmrTUfJvjNnYPZs/VFZGq/bn1L6k8XKlbBrl+k0RkgBBubunMvJyyfpV0s+ZjpKt9Bu5PHLw9ebvjYdJfvi4/VHZJl+cJw+fXRfDS8dBUsBBqZsnkKp/KVoeU9L01E8VnBgMF2qdyEuMY4rqVdMx8meKVOgenWoU8d0Es9111361ORvvvHKU5O9vgCfuHSCuTvn0rtGb/x8pMG2I/Wr1Y8zV87w4x8/mo5ye7t3w4oVevQrx4Q7Vr9++tTkXz2oc142eX0Bnrp5KinpKTL94ATN7m5GSIEQvtz4pekotzd5sv5o3KeP6SSer1MnKFAAvnSD94WdeXUBtiyLiRsmUq9MPWqUrGE6jsfz9fFlQMQAFu5eyMGzB03HyVpqqi7AbdtCmTKm03i+vHn1mXHffad3fHoRry7Aq5NWk3g8kUGRg0xH8RqP3vsoAJN/n2w4yS3Mn68vnz5I3hdOM2gQXLnidQ16vLoAT9wwkXz++YgKizIdxWtUKFSBlve05MuNX5KWnmY6TuYmToRSpeTYX2eKjIR774UJE3TzIy/htQX4XPI5YhNj6RneU6567GSDIgdx8OxBFu9dbDrKPx05AnPnQv/+4O9vOo13GTQINm2CDRtMJ3Eary3AsVtjuZRyiYGRA01H8Todq3akWN5iTNww0XSUf/rqK0hLg8ceM53E+/TsqfsET5hgOonTeG0BnrhhIuElwqlXpp7pKF4nwDeAfrX68f0f33Ps4jHTca5LT4dJk+DBB6GSNGRyukKFoHt33fz+4kXTaZzCKwvwpqObWHdkHYMiB0nfX0MGRg4kNT2Vrze60JlxP/8Me/fCQPlUZMygQXD+vNc06PHKAjxm3RiC/ILoU1OO8TSlWrFqPFD+AcatH0e65SJnQI0ZA0WLQufOppN4r4YNdYOeMWNMJ3EKryvAZ66cYeqWqfQK70WRPEVMx/FqT9Z9kr2n97Jg9wLTUeDQId2b9rHH9DykMEMpeOIJSEiAtWtNp3E4ryvAX238ikspl3iy3pOmo3i9ztU6Uzp/aUavHW06Cowfr+eAH3/cdBLRt6++btxoF3hfOJhXFeB0K50x68Zwf8j9RJaONB3H6/n7+jOk9hDm757P7lO7zQVJTtZ73h96CCpUMJdDaMHBugjHxcHx46bTOJRXFeDFexaz69Qu/lXvX6ajCJvBtQfj5+PH2HVjzYWYPh2OHYMn5VORy3jySbh6VZ8U48G8qgB/vu5zSuQrQdfqXU1HETalg0vTtXpXvtz4JRevGjr06PPPoUoV3RZRuIbq1aFZMxg3Tvfm8FBeU4D3nd7HnJ1zGBw5mEA/ubqBK/lXvX9x5soZpm2Z5vyVb9gAq1bpEZeP1/x3cA//+hccPAhz5phO4jBe844btWYUvj6+DKkzxHQUcZOGZRsSUSqCkWtGOv+QtE8+gfz55aoXruihh6BcOf078lBeUYBPXz7NhA0TiA6PJqRAiOk44iZKKYbfP5xtx7c595C0Q4cgNlYf/F+woPPWK7LHzw+GDdON2j30kDSvKMDj14/nYspFRtw/wnQUkYWosChCCoTw4coPnbfSTz/VnbeeecZ56xQ5M3Cg/uP40UemkziExxfg5NRkPl3zKS0rtqRWqVqm44gs+Pv6M6z+MJbtX0bCkQTHr/DsWfjiC4iKgvLlHb8+kTvBwTB0KMyYoU8T9zAeX4CnbZnG0QtHea7Bc6ajiNsYVHsQBQIL8NFKJ4x2xo/XPQdGyKcil/f00+DrCx9/bDqJ3Xl0AU630vlo1UfUKlmLFhXlECNXVyCwAENrD+W7bd+x7/Q+x63o6lU9/dC8uW4CLlzbXXfpa/N9+SWcOGE6jV15dAGet2se245vY0SDEdL1zE08Xf9pfJUvH69y4GgnJkY3XpfRr/sYPhwuX9bHbHsQjy3AlmXxn1/+Q4VCFeSSQ26kTIEy9K3VlwkbJnDk/BH7ryA1Fd5+GyIioHVr+y9fOEZYGHTsCCNH6vl7D+GxBXj+7vmsO7KOVxu/ir+vXFrGnbzc+GXSrDTeX/6+/RceEwN79sAbb+jOW8J9/N//6asmjxplOondeGQBtiyLN5a9QYVCFehbq6/pOCKHKhauSL9a/Ri/frx9R8GpqfDWW3r027Gj/ZYrnCMyUv/ePv7YY0bBHlmAZfTr/hwyCpbRr/vzsFGwxxVgGf16BruPgmX06xk8bBTscQV47q65Mvr1ENdGwe/+9u6dL+ybb2T06ymujYI//dR0kjvmUQU4NT2VF5a8QKUilWT06wEqFq7IoxGPMm79OHad3JX7BV26BK+/DnXqyOjXE0RG6uv2ffgh/PWX6TR3xKMK8OTfJ7Pt+Dbeb/G+jH49xBtN3yDQN5CXfnop9wsZORKSknQ/ARn9eob33oMrV/QnGjfmMQX4wtULvPbzazQs25DO1eSqtp6idHBpnm/4PDO2z2DFwRU5X8Bff8G778LDD0OTJvYPKMyoUkVfv2/CBNi2zXSaXPOYAvzhig/56+Jf/K/V/+SsNw8z/P7h3BV8F8MXDceyrJy9+M039Ujpgw8cE06Y8/rrkC8fvPCC6SS55hEF+PC5w3y48kOiwqKoH1LfdBxhZ/kC8vHfB//LmsNr+G7bd9l/4fbtuuPZ0KF6xCQ8S7Fi8Mor+ooZS5eaTpMrHlGAn1/yPGlWGu80f8d0FOEgfWv1pVbJWoxYNIILVy/c/gXX+vzmy6dHSsIzPf20bif6zDOQkmI6TY65fQFetGcR07ZM46VGL1GxcEXTcYSD+Pr48nm7zzl07hBvLHvj9i/49ltYvFj3fShe3OH5hCFBQfqkjK1b3bJdpVsX4Mspl3li7hNULlKZFxu9aDqOcLCG5RoyKHIQI1ePZNPRTVk/8fRpePZZqFtX76gRnq1jR+jUSc/373NgG1MHcOsC/PZvb7Pn9B7GdRhHkF+Q6TjCCd5r8R5F8hRhyJwhpKWnZf6kl17SfWPHj9eNvIXnGzVK/66feEJPP7kJty3A245v44MVH9C3Vl+a3d3MdBzhJEXyFOGT1p+w5vAaxq8f/88nrFypC++wYdJs3ZuULQv//S8sWADf5WBHrWFuWYBT0lLoN7sfwYHBfNTSMy/WJ7LWq0YvWlRswQtLXmDPqT3Xv3HxIgwYoP8zvvmmuYDCjH/9C2rXhiefhKNHTafJFrcswG8se4OEIwl80eELiueTHSzeRinFpI6T8FW+9JnVBwvbR85//xt27YKvv4b8+c2GFM7n6wtTpsCFC/oPsRtMRbhdAf7twG+8u/xdBkQMoGtoV9NxhCHlCpZjXIdxrE5azYEzB/Sc7xdf6MsMPfig6XjClNBQ3SNiwQIYM8Z0mttyqwJ89spZHpn1CBULV+TTNu7fCUncmejwaPrU7MOfJw9g/bFDt5p86y3TsYRpTz4JbdvqP8Yufpqy2xRgy7IY+ONAks4lMbXLVIIDg01HEi5gdMuRhJ5UWGlpnJ70OQQGmo4kTFNKX0E5f36IitJTEi7KbQrwu8vfZfq26bzX4j3uC7nPdBzhIgq+9l8KXrbYXRS6bnmVlDT3OxtKOECpUjBtmh4B9+/vsvPBblGA5+ycw6tLX6V3jd4Mv3+46TjCVUyZoltNlilDgfJV+Xn/z4xYJJeaFzYtW+omTDNm6DMiXZCf6QC3s+PEDnrN6MW9pe9lwkMTpNOZ0Natg8GD9Q63tDRKKUWb+m34dM2nRJSKYMC9A0wnFK7g3/+GjRvhtdegZk2Xa8jv0iPgpHNJtJnahiC/IGZFzSKPfx7TkYQr2LkT2reH0qUhPj6jyfpHrT6i+d3NGTJnCPN3zTccUrgEpfTRMbVrQ8+esGqV6UR/47IF+PjF47T8piWnLp9ifu/5lCtYznQk4QoOHoQWLfT9hQt1S0IbPx8/ZvSYQXiJcLrGd+W3A78ZCilcSp48umXlXXdBu3aw6RZ9RJzMJQvw2StnaRPThv1n9jOn1xxq31XbdCThCv76S8/rnTsHixZl2uO3YFBBFvRZQLmC5ejwbQc2/LnBQFDhckqVgiVL9JERrVrpE3ZcgMsV4JOXTtJ6ams2/7WZGT1m8ED5B0xHEq4gKQmaNdNf587Vx/xmoUS+Eix+ZDGFggrR6ptWrD281olBhcsqX163KLUsve/ABY4RdqkCfOjsIRpPbszGoxuZ3n067Sq3Mx1JuIIdO6BhQzh0SBffhg1v+5KyBcuytO9SCgYVpNnXzVi0Z5ETggqXV60a/PQTpKVB48bG54RdpgBvO76NBl824Mj5Iyx6ZBEPV3vYdCThCtauhUaN9HXdfvkFmjbN9kvvKXIPKx5dQeWilWk/rT3TtkxzXE7hPmrU0F3zihSB5s1h3jxjUVyiAE/fNp37Jt5Hanoqv/T/RaYdhDZpEjzwABQoACtW5Kq9ZKn8pVjWbxkNyzak98zePL/4eVLTUx0QVriVu+/W76nq1eGhh+CddyA93ekxjBbglLQU/r3w33T/rjuhxUNZO3AttUrVMhlJuILLl+Gxx2DgQP0xcc0aqFQp14srGFSQhX0W8nidx/lw5Yc0n9Kcoxfco12hcKASJfSnqqgofXHPhx/WV1NxImMFOPFYIo0mN+KT1Z/wVL2n+HXAr5QtWNZUHOEq1q2DevX0ufyvvqq7Wtnhmm6BfoGMaT+GKZ2msO7wOiLGRTBz+0w7BBZuLX9+iImB0aP1YY0REfoIGydxegG+mnaVN5e9yb3j72XPqT3EdYtjVNtRBPgGODuKcCWXLunuVffdB6dO6Xm5t96y+yWFHqn1CGsGrqF0cGm6xnelW3w3GQ17O6V0B7XfftPHDLdurfsJnzrl8FU7rQBblsWs7bOoObYmb/zyBt3DurP9ye30COvhrAjCFaWnw9Spuo/r//4Hgwbpw4PatnXYKmuUrMHagWt5t/m7zNk5h2qjq/H+8ve5nHLZYesUbqB+fX3a8ssvwzff6CMmRo+Gq1cdtkqHF2DLsli6bykNvmxAl/guKKWY03MOMV1i5GoW3iw9XZ+dFBkJjzyi90j//DOMGwcFCzp89f6+/rzY6EU2Dd1E4/KNefGnF6n8WWUmrJ9Acmqyw9cvXFRQkG7ck5AAYWHw1FN6R11MDKTaf+etwwpwcmoyUzZNIfKLSJpPac6hs4eY+NBEtjy+hfZV2jtqtcLVXbqkL5oZFqb3Pp8/D99+q9/wOTjEzF6qFqvKjz1/5Jf+v1C2YFkGzxlM+ZHleeuXtzh+8bjT8wgXEREBS5fC/PkQHAx9+kDFivpqG2fO2G01di3AlmWx9vBanpr3FCGfhNBvdj+upl1lwkMT2PXULh6LfAw/H5dvwCbsLT1dz68NHgxlysDQoZAvn5562L4doqPBx+wRkQ+Uf4CVj65kUZ9FRJaO5PVlrxPySQjd4rsxe8dsrqY57mOocFFKQZs2sGEDfP893HMPPP+8fg/37q13EN/hqPiOq+GV1Cv8sv8X5u2ax5xdc9h7ei+BvoF0rNqRgZEDaVmxpbSQ9EYXL+oRxLx5+uy1Q4d00e3S5frhZS72vlBK0fKelrS8pyXbj29nXMI4YhNjmbF9BoWDCtOmUhvaV25P60qtKZa32O0XKDyDj49uY9mxI/z+u/4EFx+vG74XL64b/LRrp3tMFCqUo0UrKwed4uvUqWPN/GkmG49uZNWhVaw4tIK1h9eSnJZMkF8Qze5uRpdqXegW2o2CQY6fx/NmTW0f15ctW2Y0B6DPrd+/X785V67UB7ivXw8pKbrotmwJ3bpBp07633bmyG2Rmp7Kkr1LiN0ay/zd8zl28RgKRViJMBqVbUTDcg25t9S9VC1W1SU+3bnU+8KTJSfr6Yn4eD0SPn1aF+patfSp8g0b6hOHKlUCX1+UUusty6pz82JyVID9QvystEFp+r6PH7VL16Zh2YY0r9icBys8KP16ncjp/9HS0uDoUT2SPXgQ9uzRHaV27oQtW3SHMoCAAKhbV78BW7bUI10HX6fNWdsi3Upn/ZH1LNyzkOUHl7MqaRXnkvXPHegbSFiJMKoWrUrlIpWpXLQy5QuWJ6RACGUKlHHaYZZSgA1ITdUnCy1aBMuXw+rVel8HQN68EB6OWrs20wKcoz/ZhfMU5s12b1KrZC3uLX0vef3z2ucHELmXmqrnWNPS/nlLTdW3lBR9KE1Kiu6pkJysv16+rG8XL+oLF54/rwvp6dP6duoUHD8Ox47pr2lpf193qVJQubLeQVGr1vVbUJCZbeFgPsqHumXqUrdMXQDS0tPYdnwbG49uZNNfm9j812ZWJa0idmssFn8f2BTNU5QS+UpQIl8JiuYtSuGgwhQOKkyBwAIEBwYTHBBMvoB85PHLQ17/vAT5BRHoF0igbyABvgEE+Abg7+uPn49fxs1X+eLr44uP8sFX6a8WFgrXmtrxeH5+10e9oP/Pbdmi+w5v2gSbN2f50hxPQSQkJNxpXPd27JiejL8mO9svq+fc+Pi1+zd+vfn+Dbemtpcty0H0bPH3h8KFr99KlLh+CwmBsmX17Z579FlELsDVRn3JqcnsO7OPg2cPcujsIQ6dO8Sxi8cybqcun+L0ldOcvnyay6l2PvZ4MuQLyMeFna57JWBvZJcpCKXUceCAPYPlQjHghOEMrkK2xXWyLa6TbXGdq2yL8pZl/ePEhxwVYFeglErI7C+JN5JtcZ1si+tkW1zn6tvCJdpRCiGEN5ICLIQQhrhjAf7CdAAXItviOtkW18m2uM6lt4XbzQELIYSncMcRsBBCeAS3LsBKqRFKKUsp5bUn5iulPlRK7VBKbVZKzVJK5exkdA+glGqjlPpDKbVbKfWi6TymKKXKKqV+VkptV0olKqWeMZ3JJKWUr1Lqd6XUHNNZsuK2BVgpVRZoCRw0ncWwxUC4ZVk1gZ3AS4bzOJVSyhf4HGgLhAI9lVKhZlMZkwoMtyyrOnAf8KQXbwuAZ4DtpkPcitsWYOAT4HnAqyexLctaZFnWtZ54q4EQk3kMqAfstixrr2VZV4FY4GHDmYywLOtPy7I22O6fRxefMmZTmaGUCgHaAxNNZ7kVtyzASqmOwGHLsjaZzuJiHgXmmw7hZGWAQzf8OwkvLTo3UkpVAO4F1phNYsxI9ADN+deazwHz/fOyoJRaApTK5FuvAC8DrZybyJxbbQvLsr63PecV9EfQGGdmcwGZdZ7x6k9FSqn8wAxgmGVZ50zncTalVAfgmGVZ65VSTU3nuRWXLcCWZbXI7HGlVA3gbmCTrdF7CLBBKVXPsiyPvLxtVtviGqVUP6AD0NzyvuMKk4CyN/w7BDhiKItxSil/dPGNsSxrpuk8hjQEOiql2gFBQAGl1FTLsvoYzvUPbn8csFJqP1DHsixXaLjhdEqpNsDHQBPLsrzuImZKKT/0zsfmwGFgHdDLsqxEo8EMUHpE8jVwyrKsYabzuALbCHiEZVkdTGfJjFvOAYu/GQ0EA4uVUhuVUuNMB3Im2w7IfwEL0Tud4r2x+No0BB4BmtneCxtto0Dhotx+BCyEEO5KRsBCCGGIFGAhhDBECrAQQhgiBVgIIQyRAiyEEIZIARZCCEOkAAshhCFSgIUQwpD/B/qlfbnDiDFSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import scipy.stats as stats\n",
    "import math\n",
    "\n",
    "mu1 = -2\n",
    "mu2 = 2\n",
    "variance = 1\n",
    "sigma = math.sqrt(variance)\n",
    "x1 = np.linspace(mu1 - 5*sigma, mu1 + 4*sigma, 100)\n",
    "x2 = np.linspace(mu2 - 5*sigma, mu2 + 4*sigma, 100)\n",
    "plt.plot(x1, stats.norm.pdf(x1, mu1, sigma)/1,color=\"green\")\n",
    "plt.plot(x2, stats.norm.pdf(x2, mu2, sigma)/1,color=\"red\")\n",
    "plt.axvline(x=-2,color=\"black\")\n",
    "plt.axvline(x=0,color=\"black\")\n",
    "plt.axvline(x=+2,color=\"black\")\n",
    "plt.text(-2.7,0.55,\"Sensitive\")\n",
    "plt.text(-0.7,0.55,\"Balanced\")\n",
    "plt.text(1.7,0.55,\"Specific\")\n",
    "plt.ylim([0,0.53])\n",
    "plt.xlim([-5,5])\n",
    "plt.legend(['Negative','Positive'])\n",
    "plt.yticks([])\n",
    "#plt.set_yticklabels([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import zscore\n",
    "\n",
    "# Prepare data - apply z-score to ALL x columns\n",
    "# Only do this if you have no categoricals (and are sure you want to use z-score across the board)\n",
    "x_columns = df.columns.drop('diagnosis').drop('id')\n",
    "for col in x_columns:\n",
    "    df[col] = zscore(df[col])\n",
    "\n",
    "# Convert to numpy - Regression\n",
    "x = df[x_columns].values\n",
    "y = df['diagnosis'].map({'M':1,\"B\":0}).values # Binary classification, M is 1 and B is 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "# Plot a confusion matrix.\n",
    "# cm is the confusion matrix, names are the names of the classes.\n",
    "def plot_confusion_matrix(cm, names, title='Confusion matrix', cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(names))\n",
    "    plt.xticks(tick_marks, names, rotation=45)\n",
    "    plt.yticks(tick_marks, names)\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    \n",
    "\n",
    "# Plot an ROC. pred - the predictions, y - the expected output.\n",
    "def plot_roc(pred,y):\n",
    "    fpr, tpr, _ = roc_curve(y, pred)\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "\n",
    "    plt.figure()\n",
    "    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operating Characteristic (ROC)')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Chart Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 426 samples, validate on 143 samples\n",
      "Epoch 1/1000\n",
      "426/426 - 1s - loss: 1.5214 - accuracy: 0.6291 - val_loss: 0.9492 - val_accuracy: 0.6224\n",
      "Epoch 2/1000\n",
      "426/426 - 0s - loss: 0.7552 - accuracy: 0.6338 - val_loss: 0.4821 - val_accuracy: 0.7063\n",
      "Epoch 3/1000\n",
      "426/426 - 0s - loss: 0.3770 - accuracy: 0.7770 - val_loss: 0.2008 - val_accuracy: 0.9021\n",
      "Epoch 4/1000\n",
      "426/426 - 0s - loss: 0.2153 - accuracy: 0.9343 - val_loss: 0.2071 - val_accuracy: 0.9510\n",
      "Epoch 5/1000\n",
      "426/426 - 0s - loss: 0.1518 - accuracy: 0.9460 - val_loss: 0.0856 - val_accuracy: 0.9720\n",
      "Epoch 6/1000\n",
      "426/426 - 0s - loss: 0.0896 - accuracy: 0.9718 - val_loss: 0.0636 - val_accuracy: 0.9790\n",
      "Epoch 7/1000\n",
      "426/426 - 0s - loss: 0.0715 - accuracy: 0.9742 - val_loss: 0.0560 - val_accuracy: 0.9860\n",
      "Epoch 8/1000\n",
      "426/426 - 0s - loss: 0.0617 - accuracy: 0.9859 - val_loss: 0.0536 - val_accuracy: 0.9860\n",
      "Epoch 9/1000\n",
      "426/426 - 0s - loss: 0.0537 - accuracy: 0.9859 - val_loss: 0.0494 - val_accuracy: 0.9790\n",
      "Epoch 10/1000\n",
      "426/426 - 0s - loss: 0.0490 - accuracy: 0.9859 - val_loss: 0.0485 - val_accuracy: 0.9860\n",
      "Epoch 11/1000\n",
      "426/426 - 0s - loss: 0.0451 - accuracy: 0.9859 - val_loss: 0.0502 - val_accuracy: 0.9860\n",
      "Epoch 12/1000\n",
      "426/426 - 0s - loss: 0.0409 - accuracy: 0.9859 - val_loss: 0.0488 - val_accuracy: 0.9860\n",
      "Epoch 13/1000\n",
      "426/426 - 0s - loss: 0.0365 - accuracy: 0.9859 - val_loss: 0.0500 - val_accuracy: 0.9860\n",
      "Epoch 14/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "426/426 - 0s - loss: 0.0328 - accuracy: 0.9906 - val_loss: 0.0514 - val_accuracy: 0.9860\n",
      "Epoch 00014: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a493a9110>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(1,activation='linear',kernel_initializer='random_normal'))\n",
    "model.compile(loss='binary_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "    patience=5, verbose=1, mode='auto', restore_best_weights=True)\n",
    "\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxN9f/A8dfbvgvTItTYMmNfJtmSkiUJSUmUREK2SJQWJSUJ+eIrP0mlIpWltKAs34o0su9SMShLjJ0x3r8/zpnpGrNc09w5d2bez8fjPuae/X0/c+553/P5nPM5oqoYY4wxScnmdQDGGGOCmyUKY4wxybJEYYwxJlmWKIwxxiTLEoUxxphkWaIwxhiTLEsUmZyIdBSRhV7HEUxE5ISIlPFgu6EioiKSI723HQgisklEGqViuVTvkyLSVETmpmbZ1BKR3CKyVUSuSs/tBhNLFOlIRH4XkdPugepPEZkuIgUCuU1V/UBVmwZyG75EpJ6IfCcix0UkWkQ+F5GK6bX9ROJZKiLdfMepagFV3RWg7d0gIrNF5JD7+deLyAARyR6I7aWWm7DK/Zt1qGolVV2awnYuSY7/cp98BRjps34VkZPud2qviIxJWNYi0lJEVrnzHRaRD0SkZIJ5iovI2yKy3913t4rIiyKSX1XPAtOAwamMOcOzRJH+7lLVAkB1oAbwtMfxpEpiv4pFpC6wEJgHXAuUBtYBPwTiF3yw/TIXkbLAT8AeoIqqFgbuBSKAgmm8Lc8+u1fbFpEbgcKqujLBpGrud+oWoD3wiM8y7YAPgTeBEKAScBb4XkSKuPMUBVYAeYG6qloQaAJcAZR1V/Uh0FlEcgfo4wU3VbVXOr2A34HbfYZHAQt8hnMDo4HdwF/AZCCvz/TWwFrgGPAr0NwdXxh4G9gP7AVeBrK70x4GvnffTwZGJ4hpHjDAfX8t8ClwEPgN6Osz3zDgE2CGu/1uiXy+/wGTEhn/FfCe+74REAU8Axxyy6SjP2Xgs+xg4E/gfaAI8IUb8xH3fUl3/hFALHAGOAFMcMcrUM59Px2YCCwAjuMc6Mv6xNMU2AZEA5OAZYl9dnfeGb7/z0Smh7rb7ux+vkPAUJ/ptXEOWEfd/+UEIJfPdAUeB3YAv7nj3sRJTMeA1cDNPvNnd8v5V/ezrQZKAcvddZ10y6W9O39LnP3rKPAjUDXBvjsYWI9zoM2Bz/7sxh7pxvEXMMYdv9vd1gn3VReffdKdpxKwCPjbXfaZJMrveWBqgnHx/0t3+GNgovtegD+ApxIskw3YCLzkDr8MbACypfD93QHc4vVxxIuX5wFkpVeCL1ZJd+d802f6OGA+UBTnF+jnwKvutNruwaqJu6OXAMLcaXOBt4D8wFXAKuAxd1r8lxJo6B5UxB0uApzGSRDZ3APJ80AuoAywC2jmzjsMiAHauPPmTfDZ8uEclG9N5HN3Afa77xsB54ExOEnhFpwDVgU/yiBu2dfcZfMCxYB73O0XBGYDc322vZQEB3YuTRR/u+WbA/gAmOlOC8E58LV1p/VzyyCpRPEn0CWZ/3+ou+3/c2OvhnPQDXen1wLquNsKBbYA/RPEvcgtm7jk2cktgxzAQDeGPO60QTj7WAWcg2Y1oFjCMnCHawIHgJtwEkxnnP01t8++uxYn0eT1GRe3P68AHnTfFwDqJPjMOXy29TD/7JMFcZLiQCCPO3xTEuU3GxiUzP8yzF3XEz7DCpROZF0vAivc9yuBF/34/s7H58dTVnp5HkBWerlfrBM4v+4U+Ba4wp0mOAdM31+zdfnnl+NbwNhE1nm1e7DxPfPoACxx3/t+KQXnF15Dd/hR4Dv3/U3A7gTrfhp4x30/DFiezGcr6X6msESmNQdi3PeNcA72+X2mfww850cZNALO4R4Ik4ijOnDEZ3gpKSeKqT7TWgBb3fcPxR1MfMpvT8L1+UyPwT3LS2J6qLvtkj7jVgH3JzF/f2BOgrhvS2EfO4JTFQPOmVDrJOZLmCj+CwxPMM823F/Q7r77SCL7c1yiWI5z8A1J4jMnlSg6AGv8/P4sAnok8jmOufuNAh/xT3Jr4I67ZH8BegA73Pc7Eq43ie1/ADzvT6yZ7WVtFOmvjTp1oI1wfvGEuOOvxPlVvFpEjorIUeBrdzw4v+R+TWR91wM5gf0+y72Fc2ZxEXX29pk4X06AB3B2/rj1XBu3Dnc9z+Akojh7kvlcR4ALQPFEphXHqWaJn1dVT/oM/4FzVpNSGQAcVNUzcQMikk9E3hKRP0TkGM4B64rLbDz+0+f9KZxfxLgxxX9mt/yiklnPYRL//H5tz20I/8K90OEYTsNtSIJlL/ofiMhAEdniNpwfxamGjFsmqX0mMdcDAxP8/0vhlEGi206gK3ADsFVEfhaRln5u93JiPELibT01ccqwPc4Pnvzu+Lh9LqV90t//W0GcarksxxKFR1R1Gc6v2dHuqEM41UCVVPUK91VYnUY6cL6kZS9dE3twzihCfJYrpKqVktj0R0A7Ebke50v1qc96fvNZxxWqWlBVW/iGncznOYlT/XBvIpPvwzl7ilNERPL7DF8H7POjDBKLYSBO1cpNqloIp3oNnF//ycbsh/04Z0rOCkXEdzgRi3GqwVLrv8BWoLz7WZ7hn88RJ/7ziMjNOO0G9wFFVPUKnOrJuGWS2mcSswcYkeD/n09VP0ps2wmp6g5V7YDzA+U14BP3f5xS+V9OjOtxklFi21dV/RhnH3zeHb0NJ7FftE+KSDac/1PcPrkYuNsdn5xwnIszshxLFN4aBzQRkeqqegGn7nps3PXaIlJCRJq5874NdBGRxiKSzZ0Wpqr7ca40ekNECrnTyorILYltUFXX4DT8TgW+UdW4X0irgGMiMlhE8opIdhGp7F5p4q8hOFeG9BWRgiJSRERexqk+ejHBvC+KSC73YNcSmO1HGSSmIE5yOepevfJCgul/4bS3pMYCoIqItHGv9HkcuCaZ+V8A6onI6yJyjRt/ORGZISJX+LG9gjjVKCdEJAzo6cf853H+nzlE5HmgkM/0qcBwESkvjqoiUsydlrBc/g/oISI3ufPmF5E7RcSvq7VEpJOIXOn+D+P2qVg3tgsk/T/4ArhGRPqLc79CQRG5KYl5v8Rp00rOSKC7iFzjngE+CTwrIg+4+/U1OOVSCBjrLjPGHX7X/QEVt9+NEZGqccM4bUMJr7jKEixReEhVDwLv4dTPg/PrcCew0q16WIzzaxlVXYXTKDwW51fjMpzqAnDq0nMBm3FOzz8h+VPpj4DbcS75i4slFrgLp47/N5xf91NxqjL8/TzfA81wGn/341Qp1QAaqOoOn1n/dOPch1P11UNVt6ZUBkkYh9MwfAjnS/x1gulv4pxBHRGR8f5+FvfzHML5NToKp3qiIs6VPWeTmP9XnKQYCmwSkWicM7ZInHaplDyJUx14HOfAPSuF+b/BuaJsO05Zn+Hi6qExOO0/C3ES0Ns4ZQVOm9O7bjXTfaoaidNmNQHnf7MTpy3BX81xPvMJnDK/X1XPqOopnKvPfnC3Vcd3IVU9jnOBxl04+8UO4NbENqCqvwDRySQSVHUDzndjkDs8C3gQeAJnH9nslkF9VT3szvM3UA+njeknETmOc7YR7ZYDOP+Xd9W5pyLLibv6xZh0Ic6dvDNUNbkqnKDkVk1E4VzOu8TreLIiEWkK9FLVNum4zdw4VU4NVfVAem03mATVDUvGBBu32usnnOqtQTj1/1my+iEYqOpCnDOk9NzmWZwLT7Isq3oyJnl1ca7KOYRTPdJGVU97G5Ix6cuqnowxxiTLziiMMcYkK8O1UYSEhGhoaKjXYRhjTIayevXqQ6p6ZcpzXirDJYrQ0FAiIyO9DsMYYzIUEfkjtcta1ZMxxphkWaIwxhiTLEsUxhhjkmWJwhhjTLIsURhjjEmWJQpjjDHJCliiEJFpInJARDYmMV1EZLyI7BSR9SJSM1CxGGOMSb1AnlFMx+l6OCl3AOXdV3ech7YYY4wJMgG74U5Vl4tIaDKztAbecx8uslJErhCR4u6DeNLdhz/tZt7avV5s2hhjAkJV2bt2GXvXLvtX6/HyzuwSXPyQlSh33CWJQkS645x1cN111wUkmHlr97J5/zEqFi+U8szGGBPkTh7azy+z3mD/hh8pXKLcv1qXl4ki4bOAIYnn66rqFGAKQERERMC6u61YvBCzHqsbqNUbY0y6UFUiIiI4tmsbb7zxBn379iVnzpypXp+XiSIKKOUzXBLn0ZjGGGNS4ccff6RKlSoULFiQqVOnEhISQqlSpVJeMAVeJor5QG8RmQncBERfbvtEWrYrWLWTMSajOnz4MEOGDGHq1Km88MILDBs2jBo1aqTZ+gOWKETkI6ARECIiUcALQE4AVZ0MfAm0wHl4+Smgy+VuIy3bFSoWL0Tr6iX+9XqMMSa9qCrvvfceTz75JEeOHGHQoEEMGjQozbcTyKueOqQwXYHH/+12rF3BGJNVDR48mNdff5169eoxefJkqlSpEpDtZLjnUew6eJL2b60ArLrIGJP1nD59mpMnTxISEkLXrl0pX748Xbt2JVu2wN0Wl+G68DgdExv/3qqLjDFZyddff03lypV57LHHAKhQoQKPPvpoQJMEZMAzirw5s1tVkzEmS9m3bx/9+/dn9uzZVKhQgd69e6fr9jNcojDGmKzk22+/5e677+bcuXMMHz6cQYMGkTt37nSNwRKFMcYEoZiYGHLmzEm1atVo0aIFL7/8MuXK/bs7rFMrw7VRGGNMZnbs2DH69evHzTffTGxsLCEhIcycOdOzJAGWKIwxJiioKrNnzyYsLIz//Oc/REREcPbsWa/DAqzqyRhjPHfw4EE6d+7MV199RY0aNZg3bx433nij12HFszMKY4zxWKFChTh06BDjxo1j1apVQZUkwBKFMcZ4Yvny5TRr1owTJ06QO3duVq5cSb9+/ciRI/gqeixRGGNMOjp06BBdunThlltuYfv27fz+++8AAb9p7t8I3siMMSYTUVWmTZtGhQoVmDFjBk8//TSbNm2icuXKXoeWouA7xzHGmExqxowZVKxYkcmTJ1OpUiWvw/GbnVEYY0yAnDp1imeffZaoqChEhE8//ZRly5ZlqCQBliiMMSYgvvzySypVqsSIESP4/PPPAShSpEhQt0UkJeNFbIwxQSwqKop27dpx5513kjdvXpYtW0bPnj29DutfsURhjDFpaMSIESxYsIBXXnmFtWvX0rBhQ69D+tfEedBcxlH0+nD9+48tXodhjDHxVq1aRd68ealSpQqHDx8mOjqaMmXKeB3WRURktapGpGZZO6MwxphUio6O5vHHH6dOnToMHToUgGLFigVdkvi3LFEYY8xlUlVmzpxJWFgYkydPpk+fPsyYMcPrsALG7qMwxpjLNGPGDB566CEiIiL44osvqFWrltchBZQlCmOM8cPZs2fZtWsX4eHh3HfffZw/f56HHnqI7Nmzex1awFnVkzHGpGDJkiVUq1aNZs2acfbsWXLnzk2XLl2yRJIASxTGGJOkAwcO8NBDD3HbbbcRExPDlClT0v151cHAqp6MMSYRO3fupHbt2pw4cYKhQ4cydOhQ8ubN63VYnrBEYYwxPo4dO0ahQoUoW7YsXbt25ZFHHiE8PNzrsDxlVU/GGAOcPHmSwYMHExoaGt+J3+uvv57lkwTYGYUxxvD555/Tu3dvdu/eTdeuXcmXL5/XIQUVSxTGmCzr/Pnz3HfffcyZM4dKlSrxv//9jwYNGngdVtCxqidjTJYT18ddjhw5KF68OCNHjuSXX36xJJEESxTGmCxl5cqVRERE8MsvvwAwceJEBg8eTK5cuTyOLHhZojDGZAlHjhyhZ8+e1KtXj7/++osjR454HVKGEdBEISLNRWSbiOwUkSGJTL9ORJaIyBoRWS8iLQIZjzEma5o1axZhYWFMmTKF/v37s2XLFho3bux1WBlGwBqzRSQ7MBFoAkQBP4vIfFXd7DPbs8DHqvpfEakIfAmEBiomY0zWtHXrVkJDQ/n666+pUaOG1+FkOIE8o6gN7FTVXap6DpgJtE4wjwKF3PeFgX0BjMcYk0WcOXOGF198Mf5Z1c888ww//vijJYlUCmSiKAHs8RmOcsf5GgZ0EpEonLOJPomtSES6i0ikiETGxMQEIlZjTCaxePFiqlatyrBhw1i2bBkAOXPmzDId+AVCIBOFJDIu4XNXOwDTVbUk0AJ4X0QuiUlVp6hqhKpG5MyZMwChGmMyur/++ouOHTvSpEkTVJWFCxcyevRor8PKFAKZKKKAUj7DJbm0aqkr8DGAqq4A8gAhAYzJGJNJLVq0iE8++YTnn3+eDRs20KRJE69DyjQCeWf2z0B5ESkN7AXuBx5IMM9uoDEwXUTCcRLFwQDGZIzJRNatW8eOHTto164dHTt2pH79+pQuXdrrsDKdgJ1RqOp5oDfwDbAF5+qmTSLykoi0cmcbCDwqIuuAj4CHNe6WSWOMScKJEycYOHAgtWrVYsiQIZw/fx4RsSQRIJLRjstFrw/Xv//Y4nUYxhiPzJ07lz59+hAVFUX37t159dVXKVq0qNdhBT0RWa2qEalZ1joFNMZkGBs2bODuu++mSpUqzJo1i3r16nkdUpZgXXgYY4JaTEwM3333HQBVqlRhwYIFrF692pJEOrJEYYwJWj/++CO1atWiSZMm7Ny5E4AWLVpgl8mnL0sUxpig8/fff9O9e3fq16/P0aNH+eyzzyhXrpzXYWVZ1kZhjAkqZ86coXr16uzbt4+BAwcybNgwChQo4HVYWZolCmNMUIiKiqJkyZLkyZOH4cOHU716dapVq+Z1WAarejLGeOz06dM8//zzlC1bNr4Tv86dO1uSCCJ+nVGISC7gOlXdGeB4jDFZyMKFC+nVqxe//vornTp1onbt2l6HZBKR4hmFiNwJbAAWucPVRWROoAMzxmRuffr0oVmzZmTLlo3Fixfz/vvvc/XVV3sdlkmEP2cULwE3AUsAVHWtiNjlB8aYyxYbGwtA9uzZqVOnDiEhIQwePJg8efJ4HJlJjj9tFDGqejTBuIzV74cxxnO//PILdevWZdKkSQB07NiRF154wZJEBuBPotgiIvcB2USktIiMA1YGOC5jTCZx/PhxnnjiCW688UZ2795N8eLFvQ7JXCZ/EkVvoBZwAfgMOAP0C2RQxpjMYeHChYSHh/Pmm2/y2GOPsXXrVtq1a+d1WOYy+dNG0UxVBwOD40aISFucpGGMMUnKlSsXV111FZ9++ik33XST1+GYVEqxm3ER+UVVayYYt1pVawU0siRYN+PGBK+YmBjGjBnDsWPHGDFiBAAXLlwgWza7ZctrAelmXESaAc2BEiIyxmdSIZxqKGOMiff999/To0cPNm3axL333hufICxJZHzJ/QcPABtx2iQ2+bwWAncEPjRjTEZw+PBhunXrxs0338zx48f5/PPP+fjjjy1BZCJJnlGo6hpgjYh8oKpn0jEmY0wGcvjwYWbOnMlTTz3F888/T/78+b0OyaQxfxqzS4jICKAiEH/Bs6reELCojDFBbcuWLXz88ce88MIL3HDDDezevdseR5qJ+XNuOB14BxCcKqePgZkBjMkYE6ROnTrF0KFDqVatGm+++SZRUVEAliQyOX8SRT5V/QZAVX9V1WeBWwMbljEm2Hz99ddUrlyZV155hQceeIBt27ZRsmRJr8My6cCfqqezIiLAryLSA9gLXBXYsIwxweTEiRM8+OCDFCtWjCVLltCoUSOvQzLpyJ8ziieAAkBfoD7wKPBIIIMyxngvNjaWGTNmEBsbS4ECBVi8eDHr1q2zJJEFpXhGoao/uW+PAw8CiIidbxqTia1evZrHHnuM1atXkzdvXu655x57kFAWluwZhYjcKCJtRCTEHa4kIu9hnQIakylFR0fTt29fateuzd69e5k5cyZt27b1OizjsSQThYi8CnwAdAS+FpGhOM+kWAfYpbHGZEL33HMPEyZMoFevXmzdupX27dvjNFGarCy5qqfWQDVVPS0iRYF97vC29AnNGJMedu3axZVXXknBggUZMWIE2bJl48Ybb/Q6LBNEkqt6OqOqpwFU9W9gqyUJYzKPc+fO8corr1CpUiVefvllAG666SZLEuYSyZ1RlBGRuK7EBQj1GUZVreLSmAxq+fLl9OjRgy1bttCuXTv69u3rdUgmiCWXKO5JMDwhkIEYY9LH2LFjGTBgAKGhoSxYsIAWLVp4HZIJcsl1CvhtegZijAmcCxcucPLkSQoWLMidd97JwYMHefbZZ8mXL5/XoZkMIMUHFwUbe3CRMZdn06ZN9OjRI/5JcyZr+jcPLgpoh/Ei0lxEtonIThEZksQ894nIZhHZJCIfBjIeY7KSU6dO8fTTT1O9enW2bNlCy5YtyWg/DE1w8KevJwBEJLeqnr2M+bMDE4EmQBTws4jMV9XNPvOUB54G6qvqERGxPqSMSQNr1qyhbdu2/P7773Tp0oVRo0YREhLidVgmg0rxjEJEaovIBmCHO1xNRP7jx7prAztVdZeqnsPpmrx1gnkeBSaq6hEAVT1wWdEbYy4Sd8Zw3XXXcd1117Fs2TKmTZtmScL8K/5UPY0HWgKHAVR1Hf51M14C2OMzHOWO83UDcIOI/CAiK0WkuR/rNcYkcP78ecaNG0fjxo2JjY2lWLFiLFu2jIYNG3odmskE/EkU2VT1jwTjYv1YLrH7/hNWkOYAygONgA7AVBG54pIViXQXkUgRiYyJifFj08ZkHatWraJ27do88cQT5MmTh2PHjnkdkslk/EkUe0SkNqAikl1E+gPb/VguCijlM1wSpxuQhPPMU9UYVf0N2IaTOC6iqlNUNUJVI3LmzOnHpo3J/E6cOMHjjz9OnTp1+Ouvv5g9ezYLFiygSJEiXodmMhl/EkVPYABwHfAXUMcdl5KfgfIiUlpEcgH3A/MTzDMXtxrL7aH2BmCXf6Ebk7XlzJmTpUuX0qdPn/g7rK0DPxMI/lz1dF5V77/cFavqeRHpDXwDZAemqeomEXkJiFTV+e60piKyGac6a5CqHr7cbRmTVezcuZOXXnqJiRMnUrBgQVavXk2ePHm8DstkcinecCciv+JUCc0CPlPV4+kRWFLshjuTFZ09e5ZRo0YxYsQIcuXKxYIFC7j55pu9DstkIAG94U5VywIvA7WADSIyV0Qu+wzDGJM6S5YsoVq1ajz//PO0adOGrVu3WpIw6cqvO7NV9UdV7QvUBI7hPNDIGBNgqsqIESOIiYnh66+/ZubMmVx77bVeh2WymBTbKESkAM6NcvcD4cA8oF6A4zImy7pw4QJvv/02zZs3p1SpUrz//vtcccUV5M2b1+vQTBblzxnFRpwrnUapajlVHaiqPwU4LmOypPXr19OgQQO6d+/O1KlTAShevLglCeMpf656KqOqFwIeiTFZ2IkTJ3jxxRcZO3YsRYoUYfr06Tz00ENeh2UMkEyiEJE3VHUg8KmIXHJplD3hzpi0M2zYMN544w26devGyJEjKVasmNchGRMvyctjRaS2qq4SkcaJTffqwUZ2eazJLPbs2cPJkycJCwvj0KFDbN26lQYNGngdlsmkAnJ5rKquct+Gq+q3vi+cRm1jTCqcP3+eMWPGEB4ezmOPPQZASEiIJQkTtPxpzH4kkXFd0zoQY7KClStXEhERwcCBA2nUqBHvvvuu1yEZk6Lk2ija41wSW1pEPvOZVBA4GujAjMlsFixYwF133cW1117LZ599Rps2baxvJpMhJHfV0yqcZ1CUxHlSXZzjwJpABmVMZqGq7Nu3jxIlSnD77bfz0ksv0a9fPwoWLOh1aMb4LcW+noKNNWabjGL79u306tWL7du3s3nzZgoUKOB1SCYLC0hjtogsc/8eEZG/fV5HROTv1AZrTGZ35swZhg0bRpUqVYiMjOTpp5+2G+ZMhpZc1VPc407tYbvG+OnPP/+kYcOG7Nixgw4dOjBmzBiuueYar8My5l9J7vLYuLuxSwHZVTUWqAs8BuRPh9iMyTDiHtF79dVX07BhQxYuXMiHH35oScJkCv5cHjsX5zGoZYH3cO6h+DCgURmTQVy4cIHJkydTtmxZoqKiEBGmTp1KkyZNvA7NmDTjT6K4oKoxQFtgnKr2AUoENixjgt+6deuoV68ePXv2pHz58vFnFcZkNv4kivMici/wIPCFOy5n4EIyJripKk8++SS1atVi165dvP/++yxevJjSpUt7HZoxAeHvndm34nQzvktESgMfBTYsY4KXiHDkyBG6du3Ktm3b6NSpk904ZzI1v+6jEJEcQDl3cKeqng9oVMmw+yiMF/744w/69evH888/T82aNblw4QLZsvn1gEhjgkJAn5ktIjcDO4G3gWnAdhGpn5qNGZPRxMTEMGrUKCpWrMiiRYvYtm0bgCUJk6X48+CisUALVd0MICLhwPtAqjKTMRnFjz/+yGOPPcbGjRtp3bo148eP57rrrvM6LGPSnT+JIldckgBQ1S0ikiuAMRkTFBYvXkx0dDRz586ldevWXodjjGdSbKMQkenAWZyzCICOQD5V7RzY0BJnbRQmUFSV999/nyuvvJI77riDs2fPEhMTY300mUwhoG0UQA/gV+ApYDCwC+fubGMyja1bt3LbbbfRuXNn3nnnHQBy585tScIYUqh6EpEqQFlgjqqOSp+QjEk/p0+f5pVXXuG1114jf/78vPXWW3Tr1s3rsIwJKsn1HvsMTvcdHYFFIpLYk+6MydA+//xzXn75Zdq3b8/WrVvp3r27XdFkTALJnVF0BKqq6kkRuRL4EufyWGMytD///JO1a9fSvHlz7r33XkJDQ6ldu7bXYRkTtJL76XRWVU8CqOrBFOY1JujFxsYyadIkKlSowIMPPsjp06cREUsSxqQguTOKMj7PyhagrO+zs1W1bUAjMyYN/fLLL/To0YOff/6Z22+/nUmTJtnDhIzxU3KJ4p4EwxMCGYgxgfLbb79Ru3ZtQkJC+PDDD7n//vutbyZjLkOSiUJVv03PQIxJS6rKhg0bqFq1KqVLl+add97hrrvu4oorrvA6NGMyHGt3MJnOb7/9RsuWLalRowbr168H4MEHH7QkYUwqBTRRiEhzEdkmIjtFZEgy87UTERUR6z/KpNq5cxHmw+QAABxdSURBVOcYOXIklSpVYtmyZYwePZqKFSt6HZYxGZ4/fT0BICK5VfXsZcyfHZgINAGigJ9FZL5vv1HufAWBvsBP/q7bmIRiY2OpV68eq1evpm3btowbN45SpUp5HZYxmYI/3YzXFpENwA53uJqI/MePddfGeXbFLlU9B8wEEutZbTgwCjjjf9jGOI4dOwZA9uzZeeSRR/j888/59NNPLUkYk4b8qXoaD7QEDgOo6jqcJ96lpASwx2c4igTP2haRGkApVf2CZIhIdxGJFJFIey6xAaexevr06ZQpU4Z58+YB0KtXL1q2bOlxZMZkPv4kimyq+keCcbF+LJfY9YfxXdWKSDacZ10MTGlFqjpFVSNUNSJnTntcd1a3efNmGjVqRJcuXQgLC6Ns2bJeh2RMpuZPotgjIrUBFZHsItIf2O7HclGA7/l/SWCfz3BBoDKwVER+B+oA861B2yRn1KhRVKtWjY0bNzJ16lSWL19O5cqVvQ7LmEzNn0TRExgAXAf8hXNA7+nHcj8D5UWktPugo/uB+XETVTVaVUNUNVRVQ4GVQCtVjbzMz2CygLjnplxzzTV07NiRrVu30rVrV+vAz5h0kOKDi/7VykVaAOOA7MA0VR0hIi8Bkao6P8G8S4EnU0oU9uCirGXfvn3069ePm2++mb59+3odjjEZ1r95cFGKl8eKyP/h07YQR1W7p7Ssqn6J0+us77jnk5i3UUrrM1lHXAd+Q4cOJSYmhnr16nkdkjFZlj/3USz2eZ8HuJuLr2YyJk2tXbuWbt26sXr1apo2bcqkSZOswdoYD6WYKFR1lu+wiLwPLApYRCbLi46OZt++fcyaNYt7773XOvAzxmN+35ntozRwfVoHYrIuVWX27Nns2LGDoUOHcsstt7Br1y7y5MnjdWjGGPy7M/uIiPztvo7inE08E/jQTFbw66+/0qJFC9q3b8+8efOIu6HSkoQxwSPZRCHOOX814Er3VURVy6jqx+kRnMm8zp49y4gRI6hcuTI//PADb775Jj/++CN2Q6UxwSfZqidVVRGZo6q10isgkzXs2bOH4cOHc9dddzFu3DhKlCiR8kLGGE/4c7fSKhGpGfBITKZ38OBBJkxwHpRYrlw5Nm/ezOzZsy1JGBPkkkwUIhJ3ttEAJ1lsE5FfRGSNiPySPuGZzODChQu8/fbbhIWFMWDAALZt2wZAmTJlPI7MGOOP5KqeVgE1gTbpFIvJhDZu3EjPnj35/vvvufnmm5k8eTIVKlTwOixjzGVILlEIgKr+mk6xmEzm3LlzNG3alHPnzjFt2jQefvhhuyfCmAwouURxpYgMSGqiqo4JQDwmE/juu++45ZZbyJUrFx9//DFhYWGEhIR4HZYxJpWSa8zODhTA6Q48sZcxF4mKiuKee+6hcePGvPfeewA0aNDAkoQxGVxyZxT7VfWldIvEZFjnz59nwoQJPPfcc8TGxvLqq6/SsWNHr8MyxqSRFNsojEnJgw8+yMyZM7njjjuYOHEipUuX9jokY0waSvJ5FCJSVFX/Tud4UmTPowgOR48eJUeOHBQoUIDvv/+eP//8k3vuuccaq40JUv/meRRJtlEEY5Iw3lNVZs6cSXh4OM899xzgtEO0a9fOkoQxmZQ9R9L4befOnTRr1owOHTpQsmRJOnXq5HVIxph0YInC+OXDDz+kcuXK/PTTT0yYMIGVK1dSq5Z1AWZMVpCa51GYLCQmJoacOXMSERFBu3btGDVqFNdee63XYRlj0lGSjdnByhqz08eBAwcYOHAgJ0+e5LPPPvM6HGPMvxSQxmyTNV24cIEpU6ZQoUIFZs2aRaVKlYiNjfU6LGOMh6zqycTbtWsXnTp1YsWKFTRq1Ij//ve/hIWFeR2WMcZjlihMvMKFC3P06FHeffddHnzwQbvc1RgDWNVTljd//nzatm1LbGwsxYoVY+PGjTz00EOWJIwx8SxRZFG7d++mTZs2tG7dmu3bt7N//34AsmWzXcIYczE7KmQx58+fZ/To0YSHh7Nw4UJee+011qxZQ8mSJb0OzRgTpKyNIouJjY1l6tSp3HbbbfznP/8hNDTU65CMMUHOziiygCNHjjB48GCOHz9O7ty5+eGHH5g/f74lCWOMXyxRZGKqygcffEBYWBhvvPEGS5YsAaBYsWLWWG2M8Zslikxq+/btNGnShE6dOhEaGkpkZCStWrXyOixjTAZkbRSZVP/+/YmMjGTSpEl0796d7Nmzex2SMSaDskSRiSxatIiwsDBKlSrFf//7X3Lnzs0111zjdVjGmAwuoFVPItJcRLaJyE4RGZLI9AEisllE1ovItyJyfSDjyaz+/PNPHnjgAZo2bcprr70GwPXXX29JwhiTJgKWKEQkOzARuAOoCHQQkYoJZlsDRKhqVeATYFSg4smMLly4wOTJkwkLC+PTTz/lhRdeYPTo0V6HZYzJZAJ5RlEb2Kmqu1T1HDATaO07g6ouUdVT7uBKwO76ugyvvvoqPXv2pFatWqxfv55hw4aRJ08er8MyxmQygWyjKAHs8RmOAm5KZv6uwFeJTRCR7kB3gALFy6ZVfBnS8ePHOXToEKVLl6ZHjx6ULl2aDh062OWuxpiACeQZRWJHrkSfkiQinYAI4PXEpqvqFFWNUNWInDlzpmGIGYeqMmfOHCpWrEj79u1RVYoVK8YDDzxgScIYE1CBTBRRQCmf4ZLAvoQzicjtwFCglaqeDWA8GdYff/xBq1ataNu2LUWLFmX8+PGWHIwx6SaQVU8/A+VFpDSwF7gfeMB3BhGpAbwFNFfVAwGMJcNasWIFt99+OwCjR4+mX79+5MhhVzUbY9JPwM4oVPU80Bv4BtgCfKyqm0TkJRGJu0X4daAAMFtE1orI/EDFk9EcO3YMgJo1a/LII4+wZcsWBg4caEnCGJPuRDXRZoOgVfT6cP37jy1ehxEwhw8fZsiQISxcuJBNmzZRoEABr0MyxmQCIrJaVSNSs6z19RQkVJX33nuPsLAw3nnnHdq3b2/tEMaYoGD1GEEgOjqaNm3asHTpUurWrcvkyZOpWrWq12EZYwxgicJTqoqIUKhQIUJCQpgyZQpdu3a1x5EaY4KKHZE88s0331CzZk2ioqIQEWbPns2jjz5qScIYE3TsqJTO9u/fz/3330/z5s05deoUBw7YVcHGmOBmiSIdTZw4kbCwMObOncuLL77I+vXrqVmzptdhGWNMsqyNIh2tXr2am266iYkTJ1K+fHmvwzHGGL/YGUUAHTt2jP79+7N69WoAJk2axDfffGNJwhiToViiCABV5ZNPPiE8PJzx48ezbNkyAPLkyWP3RhhjMhxLFGnst99+o2XLltx7771cddVVrFixggEDBngdljHGpJolijT2wQcfsHz5csaOHcvPP//MTTcl9wgOY4wJftbXUxr43//+x9mzZ7n99ts5e/YsBw8epGRJe1ifMSZ4WF9PHjl06BCPPPIIDRs25KWXXgIgd+7cliSMMZmKXR6bCqrK9OnTGTRoENHR0QwePJjnnnvO67BMkImJiSEqKoozZ854HYrJQvLkyUPJkiVJy6eBWqJIhS+//JJHHnmE+vXrM3nyZCpXrux1SCYIRUVFUbBgQUJDQ+1qN5MuVJXDhw8TFRVF6dKl02y9VvXkp1OnTvHDDz8A0KJFC+bNm8fy5cstSZgknTlzhmLFilmSMOlGRChWrFian8VaovDDV199ReXKlbnjjjs4evQoIkKrVq2sAz+TIksSJr0FYp+zI10y9u7dy7333kuLFi3InTs3n3/+OVdccYXXYRljTLqyRJGEAwcOULFiRb744gtefvll1q1bxy233OJ1WMZcluzZs1O9enUqV67MXXfdxdGjR+Onbdq0idtuu40bbriB8uXLM3z4cHwvl//qq6+IiIggPDycsLAwnnzySS8+QrLWrFlDt27dvA4jSYcPH+bWW2+lQIEC9O7dO8n5/v77b5o0aUL58uVp0qQJR44cAZw2h759+1KuXDmqVq3KL7/8AsDBgwdp3rx5unwGsERxib179wJw1VVXMXz4cDZu3MjQoUPJlSuXx5EZc/ny5s3L2rVr2bhxI0WLFmXixIkAnD59mlatWjFkyBC2b9/OunXr+PHHH5k0aRIAGzdupHfv3syYMYMtW7awceNGypQpk6axnT9//l+v45VXXqFPnz7pus3LkSdPHoYPH87o0aOTnW/kyJE0btyYHTt20LhxY0aOHAk4yXrHjh3s2LGDKVOm0LNnTwCuvPJKihcvHt9uGmh21ZMrOjqaZ599lrfeeouVK1dSs2ZN+vbt63VYJpN48fNNbN53LE3XWfHaQrxwVyW/569bty7r168H4MMPP6R+/fo0bdoUgHz58jFhwgQaNWrE448/zqhRoxg6dChhYWEA5MiRg169el2yzhMnTtCnTx8iIyMREV544QXuueceChQowIkTJwD45JNP+OKLL5g+fToPP/wwRYsWZc2aNVSvXp05c+awdu3a+CrdcuXK8cMPP5AtWzZ69OjB7t27ARg3bhz169e/aNvHjx9n/fr1VKtWDYBVq1bRv39/Tp8+Td68eXnnnXeoUKEC06dPZ8GCBZw5c4aTJ0/y3Xff8frrr/Pxxx9z9uxZ7r77bl588UUA2rRpw549ezhz5gz9+vWje/fufpdvYvLnz0+DBg3YuXNnsvPNmzePpUuXAtC5c2caNWrEa6+9xrx583jooYcQEerUqcPRo0fZv38/xYsXp02bNnzwwQeXlEsgZPlEoarMnj2b/v378+eff9K7d2/Kli3rdVjGpKnY2Fi+/fZbunbtCjjVTrVq1bponrJly3LixAmOHTvGxo0bGThwYIrrHT58OIULF2bDhg0A8VUmydm+fTuLFy8me/bsXLhwgTlz5tClSxd++uknQkNDufrqq3nggQd44oknaNCgAbt376ZZs2Zs2XJxjwyRkZEXXXUYFhbG8uXLyZEjB4sXL+aZZ57h008/BWDFihWsX7+eokWLsnDhQnbs2MGqVatQVVq1asXy5ctp2LAh06ZNo2jRopw+fZobb7yRe+65h2LFil203SeeeIIlS5Zc8rnuv/9+hgwZkuLnT8xff/1F8eLFAShevHj8A8327t1LqVKl4ucrWbIke/fupXjx4kRERPDss8+manuXK0snClWlbdu2zJ07l5o1azJ//nwiIlJ1h7sxybqcX/5p6fTp01SvXp3ff/+dWrVq0aRJE+Cf57Un5nKumlm8eDEzZ86MHy5SpEiKy9x7771kz54dgPbt2/PSSy/RpUsXZs6cSfv27ePXu3nz5vhljh07xvHjxylYsGD8uP3793PllVfGD0dHR9O5c2d27NiBiBATExM/rUmTJhQtWhSAhQsXsnDhQmrUqAE4Z0U7duygYcOGjB8/njlz5gCwZ88eduzYcUmiGDt2rH+FkwYS62Ip7v9z1VVXsW/fvnSJI0smipiYGHLmzImI0KBBA2677TZ69eoVv/Mak1nEtVFER0fTsmVLJk6cSN++falUqRLLly+/aN5du3ZRoEABChYsSKVKlVi9enV8tU5Skko4vuMSXtOfP3/++Pd169Zl586dHDx4kLlz58b/Qr5w4QIrVqwgb968yX4233U/99xz3HrrrcyZM4fff/+dRo0aJbpNVeXpp5/mscceu2h9S5cuZfHixaxYsYJ8+fLRqFGjRO9HCMQZxdVXXx1fpbR//36uuuoqwDmD2LNnT/x8UVFRXHvttYBTrsmVT1rKco3ZS5cupWrVqsybNw+AgQMH0qdPH0sSJlMrXLgw48ePZ/To0cTExNCxY0e+//57Fi9eDDhnHn379uWpp54CYNCgQbzyyits374dcA7cY8aMuWS9TZs2ZcKECfHDcVVPV199NVu2bImvWkqKiHD33XczYMAAwsPD43+9J1zv2rVrL1k2PDz8orr/6OhoSpQoAcD06dOT3GazZs2YNm1afBvK3r17OXDgANHR0RQpUoR8+fKxdetWVq5cmejyY8eOZe3atZe8UpskAFq1asW7774LwLvvvkvr1q3jx7/33nuoKitXrqRw4cLxVVTbt29Ptxt+s0yiOHjwIJ07d+bWW2/l7NmzF53CGpMV1KhRg2rVqjFz5kzy5s3LvHnzePnll6lQoQJVqlThxhtvjL+Es2rVqowbN44OHToQHh5O5cqV2b9//yXrfPbZZzly5AiVK1emWrVq8b+0R44cScuWLbntttviD2xJad++PTNmzIivdgIYP348kZGRVK1alYoVKzJ58uRLlgsLCyM6Oprjx48D8NRTT/H0009Tv359YmNjk9xe06ZNeeCBB6hbty5VqlShXbt2HD9+nObNm3P+/HmqVq3Kc889R506dVIuVD+EhoYyYMAApk+fTsmSJeOr1Lp160ZkZCQAQ4YMYdGiRZQvX55FixbFJ50WLVpQpkwZypUrx6OPPhp/VRrAkiVLuPPOO9MkxpRkiW7GP/roIx5//HFOnDjBoEGDGDp0KPny5QtQhMY4tmzZQnh4uNdhZGpjx46lYMGCQX0vRaA0bNiQefPmJdoulNi+Z92Mp+D8+fNUrlyZtWvXMmLECEsSxmQSPXv2JHfu3F6Hke4OHjzIgAED/Lp4IC1kyjOKkydPMnz4cK677jp69eoVf+WA9btj0pOdURiv2BlFCr744gsqVarEa6+9Ft8QJyKWJIwnMtoPMZPxBWKfyzSJIioqirZt23LXXXeRP39+li9fzrhx47wOy2RhefLk4fDhw5YsTLqJex5Fnjx50nS9meY+il27dvHNN9/w6quvMmDAAOubyXiuZMmSREVFcfDgQa9DMVlI3BPu0lKGbqNYtWoVK1asoF+/foDTU2PCuyiNMcYEcRuFiDQXkW0islNELrkbRURyi8gsd/pPIhLqz3qPHj1Kr169qFOnDmPGjOHkyZMAliSMMSYAApYoRCQ7MBG4A6gIdBCRiglm6wocUdVywFjgtZTWe+5UNGFhYbz11lv07duXDRs2XHR7vjHGmLQVyDOK2sBOVd2lqueAmUDrBPO0Bt51338CNJYULk86eehPSpUqxc8//8y4ceMoVKhQmgdujDHmH4FszC4B7PEZjgJuSmoeVT0vItFAMeCQ70wi0h2I6xj+bGRk5MaEXSRnUSEkKKsszMriH1YW/7Cy+EeF1C4YyESR2JlBwpZzf+ZBVacAUwBEJDK1DTKZjZXFP6ws/mFl8Q8ri3+ISGRqlw1k1VMUUMpnuCSQsPP0+HlEJAdQGPg7gDEZY4y5TIFMFD8D5UWktIjkAu4H5ieYZz7Q2X3fDvhOM9r1usYYk8kFrOrJbXPoDXwDZAemqeomEXkJiFTV+cDbwPsishPnTOJ+P1Y9JVAxZ0BWFv+wsviHlcU/rCz+keqyyHA33BljjElfmaavJ2OMMYFhicIYY0yygjZRBKr7j4zIj7IYICKbRWS9iHwrItd7EWd6SKksfOZrJyIqIpn20kh/ykJE7nP3jU0i8mF6x5he/PiOXCciS0Rkjfs9aeFFnIEmItNE5ICIbExiuojIeLec1otITb9WrKpB98Jp/P4VKAPkAtYBFRPM0wuY7L6/H5jlddwelsWtQD73fc+sXBbufAWB5cBKIMLruD3cL8oDa4Ai7vBVXsftYVlMAXq67ysCv3sdd4DKoiFQE9iYxPQWwFc497DVAX7yZ73BekYRkO4/MqgUy0JVl6jqKXdwJc49K5mRP/sFwHBgFHAmPYNLZ/6UxaPARFU9AqCqB9I5xvTiT1koENffT2EuvacrU1DV5SR/L1pr4D11rASuEJHiKa03WBNFYt1/lEhqHlU9D8R1/5HZ+FMWvrri/GLIjFIsCxGpAZRS1S/SMzAP+LNf3ADcICI/iMhKEWmebtGlL3/KYhjQSUSigC+BPukTWtC53OMJELwPLkqz7j8yAb8/p4h0AiKAWwIakXeSLQsRyYbTC/HD6RWQh/zZL3LgVD81wjnL/J+IVFbVowGOLb35UxYdgOmq+oaI1MW5f6uyql4IfHhBJVXHzWA9o7DuP/7hT1kgIrcDQ4FWqno2nWJLbymVRUGgMrBURH7HqYOdn0kbtP39jsxT1RhV/Q3YhpM4Mht/yqIr8DGAqq4A8uB0GJjV+HU8SShYE4V1//GPFMvCrW55CydJZNZ6aEihLFQ1WlVDVDVUVUNx2mtaqWqqO0MLYv58R+biXOiAiITgVEXtStco04c/ZbEbaAwgIuE4iSIrPqN2PvCQe/VTHSBaVfentFBQVj1p4Lr/yHD8LIvXgQLAbLc9f7eqtvIs6ADxsyyyBD/L4hugqYhsBmKBQap62LuoA8PPshgI/J+IPIFT1fJwZvxhKSIf4VQ1hrjtMS8AOQFUdTJO+0wLYCdwCuji13ozYVkZY4xJQ8Fa9WSMMSZIWKIwxhiTLEsUxhhjkmWJwhhjTLIsURhjjEmWJQoTdEQkVkTW+rxCk5k3NKmeMi9zm0vd3kfXuV1eVEjFOnqIyEPu+4dF5FqfaVNFpGIax/mziFT3Y5n+IpLv327bZF2WKEwwOq2q1X1ev6fTdjuqajWcziZfv9yFVXWyqr7nDj4MXOszrZuqbk6TKP+JcxL+xdkfsERhUs0ShckQ3DOH/4nIL+6rXiLzVBKRVe5ZyHoRKe+O7+Qz/i0RyZ7C5pYD5dxlG7vPMNjg9vWf2x0/Uv55Bshod9wwEXlSRNrh9Ln1gbvNvO6ZQISI9BSRUT4xPywi/0llnCvw6dBNRP4rIpHiPHviRXdcX5yEtURElrjjmorICrccZ4tIgRS2Y7I4SxQmGOX1qXaa4447ADRR1ZpAe2B8Isv1AN5U1eo4B+oot7uG9kB9d3ws0DGF7d8FbBCRPMB0oL2qVsHpyaCniBQF7gYqqWpV4GXfhVX1EyAS55d/dVU97TP5E6Ctz3B7YFYq42yO001HnKGqGgFUBW4RkaqqOh6nL59bVfVWtyuPZ4Hb3bKMBAaksB2TxQVlFx4myzvtHix95QQmuHXysTj9FiW0AhgqIiWBz1R1h4g0BmoBP7vdm+TFSTqJ+UBETgO/43RDXQH4TVW3u9PfBR4HJuA862KqiCwA/O7SXFUPisgut5+dHe42fnDXezlx5sfprsL3CWX3iUh3nO91cZwH9KxPsGwdd/wP7nZy4ZSbMUmyRGEyiieAv4BqOGfClzyUSFU/FJGfgDuBb0SkG063yu+q6tN+bKOjbweCIpLo803cvoVq43Qydz/QG7jtMj7LLOA+YCswR1VVnKO233HiPMVtJDARaCsipYEngRtV9YiITMfp+C4hARapaofLiNdkcVb1ZDKKwsB+9/kBD+L8mr6IiJQBdrnVLfNxqmC+BdqJyFXuPEXF/2eKbwVCRaScO/wgsMyt0y+sql/iNBQnduXRcZxuzxPzGdAG5xkJs9xxlxWnqsbgVCHVcautCgEngWgRuRq4I4lYVgL14z6TiOQTkcTOzoyJZ4nCZBSTgM4ishKn2ulkIvO0BzaKyFogDOeRj5txDqgLRWQ9sAinWiZFqnoGp3fN2SKyAbgATMY56H7hrm8ZztlOQtOByXGN2QnWewTYDFyvqqvccZcdp9v28QbwpKquw3k+9iZgGk51VpwpwFciskRVD+JckfWRu52VOGVlTJKs91hjjDHJsjMKY4wxybJEYYwxJlmWKIwxxiTLEoUxxphkWaIwxhiTLEsUxhhjkmWJwhhjTLL+H5ZKKedEYCQxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred = model.predict(x_test)\n",
    "plot_roc(pred,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiclass Classification Error Metrics\n",
    "\n",
    "The following sections will examine several metrics for evaluating classification error. The following classification neural network will be used to evaluate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1500 samples, validate on 500 samples\n",
      "Epoch 1/1000\n",
      "1500/1500 - 1s - loss: 1.5465 - accuracy: 0.4513 - val_loss: 1.1468 - val_accuracy: 0.3740\n",
      "Epoch 2/1000\n",
      "1500/1500 - 0s - loss: 1.1420 - accuracy: 0.4673 - val_loss: 1.1057 - val_accuracy: 0.4080\n",
      "Epoch 3/1000\n",
      "1500/1500 - 0s - loss: 1.0928 - accuracy: 0.4807 - val_loss: 1.0325 - val_accuracy: 0.5700\n",
      "Epoch 4/1000\n",
      "1500/1500 - 0s - loss: 0.9624 - accuracy: 0.6100 - val_loss: 0.9007 - val_accuracy: 0.6680\n",
      "Epoch 5/1000\n",
      "1500/1500 - 0s - loss: 0.8312 - accuracy: 0.6720 - val_loss: 0.8426 - val_accuracy: 0.6500\n",
      "Epoch 6/1000\n",
      "1500/1500 - 0s - loss: 0.7847 - accuracy: 0.6807 - val_loss: 0.8628 - val_accuracy: 0.6280\n",
      "Epoch 7/1000\n",
      "1500/1500 - 0s - loss: 0.7781 - accuracy: 0.6700 - val_loss: 0.7909 - val_accuracy: 0.6940\n",
      "Epoch 8/1000\n",
      "1500/1500 - 0s - loss: 0.7470 - accuracy: 0.6873 - val_loss: 0.7994 - val_accuracy: 0.6540\n",
      "Epoch 9/1000\n",
      "1500/1500 - 0s - loss: 0.7326 - accuracy: 0.6967 - val_loss: 0.7821 - val_accuracy: 0.6920\n",
      "Epoch 10/1000\n",
      "1500/1500 - 0s - loss: 0.7149 - accuracy: 0.7020 - val_loss: 0.7616 - val_accuracy: 0.7040\n",
      "Epoch 11/1000\n",
      "1500/1500 - 0s - loss: 0.7071 - accuracy: 0.6973 - val_loss: 0.7604 - val_accuracy: 0.7040\n",
      "Epoch 12/1000\n",
      "1500/1500 - 0s - loss: 0.6948 - accuracy: 0.7013 - val_loss: 0.7666 - val_accuracy: 0.6700\n",
      "Epoch 13/1000\n",
      "1500/1500 - 0s - loss: 0.6958 - accuracy: 0.7020 - val_loss: 0.7492 - val_accuracy: 0.6920\n",
      "Epoch 14/1000\n",
      "1500/1500 - 0s - loss: 0.6820 - accuracy: 0.7040 - val_loss: 0.7565 - val_accuracy: 0.6820\n",
      "Epoch 15/1000\n",
      "1500/1500 - 0s - loss: 0.6794 - accuracy: 0.7033 - val_loss: 0.7830 - val_accuracy: 0.6640\n",
      "Epoch 16/1000\n",
      "1500/1500 - 0s - loss: 0.6793 - accuracy: 0.7040 - val_loss: 0.7424 - val_accuracy: 0.7020\n",
      "Epoch 17/1000\n",
      "1500/1500 - 0s - loss: 0.6741 - accuracy: 0.7073 - val_loss: 0.7433 - val_accuracy: 0.7000\n",
      "Epoch 18/1000\n",
      "1500/1500 - 0s - loss: 0.6643 - accuracy: 0.7167 - val_loss: 0.7504 - val_accuracy: 0.7020\n",
      "Epoch 19/1000\n",
      "1500/1500 - 0s - loss: 0.6660 - accuracy: 0.7100 - val_loss: 0.7402 - val_accuracy: 0.6940\n",
      "Epoch 20/1000\n",
      "1500/1500 - 0s - loss: 0.6688 - accuracy: 0.7053 - val_loss: 0.7461 - val_accuracy: 0.6920\n",
      "Epoch 21/1000\n",
      "1500/1500 - 0s - loss: 0.6682 - accuracy: 0.7173 - val_loss: 0.7432 - val_accuracy: 0.6940\n",
      "Epoch 22/1000\n",
      "1500/1500 - 0s - loss: 0.6539 - accuracy: 0.7187 - val_loss: 0.7393 - val_accuracy: 0.7020\n",
      "Epoch 23/1000\n",
      "1500/1500 - 0s - loss: 0.6611 - accuracy: 0.7173 - val_loss: 0.7341 - val_accuracy: 0.7040\n",
      "Epoch 24/1000\n",
      "1500/1500 - 0s - loss: 0.6509 - accuracy: 0.7193 - val_loss: 0.7348 - val_accuracy: 0.6960\n",
      "Epoch 25/1000\n",
      "1500/1500 - 0s - loss: 0.6436 - accuracy: 0.7213 - val_loss: 0.7417 - val_accuracy: 0.6940\n",
      "Epoch 26/1000\n",
      "1500/1500 - 0s - loss: 0.6392 - accuracy: 0.7253 - val_loss: 0.7494 - val_accuracy: 0.6920\n",
      "Epoch 27/1000\n",
      "1500/1500 - 0s - loss: 0.6342 - accuracy: 0.7327 - val_loss: 0.7366 - val_accuracy: 0.6980\n",
      "Epoch 28/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "1500/1500 - 0s - loss: 0.6351 - accuracy: 0.7340 - val_loss: 0.7346 - val_accuracy: 0.7040\n",
      "Epoch 00028: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a4c5f1dd0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(y.shape[1],activation='softmax',kernel_initializer='random_normal'))\n",
    "model.compile(loss='categorical_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, \n",
    "                        verbose=1, mode='auto', restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Accuracy\n",
    " \n",
    "Accuracy is the number of rows where the neural network correctly predicted the target class.  Accuracy is only used for classification, not regression.\n",
    "\n",
    "$ accuracy = \\frac{c}{N} $\n",
    "\n",
    "Where $c$ is the number correct and $N$ is the size of the evaluated set (training or validation). Higher accuracy numbers are desired.\n",
    "\n",
    "As we just saw, by default, Keras will return the percent probability for each class. We can change these prediction probabilities into the actual iris predicted with **argmax**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(x_test)\n",
    "pred = np.argmax(pred,axis=1) # raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the actual iris flower predicted, we can calculate the percent accuracy (how many were correctly classified)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score: 0.704\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "y_compare = np.argmax(y_test,axis=1) \n",
    "score = metrics.accuracy_score(y_compare, pred)\n",
    "print(\"Accuracy score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Log Loss\n",
    "\n",
    "Accuracy is like a final exam with no partial credit.  However, neural networks can predict a probability of each of the target classes.  Neural networks will give high probabilities to predictions that are more likely.  Log loss is an error metric that penalizes confidence in wrong answers. Lower log loss values are desired.\n",
    "\n",
    "The following code shows the output of predict_proba:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy array of predictions\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.    , 0.1029, 0.7342, 0.1614, 0.0014, 0.    , 0.    ],\n",
       "       [0.    , 0.6965, 0.3013, 0.    , 0.0022, 0.    , 0.    ],\n",
       "       [0.    , 0.689 , 0.3062, 0.0001, 0.0047, 0.0001, 0.    ],\n",
       "       [0.    , 0.3125, 0.6783, 0.0062, 0.0029, 0.    , 0.    ],\n",
       "       [0.    , 0.0392, 0.7529, 0.2077, 0.0002, 0.    , 0.    ]],\n",
       "      dtype=float32)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As percent probability\n",
      "[ 0.0001 10.2942 73.4207 16.1436  0.1407  0.0007  0.    ]\n",
      "Log loss score: 0.7341072283834219\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "\n",
    "# Don't display numpy in scientific notation\n",
    "np.set_printoptions(precision=4)\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "# Generate predictions\n",
    "pred = model.predict(x_test)\n",
    "\n",
    "print(\"Numpy array of predictions\")\n",
    "display(pred[0:5])\n",
    "\n",
    "print(\"As percent probability\")\n",
    "print(pred[0]*100)\n",
    "\n",
    "score = metrics.log_loss(y_test, pred)\n",
    "print(\"Log loss score: {}\".format(score))\n",
    "\n",
    "pred = np.argmax(pred,axis=1) # raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Log loss](https://www.kaggle.com/wiki/LogarithmicLoss) is calculated as follows:\n",
    "\n",
    "$ \\mbox{log loss} = -\\frac{1}{N}\\sum_{i=1}^N {( {y}_i\\log(\\hat{y}_i) + (1 - {y}_i)\\log(1 - \\hat{y}_i))} $\n",
    "\n",
    "The log function is useful to penalizing wrong answers.  The following code demonstrates the utility of the log function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jheaton/miniconda3/envs/tensorflow/lib/python3.7/site-packages/ipykernel_launcher.py:12: RuntimeWarning: divide by zero encountered in log\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAE0CAYAAAASSJRcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxc9Xnv8e+jfV8syZJ3G2NjjA3EyIQlCTIQAm0CzXppGkq4SWhIk5S2SZuWNGmSJm3olvbV3jbcJr1pm8ShIQs3IQRIYkgLBmw22xjjDRtbsiXL1r6Pnv4xR2awZXRGo9EZaT7v10svzZwzyzOPj0Zf//Sb3zF3FwAAAICplRN1AQAAAMBsRNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAkOHM7CUzu3oKHucaM/tByNs+YWbnpfqcAJDNCNoAkD2+JOkvQt72ryR9Po21AMCsR9AGgCxgZuslVbr75pB3uVfSBjObl8ayAGBWI2gDwAxhZoVm9hUzaw6+vmJmhQn7/8DMWoJ9HzQzN7Ozg93XSXo44baXmdkxM1sUXL/AzDrMbJUkufuApK2Srpm+VwgAswtBGwBmjjskXSLpQkkXSLpY0qclycyulfR7kq6WdLakK06571pJu8auuPujkr4q6RtmVizp3yV92t1fSLjPzuB5AACTQNAGgJnjNyR93t1b3b1N0uck3RTse4+kf3X3He7eF+xLVCWp+5RtfyqpUtITkpol/eMp+7uD+wEAJoGgDQAzx3xJBxKuHwi2je17OWFf4mVJOiGpPHGDuw9L+n+S1kj6a3f3U+5TLqkjtZIBIHsRtAFg5miWtCTh+uJgmyS1SFqYsG/RKfd9TtLKxA1mtkDSZyX9q6S/TpzvHThX0rMp1gwAWYugDQAzx7clfdrM6sysVtJnJP1HsO9uSbeY2blmVhLsS3SfEuZtm5kpPpr9NUkfUDyofyFhf6GkiyQ9mJ6XAgCzH0EbAGaOP5O0RfHR6W2Sngq2yd1/IunvJf1C0h5JjwX3GQz2PyWp08xeH2z/uKR6SX8STBm5RfGg/sZg//WSNrn72Ig5ACBJdvqUPADATGdm50raLqnQ3UeCbddI+oi7/1qI+z8u6QPuvj29lQLA7EXQBoBZwszeLunHkkolfUPSaJhQDQBID6aOAMDs8VuS2iTtlRSTdFu05QBAdmNEGwAAAEgDRrQBAACANCBoAwAAAGmQF3UByaitrfWlS5dG9vy9vb0qLS2N7PlnOvo3efQuNfQvNfRv8uhdauhfauhfarZu3XrM3etSeYwZFbSXLl2qLVu2RPb8mzZtUlNTU2TPP9PRv8mjd6mhf6mhf5NH71JD/1JD/1JjZgdSfQymjgAAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpEGnQNrNrzWyXme0xs09FWQsAAAAwlSIL2maWK+kfJV0nabWkXzez1VHVAwAAAEylKEe0L5a0x933ufuQpI2SboiwHgDAKYZGRjU86lGXAQAzUl6Ez71A0ssJ1w9Jen1EtQBAVhgddXX0D6u9Z1DHeobU3juo9p6h+PXe+Pf2niG1B5e7Bkb0vnML9OaoCweAGcjcoxmpMLN3S3qLu38wuH6TpIvd/WOn3O5WSbdKUn19/UUbN26c9lrH9PT0qKysLLLnn+no3+TRu9TM5v65uwZiUveQq2vQ1T0c/941FP/qDr7Ht8VvN967vkkqK5AqCkwVBaby4KuiwLS8dEhr5s3O/qXbbD72pgP9Sw39S82GDRu2untjKo8R5Yj2IUmLEq4vlNR86o3c/S5Jd0lSY2OjNzU1TUtx49m0aZOifP6Zjv5NHr1LzUzrn7urq39EbT0Dau0eVNvYV88ro8/xEechHesZ1ODI6LiPU1aYp5qyQtWUFmhBfaFqywpUU1qomrIC1ZQVqrY0/r2mrEDVJQXKzbFxH2em9S+T0LvU0L/U0L/oRRm0n5S0wsyWSTos6UZJ742wHgBIq4Hh2MnA3NY9+OoQHWw/Flweip0envNzTbVBMK4pLdTZc8vi1xMC88nLpQUqys+N4FUCAMZEFrTdfcTMPirpp5JyJX3d3XdEVQ8ATMboqOt435BauwbV2j3wqiCdGKDbugfVPTBy2v3NpJrSAtWWFaquvFDL60pVV16ouuB6XXmh5pYXqq6sSBXFeTIbf9QZAJB5ohzRlrvfJ+m+KGsAgPG4u7oHR9TaNaAjnYM62jWgo90DOto5oKNdgycvt3YPamScVTnKCvNOBuZzGyr0phWFpwXouvL4yHNeLucOA4DZKNKgDQBRGJvCcaRrQEe7BnQkCMynXu4bip123/KiPDVUFKm+okiXLK85eXlueaHmVsRHnmvLC1RSwNsrAGQ7fhMAmFX6h2Jq7uxXS8eAmjv7daRzQFt3Dupf9z0RD9JdA+roGz7tfgV5OUFoLtR58yt05aq5qq8oVH0QpBsqijS3opAADQAIjd8YAGaMwZGYjnQOqLljQC2d/WrpDL53DKg5uDxeiK4okJbUDWlhdYkal1arvrxI9ZWvBOj6ikJVFucz/xkAMKUI2gAygrurrWdQh07069CJfjV39Kulo1/NnfHpHC2d/TrWM3Ta/apK8jWvsljzK4u0bnGV5lcVa15lUXxbVTxMb/7vX6qp6Q0RvCoAQDYjaAOYFqOjrtbuQR060afDHf1BoO7ToRP9OnyiX4c6+jV0ynrQ5YV5mlcVD83nza/QvMpizasq0vzg+7zKIqZyAAAyFr+hAEyJ2KjrSNdAPDS/KkDHL7d0DJy2NnRNaYEWVhdr1bxyXb26XguqirWwulgLq0s0r6pIFUX5Eb0aAABSR9AGEFrv4IgOHu+Lf7X36cDxXh083q+D7b06dKL/tGXu6soLtbC6WGsXVOq6NfO0oDoepBdVF2t+VTGj0QCAWY3fcgBOGpsnfbA9HqYPBN/HLh/rGXzV7SuK8rSkplTnLajUdWvnaVF1STAiHQ/SnJkQAJDNCNpAlnF3tXUPat+xXu1P+Ho5CNSJa0ebSfMri7VoTrGuWjVXi2tKtHhOiZbUlGjJnFJVljC1AwCAMyFoA7NU18Cw9rfFQ/QrobpH+9t61ZsQpgvycrQkCM+XLa/VkiBML66Jj04X5jEqDQDAZBC0gRlsODaqA+292tOaEKSDUJ24FF6OSQurS7SstlSNS+borLpSLauNf82rLFZuDutHAwAw1QjawAwwFHM939yl3a3d2tvao93B10vHel/1AcS68kItqy3V1efWnwzSZ9WVatGcEkamAQCYZgRtIIP0DI6cDNJ7Wnu0p7Vbu1t7dLC9T/7gLyXFR6eX1pTq7LllumZ1vVbUl2l5XZmW1ZaqnOXwAADIGARtIALDsVHta+vVC0e69MKRbr3Q0qUXj/bocEf/ydvk55rOqi3TmgWVurBqWG9+/RqtmFuupbWMTgMAMBMQtIE0co+fDXFnSzxQ7zrSrZ0tXdrb1qPhWHzKR36uaXldmdYvrdZ76xfr7LllWjG3TIvnlCgvN0eStGnTJjWdPz/KlwIAAJJE0AamyNDIqHa3dmvH4S4939J1crS6o2/45G3mVRbpnIZyNZ0zV+fOK9c5DeU6q7ZMBXk5EVYOAADSgaANTMLAcEy7jnRre3Onth/u1PbDXdp1pPvkKcZLCnK1sr5c161p0KqGCp3TUK5VDeWqKimIuHIAADBdCNrABHoHR7SzpSseqJvj33e39igWrPZRWZyvNQsqdMvlS3XegkqtmV+hpTWlymHJPAAAshpBG0gQG3Xtbu3WMwc79MzL8a8Xj3ZrbAW9mtICrVlQqavOnau1Cyp13vxKLawulhmhGgAAvBpBG1ntaNeAnj4Zqk9o26HOk2dNrCzO1wWLqnTN6nqtXViltQsqVV9RSKgGAAChELSRNUZio3q+pUtPvnRCWw8c19MHO9TSOSApvvLH6nkVeudFC3XhoipduKhKy2pLCdUAAGDSCNqYtXoHR/TMyx168qXj2vLSCT118IT6gtHqhdXFWr90TjxUL67S6nkVKspnbWoAADB1CNqYNU70Dunx/e168qUT2vLScW1v7lJs1GUmndtQoXdftFDrl81R45I5aqgsirpcAAAwyxG0MWN1Dwzrif3H9ejedj26t107W7okSYV5OXrd4ip9pGm5GpfO0esWV6mCU5MDAIBpRtDGjNE/FNPWAyf06N5jenRvu7Yd7lRs1FWYl6PGpdX6xDUrdenyGq1dUMUJYAAAQOQI2shY7q5dR7v18K42Pfxim7a8dEJDsVHl5ZguWBQfsb50eY3WLa5mfjUAAMg4BG1klM7+Yf33nmMnw/WRrviqIKsaynXzZUt02dm1Wr90jsoKOXQBAEBmI60gUu6uF4/26KGdR7VpV6ueOtih2KirvDBPb1hRq6Zz6vSmlXWaV1kcdakAAABJIWhj2sVGXVsPnNADO47owZ1HdaC9T5J03vwKffiKs3TFyrl63eIq5ecyzxoAAMxcBG1Mi/6hmJ5uHdGP//NZ/eyFVh3vHVJBbo4uXV6jD73xLL15db3qK1hyDwAAzB4EbaTNwHBMm3a16UfPNetnO1vVPxxTedERXblqrt68ul5XrKxTOcvuAQCAWYqgjSk1NDKq/9rTph8926IHnj+qnsERzSkt0NvXLdCCWKs+9GsbWHoPAABkBYI2UubuemL/cX3/6cP6yfYj6uwfVkVRnn5lbYPeev58Xba8Rnm5Odq0aRMhGwAAZA2CNibt5eN9+t5Th3XPU4d08HifSgpy9ZbzGvTW8+fpjSvqCNUAACCrEbSRlL6hEf1k2xF9d+shPbavXZJ02fIa3X71Cl27pkElBRxSAAAAEkEbIe1p7dZ/bD6oe7YeUvfgiBbPKdHvvXml3v66BVo0pyTq8gAAADIOQRtnNDQyqgeeP6L/2HxAm/cdV0Fujq5b26D3XrxYFy+bIzOLukQAAICMRdDGadp7BvVvjx3Qt544qLbuQS2sLtYfXrtK725cqNqywqjLAwAAmBEI2jhpb1uPvvZf+3XP1kMaHBnVlavm6qZLluhNK+uUm8PoNQAAQDII2tCTLx3XXY/s00M7jyo/N0fvXLdQH3jDMp09tyzq0gAAAGYsgnYWe2xvu77y0It6fP9xVZfk62NXrtBvXrqE6SEAAABTgKCdZdxdj+1r11ce2q0n9h/X3PJCffZtq3Xj+sUqLsiNujwAAIBZg6CdRbYeOK4v37/rZMD+07et1o0XL1ZRPgEbAABgqkUStM3sLyW9TdKQpL2SbnH3jihqyQb7j/Xqzvtf0E+2H1FdeaE+d/15+l/rFxGwAQAA0iiqEe0HJf2Ru4+Y2Zcl/ZGkP4yollmrvWdQf/+z3frm4wdVkJej3716pT70pmWcvREAAGAaRJK43P2BhKubJb0rijpmq9io65uPH9Bf/nSX+oZiunH9It1+9UrVlfMhRwAAgOmSCUOb/1vSd6IuYrZ4+uAJ/ckPt2v74S5dfnaNPnf9eTp7bnnUZQEAAGQdc/f0PLDZQ5Iaxtl1h7v/MLjNHZIaJb3Dz1CImd0q6VZJqq+vv2jjxo1pqTeMnp4elZVl5trSfcOuu18c0sMvj6iy0PTrqwp0cUNuRp0mPZP7l+noXWroX2ro3+TRu9TQv9TQv9Rs2LBhq7s3pvIYaQvaEz6x2c2SPizpKnfvC3OfxsZG37JlS3oLew2bNm1SU1NTZM9/Jg+/2KZP3fOcjnYN6JbLl+n2q1eovCg/6rJOk6n9mwnoXWroX2ro3+TRu9TQv9TQv9SYWcpBO6pVR65V/MOPV4QN2Thd98Cwvvjjndr45Ms6e26ZvveRy3XhoqqoywIAAICim6P9D5IKJT0YTG3Y7O4fjqiWGWnrgRP6+LefVktnvz58xXLdfvUKlusDAADIIFGtOnJ2FM87G4yOur76yD791QO7NL+qSN+97TKtW1wddVkAAAA4RSasOoKQ2nsG9bt3P6tHXmzTr54/T3/+jrWqyMC52AAAACBozxg7mjv1oW9s0bHeIX3x7Wv03osXZ9SKIgAAAHg1gvYMcN+2Fv3+3c+qqiRf93z4Mq1dWBl1SQAAAJgAQTuDubv+7me79ZWHdmvd4ir9800XaW55UdRlAQAAIASCdoaKjbo+/YPt+vYTB/XOdQv1pXesUWEeq4oAAADMFATtDDQwHNPtG5/R/TuO6Lc3LNcnrjmH+dgAAAAzDEE7w/QPxfSBbzypR/e260/eulofeMOyqEsCAADAJBC0M8jAcDxkb97Xrr95zwV6x7qFUZcEAACASSJoZ4iB4Zg+9G9b9Ni+dv31uwnZAAAAM11O1AVAGomN6qPfekq/3H1MX37n+YRsAACAWYCgHTF312fu3aGHdrbqCzecp/c0Loq6JAAAAEwBgnbE/unhvfrW4wd1W9Ny3XTp0qjLAQAAwBQhaEfovm0tuvP+Xbrhwvn65DXnRF0OAAAAphBBOyJ7Wrv1if98VusWV+nOd52vnBzWyQYAAJhNCNoR6B4Y1q3/vlUlBXn6p/ddxBkfAQAAZiGW95tm7q5PfW+bDrT36ZsffL3qK4qiLgkAAABpwIj2NPv+04f14+da9PvXrNQlZ9VEXQ4AAADShKA9jQ539OuzP9yh9Uur9VtvWh51OQAAAEgjgvY0cXd94u5nNequv3nPhcrlw48AAACzGkF7mtzz1GE9tq9dd/zqai2aUxJ1OQAAAEgzgvY06Ogb0pfu26l1i6t043rO/AgAAJANCNrT4Mv371Jn/7C++Pa1rJcNAACQJQjaabazpUsbnzyomy9dqnPnVURdDgAAAKYJQTvN7rz/BZUX5ul3rloRdSkAAACYRgTtNNq8r12/2NWmj2w4W5Ul+VGXAwAAgGlE0E4Td9ed97+ghooivf+ypVGXAwAAgGk2YdA2s4+aWfV0FDObbN53XE8d7NBvb1iuovzcqMsBAADANAszot0g6Ukzu9vMrjUzls0I4Z8e3qvasgK9u5Hl/AAAALLRhEHb3T8taYWkr0l6v6TdZvYlM+Mc4mewo7lTj7zYplsuX8ZoNgAAQJYKNUfb3V3SkeBrRFK1pO+a2Z1prG3G+pdf7ldZYZ7ed8mSqEsBAABARPImuoGZfVzSzZKOSfoXSZ9092Ezy5G0W9IfpLfEmeVE75B+vK1FN65fpMpiVhoBAADIVhMGbUm1kt7h7gcSN7r7qJm9NT1lzVz3PHVIQyOjeu/rF0ddCgAAACI0YdB298+8xr6dU1vOzObu+tYTB7VucZVWNXAWSAAAgGzGOtpTaMuBE9rX1qv3vp652QAAANmOoD2F7n2mWUX5ObpuTUPUpQAAACBiBO0pMhIb1X3bWnTVufUqLQwz9R0AAACzGUF7ijy2r13tvUN62/nzoy4FAAAAGYCgPUV+9GyLygrz1HROXdSlAAAAIAMQtKfA6KjrZy+0qumcOs4ECQAAAEkE7Smxo7lLx3oGdeWquVGXAgAAgAxB0J4Cv9jVKjPpipVMGwEAAEAcQXsK/GJXqy5YWKWassKoSwEAAECGIGinqLNvWM+83MFoNgAAAF4l0qBtZp8wMzez2ijrSMWWA8flLl26vCbqUgAAAJBBIgvaZrZI0pslHYyqhqnwxP7jKsjN0YWLqqIuBQAAABkkyhHtv5X0B5I8whpS9sRLx3X+wkqW9QMAAMCrRBK0zex6SYfd/dkonn+q9A2NaNuhTl28bE7UpQAAACDDmHt6BpTN7CFJDePsukPSH0u6xt07zewlSY3ufuwMj3OrpFslqb6+/qKNGzempd4wenp6VFZWdvL6zvaYvvzkgH73okJdUJcXWV0zxan9Q3j0LjX0LzX0b/LoXWroX2roX2o2bNiw1d0bU3mMtKVDd796vO1mtlbSMknPmpkkLZT0lJld7O5HxnmcuyTdJUmNjY3e1NSUrpIntGnTJiU+/+5H9knaqfdd90aW9gvh1P4hPHqXGvqXGvo3efQuNfQvNfQvetM+DOvu2ySdPIXiRCPamWzb4U7NrywiZAMAAOA0rKOdgu3NnVqzoDLqMgAAAJCBIg/a7r50Jo5m9wyOaP+xXoI2AAAAxhV50J6pnm/ukru0lqANAACAcRC0J2nX0W5J0qp55RFXAgAAgExE0J6kva09Ki3IVUNFUdSlAAAAIAMRtCdpb1uPls8tU7BEIQAAAPAqBO1J2tvao+V1LAIPAACA8RG0J6F3cETNnQNaXlcadSkAAADIUATtSdh/rFeSGNEGAADAGRG0J2EsaC+tZUQbAAAA4yNoT8KhE/2SpIXVxRFXAgAAgExF0J6Ewx19qizOV3lRftSlAAAAIEMRtCfh8Il+LahiNBsAAABnRtCehMMd/VrAtBEAAAC8BoJ2ktydEW0AAABMiKCdpM7+YfUOxfggJAAAAF4TQTtJzR0DkqT5jGgDAADgNRC0k3SsZ1CSVFdeGHElAAAAyGQE7SS1dceDdm0ZQRsAAABnRtBO0tiIdm1ZQcSVAAAAIJMRtJN0rGdQRfk5KivMi7oUAAAAZDCCdpKO9QyptqxQZhZ1KQAAAMhgBO0kHesZZH42AAAAJkTQTlJbN0EbAAAAEyNoJyk+dYQPQgIAAOC1EbST4O7q6h9WZUl+1KUAAAAgwxG0kzA8Kg3FRlVZTNAGAADAayNoJ6F32CWJoA0AAIAJEbST0DcS/15RRNAGAADAayNoJ6GPEW0AAACERNBOwtjUkQqCNgAAACZA0E7C2NQRRrQBAAAwEYJ2EsamjlQU5UVcCQAAADIdQTsJTB0BAABAWATtJPSPuIrzc5WfS9sAAADw2kiMSRiMSSUFuVGXAQAAgBmAoJ2EwZhUTNAGAABACATtJAzGnBFtAAAAhELQTkJ8RJsVRwAAADAxgnYShmKuknxGtAEAADAxgnYSmKMNAACAsAjaSRiMOUEbAAAAoRC0kzAUE1NHAAAAEApBOwmsOgIAAICwCNpJGBxh1REAAACEE1nQNrOPmdkuM9thZndGVUdYI7FRjbhUzNQRAAAAhBDJ8KyZbZB0g6Tz3X3QzOZGUUcy+odjkqTiAv4IAAAAgIlFlRpvk/QX7j4oSe7eGlEdoQ2NjEqSCvMY0QYAAMDEograKyW90cweN7OHzWx9RHWENhxzSVJ+LiPaAAAAmJi5e3oe2OwhSQ3j7LpD0hcl/VzS70haL+k7ks7ycYoxs1sl3SpJ9fX1F23cuDEt9U6krW9Un3ykXx9cW6A3LMiPpIaZrqenR2VlZVGXMSPRu9TQv9TQv8mjd6mhf6mhf6nZsGHDVndvTOUx0jZH292vPtM+M7tN0veCYP2EmY1KqpXUNs7j3CXpLklqbGz0pqam9BQ8gT2tPdIjD2vteavVdOGCSGqY6TZt2qSo/v1mOnqXGvqXGvo3efQuNfQvNfQvelHNg/iBpCslycxWSiqQdCyiWkIZjsXnaBcwdQQAAAAhRLUo9Nclfd3MtksaknTzeNNGMsnJoJ1H0AYAAMDEIgna7j4k6X1RPPdkja06wochAQAAEAapMaShGEEbAAAA4ZEaQxpb3o+pIwAAAAiD1BjS8AgfhgQAAEB4pMaQTk4dybOIKwEAAMBMQNAOieX9AAAAkAxSY0isOgIAAIBkkBpDGmIdbQAAACSB1BgSH4YEAABAMkiNIY0t75eXy4chAQAAMDGCdkgjo0HQzqFlAAAAmBipMaRRjwdtcjYAAADCIDaGFAtGtHONqSMAAACYGEE7pJNBO4egDQAAgIkRtEMadZdJMka0AQAAEAJBO6TYqIvBbAAAAIRF0A4p5i4GswEAABAWQTukUUa0AQAAkASCdkixUZoFAACA8MiOIY06I9oAAAAIj6AdEh+GBAAAQDII2iHFGNEGAABAEgjaIcU/DEnSBgAAQDgE7ZCYOgIAAIBkELRDigVnhgQAAADCIGiHxDraAAAASAZBO6SYi6ANAACA0AjaITGiDQAAgGQQtEPiw5AAAABIBkE7pPg62iRtAAAAhEPQDompIwAAAEgGQTukmDvNAgAAQGhkx5Bioy5mjgAAACAsgnZIzvJ+AAAASAJBO6RRzgwJAACAJBC0Q3IXU0cAAAAQGkE7JJdHXQIAAABmEIJ2EhjQBgAAQFgE7ZCcAW0AAAAkgaAdEjkbAAAAySBoJ4EPQwIAACAsgnZYDGkDAAAgCQTtkFysow0AAIDwCNoh8WFIAAAAJCOSoG1mF5rZZjN7xsy2mNnFUdSRLOZoAwAAIKyoRrTvlPQ5d79Q0meC6xmNAW0AAAAkI6qg7ZIqgsuVkpojqiM0d+ZoAwAAILy8iJ73dkk/NbO/UjzsXxZRHQAAAEBamKfpU35m9pCkhnF23SHpKkkPu/s9ZvYeSbe6+9VneJxbJd0aXD1H0q501BtSraRjET7/TEf/Jo/epYb+pYb+TR69Sw39Sw39S8057l6eygOkLWi/5pOadUqqcnc3M5PU6e4VE90vama2xd0bo65jpqJ/k0fvUkP/UkP/Jo/epYb+pYb+pWYq+hfVHO1mSVcEl6+UtDuiOgAAAIC0iGqO9ock/Z2Z5Uka0CtTQwAAAIBZIZKg7e7/JemiKJ47RXdFXcAMR/8mj96lhv6lhv5NHr1LDf1LDf1LTcr9i2SONgAAADDbcQp2AAAAIA0I2pLM7Foz22Vme8zsU+PsLzSz7wT7HzezpQn7/ijYvsvM3jKddWeKEP37PTN73syeM7OfmdmShH0xM3sm+Lp3eivPDCH6934za0vo0wcT9t1sZruDr5unt/Lohejd3yb07UUz60jYx7Fn9nUzazWz7WfYb2b290F/nzOzdQn7sv3Ym6h3vxH07Dkze9TMLsAyYTcAAAZQSURBVEjY95KZbQuOvS3TV3XmCNG/JjPrTPgZ/UzCvtf8uc8GIfr3yYTebQ/e7+YE+7L6+DOzRWb2CzPbaWY7zOx3xrnN1L33uXtWf0nKlbRX0lmSCiQ9K2n1Kbf5iKR/Di7fKOk7weXVwe0LJS0LHic36teUgf3bIKkkuHzbWP+C6z1Rv4YZ0L/3S/qHce47R9K+4Ht1cLk66teUSb075fYfk/T1hOtZfewFPXiTpHWStp9h/69I+okkk3SJpMeD7Vl97IXs3WVjPZF03VjvgusvSaqN+jVkeP+aJP1onO1J/dzP1q+J+nfKbd8m6ecJ17P6+JM0T9K64HK5pBfH+b07Ze99jGhLF0va4+773H1I0kZJN5xymxskfSO4/F1JV5mZBds3uvugu++XtCd4vGwyYf/c/Rfu3hdc3Sxp4TTXmMnCHH9n8hZJD7r7cXc/IelBSdemqc5MlGzvfl3St6elshnC3R+RdPw1bnKDpH/zuM2Sqsxsnjj2Juyduz8a9Ebife80IY69M0nlPXPWSLJ/vPclcPcWd38quNwtaaekBafcbMre+wja8ea+nHD9kE5v+MnbuPuIpE5JNSHvO9sl24MPKP6/xDFFZrbFzDab2a+lo8AMF7Z/7wz+fPVdM1uU5H1nq9CvP5iutEzSzxM2Z/uxF8aZepztx16yTn3fc0kPmNlWi5/9GOO71MyeNbOfmNl5wTaOvSSYWYniQfCehM0cfwGLTwV+naTHT9k1Ze99Ua2jnUlsnG2nLsVyptuEue9sF7oHZvY+SY165WRFkrTY3ZvN7CxJPzezbe6+Nw11Zqow/fv/kr7t7oNm9mHF/7pyZcj7zmbJvP4bJX3X3WMJ27L92AuD974UmdkGxYP2GxI2Xx4ce3MlPWhmLwQjlHjFU5KWuHuPmf2KpB9IWiGOvWS9TdJ/u3vi6DfHnyQzK1P8PyC3u3vXqbvHucuk3vsY0Y7/b2RRwvWFip+5ctzbWPwkO5WK/8kmzH1nu1A9MLOrJd0h6Xp3Hxzb7u7Nwfd9kjYp/j/LbDJh/9y9PaFn/1evrEGf7cdfMq//Rp3yp1OOvVDO1ONsP/ZCMbPzJf2LpBvcvX1se8Kx1yrp+8q+KYcTcvcud+8JLt8nKd/MasWxl6zXeu/L2uPPzPIVD9nfdPfvjXOTKXvvI2hLT0paYWbLzKxA8YPy1BUI7pU09snSdyn+oQIPtt9o8VVJlin+v+0npqnuTDFh/8zsdZK+qnjIbk3YXm1mhcHlWkmXS3p+2irPDGH6Ny/h6vWKzyeTpJ9KuiboY7Wka4Jt2SLMz67M7BzFP7TyWMI2jr1w7pX0m8En8C+R1OnuLeLYm5CZLZb0PUk3ufuLCdtLzax87LLivRt35YhsZmYNwWehZGYXK55X2hXy5x6SmVUq/hfkHyZsy/rjLziuviZpp7v/zRluNmXvfVk/dcTdR8zso4o3KlfxVQl2mNnnJW1x93sV/wf5dzPbo/hI9o3BfXeY2d2K/4IekfTbp/xpetYL2b+/lFQm6T+D982D7n69pHMlfdXMRhV/E/0Ld8+qsBOyfx83s+sVP8aOK74Kidz9uJl9QfFfPJL0+VP+PDirheydFP8g0MbgP8djsv7YkyQz+7biqzvUmtkhSZ+VlC9J7v7Pku5T/NP3eyT1Sbol2JfVx54UqnefUfyzPP8neN8bcfdGSfWSvh9sy5P0LXe/f9pfQMRC9O9dkm4zsxFJ/ZJuDH6Gx/25j+AlRCpE/yTp7ZIecPfehLty/MUHVm6StM3Mngm2/bGkxdLUv/dxZkgAAAAgDZg6AgAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGgFnIzNab2XNmVhScDW6Hma2Jui4AyCacsAYAZikz+zNJRZKKJR1y9z+PuCQAyCoEbQCYpcysQPFTBQ9IuszdYxGXBABZhakjADB7zZFUJqlc8ZFtAMA0YkQbAGYpM7tX0kZJyyTNc/ePRlwSAGSVvKgLAABMPTP7TUkj7v4tM8uV9KiZXenuP4+6NgDIFoxoAwAAAGnAHG0AAAAgDQjaAAAAQBoQtAEAAIA0IGgDAAAAaUDQBgAAANKAoA0AAACkAUEbAAAASAOCNgAAAJAG/wNvAXy94P6E/wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import figure, show\n",
    "from numpy import arange, sin, pi\n",
    "\n",
    "#t = arange(1e-5, 5.0, 0.00001)\n",
    "#t = arange(1.0, 5.0, 0.00001) # computer scientists\n",
    "t = arange(0.0, 1.0, 0.00001)  # data     scientists\n",
    "\n",
    "fig = figure(1,figsize=(12, 10))\n",
    "\n",
    "ax1 = fig.add_subplot(211)\n",
    "ax1.plot(t, np.log(t))\n",
    "ax1.grid(True)\n",
    "ax1.set_ylim((-8, 1.5))\n",
    "ax1.set_xlim((-0.1, 2))\n",
    "ax1.set_xlabel('x')\n",
    "ax1.set_ylabel('y')\n",
    "ax1.set_title('log(x)')\n",
    "\n",
    "show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[ 21   1   0   0   0   0   0]\n",
      " [  4 189  56   0   0   0   0]\n",
      " [  0  46 138   3   0   0   0]\n",
      " [  0   0  10   4   0   0   0]\n",
      " [  0   8   0   0   0   0   0]\n",
      " [ 10   7   0   0   0   0   0]\n",
      " [  3   0   0   0   0   0   0]]\n",
      "Normalized confusion matrix\n",
      "[[0.95 0.05 0.   0.   0.   0.   0.  ]\n",
      " [0.02 0.76 0.22 0.   0.   0.   0.  ]\n",
      " [0.   0.25 0.74 0.02 0.   0.   0.  ]\n",
      " [0.   0.   0.71 0.29 0.   0.   0.  ]\n",
      " [0.   1.   0.   0.   0.   0.   0.  ]\n",
      " [0.59 0.41 0.   0.   0.   0.   0.  ]\n",
      " [1.   0.   0.   0.   0.   0.   0.  ]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEmCAYAAAB/OxvoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZn/8c+3O7KEIBASkC0EmIAiI4FEVBAJggy4sKgIEREVjUHB3TEKCi6M/ETBURQNguAyMSBGGWBEZFSEASUJIZJhDSZDIELCvhmS8Pz+uKekqNx03+6u23Vv5/vmdV9dde+tc55OwlOnTp1FEYGZmdVXV6cDMDOzgXEiNzOrOSdyM7OacyI3M6s5J3Izs5pzIjczqzkncusXSRtK+k9Jj0m6ZADlHCPpN+2MrVMk7Svpjk7HYeseeRz50CbpncAngJcCTwDzgNMj4roBlnsscBKwd0SsGnCgFScpgHERcXenYzFr5Rb5ECbpE8A3gX8DtgTGAN8FDmtD8dsDd64LSbwIScM6HYOtwyLCxxA8gE2AJ4Eje7hnfbJEf386vgmsn65NApYAnwQeBJYC703Xvgg8C6xMdRwPnAb8pKnssUAAw9Lz9wD3kH0q+CtwTNP565petzdwE/BY+rl307XfA18Grk/l/AYYtZbfrRH/vzbFfzjwRuBO4GHgc0337wXcADya7j0HWC9duzb9Lk+l3/eopvI/A/wN+HHjXHrNTqmOPdPzrYHlwKRO/9vwMfQOt8iHrtcAGwCzerjnZODVwHhgd7JkdkrT9ZeQvSFsQ5asvyNps4g4layVPzMiRkTE+T0FImkj4FvAIRGxMVmynpdz30jginTv5sBZwBWSNm+67Z3Ae4EtgPWAT/VQ9UvI/gy2Ab4AnAe8C5gA7At8QdKO6d7VwMeBUWR/dgcAHwKIiNele3ZPv+/MpvJHkn06mdJccUQsJEvyP5U0HPghcGFE/L6HeM36xYl86NocWB49d30cA3wpIh6MiGVkLe1jm66vTNdXRsSVZK3RXfoZz3PAbpI2jIilEbEg5543AXdFxI8jYlVEzABuB97SdM8PI+LOiHgGuJjsTWhtVpJ9H7AS+BlZkv73iHgi1b8AeAVARMyJiBtTvYuA7wP7FfidTo2IFSmeF4iI84C7gD8BW5G9cZq1nRP50PUQMKqXvtutgcVNzxenc/8oo+WN4GlgRF8DiYinyLojpgJLJV0h6aUF4mnEtE3T87/1IZ6HImJ1etxItA80XX+m8XpJO0u6XNLfJD1O9oljVA9lAyyLiL/3cs95wG7AtyNiRS/3mvWLE/nQdQPwd7J+4bW5n6xboGFMOtcfTwHDm56/pPliRFwVEW8ga5neTpbgeounEdN9/YypL84li2tcRLwY+BygXl7T45AvSSPIvnc4HzgtdR2ZtZ0T+RAVEY+R9Qt/R9LhkoZLepGkQyR9Ld02AzhF0mhJo9L9P+lnlfOA10kaI2kT4LONC5K2lHRo6itfQdZFszqnjCuBnSW9U9IwSUcBuwKX9zOmvtgYeBx4Mn1aOKHl+gPAjmu8qmf/DsyJiPeT9f1/b8BRmuVwIh/CIuIssjHkpwDLgHuBE4Ffplu+AswG5gN/Aeamc/2p62pgZiprDi9Mvl1ko1/uJxvJsR/pi8SWMh4C3pzufYhsxMmbI2J5f2Lqo0+RfZH6BNmnhZkt108DLpL0qKR39FaYpMOAg8m6kyD7e9hT0jFti9gs8YQgM7Oac4vczKzmnMjNzGrOidzMrOZKTeSSfilpjqQFkqb0/gozM+urUr/slDQyIh6WtCHZuhn7pZEJzfdMIU1vHj58owk7jevvxMFyDevubUhxZ1QzKrPBs3jxIpYvX97W/xW6X7x9xKo1JuvmimeWXRURB7ez/r4qe8W2j0g6Ij3eDhhHNqzsHyJiOjAd4BXjJ8Tl11xfckj9M/rF63c6hFySU7mt2/Z51cS2lxmrnmH9XXodZQrA3+d9p7cZwKUrLZFLmgQcCLwmIp6W9HuyBYzMzCpOoPp8hVhmi3wT4JGUxF9KtsqemVn1CajRp90yE/mvgamS5gN3ADeWWJeZWXu5RQ5ppbdDyirfzKw8gq7uTgdRmLenMjPL464VM7MaE+5aMTOrN7lFbmZWe26Rm5nVnFvkZmZ15glBZmb15glBZmZDgFvkZmZ1Juhuz4QgSReQ7UX7YETsls7NBBpLvW4KPBoR4yWNBW4jmw0PcGNETKUXTuRmZq3aO478QuAc4EeNExFx1D+qkr4BPNZ0/8KIGN+XCpzIzczytKmPPCKuTS3tnCok4B3A6wdSR306gczMBk0atVLkGJh9gQci4q6mcztIulnSHyTtW6QQt8jNzPIUb5GPkjS76fn0tGFOEZOBGU3PlwJjIuIhSROAX0p6eUQ83lMhlUrkw7rFyBHrdTqMXCP3OqnTIeSa9ZNTOx3CWk3aZXSnQzDrv+Kt7eUR0edtiiQNA94KTGicS6vGrkiP50haCOwMzM4tJHHXiplZK6n40X8HArdHxJLnq9VoSd3p8Y5k22Pe01tBTuRmZnna1EcuaQZwA7CLpCWSjk+XjuaF3SoArwPmS7oF+DkwNSIe7q2OSnWtmJlVRvtGrUxey/n35Jy7FLi0r3U4kZuZrcFrrZiZ1ZvwVm9mZvXmFrmZWf159UMzs5pzi9zMrObcIjczqzG5j9zMrP7cIjczqzc5kZuZ1Ve2ZacTuZlZfUmoy4nczKzW3CI3M6u5OiXy0sbXSBor6dayyjczK5OkQkcVuEVuZtZK6aiJske8D5N0kaT5kn4uaXjJ9ZmZDZgo1hqvSou87ES+C9lGpK8AHgc+1HqDpCmSZkuavXzZspLDMTMrxon8efdGxPXp8U+A17beEBHTI2JiREwcNdqb9ZpZNdQpkZfdRx69PDczq6SqJOkiym6Rj5H0mvR4MnBdyfWZmQ2c+nBUQNmJ/DbgOEnzgZHAuSXXZ2Y2YEJ0dXUVOqqgtK6ViFgE7FpW+WZmZXLXiplZ3bWpa0XSBZIebJ4gKek0SfdJmpeONzZd+6ykuyXdIelfioTqRG5m1kptHbVyIXBwzvmzI2J8Oq4EkLQrcDTw8vSa70rq7q0CJ3IzsxztSuQRcS3wcMFqDwN+FhErIuKvwN3AXr29yInczCxHHxL5qMakxnRMKVjFiWnW+wWSNkvntgHubbpnSTrXI6+1YmbWojFFv6DlETGxj1WcC3yZbG7Nl4FvAO8jv9e91/k3bpGbmeUpcRx5RDwQEasj4jngPJ7vPlkCbNd067bA/b2V50RuZtaqvV92rlm8tFXT0yOAxoiWy4CjJa0vaQdgHPDn3spz14qZWY52jSOXNAOYRNaXvgQ4FZgkaTxZt8ki4IMAEbFA0sXA/wKrgA9HxOre6nAiNzPL0a49OyNics7p83u4/3Tg9L7U4URuZpajTjM7ncjNzFpUaYnaIpzIzcxyOJGbmdWcE3k/CRjWXc0Rkb+75CudDiHXcefd2OkQ1mrOlw7qdAhrVdV/Z1Yh9cnj1UrkZmZV4Ra5mVmdyYnczKzWBNQojzuRm5mtSXS1aULQYHAiNzPL4a4VM7M6k7tWzMxqTeCuFTOzunOL3Mys5txHbmZWZ+4jNzOrt2wceX0yuRO5mdkavIytmVnt1SiPO5Gbma1BHn5oZlZr7iM3MxsCapTHncjNzPLUqUVe+jYpkt4tab6kWyT9uOz6zMzaQSp29F6OLpD0oKRbm86dKen2lBtnSdo0nR8r6RlJ89LxvSKxlprIJb0cOBl4fUTsDny0zPrMzNoibSxR5CjgQuDglnNXA7tFxCuAO4HPNl1bGBHj0zG1SAVlt8hfD/w8IpYDRMTDrTdImiJptqTZy5YvKzkcM7PeNTaWaEeLPCKuBR5uOfebiFiVnt4IbDuQeMtO5AKipxsiYnpETIyIiaNHjS45HDOzIoq1xlOLfFSjMZqOKX2s7H3AfzU930HSzZL+IGnfIgWU/WXnNcAsSWdHxEOSRua1ys3MqqYP33Uuj4iJ/atDJwOrgJ+mU0uBMSlfTgB+KenlEfF4T+WUmsgjYoGk04E/SFoN3Ay8p8w6zcwGbBAmBEk6DngzcEBEBEBErABWpMdzJC0EdgZm91RW6cMPI+Ii4KKy6zEza5eyJwRJOhj4DLBfRDzddH408HBErJa0IzAOuKe38jyO3MwsR7sSuaQZwCSyvvQlwKlko1TWB65O9dyYRqi8DviSpFXAamBqke5oJ3IzsxztapBHxOSc0+ev5d5LgUv7WocTuZlZjjrN7HQiNzNr5R2CzMzqTd5Ywsys/mqUx53IzczydNUokzuRm5nlqFEedyI3M2slQbe3ejMzq7ch8WWnpBf39MLeFnExM6uzGuXxHlvkC8iWoG3+dRrPAxhTYlxmZh0jsiGIdbHWRB4R2w1mIGZmVVKjLvJiG0tIOlrS59LjbdM6uWZmQ1PBTSWq0o/e65edks4BXkS2Kte/AU8D3wNeWW5o1TJ+7KadDiHX/3zhwE6HYDYkVSRHF1Jk1MreEbGnpJsh23dT0nolx2Vm1jFi6E0IWimpi7T3pqTNgedKjcrMrMNqlMcL9ZF/h2x93NGSvghcB/y/UqMyM+uwIdVHHhE/kjQHaHTGHhkRt5YblplZ5wzVmZ3dwEqy7pVCI13MzOqsPmm8QFKWdDIwA9ga2Bb4D0mfLTswM7NOGlJdK8C7gAmNnZ4lnQ7MAb5aZmBmZp2SjVrpdBTFFUnki1vuGwbcU044ZmYVUKHWdhFr7VqRdLaks8gmAC2Q9ANJ5wF/AR4drADNzDpBKnb0Xo4ukPSgpFubzo2UdLWku9LPzdJ5SfqWpLslzZe0Z5FYe2qRNypdAFzRdP7GIgWbmdVZG1vkFwLnAD9qOjcNuCYizpA0LT3/DHAIMC4drwLOTT971NOiWef3O2wzsxprZx95RFwraWzL6cOASenxRcDvyRL5YcCPIiKAGyVtKmmriFjaUx1F1lrZCTgd2BXYoCm4nQv9FmZmNVRyH/mWjeQcEUslbZHObwPc23TfknSux0ReZEz4hcAPyd6kDgEuBn7Wt5jNzOpDgm6p0AGMkjS76ZgykKpzzkVvLyoyamV4RFwl6esRsRA4RdIf+xyemVmN9KFBvjwiJvax+AcaXSaStgIeTOeXAM17QWwL3N9bYUVa5CuUfcZYKGmqpLcAW/T2IjOzOit5QtBlwHHp8XHAr5rOvzuNXnk18Fhv/eNQrEX+cWAE8BGyvvJNgPf1NWpJpwFPRsTX+/paM7PB1q4uckkzyL7YHCVpCXAqcAZwsaTjgf8Djky3Xwm8EbibbOj3e4vUUWTRrD+lh08Ax/YhfjOzWhJq23rkETF5LZcOyLk3gA/3tY61JnJJs+ihkz0i3tpb4WmdlneTfQu7jGxqv5lZtRWc7FMVPbXIzxlIwWlfz6OBPVI9c8lJ5Okb3ikA240ZM5Aqzczapk5T9HuaEHTNAMveF5jVtNjWZWupZzowHWDChIm9DrMxMxsMdVqvu+h65P3lxGxmtSPq1SIv803nWuAISRtK2hh4S4l1mZm1VZeKHVVQuEUuaf2IWFH0/oiYK2kmMI9sKVxPIjKzWhhyW71J2gs4n2z8+BhJuwPvj4iTenttRJxONvbczKxWapTHC3WtfAt4M/AQQETcAuxfZlBmZp3WrvXIB0ORrpWuiFjc0vG/uqR4zMw6LlvGtiJZuoAiifze1L0SkrqBk4A7yw3LzKyzhtrwwxPIulfGAA8Av03nzMyGrBo1yAuttfIg2QxNM7N1gtS+tVYGQ5FRK+eRM7EnIgayeLqZWaXVKI8X6lr5bdPjDYAjeOFWRGZmQ06dhh8W6VqZ2fxc0o+Bq0uLyMysw8QQmxCUYwdg+3YHYmZWGRWafl9EkT7yR3i+j7wLeBiYVmZQZmadptx9kKupx0Se9urcHbgvnXou7WBhZjZkZROCOh1FcT2OeU9Je1ZErE6Hk7iZrRPqtPphkclLf5a0Z+mRmJlViKRCRxX0tGfnsIhYBbwW+ICkhcBTZJ86IiLWqeT+92erubzMRuuXvTeI2bqnbl0rPWWBPwN7AocPUixmZtVQoZUNi+gpkQsgIhYOUixmZpXRrin6knYBmufj7Ah8AdgU+ACwLJ3/XERc2Z86ekrkoyV9Ym0XI+Ks/lRoZlZ17exaiYg7gPEAaQXZ+4BZwHuBsyPi6wOto6dE3g2MgBoNpjQzawvRXU7fygHAwpw9Hgakp0S+NCK+1LaazMxqQvSpj3yUpNlNz6dHxPS13Hs0MKPp+YmS3g3MBj4ZEY/0NVboefihW+Jmtm4qOIY8db8sj4iJTUduEpe0HnAocEk6dS6wE1m3y1LgG/0Nt6cW+QH9LdTMrO5KWI/8EGBuRDwA0PgJ/1gu/PL+FrzWFnlEPNzfQs3M6qzRtdLmzZcn09StImmrpmtHALf2N17PJjEzy9HOFrmk4cAbgA82nf6apPFkixIuarnWJ07kZmY52tmzEhFPA5u3nDu2XeU7kZuZtRDFFqKqCidyM7NWojILYhXhRG5mlqM+adyJ3MxsDYKyZnaWwonczCxHjfK4E7mZ2Zqqs2lEEU7kZmYtPGrFzGwIqFOLvNQ3HUnvkvRnSfMkfT+txWtmVnkqeFRBaYlc0suAo4B9ImI8sBo4Jue+KZJmS5q9bPmy1stmZoNPQ2Tz5TY4AJgA3JR+2Q2BB1tvSks+TgeYMGFilBiPmVkh7iN/noCLIuKzJdZhZlaKqrS2iyjzTeca4O2StgCQNFLS9iXWZ2bWNn3YWKLjSmuRR8T/SjoF+I2kLmAl8GFgcVl1mpm1Q9a1UpEsXUCpww8jYiYws8w6zMzKUKOeFY8jNzNbk5Bb5GZm9eYWuZlZjbmP3Mys7vq+sXJHOZGbmeVwIjczqzl/2WlmVmOiOpN9inAiNzPL0dXGvhVJi4AnyBYPXBUREyWNJJtnMxZYBLwjIh7pT/l1WhfGzGzQqOB/fbB/RIyPiInp+TTgmogYR7akybT+xupEbmbWotG1UvJaK4cBF6XHFwGH97cgJ3IzszUUbY8LYFRjT4V0TMkpMMjWnZrTdH3LiFgKkH5u0d9o3UduZtaqb+PIlzd1l6zNPhFxf1oN9mpJtw8ovhZukZuZ5WjnVm8RcX/6+SAwC9gLeEDSVgDp5xob7xRVqRb56gieWrGq02Hk6q7TWCQzG5Csj7w9/89L2gjoiogn0uODgC8BlwHHAWekn7/qbx2VSuRmZlXRxqbblsCstOPQMOA/IuLXkm4CLpZ0PPB/wJH9rcCJ3MwsT5syeUTcA+yec/4hsr2NB8yJ3MwsRzsnBJXNidzMLEd90rgTuZlZvhplcidyM7MW2dDC+mRyJ3Izs1beWMLMrP5qlMedyM3MctUokzuRm5mtoc9L1HaUE7mZWQ73kZuZ1VhfFsSqAidyM7McqlGT3InczCxHjfK4E7mZWZ4a5fHB2VhC0kck3Sbpp4NRn5nZgBTdVaIi2X6wWuQfAg6JiL8OUn1mZgNSp+GHpbfIJX0P2BG4TNLHy67PzGygRNZHXuSogtJb5BExVdLBwP4Rsbzs+szM2qEiObqQjm++LGmKpNmSZj+03HnezCqiRn3kHU/kETE9IiZGxMTNR43qdDhmZkBjkn7v/1WBhx+ameXoqkaOLsSJ3MwsT40S+aB0rUTEWH/RaWZ10dghqB1dK5K2k/S7NJdmgaSPpvOnSbpP0rx0vLG/8bpFbmbWqr1DC1cBn4yIuZI2BuZIujpdOzsivj7QCpzIzcxytCuPR8RSYGl6/ISk24Bt2lQ8UIFRK2ZmlVTC8ENJY4E9gD+lUydKmi/pAkmb9TdUJ3IzszUU7SEXwKjGXJh0TMktURoBXAp8LCIeB84FdgLGk7XYv9HfaN21YmaWow995MsjYmLPZelFZEn8pxHxC4CIeKDp+nnA5f2L1C1yM7M1tHPxQ2U7VJwP3BYRZzWd36rptiOAW/sbr1vkZmZ52jdqZR/gWOAvkualc58DJksaDwSwCPhgfytwIjczy9HVpvGHEXEd+W8LV7alApzIzcxy1WhipxO5mdkaKrTWeBFO5GZmueqTyZ3IzcxaNHYIqgsncjOzHDXK407kZmZ53CLvpy6J9YdVc47SsO5qxmVm5ajK7j9FVCqRm5lVRn3yuBO5mVmeGuVxJ3Izs1ZS+2Z2DgYncjOzPPXJ407kZmZ5apTHncjNzPLUqGfFidzMbE3y8EMzszqr2xR9z3IxM6s5t8jNzHLUqUXuRG5mlsN95GZmNZZNCOp0FMU5kZuZ5XEiNzOrN3etmJnVXJ2+7PTwQzOzHCp4FCpLOljSHZLuljSt3bE6kZuZ5WlTJpfUDXwHOATYFZgsadd2hlpqIpf0eUm3S7pa0gxJnyqzPjOzdlHB/wrYC7g7Iu6JiGeBnwGHtTPW0vrIJU0E3gbskeqZC8zJuW8KMCU9fXLjDbrvaFMIo4DlbSqr3aoaW1XjgurGVtW4YN2Jbfs2lfMPN8+dc9Xw9TSq4O0bSJrd9Hx6RExver4NcG/T8yXAqwYaY7Myv+x8LfCriHgGQNJ/5t2UfuHpedcGQtLsiJjY7nLboaqxVTUuqG5sVY0LHNtARMTBbSwur9kebSy/1K6VGn3na2ZWmiXAdk3PtwXub2cFZSby64C3SNpA0gjgTSXWZWZWVTcB4yTtIGk94GjgsnZWUFrXSkTcJOky4BZgMTAbeKys+nK0vbumjaoaW1XjgurGVtW4wLFVQkSsknQicBXQDVwQEQvaWYci2tpV88LCpRER8aSk4cC1wJSImFtahWZm66CyZ3ZOT+MlNwAuchI3M2u/UlvkZja4JA2PiKc7HYcNLs/stMqStKVUpxUvOkvSocC30xdqtg5xIh8EkrrSTyelgiRtA5xCNp25kn9uVYpL0ubAR4BvANtKGtnhkGwQDblELmnDTsfQKiKeSw+3lzSsSgmgQdLLOh1Di/vJZgLvAby1in9mwNYAkqqwiuizwCrg88BZwHM9394Zkl5S0b/LWhtSiTwN8fmapK9K2qQC8ewt6ej0+MPApcAPgWmNVnoVSDoBOFPSlp2OBbKWbmRf3jwHvBT4DHBYlRJA+rf2PUlnAB+StH4n44mIJ4BrgLcA8yPi0Sr9eQFI+mfgy8DbqhZb3VWhJdEWkj4EHAm8k2xdl20kfTki7upgWJsBX02t3bEpvrHAJOAMSdOaWusdkfpVpwKHRsQDnYylISJC0jHAScDxwPuA/YFhki6NDn9DL+lw4B3AocAsYEFErOhkTMnFZP/2z5H0SESc3emAGiS9GTgRGEFaG6UKf5dDRWVahQMh6cXAnmQzpt4G3JwufUvSuE7FFRFXkC0I9jZgWETcQzbj9cfAlmRLWnba1sDMiFgs6UWdDqbJLsDFETEf+DRwN1liP7ICrblNgG8ChwMrgU8ASNq5k0FFxOKIuIasMXNC+tTQcemT3jTgoxHxWuAXZI2ZSn3KqrMhkcgj4nHgw8AWwBFpwZvjgFcCx3byW/yIuBo4GXijpKMi4tn0KWEEUIV+6cXAvpJ2iYiVAJKOTa3OTpoL7CPp5RGxMiK+TTYrbneyP7tOWgScCRwfEQdFxLOSPgK8vwpvhhFxM/B24DRJH+x0PGT9913A6PT8fOBFwEeBf+lUUEPJkOlaiYgVkp4m+/j9z2SL1Pwa+EFaA7iTsf1K0rFknxB2Bf4M7ECWrDrtemAf4DhJ/wNsTDb6YXJHo4Lfk70RT5b038CGwMPAt1N/cCfNAX4FPCdpEjCGrOFwXOPNsNMiYn6K7ZkKxPKIpJ8D+0t6NCJulXQJ2afVoyT9riJdU7U1pCYEpS+cPgYcSNZ18Y6IuL2zUT0vtXIvBS4BpkXEos5GlJG0FdlC94eSrYfz1dSl0VGStgbemo5VwCcj4i+djSqT/swOTcdDwJlVia2KJG1L9l3MBLIGzOHAu4HTgFMi4pbORVd/QyqRA6SPti8BnouI+zodTytJ+wGLImJxp2Np1eiC6vQnmFaSNiL7t/pkp2Np1ehKqUpLvMrSd1l7k3WPXQlsRLZ41huq8kV7XQ25RG5m1Sdpf+CrwAfdGh84J3IzG3Spa2q9Kn4yrSMncjOzmhsSww/NzNZlTuRmZjXnRG5mVnNO5LYGSaslzZN0q6RL0lZ9/S1rkqTL0+NDJU3r4d5N05o5fa3jNEmfKnq+5Z4LJb29D3WNlXRrX2M0K5MTueV5JiLGR8RuZNOrpzZfVKbP/3Yi4rKIOKOHWzYF+pzIzdZ1TuTWmz8C/5RaordJ+i7ZzLztJB0k6QZJc1PLfQSApIMl3S7pOrJZmaTz75F0Tnq8paRZkm5Jx97AGcBO6dPAmem+T0u6SdJ8SV9sKutkSXdI+i3ZAls9kvSBVM4tki5t+ZRxoKQ/SrozrdKHpG5JZzbVXYU1S8xyOZHbWinbMOEQoDH1fBfgRxGxB/AU2Q4+B0bEnsBs4BOSNgDOI1sXe1+yWbZ5vgX8ISJ2J1u5cgHZCnkL06eBT0s6CBgH7AWMByZIep2kCWQrXe5B9kbxygK/zi8i4pWpvtvIlsdtGAvsB7yJbI3xDdL1xyLilan8D0jaoUA9ZoNuyCyaZW21oaR56fEfyVar2xpYHBE3pvOvJluG9/q0Eul6wA1kG0H8tbEOvKSfkC2O1Or1ZGttEBGrgcckbdZyz0HpaCxLPIIssW8MzGpsMizpsgK/026SvkLWfTMCuKrp2sVpXfi7JN2TfoeDgFc09Z9vkuq+s0BdZoPKidzyPBMR45tPpGT9VPMp4OqImNxy33igXbPMRLaA1/db6vhYP+q4EDg8Im6R9B6y9bAbWsuKVPdJEdGc8JE0to/1mpXOXSvWXzeSrRf+TwCShqeNFW4HdpC0U7pvbcvhXgOckF7bnRZUeoKstd1wFfC+pr73bSRtAVwLHCFpQ0kbk3Xj9GZjYGla5OqYlmtHSupKMe8I3JHqPqGxKJakndPiXWaV4xa59UtELEst2xl6fr/KUyLiTklTgCskLSfbEWm3nCI+CkyXdDywGjghIm6QdH0a3vdfqZ/8ZcAN6RPBk8C7ImKupJnAPLKNMf5YIOTPA39K9/+FF75h3HUARSQAAABNSURBVAH8gWzp46kR8XdJPyDrO5+rrPJlZEuvmlWO11oxM6s5d62YmdWcE7mZWc05kZuZ1ZwTuZlZzTmRm5nVnBO5mVnNOZGbmdXc/wevHeE2GmIQ0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAEmCAYAAACtaxGwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3debwcVZn/8c/33hASTAhCECQJBGUZgZEtoLIoCjIwSuKKoID8WCIooCJo/IEMogwoo44LilEQ0BHCIppRFBUHWQRJWCVgmBCJBBQSCCA7hGf+qHOh0+nbXTfpulV1833zqhddXdXnPN238/Tp0+ecUkRgZmb10lN2AGZmNnBO3mZmNeTkbWZWQ07eZmY15ORtZlZDTt5mZjXk5L2KkXSypB+l2xtKekJSb5fruFfSHt0sM0edR0p6MD2fdVainCckvaabsZVF0hxJu5UdhxXDybvLUuJ6UNIrGu47TNJVJYbVUkT8NSJGRcTSsmNZGZJWA74K7Jmez8MrWlZ6/PzuRdd9ks6V9MVO50XElhFx1SCEZCVw8i7GMODjK1uIMv4bdbYeMAKYU3YgVSBpWNkxWPGcGIpxBnCcpLVaHZS0k6RZkh5L/9+p4dhVkk6VdB3wFPCadN8XJf0hfa3/b0nrSPovSY+nMiY2lPF1SfelYzdJ2rWfOCZKCknDJL0pld23PSPp3nRej6Rpku6R9LCkiySt3VDOgZIWpGMntHthJI2U9JV0/mOSrpU0Mh2bnL7qP5qe8+saHnevpOMk3Z4eN0PSCEmbAXPTaY9K+l3j82p6XQ9LtzeR9PtUzmJJMxrOC0mbpNtjJJ0vaVGK98S+D1NJB6fY/0PSEkl/kbR3m+d9r6TjU/xPSjpb0nqSfinpH5J+K+mVDedfLOnvKcarJW2Z7p8KfAj4dN97oaH8z0i6HXgy/U1f6r6SdLmkrzSUP0PSOe3+VlZxEeGtixtwL7AH8BPgi+m+w4Cr0u21gSXAgWQt9P3T/jrp+FXAX4Et0/HV0n3zgNcCY4A7gbtTPcOA84EfNMRwALBOOvYp4O/AiHTsZOBH6fZEIIBhTc+hr87T0v4ngBuA8cDqwHeBC9KxLYAngDenY18FXgD26Of1OTOVPQ7oBXZKj9sMeBJ4e6r/0+k5D294XW8ENkiv4V3AEa2eR6vnleo8LN2+ADiBrPEyAtil4bwANkm3zwd+BoxOZd4NHJqOHQw8DxyenseRwAOA2rwvbiD7ljAOeAi4Gdg2Pf/fAf/WcP4hqd7Vgf8Ebm04di7pvdVU/q3ABGBk43sx3V4/1fk2suQ/Hxhd9r8XbyuRa8oOYKhtvJy8twIeA9Zl2eR9IHBj02OuBw5Ot68CTmk6fhVwQsP+V4BfNuzv0/iPu0VMS4Ct0+2T6Zy8vwP8AuhJ+3cBuzccf3VKXMOAk4ALG469AniOFsk7Jcun+2JpOvY54KKmc+8Hdmt4XQ9oOP5l4KxWz6PV82LZ5H0+MB0Y3yKOADYhS8jPAls0HPtIw9/xYGBew7E10mPXb/O++FDD/qXAdxr2jwZ+2s9j10plj0n759I6eR/S6r3YsP8e4D5gMQ0fWN7qubnbpCARcQfwc2Ba06ENgAVN9y0ga431ua9FkQ823H66xf6ovh1Jn5J0V/rK/ShZa31snrglfQTYDfhgRLyY7t4IuCx1ZzxKlsyXkrUiN2iMNyKeBPr7wXAsWUv3nhbHlnldUt33sezr8veG20/R8JwH6NOAgBtTN80h/cQ6nGX/Vs1/p5fiiYin0s12MeX6G0rqlXR66qZ6nCwJ98XUTqv3TaOfk30ozY2IazucaxXn5F2sfyP7Wt34D/4BsmTYaEOyVmafFV7qMfVvfwbYF3hlRKxF9g1AOR/7BWBKRDzWcOg+YO+IWKthGxER9wN/I/uq3lfGGmRdNq0sBp4h6/5ptszrIkmp3PtbnNvJk+n/azTct37fjYj4e0QcHhEbkLWmv93Xz90U6/Ms+7dq/jsV5YPAFLJvcGPIvknAy3/D/t4fnd43p5J98L5a0v4rGaOVzMm7QBExD5gBHNNw9+XAZpI+mH5U+gBZv/HPu1TtaLI+50XAMEknAWt2epCkCSnWgyLi7qbDZwGnStoonbuupCnp2CXAOyXtImk4cAr9vK9Sa/oc4KuSNkgtzDdJWh24CHiHpN2VDf37FFm3xR8G9OyzehaRJdkDUh2H0PCBIen9ksan3SVkSW9pUxlLU0ynShqdnvuxwI8GGs8KGE323B8m+wD696bjDwIDGosu6c3A/wMOSts3JY1r/yirMifv4p1C1g8MQGRjkN9JlpweJvsK/86IWNyl+q4Afkn249oCspZup6/TALuTtU4v0csjTvqG3n0dmAn8WtI/yH54e0N6PnOAjwE/JmuFLwEWtqnnOOBPwCzgEeBLZH3rc8l+aP0mWat3H2CfiHgu5/NudjhwPNlrvCXLfgjsAPxR0hPpeX08Iv7SooyjyVrx84Fr03McjBEa55P97e4n+3H6hqbjZwNbpG6sn3YqTNKaqcyjIuL+1GVyNvCD9A3HakgRvhiDmVnduOVtZlZDTt5mZgWSdI6khyTd0c9xSfqGpHlpEtd2ecp18jYzK9a5wF5tju8NbJq2qWTzLDoqNHlL+qmy6dlz0rReM7NVSkRcTfbjfH+mAOdH5gZgLUmv7lRu0QvYHBIRjyhbu2KWpEujacW3lNSzxN6z2vYaucKreRZq682qOaqqx2MFbBW3YMG9LF68uKv/EnrX3CjihadznRtPL5pDNqqrz/SImD6A6sax7Iiwhem+v7V7UNHJ+xhJ7063J5B9LVgmeacnOR2gZ9SrY/VtDi04pBXzu19/vuwQWho5vKtLcZvVzs5vmNT1MuOFp1l9831znfvMrWc+ExErE0SrD56OwwALS97KFoHfA3hTRDylbD3rEUXVZ2bWPYLBW415IQ2zlMkWgHug04OKjG4MsCQl7n8C3lhgXWZm3SNAyretvJnAQWnUyRuBxyKibZcJFNtt8ivgiLS+8FyWnyVmZlZdXWp5S7qAbLG3sZIWkq15tBpARJxFtmTGv5ItgfwU2TIGHRWWvCPiWbIhMGZmNSPo6c7vSRHRdhGwyKa5f2yg5fpySWZmrVR82RcnbzOzZmIwf7BcIU7eZmbL6dqPkYVx8jYza8UtbzOzGnLL28ysbgZ1ks4KcfI2M2vWN0mnwpy8zcxaccvbzKxuBL3VXvTNydvMrJnHeZuZ1ZT7vM3M6sajTczM6skt7/y22XwcV195StlhtLTlp39RdggtXfqJN5cdQr+2mjCm7BDMVpxb3mZmNdO9Cy0UxsnbzKwVt7zNzGrILW8zs7rxaBMzs/oRXbsMWlGcvM3MluOWt5lZPbnP28yshtzyNjOrIbe8zcxqRu7zNjOrJ7e8zczqR07eZmb1kl3C0snbzKxeJNTj5G1mVjtueZuZ1VDVk3dhY2EkTZR0R1Hlm5kVSVKurSxueZuZNVPaKqzoUejDJJ0n6XZJl0hao+D6zMxWmsjX6i6z5V108t4cmB4RrwceBz7afIKkqZJmS5q9eNGigsMxM8tnVU/e90XEden2j4Bdmk+IiOkRMSkiJo1dd92CwzEzy6fqybvoPu/osG9mVkmr7GiTZENJb0q39weuLbg+M7OVpwFsJSk6ed8FfFjS7cDawHcKrs/MbKUJ0dPTk2srS2HdJhFxL7BFUeWbmRWpm90mkvYCvg70At+PiNObjm8InAeslc6ZFhGXtyuz2gvWmpmVpUvdJpJ6gTOBvckatPtLam7YnghcFBHbAvsB3+5UrpO3mVkzdXW0yY7AvIiYHxHPARcCU5rOCWDNdHsM8ECnQj3D0syshQF0m4yVNLthf3pETG/YHwfc17C/EHhDUxknA7+WdDTwCmCPTpU6eZuZtTCA5L04Iia1K6rFfc3DpvcHzo2Ir6QRej+UtFVEvNhfoU7eZmZN+qbHd8lCYELD/niW7xY5FNgLICKulzQCGAs81F+h7vM2M2ule+O8ZwGbStpY0nCyHyRnNp3zV2B3AEmvA0YAbdcLccvbzKyZujdUMCJekHQUcAXZMMBzImKOpFOA2RExE/gU8D1JnyTrUjk4ItrOSHfyNjNroZvjvNOY7cub7jup4fadwM4DKdPJ28ysBV/D0syshqq+MJWTt5lZk7KXe83DydvMrAUnbzOzGnLyHgABw3qrOfR8xtG7lh1CS/t/+w9lh9CvW774L2WH0K+qvs+sQqqdu6uVvM3MqsItbzOzuuniJJ2iOHmbmTURUPHc7eRtZrY80eNJOmZm9eNuEzOzupG7TczMakfgbhMzszpyy9vMrIbc521mVjfu8zYzq59snHe1s7eTt5nZcrwkrJlZLVU8dzt5m5ktRx4qaGZWO+7zNjOrqYrnbidvM7NWqt7yLvxyIpIOknS7pNsk/bDo+szMukHKt5Wl0Ja3pC2BE4CdI2KxpLWLrM/MrCtqcDGGolvebwMuiYjFABHxSPMJkqZKmi1p9qLFiwoOx8yss76LMVS55V108hYQ7U6IiOkRMSkiJq07dt2CwzEzyyObpJNnK0vRyftKYF9J6wC428TM6qLqLe9C+7wjYo6kU4HfS1oK3AIcXGSdZmYrzZN0ICLOA84ruh4zs27xJB0zs5py8jYzq6GK524nbzOzVtzyNjOrG19Jx8ysfuSLMZiZ1VPFc7eTt5lZKz0Vz96FrypoZlZH3ZxhKWkvSXMlzZM0rZ9z9pV0p6Q5kn7cqUy3vM3MmkjQ26UZlpJ6gTOBtwMLgVmSZkbEnQ3nbAp8lmwF1iWSXtWpXLe8zcxa6OLCVDsC8yJifkQ8B1wITGk653DgzIhYAhARD3UqtN+Wt6Q12z0wIh7vGLKZWU0NoMt7rKTZDfvTI2J6w/444L6G/YXAG5rK2CyrU9cBvcDJEfGrdpW26zaZQ7aca+NT6NsPYMN2BZuZ1ZXIhgvmtDgiJnUorlnzUtnDgE2B3YDxwDWStoqIR/srtN/kHRET2gRjZjakdXFRwYVAYz4dDzzQ4pwbIuJ54C+S5pIl81n9xpenZkn7Sfr/6fZ4SdsPJHIzs1rJ2d+ds897FrCppI0lDQf2A2Y2nfNT4K1Z1RpL1o0yv12hHUebSPoWsBrwZuDfgaeAs4Ad8kQ9VGwzca2yQ2hp+PDeskPo1/XzHy47hH7tuqmv2mTtdWuYd0S8IOko4Aqy/uxz0rUOTgFmR8TMdGxPSXcCS4HjI6LtP6A8QwV3iojtJN2SAnkkfXqYmQ1JoruTdCLicuDypvtOargdwLFpyyVP8n5eUg+pgz1d0uzFvBWYmdVRxSdY5urzPhO4FFhX0ueBa4EvFRqVmVnJqn4B4o4t74g4X9JNwB7prvdHxB3FhmVmVp5uzrAsSt7p8b3A82RdJ56VaWZDXrVTd45ELOkE4AJgA7LxiT+W9NmiAzMzK1Ptu02AA4DtI+IpAEmnAjcBpxUZmJlZWbLRJmVH0V6e5L2g6bxhdBg8bmZWayW3qvNotzDV18j6uJ8C5ki6Iu3vSTbixMxsyKp47m7b8u4bUTIH+EXD/TcUF46ZWTXUtuUdEWcPZiBmZlUxJPq8Jb0WOBXYAhjRd39EbFZgXGZmpap6yzvPmO1zgR+QfRjtDVxEdiUIM7MhSYJeKddWljzJe42IuAIgIu6JiBNJSxeamQ1V3bwAcRHyDBV8Vtn3h3skHQHcD3S8OKaZWZ1VvdskT/L+JDAKOIas73sMcMhAK5J0MvBERPzHQB9rZjbYKp67cy1M9cd08x/AgcWGY2ZWPqGuruddhHaTdC5j+YtkviQi3tOp8LQuykFkV05eRDat3sys2kruz86jXcv7WytTcLrO5X7Atqmem2mRvCVNBaYCTNjQF6Q3s2qobZ93RFy5kmXvClzWsKBV8wU3++qZDkwH2H77Sf229M3MBlPV177Ou573inIyNrPaEdVveRf54XI18G5JIyWNBvYpsC4zs67qUb6tLLlb3pJWj4hn854fETdLmgHcSras7DUrEJ+Z2aAbEpdBk7QjcDbZ+O4NJW0NHBYRR3d6bEScSjY23MysViqeu3N1m3wDeCfwMEBE3Ianx5vZEDcUpsf3RMSCps77pQXFY2ZWumxJ2Go3vfMk7/tS10lI6gWOBu4uNiwzs3INhaGCR5J1nWwIPAj8Nt1nZjZkVbzhnWttk4fIZkqama0SpBqvbdJH0vdoMdkmIqYWEpGZWQVUPHfn6jb5bcPtEcC7yRaaMjMbsqo+VDBPt8mMxn1JPwR+U1hEZmYlE0Ngkk4LGwMbdTsQM7PKKHnqex55+ryX8HKfdw/wCDCtyKDMzMomqp292ybvdO3KrcmuWwnwYkR4pUAzG9KySTplR9Fe23HoKVFfFhFL0+bEbWarhKqvKphnEtGNkrYrPBIzswqRlGsrS7trWA6LiBeAXYDDJd0DPEn2jSIiYpVK6K/c4aiyQ2hpyayVulqdmbVQh26Tdn3eNwLbAe8apFjMzKqhyysGStoL+DrQC3w/Ik7v57z3ARcDO0TE7HZltkveAoiIe1YsXDOz+urW9Pi0oN+ZwNuBhcAsSTMj4s6m80YDxwB/zFNuu+S9rqRj+zsYEV/NU4GZWd10udtkR2BeRMwHkHQhMAW4s+m8LwBfBo7LU2i7Hyx7gVHA6H42M7MhSvQq3waMlTS7YWte92kcyy4psjDd93Jt0rbAhIj4ed4I27W8/xYRp+QtyMxsqMiuHp/79MURMalDcc1eGnYtqQf4GnBw7hrJ0edtZrbK6e4Y7oXAhIb98cADDfujga2Aq9LQw/WBmZImt/vRsl3y3n3FYzUzq7curuc9C9hU0sZks9X3Az7YdzAiHgPG9u1Lugo4rtNok377vCPikZUM2Myslvq6TbpxAeI0X+Yo4ArgLuCiiJgj6RRJk1c0xhVZVdDMbMjr5pV0IuJy4PKm+07q59zd8pTp5G1m1sJQuJKOmdkqRQyNq8ebma1aRKmLTuXh5G1m1kK1U7eTt5nZcgR9sycry8nbzKyFiuduJ28zs+WVe6GFPJy8zcyaeLSJmVlNVb3lXeiHi6QDJN0o6VZJ302LkpuZVZ5ybmUpLHlLeh3wAWDniNgGWAp8qMV5U/vWwV20eFFR4ZiZ5acaX4C4C3YHtie75A/ASOCh5pMiYjowHWD77SdF83Ezs8G2qvd5CzgvIj5bYB1mZoVYlfu8rwTeJ+lVAJLWlrRRgfWZmXVNj/JtZSms5R0Rd0o6Efh1uszP88DHgAVF1Wlm1g1Zt0m1W96FDhWMiBnAjCLrMDMrQsV7TTzO28xseUKrcsvbzKyu3PI2M6uZVb7P28yslnJeXLhMTt5mZi04eZuZ1ZB/sDQzqxlR7gScPJy8zcxa6Kl4v4mTt5lZC+42MTOrGXebmJnVkmdYmpnVj8d5m5nVU8Vzd7WS9z0PP8kHfjCr7DBaOu3rx5YdgpkNkqzPu9rpu1LJ28ysKqqdup28zcxaq3j2dvI2M2vB3SZmZjVU7dTt5G1m1lrFs7eTt5lZE+Hp8WZm9eNJOmZm9VTx3E1P2QGYmVWScm55ipL2kjRX0jxJ01ocP1bSnZJul3SlpI06lenkbWa2HOX+r2NJUi9wJrA3sAWwv6Qtmk67BZgUEa8HLgG+3KlcJ28zsxakfFsOOwLzImJ+RDwHXAhMaTwhIv4nIp5KuzcA4zsV6uRtZtYkb49Jyt1jJc1u2KY2FTcOuK9hf2G6rz+HAr/sFKN/sDQza0H5h5ssjohJ7YpqcV/0U+cBwCTgLZ0qdfI2M2uhi0MFFwITGvbHAw8sX5/2AE4A3hIRz3Yq1N0mZmYtdHGwySxgU0kbSxoO7AfMXKYuaVvgu8DkiHgoT6GDkrwlHSPpLkn/NRj1mZmtlAF2ercTES8ARwFXAHcBF0XEHEmnSJqcTjsDGAVcLOlWSTP7Ke4lg9Vt8lFg74j4yyDVZ2a2Uro5PT4iLgcub7rvpIbbewy0zMJb3pLOAl4DzJT0yaLrMzNbWaKrQwULUXjLOyKOkLQX8NaIWFx0fWZm3VD16fGljzZJYyKnAoxce/2SozEzSyqevUsfbRIR0yNiUkRMGj56rbLDMTMD8k+QL0vpLW8zsyrqqXjL28nbzKwVJ2+IiImDUY+ZWTf4SjpmZnXkK+mYmdVTxXO3k7eZWUsVz95O3mZmyyl3GGAeTt5mZi24z9vMrGYGsNxraZy8zcxaqXj2dvI2M2uhp+L9Jk7eZmYtVDt1O3mbmS3Pk3TMzOqq2tnbydvMrEnflXSqzMnbzKyFiuduJ28zs1bc8h6Axx96mF9/+7yyw2hpxqxvlR2CmQ0iT483M6ujauduJ28zs1YqnrudvM3MmkmeYWlmVk/Vzt1O3mZmrVQ8dzt5m5m1UvFeEydvM7Pl+Uo6Zma1U4fp8T1lB2BmZgPnlreZWQtVb3k7eZuZteA+bzOzmskm6ZQdRXtO3mZmrTh5m5nVj7tNzMxqqOo/WHqooJlZC8q55SpL2kvSXEnzJE1rcXx1STPS8T9KmtipTCdvM7NWupS9JfUCZwJ7A1sA+0vaoum0Q4ElEbEJ8DXgS53KLTR5S/qcpD9L+o2kCyQdV2R9Zmbdopz/5bAjMC8i5kfEc8CFwJSmc6YAfZcRuwTYXWrfcVNYn7ekScB7gW1TPTcDN7U4byowNe0+8cytZ87tUghjgcVdKouRq53ZraKgy7F1UVXjgurGVtW4YNWJbaMulfOSW26+6Yo1hmtsztNHSJrdsD89IqY37I8D7mvYXwi8oamMl86JiBckPQasQ5vXqMgfLHcBfhYRTwNI+u9WJ6UnOb3VsZUhaXZETOp2ud1Q1diqGhdUN7aqxgWObWVExF5dLK5VCzpW4JxlFNltUvHfas3MBsVCYELD/njggf7OkTQMGAM80q7QIpP3tcA+kkZIGgW8o8C6zMyqahawqaSNJQ0H9gNmNp0zE/hwuv0+4HcR0bblXVi3SUTMkjQTuA1YAMwGHiuqvha63hXTRVWNrapxQXVjq2pc4NgqIfVhHwVcAfQC50TEHEmnALMjYiZwNvBDSfPIWtz7dSpXHZL7SpE0KiKekLQGcDUwNSJuLqxCM7NVRNEzLKen8YwjgPOcuM3MuqPQlreZDS5Ja0TEU2XHYcXzDEurLEnrdZqoYC+TNBn4ZvpRzIY4J+9BIKkn/d+JKCdJ44ATyaYSV/J1q1JcktYBjgG+AoyXtHbJIVnBhlzyljSy7BiaRcSL6eZGkoZV6R99H0mvKzuGJg+QzcjdFnhPFV8zYAN4aVxu2Z4DXgA+B3wVeLH96eWQtH5F/5a1M6SSdxqO82VJp0kaU4F4dpK0X7r9MeBS4AfAtL7WeBVIOhI4Q9J6ZccCWYs2jXF9Efgn4DPAlCr9o0/vtbMknQ58VNLqZcYTEf8ArgT2AW6PiEer9HoBSPpn4AvAe6sWWx1VocXQFZI+Crwf+CDZOirjJH0hIv63xLBeCZyWWrUTU3wTgd2A0yVNa2iVlyL1kx4BTI6IB8uMpU9EhKQPAUeTrbZ2CPBWYJikSztNXiiapHcB+wKTgcuAORHxbJkxJReRvfe/JWlJRHyt7ID6SHoncBQwirQWSRX+lnVWmdbfypC0JrAd2cD29wK3pEPfkLRpWXFFxC/IFt16LzAsIuaTzTz9IbAe2fKQZdsAmBERCyStVnYwDTYHLoqI24HjgXlkyfz9FWi1jQH+E3gX8DxwLICkzcoMKiIWRMSVZA2YI9O3g9Klb3TTgI9HxC7AT8gaMJX6NlU3QyJ5R8TjwMeAVwHvTovKfBjYATiwzF/fI+I3wAnAv0r6QEQ8l74NjAKq0M+8ANhV0uYR8TyApANT67JMNwM7S9oyIp6PiG+SzU7bmuy1K9O9wBnAoRGxZ0Q8J+kY4LAqfABGxC1kU6xPlvSRsuMh64/vAdZN+2cDqwEfB/6lrKDqbsh0m0TEs5KeIvtq/c9ki7z8Cvh+WkO3zNh+JulAsm8CWwA3AhuTJaiyXQfsDHxY0h+A0WSjFvYvNSq4iuzDd39JvwNGkk0b/mbq3y3TTcDPgBcl7QZsSNZY+HDfB2DZIuL2FNvTFYhliaRLgLdKejQi7pB0Mdm30g9I+p+KdDvVypCapJN+NPoEsAdZt8S+EfHncqN6WWrNXgpcDEyLiHvLjSgj6dVki8FPJlt/5rTUXVEqSRsA70nbC8CnIuJP5UaVSa/Z5LQ9DJxRldiqSNJ4st9WtidrtLwLOAg4GTgxIm4rL7p6GlLJGyB9bV0feDEi7i87nmaS3gLcGxELyo6lWV/3UtnfVJpJegXZe/WJsmNp1tdNUpUWd5Wl36Z2Iuv6uhx4BdkCVW+vyo/ldTLkkreZVZ+ktwKnAR9xq3vFOHmb2aBL3U7Dq/gNtC6cvM3MamhIDBU0M1vVOHmbmdWQk7eZWQ05edtyJC2VdKukOyRdnC5jt6Jl7Sbp5+n2ZEnT2py7VlqjZqB1nCzpuLz3N51zrqT3DaCuiZLuGGiMZt3m5G2tPB0R20TEVmRTm49oPKjMgN87ETEzIk5vc8pawICTt9mqyMnbOrkG2CS1OO+S9G2yGXITJO0p6XpJN6cW+igASXtJ+rOka8lmR5LuP1jSt9Lt9SRdJum2tO0EnA68NrX6z0jnHS9plqTbJX2+oawTJM2V9FuyRazaknR4Kuc2SZc2fZvYQ9I1ku5Oq98hqVfSGQ11V2GNELOXOHlbv5RdZGBvoG/a9+bA+RGxLfAk2ZVu9oiI7YDZwLGSRgDfI1tXeley2a6tfAP4fURsTbYi5ByylefuSa3+4yXtCWwK7AhsA2wv6c2StidbQXJbsg+HHXI8nZ9ExA6pvrvIlprtMxF4C/AOsjW6R6Tjj0XEDqn8wyVtnKMes0ExZBamsq4aKenWdPsaslXgNgAWRMQN6f43ki1pe11a1XM4cD3ZxRP+0reOuqQfkS1A1OxtZGtbEBFLgcckvbLpnD3T1rfE7yiyZD4auKzvQruSZuZ4TltJ+iJZ18wo4IqGYxelddX/V9L89Bz2BF7f0B8+JtV9d466zArn5G2tPB0R2zTekRL0k413Ab+JiP2bztsG6NbML5EtkvXdpjo+sQJ1nAu8Ky2KIXcAAAEhSURBVCJuk3Qw2XrSfZrLilT30RHRmOSRNHGA9ZoVwt0mtqJuIFtvexMASWukixH8GdhY0mvTef0tLXslcGR6bG9atOgfZK3qPlcAhzT0pY+T9CrgauDdkkZKGk3WRdPJaOBvaSGpDzUde7+knhTza4C5qe4j+xaekrRZWiDLrBLc8rYVEhGLUgv2Ar18/cYTI+JuSVOBX0haTHbloK1aFPFxYLqkQ4GlwJERcb2k69JQvF+mfu/XAdenlv8TwAERcbOkGcCtZBeTuCZHyJ8D/pjO/xPLfkjMBX5PtozwERHxjKTvk/WF36ys8kVky5iaVYLXNjEzqyF3m5iZ1ZCTt5lZDTl5m5nVkJO3mVkNOXmbmdWQk7eZWQ05eZuZ1dD/Aci5Ng5CO8lhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import svm, datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "# Compute confusion matrix\n",
    "cm = confusion_matrix(y_compare, pred)\n",
    "np.set_printoptions(precision=2)\n",
    "print('Confusion matrix, without normalization')\n",
    "print(cm)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, products)\n",
    "\n",
    "# Normalize the confusion matrix by row (i.e by the number of samples\n",
    "# in each class)\n",
    "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "print('Normalized confusion matrix')\n",
    "print(cm_normalized)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm_normalized, products, title='Normalized confusion matrix')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}