09_kmeans/09_01_roc_comp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ROC Curves\n",
    "\n",
    "<img src=\"https://kaggle2.blob.core.windows.net/competitions/inclass/4050/media/AUC.jpeg\"/, width=\"500\">\n",
    "The ROC (Receiver Operating Characteristic) curve or ROC curve is a plot with the True Positive Rate (Sensitivity) on the y-axis, and the False Positive Rate (1 - Specificity) on the x-axis.\n",
    "\n",
    "\n",
    "    True Positive Rate: TP / (TP + FN) [aka. Recall]\n",
    "    False Positive Rate: 1 - TN/(TN + FP) = FP/(FP + TN)\n",
    "\n",
    "---\n",
    "per Wiki:\n",
    "A receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied. It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate) vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.\n",
    "\n",
    "-----\n",
    "\n",
    "The 'ideal' decision boundary provided by logistic regression is the boundary at which the probability of binary classes are even (i.e. P(A) == P(A') )\n",
    "\n",
    "We can modulate the threshold for acceptance if we want a higher or lower True Positive Rate, traded off for a higher or lower False Positive Rate.\n",
    "\n",
    "-----\n",
    "\n",
    "The dotted line in an ROC curve represents when TPR == FPR. Models with curves above this line imply models better than random, while lines below the line imply models worse than random choice. \n",
    "\n",
    "\n",
    "The AUC (Area Under Curve) is the integral of the ROC curve. It can be used as a rough comparison measure when comparing multiple classifiers. An AUC of 1 implies a perfect model (i.e. the integral == 1 across the entire curve). In addition, AUC is a better metric than Accuracy for imbalanced classes. \n",
    "\n",
    "-----\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Heart Disease\n",
    "\n",
    "\n",
    "The following dataset is a set of data from the UCI data storage. Patients who visited Cleveland hospitals had 76 attributes about them collected, as well as the number of heart diseases they had.\n",
    "\n",
    "Typically, only 14 of the attributes have been used for learning, and rather than predicting how many heart diseases a patient has, we will look at the presence or absence of heart disease.\n",
    "\n",
    "\n",
    "\n",
    "[UCI Heart Disease Link](https://archive.ics.uci.edu/ml/datasets/Heart+Disease)\n",
    "\n",
    "\n",
    "** Unlike in lecture 8, we would like to compare multiple models' performance using the ROC curve and AUC **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import linear_model\n",
    "from sklearn import naive_bayes\n",
    "from sklearn import metrics\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sb\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "features = [\"age\",\"sex\",\"cp\",\"trestbps\",\"chol\",\"fbs\",\n",
    "           \"restecg\",\"thatlach\",\"exang\",\"oldpeak\",\"slope\",\n",
    "           \"ca\",\"thal\",\"num\"]\n",
    "data = pd.read_csv(\"https://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data\",names=features  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thatlach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>120</td>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thatlach  exang  oldpeak  \\\n",
       "0   63    1   1       145   233    1        2       150      0      2.3   \n",
       "1   67    1   4       160   286    0        2       108      1      1.5   \n",
       "2   67    1   4       120   229    0        2       129      1      2.6   \n",
       "3   37    1   3       130   250    0        0       187      0      3.5   \n",
       "4   41    0   2       130   204    0        2       172      0      1.4   \n",
       "\n",
       "   slope   ca thal  num  \n",
       "0      3  0.0  6.0    0  \n",
       "1      2  3.0  3.0    2  \n",
       "2      2  2.0  7.0    1  \n",
       "3      3  0.0  3.0    0  \n",
       "4      1  0.0  3.0    0  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thatlach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>54.438944</td>\n",
       "      <td>0.679868</td>\n",
       "      <td>3.158416</td>\n",
       "      <td>131.689769</td>\n",
       "      <td>246.693069</td>\n",
       "      <td>0.148515</td>\n",
       "      <td>0.990099</td>\n",
       "      <td>149.607261</td>\n",
       "      <td>0.326733</td>\n",
       "      <td>1.039604</td>\n",
       "      <td>1.600660</td>\n",
       "      <td>0.937294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.038662</td>\n",
       "      <td>0.467299</td>\n",
       "      <td>0.960126</td>\n",
       "      <td>17.599748</td>\n",
       "      <td>51.776918</td>\n",
       "      <td>0.356198</td>\n",
       "      <td>0.994971</td>\n",
       "      <td>22.875003</td>\n",
       "      <td>0.469794</td>\n",
       "      <td>1.161075</td>\n",
       "      <td>0.616226</td>\n",
       "      <td>1.228536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>94.000000</td>\n",
       "      <td>126.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>48.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>211.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>133.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>56.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>241.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>153.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>61.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>275.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>166.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>77.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>564.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>202.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              age         sex          cp    trestbps        chol         fbs  \\\n",
       "count  303.000000  303.000000  303.000000  303.000000  303.000000  303.000000   \n",
       "mean    54.438944    0.679868    3.158416  131.689769  246.693069    0.148515   \n",
       "std      9.038662    0.467299    0.960126   17.599748   51.776918    0.356198   \n",
       "min     29.000000    0.000000    1.000000   94.000000  126.000000    0.000000   \n",
       "25%     48.000000    0.000000    3.000000  120.000000  211.000000    0.000000   \n",
       "50%     56.000000    1.000000    3.000000  130.000000  241.000000    0.000000   \n",
       "75%     61.000000    1.000000    4.000000  140.000000  275.000000    0.000000   \n",
       "max     77.000000    1.000000    4.000000  200.000000  564.000000    1.000000   \n",
       "\n",
       "          restecg    thatlach       exang     oldpeak       slope         num  \n",
       "count  303.000000  303.000000  303.000000  303.000000  303.000000  303.000000  \n",
       "mean     0.990099  149.607261    0.326733    1.039604    1.600660    0.937294  \n",
       "std      0.994971   22.875003    0.469794    1.161075    0.616226    1.228536  \n",
       "min      0.000000   71.000000    0.000000    0.000000    1.000000    0.000000  \n",
       "25%      0.000000  133.500000    0.000000    0.000000    1.000000    0.000000  \n",
       "50%      1.000000  153.000000    0.000000    0.800000    2.000000    0.000000  \n",
       "75%      2.000000  166.000000    1.000000    1.600000    2.000000    2.000000  \n",
       "max      2.000000  202.000000    1.000000    6.200000    3.000000    4.000000  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#looks like issues with ca and thal columns. let's remove them for now\n",
    "data = data.drop(['ca','thal'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Any heart disease amount greater > 0, we'll set to 1.\n",
    "# This way we have a binary class for \"disease\" and \"no disease\"\n",
    "data.ix[ data.num > 0, 'num'] = 1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thatlach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>299</th>\n",
       "      <td>68</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>144</td>\n",
       "      <td>193</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>141</td>\n",
       "      <td>0</td>\n",
       "      <td>3.4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>130</td>\n",
       "      <td>131</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>115</td>\n",
       "      <td>1</td>\n",
       "      <td>1.2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>301</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>174</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>138</td>\n",
       "      <td>175</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>173</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>100000</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>30000</td>\n",
       "      <td>1000000</td>\n",
       "      <td>100</td>\n",
       "      <td>2</td>\n",
       "      <td>1000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        age  sex  cp  trestbps     chol  fbs  restecg  thatlach  exang  \\\n",
       "299      68    1   4       144      193    1        0       141      0   \n",
       "300      57    1   4       130      131    0        0       115      1   \n",
       "301      57    0   2       130      236    0        2       174      0   \n",
       "302      38    1   3       138      175    0        0       173      0   \n",
       "303  100000    0  10     30000  1000000  100        2      1000      0   \n",
       "\n",
       "     oldpeak   slope  num  \n",
       "299      3.4       2    1  \n",
       "300      1.2       2    1  \n",
       "301      0.0       2    1  \n",
       "302      0.0       1    0  \n",
       "303      0.0  100000    1  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## OUTLIER BLOCK , SAVE FOR LATER\n",
    "\n",
    "#data.head()\n",
    "data.loc[303] = pd.Series([100000,0,10,30000,1000000,100,2,1000,0,0,100000,1],index=data.columns)\n",
    "data.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:\n",
      "   age  sex  cp  trestbps  chol  fbs  restecg  thatlach  exang  oldpeak  slope\n",
      "0   63    1   1       145   233    1        2       150      0      2.3      3\n",
      "1   67    1   4       160   286    0        2       108      1      1.5      2\n",
      "2   67    1   4       120   229    0        2       129      1      2.6      2\n",
      "3   37    1   3       130   250    0        0       187      0      3.5      3\n",
      "4   41    0   2       130   204    0        2       172      0      1.4      1\n",
      "y:\n",
      "0    0\n",
      "1    1\n",
      "2    1\n",
      "3    0\n",
      "4    0\n",
      "Name: num, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Create our data\n",
    "X = data.ix[:,:-1]\n",
    "y = data.num \n",
    "print 'X:\\n',X.head()\n",
    "print 'y:\\n',y[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Comparison\n",
    "\n",
    "We want to compare our classifiers by their accuracy as well as their ROC curves.\n",
    "\n",
    "To do that, we'll create separate train and test splits (not CV since we want a common test set across all the models).\n",
    "\n",
    "In practice if you have enough data, you ideally want to have a separate test set. If you need to do cross validation, you can do it within your training set, but the test set should be saved for last and untouched until you have many models to compare.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.815217391304 AUC: 0.890961262554\n"
     ]
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "\n",
    "model = linear_model.LogisticRegression() \n",
    "model.fit(X_train,y_train) \n",
    "y_pred  = model.predict(X_test) \n",
    "y_pred_prob = model.predict_proba(X_test)\n",
    "\n",
    "# We are going to pass in the true y, as well as the predicted *PROBABILITIES*\n",
    "fpr_lr, tpr_lr, thresholds_lr = metrics.roc_curve(y_test, y_pred_prob[:,1])\n",
    "roc_auc_lr = metrics.roc_auc_score(y_test,y_pred_prob[:,1]) \n",
    "\n",
    "print \"Score:\",model.score(X_test,y_test), \"AUC:\", roc_auc_lr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.445652173913 AUC: 0.519607843137\n"
     ]
    }
   ],
   "source": [
    "from sklearn import naive_bayes\n",
    "\n",
    "model = naive_bayes.GaussianNB()\n",
    "model.fit(X_train,y_train)\n",
    "y_pred  = model.predict(X_test)\n",
    "y_pred_prob = model.predict_proba(X_test)\n",
    "\n",
    "# We are going to pass in the true y, as well as the predicted *PROBABILITIES*\n",
    "fpr_nb, tpr_nb, thresholds_nb = metrics.roc_curve(y_test, y_pred_prob[:,1])\n",
    "roc_auc_nb = metrics.roc_auc_score(y_test,y_pred_prob[:,1]) \n",
    "\n",
    "print \"Score:\",model.score(X_test,y_test), \"AUC:\", roc_auc_nb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K Nearest Neighbors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.684782608696 AUC: 0.708273553324\n"
     ]
    }
   ],
   "source": [
    "from sklearn import neighbors\n",
    "model = neighbors.KNeighborsClassifier()\n",
    "model.fit(X_train,y_train)\n",
    "y_pred_prob = model.predict_proba(X_test)\n",
    "\n",
    "# We are going to pass in the true y, as well as the predicted *PROBABILITIES*\n",
    "fpr_knn, tpr_knn, thresholds_knn = metrics.roc_curve(y_test, y_pred_prob[:,1])\n",
    "roc_auc_knn = metrics.roc_auc_score(y_test,y_pred_prob[:,1]) \n",
    "\n",
    "print \"Score:\",model.score(X_test,y_test), \"AUC:\", roc_auc_knn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot Out ROC curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x10e2f44d0>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAECCAYAAAAFL5eMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XdgVNeZ+P3v9JFmRqMuIYE6XBW6JJrpGGxjTLcx2Imd\n2InXKfvuJtkkjjdxkk3WKa+TTVmXOGXjBLBNNcYNjOnGNhIdpKtGEah3jUbStPv7Y0CAwQiwpCk6\nn7+YuVczD1fSozPnOfc8KkVREARBEIKL2tcBCIIgCH1PJHdBEIQgJJK7IAhCEBLJXRAEIQiJ5C4I\nghCERHIXBEEIQjeV3CVJmihJ0s7rPH+fJEmfSJL0oSRJj/d9eIIgCMLt6DW5S5L0XeBlwPCp53XA\nb4C5wAzgq5IkxfZHkIIgCMKtuZmRexmwFFB96vksoEyW5VZZlp3APmB6H8cnCIIg3IZek7ssyxsB\n13UOhQGtVzxuB6x9FJcgCILwOXyegmorYLnisQVo/nzhCIIgCH1B+zm+thgYLklSBNCBd0rm1zf6\nAkVRFJXq07M7giAMVo/9bBsNrV1EW42+DsVnNB4Xce01JLSdJ6HtAlH2BlRAp9bMJ8MWEOvcxuP/\n+/ItJ85bSe4KgCRJKwGzLMsvS5L0LeA9vJ8A/iLLcvWNXkClUlFf336rMQalmBiLuBYXiWtx2WC7\nFm63QoTZwC+emHzNsWC9ForHQ3flOeynTmE/dZLOshIUp9N7TKvFnhzP8YgOzsRpibO0k5f31G29\nz00ld1mWzwBTLv577RXPbwW23tY7C4IgDBLOhnrsp07Rceok9uJTeGy2nmP6ocMwZefQnhzD5rMX\nsNsU7CPO8cCIxYyNHXXb7/l5pmUEQRA+0+sflHGwuO6G5zS3dxNhMdzwnEDktndgLyrCXuQdnTvr\nanuOaSMiME+ZSmhODqGZ2bhMRraefo9dldtQhWsY45zBg3nLMRtNnysGkdwFQegXB4vrek3eERYD\n+ZmBf3uMx+mkq6Ic+6mT2E+dpOvMabjYK0NtNGIaO47Q7BxMWdno4odwqfZ4slHm3VeOcj6uiJjI\nKFZlLmN4RHqfxCSSuyAI/SbCYuDXX5vi6zD6nKIoOC6cx37qJB2nTtFZUozicHgPajSEZAwnNCub\n0OwcjCmpqLRXp1qbo4P1pW9ysPYQUSEpjO++gwcnzECn0fVZjCK5C4Ig3ARnc3PPyNxedBJ3W1vP\nMf2QBEKzcwjNyiZEykQTEnLd11AUhQNnC9n50RGqYktIsgxlVd4ihloS6OuVhCK5C4Ifupn5an8X\n6PPpnq5O7MXFPfPmjuqqnmOasDAsEyf3JHRdZGSvr9fY2cyr8kaK6ksZXjOde9KHMz9vGmpV/+zf\nKJK7IPihm5mv9neBNp+uuFx0nTl9carlJF2nK8DtBkCl1xM6cjSmbO9Uiz5x6E2PtD2Kh/dL9rOj\nYi82QwtZMSNY9HAuQ6Ni+3y0fiWR3AXBTwXrfLW/UBQFZ021d3li0Sk6i4vwdHV5D6pUGFNSe0bm\nxvQM1Lpbnw+/YKtmdfF6WivcDKkZzcSl8UxKzO3XpH6JSO6CIAwartZW7MWnsJ/0JnRXc1PPMV1s\nHJZJU7yF0MwsNKbbX4rodDt5q+QDdlTvxIOH3OyxTB49lszExAFJ7CCSuyAIQczjdNApF/fcQOQ4\nX9lzTG02Y86bgCk7h9DsbHTRMX3ynmUtp1lTvJ7QwjTih6SzaPpURkZn9clr3wqR3AXhNvRXwVOj\nUeF2KwE/3+5r3VVVtO7ZSduHH+KxdwCg0moJzfIm8tDsHAzDklCp+66Y2enqZFPZO+yv+ggVKu7I\ny2SU8W5GRg/rs/e4FSK5C8Jt6O+CZ6AVI/2Bx+nAVlhA6+5ddJaWAN5VLRFz7yJ05ChCho9Ardf3\ny3sfrT/J+mNvEX5qBENy43koZxmp1uR+ea+bJZK7INym/ih4ButmWf3JUV1Fy57dtH24D0+Hd5Qe\nmpWDdcZMzGPHXXMDUV9q7W5jXckbHK4/jlatYURcGLOHPUqitfelkf1NJHdBEAKOx+nEdqiQ1t07\n6SyRAdBYLETcPR/rtBno4+L69f0VReFA9UHePXgAd5eKtIwUHspcRrypf9/3VojkLvidQLiBR8yJ\n+4ajtobWPbto278ft837CSckM4vwGbMwjxvfr6P0S+rs9awt3khJSzkWQwQZlZP54tKp6HX+lU79\nKxpBIDBu4BFz4gNHcbmwHT5Ey+6ddBYXAaAxW4i4626s02aij48fkDjcHjc7zu1hb8FxWsJqGBWf\nzYoRi7HMsaDVaQYkhlshkrvgl8QNPEJndQ31m9+ibf8+3O3efVxCRkhYZ8zCPD73tm4qul3n2s6z\nung9521VJNvGMVabz9JRkwZszfrtEMldEAS/obhc2I4cpnX3LuxFJwFQm0yEz72L8Okz0A9JGNB4\nHG4Hb1Zs40P5MF2h7UwZks+9E+5G49b5dWIHkdwFQfADzvp6WvfupnXfnp7dFsOyswidPA1zXh5q\nXf8sYbyRoqYS1hZvpLXNxojiGUxeMpTx6QN/M9LtEsldEASfUFwubMeO0rp7J/ZTJ0FRUIeGEn7n\nXKzTZzJ0TKZPloXanB1sKNnKwerDqDQwJ2M6I9NGERcdPuCxfB4iuQuCMKCcjQ207tlN6769uFtb\nADCmZ3hXvOTl99uNRr1RFIXCuqOsL9mCtiqc4R2TWHp/PsMsiT6J5/MSyV0QhH6nuN10HDtKy+5d\n2E8e947SQ0IInz0H64xZGBKH+jS+pq5mXpM3caKxGJ1ax/z80eiK4xliHJiVOP1BJHdBEPqNs6mR\n1r17aNu3B1dzMwDGtHSsM2ZiyZuA2uDb5a4excOeCwfYUv4O0aUSmRk5PDhhATGhUZDm09A+N5Hc\nBUHoU4rHQ8fxY7Tu3knH8WM9o3TrrNmET5+JYViSr0MEoMpWw5riDZxuO0uoNoSxWRmYbVHexB4E\nRHIXBKFPOJuaaNu3h9a9e3r2STekpBI+YyaWCZN8Pkq/xOlxse3MB2wv30t4zTDGjx3D/dJCLDqz\nr0PrUyK5C4Jw2xSPh44Tx2nds4uOo0dAUVAZjFhnzMQ6YxbGJN/ujPhpFa1nWF20nhp7HeHGcNK6\nspkRkkWY3uLr0PqcSO6CINwyV0szrfv20rpnN66mRgAMSclYZ8wibOJE1MYQH0d4tU5XF1vK3+XA\n6UNonQamj5jCwvS7IVeNMWTg7nQdSCK5C33uVjf+utSg4hJ/31dmsFI8HuynTtCy++Io3eNBZTBg\nnT4D6/RZGFNSfB3idR1vOMWr8iZaulsZ5hpOfGUWS++ejE6rCeoMGMT/NcFXPu/GX2JTLv/iam3x\njtL37sbV0ACAYViSd8XLxMloQvxrlH5Jm6Od9SVbOHq+CEXvZn7KncxLmU3tuXa02r7rwOSvRHIX\n+sWtbPwlGlT4H8XjwV50itY9u7AdOQxuNyq9nrCp07yj9NRUv91bRVEUPqopZGPpm9hdnWSen8rY\nrAympWUDMDQlwscRDgyR3AVB6OFqa6Ntv3cu3VnvnVrTDx3mXfEycTKa0FAfR3hj9fZG1sobkJvK\nMGj13D9iEaPHjKHuwuAbPIjkLgiDnOLx0CkX07J7F7bDhZdH6VOmYp0xE2Naut+O0i9xe9zsPL+P\nrRXb8HRDTumdLH14HPHhMQBERgXXMsebIZJ7kPGHLkaiIBoYXO1ttO3f5x2l19UCoE9IxDpjJmGT\np6AJNfk4wptT2X6B1cXrqWy/gFln4v6xi3DqLbhbtRBYe331KZHcg4w/dDESBVH/pSgKnXKxdy79\nUCGKy4VKq8UyeQrh02dhzMjw+1H6JQ63k7dPb2dH5R5CWiIZr53GirtnY9aZwH9amfqMSO5BSHQx\nEj7NbbPR9uE+WvbswllTA4A+fsjFUfodaMyBNW0hN5WxRt5AQ2cjUcZIluTex6m3W9F7xCfGS0Ry\nF4QgpSgKnaUltO7eia2w4PIofeJkrDNmEjJ8RMCM0i+xO+1sLHuLA9UHCW9IZGbWDBZmz8Wg0TP6\ncQ+aQbDE8WaJ5H4L+nI++9M37vQVX0/JCL6lKAqO85W0Fx6kveBgzyhdFxdP+IyZhE2ZGnCjdPD+\nvw7XH+f1ks20O2wkmoeQ65yB9owBwyjv/u8isV9NJPdb4A/z2b0R892Dj6IodJ89S3vhQWyFBT3F\nUZVOh2XCRKzTZxIiZQbcKP2S5q4WXivZzPH6U5g6I1g08h7mJE1H8UBnh9PX4fmtGyZ3SZLUwPPA\naKAbeFyW5fIrjj8EfAtwA3+VZfnFfozVL/TVfLa4cUf4PBSPh67TFdgOFdBeWNBz56hKr8ecl48l\nNx/TqNGojUYfR3r7PIqHPecP8Eb523S5uxluziDsZDZjxo9Go9aAGixWja/D9Fu9jdwXA3pZlqdI\nkjQReO7ic5f8GsgGOoBTkiStlWW5tX9CFYTBTfF46CovuzhCL+zZVldtNGKZOAlzbj6mnJF+s7Xu\n51HTUccfjm2iuL6cULWJVZkLmDJkAlXxLYSafNOGL9D0ltzvAN4FkGX5Y0mS8j51/BjelaQeQAX0\n/STyAOptTt3fp2SE4KN4PHSWyLQXFmA7VNjTc1QdGkrY5Dsw5+YRmpODWhccCc/lcbH97C7ePbMD\nl+JmtGsCsY2pTJk5HpVKRWLy4Ng6oC/0ltzDgLYrHrslSVLLsuy5+PgkUIh35L5BluW2T79AIOlt\nTl3MZwsDQXG5sMvF2AoLsB0uxN3unb5Tm0yETZ2OJS+P0MxsVNrgKpmdbj3L6uL1VHfUYtWH8ZX8\nlSTrUvlwRzlOhxu9Ibj+v/2tt6vVBly5i31PYpckaTQwH0gG7MA/JUlaLsvy+hu9YEyM/26Kr9Go\niA438pf/nDcg7+fP12KgDfZr4XE6aTl6jNJXP6Lp409wtdsA0FnDiL5rHtFTJhE2Mgd1kCV0gC5n\nF2uPb+Hd0l0oKOQ33smCmZPIGpoIwJJV430cYWDq7SdlP3AfsE6SpEl4p2EuaQU6gW5Zlj2SJNVx\nEzf7+nMR8dLSxIGIURRULxus18LjdGA/eZL2woN0HDmMp7MTAI01nPDZczDn5nvXoqvVOIHG5k7f\nBtwPTjYWs7Z4I83dLcSGRrNKWo6+ycrxg1VkZScOyp+L67mdwU9vyX0TMFeSpP0XH39JkqSVgFmW\n5ZclSXoJ2CdJkgMoA/7vliMQhEHE091Nx4nj3imXo0dQursA0EZGEnbHNIbdOYOuyCGo1MG9Zrvd\nYWN96RYKao+gdeuY3DGP+6dNx6DTQwQkpUX6OsSAd8PkLsuyAjz5qadLrjj+EvBSP8QlCEHD09VF\nx7Gj3hH68WMoDgcAuugYzDNnYc7N79kfPSzGQncQj1YVReGTmkNsKHuTDqed5LBhrJKWUbCllsqy\nFjKyvDWtQF2T70+CbwJPEPyA226n49gR2gsLsJ84juL03myji4vDkpuPOTcPQ1LyoEpiDZ1NvCpv\npKiphFCXmfnRC7hn3FTUKjUxy2JEwbSPiaspCH3EbbNhO3oYW2EB9lMnUVwuAPQJCZhz87Hk5qFP\nHDqoEjp4b0baVbmPNyvew+FxkhU5gjvN8/jk3fM4s90YjGoMxuBsUu1LIrkLwufgbm/HdvgQ7YUH\nsRcXgdsNeLsXWXLzvCP0hEQfR+k7F2zVrC5az9n2SsI84azIuouJCd4169bFVjFa70fiygrCLXK1\ntmA75E3onSUyeLy3fRiSU3oSuj4u3sdR+pbT7eSdMzvYfm4XHsVDftx4Ykuy0Jy2okr0fnIZMtTq\n4yiDm0jugnATnM3N2A4VYCssoLO0BBTvslljWpp3ymV8HrqYGB9H6R9Km8tZI2+gzt5ApCGCBzOX\nkhMlYUvqprKiydfhDRoiuQvCZ3A2NmAr9G7M1VVe5n1SpSIkYzjm8bmYx+ehi4rybZB+xO7sZHP5\n2+yv+hgVKmbET8W5N5aUMSkAmC0GssYM8W2Qg4hI7oJwBUdd3cWEfpDuM6e9T6pUhEiZ3imX8blo\nw8X+Jp92pP4Er8ubaHW0k2CKZ1XmclKtSRxqPUtjXQcms9iTaaANquQuNgYTrsdRU+3dmKuwgO5z\nZ71PqtWEZudgzs3HPG482rAw3wbpp1q6W1lX8gZH6k+gVWm403QXCY5UUq1JAIyfnOzjCAevQZXc\nxcZgAlzsVlRVha3wIO2FBTgunPce0GgIHTkaS14e5rHjA7Jj0UDxKB4OVB1kU/lbdLq6SLemsipz\nGVZVOJv+eZgxecMwhojljb40qJI7iObRg9WV7edsBQU4aqoBUGm1mMaOwzI+D9OYsWhMJh9H6v9q\n7fWsLd5AaUsFRo2Ru3WLmJ4+DqspFIAHH89HHeTbJwSCQZfchcHD237uDO0FF9vP1Xun5FQ6nbcg\nmpuPafQYNCEhPo40MLg9bt4/t5u3z7yPy+NidHQOK6TFnD7cyofvl3PP8lEAIrH7CZHchaDS037u\nYlHU1dgIgMpgwJw3AUteHqaRgd1+zhfOtlWyung9F2zVhOks3BN1L9NGjkOlUjFmgoXh2WI609+I\n5C4EPMXjobOs1LvT4qECXM3NwKX2c5O9zS1yRqHWB0e3ooHU7XawteI9dlbuQ0FhypAJLEi6iy1/\nP0GquZlhqZGo1WrMYeKPpb8RyV0ISIrbTWdpiXcO/VAh7lZv6151aChhU+7AnJtPaHYOap0o6t2u\nosYS1sobaOxqJsYYxf1pS8iJHwHAvMU5opepnxPJXQgYl9vPHcR26BBu28X2c2YzYdOmY8nNJzQz\nK+jazw00m7ODjaVb+bimELVKzbzkWeS4xnN023myHlZQq1XEJYilof5O/BYIfs3jdGIvOnWxn+gh\nPPYOADRhYVhnzMKSl0/ICAmVRuPjSAOfoigU1B5hfekWbM4OksyJrMq6n2GWBBRFoaG6A0e3Syxx\nDBBBldzFTUrBweNwYD95wtvc4uiRy+3nwsMJn3Qn5rx8QjKGB323ooHU1NXMq/ImTjYWo1PrWJJx\nL/rieJxVOpC8zTMmzUzzdZjCLQiq5C5uUgpc3vZzxy62nzt6Rfu5KMKmTseSm4cxLV0k9D7mUTzs\nOX+ANyreweF2kBkxnJWZS4kOiaJG3crJw1WkSWJDtEAUVMkdxE1KgcTT1Ynt2FEaTxylqaDwcvu5\nmBjMubOx5OZhSEkddM0tBkqVrYbVxes503aOUG0I9w+/H3dxOBF679458UOtxItteQNW0CV3wb+5\n7XY6jh7xNrc4cbynW5EuLt67MVdePoZhSSKh9yOnx8V7Z3aw7ewu3Iqb3NgxLB+xEIvOzDsfn6Dk\nRK3YvTEIiOQu9Du3zYbtyGFshwroOHnicreihETMuXkkzZ1JR0i4SOgDoKzlNGuKN1BrryPCEM6S\noQuJUxII01sAmLswC61OFKeDQUAld1EwDRyu9jZshw95+4le0X7OMGyYd6fF8XkYEhIAMMVYsNe3\n+zLcoNfp6uKN8nfYe+GAd6/1oVNYmHY3tkYnb752jBWP5RNq0qPTB1RKEG4goL6TomDq365qPycX\n93QrMqSkXtwLPQ99XJyPoxx8jtWf5LWSzbR0txJvimPZ0MUMj01Bp9VgjDOy4IHRhISK5Y3BJqCS\nO4iCqb9xNjVhO1SIrfAgnWWlV7SfS8ecm4clNw9dtFht4Qut3e2sK32Dw3XH0Kg03Js6l7nJs9j/\nXjlNugqm3jkcgJh4i48jFfpDwCV3wfecDfXe5haHCq9tP5ebj3l8LrrISN8GOYgpisKB6gI2lm2l\n09VJmjWZB4cvIzHM27R70sw0zpQ2+DhKob+J5C7cFEdtLbZD3n6iV7Wfy8zyTrmMy0UbHu7bIAXq\n7A2slTdS0lyGQaNnxYjFTIrNZ/3fDnHvA1asESEYQ3RkjharYYKdSO7CZ3JUV11uP1d5zvukWk1o\nzkjMud5uRaL9nH9we9x8ULmXt05vw+lxMTIqiwelJUQYvX9wR+cPpbmhA2uE2Lt+sBDJXejhbT93\noae5haPqgveARoNp1GjvlMvYcaL9nJ85136eNUXrqbRVYdGZ+ULWIhIcyZza38Adc7zJfeT4RB9H\nKQw0kdwHOUVR6K4819PcwllTA1zRfi73Yvu5UNF+zt843A7eOr2dDyr34lE8TBqSx9KMBZh0oTi6\nXezZVkpHezcmsTx4UBLJfRBSFIXuM6cvTrkcxFlfD4BKr/dOt+TmYR49BrVRfIT3V8VNpawt3kBD\nVxPRxkhWZi5DXR1Gd4uCKQb0Bi33fykPtVrcGDZYieQ+SPS0nys4SHthAa6my+3nLBMmYs692H7O\nIEZ5/qzDaWdj2VY+qi5AhYo7k2Zwb+pc9Bo9p85VsXd7KYtWjQUQiX2QE8k9iPW0nys4iO1w4eX2\ncyEhWCZN9ja3yBkp2s8FAEVROFR3jHUlb9DutDHUnMAqaRmaZjN6jff7lzVmCMnpUT6OVPAXIrkH\nGcXtprNEvrgOvQB3WxsA6lATYXdMw5ybS2iWaD8XSJq7WnitZBPHG4rQqbUsTp/P7GHTwKPi9dcL\nyJ+aQkZWLCqVSsyvCz1Ecg8CisuFvbgI26GCq9rPacwWrNNnePuJSpmi/VyA8Sge9l34iDfK36HL\n3c2I8HQelJYSprKiUWtADfMWZaM3iO+rcC3xUxGgvO3nTmIrKMB25PDV7edmevdCF+3nAld1Ry1r\nitdT0XqWEG0ID2Xez+QheVSda+GN7UdY/mguWq2GqFixLFW4PpHcA4i3/dxx2gsLrmo/p42IIGzy\nFMy5eaL9XIBzuV28fXo77535AJfiZlzsaJZnLMRqsKBSqUhICmd4dhxOhxutVvzhFj7bDZO7JElq\n4HlgNNANPC7LcvkVx/OB5wAVUAM8LMtyd/+FO/h4urvpOH4MW+FBbMeOonR7L682KgrrtBmYc/Mw\npqaJhB4EKlrP8nrBRirbqrHqw1ghLWFMTA4HdpYTHtlB1pghqFQqcqck+zpUIQD0NnJfDOhlWZ4i\nSdJEvIl8MYAkSSrgT8AyWZYrJEl6DEgGSvoz4MHgUvs5W2EBHcePXdF+Lta702JePobkFNHcIkh0\nubrYUvEee85/iILCtMTJLEq/mxCt9z6DETlxHPronOiOJNyS3pL7HcC7ALIsfyxJUt4Vx0YAjcC3\nJEkaCbwly/JtJ/beGnFA8Dfj8Did1L7yN2wHP7ncfi4+HktuPubcPNF+LgidaCjiVXkTzd0txIXG\n8vVJXyDcHcO+7aVMmZ2OTq8lKtbM3IXZvg5VCDC9JfcwoO2Kx25JktSyLHuAaGAK8HWgHNgqSVKB\nLMs7byeQ3hpxQHA341AUhbo1/6D9wIfo4uMJmzAJc24++oQEkdCDULvDxvrSLRTUHkGtUnNPyhzu\nSp5NQkwk9fXtuF0eio7VMDpvqK9DFQJUb8m9DbhyJ/9LiR28o/YyWZZlAEmS3gXygNtK7jC4G3G0\n7tlF2949GIYlMez7T4s7RYOUoih8XFPIxtKtdLjspIQlsSpzGeFKJOcrWkmI9+6DP/2uEWi0oo4i\n3L7ekvt+4D5gnSRJk4BjVxyrAMySJKVfLLJOA/7c2xvGxFy/64tGo7rh8WB06f/aVlRM/drVaC0W\nRv7wKYxx0T6ObOANhu97na2BPxWs4VhtEQatgUfH3c/dGTNRq9U0NXSw7m8FjMiMGxTX4maJa3H7\nekvum4C5kiTtv/j4S5IkrQTMsiy/fLGIuuZicXW/LMvv9PaG9Z/RCNntVm54PNjExFior2/H1dLM\n2Wd/heJ2E//VJ2lXh9A+SK7BJZeuRbBye9zsOr+frRXv4fA4yY6SeHDEUoyuUKoutGAweu8Wvm/l\nGKwRIUF9LW5FsP9c3Irb+SN3w+Quy7ICPPmpp0uuOL4TmHizb/bYz7b1JPFPC/Zi6fV4nE6qnv8j\n7tZWYh54kNAsUTQLNpXtVawpXse59guYdSZWZS4nL24sKpWKDz8oo9PuZM6CLACiYsyiviL0mQG9\niamhtYsI8/UTeDAXSz9L/dp/0lVRjmXiZMLn3uXrcIQ+5HA7eefM+7x/bjcexcOE+PEsy7gPo8rY\nk8Dzp6ZSIdejKIpI6kKfG9DkHm018osnJg/kW/qtmne30bpnN4ZhScR98VHxyx1ESprLWVu8gbrO\nBiKNEayUlpIdJeF2e3jtzwe5a0kOUbFmdHoN0qh4X4crBCmx/YAPdJaWcv7lv6A2m0n4+jfFypgg\nYXfa2VT2Nh9Wf4IKFbOHTePe1HkYtd7vr0ajJm9qCq3NnWJPGKHfieQ+wFwtzVS9+EcUj4fEJ76G\nLjrG1yEJfeBw3XFeL9lMm6OdRPMQHspcTnLYMOpr2jlQeJqZ8yVUKhUjcuJ8HaowSIjkPoCuLKCm\nfPkR9KKAGvBault5Xd7M0YaTaNVa7ku7m7lJM7xb8gKR0SaaG+20tXRhjRBtC4WBI5L7ALpcQJ1E\nwsL7aGiw+Tok4TZ5FA/7qz5hc9nbdLm7yAhPZZW0jDhTLGVFdZjDDMQnWtFo1Sz5wjhRUxEGnEju\nA6Rl964rCqhfEr/sAay2o47VxRsobz1NiNbIKmkZkxPyUau8d5SqVLBvexnLHhmPSqUS32vBJ0Ry\nHwCdZaXUrfmHKKAGOJfHxfvndvPOmR24PC7GxIzkgRGLsOrDOFfRRFJaJCqVivTMWBKSwkVSF3xK\nJPd+5mpppuqFP4LHQ4IooAasM23nWF20nqqOGqx6Cw+MWMzY2FEAeDweCvadoaO9m+yxCQCEhIqm\n44JvieTejzxOJ1Uv/K+4AzWAdbm62Xr6PXZV7kdB4Y6EiSxOn0+I1kiHrRuT2YBarWbOfVloNGKj\nL8F/iOTej+rXrqarvAzLxEniDtQAdLJR5lV5I01dzcSGRLMqcxnDI9IBqKtuY9umkzzwWD56g5bw\nyFAfRysIVxPJvZ94C6i7RAE1ANkcHawvfZODtYdQq9TclTybu1PmoFNre7YKiB0SxugJw3A53egN\n4tdI8D/ip7IfdJaXeQuoJpMooAYQRVE4WHuYDaVvYnN2kGQZykOZyxlq8c6jf7LnNHqDhrETkwBE\nIw3Br4kOWxMmAAAgAElEQVTk3sdcLc1UPe8toA4RBdSA0djZzKvyRk41yejVOpZlLGDmsKk9yxsB\nMkfH8/HuCrHRlxAQRHLvQ5cLqC1E378CU3aOr0MSeuFRPOw+/yFbKt7F4XaQFTmCB6WlRIdE4vEo\nfPB2MVNmp2MM0REWHsLcReJ7KgQGkdz70JUF1Ih5d/s6HKEXF2zVrC5ez9m2SkzaUB7MWsKE+PE9\no3K1WoXeoOHUkSrGT072cbSCcGtEcu8jlwuow0QB1c853U7ePbODbed24VE85MWNZfnwhVj0Zuy2\nbi6ca2F4tneDr8mz0lGrxfdSCDwiufeBqwqoX/tXUUD1Y2Utp1lTvJ5aez0RhnAelJYwMjqr57ii\nwL73y4iOsxARFSrWrgsBSyT3z+maAmqMKKD6o05XJ5vL3mZf1ceoUDFz6B3cl3YXRq2RjvZuVCoI\nNRswWQwseXic2MFRCHgiuX8OisslCqgB4Gj9CV6TN9PqaGOIKY6HMpeTar08h150rJq66nbuWTYS\nlUolbkgSgoJI7p9D3dp/eguoE0QB1R+1drexruQNDtcfR6vSsCB1HnOTZ6JVa3E6XOj03h//cZOS\nKC+u93G0gtC3RHK/TS17dtG6+2IB9RFRQPUniqLwYfUnbCp7i05XF2nWFB7KXEa8Ka7n+IZXDjHj\nbokhQ61oNGrRIUkIOiK534bO8jLqVosCqj+qs9ezpngDpS0VGDUGVoxYwtTEiVfdjKRSqZg4I42O\n9m4fRioI/Usk91vkamkRBVQ/5Pa42XFuD2+f2Y7T42JUdBYrRiwhwhgOQGO9jYN7z3DXkhxUKhWp\nw6N9HLEg9C+R3G+Bt4D6R1FA9TNn2ypZXbyeC7ZqLHozXxyxmHExo66aKouIMuF0uGlutBMZbfJh\ntIIwMERyvwV1l+5AFQVUv9DtdvBWxTY+qNyLgsLkIfksybgXk8672qW8uB69QcOw1EjUahULVowW\ntRFh0BDJ/Sa17tlN6+6dooDqJ4qaSlhbvJHGriaiQ6JYKS0lM3L4VefoDRr27yhjxWP5opepMOiI\n5H4TxB2o/sPm7GBj6VY+rilErVIzN2km81PvRK/RoygKZ0obSM6IQq1WMyw1kiUPjxNJXRiURHLv\nxaUCquJ2e3ugigKqTyiKQmHdUdaVvIHN2cEwcwKrspaTZLl6T/XjhRdoae5k3MU91w1GnS/CFQSf\nE8n9BhSXi6oXxR2ovtbU1cxr8iZONBajU2tZnD6f2cOmoVFrUBQFW1s3FqsRlUrF7HszURRfRywI\nvieS+w3UrV1NV1kplgkTRQHVBzyKhz0XDrCl/B263Q6kiAxWSsuICY3qOae5wc6brx7lgcfyCAnV\nYw4z+jBiQfAfIrl/hksFVP3QYcQ98mUxbzvAqmw1rCnewOm2s4RqQ3g4834mDclDpVKhKAqKoqBW\nq4mMMTFheiputxiuC8KVRHK/jisLqIlfFwXUgeT0uNh25gPeO7sTt+JmfOxolg9fhNVg6Tnn8Efn\ncDrdTJyeBkDWmCG+ClcQ/JZI7p/iam2h6oWLBdSvPikKqAOoovUMq4vWU2OvI9xg5UFpCaOis685\nL3NUPPt3lItepoJwAyK5X6FnC9+WFqKXP4ApZ6SvQxoU7M5OXpM3s/fCARQUpidOYWH63YRovfPn\niqLwwVvFTJyeijnMSKjZwNxF1yZ9QRAuE8n9CnWvrrlcQL3rHl+HMygcbzjFugNv0NjZTHxoLKsy\nl5MennLVOSqVCmt4CKeOVDNheqpvAhWEACOS+0Wte3fTuusDUUAdIG2OdtaXbKGw7igatYb5KXcy\nL2U2OrX3R9Le4eBMWQPZYxIAGD8lSXxPBOEW3DC5S5KkBp4HRgPdwOOyLJdf57w/AY2yLD/VL1H2\ns54tfENNJHz9m6KA2o8UReGjmkI2lr6J3dVJalgS35jyCEaH5arz1GoVBXvPEBNnISbeglotepkK\nwq3obeS+GNDLsjxFkqSJwHMXn+shSdITwEhgV79E2M+uKqA+8ST6mFhfhxS06u2NrJU3IDeXYdDo\nuX/EIqYnTibOaqW+vp2O9m48HgWL1YgxRMeSL4zHHCb+0ArC7egtud8BvAsgy/LHkiTlXXlQkqQp\nwATgJSCzXyLsR6KAOjDcHjc7z+9ja8U2nB4nOVGZPCgtIdIYcdV55XI9FcX1LHpoLCqVCotV3JAk\nCLert+QeBrRd8dgtSZJalmWPJElDgB8BS4AV/RVgf+opoOZPEAXUflLZfoHVxeupbL+AWWfi4az7\nyY0d0zN/3t3l7Dl35PhETGa9r0IVhKDSW3JvA66cDFXLsuy5+O/lQDTwNhAPhEqSVCTL8it9H2bf\nu6qA+uhjoljXxxxuJ2+f3s6Oyj14FA8T43NZOnwBZt3lRhmKorBl7VHuXJBNREwoarWK9EwxLSYI\nfaG35L4fuA9YJ0nSJODYpQOyLP8B+AOAJEmPAJk3k9hjYiy9ndLv2uUSSlf/A63ZzKgffh9jvG9a\nrvnDtegPJ2qLealwDbW2emJNUXw17yFGx2f1HL/y5qP5S0fR3toVtNfidohrcZm4Frevt+S+CZgr\nSdL+i4+/JEnSSsAsy/LLnzr3pjb3qK9vv8UQ+5artYWz//1LFLebuK/8C+0aE+0+iCkmxuLza9HX\n7E47G8ve4kD1QVSomDNsOvemzcOg0ff8X1ub7ex5r5T5949Co1FjshpIyYgOumtxu4Lx5+J2iWtx\n2e38kbthcpdlWQGe/NTTJdc57++3/M4+cFUBdZkooPYVRVE4XH+c10s20+6wkWgewkOZy0kOG3bN\nuWHhIegNWprqO4iJF6MyQegvg+omprrXriig3i0KqH2huauF10o2c7zhFFq1lkVp9zAnaToatabn\nnNMlDSiKQpoUg0ql4q4lYl98Qehvgya5t+7bQ+tOUUDtKx7Fw74LH/NG+dt0ubsZHp7GqsxlxIZe\nu9FaqFnPjjeLSBkeJW5GEoQBMiiSe2dFOXX/fEXcgdpHajrqWF28norWM4RojazKXMbkIfmoVd7E\nrSgKFXI9yRlRaLUa4hLCWP5orkjsgjCAgj65u1pbqBZ3oPYJl8fF9rO7ePfMDlyKm7Exo3hgxCKs\nhrBrzi0rqqOh1sbEGd491/WGoP9REwS/EtS/cYrLRfWLz+NqbhYF1M/pdOtZVhevp7qjFqvewgPS\nEsbGXL6eiqLQ1tKJNSIUlUrFtHkjcLs8N3hFQRD6U1An97rX1tBZWoI5TxRQb1eXq5s3K95l9/kP\nUVCYmjCRRenzCdWFXHVee2sXG185zPJHc7FYjYSaxJ2mguBLQZvcewqoiUOJ/5IooN6Ok43FrC3e\nSHN3C7Gh0aySljM8Iq3nuKIoeDwKGo2asPAQps7NQFxmQfAPQZncOysqLhdQvyF6oN6qdoeN9aVb\nKKg9glql5u7k2dydMgedRnfVeccLLtDSZGf6XSMAGJ4d54twBUG4jqBL7t4C6h9EAfU2KIrCJzWH\n2FD2Jh1OO8mWYTyUtZxE8/UbUEuj4ti3vQy324NGI1bCCII/CarkLgqot6+hs4lX5Y0UNZWgV+tY\nNvw+Zg69o2d54yU7thaROyWZ8MhQDEYdc+7L+oxXFATBl4Iqude/vlYUUG+RR/Gwq3Ifb1a8h8Pj\nJCtyBCulpUSFRF73/Nh4C6eOVDNldvoARyoIwq0ImuTubGyg5YMd6BMSRAH1Jl2wVbO6aD1n2ysx\n6UJZmbmM/LhxV127TruD0pN1jM4fCsDI3ERfhSsIwi0ImuTecdy7G3H4zNmigNoLp9vJO2d2sP3c\nLjyKh/y48SwbvgCL3nzNuVqthuOF54mOM5OQFC7+aApCgAi65B46arSPI/Fvpc3lrJE3UGdvIMIQ\nzsrMZeRESVed09HejcPhIiLKhE6vYfFD4wgVHZIEIaAERXL3OJ3Yi06hi48Xq2M+g93Zyebyt9lf\n9TEqVMwaOpUFaXdh1F77Kef8mWaOHqxk2SO53j3XLeKTkCAEmqBI7p0lMorDgWnUGF+H4peO1J/g\ndXkTrY52EkzxrMpcTqo16apzOu0OjCE6VCoVI0bGYQjRolaLKRhBCFRBkdw7ThwHwDRylI8j8S8t\n3a2sK3mDI/Un0Ko03Jd2F3cmzUCrvvbb/t6mk2SPTWBEThwqlYqUDN+0HhQEoW8ER3I/fhSVXk/I\nCKn3kwcBj+LhQNVBNpW/Raeri3RrKqsylxFvunrK6spepnfMyaCpvsMX4QqC0A8CPrk76utw1tRg\nGjMWtU7X+xcEuVp7PWuLN1DaUoFRY+RBaSl3JEy45mYkW1sX720+yaKVY9HqNMTEW0TbO0EIIgGf\n3O0XV8mYBvkqGbfHzfZzu3nnzPu4PC5GR+ewQlpMuMF63fPNYUaiYsw01NqIH3r9cwRBCFwBn9x7\n5tsHcXI/21bJ6uL1XLBVE6a38MCIxYyNGXnNmvSz5Y10djjIHO3dK2bmPWIaSxCCVUAnd4/Tgb24\nCP2QBHRRg68A2O12sLXiPXZW7kNBYcqQCSzJmE+oLvS654dZjezdVkpGdixarea65wiCEBwCOrl3\nypeWQA6+UfupRplX5Y00djUTExLFqsxljIjIuOa80lO1JKdHoTdoiYg2seLxfJHYBWEQCOjk3jEI\n59ttzg42lL7JJzWHUKvUzEuexT0pd6LXXL+YfOFsCzXnW5k2z7vnuk4nErsgDAaBndxPHENlMGLM\nGO7rUPqdoigU1B5hfekWbM4OkiyJrMq8n2GWhGvOa26wExljAmDK7HS6u1y+CFkQBB8K2OTuqK3F\nWVuLaey4oF8C2djZzKslGznVKKNT61iScS+zhk5Fo752FG7vcPDGmsMsfmgcEdEm9AYtekPAfpsF\nQbhNAftb33Hi0pRM8G454FE87D7/IVsq3sXhdpAZMZyVmUuJDom66jxFUXC7PWi1GkxmA7PmZ6IV\n0y+CMKgFbnI/fmkJZHBuOVBlq2F18XrOtJ0jVBvCiqwHmBife90td4uOVnPhbAtzF2UDkDJ88K0c\nEgThagGZ3D0OB51yEfrEoegio3r/ggDi9Lh478wOtp3dhVtxkxs7huUjFhKm/+y7R4fnxFFb1YbL\n6RYjdkEQgABN7p1yMYrTGXQbhZW1nGZN8QZq7XWEG6w8KC1hVHT2dc/d+VYxI3MTiYm3oNNpmDU/\nc4CjFQTBnwVkcu84fhQIniWQna4u3ih/h70XDqBCxYyhU1iYdjdGrfEzvyYxOZziY9ViPxhBEK4r\n4JK7oih0HD+G2mgkJAiWQB6rP8lrJZtp6W4l3hTHQ5nLSbMmX3NeV6eTk4cuMH5KMiqViuE5cQzP\nifNBxIIgBIKAS+7O2lqc9fWYx+ei0gZc+D1aOlv584k1HK47hkalYX7qXOYlz0J3nb3WAXR6DeVy\nPdHxFpLTo0QvU0EQbijgsmPPEsiRgTkloygKB6oL2Fy+lQ5nJ6lhyTyUtZwhpmtH4bb2brrsDqLj\nLGg0au57cAzGkOBe0y8IQt8IvOQewI2w6+wNrJU3UtJchlFrYMWIxUxNnHTNXuuX1Ne08+GOMh54\nLB+dTkNIqGhSLQjCzQmo5O7p7qZTLkY/dBi6iAhfh3PT3B43H1Tu5a3T23B6XIyMyuJrUx5G6bh2\nFG7vcBAS6u1lmjo8Gp1OjVZ7/eQvCILwWQIquduLi1BcroBaJXOu/TxritZTaavCojPzhayFjI8d\nQ3RoGPUd7decv/OtYpLSIxmVOxSAoSmRAx2yIAhB4IbJXZIkNfA8MBroBh6XZbn8iuMrgf8PcAHH\nga/Jsqz0V7CXtxzw/+TucDt46/R2Pqjci0fxMGlIHkszFmC6zl7rV/YynTIng9oLrQMdriAIQaa3\nz/uLAb0sy1OA7wPPXTogSVII8F/ATFmWpwJWYEF/BdqzBDIkhJC09P56mz5R3FTKzz/+De+f202k\nIZxvjv0KX8h64LqJvdPu4PW/FtDd5QQgIiq0p1OSIAjC7eptWuYO4F0AWZY/liQp74pjXcBkWZa7\nrnitzr4P0ctZU42roQFzbp7fLoHscNrZWLaVj6oLUKHizqQZ3Js6F73mswuhIaF6hqVG0lBrIzE5\ncOoIgiD4t96yZBjQdsVjtyRJalmWPRenX+oBJEn6JmCSZfn9forzisYc/rcLpKIoHKo7xrqSN2h3\n2hhqTuChzOUkhQ297vmVp5uoKK4nLTMG8O65Lgh96dChAt54YyM/+cl/3/Zr/OUvL/H+++8RHe39\nOW1ra2XOnHl88Ytf7qswWb3677z++lrWrduCXu8dBP385z/mzjvvYsGCeT3nLVx4F1u2vAfAnj27\nWL/+VRRFobu7m1WrvsDMmXM+8z327dvD3//+ZzQaLffeu5D77lt81fGamhp+9rMfAWCxhPHjH/8M\ng8HIu+++xdq1/8RsNnPPPQtYsGBRn/2/B0Jvyb0NuPL+drUsy55LDy7Oyf8KyACW3cwbxsTc3u3y\ntfIpAJJmTEYf6T+33DfYm/hr4asUVh1Hp9Hx0Ogl3CvNQXudvdYv0Ws1/P35A4ybkCT2Wr/odn8u\nglFfXIuICBNGo+5zvZbZbOQrX3mcFStWAOBwOLj33nv50pe+QGRk3xT6d+x4j4UL7+OTT/awZMkS\nAEJC9ISHe6cwL8Wv0aiJibFw6NAhNm9+nb/+9c+EhITQ0tLCihUrGD9+FOnp1w6SnE4nL7zwOzZs\n2IDRaGTlypUsWjSfqKjLGw6+/PIfWLx4IStXruS3v/0tu3a9x/z58/nb3/7E5s2bsVgsPProo8yb\nN4vExMQ++X8PhN4yy37gPmCdJEmTgGOfOv4S3umZJTdbSK2vv3aFSG88XV20nTyFISmZVrcWbuM1\n+ppH8bDvwke8Uf4OXe5uRoSnszJzGbGh0TQ32q85v+RkLUNTIgg1eUcnX/veTJqbrz1vMIqJsdzW\nz0WgeP2DMg4W193UuRqNCre791+l/MxYHph9bc/cS5qbO+jqcl5zXQ8e/IiXX34RvV6P1Wrlqaee\nwWQy8dxzv0SWi4iKiqK6uopf/vK3dHR0YzB09rxGQ0M93d0ObDYXra3V/OIXP6WtzfvB/t/+7Tuk\npWWwdetmNm5ch8ViRafTMmfOPO655/qluEOHChgyJJG5cxfwX//1Q6ZOvROAri4nLS3e341L7+3x\neKivb+cf/1jDkiUPYLO5sNnaAQ0vvvh/mM1mDh48xoYNr/Ptb3+v5z3KykqJj0+kqwu6urrIzh7F\njh17mDXrzp5zhg5Npaqqjvr6durrm7BYIjlxooS0tAy6u1V0d9tIT5fYu/cj5sy5/GliIN3OH+ne\nkvsmYK4kSfsvPv7SxRUyZqAA+DKwB/hAkiSA38myvPmWo+hFzxJIP9kFsrqjljXF66loPUuINoSH\nMu9n8pC8G24J0Fhno7KiiTn3ZQGIJtXCgFMUhV/96lleeOEvREdHs27dq/z9739hzJixtLe38vLL\nf6elpYUHH1zSc/5rr63m/fffo66ulpiYWL7//R8SEhLC88//nry8CSxevJzKynM8++xPefbZ/5/V\nq1/h//5vLTqdjn/913+5YTxbt77BggWLSEpKRqfTc+rUCbKzRwJc53fJ+7ihoYGEhKunO81mMwAp\nKalXJXaAjo4OzGZTz+PQUBMdHbarzomJieXFF//I9u3v4nS6eOyxJwAVp09X0NzcREhIKIWFB0lK\nunbPJ392w+R+cTT+5KeeLrni3wOSoS7vAunb+Xanx8W2szvZduYDXIqbcbGjuX/4IqyGa/+qKopC\nQ62tZ9fG/GkpdHY4BzpkwQ88MDvjhqPsK/Xnp5iWlhZMJhPR0d5mLmPHjuOll/4Xq9VKTo53eXF4\neDjJySmAN8E++ODDLFq0FFku5plnfsCwYUkAVFSUcfhwATt2bAegvb2N8+fPk5KShsFgAGDkDbYI\naWtr46OPPqSlpZn161+no8PGhg2vk509Er3egMPRfdX5brcbgPj4eGpra0hPv3w9jx07QlRUNImJ\n19a4zGYzdvvlT8h2ewcWS9hV5zz//O/4z//8Mfn5kzhwYB8/+9kz/OpX/8M3v/ktnn76u1itVkaM\nkLBaw3u/yH7E72999C6BPI46NBRjWprP4qhoPcsvDv6Ot09vx6Qz8dVRj/D4yIevm9gBurtcvLXu\nGHXV3o+tWq0Gi/Wzt/AVhP4WHh5OR0cHjY0NABw+XEhSUjJpaRmcPOmdcW1ra6Oy8lzP1yiKd4pI\nkjJ5+OFHeOaZH6AoCsnJqTzwwCr+8IeX+OlPn2XevHsYOnQo586dobu7G4/HQ1HRyc+MZdu2t1mw\nYBG/+c0fee653/OnP/0fBw9+TEtLC5KUye7dO3vOPXr0MKmp3t/9+fMXsmbNK3R1eRfpNTc38eyz\nP6W7u+u675OcnEJlZSVtbW04nU6OHDnc84fsEovFSmiod/QfFRVNe3s7brebkpJinn/+z/zkJ89y\n7txZRo/2v8UcN+L31TxHVRWupkYs+RNQaQZ+KqPL1cWWivfYc/5DFBSmJU5mUfrdhGhDrjlXURRc\nTjc6vRZjiI4778sW+8EIPqFSqTh48GMef/yLPc8988zP+N73nubpp7+LSqUiLCyMp5/+MWFhVj76\naD9PPvllIiOjMBqNaC8uN75yemTBgkV88MF2Nm/ewCOPfJlnn/0vtmzZREdHB4899gRWazgPPfQI\nX//6VwgLC6O7uxutVktTUyO/+91zV63c2bp1Cz/60U97HhsMRmbMmM2bb25m5cqHKS2VWbx4MXq9\nEZ1Ox3e/+zQAI0eOYuHCpfz7v38NjUZLd3c3//Iv3yQtLYPTpyvYuHHdVVMzWq2Wb37z3/n2t7+B\nx6OwYMEioqOjaWtr5Ze//Bk///mv+fd//w9++9tf9Xw6+Na3vofmYq758pcfQq83sHLlw4SFWfvh\nO9V/VJf+Mg+Ex362TfnFE5Nv6Wua3nuHhnWvEfelx7HeMbWfIru+Ew1FvCpvorm7hbjQGFZlLicj\nPPUzzy85WUvJiRrufWB0r1vyBnsR8VaIa3GZL67FuXNnKC0tYc6cebS2tvDFL65gw4a3ehL8zXK7\n3axe/Xe++MUvoygK3/jGV/nqV7/OyJGjeOGFP/CNb/zbLb2e+Lm4LCbGcst7fPv9yL1nffsAFlPb\nHTbWlbxBYd1R1Co196TM4a7k2eg0N95uNz0zhtoLbTgdbrHEUQgYsbHxvPDCH3j99bV4PG6efPJf\nbzmxA2g0Gjo7O/nylx9Gp9ORkzOSMWPG4nK5WLXqC/0QuXAjfp2B3J2ddJaWYEhOQWvt/49EiqLw\ncU0hG0u30uGykxKWxKrMZSSaP3s7gF3vyAzPjiUxOQKNRs20eYHfHUoYXIxGI88++1zvJ96EJ574\nOk888fWrntNqtUQGWSP7QODXyd1edArc7gHZKKyhs5G1xRspbi5Fr9GzfPhCZgyd8pl7rV+SOjya\nkpO1YusAQRD8in8n9wHYBdLtcbPz/D62VmzD6XGSHSXx4IilRIVcP1k7ul0cOnCOCdNTUatVJGdE\nkZQutuUVBMG/+G1y79kF0mTCmNo/SyAr26tYU7yOc+0XMOtMPJS5nLy4sTcshur0GmrOt1Ih15OR\nFQtc74YLQRAE3/Lb5O64cB5XczOWCZNQqft2Ob7D7eSdM+/z/rndeBQPE+LHsyzjPsx603XPt7V3\nY2vrIj7Rikql4p7lI0XBVBAEv+a3GeryLpB9OyVT0lzOmuL11Hc2EmmMYKW0lOwo6YZf09bcyftb\nTrHi8QkYjFoMRtGkWhAE/+bfyV2lInTkyD55PbvTzqayt/mw+hNUqJg9bBr3ps7DqDVc//3buzGG\n6tBo1CQkhTN3cQ56g9gPRggcFy6c54UXfk99fT1GoxGDwcCTT/5rz92efa2pqZG//e3P1+zv0pu/\n/OUlPvroQ1588a89Nw999auP8sc//p4TJ0r40Y+eIjU1DUVRcDqdfOc732f48BsPyB59dBWjR4/h\nW9+6HMuV2wYDfPTRh3zwwXZ+8INn6O7u5uWXX6Co6CQqlYqQkBD+4z9+QGxs3HVf3+Px8Nxzv6C8\nvAydTsf3v//Da7Y/eO211Wzd+gbh4d763Xe/+zRDhiTw7LM/oaamBofDwSOPPMbUqdNv6XrdLL9M\n7m67nc6yUowpqWg/tQ/ErVIUhSP1J3i9ZDNtjnYSTPE8lLWclLCkG37d/h1lRMaYyLsjBYAhQwPr\n7jTBf2ws28rhuuM3da5GrcLt6f3GwnGxo1ia8dmNz7q6unjqqW/zve/9kJwc7wCpqOgkv/nNL/nD\nH166ucBvUWRk1C0n9ktqaqr5xz/+xqOPPg5461iXall5eRP48Y9/Dlze1fJXv/rtZ77WsWNHSE/P\n4NChAux2O6GhoRdf8+rzrqyV/f73z5GSktZzo9WePbv40Y+e4sUX/3rd99i7dxdOp5MXX/wrJ0+e\n4I9//O01y0lLSor54Q9/yogRmT3Pvf32m4SHR/DDH/4XbW1tfOlLqwZXcrcXnQSPh9DPeeNSS3cr\nr8ubOdpwEq1ay31pdzM3aQaaz9hr3ePxoL44vz9ldjqVp5s/1/sLgq/s37+H3NwJPYkdICsrpyex\nV1SU8cc//g9ut4fW1ha+853vM3Lk6KtGt8888xSLFy8nKiqaZ5/9CRqNFkVReOaZn6HT6XnmmadQ\nFAWHw8F3vvMUZrOZH//4aV566W/s3Pk+mzatx+VyoVKp+O///jXl5WWsXv0Ker2OqqoLPY0/VCoV\nq1Z9ka1bN3PHHdOuGZVfeRd9W1tbr3vJb936BrNm3UlsbBzvvLOVZcseuO55l17X5XKxb98e/uM/\nftBzbPr0mYwdOx7wjsATE4ddlYSPHTvKxIlTAMjJGUlxcdE1ry/Lxbzyyt9oampk8uSpfOELjzJr\n1p09jUUUxdPzSaU/+GVy/7xdlzyKh/1Vn7C57G263F1khKeySlpGnCn2M7/G0e1iw98LWbRqLKFm\nA2moZcYAABGoSURBVOYwI1ljRC9T4fNbmrHghqPsK/XVLffV1VVXNZZ46qlvY7PZaGxs4He/e4HT\np0/zjW/8G2lpGWzf/i5vvfUmI0eO/tTo1vugoOATsrNH8eST3+TYsSPYbDZqa6uxWsP5z//8CWfO\nnKarqxOL5fImeufPV/LrX/8PBoORX//6v/n444+IiYmhtraGV155FYfDweLFd/d0dQoJCeG7332a\nn//8J/y/9u49PKZrb+D4dyaZjFwJQoiWlmZHG/eICG21OO6qSKuHqlJJ0cTjUnfl1aI9TWjoDdXW\n+3LqbVUdHJeipXWpJmjdahFVIo4gKpHJfWbOHzuZZCSZhMokmazP8/hjZu3L2svOb9asWfu3Vq1a\nY3UtR4/GExkZQW5uLgkJZ1m8OLrU6zYY0jl+/BemT59D06bNmDVraqnBvaDnnpp6y2rxjgJeXuqo\nwfPPDy9WlpFhwN29cAKGVqu16hwC9OjRi8GDw3Bzc2fWrKkcPLif0NCulv3nzp1BePj4Uq/lr6py\nwd1sNmM4eQInD09qNWt21/snG66x7szXnE+9QC2nWrygDCa0cXCZDyO56J3xD/TlxjUDD3qUPA4v\nSdVFgwa+iPzVywDLkEFExMsYjUbq1/fh889Xo9fr8wOVRwlHMaPRaOjf/xnWrVvDlClReHi4ExEx\ngZCQLiQmJjJz5hScnZ0ZOXKMVQ+7Th1v3nprPq6urly6dNGS/rd58+ZotVrLbwBFtWnTjqCgYFat\n+kg9e/7x2rcPsiQdu3TpIq++OppNm7ZbluUr6ttvd2AymZg2bRKg/g5w5EgcHTp0RHNHDMjIMFCr\nVi1q167D7dvpJRxrO08/3bPEVAxubu5WqYTNZrNVYAcICxtmadfOnbty9uwZQkO7kpx8ldmzpzF4\ncBg9evQq3uz3SZVL+ZtzORHjrVu4BQbe1RTIPFMeO/7Yw6Kfl3I+9QJtfAKZGzKFrn4hpQb2pIt/\nEn/gD8vrDqFNefBh+UCSVP09/viTxMf/zKlTJy3vXb6cyPXr1wANsbHRjBkTwezZ83n44RZWQxSZ\nmZnk5uZy4cLvmM1mfvxxH23atCM29kO6devO2rVrOHbsCPXq1WfJkvcZOXI0K1d+YOkJGwzpfPrp\nShYsWMz06XPQ6/VFAn/xZ0KKfiiEh4/n8OGDJCUllnhd3t5lD8n84x/vEROzjJiYZUyc+DobN34J\nQKNGjTl6NN6y7eHDh2jZ8jGcnZ3p1CmEDRvWW8q++243X321vtQcO61bt+Gnn9Q1jE6ePGGVXx4g\nPT2dF198nszMTMxmM0eOxBEQ8Cg3b6YwefJrjB8fRd++A2xey19V5Xru9zIF8o+0S6z7bQNXDFep\n7eLJc/6DaNug7PH6uj7u7NtxlsD2ftRyldMbJcfh6urKO+8s5aOPlpOScgOj0YiTkxNRUZPx9fWl\nV68+zJ07HU9PL3x8GpCWlgpAWNgLRESMonFjP3x9G6PRaAgIaMnChfPR6XSYTCaioibTsKEv8+bN\nYtOmDRiNRl5+eSygDnW4u3vQqlUbwsNHqesYeNYmJeUGjRo1vuOBP41ln4L3XVxcmDlzHuPGjba8\nXzAso9U6kZFhIDJyEi4uLmzfvhXAsoyfEGcAdUWmAk8++RTLly/h+vVrTJ8+h5iYt1mx4gNMJhOB\nga3p1asvAJGRk1i+fCnjxo0G1HTIixa9C5Q85v7EE08RF3c4f3uYOXMeALt27SAzM5OBA58lImIC\nUVER6HQuBAUFExISynvvRZOens5nn63is89WARAdvazYt5j7ocql/E18ZxGZCedovnQ5Th4lfVUs\nlJWXzdYLO9mbeAAzZro0DmZQ83646YrnWi9w9lQyjZrUtiycYTSacHKy/xcYmc60kGyLQrItCpXV\nFufPJ3DmzGn69Rtox1pVjmqf8teYYSDzfAK1Hnq4zMB+KkWwXmzkZtafNHCtzwsBQ/D3Lr76+Z3S\n07LYv/safYaoPfvKCOySJP11Xl5eNSKw36sqFdwzTqtTIG0NyaTnGNhwbjNxycfQarT0avo0vZt1\nx6WUXOtms5nkK2n4+qnz1Nt2eoCM9JwKqb8kSfbj41P67DepigV3w/HSx9vNZjNxycf4+twW0nMN\nPOjZhOEBQ2ni2djmMfNyjez612me6qvQpFldtFotHl5yLVNJkhxblQnuZpMJw6kTOHl6oX+wqVVZ\nSuZNvhAb+e3mWVy0Oga36E+3Jl1KfRjJbDaTk52HvpYOnYszPZ95FDd3uZapJEk1R5UJ7tmJlzCm\npuLVuYtlCqTJbGLv5QNsOb+DHFMuLev6M0wZTH1X29Oh/khI4dihSwwa0Q6tVmMZkpEkSaopqkxw\nTzvwIwDubdoCkJT+H9ad2cDFtETcnd0Ypgwm2Ld9qbnTzWazpaxZi3okJ6WSk50npzhKklQjVYmp\nInm300jd/yPOdeuhbx3IlvM7eDsulotpiQQ1bMvckKl0atTB5qIY+3ed48LZ64A6bzakW3MZ2KUa\n6+jReObNK8yV8v33uxk58nmSk6+ycOF8Zs9+3Wr7Z55Rn5Tctm0LYWEDrZ6+nDdvJseOHbF5vj17\ndtGz5+PcuHHD8t7q1SvYtOlrq+3Cw0dx9epVAH799RiTJk0gMjKCsWNH8s03G2ye4+TJE4SHj2Lc\nuDGWOeJFrV37OZGREURGRjBq1N8t1wRgNBqZM2cahw8fsnkOR1Ileu639uzGnJND3uMdWXx0Odcy\nbuCtr8Mw5VkC67cs1zFatGzA6V/+w0P+PhVcW0m6O9e/Ws/t+LhybXvRSYvRaCpzO8+gjviEDSvX\nMXft2sH69euIjf0Yb281/ezx47+yc+c2y0M8RWVlZbFsWQwzZszNf0dT5mpjW7Z8w9Chw9i8eSOj\nR4ere5WwT8F7SUmXiY2NJibmfby9vcnOziYq6lX8/JoQHBxS4jliYhazcOG7NG7sx+uvT+TcOWGV\nZGzEiFGMGDEKgGnTJjFhwkTLud566w2uX7/OwIGDbV6HI6n0nrspK4tb3+0mz1XHB/ojXM9IoVuT\nLszpNNlmYM/NMbJ/1znLH0KjB+rQfUD5PggkydEVBNEdO/7Nl19+QWzsR5bArtFoiIiYwOrVK/LT\nEVjv16dPPy5evMDBg/st79t62PHKlSTS09MZPvwldu7chtFoLKN2Znbu3Ebv3v0tddLr9Sxd+j5B\nQcGkpaUW+2ZhMKSTm5tL48ZqMrTg4M7Exf1c4tH37fsOLy8vOnbsBEBmZiYzZrxB+/ZBNq/D0VR6\nz/309vW4ZGQQ38odn9q+DA8YykO1m5a5n7NOS9qtLM6eTJbZG6UqzSdsWLl72ffrCVWz2czx479w\n48Z1bt++TV5e3h3nacArr4xj8eI3WbJkudV+Wq0Ts2f/D1OnRhFYjrTbW7f+i759B+Dh4UFgYCv2\n7v2O7t172thDQ0rKDfz9rVP7urmpWRa9vGqzcOG7VmUGg8FSrm7rxpUrSSUefe3az5k/f5HldYsW\nj5R5DY6o0nruqdlpfPLLGrK//4FcJw0Ne/RlRseJNgN7+u1sLv+h5ljXaDT0fKYlAa197VVlSapW\n6tWrz3vvfUhY2DAWLJhr1WvVaDT87W+9cXNzK3Gsu0mTBwgLG0Z09Ns2z2E0Gvn22+3s3buHKVOi\nuHTpkiVRl16vJzc312r7zMwM9Ho9vr6NSE5Otio7d+4s586JEs/j7m6dhdFgMODh4VlsuwsXfsfD\nw7PYqkg1kd2Du9ls5sCVw7x5OJqs+Hg8M0y4dw2l12P9cNba/iKRachh1+bTZBjUJ0x1Ls5ljgVK\nUk3l59cEnU7HkCHPodM5s2bNaktZQaCfOnUmX3zxf1aBs8CQIc+TmprK0aPxpf6dHTp0gEcfDWTZ\nso+JiVnGqlVruHkzhfPnE/D3D2D//h8swzRJSZfJycnF29ubnj17s3XrJm7dugVARkYG0dGLSUlJ\nKfE87u4e6HTOJCVdVh9ojPuJtm3bFdsuPv5nOnfucncN5aDsGtxNunRij63gn2e+xmwy8XSCEzg5\n4de39B850m9nk5ur3hw+vp70C2uFq5ucBSNJthTNtAhq1sLNm7+xpLwtKKtTpw5RUZPJzs4ucb9Z\ns96w6n1HRkZYnWfr1k3FfpQdMGAQGzd+SceOnWjdug1jxrzIhAljWbhwPnPnLgDA17cR48dHMXv2\n65YZLn369CckJLTEMXeAqVNnsWDBXMLDX8LfP4CWLR8DYPLk1yzDTomJFy3j8qW1S01h16yQz30x\nwYzWRKv6LRmU9QhpKz/Bq3MXfMeMLXWfvdsFOp0TXXq0KHWb6khm/ysk26JQVW+LZctiiIqaYpdz\nVfW2sKd7yQpp32EZrYnn/AcRHvgSOXv2AuDdu/hUrKJTwUK6PUy9Bu7FtpEkyf6GDRtR2VWQysnu\nY+4PeT1IVsI5ss4n4N6mLXo/669QeblG/v+TONJuZQJQy1VHQGs5G0aSqoIGDRpWdhWkcqqU2TJ/\nbv83AHX79CtW5qxzonVQE27eMNi7WpIkSQ7D7sHddOUqhhPHcX3EH9f8+adXL6dycE+CZZvADn40\na1Hf3lWTJElyGDbnHiqKogU+BFoD2cArQojzRcoHAHOBPOBTIcQnZZ5xn5rbwbtP4Vh7XR939u08\niyE9G3eP+7+WoCRJUk1TVs99EOAihAgFZgAxBQWKouiAJUBP4EkgXFEUm0ujLGr3KjlHjuHi14Qr\nTo0sQy8uemfCXg6SgV2SJOk+KSu4dwF2AAghDgNBRcpaAglCiFQhRC6wH3ii+CGKnGxvPJhM1O3d\n15IbxlKmrTnzTyVJkipaWbllvIC0Iq+NiqJohRCm/LLUImW3AZurYvz2w3Hq162HZ8dgHnVyolmL\nevdUaUmSJMm2soJ7GlA0gUNBYAc1sBct8wT+tHWw37w70i7AHY2zelp3TzkMI0mSVBHKCu4HgAHA\nV4qihADHi5SdAR5RFMUbMKAOybxb/BCFpi4fIcdeivDxKZ74qKaSbVFItkUh2Rb3zmb6AUVRNBTO\nlgF4GegAeAghVimK0h94A3XsfrUQ4qMKrq8kSZJUDnbNLSNJkiTZR6WvxCRJkiTdfzK4S5IkOSAZ\n3CVJkhyQDO6SJEkOqEIWyK6QnDTVVDna4gVgImpbnADGCyEc8lfustqiyHYrgRQhxEw7V9FuynFf\ndERN96EBrgIjhBDZlVHXilaOthgOTAaMqPHi40qpqJ0oitIJeFsI8dQd799V3Kyonvt9zUlTzdlq\nC1fgTaCbEKIr6hO+/SullvZRalsUUBQlAggEHPIDrghb94UGWAmMEkI8jpoCpPSV46u/su6Ld4Hu\nqOlQpiiKYvNJ+OpMUZRpwCpAf8f7dx03Kyq439ecNNWcrbbIAjoLIbLyXzsDmfatnl3ZagsURQkF\ngoEVqD1WR2arLfyBFGCyoih7gbpCiLN2r6H92LwvUB+erAO4ot4XjvzBnwAMpvj9f/e5vCqmfiXn\npClSdlc5aaq5UttCCGEWQlwHUBQlEnAXQuyuhDraS6ltoShKI9QH4l7D8QM72P4bqQ+EAsuBHkB3\nRVGewnHZaguAU8AR4CSwRQhRdFuHIoTYiDrscqe7jpsVFdzva06aas5WW6AoilZRlGjUr51D7F05\nO7PVFkNRg9o2YDrwd0VRRtq5fvZkqy1SUHtpQgiRh9qrvbM360hKbQtFUVoDfVGHpZoBDRVFGWr3\nGla+u46bFRXcD6D+h2ArJ42iKC6oXy0OVVA9qgJbbQHqEIQeeLbI8IyjKrUthBDLhRBB+T8ivQ38\nUwjxv5VTTbuwdV/8DngoitI8//XjqL1WR2WrLVJRhyqz8wP+NdQhmprmruNmhaQfkDlpCtlqCyA+\n/98PRXaJFUJssmsl7aSs+6LIdi8BihBilv1raR/l+Bsp+JDTAAeEEJMqp6YVrxxtEQGMBnJQx6TH\n5n+jcUiKojRD7dyE5s+mu6e4KXPLSJIkOSD5EJMkSZIDksFdkiTJAcngLkmS5IBkcJckSXJAMrhL\nkiQ5IBncJUmSHJAM7pIkSQ5IBndJkiQH9F8wd38cVaxO8gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10dd0de10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(fpr_lr,tpr_lr,label=\"LogReg, AUC: %.2f\" % roc_auc_lr)\n",
    "plt.plot(fpr_nb,tpr_nb,label=\"GaussianNB, AUC: %.2f\" % roc_auc_nb)\n",
    "plt.plot(fpr_knn,tpr_knn,label=\"KNN, AUC: %.2f\" % roc_auc_knn)\n",
    "plt.plot(np.linspace(0,1,100),np.linspace(0,1,100),\":\")\n",
    "\n",
    "plt.legend(loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression C Tuning\n",
    "\n",
    "Perhaps this is because we didn't tune our LogisticRegression regularization parameter, **C**.\n",
    "\n",
    "If you recall from last lecture, **C** is the inverse of regularization strength. A low C value (just a scalar, no interpretation) corresponds with high regularization. A high C value corresponds with low amounts of regularization.\n",
    "\n",
    "We can use the provided `LogisticRegressionCV` class, which performs cross validation over a range of C values.\n",
    "\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html\n",
    "\n",
    "By default it runs over Cs values of C on a logarithmic scale between 1e-4 and 1e4, though you can provide your own values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.815217391304 AUC: 0.890961262554\n"
     ]
    }
   ],
   "source": [
    "model = linear_model.LogisticRegressionCV(Cs=1000)\n",
    "\n",
    "\n",
    "\n",
    "model.fit(X_train,y_train)\n",
    "y_pred  = model.predict(X_test)\n",
    "y_pred_prob = model.predict_proba(X_test)\n",
    "\n",
    "\n",
    "# We are going to pass in the true y, as well as the predicted *PROBABILITIES*\n",
    "fpr_lr, tpr_lr, thresholds_lr = metrics.roc_curve(y_test, y_pred_prob[:,1])\n",
    "roc_auc_lr = metrics.roc_auc_score(y_test,y_pred_prob[:,1]) \n",
    "\n",
    "print \"Score:\",model.score(X_test,y_test), \"AUC:\", roc_auc_lr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the very similar results, I would imagine the betas are regularized in LogisticRegressionCV() to a similar level of the default in LogisticRegression()."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New ROC Curve With CV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x10e3ba790>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAECCAYAAAAFL5eMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XdA1Hee//Hn9GEKQxUEpSjypdgBW2zRaHqsSYy66eWy\n2XK/3exms3u53N7tXbZcbm832ZTLZruaxBqjSTQx9hgj2BG+gNjpSBtmYNr398coYixYgIHh8/hL\n5vtl5u2X4TVfPlWlKAqCIAhCcFEHugBBEASh84lwFwRBCEIi3AVBEIKQCHdBEIQgJMJdEAQhCIlw\nFwRBCELXFO6SJI2VJGnzZR6/V5KkryVJ+lKSpCc7vzxBEAThRnQY7pIk/Rh4BzB843Ed8D/ADGAK\n8LQkSf26okhBEATh+lzLnXsJMBdQfePxdKBEluUGWZbdwA5gcifXJwiCINyADsNdluVVgOcyh0KB\nhnZfNwG2TqpLEARBuAk306HaAFjbfW0F6m6uHEEQBKEzaG/iewuBIZIkhQPN+JtkfnO1b1AURVGp\nvtm6IwiCcHlP/GIjNQ0tRNmMgS6ly2h8HmKaKohrPE1c4xkiHTWoAKfWwtcD76GfeyNP/uGd6w7O\n6wl3BUCSpIcAiyzL70iS9ANgA/6/AN6VZbn8ak+gUqmorm663hqDUnS0VVyLc8S1uEBciwuio614\nvQrhFgO/fGZ8oMvpNIrPR+upkziOHMFxJB9nSRGK2+0/ptXiSIzlUHgzx2O0xFibyM5+8YZe55rC\nXZbl48CEc/9e1u7xdcC6G3plQRCEPsJdU43jyBGaj+TjKDyCz25vO6YfMBBzRiZNidGsOXEGh13B\nkXqSB1JnM7LfsBt+zZtplhEEIYh98EUJewqrAvb6Go2KuqZWwq2Gjk/uYbyOZhwFBTgK/Hfn7qrK\ntmPa8HAsEyZiyszElJaBx2xk3bENbDm1EVWYhhHuKSzIno/FaL6pGkS4C4JwWXsKqwIeruFWAzlp\nPX/6jM/tpqX0KI4j+TiO5NNy/Bic2ytDbTRiHjkKU0Ym5vQMdLH9Od/3mF8r8+nfDnA6poDoiEgW\nps1jSPjgTqlJhLsgCFcUbjXwm29PCMhr9+T+B0VRcJ05jeNIPs1HjuAsKkRxufwHNRpCUoZgSs/A\nlJGJMSkZlfbiqLW7mllR/BF7KvcSGZLE6NZbWDBmCjqNrtNqFOEuCIJwDdx1dW135o6CfLyNjW3H\n9P3jMGVkYkrPIERKQxMSctnnUBSFXSfy2PzVfsr6FZFgHcDC7FkMsMbR2SMJRbgLQgBcqT1bo1Hh\n9faMrS8D3SQTaL4WJ47CwrZ2c1d5WdsxTWgo1rHj2wJdFxHR4fPVOut4T15FQXUxQyomc+fgIdyV\nPQm1qmvWbxThLggB0BPaszvSW9q7O4vi8dBy/Ni5ppZ8Wo6VgtcLgEqvxzR0OOYMf1OLPn7ANd9p\n+xQfnxftZFPpduyGetKjU5m1OIsBkf06/W69PRHughAgl2vP7sntzMFGURTcFeX+4YkFR3AWFuBr\nafEfVKkwJiW33ZkbB6eg1l1/e/gZezlLClfQUOqlf8Vwxs6NZVx8VpeG+nki3AVB6DM8DQ04Co/g\nyPcHuqfubNsxXb8YrOMm+DtC09LRmG98KKLb62Z90RdsKt+MDx9ZGSMZP3wkafHx3RLsIMJdEIQg\n5nO7cMqFbROIXKdPtR1TWyxYssdgzsjElJGBLiq6U16zpP4YSwtXYMobRGz/wcyaPJGhUemd8tzX\nQ4S7EHQCPfnmWvT09vberrWsjIZtm2n88kt8jmYAVFotpnR/kJsyMjEMTECl7rzOTKfHyeqST9hZ\n9hUqVNySncYw4x0MjRrYaa9xPUS4C0FHdFb2TT63C3teLg1bt+AsLgL8o1rCZ9yOaegwQoakotbr\nu+S1D1Tns+LgesKOpNI/K5ZFmfNItiV2yWtdKxHuQlAK5OQboXu5ysuo37aVxi934Gv236Wb0jOx\nTZmKZeSoSyYQdaaG1kaWF33IvupDaNUaUmNCmTbwUeJtHQ+N7Goi3AVB6HV8bjf2vXk0bN2Ms0gG\nQGO1En7HXdgmTUEfE9Olr68oCrvK9/Dpnl14W1QMSkliUdo8Ys1d+7rXQ4S70OPcbJt5T2+SEW6c\nq7KChm1baNy5E6/dP2Q0JC2dsCm3Yhk1ukvv0s+rclSzrHAVRfVHsRrCSTk1nofnTkSv61lx2rOq\nEQRuvs1ctGcHF8Xjwb5vL/VbN+MsLABAY7ESfvsd2CZNRR8b2y11eH1eNp3cxvbcQ9SHVjAsNoMH\nU2djnW5Fq9N0Sw3XQ4S70COJNnPBWV5B9Zr1NO7cgbfJv45LSKqEbcqtWEZn3dCkoht1svE0SwpX\ncNpeRqJ9FCO1OcwdNq7bxqzfCBHugiD0GIrHg33/Phq2bsFRkA+A2mwmbMbthE2egr5/XLfW4/K6\n+Kh0I1/K+2gxNTGhfw53j7kDjVfXo4MdRLgLAfDNNvVvLpYl2sz7Hnd1NQ3bt9KwY1vbaouhGemY\nxk/Ckp2NWtc1QxivpuBsEcsKV9HQaCe1cArj5wxg9ODun4x0o0S4C92uozZ10WbeNygeD/aDB2jY\nuhnHkXxQFNQmE2G3zcA2eSoDRqQFZJ0du7uZlUXr2FO+D5UGpqdMZuigYcREhXV7LTdDhLsQEO3b\n1MViWX2Lu7aGhm1badixHW9DPQDGwSn+ES/ZOV020agjiqKQV3WAFUVr0ZaFMaR5HHPvz2GgNT4g\n9dwsEe6CIHQ5xeul+eAB6rduwZF/yH+XHhJC2LTp2KbciiF+QEDrO9tSx/vyag7XFqJT67grZzi6\nwlj6G7tnJE5XEOEuCEKXcZ+tpWH7Nhp3bMNTVweAcdBgbFOmYs0eg9oQ2L4Vn+Jj25ldrD36CVHF\nEmkpmSwYcw/RpkgYFNDSbpoId+G6iUlGwtUoPh/Nhw7SsHUzzYcOtt2l226dRtjkqRgGJgS6RADK\n7BUsLVzJscYTmLQhjExPwWKP9Ad7EBDhLlw3MclIuBz32bM07thGw/ZtbeukG5KSCZsyFeuYcQG/\nSz/P7fOw8fgXfHZ0O2EVAxk9cgT3S/dh1VkCXVqnEuEu3BAxyUiAc3fphw/RsG0LzQf2g6KgMhix\nTZmKbcqtGBMCuzLiN5U2HGdJwQoqHFWEGcMY1JLBlJB0QvXWQJfW6US4C4Jw3Tz1dTTs2E7Dtq14\nztYCYEhIxDblVkLHjkVtDAlwhRdzelpYe/RTdh3bi9ZtYHLqBO4bfAdkqTGGdN9M1+4kwl24yLW0\np4s2875J8flwHDlM/dZzd+k+HyqDAdvkKdgm34oxKSnQJV7WoZojvCevpr61gYGeIcSeSmfuHePR\naTVBnYBB/F8TbsS1tKeLNvO+xdNQ779L374VT00NAIaBCf4RL2PHownpWXfp5zW6mlhRtJYDpwtQ\n9F7uSrqNmUnTqDzZhFbbeTsw9VQi3IVLiPZ0QfH5cBQcoWHbFuz794HXi0qvJ3TiJP9denJyj11b\nRVEUvqrIY1XxRzg8TtJOT2RkegqTBmUAMCApPMAVdg8R7oIgtPE0NtK409+W7q72N8/pBwz0j3gZ\nOx6NyRTgCq+u2lHLMnkl8tkSDFo996fOYviIEVSd6XszoEW4C0Ifp/h8OOVC6rduwb4v78Jd+oSJ\n2KZMxThocI+9Sz/P6/Oy+fQO1pVuxNcKmcW3MXfxKGLDogGIiAyuYY7XQoS7IPRRnqZGGnfu8N+l\nV1UCoI+LxzZlKqHjJ6AxmQNc4bU51XSGJYUrONV0BovOzP0jZ+HWW/E2aKF3rfXVqUS4C0IfoigK\nTrnQ35a+Nw/F40Gl1WIdP4GwybdiTEnp8Xfp57m8bj4+9hmbTm0jpD6C0dpJPHjHNCw6M/ScrUwD\nRoS7IPQBXrudxi93UL9tC+6KCgD0sf3P3aXfgsbSu5ot5LMlLJVXUuOsJdIYwZyseznycQN6nxii\ne54Id0EIUoqi4CwuomHrZux5uRfu0seOxzZlKiFDUnvNXfp5DreDVSXr2VW+h7CaeKamT+G+jBkY\nNHqGP+lD0weGOF4rEe59TEeTlMQEpd5NURRcp0/RlLeHptw9bXfpuphYwqZMJXTCxF53lw7+/9e+\n6kN8ULSGJpedeEt/stxT0B43YBjmX/9dBPvFRLj3MWIXpOCjKAqtJ07QlLcHe15uW+eoSqfDOmYs\ntslTCZHSet1d+nl1LfW8X7SGQ9VHMDvDmTX0TqYnTEbxgbPZHejyeqyrhrskSWrgDWA40Ao8Kcvy\n0XbHFwE/ALzAn2RZfqsLaxU6iZik1PspPh8tx0qx782lKS+3beaoSq/Hkp2DNSsH87DhqI3GAFd6\n43yKj22nd/Hh0Y9p8bYyxJJCaH4GI0YPR6PWgBqsNk2gy+yxOrpznw3oZVmeIEnSWODVc4+d9xsg\nA2gGjkiStEyW5YauKVUQ+jbF56PlaMm5O/S8tmV11UYj1rHjsGTlYM4c2mOW1r0ZFc1VvHZwNYXV\nRzGpzSxMu4cJ/cdQFluPyRyYbfh6m47C/RbgUwBZlndLkpT9jeMH8Y8k9QEqQEG4KTe7EUZHRJt6\n76L4fDiLZJrycrHvzWvbc1RtMhE6/hYsWdmYMjNR64Ij8Dw+D5+d2MKnxzfhUbwM94yhX20yE6aO\nRqVSEZ/YN5YO6AwdhXso0Njua68kSWpZln3nvs4H8vDfua+UZbnxm08gXJ+b3QijI6JNvedTPB4c\nciH2vFzs+/LwNvmnzqvNZkInTsaanY0pLQOVNri6zI41nGBJ4QrKmyux6UN5KuchEnXJfLnpKG6X\nF70huP6/Xa2jq9UItF/Fvi3YJUkaDtwFJAIO4B+SJM2XZXnF1Z4wOjr4FsW/UZe7FhqNiqgwI+/+\ny8wAVBQ4ff194XO7qT9wkOL3vuLs7q/xNNkB0NlCibp9JlETxhE6NBN1kAU6QIu7hWWH1vJp8RYU\nFHJqb+OeqeNIHxAPwJyFowNcYe/U0TtlJ3AvsFySpHH4m2HOawCcQKssyz5Jkqq4hsm+1dV9bwGf\ny4mOtl72Wni9/patvnSdrnQtgp3P7cKRn09T3h6a9+/D53QCoLGFETZtOpasHP9YdLUaN1Bb5wxs\nwV0gv7aQZYWrqGutp58pioXSfPRnbRzaU0Z6RnyffF9czo3c/HQU7quBGZIk7Tz39WOSJD0EWGRZ\nfkeSpLeBHZIkuYAS4C/XXYEg9CG+1laaDx/yN7kc2I/S2gKANiKC0FsmMfC2KbRE9EelDu4x200u\nOyuK15JbuR+tV8f45pncP2kyBp0ewiFhUESgS+z1rhrusiwrwLPfeLio3fG3gbe7oK4eqTM7OzUa\nVdtdenuiwzP4+FpaaD54wH+HfuggissFgC4qGsvUW7Fk5bStjx4abaU1iO9WFUXh64q9rCz5iGa3\ng8TQgSyU5pG7tpJTJfWkpPv7g3rrmPyeJPga8LpQV3d2gujwDBZeh4Pmg/tpysvFcfgQits/2UYX\nE4M1KwdLVjaGhMQ+FWI1zrO8J6+i4GwRJo+Fu6Lu4c5RE1Gr1ETPixYdpp1MXM3r1FkTgPpqO3Mw\n89rt2A/sw56Xi+NIPorHA4A+Lg5LVg7WrGz08QP6VKCDfzLSllM7+Kh0Ay6fm/SIVG6zzOTrT0/j\nzvBiMKoxGINzk+pAEuEuCDfB29SEfd9emvL24CgsAK8X8O9eZM3K9t+hx8UHuMrAOWMvZ0nBCk40\nnSLUF8aD6bczNs4/Zt022ybu1ruQuLLtiEW1hGvhaajHvtcf6M4iGXz+aR+GxKS2QNfHxAa4ysBy\ne918cnwTn53cgk/xkRMzmn5F6WiO2VDF+/9y6T/AFuAqg5sI93bEolrClbjr6rDvzcWel4uzuAgU\nf2e4cdAgf5PL6Gx00dEBrrJnKK47ylJ5JVWOGiIM4SxIm0tmpIQ9oZVTpWcDXV6fIcL9G8SiWsJ5\n7toa7Hn+hblajpb4H1SpCEkZgmV0FpbR2egiIwNbZA/icDtZc/RjdpbtRoWKKbETcW/vR9KIJAAs\nVgPpI/oHtsg+RIS7ILTjqqo6F+h7aD1+zP+gSkWIlOZvchmdhTZMrG/yTfurD/OBvJoGVxNx5lgW\nps0n2ZbA3oYT1FY1Y7aI5szu1qfCXbSpC5fjqij3L8yVl0vryRP+B9VqTBmZWLJysIwajTY0NLBF\n9lD1rQ0sL/qQ/dWH0ao03Ga+nThXMsm2BABGj08McIV9V58Kd9GmLsC53YrKyrDn7aEpLxfXmdP+\nAxoNpqHDsWZnYxk5ulfuWNRdfIqPXWV7WH10PU5PC4NtySxMm4dNFcbqf+xjRPZAjCFieGMg9alw\nB9Gm3le1337OnpuLq6IcAJVWi3nkKKyjszGPGInGbA5wpT1fpaOaZYUrKa4vxagxcoduFpMHj8Jm\nNgGw4Mkc1EG+fEJv0OfCXeg7/NvPHacp99z2c9X+JjmVTufvEM3KwTx8BJqQkABX2jt4fV4+P7mV\nj49/jsfnYXhUJg9Kszm2r4EvPz/KnfOHAYhg7yFEuAtBpW37uXOdop7aWgBUBgOW7DFYs7MxD+3d\n288FwonGUywpXMEZezmhOit3Rt7NpKGjUKlUjBhjZUiGaM7saYIq3EWHad+k+Hw4S4r9Ky3uzcVT\nVwec335uvH9zi8xhqPXBsVtRd2r1ulhXuoHNp3agoDCh/xjuSbidtX89TLKljoHJEajVaiyh4sOy\npwmqcBcdpn2H4vXiLC7yt6HvzcPb4N+6V20yETrhFixZOZgyMlHrRKfejSqoLWKZvJLaljqijZHc\nP2gOmbGpAMycnSn2Mu3hgircQXSYBrML28/twb53L177ue3nLBZCJ03GmpWDKS096Laf6252dzOr\nitexuyIPtUrNzMRbyfSM5sDG06QvVlCrVcTEiaGhPZ34LRB6NJ/bjaPgyLn9RPficzQDoAkNxTbl\nVqzZOYSkSqg0mgBX2vspikJu5X5WFK/F7m4mwRLPwvT7GWiNQ1EUasqbcbV6xBDHXqJXhbtoU+8b\nfC4XjvzD/s0tDuy/sP1cWBhh427Dkp1DSMqQoN+tqDudbanjPXk1+bWF6NQ65qTcjb4wFneZDiT/\n5hnjpg4KdJnCdehV4S7a1IOXf/u5g+e2nzvQbvu5SEInTsaalY1x0GAR6J3Mp/jYdnoXH5Z+gsvr\nIi18CA+lzSUqJJIKdQP5+8oYJIkF0XqjXhXuINrUg4mvxYn94AFqDx/gbG7ehe3noqOxZE3DmpWN\nISm5z21u0V3K7BUsKVzB8caTmLQh3D/kfryFYYTr/WvnxA6wESuW5e21el24C72b1+Gg+cB+/+YW\nhw+17Vaki4n1L8yVnYNhYIII9C7k9nnYcHwTG09swat4yeo3gvmp92HVWfhk92GKDleK1RuDgAh3\noct57Xbs+/dh35tLc/7hC7sVxcVjycomYcZUmkPCRKB3g5L6YywtXEmlo4pwQxhzBtxHjBJHqN4K\nwIz70tHqROd0MBDhLnQJT1Mj9n17/fuJttt+zjBwoH+lxdHZGOLiADBHW3GI/WS7lNPTwodHP2H7\nmV3+tdYHTOC+QXdgr3Xz0fsHefCJHExmPTq9iIRgIX6SQqe5aPs5ubBttyJDUvK5tdCz0cfEBLjK\nvudgdT7vF62hvrWBWHMM8wbMZki/JHRaDcYYI/c8MJwQkxjeGGxEuAs3xX32LPa9edjz9uAsKW63\n/dxgLFnZWLOy0UWJ0RaB0NDaxPLiD9lXdRCNSsPdyTOYkXgrOzcc5ayulIm3DQEgOtYa4EqFriDC\nXbhu7ppq/+YWe/Mu3X4uKwfL6Cx0ERGBLbIPUxSFXeW5rCpZh9PjZJAtkQVD5hEf6t+0e9zUQRwv\nrglwlUJX6zHh3tEEJRCTlALJVVmJfa9/P9GLtp9LS/c3uYzKQhsWFtgiBaocNSyTV1FUV4JBo+fB\n1NmM65fDij/v5e4HbNjCQzCG6EgbLkbDBLseE+4dTVACMUmpu7nKyy5sP3fqpP9BtRpT5lAsWf7d\nisT2cz2D1+fli1PbWX9sI26fh6GR6SyQ5hBu9H/gDs8ZQF1NM7ZwsXZ9X9Fjwh3EBKVA828/d6Zt\ncwtX2Rn/AY0G87Dh/iaXkaPE9nM9zMmm0ywtWMEpexlWnYVvpc8izpXIkZ013DLdH+5DR8cHuEqh\nu/WocBe6n6IotJ462ba5hbuiAmi3/VzWue3nTGL7uZ7G5XWx/thnfHFqOz7Fx7j+2cxNuQezzoSr\n1cO2jcU0N7ViFk2ZfZII9z5IURRajx871+SyB3d1NQAqvd7f3JKVjWX4CNRG8Sd8T1V4tphlhSup\naTlLlDGCh9LmoS4PpbVewRwNeoOW+x/LRq0WE8P6KhHufUTb9nO5e2jKy8Vz9sL2c9YxY7Fkndt+\nziDu8nqyZreDVSXr+Ko8FxUqbkuYwt3JM9Br9Bw5Wcb2z4qZtXAkgAj2Pk6EexBr234udw/2fXkX\ntp8LCcE6brx/c4vMoWL7uV5AURT2Vh1kedGHNLntDLDEsVCah6bOgl7j//mlj+hP4uDIAFcq9BQi\n3IOM4vXiLJLPjUPPxdvYCIDaZCb0lklYsrIwpYvt53qTupZ63i9azaGaAnRqLbMH38W0gZPAp+KD\nD3LJmZhESno/VCqVaF8X2ohwDwKKx4OjsAD73tyLtp/TWKzYJk/x7ycqpYnt53oZn+Jjx5mv+PDo\nJ7R4W0kNG8wCaS6hKhsatQbUMHNWBnqD+LkKlxLvil7Kv/1cPvbcXOz79128/dxU/1roYvu53qu8\nuZKlhSsobThBiDaERWn3M75/NmUn6/nws/3MfzQLrVZDZD8xLFW4PBHuvYh/+7lDNOXlXrT9nDY8\nnNDxE7BkZYvt53o5j9fDx8c+Y8PxL/AoXkb1G878lPuwGayoVCriEsIYkhGD2+VFqxUf3MKVXTXc\nJUlSA28Aw4FW4ElZlo+2O54DvAqogApgsSzLrV1Xbt/ja22l+dBB7Hl7sB88gNLqv7zayEhsk6Zg\nycrGmDxIBHoQKG04wQe5qzjVWI5NH8qD0hxGRGeya/NRwiKaSR/RH5VKRdaExECXKvQCHd25zwb0\nsixPkCRpLP4gnw0gSZIK+D9gnizLpZIkPQEkAkVdWXBfcH77OXteLs2HDrbbfq6ff6XF7BwMiUli\nc4sg0eJpYW3pBrad/hIFhUnx45k1+A5CtP55BqmZMez96qTYHUm4Lh2F+y3ApwCyLO+WJCm73bFU\noBb4gSRJQ4H1siyLYL8JPrebyr/9Gfuery9sPxcbizUrB0tWtth+LggdringPXk1da31xJj68dy4\nbxHmjWbHZ8VMmDYYnV5LZD8LM+7LCHSpQi/TUbiHAo3tvvZKkqSWZdkHRAETgOeAo8A6SZJyZVne\n3DWlBjdFUaha+neadn2JLjaW0DHjsGTloI+LE4EehJpcdlYUryW3cj9qlZo7k6Zze+I04qIjqK5u\nwuvxUXCwguHZAwJdqtBLdRTujUD7lfzPBzv479pLZFmWASRJ+hTIBkS434CGbVto3L4Nw8AEBv7k\nZ2KmaJBSFIXdFXmsKl5Hs8dBUmgCC9PmEaZEcLq0gbhY/zr4k29PRaMV/SjCjeso3HcC9wLLJUka\nBxxsd6wUsEiSNPhcJ+sk4I8dvWB09OV3fdFoVFc9HozO/18bCwqpXrYErdXK0JdexBgTFeDKul9f\n+LlX2Wv4v9ylHKwswKA18Oio+7kjZSpqtZqzNc0s/3MuqWkxfeJaXCtxLW5cR+G+GpghSdLOc18/\nJknSQ4BFluV3znWiLj3XubpTluVPOnrB6itshOz1Klc9Hmyio61UVzfhqa/jxCu/RvF6iX36WZrU\nITT1kWtw3vlrEay8Pi9bTu9kXekGXD43GZESC1LnYvSYKDtTj8Hony1870MjsIWHBPW1uB7B/r64\nHjfyIXfVcJdlWQGe/cbDRe2ObwbGXverCoC/A7XsjdfxNjQQ/cACTOmi0yzYnGoqY2nhck42ncGi\nM7MwbT7ZMSNRqVR8+UUJToeb6fekAxAZbRH9K0KnEZOYAqh62T9oKT2Kdex4wmbcHuhyhE7k8rr5\n5PjnfH5yKz7Fx5jY0cxLuRejytgW4DkTkymVq1EURYS60OlEuAdIxacbadi2FcPABGIeflT8cgeR\norqjLCtcSZWzhghjOA9Jc8mIlPB6fbz/xz3cPieTyH4WdHoN0rDYQJcrBCkR7gHgLC7m9DvvorZY\niHvuu2JkTJBwuB2sLvmYL8u/RoWKaQMncXfyTIxa/89Xo1GTPTGJhjqnWBNG6HIi3LuZp76Osrde\nR/H5iH/m2+iiogNdktAJ9lUd4oOiNTS6moi39GdR2nwSQwdSXdHErrxjTL1LQqVSkZoZE+hShT5C\nhHs3at+BmvT4I+hFB2qvV9/awAfyGg7U5KNVa7l30B3MSJjiX5IXiIgyU1froLG+BVu42LZQ6D4i\n3LvRhQ7UccTddy81NfZAlyTcIJ/iY2fZ16wp+ZgWbwspYckslOYRY+5HSUEVllADsfE2NFo1c741\nSvSpCN1OhHs3qd+6pV0H6mPil70Xq2yuYknhSo42HCNEa2ShNI/xcTmoVf4ZpSoV7PishHmPjEal\nUomftRAQIty7gbOkmKqlfxcdqL2cx+fh85Nb+eT4Jjw+DyOih/JA6ixs+lBOlp4lYVAEKpWKwWn9\niEsIE6EuBJQI9y7mqa+j7M3XwecjTnSg9lrHG0+ypGAFZc0V2PRWHkidzch+wwDw+Xzk7jhOc1Mr\nGSPjAAgxiU3HhcAS4d6FfG43ZW/+QcxA7cVaPK2sO7aBLad2oqBwS9xYZg++ixCtkWZ7K2aLAbVa\nzfR709FoxEJfQs8hwr0LVS9bQsvREqxjx4kZqL1Qfq3Me/IqzrbU0S8kioVp8xgSPhiAqvJGNq7O\n54EnctAbtIRFmAJcrSBcTIR7F/F3oG4RHai9kN3VzIrij9hTuRe1Ss3tidO4I2k6OrW2bamAfv1D\nGT5mIB63F71B/BoJPU+3viuf+MXGttUfv6muqZVwa3B0NDqPlvg7UM1m0YHaiyiKwp7Kfaws/gi7\nu5kE6wC/RPeCAAAgAElEQVQWpc1ngNXfjv71tmPoDRpGjk0AEBtpCD1at4Z7TUML4ZbLB1241UBO\nWr/uLKdLeOrrKHvD34HaX3Sg9hq1zjrek1dx5KyMXq1jXso9TB04sW14I0Da8Fh2by0VC30JvUK3\nhnuUzcgvnxnfnS/ZrS50oNYTdf+DmDMyA12S0AGf4mPr6S9ZW/opLq+L9IhUFkhziQqJwOdT+OLj\nQiZMG4wxREdoWAgzZomfqdA7iMbCTtS+AzV85h2BLkfowBl7OUsKV3Ci8RRmrYkF6XMYEzu67a5c\nrVahN2g4sr+M0eMTA1ytIFwfEe6d5EIH6kDRgdrDub1uPj2+iY0nt+BTfGTHjGT+kPuw6i047K2c\nOVnPkAz/Al/jbx2MWi1+lkLvI8K9E1zUgfrt74kO1B6spP4YSwtXUOmoJtwQxgJpDkOj0tuOKwrs\n+LyEqBgr4ZEmMXZd6LVEuN+kSzpQo0UHak/k9DhZU/IxO8p2o0LF1AG3cO+g2zFqjTQ3taJSgcli\nwGw1MGfxKLGCo9DriXC/CYrHIzpQe4ED1Yd5X15Dg6uR/uYYFqXNJ9l2oQ294GA5VeVN3DlvKCqV\nSkxIEoKCCPebULXsH/4O1DGiA7UnamhtZHnRh+yrPoRWpeGe5JnMSJyKVq3F7fKg0/vf/qPGJXC0\nsDrA1QpC5xLhfoPqt22hYeu5DtRHRAdqT6IoCl+Wf83qkvU4PS0MsiWxKG0eseaYtuMr/7aXKXdI\n9B9gQ6NRix2ShKAjwv0GOI+WULVEdKD2RFWOapYWrqS4vhSjxsCDqXOYGD/2oslIKpWKsVMG0dzU\nGsBKBaFriXC/Tp76etGB2gN5fV42ndzGx8c/w+3zMCwqnQdT5xBuDAOgttrOnu3HuX1OJiqViuQh\nUQGuWBC6lgj36+DvQH1ddKD2MCcaT7GkcAVn7OVY9RYeTp3NqOhhFzWVhUeacbu81NU6iIgyB7Ba\nQegeItyvQ9X5GaiiA7VHaPW6WF+6kS9ObUdBYXz/HOak3I1Z5x/tcrSwGr1Bw8DkCNRqFfc8OFz0\njQh9hgj3a9SwbSsNWzeLDtQeouBsEcsKV1HbcpaokEgekuaSFjHkonP0Bg07N5Xw4BM5Yi9Toc8R\n4X4NxAzUnsPubmZV8Tp2V+ShVqmZkTCVu5JvQ6/RoygKx4trSEyJRK1WMzA5gjmLR4lQF/okEe4d\nON+Bqni9/j1QRQdqQCiKQl7VAZYXfYjd3cxASxwL0+eTYL14TfVDeWeor3My6tya6wajLhDlCkLA\niXC/CsXjoewtMQM10M621PG+vJrDtYXo1FpmD76LaQMnoVFrUBQFe2MrVpsRlUrFtLvTUC6/H4wg\n9Cki3K+iatkSWkqKsY4ZKzpQA8Cn+Nh2Zhdrj35Cq9eFFJ7CQ9I8ok2RbefU1Tj46L0DPPBENiEm\nPZZQYwArFoSeQ4T7FZzvQNUPGEjMI4+LdttuVmavYGnhSo41nsCkDWFx2v2M65+NSqVCURQURUGt\nVhMRbWbM5OQrbt8oCH2VCPfLaN+BGv+c6EDtTm6fh43Hv2DDic14FS+j+w1n/pBZ2AzWtnP2fXUS\nt9vL2MmDAEgf0T9Q5QpCjyXC/Rs8DfWUvXmuA/XpZ0UHajcqbTjOkoIVVDiqCDPYWCDNYVhUxiXn\npQ2LZeemo2IvU0G4ChHu7bQt4VtfT9T8BzBnDg10SX2Cw+3kfXkN28/sQkFhcvwE7ht8ByFaf/u5\noih8sb6QsZOTsYQaMVkMzJh1aegLgnCBCPd2qt5beqED9fY7A11On3Co5gjLd31IrbOOWFM/FqbN\nZ3BY0kXnqFQqbGEhHNlfzpjJyYEpVBB6GRHu5zRs30rDli9EB2o3aXQ1saJoLXlVB9CoNdyVdBsz\nk6ahU/vfko5mF8dLasgYEQfA6AkJ4mciCNfhquEuSZIaeAMYDrQCT8qyfPQy5/0fUCvL8otdUmUX\na1vC12Qm7rnvig7ULqQoCl9V5LGq+CMcHifJoQl8Z8IjGF3Wi85Tq1Xkbj9OdIyV6FgrarXYy1QQ\nrkdHd+6zAb0syxMkSRoLvHrusTaSJD0DDAW2dEmFXeyiDtRnnkUf3S/QJQWtakcty+SVyHUlGDR6\n7k+dxeT48cTYbFRXN9Hc1IrPp2C1GTGG6JjzrdFYQsUHrSDciI7C/RbgUwBZlndLkpTd/qAkSROA\nMcDbQFqXVNiFRAdq9/D6vGw+vYN1pRtx+9xkRqaxQJpDhDH8ovOOytWUFlYza9FIVCoVVpuYkCQI\nN6qjcA8FGtt97ZUkSS3Lsk+SpP7AvwJzgAe7qsCu1NaBmjNGdKB2kVNNZ1hSuIJTTWew6MwsTr+f\nrH4j2trPW1vcbecOHR2P2aIPVKmCEFQ6CvdGoH1jqFqWZd+5f88HooCPgVjAJElSgSzLf+v8Mjvf\nRR2ojz4hOus6mcvr5uNjn7Hp1DZ8io+xsVnMHXIPFt2FjTIURWHtsgPcdk8G4dEm1GoVg9NEs5gg\ndIaOwn0ncC+wXJKkccDB8wdkWX4NeA1AkqRHgLRrCfboaGtHp3S5JrmI4iV/R2uxMOyln2CMDcyW\naz3hWnSFw5WFvJ23lEp7Nf3MkTydvYjhseltx9tPPrpr7jCaGlqC9lrcCHEtLhDX4sZ1FO6rgRmS\nJO089/VjkiQ9BFhkWX7nG+de0+Ie1dVN11li5/I01HPiv36F4vUS89Q/0aQx0xSAmqKjrQG/Fp3N\n4XawqmQ9u8r3oELF9IGTuXvQTAwafdv/taHOwbYNxdx1/zA0GjVmm4GklKiguxY3KhjfFzdKXIsL\nbuRD7qrhLsuyAjz7jYeLLnPeX6/7lQPgog7UeaIDtbMoisK+6kN8ULSGJpedeEt/FqXNJzF04CXn\nhoaFoDdoOVvdTHSsuCsThK7SpyYxVb3frgP1DtGB2hnqWup5v2gNh2qOoFVrmTXoTqYnTEaj1rSd\nc6yoBkVRGCRFo1KpuH2OWBdfELpanwn3hh3baNgsOlA7i0/xsePMbj48+jEt3laGhA1iYdo8+pku\nXWjNZNGz6aMCkoZEislIgtBN+kS4O0uPUvWPv4kZqJ2kormKJYUrKG04TojWyMK0eYzvn4Na5Q9u\nRVEolatJTIlEq9UQExfK/EezRLALQjcK+nD3NNRTLmagdgqPz8NnJ7bw6fFNeBQvI6OH8UDqLGyG\n0EvOLSmooqbSztgp/jXX9Yagf6sJQo8S1L9xisdD+Vtv4KmrEx2oN+lYwwmWFK6gvLkSm97KA9Ic\nRkZfuJ6KotBY78QWbkKlUjFpZipej+8qzygIQlcK6nCven8pzuIiLNmiA/VGtXha+aj0U7ae/hIF\nhYlxY5k1+C5MupCLzmtqaGHV3/Yx/9EsrDYjJrOYaSoIgRS04d7WgRo/gNjHRAfqjcivLWRZ4Srq\nWuvpZ4pioTSfIeGD2o4rioLPp6DRqAkNC2HijBTEZRaEniEow91ZWnqhA/U7Yg/U69XksrOieC25\nlftRq9TckTiNO5Kmo9PoLjrvUO4Z6s86mHx7KgBDMmICUa4gCJcRdOHu70B9TXSg3gBFUfi6Yi8r\nSz6i2e0g0TqQRenzibdcfgNqaVgMOz4rwev1odGIkTCC0JMEVbiLDtQbV+M8y3vyKgrOFqFX65g3\n5F6mDrilbXjjeZvWFZA1IZGwCBMGo47p96Zf4RkFQQikoAr36g+WiQ7U6+RTfGw5tYOPSjfg8rlJ\nj0jlIWkukSERlz2/X6yVI/vLmTBtcDdXKgjC9QiacHfX1lD/xSb0cXGiA/UanbGXs6RgBSeaTmHW\nmXgobR45MaMuunZOh4vi/CqG5wwAYGhWfKDKFQThOgRNuDcf8q9GHDZ1muhA7YDb6+aT45v47OQW\nfIqPnJjRzBtyD1a95ZJztVoNh/JOExVjIS4hTHxoCkIvEXThbho2PMCV9GzFdUdZKq+kylFDuCGM\nh9LmkRkpXXROc1MrLpeH8EgzOr2G2YtGYRI7JAlCrxIU4e5zu3EUHEEXGytGx1yBw+1kzdGP2Vm2\nGxUqbh0wkXsG3Y5Re+lfOaeP13FgzynmPZLlX3PdKv4SEoTeJijC3Vkko7hcmIeNCHQpPdL+6sN8\nIK+mwdVEnDmWhWnzSbYlXHSO0+HCGKJDpVKROjQGQ4gWtVo0wQhCbxUU4d58+BAA5qHDAlxJz1Lf\n2sDyog/ZX30YrUrDvYNu57aEKWjVl/7YN6zOJ2NkHKmZMahUKpJSArP1oCAInSM4wv3QAVR6PSGp\nUscn9wE+xceusj2sProep6eFwbZkFqbNI9Z8cZNV+71Mb5mewtnq5kCUKwhCF+j14e6qrsJdUYF5\nxEjUOl3H3xDkKh3VLCtcSXF9KUaNkQXSXG6JG3PJZCR7Ywsb1uQz66GRaHUaomOtYts7QQgivT7c\nHedGyZj7+CgZr8/LZye38snxz/H4PAyPyuRBaTZhBttlz7eEGomMtlBTaSd2wOXPEQSh9+r14d7W\n3t6Hw/1E4ymWFK7gjL2cUL2VB1JnMzJ66CVj0k8crcXZ7CJtuH+tmKl3imYsQQhWvTrcfW4XjsIC\n9P3j0EX2vQ7AVq+LdaUb2HxqBwoKE/qPYU7KXZh0psueH2ozsn1jMSkZ/dBqNZc9RxCE4NCrw90p\nnx8C2ffu2o/Uyrwnr6K2pY7okEgWps0jNTzlkvOKj1SSODgSvUFLeJSZB5/MEcEuCH1Arw735j7Y\n3m53N7Oy+CO+rtiLWqVmZuKt3Jl0G3rN5TuTz5yop+J0A5Nm+tdc1+lEsAtCX9C7w/3wQVQGI8aU\nIYEupcspikJu5X5WFK/F7m4mwRrPwrT7GWiNu+S8uhoHEdFmACZMG0xriycQJQuCEEC9NtxdlZW4\nKysxjxwV9EMga511vFe0iiO1Mjq1jjkpd3PrgIlo1JfehTuaXXy4dB+zF40iPMqM3qBFb+i1P2ZB\nEG5Qr/2tbz58vkkmeJcc8Ck+tp7+krWln+LyukgLH8JDaXOJCom86DxFUfB6fWi1GswWA7felYZW\nNL8IQp/We8P90PkhkMG55ECZvYIlhSs43ngSkzaEB9MfYGxs1mWX3C04UM6ZE/XMmJUBQNKQvjdy\nSBCEi/XKcPe5XDjlAvTxA9BFRHb8Db2I2+dhw/FNbDyxBa/iJavfCOan3keo/sqzR4dkxlBZ1ojH\n7RV37IIgAL003J1yIYrbHXQLhZXUH2Np4UoqHVWEGWwskOYwLCrjsuduXl/I0Kx4omOt6HQabr0r\nrZurFQShJ+uV4d586AAQPEMgnZ4WPjz6CdvP7EKFiikDJnDfoDswao1X/J74xDAKD5aL9WAEQbis\nXhfuiqLQfOggaqORkCAYAnmwOp/3i9ZQ39pArDmGRWnzGWRLvOS8Fqeb/L1nGD0hEZVKxZDMGIZk\nxgSgYkEQeoNeF+7uykrc1dVYRmeh0va68tvUOxv44+Gl7Ks6iEal4a7kGcxMvBXdZdZaB9DpNRyV\nq4mKtZI4OFLsZSoIwlX1unRsGwI5tHc2ySiKwq7yXNYcXUez20lyaCKL0ufT33zpXbi9qZUWh4uo\nGCsajZp7F4zAGBLcY/oFQegcvS/ce/FG2FWOGpbJqyiqK8GoNfBg6mwmxo+7ZK3186ormvhyUwkP\nPJGDTqchxCQ2qRYE4dr0qnD3tbbilAvRDxiILjw80OVcM6/PyxentrP+2EbcPg9DI9P59oTFKM2X\n3oU7ml2EmPx7mSYPiUKnU6PVXj78BUEQrqRXhbujsADF4+lVo2RONp1macEKTtnLsOosfCv9Pkb3\nG0GUKZTq5qZLzt+8vpCEwREMyxoAwICkiO4uWRCEIHDVcJckSQ28AQwHWoEnZVk+2u74Q8D3AQ9w\nCPi2LMtKVxV7YcmBnh/uLq+L9cc+44tT2/EpPsb1z2Zuyj2YL7PWevu9TCdMT6HyTEN3lysIQpDp\n6O/92YBeluUJwE+AV88fkCQpBPgPYKosyxMBG3BPVxXaNgQyJISQQYO76mU6ReHZYv5z9//w+cmt\nRBjC+O7Ip/hW+gOXDXanw8UHf8qltcUNQHikqW2nJEEQhBvVUbPMLcCnALIs75YkKbvdsRZgvCzL\nLe2ey9n5Jfq5K8rx1NRgycrusUMgm90OVpWs46vyXFSouC1hCncnz0CvuXJHaIhJz8DkCGoq7cQn\n9p5+BEEQeraOUjIUaGz3tVeSJLUsy75zzS/VAJIkfRcwy7L8eRfV2W5jjp63CqSiKOytOsjyog9p\nctsZYIljUdp8EkIHXPb8U8fOUlpYzaC0aMC/5rogdKa9e3P58MNV/Pzn/3XDz/Huu2/z+ecbiIry\nv08bGxuYPn0mDz/8+HU/V2npUd566zVaWlpwOh2MG3cLTzzxDL/4xcuMGpXF3Xff13bu++8vobGx\nkZ/+9MeXfa7//u9fcuTIIf70pyVtj33nO0/z4x//lISEJABaW1tZvPh+li9fC8CHH67is88+RaVS\n4fF4ePrpbzNqVNYV6127djVr165Go9HwyCNPMGHCxIuOFxfL/OY3r6DVahk4MIGf/OQlVCoV//jH\nX9i0aSNms4WFCx++5Pu6U0fh3gi0n9+ulmXZd/6Lc23yvwZSgHnX8oLR0Tc2Xb5SPgJAwpTx6CN6\nzpT7GsdZ/pT3Hnllh9BpdCwaPoe7peloL7PW+nl6rYa/vrGLUWMSxFrr59zo+yIYdca1CA83YzTq\nbuq5LBYjTz31JA8++CAALpeLu+++m8ce+xYREdfe0d/Y2MgvfvESf/jDH0hISMDn8/H973+fTZvW\n8/DDi/jf//1fHn10Udv5n3/+KW+88QZw6bVwOp0UFBxCklI5dqyAMWPGAKDXa4mIsLSd39qqR61W\nER1tZf369Rw6tJelS/+BRqPh9OnTLF68mDVr1hAWFnZJvdXV1axZs5xVq1bR2trKQw89xJ13Tkev\nv/AX+M9//hf+3//7PpMnT+b5558nPz+PuLg4tmz5nFWrVgKwYMECbr/9VozGKy8j0pU6SpadwL3A\nckmSxgEHv3H8bfzNM3OutSO1uvrSESId8bW00Jh/BENCIg1eLdzAc3Q2n+Jjx5mv+PDoJ7R4W0kN\nG8xDafPoZ4qirtZxyflF+ZUMSArHZPa/Qb79wlTq6i49ry+Kjrbe0Puit/jgixL2FFZd07kajQqv\nt+NfpZy0fjww7dI9c8+rq2umpcV9yXXds+cr3nnnLfR6PTabjRdffBmz2cyrr/4KWS4gMjKS8vIy\nfvWr39Lc3IrB4Gx7jpqaalpbXdjtHhoayvnlL/+dxkb/H/b//M/PM2hQCuvWrWHVquVYrTZ0Oi3T\np88EYOTILEJCwtue60c/egmdTodWq6W6uoZDh4qJjY2loCCf0NBwdDp/SH+z/o8//oiRI7MZN248\n7777F5KT0wFwu72cPduMxeI/v7W1FZ9Pobq6ib//fQnf+94POXvW//tmMNh4990luN0aNmzYzMGD\n+3n00SfbXmPHjt1kZAyjvt7f4hwbG8/u3ftIS7uwiF9i4mBOnaqgqqqRuroGHA4P+/cfYdiwUTQ0\ntJ77vji++mofmZlDO/x5duRGPqQ7CvfVwAxJknae+/qxcyNkLEAu8DiwDfhCkiSA38myvOa6q+hA\n2xDIHrIKZHlzJUsLV1DacIIQbQiL0u5nfP/sqy4JUFtl51TpWabf638zik2qhe6mKAq//vUrvPnm\nu0RFRbF8+Xv89a/vMmLESJqaGnjnnb9SX1/PggVz2s5///0lfP75BqqqKomO7sdPfvISISEhvPHG\n78nOHsPs2fM5deokr7zy77zyyn+zZMnf+MtflqHT6fje9/4JgJqaGvr3j7+olpCQkLZ/33PPLDZu\n/JiHH36c9es/YvbsuVf8P3z00Rp+/OOfkZiYxH//9y+pqakhKupK+xf4fx9ra2uIj7/49UNDQwEY\nPTqb0aOzLzrmcDgwmy1tX5tMJux2+0XnxMcP4Le//Q1//eu7WCxWRo4cTXl5Gf/4x59xOBy43S4O\nHTrIrFnX1KDRJa4a7ufuxp/9xsNF7f7dLQl1YRXIwLa3u30eNp7YzMbjX+BRvIzqN5z7h8zCZrj0\nU1VRFGoq7W2rNuZMSsLZ7O7ukoUe4IFpKVe9y26vK/+Kqa+vx2w2t4XhyJGjePvtP2Cz2cjM9A8v\nDgsLIzExCQCVSsWCBYuZNWsuslzIyy//lIEDEwAoLS1h375cNm36DICmpkZOnz5NUtIgDAYDAEPP\nLRESGxtLUZF8US1lZWeorq5ixIhR3HHH3Xz/+99mwYLF7N+/lx/84PJt7cePH+PYsVJef/1/z9Wn\nZs2aFTz55D9hMBhwuVxt5zqdzrY6YmP7U1FRwaB2o+x2795FSsoQIiMv/WAwmcw4HBf+qnY4HFit\noRed87vfvcobb/yRpKRkVq1azuuv/5Yf/OAF5s59gB/+8LvExMSSkTEUm+3SZp/u0uOnPvqHQB5C\nbTJhHDQoYHWUNpzgl3t+x8fHPsOsM/P0sEd4cujiywY7QGuLh/XLD1JV7v+zVavVYLUFpu1NEMAf\n3M3NzdTW1gCwb18eCQmJDBqUQn6+v8W1sbGRU6dOtn2PovibiCQpjcWLH+Hll3+KoigkJibzwAML\nee21t/n3f3+FmTPvZMCAAZw8efxck4iPgoJ8AG65ZRK7d3/JmTOnAfB4PLz22m85dsw/ZcZm83+g\n/PnP7zBlyq2o1ZePpY8+WsMzzzzHq6/+nldf/T2/+90brF+/Fo/HQ2pqGlu3ftF27ldf7SQjIxOA\nu+++j7/+9Y94vV4ATp48wa9+9Qs0msvf22ZkZHLw4D5cLhd2u50TJ45d9MHgr9mGyeQf2hwZGYXd\nbqe+vh6Hw8Gbb77L88+/SFVV5SXf1516fG+eq6wMz9larDljUGm6vymjxdPC2tINbDv9JQoKk+LH\nM2vwHYRoQy45V1EUPG4vOr0WY4iO2+7NEOvBCAGhUqnYs2c3Tz75cNtjL7/8C1544Wf87Gc/RqVS\nERoays9+9m+Ehtr46qudPPvs40RERGI0GtGeG27cvqnxnntm8cUXn7FmzUoeeeRxXnnlP1i7djXN\nzc088cQz2GxhLFr0CM899xShoaG0trai1Woxmcz87Gc/59e//k98Ph8Oh4OJEycze/b8tue+7745\n/OhH/8zSpf7OyOLiIt55ZwNPPfVdANxuN5s2beRvf3uv7XtiYmJJSRnCli2bWLz4EX796//i8ccX\nYzAYsNlsvPDCvwAwffpMamtr+Pa3n0Sn0+H1enn55V8QFhbG3r25l7S5R0REMn/+Ap577kl8PoWn\nn34OnU7HsWOlrFq1nB/+8AVeeOFfePnln6LRaNDr9fz4x/9CWFgYJ04c46mnHkar1fHcc/8c0NVb\nVec/mbvDE7/YqPzymfHX9T1nN3xCzfL3iXnsSWy3dO+wosM1Bbwnr6autZ4YUzQL0+aTEpZ8xfOL\n8ispOlzB3Q8M7/CHGuydiNdDXIsLAnEtTp48TnFxEdOnz6ShoZ6HH36QlSvXtwX8tfJ6vSxZ8lce\nfvhxFEXhO995mqeffo4RI0Zed00tLS2sWPEPFi9+suOT+4DoaOt1f0r0+Dv3tvHt3diZ2uSys7zo\nQ/KqDqBWqbkzaTq3J05Dp7n6cruD06KpPNOI2+UVQxyFXqNfv1jefPM1PvhgGT6fl2ef/d51BzuA\nRqPB6XTy+OOL0el0ZGYOvaFgB/B6PTz11FM4nd138xlsenQCeZ1OnMVFGBKT0NpsXf56iqKwuyKP\nVcXraPY4SApNYGHaPOItV14OYMsnMkMy+hGfGI5Go2bSzN6/O5TQtxiNRl555dWOT7wGzzzzHM88\n89xNP4/ZbMFiseB0ir/oblSPDndHwRHwertlobAaZy3LCldRWFeMXqNn/pD7mDJgwhXXWj8veUgU\nRfmVYukAQRB6lJ4d7t2wCqTX52Xz6R2sK92I2+cmI1JiQepcIkMuH9auVg97d51kzORk1GoViSmR\nJAwWy/IKgtCz9Nhwb1sF0mzGmNw1QyBPNZWxtHA5J5vOYNGZWZQ2n+yYkVftDNXpNVScbqBUriYl\nvR+A2M9UEIQep8eGu+vMaTx1dVjHjEN1hXGvN/zcXjefHP+cz09uxaf4GBM7mnkp92LRmy97vr2p\nFXtjC7HxNlQqFXfOHyo6TAVB6NF6bEJdWAWyc5tkiuqOsrRwBdXOWiKM4TwkzSUjUrrq9zTWOfl8\n7REefHIMBqMWg1FsUi0IQs/Ws8NdpcI09OYX3QFwuB2sLvmYL8u/RoWKaQMncXfyTIxaw+Vfv6kV\no0mHRqMmLiGMGbMz0RvEejBC73HmzGnefPP3VFdXYzQaMRgMPPvs90juombOs2dr+fOf/8gPf/jC\ndX3fu+++zVdffclbb/0JzbmJik8//Sivv/57Dh8u4l//9UWSkwehKAput5vnn/8JQ4Zc/Ybs0UcX\nMnz4CH7wgwu13Hff7axdu6Ht66+++pIvvviMn/70ZVpbW3nnnTcpKMhHpVIREhLCj370U/r1i7ns\n8/t8Pl599ZccPVqCTqfjJz95ifj4i5f4fv/9Jaxb9yFhYf7+ux//+Gf07x/HK6/8nIqKClwuF488\n8gQTJ06+rut1rXpkuHsdDpwlxRiTktF+Y02H66UoCvurD/NB0RoaXU3EmWNZlD6fpNCEq37fzk0l\nRESbyb4lCYD+A7p+KKYQnFaVrGNf1aFrOlejVuH1dTy2e1S/YcxNufLGZy0tLbz44g954YWX2lYl\nLCjI53/+51e89trb11b4dYqIiLzuYD+voqKcv//9z20zRVUqVVtfVnb2GP7t3/4TuLCq5a9//dsr\nPtfBg/sZPDiFvXtzcTgcbcsEfLNrrH1f2e9//ypJSYP4znf+GYBt27bwr//6Im+99afLvsb27Vtw\nu9289dafyM8/zOuv//aS4aRFRYW89NK/k5qa1vbYxx9/RFhYOC+99B80Njby2GML+1a4OwrywefD\ndOuoZmEAABL7SURBVJMTl+pbG/hAXsOBmny0ai33DrqDGQlT0FxhrXWfz9e2rsWEaYM5dazupl5f\nEAJl585tZGWNuWi52fT0zLZgLy0t4fXX/xev10dDQz3PP/8Thg4dftHd7csvv8js2fOJjIzilVd+\njkajRVEUXn75F+h0el5++UUURcHlcvH88y9isVj4t3/7GW+//Wc2b/6c1atX4PF4UP3/9s41Lqqq\n7cPXDAwjR0UFUSxNja2GhwwVNUtTH8+HVMrKzLQgNej1fE6z1HrUCu2oWemrT75lZkkeMktLMxO1\nyMql8KCiJComOMN5Zt4PG4YZgQFLBhjX9fv5YfZae6+1bzf/Wfuetf5Lo2HJkmUkJSWyceN6PDx0\npKaet278odFoePTRMcTFbaVbt+4lRuW2q+gzMzPL9ZKPi/ucnj17ExjYgB074hgx4qFS6xVdt6Cg\ngP37v2P69DnWsvvu60H79h0AdQQeHHybnQgnJPxC585dAbjrrlBOnPijxPWFOMH69R9w5Uo6Xbrc\ny+OPj6Vnz9706NGrsH2z9U2lMqiW4v5Pd10yW8wcSP2JrYnbyTHl0KLOHTyqjKCBd2CZ5+TlFvDp\nuiMMfbQ9Xj56fPxq0aqd3MtU8s8Z3mKQw1G2LTfLfuDPP1PtbG5nz56KwWAgPf0ysbFvk5yczLPP\n/g/NmrVg9+6dfPnlNkJD2143ulU/xMf/ROvWbZgwIZqEhJ8xGAykpf1J7dp1mDfvBU6fTiYnJxtf\n32ITvXPnUli27HX0+losW7aEQ4d+JCAggLS0C6xfv4m8vDyGDetn3dXJ09OTGTPmsnjxC6xZs87u\nXo4ejSc6Oor8/HwSE0+ydOnyMu/baDSQkPAzM2fOo0mTpsyZM61McS8auWdkXKVevXolyotsgR9+\n+LESZVlZRry9iydgaLVau8EhQO/efRk+PAIvL2/mzJnGDz/st+7MlJVlZP78WURGTizzXv4p1U7c\nLRYLxuO/4ubjS62mTW/4/DTjRTae+JSkjGRqudXiEWU4XRt1KncxkofenZDQIC5fNHK7T+l5eImk\nphAYGIQo3L0MsKYMoqKexGQyUb9+AB9+uBa9Xl8oVD6lXMWCRqNh0KChbNy4jqlTY/Dx8SYqahLh\n4d1ISUlh9uypuLu7M2bMeLsRdp06/rz00kI8PT05e/aM1f63efPmaLVa628AtrRrdzdhYZ1Ys+Zt\ntfXC63XoEGbdLvDs2TM888w4tm7dYbczUhFffbUTs9nMjBmTAfV3gCNHDnPPPR3RXKcBWVlGatWq\nRe3adbh2zVDKtXbwwAN9SrViuN4W2GKxlHCzjIgYZY1rly73cvLkCbp2vZe0tAvMnTuD4cMj6N27\nb8mw3ySqneVv3rkUTFev4hUaekNTIAvMBew8vYclP71GUkYy7QJCmR8+lXuDw8sU9vNn/iL+wGnr\n53u6NuH2ZnJBkqTm0737/cTH/8Rvvx23Hjt3LoVLly4CGmJjlzN+fBRz5y6kWbMWdimK7Oxs8vPz\nSU7+LxaLhe+/30e7dncTG/sWPXr0YsOGdRw7doR69erz6qtvMGbMOFavftM6EjYaDbz//moWLVrK\nzJnz0Ov1NsJfck2I7ZdCZOTEQnvglFLvy9+//JTMv//9utUW+LnnprNly8cANGzYiKNH4611Dx06\nSKtWd+Hu7k7nzuFs3lzsOPnNN1/zySebyvTYadu2HT/+qO5hdPz4rzRvbu/XbzAYePzxh8nOzsZi\nsXDkyGFatmzNlSvpTJnyLBMnxjBgwGCH9/JPqXYj978zBfJ05lk2/rGZVOMFanv48lDIMNoHlp+v\nrxvgzb6dJwntEEwtTzm9UeI6eHp68sorr/H226tIT7+MyWTCzc2NmJgpBAUF0bdvf+bPn4mvrx8B\nAYFkZmYAEBHxCFFRY2nUKJigoEZoNBpatmzF4sUL0el0mM1mYmKm0KBBEAsWzGHr1s2YTCaefPJp\nQE11eHv70KZNOyIjx6r7GPjWJj39Mg0bNrpuwZ/Gek7RcQ8PD2bPXsCECeOsx4vSMlqtG1lZRqKj\nJ+Ph4cGOHXEA9O+vpryEOAFA06bFzq3339+TVate5dKli8ycOY8VK17m3XffxGw2Exralr59BwAQ\nHT2ZVateY8KEcYBqh7xkyTKg9Jz7fff15PDhQ4X1YfbsBQDs3r2T7Oxshgx5kKioScTERKHTeRAW\n1onw8K68/vpyDAYDH3ywhg8+WAPA8uUrS7zF3AyqneVvyitLyE48RfPXVuHmU9qrYjE5BbnEJe9i\nb8oBLFjo1qgTw5oPxEtX0mu9iJO/pdGwcW3rxhkmkxk3N+e/wEib22JkLIqRsSimvFgkJSVy4sTv\nDBw4xIm9qhpqvOWvKctIdlIite5oVq6w/5Yu2CS2cCXnLwI96/NIyxGE+Je/64khM4f9X1+k/wh1\nZF8Vwi6RSP45fn5+t4Sw/12qlbhn/a5OgXSUkjHkGdl86gsOpx1Dq9HSt8kD9GvaC48yvNYtFgtp\nqZkEBavz1Nt3vo0sQ16pdSUSSc0hIKDs2W+SaibuxoSy8+0Wi4XDacf49NQ2DPlGbvdtzGMtR9LY\nt5HDaxbkm9j9+e/0HKDQuGldtFotPn5yL1OJROLaVBtxt5jNGH/7FTdfP/S3N7ErS8++wkdiC39c\nOYmHVsfwFoPo0bhbmYuRLBYLebkF6Gvp0Hm402doa7y85V6mEonk1qHaiHtuyllMGRn4delmnQJp\ntpjZe+4A25J2kmfOp1XdEEYpw6nv6Xg61OnEdI4dPMuw0Xej1WqsKRmJRCK5Vag24p554HsAvAv3\nXDxv+JONJzZzJjMFb3cvRinD6RTUoUzvdIvFYi1r2qIeaeczyMstkFMcJRLJLUm1mCpScC2TjP3f\n4163Hvq2oWxL2snLh2M5k5lCWIP2zA+fRueG9zjcFGP/7lMkn7wEqPNmw3s0l8IuuWU5ejSeBQuK\nvVK+/fZrxox5mLS0CyxevJC5c6fb1R86VF0puX37NiIihtitvlywYDbHjh1x2N6ePbvp06c7ly9f\nth5bu/Zdtm791K5eZORYLly4AMAvvxxj8uRJREdH8fTTY/jss80O2zh+/FciI8cyYcJ46xxxWzZs\n+JDo6Ciio6MYO/ZR6z0BmEwm5s2bwaFDBx224UpUi5H71T1fY8nLo6B7R5YeXcXFrMv46+swSnmQ\n0PqtKnSNFq0C+f3nP7kjJKCSeyuR3BiXPtnEtfjDFap7xk2LyWQut55vWEcCIkZV6Jq7d+9k06aN\nxMa+g7+/aj+bkPALu3Ztty7isSUnJ4eVK1cwa9b8wiOacncb27btM0aOHMUXX2xh3LhI9axSzik6\ndv78OWJjl7NixRv4+/uTm5tLTMwzBAc3plOn8FLbWLFiKYsXL6NRo2CmT3+OU6eEncnY6NFjGT16\nLAAzZkxm0qTnrG299NLzXLp0iSFDhju8D1eiykfu5pwcrn7zNQWeOt7UH+FSVjo9GndjXucpDoU9\nP8/E/t2nrH8IDW+rQ6/BFfsikEhcnSIR3bnzSz7++CNiY9+2CrtGoyEqahJr175baEdgf17//gM5\ncyaZH37Ybz3uaLFjaup5DAYDjz32BLt2bcdkMpXTOwu7dm2nX79B1j7p9Xpee+0NwsI6kZmZUeLN\nwmg0kJ+fT6NGqhlap05dOHz4p1Kvvm/fN/j5+dGxY2cAsrOzmTXreTp0CHN4H65GlY/cf9+xCY+s\nLOLbeBNQO4jHWo7kjtpNyj3PXacl82oOJ4+nSfdGSbUmIGJUhUfZN2uFqsViISHhZy5fvsS1a9co\nKCi4rp1AnnpqAkuXvsirr66yO0+rdWPu3BeYNi2G0ArYbsfFfc6AAYPx8fEhNLQNe/d+Q69efRyc\noSE9/TIhIfbWvl5eqsuin19tFi9eZldmNBqt5WpdL1JTz5d69Q0bPmThwiXWzy1a3FnuPbgiVTZy\nz8jN5L2f15H77Xfku2lo0HsAszo+51DYDddyOXda9VjXaDT0GdqKlm2DnNVliaRGUa9efV5//S0i\nIkaxaNF8u1GrRqPhX//qh5eXV6m57saNbyMiYhTLl7/ssA2TycRXX+1g7949TJ0aw9mzZ61GXXq9\nnvz8fLv62dlZ6PV6goIakpaWZld26tRJTp0Spbbj7W3vwmg0GvHx8S1RLzn5v/j4+JbYFelWxOni\nbrFYOJB6iBcPLScnPh7fLDPe93al710Dcdc6fpHINuax+4vfyTKqK0x1Hu7l5gIlkluV4ODG6HQ6\nRox4CJ3OnXXr1lrLioR+2rTZfPTR/9oJZxEjRjxMRkYGR4/Gl/l3dvDgAVq3DmXlyndYsWIla9as\n48qVdJKSEgkJacn+/d9Z0zTnz58jLy8ff39/+vTpR1zcVq5evQpAVlYWy5cvJT09vdR2vL190Onc\nOX/+nLqg8fCPtG9/d4l68fE/0aVLtxsLlIviVHE36wzEHnuX/5z4FIvZzAOJbuDmRvCAsn/kMFzL\nJT9ffTgCgnwZGNEGTy85C0YicYSt0yKoroVffPGZ1fK2qKxOnTrExEwhNze31PPmzHnebvQdHR1l\n105c3NYSP8oOHjyMLVs+pmPHzrRt247x4x9n0qSnWbx4IfPnLwIgKKghEyfGMHfudOsMl/79BxEe\n3rXUnDvAtGlzWLRoPpGRTxAS0pJWre4CYMqUZ61pp5SUM9a8fFlxuVVwqivkQx9NsqA106Z+K4bl\n3Enm6vfw69KNoPFPl3nO3h0Cnc6Nbr1blFmnJiLd/4qRsSimusdi5coVxMRMdUpb1T0WzuTvuEI6\nNy2jNfNQyDAiQ58gb89eAPz7lZyKZTsVLLxHM+oFepeoI5FInM+oUaOruguSCuL0nPsdfreTk3iK\nnKREvNu1Rx9s/wpVkG/i/947TObVbABqeepo2VbOhpFIqgOBgQ2quguSClIls2X+2vElAHX7DyxR\n5q5zo21YY65cNjq7WxKJROIyOF3czakXMP6agOedIXgWzj+9cC6DH/YkWuuE3hNM0xb1nd01iUQi\ncRkczj1UFEULvAW0BXKBp4QQSTblg4H5QAHwvhDivXJb3Kd6O/j3L8611w3wZt+ukxgNuXj73Py9\nBCUSieRWo7yR+zDAQwjRFZgFrCgqUBRFB7wK9AHuByIVRXG4NcqSu58h78gxPIIbk+rW0Jp68dC7\nE/FkmBR2iUQiuUmUJ+7dgJ0AQohDQJhNWSsgUQiRIYTIB/YD95W8hE1je+PBbKZuvwFWbxhrmfbW\nmX8qkUgklU153jJ+QKbNZ5OiKFohhLmwLMOm7BrgcFeMP75LoH7devh27ERrNzeatqj3tzotkUgk\nEseUJ+6ZgK2BQ5GwgyrstmW+wF+OLvaHf0fubumNxl1t1ttXpmEkEomkMihP3A8Ag4FPFEUJBxJs\nyk4AdyqK4g8YUVMyy0peophpq0bL3IsNAQEljY9uVWQsipGxKEbG4u/j0H5AURQNxbNlAJ4E7gF8\nhBBrFEUZBDyPmrtfK4R4u5L7K5FIJJIK4FRvGYlEIpE4hyrfiUkikUgkNx8p7hKJROKCSHGXSCQS\nF0SKu0QikbgglbJBdqV40tRQKhCLR4DnUGPxKzBRCOGSv3KXFwubequBdCHEbCd30WlU4LnoiGr3\noQEuAKOFELlV0dfKpgKxeAyYAphQ9eKdKumok1AUpTPwshCi53XHb0g3K2vkflM9aWo4jmLhCbwI\n9BBC3Iu6wndQlfTSOZQZiyIURYkCQgGX/IKzwdFzoQFWA2OFEN1RLUDK3jm+5lPec7EM6IVqhzJV\nURSHK+FrMoqizADWAPrrjt+wblaWuN9UT5oajqNY5ABdhBA5hZ/dgWznds+pOIoFiqJ0BToB76KO\nWF0ZR7EIAdKBKYqi7AXqCiFOOr2HzsPhc4G6eLIO4In6XLjyF38iMJySz/+Ne3lVTv9K96SxKbsh\nT5oaTpmxEEJYhBCXABRFiQa8hRBfV0EfnUWZsVAUpSHqgrhncX1hB8d/I/WBrsAqoDfQS1GUnrgu\njmIB8BtwBDgObBNC2NZ1KYQQW1DTLtdzw7pZWeJ+Uz1pajiOYoGiKFpFUZajvnaOcHbnnIyjWIxE\nFbXtwEzgUUVRxji5f87EUSzSUUdpQghRgDqqvX4060qUGQtFUdoCA1DTUk2BBoqijHR6D6ueG9bN\nyhL3A6j/ITjypFEUxQP11eJgJfWjOuAoFqCmIPTAgzbpGVelzFgIIVYJIcIKf0R6GfiPEGJ91XTT\nKTh6Lv4L+CiK0rzwc3fUUaur4igWGaipytxCwb+ImqK51bhh3awU+wHpSVOMo1gA8YX/vrM5JVYI\nsdWpnXQS5T0XNvWeABQhxBzn99I5VOBvpOhLTgMcEEJMrpqeVj4ViEUUMA7IQ81JP134RuOSKIrS\nFHVw07VwNt3f0k3pLSORSCQuiFzEJJFIJC6IFHeJRCJxQaS4SyQSiQsixV0ikUhcECnuEolE4oJI\ncZdIJBIXRIq7RCKRuCBS3CUSicQF+X9k3is/uBhZ7gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e20e350>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(fpr_lr,tpr_lr,label=\"LogRegCV, AUC: %.2f\" % roc_auc_lr)\n",
    "plt.plot(fpr_nb,tpr_nb,label=\"GaussianNB, AUC: %.2f\" % roc_auc_nb)\n",
    "plt.plot(fpr_knn,tpr_knn,label=\"KNN, AUC: %.2f\" % roc_auc_knn)\n",
    "plt.plot(np.linspace(0,1,100),np.linspace(0,1,100),\":\")\n",
    "\n",
    "plt.legend(loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information on choosing an 'optimal threshold' \n",
    "\n",
    "http://stats.stackexchange.com/questions/123124/how-to-determine-the-optimal-threshold-for-a-classifier-and-generate-roc-curve\n",
    "\n",
    "The 'stages' if you will, will be to tune each model individually (if they have tuning parameters such as C or lambda). Then, once you have an optimal model, you compare them to each other on accuracy, ROC curves, and AUC.\n",
    "\n",
    "The optimal threshold once you have an optimal model is more of a subjective area. For instance, if you choose a higher threshold of acceptance (i.e. higher probability required to predict a \"Yes\" or \"true\"), there may be higher financial costs corresponding with this decision.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Effect of Outliers\n",
    "\n",
    "There's a commented code block above that adds a single row that is a massive outlier (data entry mistake).\n",
    "\n",
    "Let's see how our models perform with that outlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}