06_Lab_4_SVM_Classification_Evaluation_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introductory applied machine learning (INFR10069) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 4: Support Vector Machine (SVM) Classification and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we initially re-examine the spam filtering problem from Lab 2. This time, we train a Logistic Regression model and a linear Support Vector Machine for the spam or non-spam classification task. In the second part of the lab we examine classification evaluation by using a K-nearest neighbour classifier.\n",
    "\n",
    "\n",
    "All the datasets that you will need for this lab are located at the `./datasets` directory which is adjacent to this file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Import packages\n",
    "from __future__ import division, print_function # Imports from __future__ since we're running Python 2\n",
    "import os\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC, SVC\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import confusion_matrix, log_loss\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Spam filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.1 ==========\n",
    "Download the `spambase_binary.csv` dataset and load it into a pandas DataFrame structure called `spambase`. Display the number of instances and attributes and the first 5 samples. Remember that the attributes have been binarised. The instances have also been shuffled (i.e. their order has been randomised). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of instances: 4601, number of attributes: 55\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_freq_make_binarized</th>\n",
       "      <th>word_freq_address_binarized</th>\n",
       "      <th>word_freq_all_binarized</th>\n",
       "      <th>word_freq_3d_binarized</th>\n",
       "      <th>word_freq_our_binarized</th>\n",
       "      <th>word_freq_over_binarized</th>\n",
       "      <th>word_freq_remove_binarized</th>\n",
       "      <th>word_freq_internet_binarized</th>\n",
       "      <th>word_freq_order_binarized</th>\n",
       "      <th>word_freq_mail_binarized</th>\n",
       "      <th>...</th>\n",
       "      <th>word_freq_edu_binarized</th>\n",
       "      <th>word_freq_table_binarized</th>\n",
       "      <th>word_freq_conference_binarized</th>\n",
       "      <th>char_freq_;_binarized</th>\n",
       "      <th>char_freq_(_binarized</th>\n",
       "      <th>char_freq_[_binarized</th>\n",
       "      <th>char_freq_!_binarized</th>\n",
       "      <th>char_freq_$_binarized</th>\n",
       "      <th>char_freq_#_binarized</th>\n",
       "      <th>is_spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 55 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   word_freq_make_binarized  word_freq_address_binarized  \\\n",
       "0                         0                            1   \n",
       "1                         0                            0   \n",
       "2                         0                            0   \n",
       "3                         0                            0   \n",
       "4                         0                            0   \n",
       "\n",
       "   word_freq_all_binarized  word_freq_3d_binarized  word_freq_our_binarized  \\\n",
       "0                        0                       0                        1   \n",
       "1                        0                       0                        0   \n",
       "2                        1                       0                        0   \n",
       "3                        1                       0                        1   \n",
       "4                        0                       0                        1   \n",
       "\n",
       "   word_freq_over_binarized  word_freq_remove_binarized  \\\n",
       "0                         0                           1   \n",
       "1                         0                           0   \n",
       "2                         0                           0   \n",
       "3                         0                           0   \n",
       "4                         0                           0   \n",
       "\n",
       "   word_freq_internet_binarized  word_freq_order_binarized  \\\n",
       "0                             1                          0   \n",
       "1                             0                          0   \n",
       "2                             0                          1   \n",
       "3                             0                          0   \n",
       "4                             0                          0   \n",
       "\n",
       "   word_freq_mail_binarized   ...     word_freq_edu_binarized  \\\n",
       "0                         1   ...                           0   \n",
       "1                         0   ...                           1   \n",
       "2                         0   ...                           0   \n",
       "3                         0   ...                           0   \n",
       "4                         0   ...                           0   \n",
       "\n",
       "   word_freq_table_binarized  word_freq_conference_binarized  \\\n",
       "0                          0                               0   \n",
       "1                          0                               0   \n",
       "2                          0                               0   \n",
       "3                          0                               0   \n",
       "4                          0                               0   \n",
       "\n",
       "   char_freq_;_binarized  char_freq_(_binarized  char_freq_[_binarized  \\\n",
       "0                      0                      1                      1   \n",
       "1                      0                      0                      0   \n",
       "2                      0                      0                      0   \n",
       "3                      0                      0                      0   \n",
       "4                      0                      1                      0   \n",
       "\n",
       "   char_freq_!_binarized  char_freq_$_binarized  char_freq_#_binarized  \\\n",
       "0                      1                      1                      0   \n",
       "1                      1                      0                      0   \n",
       "2                      0                      0                      0   \n",
       "3                      1                      0                      0   \n",
       "4                      1                      1                      0   \n",
       "\n",
       "   is_spam  \n",
       "0        1  \n",
       "1        0  \n",
       "2        0  \n",
       "3        1  \n",
       "4        0  \n",
       "\n",
       "[5 rows x 55 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Your code goes here\n",
    "data_path = os.path.join(os.getcwd(), 'datasets', 'spambase_binary.csv')\n",
    "spambase = pd.read_csv(data_path, delimiter = ',')\n",
    "print('Number of instances: {}, number of attributes: {}'.format(spambase.shape[0], spambase.shape[1]))\n",
    "spambase.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.2 ==========\n",
    "Use [Hold-out validation](http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.train_test_split.html) to split the dataset into training and testing subsets. Use 90% of the data for training and the remaining 10% for testing. Store your data into matrices `X_train`, `X_test`, `y_train`, `y_test`. Make sure you don't include the target variable `is_spam` in the input features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "X_train, X_test, y_train, y_test = train_test_split(spambase.drop(\"is_spam\", axis=1), spambase[\"is_spam\"], \\\n",
    "                                                    train_size = 0.9, test_size=0.1, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.3 ==========\n",
    "Train a [`LogisticRegression`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) classifier by using training data. Use the `lbfgs` solver and default settings for the other parameters. Report the classification accuracy on both the training and test sets. Does your classifier generalise well on unseen data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification accuracy on training set: 0.935\n",
      "Classification accuracy on test set: 0.928\n"
     ]
    }
   ],
   "source": [
    "# Your code goes here\n",
    "lr = LogisticRegression(solver='lbfgs')\n",
    "lr.fit(X_train, y_train)\n",
    "print('Classification accuracy on training set: {:.3f}'.format(lr.score(X_train, y_train)))\n",
    "print('Classification accuracy on test set: {:.3f}'.format(lr.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "Indeed, the classifier generalises well since the classification accuracy on the test set is comparable to the training classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.4 ==========\n",
    "Print the coefficients for class 1 for the attributes `word_freq_hp_binarized` and `char_freq_`$`_binarized`. Generally, we would expect the string $ to appear in spam, and the string `hp` to appear in non-spam e-mails, as the data was collected from HP Labs. Do the regression coefficients make sense given that class 1 is spam? *Hint: Consider the sigmoid function and how it transforms values into a probability between 0 and 1. Since our attributes are boolean, a positive coefficient can only increase the total sum fed through the sigmoid and thus move the output of the sigmoid towards 1. What can happen if we have continuous, real-valued attributes?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'spambase' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-e0e89f4b7085>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Your code goes here\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mhp_ind\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mspambase\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'word_freq_hp_binarized'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mdollar_ind\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mspambase\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'char_freq_$_binarized'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Coefficient for word_freq_hp_binarized: {:.3f}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcoef_\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhp_ind\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Coefficient for char_freq_$_binarized: {:.3f}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcoef_\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdollar_ind\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'spambase' is not defined"
     ]
    }
   ],
   "source": [
    "# Your code goes here\n",
    "hp_ind = spambase.columns.get_loc('word_freq_hp_binarized')\n",
    "dollar_ind = spambase.columns.get_loc('char_freq_$_binarized')\n",
    "print('Coefficient for word_freq_hp_binarized: {:.3f}'.format(lr.coef_[0, hp_ind]))\n",
    "print('Coefficient for char_freq_$_binarized: {:.3f}'.format(lr.coef_[0, dollar_ind]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "Indeed, the coefficients make sense, since the attribute `word_freq_hp_binarized` has a negative coefficient, meaning that it provides support for `non-spam` classification. On the other hand, `char_freq_`$`_binarized` has a positive coefficient, thus providing support for class 1 (i.e. `spam`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.5 ==========\n",
    "Train a [`LinearSVC`](http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC) (i.e. Linear Support Vector classifier) by using default parameters. Report the classification accuracy on the training and test sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "svc_linear = LinearSVC()\n",
    "svc_linear.fit(X_train, y_train)\n",
    "print('Linear SVC classification accuracy on training set: {:.3f}'.format(svc_linear.score(X_train, y_train)))\n",
    "print('Linear SVC classification accuracy on test set: {:.3f}'.format(svc_linear.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.6 ==========\n",
    "What are the coefficients for the attributes `word_freq_hp_binarized` and `char_freq_`$`_binarized`? Compare these to the ones you found with Logistic Regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "print('Coefficient for word_freq_hp_binarized: {:.3f}'.format(svc_linear.coef_[0, hp_ind]))\n",
    "print('Coefficient for char_freq_$_binarized: {:.3f}'.format(svc_linear.coef_[0, dollar_ind]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "They are consistent. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.7 ==========\n",
    "How does a linear SVM relate to Logistic Regression? *Hint: Consider the classification boundary learnt in each model.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "They both have linear classification boundaries with respect to the inputs. The classification mechanism is, however, fundamentally different. Logistic regression applies a linear transformation to the input and the result is transformed to a probability by using a non-linear function. Support vector machine classification is based on the similiarty of a new data point to a few training instances known as support vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.8 ==========\n",
    "By using the [`SVC`](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC) class train two new support vector classifiers with Gaussian (`rbf`) and polynomial (`poly`) kernels. Again, report classification accuracies on training and test sets and compare with your results from Question 1.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "svc_rbf = SVC(kernel='rbf')\n",
    "svc_rbf.fit(X_train, y_train)\n",
    "svc_poly = SVC(kernel='poly', degree=2)\n",
    "svc_poly.fit(X_train, y_train)\n",
    "print('RBF SVC classification accuracy on training set: {:.3f}'.format(svc_rbf.score(X_train, y_train)))\n",
    "print('RBF SVC classification accuracy on test set: {:.3f}'.format(svc_rbf.score(X_test, y_test)))\n",
    "print('Poly SVC classification accuracy on training set: {:.3f}'.format(svc_poly.score(X_train, y_train)))\n",
    "print('Poly SVC classification accuracy on test set: {:.3f}'.format(svc_poly.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "It appears that there is a very slight benefit to using a radial basis kernel. However, before we can make any solid claims, we need to perform a cross-validation and try different parameters for these kernels c.f. [the sklearn doc examples](http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html#sphx-glr-auto-examples-svm-plot-rbf-parameters-py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Performance assessment\n",
    "We will now look at a few ways of assessing the performance of a classifier. To do so we will introduce a new data set, the [Splice](https://archive.ics.uci.edu/ml/datasets/Molecular+Biology+%28Splice-junction+Gene+Sequences%29) data set. The classification task is to identify `intron` and `exon` boundaries on gene sequences. Read the description at the link above for a brief overview of how this works. The class attribute can take on 3 values: `N`, `IE` and `EI`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.1 ==========\n",
    "Load the `splice_train.csv` and `splice_test.csv` into two separate dataframes. Display the shape and first 10 instances for each dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "data_path_train = os.path.join(os.getcwd(), 'datasets', 'splice_train.csv')\n",
    "splice_train = pd.read_csv(data_path_train, delimiter = ',')\n",
    "print('Number of instances: {}, number of attributes: {}'.format(splice_train.shape[0], splice_train.shape[1]))\n",
    "splice_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_path_test = os.path.join(os.getcwd(), 'datasets', 'splice_test.csv')\n",
    "splice_test = pd.read_csv(data_path_test, delimiter = ',')\n",
    "print('Number of instances: {}, number of attributes: {}'.format(splice_test.shape[0], splice_test.shape[1]))\n",
    "splice_test.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.2 ========== \n",
    "Convert the categorical attributes into numeric ones by using the [`LabelEncoder`](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html) tool. Make sure to not transform the target variable (`class`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "for column_train, column_test in zip(splice_train.drop(\"class\", axis=1).columns, splice_test.drop(\"class\", axis=1).columns):\n",
    "    le = LabelEncoder().fit(splice_train[column_train])\n",
    "    splice_train[column_train] = le.transform(splice_train[column_train])\n",
    "    splice_test[column_test] = le.transform(splice_test[column_test])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.3 ==========\n",
    "Store the training and testing data into numpy arrays `X_train`, `y_train`, `X_test` and `y_test`. Display the shapes of the four arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "X_train = splice_train.drop(\"class\", axis=1).as_matrix()\n",
    "X_test = splice_test.drop(\"class\", axis=1).as_matrix()\n",
    "y_train = splice_train[\"class\"]\n",
    "y_test = splice_test[\"class\"]\n",
    "print('X_train shape: {}'.format(X_train.shape))\n",
    "print('y_train shape: {}'.format(y_train.shape))\n",
    "print('X_train shape: {}'.format(X_test.shape))\n",
    "print('y_test shape: {}'.format(y_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.4 ==========\n",
    "Familiarise yourself with [Nearest Neighbors Classification](http://scikit-learn.org/stable/modules/neighbors.html#classification). Use a [`KNeighborsClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier)\n",
    "by using a single neighbor. Report the classification accuracy on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=1)\n",
    "knn.fit(X_train, y_train)\n",
    "print('KNN classification (k=1) accuracy on training set: {:.3f}'.format( knn.score(X_train, y_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.5 ==========\n",
    "Is the above result meaningful? Why is testing on the training data a particularly bad idea for a 1-nearest neighbour classifier? Do you expect the performance of the classifier on a test set to be as good?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "The above result is not meaningful. The 1-nearest neighbor classifier will classify each instace to the class of the nearest insance in the training set. If we test the classifier on the training set, then each input will be classified correctly since the closest instance in the training set is the instance itself. The performance on unseen data is expected to be much lower."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.6 ==========\n",
    "Now report the classification accuracy on the test set and check your expectations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "print('KNN classification (k=1) accuracy on test set: {:.3f}'.format(knn.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.7 ==========\n",
    "Plot a histogram of the target variable (i.e. `class`) in the test set. *Hint: matplotlib won't allow you to plot a histogram for categorical values. Instead, you can use Pandas bulit-in bar plot tool in conjunction with the [`value_counts`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.value_counts.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "class_counts = splice_test[\"class\"].value_counts()\n",
    "ax = class_counts.plot(kind=\"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.8 ==========\n",
    "What would be the accuracy of the classifier, if all points were labelled as `N`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "print('The baseline classifier (predict always N) would achieve a classification accuracy score of: {:.3f}'.\n",
    "      format(class_counts[\"N\"] / class_counts.values.sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.9 ==========\n",
    "Now we want to explore the effect of the `k` parameter. To do this, train the classifier multiple times, each time setting the KNN option to a different value. Try `5`, `10`, `50`, `100`, `200`, `500`, `1000`, `1500` and `2000` and test the classifier on the test set. How does the k parameter effect the results? *Hint: Consider how well the classifier is generalising to previously unseen data, and how it compares to the base rate again.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "k_list = [5, 10, 50, 100, 200, 500, 1000, 1500, 2000]\n",
    "ca = []\n",
    "for k in k_list:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)\n",
    "    score = knn.score(X_test,y_test)\n",
    "    ca.append(score)\n",
    "    print('Performance on test with {} nearest neighbours: {:.3f}'.format(k, score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.10 ==========\n",
    "Plot the results (k-value on the x-axis and classification accuracy on the y-axis), making sure to mark the axes. Can you conclude anything from observing the plot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "plt.scatter(k_list, ca)\n",
    "plt.plot(k_list,ca, )\n",
    "plt.xlabel('k')\n",
    "plt.ylabel('Classification accuracy on test set')\n",
    "plt.title('K-nearest neighbors classification')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "Increasing K improves performance up to a certain point. Beyond that point, the classifier uses virtually all training samples to classify a new instance and as a results classifies all instances to the dominant class (identically to our baseline model)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.11 ==========\n",
    "Select best value for `k` from Questions 2.9 and 2.10 and plot the normalised confusion matrix on the test set. Then plot the confusion matrix for a 5-nearest neighbor classifier. What do you observe?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "def plot_confusion_matrix(cm, classes=None, title='Confusion matrix'):\n",
    "    \"\"\"Plots a confusion matrix.\"\"\"\n",
    "    if classes is not None:\n",
    "        sns.heatmap(cm, xticklabels=classes, yticklabels=classes, vmin=0., vmax=1., annot=True)\n",
    "    else:\n",
    "        sns.heatmap(cm, vmin=0., vmax=1.)\n",
    "    plt.title(title)\n",
    "    plt.ylabel('True label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for i in [1000, 5]:\n",
    "    knn = KNeighborsClassifier(n_neighbors=i).fit(X_train, y_train)\n",
    "    pred = knn.predict(X_test)\n",
    "    cm = confusion_matrix(y_test, pred)\n",
    "    cm_norm = cm/cm.sum(axis=1)[:, np.newaxis]\n",
    "    plt.figure()\n",
    "    plot_confusion_matrix(cm_norm, classes=knn.classes_, title = str(i) + ' neighbors')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "With 1000 neighbors, the `N` class is almost always predicted correctly, but the other two classes are not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.12 ==========\n",
    "Read about the [logarithimic loss](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html) (or cross-entropy loss) metric which is often used in neural networks. \n",
    "\n",
    "This metric takes as input the true labels and the estimated probability distributions (bernouli or multinomial). It makes sense to use this metric when we are interested not only in the predicted labels, but also in the confidence (i.e. probability) that these are predicted.\n",
    "\n",
    "For instance, think of the situation where you have a single test point and two classifiers. Both classifiers predict the label correctly, however classifier A predicts tha the test point belongs to the class with probability 0.55, whereas classifier B predicts the correct class with probability 0.99. Classification accuracy would be the same for the two classifiers (why?) but the `log_loss`  metric would indicate that classifier B should be favoured.\n",
    "\n",
    "Produce a scatter plot similar to the one in Question 2.10 but this time show `log_loss` on your y axis. Which value for `k` would you pick if `log_loss` was your metric of interest?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "k_list = [5, 10, 50, 100, 200, 500, 1000, 1500, 2000]\n",
    "logloss = []\n",
    "for k in k_list:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)\n",
    "    pred_proba = knn.predict_proba(X_test)\n",
    "    this_logloss = log_loss(y_test, pred_proba)\n",
    "    logloss.append(this_logloss)\n",
    "    print('Performance on test with ', k, ' nearest neighbours: ', this_logloss)\n",
    "plt.scatter(k_list, logloss)\n",
    "plt.xlabel('k')\n",
    "plt.ylabel('Logarithmic loss on test set')\n",
    "plt.title('K-nearest neighbors classification')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "We would pick the 50-neighbours classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.13 ==========\n",
    "\n",
    "Could you use the `log_loss` metric to evaluate the performance of an SVM classifier? Explain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "No, that wouldn't be possible because SVMs are not probabilistic classifiers (i.e. they only yield a classification decision and not a posterior probability distribution). Nevertheless `sklearn` does provide a `predict_proba()` method which returns probability estimates on predictions by using cross-validation."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [iaml]",
   "language": "python",
   "name": "Python [iaml]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}