07_Assignment_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introductory applied machine learning (INFR10069)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3: Object recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Marking Breakdown\n",
    "\n",
    "**70-100%** results/answer correct plus extra achievement at understanding or analysis of results. Clear explanations, evidence of creative or deeper thought will contribute to a higher grade.\n",
    "\n",
    "**60-69%** results/answer correct or nearly correct and well explained.\n",
    "\n",
    "**50-59%** results/answer in right direction but significant errors.\n",
    "\n",
    "**40-49%** some evidence that the student has gained some understanding, but not answered the questions\n",
    "properly.\n",
    "\n",
    "**0-39%** serious error or slack work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mechanics\n",
    "\n",
    "You should produce a Jupyter notebook and a `.txt` file (see Part 2) in answer to this assignment.\n",
    "**You need to submit these two files electronically as described below.**\n",
    "\n",
    "Place your notebook and `.txt` files **only** in a directory called `iamlans` and submit this directory using the submit command on a DICE machine. The format is:\n",
    "\n",
    "`submit iaml 3 iamlans`\n",
    "\n",
    "You can check the status of your submissions with the `show_submissions` command.\n",
    "\n",
    "**Late submissions:** The policy stated in the School of Informatics MSc Degree Guide is that normally you will not be allowed to submit coursework late. See http://www.inf.ed.ac.uk/teaching/years/msc/courseguide10.html#exam for exceptions to this, e.g. in case of serious medical illness or serious personal problems.\n",
    "\n",
    "**Collaboration:** You may discuss the assignment with your colleagues, provided that the writing that you submit is entirely your own. That is, you should NOT borrow actual text or code from other students. We ask that you provide a list of the people who you've had discussions with (if any).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important Instructions\n",
    "\n",
    "1. In the following questions you are asked to run experiments using Python (version 2.7) and the following packages:\n",
    "    * Numpy\n",
    "    * Pandas\n",
    "    * Scikit-learn 0.17\n",
    "    * Matplotlib\n",
    "    * Seaborn\n",
    "\n",
    "2. Before you start make sure you have set up a vitual environment (or conda environment if you are working on your own machine) and the required packages installed. Instructions on how to set-up the working enviornment and install the required packages can be found in `01_Lab_1_Introduction`.\n",
    "\n",
    "3. Wherever you are required to produce code you should use code cells, otherwise you should use markdown cells to report results and explain answers.\n",
    "\n",
    "4. The .csv files that you will be using are located at `./datasets` (the `datasets` directory is adjacent to this file).\n",
    "\n",
    "5. **IMPORTANT:** Keep your answers brief and concise. Most questions can be answered with 2-3 lines of explanation (excluding coding questions), unless stated otherwise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "\n",
    "In this assignment you are asked to import all the packages and modules you will need. Include all required imports and execute the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import division, print_function # Imports from __future__ since we're running Python 2\n",
    "import os\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC, SVC\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, log_loss\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description of the dataset\n",
    "In this assignment our goal is to recognize objects from 19 di\u000b",
    "erent visual classes (e.g. person, dog, cat, car, ...) in realistic scenes. The dataset consists of several thousands photographs harvested from the web. Each object of a relevant class has been manually annotated with a bounding box. Images can contain none, one or multiple objects of each class. We have prepared a [website](http://www.inf.ed.ac.uk/teaching/courses/iaml/2014/assts/asst3/images.html) where you can view the images.\n",
    "\n",
    "Here we will focus on a single classiffication task: you will be required to classify images as to whether or not they contain a person. To save you time and to make the problem manageable with limited computational resources, we have preprocessed the dataset. We will use the [Bag of Visual Words](https://en.wikipedia.org/wiki/Bag-of-words_model_in_computer_vision) representation. That is, each image is represented by a 500 dimensional vector that contains the normalized count for each of 500 diffeerent visual words present in the respective image (a similar representation is used for the spambase dataset, just for real words). Visual words are based on [Scale-invariant feature transforms (SIFT)](https://en.wikipedia.org/wiki/Scale-invariant_feature_transform). SIFT features are essentially local orientation histograms and capture the properties of small image regions. They possess attractive invariance properties which make them well suited for our task (you can read more about SIFT features in [D.Lowe, IJCV 60(2):91- 110, 2004](http://link.springer.com/article/10.1023/B:VISI.0000029664.99615.94), but the details don't matter for the purpose of this assignment). Each SIFT feature is a 128 dimensional vector. From each image many SIFT features are extracted, typically > 2500 per image (features are extracted at regular intervals using a 15 pixel grid and at 4 different scales). To obtain visual words a representative subset of all extracted SIFT features from all images is chosen and clustered with k-means using 500 centres (such use of the k-means algorithm will be discussed in detail during the lecture). These 500 cluster centres form our visual words. The representation of a single image is obtained by first assigning each SIFT feature extracted from the image to the appropriate cluster (i.e. we determine the visual word corresponding to each feature by picking the closest cluster centre). We then count the number of features from that image assigned to each cluster (i.e. we determine how often each visual word is present in the image). This results in a 500 dimensional count vector for each image (one dimension for each visual word). The normalized version of this count vector gives the final representation of the image (normalized means that we divide the count vector by the total number of visual words in the image, i.e. the normalized counts sum to 1 for each image). Our dataset with all images is thus a $N \\times 500$ dimensional matrix where `N` is the number of images. The resulting representation is much more compact and can be used directly to perform classiffication.\n",
    "\n",
    "The full dataset has 520 attributes (dimensions). The first attribute (`imgID`) contains the image ID which allows you to associate a data point with an actual image. The next 500 attributes (`dim1`, ..., `dim500`) correspond to the normalized count vector. The last 19 attributes (`is_class`) indicate the presence of at least one object of a particular class in the image. In most of the experiments (unless explicitly noted otherwise) you will be asked to train classiffiers for classifying person vs. non-person images and only the `is_person` attribute and the 500 dimensional feature vector will be used. Do not use the additional class indicator attributes as features unless explicitly told to do so. \n",
    "\n",
    "In Part A we provide you with a training (`train_images partA.csv`) and a validation (`valid_images partA .csv`) dataset. In Part B we provide three data sets: a training set (`train_images partB.csv`), a validation set (`valid_images partB.csv`), and a test set (`test_images partB.csv`). The training and validation set contain valid labels. In the test set the labels are missing. The files are available from the GitHub repository. \n",
    "\n",
    "*Important: Throughout the assignment you will be given various versions of the dataset that are relevant\n",
    "to a particular question. Please be careful to use the correct version of the dataset when instructed to do so.\n",
    "If you use the wrong version of the dataset by mistake no marks will be awarded.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Exploration of the dataset [70%]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.1 ==========\n",
    "Load the datasets `train_images_partA.csv` and `valid_images_partA.csv` into two pandas DataFrame called `train_A` and `valid_A`. Display the number of data points and attributes in each of the two datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Load the dataset \n",
    "data_path = os.path.join(os.getcwd(), 'datasets', 'train_images_partA.csv')\n",
    "train_A = pd.read_csv(data_path, delimiter = ',')\n",
    "\n",
    "data_path = os.path.join(os.getcwd(), 'datasets', 'valid_images_partA.csv')\n",
    "valid_A = pd.read_csv(data_path, delimiter = ',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.2 ==========\n",
    "Display and inspect the first 10 instances in the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>imgId</th>\n",
       "      <th>dim1</th>\n",
       "      <th>dim2</th>\n",
       "      <th>dim3</th>\n",
       "      <th>dim4</th>\n",
       "      <th>dim5</th>\n",
       "      <th>dim6</th>\n",
       "      <th>dim7</th>\n",
       "      <th>dim8</th>\n",
       "      <th>dim9</th>\n",
       "      <th>...</th>\n",
       "      <th>is_cow</th>\n",
       "      <th>is_diningtable</th>\n",
       "      <th>is_dog</th>\n",
       "      <th>is_horse</th>\n",
       "      <th>is_motorbike</th>\n",
       "      <th>is_person</th>\n",
       "      <th>is_pottedplant</th>\n",
       "      <th>is_sheep</th>\n",
       "      <th>is_sofa</th>\n",
       "      <th>is_tvmonitor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2008_000008</td>\n",
       "      <td>0.002232</td>\n",
       "      <td>0.000558</td>\n",
       "      <td>0.002790</td>\n",
       "      <td>0.000837</td>\n",
       "      <td>0.001674</td>\n",
       "      <td>0.001953</td>\n",
       "      <td>0.001395</td>\n",
       "      <td>0.002232</td>\n",
       "      <td>0.003627</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2008_000015</td>\n",
       "      <td>0.001563</td>\n",
       "      <td>0.000391</td>\n",
       "      <td>0.007422</td>\n",
       "      <td>0.003516</td>\n",
       "      <td>0.003906</td>\n",
       "      <td>0.005078</td>\n",
       "      <td>0.001953</td>\n",
       "      <td>0.002344</td>\n",
       "      <td>0.001953</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2008_000019</td>\n",
       "      <td>0.000521</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.001042</td>\n",
       "      <td>0.001563</td>\n",
       "      <td>0.005729</td>\n",
       "      <td>0.000521</td>\n",
       "      <td>0.002083</td>\n",
       "      <td>0.003646</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2008_000023</td>\n",
       "      <td>0.002976</td>\n",
       "      <td>0.002232</td>\n",
       "      <td>0.004464</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>0.002232</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.003720</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2008_000028</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>0.000340</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>0.000340</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>0.002038</td>\n",
       "      <td>0.002378</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.003397</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2008_000033</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.006324</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>0.000744</td>\n",
       "      <td>0.008185</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2008_000036</td>\n",
       "      <td>0.000340</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.004416</td>\n",
       "      <td>0.000340</td>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.006114</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>0.002717</td>\n",
       "      <td>0.003057</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2008_000037</td>\n",
       "      <td>0.000837</td>\n",
       "      <td>0.002232</td>\n",
       "      <td>0.000279</td>\n",
       "      <td>0.000279</td>\n",
       "      <td>0.000837</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000279</td>\n",
       "      <td>0.006696</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2008_000041</td>\n",
       "      <td>0.002378</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>0.004755</td>\n",
       "      <td>0.001019</td>\n",
       "      <td>0.003736</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>0.001019</td>\n",
       "      <td>0.004076</td>\n",
       "      <td>0.003397</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2008_000045</td>\n",
       "      <td>0.001019</td>\n",
       "      <td>0.000340</td>\n",
       "      <td>0.006454</td>\n",
       "      <td>0.001698</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>0.003736</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.004076</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 520 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         imgId      dim1      dim2      dim3      dim4      dim5      dim6  \\\n",
       "0  2008_000008  0.002232  0.000558  0.002790  0.000837  0.001674  0.001953   \n",
       "1  2008_000015  0.001563  0.000391  0.007422  0.003516  0.003906  0.005078   \n",
       "2  2008_000019  0.000521  0.000000  0.000000  0.001042  0.001563  0.005729   \n",
       "3  2008_000023  0.002976  0.002232  0.004464  0.000372  0.000372  0.002232   \n",
       "4  2008_000028  0.001359  0.000340  0.001359  0.000340  0.001359  0.002038   \n",
       "5  2008_000033  0.000000  0.006324  0.000372  0.000372  0.000372  0.000372   \n",
       "6  2008_000036  0.000340  0.000000  0.004416  0.000340  0.000679  0.006114   \n",
       "7  2008_000037  0.000837  0.002232  0.000279  0.000279  0.000837  0.000000   \n",
       "8  2008_000041  0.002378  0.001359  0.004755  0.001019  0.003736  0.001359   \n",
       "9  2008_000045  0.001019  0.000340  0.006454  0.001698  0.001359  0.003736   \n",
       "\n",
       "       dim7      dim8      dim9      ...       is_cow  is_diningtable  is_dog  \\\n",
       "0  0.001395  0.002232  0.003627      ...            0               0       0   \n",
       "1  0.001953  0.002344  0.001953      ...            0               0       0   \n",
       "2  0.000521  0.002083  0.003646      ...            0               0       1   \n",
       "3  0.000000  0.003720  0.000000      ...            0               0       0   \n",
       "4  0.002378  0.000000  0.003397      ...            0               0       0   \n",
       "5  0.000744  0.008185  0.000372      ...            0               0       0   \n",
       "6  0.001359  0.002717  0.003057      ...            0               0       0   \n",
       "7  0.000279  0.006696  0.000000      ...            0               0       0   \n",
       "8  0.001019  0.004076  0.003397      ...            0               1       0   \n",
       "9  0.000000  0.004076  0.000000      ...            0               0       0   \n",
       "\n",
       "   is_horse  is_motorbike  is_person  is_pottedplant  is_sheep  is_sofa  \\\n",
       "0         1             0          1               0         0        0   \n",
       "1         0             0          0               0         0        0   \n",
       "2         0             0          0               0         0        0   \n",
       "3         0             0          1               0         0        0   \n",
       "4         0             0          0               0         0        0   \n",
       "5         0             0          0               0         0        0   \n",
       "6         0             0          1               0         0        0   \n",
       "7         0             0          0               0         0        0   \n",
       "8         0             0          1               0         0        0   \n",
       "9         0             0          0               0         0        0   \n",
       "\n",
       "   is_tvmonitor  \n",
       "0             0  \n",
       "1             0  \n",
       "2             0  \n",
       "3             1  \n",
       "4             0  \n",
       "5             0  \n",
       "6             0  \n",
       "7             0  \n",
       "8             0  \n",
       "9             0  \n",
       "\n",
       "[10 rows x 520 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_A.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.3 ==========\n",
    "In this part we want to select the appopriate attributes (i.e. input features) for training our classifiers. These should be the attributes `dim1, dim2, ..., dim500`. Create a list which has as elements the **names** of the attributes of interest. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attributes of Interest \\(as list\\): \n",
      "['dim1', 'dim2', 'dim3', 'dim4', 'dim5', 'dim6', 'dim7', 'dim8', 'dim9', 'dim10', 'dim11', 'dim12', 'dim13', 'dim14', 'dim15', 'dim16', 'dim17', 'dim18', 'dim19', 'dim20', 'dim21', 'dim22', 'dim23', 'dim24', 'dim25', 'dim26', 'dim27', 'dim28', 'dim29', 'dim30', 'dim31', 'dim32', 'dim33', 'dim34', 'dim35', 'dim36', 'dim37', 'dim38', 'dim39', 'dim40', 'dim41', 'dim42', 'dim43', 'dim44', 'dim45', 'dim46', 'dim47', 'dim48', 'dim49', 'dim50', 'dim51', 'dim52', 'dim53', 'dim54', 'dim55', 'dim56', 'dim57', 'dim58', 'dim59', 'dim60', 'dim61', 'dim62', 'dim63', 'dim64', 'dim65', 'dim66', 'dim67', 'dim68', 'dim69', 'dim70', 'dim71', 'dim72', 'dim73', 'dim74', 'dim75', 'dim76', 'dim77', 'dim78', 'dim79', 'dim80', 'dim81', 'dim82', 'dim83', 'dim84', 'dim85', 'dim86', 'dim87', 'dim88', 'dim89', 'dim90', 'dim91', 'dim92', 'dim93', 'dim94', 'dim95', 'dim96', 'dim97', 'dim98', 'dim99', 'dim100', 'dim101', 'dim102', 'dim103', 'dim104', 'dim105', 'dim106', 'dim107', 'dim108', 'dim109', 'dim110', 'dim111', 'dim112', 'dim113', 'dim114', 'dim115', 'dim116', 'dim117', 'dim118', 'dim119', 'dim120', 'dim121', 'dim122', 'dim123', 'dim124', 'dim125', 'dim126', 'dim127', 'dim128', 'dim129', 'dim130', 'dim131', 'dim132', 'dim133', 'dim134', 'dim135', 'dim136', 'dim137', 'dim138', 'dim139', 'dim140', 'dim141', 'dim142', 'dim143', 'dim144', 'dim145', 'dim146', 'dim147', 'dim148', 'dim149', 'dim150', 'dim151', 'dim152', 'dim153', 'dim154', 'dim155', 'dim156', 'dim157', 'dim158', 'dim159', 'dim160', 'dim161', 'dim162', 'dim163', 'dim164', 'dim165', 'dim166', 'dim167', 'dim168', 'dim169', 'dim170', 'dim171', 'dim172', 'dim173', 'dim174', 'dim175', 'dim176', 'dim177', 'dim178', 'dim179', 'dim180', 'dim181', 'dim182', 'dim183', 'dim184', 'dim185', 'dim186', 'dim187', 'dim188', 'dim189', 'dim190', 'dim191', 'dim192', 'dim193', 'dim194', 'dim195', 'dim196', 'dim197', 'dim198', 'dim199', 'dim200', 'dim201', 'dim202', 'dim203', 'dim204', 'dim205', 'dim206', 'dim207', 'dim208', 'dim209', 'dim210', 'dim211', 'dim212', 'dim213', 'dim214', 'dim215', 'dim216', 'dim217', 'dim218', 'dim219', 'dim220', 'dim221', 'dim222', 'dim223', 'dim224', 'dim225', 'dim226', 'dim227', 'dim228', 'dim229', 'dim230', 'dim231', 'dim232', 'dim233', 'dim234', 'dim235', 'dim236', 'dim237', 'dim238', 'dim239', 'dim240', 'dim241', 'dim242', 'dim243', 'dim244', 'dim245', 'dim246', 'dim247', 'dim248', 'dim249', 'dim250', 'dim251', 'dim252', 'dim253', 'dim254', 'dim255', 'dim256', 'dim257', 'dim258', 'dim259', 'dim260', 'dim261', 'dim262', 'dim263', 'dim264', 'dim265', 'dim266', 'dim267', 'dim268', 'dim269', 'dim270', 'dim271', 'dim272', 'dim273', 'dim274', 'dim275', 'dim276', 'dim277', 'dim278', 'dim279', 'dim280', 'dim281', 'dim282', 'dim283', 'dim284', 'dim285', 'dim286', 'dim287', 'dim288', 'dim289', 'dim290', 'dim291', 'dim292', 'dim293', 'dim294', 'dim295', 'dim296', 'dim297', 'dim298', 'dim299', 'dim300', 'dim301', 'dim302', 'dim303', 'dim304', 'dim305', 'dim306', 'dim307', 'dim308', 'dim309', 'dim310', 'dim311', 'dim312', 'dim313', 'dim314', 'dim315', 'dim316', 'dim317', 'dim318', 'dim319', 'dim320', 'dim321', 'dim322', 'dim323', 'dim324', 'dim325', 'dim326', 'dim327', 'dim328', 'dim329', 'dim330', 'dim331', 'dim332', 'dim333', 'dim334', 'dim335', 'dim336', 'dim337', 'dim338', 'dim339', 'dim340', 'dim341', 'dim342', 'dim343', 'dim344', 'dim345', 'dim346', 'dim347', 'dim348', 'dim349', 'dim350', 'dim351', 'dim352', 'dim353', 'dim354', 'dim355', 'dim356', 'dim357', 'dim358', 'dim359', 'dim360', 'dim361', 'dim362', 'dim363', 'dim364', 'dim365', 'dim366', 'dim367', 'dim368', 'dim369', 'dim370', 'dim371', 'dim372', 'dim373', 'dim374', 'dim375', 'dim376', 'dim377', 'dim378', 'dim379', 'dim380', 'dim381', 'dim382', 'dim383', 'dim384', 'dim385', 'dim386', 'dim387', 'dim388', 'dim389', 'dim390', 'dim391', 'dim392', 'dim393', 'dim394', 'dim395', 'dim396', 'dim397', 'dim398', 'dim399', 'dim400', 'dim401', 'dim402', 'dim403', 'dim404', 'dim405', 'dim406', 'dim407', 'dim408', 'dim409', 'dim410', 'dim411', 'dim412', 'dim413', 'dim414', 'dim415', 'dim416', 'dim417', 'dim418', 'dim419', 'dim420', 'dim421', 'dim422', 'dim423', 'dim424', 'dim425', 'dim426', 'dim427', 'dim428', 'dim429', 'dim430', 'dim431', 'dim432', 'dim433', 'dim434', 'dim435', 'dim436', 'dim437', 'dim438', 'dim439', 'dim440', 'dim441', 'dim442', 'dim443', 'dim444', 'dim445', 'dim446', 'dim447', 'dim448', 'dim449', 'dim450', 'dim451', 'dim452', 'dim453', 'dim454', 'dim455', 'dim456', 'dim457', 'dim458', 'dim459', 'dim460', 'dim461', 'dim462', 'dim463', 'dim464', 'dim465', 'dim466', 'dim467', 'dim468', 'dim469', 'dim470', 'dim471', 'dim472', 'dim473', 'dim474', 'dim475', 'dim476', 'dim477', 'dim478', 'dim479', 'dim480', 'dim481', 'dim482', 'dim483', 'dim484', 'dim485', 'dim486', 'dim487', 'dim488', 'dim489', 'dim490', 'dim491', 'dim492', 'dim493', 'dim494', 'dim495', 'dim496', 'dim497', 'dim498', 'dim499', 'dim500']\n"
     ]
    }
   ],
   "source": [
    "attributes_of_interest = train_A.columns[1:501].tolist()\n",
    "print(\"Attributes of Interest \\(as list\\): \\n{}\".format(attributes_of_interest))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.4 --- [1 mark] ==========\n",
    "By using the list from Question 1.3 now create 4 numpy arrays `X_tr`, `X_val`, `y_tr` and `y_val` and store the training features, validation features, training targets, and validation targets, respectively. Your target vectors should correspond to the `is_person` attribute of the training and validation sets (also make sure you do not include this attribute in your training features). Display the dimensionalities (i.e shapes) of the 4 arrays and make sure your input data are 500-dimensional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_tr shape: (1674, 500)\n",
      "X_val shape: (419, 500)\n",
      "y_tr shape: (1674,)\n",
      "y_val shape: (419,)\n"
     ]
    }
   ],
   "source": [
    "# split dataset\n",
    "X = train_A[attributes_of_interest]\n",
    "y = train_A['is_person']\n",
    "\n",
    "X_tr, X_val, y_tr, y_val = train_test_split(X, y, train_size=0.8, test_size=0.2, random_state=0)\n",
    "\n",
    "print(\"X_tr shape: {}\".format(X_tr.shape))\n",
    "print(\"X_val shape: {}\".format(X_val.shape))\n",
    "print(\"y_tr shape: {}\".format(y_tr.shape))\n",
    "print(\"y_val shape: {}\".format(y_val.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.5 ==========\n",
    "Show two [countplots](https://seaborn.github.io/generated/seaborn.countplot.html?highlight=countplot#seaborn.countplot) of the targets in a single plot, one for the training and one for the validation set. Label axes appropriately and add a title to your plot. Use descriptive `xticklabels` instead of the default numeric ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA38AAAHxCAYAAADdkgUOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe4JmdZP/DvZjehZRMSEpEiRAK/GxQpQcGIEgRCkyJI\nR2lSDWAUUIk00UgLoUoPTbGBgBQBS1ASmoCIBOJNCREVhGBCCkXIZn9/zBxydjm7Z0NO2d3n87mu\nvfa8M/PO3HPO7Ln3O/PMvBu2bt0aAAAA9m77rHcBAAAArD7hDwAAYADCHwAAwACEPwAAgAEIfwAA\nAAMQ/gAAAAawab0LAGAMVbUxyROTPDDJjyT5WpJ3JnlSd581L/PeJKd091NWaJuvSbKxux+wEutb\ntN7nJnlokut091e3m/fiJHdJ8mPdfcEO3n/VJP+Z5B3dfddLsN2nJrl1d9/iBy4egGG58gfAWnlm\nknsneUSS68xf/0SSd63iNh+b5JhVWO9Tk5yX5FmLJ1bVDTPt36N3FPxm907y+SS3r6pDLsF2n5Mp\nWALAJSb8AbBWHpzkqd19cnf/Z3e/P8n9kty4qm66Ghvs7vO7+/xVWO8FSR6X5Feq6maLZr04ydu6\n+23LrOJ+SV6R5CtJ7n8JtvvN7v76Ja0XABLDPgFYO1uT3Kqq3trdFyVJd59ZVT+W5MxFy121qt6Z\n5FZJvpjkMd39t0lSVddNcmKSn02yb5KPJnl4d59eVUcl+ZMkb03yK/NyP5p52Oc8ZPK6mYab/kqS\n7yQ5sbufOa97Q5JnJPnVuY7nJ3lQkl/t7vdtvzPd/ZdV9dAkL0xys6q6X6Yrmdfb2Tehqq6T5Igk\nj0py7UzDYF+waP4/JtnQ3UfNr387yW8k+bEkj0lym+7+uaralORFSe6eZP8k7890xfEzO9s+AONy\n5Q+AtfKCJL+W5D+q6hVVde+qOrAn/7douV9O8sZMYecjSV6/aN5fJ/lCkhskOTJTH3vOovlXS7I5\nyY2SvHaJGu6eKfQdkeTZSf5wDpRJclymUHjfJLdJcqdM4XFnjklyg6p6UJLjkxzX3V9e5j33S/I/\n3f3ReX9uVFU3WDT/4Ul+qqruU1XXzjTE9Ne6++x5/tb578fMdd4hU+g8L8lrltk2AAMT/gBYE939\nB5nudftcpitqf5bky1X1+O0WfWt3v7a7v5ApoB1aVVepqstnGir5hO4+s7v/Ncnrkvz4ovduTfKs\nef4XlyjjnCSP7+4zuvuEJGcn+cl53qOSPKW7/767P5HpitxO+2R3fzbJCXNdX+7ul+zCt+I+mUJf\nkvx9ptD2wEXr/EySP5j3/VWZhpG+eYn1XDPJt5J8sbvPmOt/wi5sH4BBCX8ArJnufmN3/3ySQ5Lc\nI8kpSZ5VVXdatNjnF3197vz3Zbv7m0leluQBVfWqqjo109DMjdtt5j92UsKZ3b110evzk+xbVVdK\nctVMw0gXav1MprC4nOMz3UbxB8stWFVHJKnM4a+7v5vpiaf3q6rFPfnZc21HZMcPrHl5pu/jl6rq\n5ExXFE/bhXoBGJTwB8Cqq6qfqKrnL7zu7vO6+83dfbskH0ty20WLb9nu7RuSbKiqK2QKZ/dPcnqS\np2TpK13f3kkp31li2oYkFy76evt5O9XdC9vb2XYX3Hf+++1V9d2q+m6SeyX5oSR3XLTcoZnC6GWT\nLPkwnO4+PclhSe6Z6WrqcUk+UFWX2YU6ABiQ8AfAWtiU5LE7eKrnuUm+usT0BQtX6m6Z6Z6+o7r7\nud19cqahj8sGtOV097lJvpTkJgvTqupaSa54ade9nXtneijNjZLccP5zo0z7/8BFy/1Rkg8leXqS\nl81DXrdRVb+S5G7d/dfd/fBMVwl/LNP9kADwfTztE4BV190fr6q3J3lzVR2X5H1JrpTpASw3TLKz\nD2FfCHf/m+TySe5RVR9OcnSmIZHfWKEyX5TkaVX1H0nOyvSAmq25OHxeKlV1iyRXT/Ki7v70dvNO\nSvK4qrpippB7hyTXT/JfmULhM5L8+narPDDJk6rq7CSfyfQ9PH/+GgC+jyt/AKyVeyV5ZZLfSfKp\nJH+X6UrVLRY9IXOpoLU1Sbr7Q0l+L9NHKyw8kOVRSa5UVVf/AWtavL0Tkrwp05NG/z7JOzINB11q\nqOjO1rMj90lyWnd/ZIl5r8h0QvaXM+3fc7r78/NTUB+T5NeWuGr6R0lePf85Pcmdk/zCfBUTAL7P\nhq1bV+SEJgDs0arqdkk+2t3/O78+JNNwzMN28ORQANijCH8AkKSq3pzpg+N/a5709CRX7+4j168q\nAFg5hn0CwOSYJN9N8v4kH5in3X39ygGAleXKHwAAwABc+QMAABiA8AcAADAA4Q8AAGAAwh8AAMAA\nhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAMIfAADAAIQ/AACAAQh/\nAAAAAxD+AAAABiD8AQAADED4AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAA\nAAYg/AEAAAxA+AMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAM\nQPgDAAAYgPAHAAAwAOEPAABgAMIfAADAAIQ/AACAAQh/AAAAAxD+AAAABiD8AQAADED4AwAAGIDw\nBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcAADAA4Q8A\nAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAJvWuwD2\nflV1zSRfSPLQ7n71oumPS3L97n7wCm3nC0l+qbv/ZSXWt8y2Nid5d5IDkjylu9+yaN5rkhyd5KtJ\ntibZL8nnkjysu7+2CrVcov2uqqskeWN3/+wl2Mb7k1wuyWWSVJJ/m2d9qrt/5RKs5/eSfLa7/2Qn\ny9w5ya27+9hdXe8y26wkz01y9SQbkpyd5End/f5l3ndAkrd0961Xog4A/XDP7odVdYUkX0py2+7+\n8Hbz3pbk5O5+/nL1ZepFv93d91pimbfPNb1+J+vZpj9V1b8kuWV3n7cr+7EzVbUpyfFJbp/pZ5Yk\nf9Hdz9iF9z45yb9299svbR2sHuGPtXJRkhOq6n3d/blF07fu6A27uRsnObS7/98O5p/Y3ScuvKiq\nE5K8NMk916K4nenuLyfZ5eA3v+fmyff+4/LJ7j7iB9z2U3dhmbcnWcnG8VdJjuvutyVJVf1ckndU\n1Y9299d38r6Dk/zUCtYBkOiHe2w/7O5vVNXrkjwkyffCX1VdLcktktx/F9fzsSTfF/wugW360w/a\nk3fg2CQ/muRG3b11DvfvraqzuvtVy7z3Vkk+tYK1sAqEP9bKtzJdffnzqvrp7r5w8cz57OAnFxrE\n4tfzmbI/TfILmX7hPS3JzZPcJMl3ktylu/9nXtWjq+qGmc4untjdr5nXd6ckT0qyb5JvJnl8d3+4\nqp6a5MgkV0nyie5+wHZ1/WKSp2QaIn1eksclOTfJSUmuOp9tO7K7/2+Z/f+HJM+a13nVJC9O8iNz\nPX/e3c/c0fa6+yNznT+e5IeTXDnJxzOdOb5gu3rvnOR3F+3nE7r7Q9stc80kp3X35nm9h837f81M\nZ2fvvej7uayqOirJC5J8I8nlk9wsyXOS3DTJ5kxnOB/a3R/c7uf6rSTPzHRW+CpJXtDdL6yqBya5\nR3ffuarem+SDmX7e10hyysLPqKoelOS35/18b5Jf7+59lyjxh5Psv/Ciu0+pqnsl2TKv58hMP5vL\nZ/pP2dO6+2+SvDrJ5eef8U26e0/9jxmwe9EP9+x++NIkH6iqY7v7W/O0X810dez8qvqhJC9P8kNz\njf+R5F6Lr3TOffPF3f0T89XH183b/eL8voXlHpLk4fM+HJzkmd398mzbn34yyYVJDunus+erb/dJ\n8t0kn0ny6O7+6s766XauMm/vckm+Oe/TL88/h4Wrji9Icv15uX9I8ltJHjHX8pyq2tLdf73EutkN\nuOePtbK1u4/PFBCWHTqwhMt0942SPD7JK5I8b379X0ketGi5b3b3TZLcNskzq+p6VXXtJH+Y5A7z\nvEckeUtVXW5+zzUyneHavtFVpl/yd5u39dQkf51pyMdDk3y+u49YrtHN23lAkpPnSX+c5KTu/qlM\nQenoqrrHjrZXVQvB5WZJ7t7dlSm4PGW77Vw701CNxfv55kX7udjiIPOzmYbJXC/J1+f3XVI/nqlJ\n3jjJEUl+uLuP7O7rJ3l9kt9Z4j2XSfLVebjNPZM8q6r2W6K+a3X3UUl+IsmtquqoqrpepuB4q3lf\nz8uOf58dk+TFVfVfVfUXVXVMko/ODe2KSV6T5Je7+yeT3DXJy6rq6kkenOl4OkLwA1aQfrgH98Pu\nPj3Jv2S+cllVGzJdCXzRvMh9knygu2/e3YdnCvtL3R6xsN2XJPlgd/9Ekscmue683itkCpUL+3Cf\nTCdWk23700UL66qqBye5XaYTljfKdBXudYu2+X39dIm6Tsx0m8RZVfXeqvqDJJft7k/P85+XqYf+\nVKZ+f2iS3+julyT5aKaQLfjtxoQ/1tovJ3lQVd3mEr7vr+a/P5/ky9192qLXBy9a7uXJ94ZyvDvJ\nrXPxlaV/qKqPJ3lDprNk157f86Ed/Of+Vkn+vrv/Y17ne5N8JdMZ1uX8ZlX9y7y9Dyf5WpLjqury\nSY5K8vvzvA9lOuN5o13Y3hsXnTk8KdMv+MWOznSWcUf7uSP/2N3fmL/+eLb9fu6q/+zu/5rr/lCS\nJ1fVI6vqOUnukUVX3rbztvk9/5Lp7PQVlljm7fMyF2S6V+TgTPv+nvnnnFzcdL9Pd/9Fpu/LA5Kc\nnqlJf6qqrpGLz3K/df6e/U2m/0jcYBf3G+AHpR/uuf3wpZmCWZLcMckXF34O3f3CJB+sqt+oqpdk\nOjm6ox6YTD+X187v/XzmYDzXceckd6qqp2e6irlUj1zs9kle093fnl+/IFPIWxjpt1Q/3UZ3//cc\n7G6S5C8z3ef/gap65LzInZI8Yv6+fizT8NPrL1MXuxHDPllT3f2fVfWoTGeiFt/MvDXT8MAF+2Vb\ni88mfncnm9iy6Ot95mX3zdRE7rswY76y86Ukd0+yzVCR7d6/vY3z+nZWQ7LdPQ6Ltrt5/vJ7Q2Oq\n6kpJvp1tz9huv71kalyLa9uyxLL/sMR+/vcytX5r0dfb/xx21fe+h1X1C0men+SEJG9N8u/Z8X0Q\n39ru9VLbXqq+C7Ptz+eipVY+nz1+UHc/MVNDPTnJ06rqbzOF0s8k+XR3H7noPVfJNNzn6juoGeBS\n0w/36H74liTPm68wPjTT0NWF7Twr0/DHV2fqOfvuZD1LbefCeT1XyzRM8+VJTknypkzDfXdm+5/T\nxkz/119Y/7L7N9f/qu7+90z9+6VVdf9Mt1m8bF7nPbu75+UPzA56MLsnV/5YK9/7BdPdb0ryrkw3\nFS84K9Mvy1TVIUl+7gfczoPmdVwjyW0yjUU/Oclt5yCQqrpjkk9kGna4MwvvO2x+360yBYIP7+xN\nO9Pd52c6u/n4eZ1XTPL+JHfZhe3dtao2V9U+SR6W+arZEvVuv5+X/UHr3YHlwuFtkrxtvi/hY0l+\nMVOzuLTrXew9SW49B7Vkar5L+UqSh1XV3RcmVNXBme6p+Fimn8V1anoITKrqRkk+m+SqmRrwrtQN\ncEnoh9mz+2F3b0nyyiS/numBN3+1aPZtkzy/u9+Q6Srn0dl5L3lXpvv6Fn5WPz9P/8lMt0Yc391/\nl+kq4MIw0+3708Ix9Z4kD56vqibTMNJ/6u7lAvpiP5Tk6QtDZOftVaaeubCN35znXSbT9/7R87wL\nc3FAZzflyh9rZfthJI/NdNPxwvQXJXlDVZ2e5MxMD/DY0Xt3to3LVtXHMv3yeXTPT1Krqodnurk+\nmX453bm7vzW/XlJ3n15Vv5bpfoiNmW4Yv9N8r9hydezM/ZL8UVX921znG7r7z+Y6d7a9r2QalnhI\nkvfl4ntFts71fnpH+7lMPZfUcvv3siR/WlX/muls7PsyPdp6ufUstd4ll+nuz1bVbyb525oeHPOJ\nTN+vbXT31+f/NDyzpifMfSPTWfNnd/c/JUlV/VKmG9Qvm6mB3n8+I79Pko9X1aeT3Ly7z1lmvwF2\nhX54sT25H74yyRlJnjGHwQVPT/LcqnrKvN1TcvFw06W+H49O8pqq+lSm+zY/Pk//2yQPqarOdEX2\nnzOdGLh2piG+C/3pZxet96RMIfmf59D2uUzDi5fa9o5+No/KdF/ov1XVtzNlhZNzccB7bJLnV9Un\n53l/l+TZ87y3Z3qS7X7d/cc7WD/rbMPWrav7HIOqulmmpxP9fFUdnmlc80WZnq50zLzMwzKd9fhu\nkuO7+53zf8T+JNMZiPOSPLC7/3dVi4XdVE1PIbtSdz92vWvZHcxngx/Q3U+fX98tyW8tHr4JewI9\nEi4Z/RAunVUd9llVT8h0ZmRhOMGJmT5v66gk+1TVXavqykkek+nBC7dP8oyq2jfTmYd/6+5bZHoa\n1JNXs1Zgj/JfmR4tftp8hfHXMz3IBfYYeiQAa221h31+LsndMjWmZHr07Cnz1+/KNC76oiSn9vQ5\nN+dV1WeT3DDTZexnLVpWY2NY3f17613D7mT+ffHIZReE3ZseCZeQfgiXzqpe+evut2TbJzItfqDD\n+UkOyPQh0Ocumn5BkgO3m76wLADsFfRIANbaWj/wZfGjYDdn+gDN87Jt09qc5Jx5+ubtll3WhRdu\n2bppkwf0AQzgB/lYkt3ZqvZI/RFgKEv2yLUOf/9SVbfo7vcluUOmpwd9JMnxVbVfkssluW6S05J8\nINMHZ350/vuUpVe5rXPO+b4H/gGwFzr00M3LL7RnWdUeqT8CjGNHPXKtP+fv8Zk+O+T9mR7p+6bu\n/kqSFyY5NcnfZ7rZ/TtJXprk+lV1SqbP8DLGG4C9mR4JwKpa9Y96WGtnnXX+3rVDACzp0EM3723D\nPleV/ggwjh31yLW+8gcAAMA6EP4AAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAMIfAADAAIQ/\nAACAAQh/AAAAAxD+AAAABiD8AQAADED4AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAA\nAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAAD2LTeBeyu\ntmzZkjPPPGO9y2Avdthh18rGjRvXuwwAAAYh/O3AmWeekSc+9y9yhQMPXe9S2At949yz8ozH3TuH\nH36d9S4FAIBBCH87cYUDD80BB19lvcsAAAC41NzzBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAA\nAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAG\nIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAMIfAADAAIQ/AACAAQh/AAAAAxD+AAAABiD8AQAADED4\nAwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcA\nADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMQPgDAAAYwKa13mBVbUryuiSH\nJbkwycOSbEny2iQXJTmtu4+Zl31Ykocn+W6S47v7nWtdLwCsFT0SgNW0Hlf+7phkY3ffPMnvJ/nD\nJCcmOa67j0qyT1XdtaqunOQxSY5Mcvskz6iqfdehXgBYK3okAKtmPcLfZ5JsqqoNSQ7MdMbyiO4+\nZZ7/riRHJ7lpklO7+8LuPi/JZ5PcYB3qBYC1okcCsGrWfNhnkguS/GiSf09ypSR3TvJzi+afn+SA\nJJuTnLvd+w5coxoBYD3okQCsmvUIf7+R5N3d/btVdbUk/5hkv0XzNyf5epLzMjW47afv1EEHXT6b\nNm281EWec87+l3odsDMHH7x/Dj1083qXAexeVq1HrlR/BGDPtR7h7+xMw1iSqVFtSvLxqjqqu/8p\nyR2SnJzkI0mOr6r9klwuyXWTnLbcys8555srU+TZF6zIemBHzj77gpx11vnrXQbssfbSkyer1iNX\nqj8CsPvbUY9cj/D3/CSvrqr3Jdk3ye8k+ViSV803q5+e5E3dvbWqXpjk1CQbMt3s/p11qBcA1ooe\nCcCq2bB169b1rmFFnXXW+SuyQ5///GfzB686OQccfJWVWB1s47yzv5wnPfRWOfzw66x3KbDHOvTQ\nzRvWu4Y9yUr1RwB2fzvqkT7kHQAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAAAAYg/AEAAAxA+AMA\nABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMYNN6FwDsPrZs2ZIz\nzzxjvctgL3bYYdfKxo0b17sMABiS8Ad8z5lnnpEnv/Hp2f+QA9a7FPZCF3ztvPz+PZ+Sww+/znqX\nAgBDEv6Abex/yAE58IcPWu8yAABYYe75AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAA\nAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAG\nIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAMIfAADAAIQ/AACAAQh/AAAAAxD+AAAABiD8AQAADED4\nAwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcA\nADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABg\nAMIfAADAAIQ/AACAAQh/AAAAAxD+AAAABiD8AQAADED4AwAAGIDwBwAAMADhDwAAYACb1mOjVfU7\nSe6SZN8kL0nyviSvTXJRktO6+5h5uYcleXiS7yY5vrvfuR71AsBa0SMBWC1rfuWvqo5KcmR3/0yS\nWya5RpITkxzX3Ucl2aeq7lpVV07ymCRHJrl9kmdU1b5rXS8ArBU9EoDVtB7DPm+X5LSqemuStyV5\nR5IjuvuUef67khyd5KZJTu3uC7v7vCSfTXKDdagXANaKHgnAqlmPYZ+HZDqTeack18rU3BaH0POT\nHJBkc5JzF02/IMmBa1QjAKwHPRKAVbMe4e9/k5ze3Rcm+UxVfTvJ1RfN35zk60nOy9Tgtp8OAHsr\nPRKAVbMe4e/UJI9N8ryqumqSKyT5h6o6qrv/Kckdkpyc5CNJjq+q/ZJcLsl1k5y23MoPOujy2bRp\n46Uu8pxz9r/U64CdOfjg/XPooZvXu4xtOO5Zbbvjcb+bWbUeuVL9EYA915qHv+5+Z1X9XFX9c5IN\nSR6V5Mwkr5pvVj89yZu6e2tVvTBTI9yQ6Wb37yy3/nPO+eaK1Hn22ResyHpgR84++4Kcddb5613G\nNhz3rLaVPO73xhC5mj1ypfojALu/HfXIdfmoh+7+nSUm33KJ5U5KctKqFwQAuwk9EoDV4kPeAQAA\nBiD8AQAADED4AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAAAAYg/AEAAAxA\n+AMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMQPgDAAAYgPAH\nAAAwAOEPAABgAMIfAADAAHYp/FXVjy8x7adXvhwAAABWw6adzayqmyfZmORVVfWrSTYset/Lkvy/\n1S0PAACAlbDT8Jfk6CRHJblKkqcvmn5hkpevVlEAAACsrJ2Gv+5+WpJU1a909x+vSUUAAACsuOWu\n/C14X1U9J8nBuXjoZ7r7IatSFQAAACtqV8PfXyY5Zf6zdfXKAQAAYDXsavjbt7sfv6qVAAAAsGp2\n9XP+Tq2qO1fVfqtaDQAAAKtiV6/83SPJo5Okqhambe3ujatRFAAAACtrl8Jfd191tQsBAABg9exS\n+Kuqpyw1vbufvtR0AAAAdi+7es/fhkV/9ktylyRXXq2iAAAAWFm7Ouzz9xa/rqrfT/K3q1IRAAAA\nK25Xr/xtb/8k11jJQgAAAFg9u3rP3xdy8Ye775Pkikmes1pFAQAAsLJ29aMebrno661Jvt7d5618\nOQAAAKyGXR32+cUkd0zy3CQvTPKgqvpBh4wCAACwxnb1yt+zk1wnyaszPfHzwUmuleTYVaoLAACA\nFbSr4e+2SW7c3RclSVW9M8knV60qAAAAVtSuDt3clG2D4qYkW1a+HAAAAFbDrl75e0OSf6yqP5tf\n3zfJn65OSQAAAKy0ZcNfVR2U5JVJPp7kVvOf53f3H69ybQAAAKyQnQ77rKobJ/l0kpt097u6+wlJ\n3pPkmVV1g7UoEAAAgEtvuXv+Tkhy3+5+98KE7j4uyUOSnLiahQEAALBylgt/B3X3P24/sbvfk+SQ\nVakIAACAFbdc+Nt3qQ9zn6fttzolAQAAsNKWC3//lOSpS0x/UpKPrnw5AAAArIblnvb5xCR/U1X3\nT/KRJBuSHJHkq0nussq1AQAAsEJ2Gv66+/yqukWSn09y4yQXJfmj7j5lLYoDAABgZSz7OX/dvTXJ\nyfMfAAAA9kDL3fMHAADAXkD4AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAA\nAAYg/AEAAAxA+AMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGsGm9NlxV\nP5Tko0luk2RLktcmuSjJad19zLzMw5I8PMl3kxzf3e9cn2oBYG3ojwCslnW58ldVm5K8LMk350kn\nJjmuu49Ksk9V3bWqrpzkMUmOTHL7JM+oqn3Xo14AWAv6IwCrab2GfZ6Q5KVJvpRkQ5IjuvuUed67\nkhyd5KZJTu3uC7v7vCSfTXKD9SgWANaI/gjAqlnz8FdVD0ry1e7+u0yNbfs6zk9yQJLNSc5dNP2C\nJAeuRY1jYzIqAAAM6ElEQVQAsNb0RwBW23rc8/fgJBdV1dFJbpjk9UkOXTR/c5KvJzkvU5PbfjoA\n7I30RwBW1ZqHv/m+hSRJVZ2c5JFJnlNVt+ju9yW5Q5KTk3wkyfFVtV+SyyW5bpLTllv/QQddPps2\nbbzUdZ5zzv6Xeh2wMwcfvH8OPXTzepexDcc9q213PO53F3tKfwRgz7VuT/vczuOTvHK+Yf30JG/q\n7q1V9cIkp2Ya/nJcd39nuRWdc843l1tkl5x99gUrsh7YkbPPviBnnXX+epexDcc9q20lj/tBQuRu\n1x8B2P3tqEeua/jr7lstennLJeaflOSkNSsIAHYD+iMAq8GHvAMAAAxA+AMAABiA8AcAADAA4Q8A\nAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAMIfAADA\nAIQ/AACAAQh/AAAAAxD+AAAABiD8AQAADED4AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEI\nfwAAAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4A\nAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAMIfAADAAIQ/AACAAQh/AAAAAxD+AAAABiD8AQAA\nDED4AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAAAAYg/AEAAAxA+AMAABiA\n8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEP\nAABgAJvWuwAAYM+xZcuWnHnmGetdBnuxww67VjZu3LjeZcBeSfgDAHbZmWeekSc+9y9yhQMPXe9S\n2At949yz8ozH3TuHH36d9S4F9krCHwBwiVzhwENzwMFXWe8yALiE3PMHAAAwAOEPAABgAMIfAADA\nAIQ/AACAAQh/AAAAA1jzp31W1aYkr05yWJL9khyf5NNJXpvkoiSndfcx87IPS/LwJN9Ncnx3v3Ot\n6wWAtaJHArCa1uPK3y8n+Vp33yLJ7ZO8OMmJSY7r7qOS7FNVd62qKyd5TJIj5+WeUVX7rkO9ALBW\n9EgAVs16fM7fXyZ54/z1xiQXJjmiu0+Zp70ryW0zneE8tbsvTHJeVX02yQ2SfGyN6wWAtaJHArBq\n1jz8dfc3k6SqNmdqcL+b5IRFi5yf5IAkm5Ocu2j6BUkOXKMyAWDN6ZEArKZ1eeBLVf1IkpOTvK67\n/zzTGcwFm5N8Pcl5mRrc9tMBYK+lRwKwWtbjgS9XTvKeJMd093vnyR+vqlt09/uS3CFT0/tIkuOr\nar8kl0ty3SSnLbf+gw66fDZt2nip6zznnP0v9TpgZw4+eP8ceujm9S5jG457VtvueNzvTlazR+qP\n7Cn8noDVsx73/D0xyRWTPLmqnpJka5JfT/Ki+Wb105O8qbu3VtULk5yaZEOmm92/s9zKzznnmytS\n5NlnX7Ai64EdOfvsC3LWWeevdxnbcNyz2lbyuN9L/3O4aj1Sf2RPsTv2R9jT7KhHrsc9f8cmOXaJ\nWbdcYtmTkpy02jUBwO5AjwRgNa3HlT8AANhjbNmyJWeeecZ6l8Fe7rDDrpWNGy/98PydEf4AAGAn\nzjzzjDz5jU/P/occsPzC8AO44Gvn5ffv+ZQcfvh1VnU7wh8AACxj/0MOyIE/fNB6lwGXyrp81AMA\nAABrS/gDAAAYgPAHAAAwAOEPAABgAMIfAADAAIQ/AACAAQh/AAAAAxD+AAAABiD8AQAADED4AwAA\nGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcAADAA\n4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAADEP4AAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAMIf\nAADAAIQ/AACAAQh/AAAAAxD+AAAABiD8AQAADED4AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAA\ngAEIfwAAAAMQ/gAAAAYg/AEAAAxA+AMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8AAIABCH8AAAAD\nEP4AAAAGIPwBAAAMQPgDAAAYgPAHAAAwAOEPAABgAMIfAADAAIQ/AACAAQh/AAAAAxD+AAAABiD8\nAQAADED4AwAAGIDwBwAAMADhDwAAYADCHwAAwACEPwAAgAEIfwAAAAMQ/gAAAAYg/AEAAAxA+AMA\nABjApvUuYGeqakOSlyS5YZJvJ3lod5+xvlUBwPrTIwG4pHb3K3+/mOQy3f0zSZ6Y5MR1rgcAdhd6\nJACXyO4e/n42ybuTpLs/nOQn17ccANht6JEAXCK79bDPJAckOXfR6wurap/uvmgtNv6Nc89ai80w\noN352Lrga+etdwnspRxbK27deuTu/DuMPdvufGz5HcZqWqvja8PWrVvXZEM/iKp6bpIPdveb5tdf\n7O5rrHNZALDu9EgALqndfdjn+5PcMUmq6qeTfHJ9ywGA3YYeCcAlsrsP+3xLkqOr6v3z6wevZzEA\nsBvRIwG4RHbrYZ8AAACsjN192CcAAAArQPgDAAAYgPAHAAAwgN39gS9cAlV1VJK/TvLj3f3f87Rn\nJDm9u19fVVdIcnySGyfZmunzoR7f3Z+tqhOS3CTJDye5fJLPJzmru++9DrsCl1pVXTPJn3f3kVV1\nSJKXJdk/yeYkn0ry2O7+9nrWCKwdPRIupkeOS/jb+/xfktckue0S816Z5P3dfWySVNUNkry1qn66\nux8/T3tgkuru49aqYFhFC0+0ekKSv+3uVyRJVZ2Y5BFJXrBehQHrQo+Ei+mRAxL+9j4nJ9lQVcd0\n9x8tTKyqKyW5fnffb2Fad/9bVb0tyd2TvG65FVfV1ZK8NMllklwlyZO6+22L5l8zyRuTfCnJ1ZO8\nu7ufVFVXT/KKJJdN8q0kD8907L0jyVlJ/ibJN5I8MMmWJB/p7mPn9b06ycZMv6Ae292frKrPJDk1\nyXWT/E+SX+puj61lZ76S5B5V9flMn432hCQXLV6gqjYneVWSA5NcNclLuvtl2y3z+SQfSnLtJJ/s\n7odW1QFJTkpy8LzYY7v7U1X1H0k+Pf85NclvJ/lOki91932q6sAkf5LkgEzH+JO6+x+r6hNJ/inJ\nDeYa79rd56/stwOGpUfC99MjB+Kev73P1iSPSnJsVR2+aPq1Mg1T2d4XklxzF9d93SQndPftMp0R\nevQSy1wzU4O6aZKfr6obJzkhyQu6+1ZJnpvkWfOyP5Tk6O4+IcmDkhzT3TdPcnpVbZzf97zuvmWS\nYzM1uYV9eVJ3/8y8jp/axfoZ1/OS/GmmhvalJG9OcrXtlrl2kj/r7tsnuV2S31xiPVfLdOzdLMn+\nVXW3JMcl+fvuvnWmfxcLzfDqSe7b3Y9Lct8kz+7uWyR5x9zUnpTpTOtRSe6Vi4/vA5K8YT7uv5Tk\nDpd254Hv0SPh++mRAxH+9kLdfU6S38h0pnLDPPlLSQ5bYvHrJPniLq76y0keWVWvS/LILH3l+BPd\nfW53X5Tkn5NUkp9IclxVnZzkyZmaUZJ8obu3zF8/JMmjq+q9mZrjhiTXS3LKvE+fyPSLIkm+1t1f\nmr/+z0xnS2FnbpXk9XPTunKSjyR5/nbLfCXJ3arq9ZmazlLH9xe7+wvz1x/Mxcf3Q+bj+5VJDprn\nn9XdX5+//s0kt56P75/JdLbyeknelyTz8XxuVS382/jX+W/HN6wwPRK+jx45EOFvL9Xd70jSSR48\nv/7vJJ+rqkctLFNVRyS5U6YzPLvi95O8rrsfmOS9ubhpLvZjVXXZ+azkzTLdNHx6kt+ez2o+MtOw\nl+TiseZJ8rAkj+jun09yRJIjMw0FuMVc640yDV/Z/n2wKx6b5P5J0t3fzXRcbn8j++OSfKC7H5Dp\nGF3q+L76ouZz8ySnZTq+nzcf3/fKNEwl2fY4fXiSp87H9z5JfjHbHt9Xy9QQ/3eJ9wIrTI+EbeiR\nA3HP397t2ExncxY8IMkJVfWhJBcmOSfJL3b3ebu4vjcmeW5VPTHJfyU5ZIllvjMvd+Ukb5zvP3hC\nkpdW1WUznaH59XnZxf94P5nk1Ko6f173hzOdbX1lVT0+07H6kCXe5xcAu+KRSV5SVcdmuqfmrExD\nvxZ7e5IXVdV9Mj3l78Kq2nduhAv+L8mLq+oaST7Y3e+oqg8kOamqHpHpKWlPm5ddfGz+c5J3zsf3\n+Znu5XlHkldX1T0y/bt4WHdvqSrHN6wNPRImeuRANmzd6vvGyphvPv+z+T4D2OtU1Ze7+yrrXQew\n59Ej2dvpkXsGwz4Bdp2zZQCwND1yD+DKHwAAwABc+QMAABiA8AcAADAA4Q8AAGAAwh8AAMAAhD8A\nAIABCH8AAAAD+P/xWgCWwzJnxQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x122a89e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "\n",
    "ax1 = fig.add_subplot(221)\n",
    "ax1 = sns.countplot(x=y_tr)\n",
    "ax1.set_title('Number of People in Training Set')\n",
    "ax1.set_ylabel('Count')\n",
    "ax1.set_xlabel('')\n",
    "ax1.set_xticklabels(['NOT a person', 'IS a person'])\n",
    "\n",
    "ax2 = fig.add_subplot(222)\n",
    "ax2 = sns.countplot(x=y_val)\n",
    "ax2.set_ylim([0, 1000])\n",
    "ax2.set_title('Number of People in Validation Set')\n",
    "ax2.set_ylabel('')\n",
    "ax2.set_xlabel('')\n",
    "ax2.set_xticklabels(['NOT a person', 'IS a person'])\n",
    "\n",
    "\n",
    "fig.suptitle('Sharing Y Axis', fontsize=14)\n",
    "fig.set_figheight(15)\n",
    "fig.set_figwidth(15)\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.6 ==========\n",
    "What would be a good choice for a baseline classifier to compare your results in the next steps against? What classification accuracy score does the baseline classifier achieve in the validation set?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the graphs show, it is more likely that the image is not a person. So as a baseline classifier we could simply predict that the image does not contain a person (0) every time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Accuracy Score: 0.5513\n"
     ]
    }
   ],
   "source": [
    "zeros = pd.Series(0, index=np.arange(len(y_val)))\n",
    "ca = accuracy_score(y_val, zeros)\n",
    "print(\"Classification Accuracy Score: {:.4f}\".format(ca))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.7 ==========\n",
    "Train a [`LogisticRegression`](http://scikit-learn.org/0.17/modules/generated/sklearn.linear_model.LogisticRegression.html) classifier by using default settings, except for the `solver` parameter which you should set to `lbfgs`. Report the classification accuracy score in the training and validation sets and compare to that of the baseline classifier. Comment in 1-2 sentences the results. You may include any additional plot(s) if you wish to justify your explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.8 ==========\n",
    "Display the means and standard deviations of the first 5 features in the training set. *Hint: you want to compute the means and standard deviations across the columns of your arrays. Make sure you make appropriate use of the `axis` parameter.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "np.set_printoptions(precision=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.9 ==========\n",
    "Feature standardisation is a pre-processing technique used to transform data so that they have zero mean and unit standard deviation. For many algorithms, this is a very important step for training models (both regression and classification). Read about [feature standardisation](http://scikit-learn.org/0.17/modules/preprocessing.html) and make sure you understand what kind of transformation this method applies to the data.\n",
    "\n",
    "`Scikit-learn` offers an [implementation](http://scikit-learn.org/0.17/modules/generated/sklearn.preprocessing.StandardScaler.html) of feature standardisation. Create a standardiser and fit it by using training features only. Then transform both your input and validation input features. \n",
    "\n",
    "Once your training and validation input data have been transformed, display the means and standard deviations of the first 5 attributes for **both** the training and validation sets. Are the results as you expected? Explain your answer in 2-3 sentences. \n",
    "\n",
    "**IMPORTANT: You should use the transformed data for the rest of part A**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.10 ==========\n",
    "By using the transformed input data, retrain a `LogisticRegression` classifier. Again, set the `solver` parameter to `lbfgs` and use default settings for the other parameters. Report the classification accuracy in both the training and validation sets. How does your model compare to the baseline classifier from Question 1.6? You may use additional plot(s) to support your explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.11 ==========\n",
    "So far we have used default settings for training the logistic regression classifier. Now, we want to use [K-fold cross-validation](http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.KFold.html) to optimise the regularisation parameter `C`. The regularisation parameter controls the degree to which we wish to penalise large magnitudes for the weight vector. Thus, it helps us prevent overfitting and, for logistic regression, it additionally controls the level of confidence in making predictions.\n",
    "\n",
    "We would like to optimise this parameter **by using the transformed training dataset only** and not the validation set. Create a 3-fold cross-validation object for the training dataset. Set the `shuffle` parameter to `True` and the `random_state` to `0`. By using the cross-validation iterator, display the number of test samples in each iteration fold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.12 ========== \n",
    "Now we want to test out different settings for the regularisation parameter `C` by using the cross-validation iterator from the previous question. For each iteration, we want to train multiple classifiers by using a range of values for `C` and then compute the performance within each CV fold. You should use a log-range for `C` from `1e-5` to `1e5` by using 20 equally-spaced values *(hint: look at the `logspace()` function in numpy)*. \n",
    "\n",
    "Create a 2-dimensional array and, for each cross-validation fold and parameter setting pair, compute and store the classification accuracy score e.g. store the score of fold 0 with parameter setting 1 at score_array[0,1]. As previously, set the `solver` parameter to `lbfgs` and use default settings for the other parameters.\n",
    "\n",
    "*(hint: your may want to use two loops in your code; one iterating over CV folds and another one iterating over the values for `C`)*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.13 ========== \n",
    "Plot the mean classification performance (across CV folds) of the logistic regression classifier against the regularisation parameter `C` by using the range from Question 1.12. Use a logarithmic scale for the x-axis and label both axes appropriately. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.14 ==========\n",
    "Find and display the optimal value for the regularisation parameter `C` of the logistic classifier by using your results from Question 1.12. Similarly to Question 1.13, consider the mean classifiation accuracy across CV folds. By using the optimal value (i.e. the one that yields the highest average classification accuracy) train a new `LogisticRegression` classifier and report the classification accuracy on the validation set. *(Hint: Do not pick the optimal value \"by hand\", instead use an appropriate numpy function).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.15 ========== \n",
    "Scikit-learn offers a [`LogisticRegressionCV`](http://scikit-learn.org/0.17/modules/generated/sklearn.linear_model.LogisticRegressionCV.html) module which implements Logistic Regression with builtin cross-validation to find out the optimal `C` parameter. You can specify the range for the `C` parameter, as well as the cross-validation method you want to use with the `Cs` and `cv` parameters, respectively. Use the `C` range you set up in Question 1.12 and the 3-fold cross-validation iterator from Question 1.11. Once again, train the models by using the `lbfgs` optimisation method and display the optimal value for the parameter `C`. Finally, display the classification accuracy on the validation set. Can you verify that your results are consistent with those from Question 1.14?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.16 ==========\n",
    "Now, we want to validate the importance of various features for classification. For this purpose, we will use a [`RandomForestClassifier`](http://scikit-learn.org/0.17/modules/generated/sklearn.ensemble.RandomForestClassifier.html) (you might want to refer to Lab_3 if you are unsure how we can estimate feature importances with decision tree and random forest models).\n",
    "\n",
    "Initialise a random forest classifier and fit the model by using training data only and 500 trees (i.e. `n_estimators`). Set the `RandomState` equal to 31 to ensure reproducible results. Report the accuracy score on both the training and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.17 ==========\n",
    "Comment on the results above. Do you find the discrepancy between training and validation accuracies surprising?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.18 ==========\n",
    "By using the random forest model from the previous question order the features by descending importance and display the names of the 50 most important features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.19 ==========\n",
    "Next, we would like to test out the performance of support vector classification and experiment with different kernels. \n",
    "By using training data only and default parameter settings, train three support vector classifiers with the following kernels: linear, radial basis function, and polynomial. Report the classification accuracy of each of the three classifiers on both the training and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.20 ==========\n",
    "At this point, we would like to get an idea of what kind of decision boundaries we can get with the three types of SVM kernels we introduced in the previous question. For visualisation, however, we can only make use of 2D input data. For this purpose, we select to use the 21st and 76th columns of our training features (*hint: remember that Python uses 0-based indexing*). \n",
    "\n",
    "Execute the cell below to define a useful function which we will be using to plot the decision boundaries *(it is also not a bad idea to try to understand what this functions does)*. \n",
    "\n",
    "Then train three distinct SVM classifiers by using the 2D input data mentioned above and default parameters:\n",
    "* a linear SVC\n",
    "* an RBF SVC \n",
    "* a polynomial SVC\n",
    "\n",
    "Finally, create a list containing the three classifiers you have just trained. Use this list as an input to the provided function along with the used training features and observe the outcome. You can use the additional `title` parameter to set the titles in the subplots. Comment on the results by using 1-2 sentences.\n",
    "\n",
    "*(Acknowledgement: this Question has been heavily based on [this example](http://scikit-learn.org/0.17/auto_examples/svm/plot_iris.html) from scikit-learn's documentation.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def show_svc_decision_boundaries(clfs, X, title=None):\n",
    "    \"\"\"Plots decision boundaries for classifiers with 2D inputs.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    clf : list\n",
    "        Classifiers for which decision boundaries will be displayed.\n",
    "    X : array\n",
    "        Input features used to train the classifiers.\n",
    "    title : list, optional\n",
    "        Titles for classifiers.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    assert X.shape[1] == 2 # Input should be 2D\n",
    "    if title is not None:\n",
    "        assert len(clfs) == len(title)\n",
    "    \n",
    "    h = .04 # step size in the mesh\n",
    "    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                         np.arange(y_min, y_max, h))\n",
    "    plt.figure(figsize=(15,5))\n",
    "    for i, clf in enumerate(clfs):\n",
    "        plt.subplot(1, len(clfs), i + 1)\n",
    "        plt.subplots_adjust(wspace=0.4, hspace=0.4)\n",
    "        Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "        Z = Z.reshape(xx.shape)\n",
    "        plt.contourf(xx, yy, Z, cmap=plt.cm.Paired, alpha=0.8)\n",
    "        # Training points\n",
    "        plt.scatter(X[:, 0], X[:, 1], c=y_tr, cmap=plt.cm.Paired)\n",
    "        plt.xlabel('Feature 1')\n",
    "        plt.ylabel('Feature 2')\n",
    "        plt.xlim(xx.min(), xx.max())\n",
    "        plt.ylim(yy.min(), yy.max())\n",
    "        plt.xticks(())\n",
    "        plt.yticks(())\n",
    "        if title is not None:\n",
    "            plt.title(title[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.21 ==========\n",
    "So far we have used default parameters for training the SVM classifiers. Now we want to tune the parameters by using cross-validation. \n",
    "\n",
    "By using the `K-fold` iterator from Question 1.11 and training data only, estimate the classification accuracy of an SVM classifier with RBF kernel, while you vary the penalty parameter `C` in a logarithmic range `np.logspace(-2, 3, 10)`. Set the kernel coefficient parameter `gamma` to `auto` for this question. \n",
    "\n",
    "Plot the mean cross-validated classification accuracy against the regularisation parameter `C` by using a log-scale for the x-axis. Display the highest obtained mean accuracy score and the value of `C` which yielded it. Label axes appropriately. \n",
    "\n",
    "Finally, train a classifier by using the optimal value for this parameter (without using cross-validation at this stage) and report the classification accuracy on the training and validation sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.22 ==========\n",
    "Now we turn to the kernel coefficient `gamma` parameter. By using the same procedure as in the previous question, estimate the classification accuracy of an SVM classifier with RBF kernel while you vary the `gamma` parameter in a logarithmic range `logspace(-5, 0, 10)`. Fix the penalty parameter `C=1.0`.\n",
    "\n",
    "Plot the mean cross-validated classification accuracy against the parameter `gamma` by using a log-scale for the x-axis. Display the highest obtained mean accuracy score and the value of `gamma` which yielded it.  Label axes appropriately.\n",
    "\n",
    "Finally, train a classifier by using the optimal value for this parameter (without using cross-validation at this stage) and report the classification accuracy on the training and validation sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.23 ==========\n",
    "Now we wish to tune both the `C` and `gamma` parameters simultaneously. To save computational time, we will now constrain the parameter search space. Define a `4 X 4` grid for the two parameters, as follows:\n",
    "* `C`: `np.logspace(-2, 1, 4)`\n",
    "* `gamma`: `np.logspace(-4, -1, 4)`\n",
    "\n",
    "Estimate the mean cross-validated classification accuracy by using training data only and all possible configurations for the two parameters. \n",
    "\n",
    "Use a [heatmap](https://seaborn.github.io/generated/seaborn.heatmap.html?highlight=heatmap#seaborn.heatmap) to visualise the mean cross-validated classification accuracy for all `C`-`gamma` pairs. Label axes appropriately and display the values for `C` and `gamma` for the best performing configuration. \n",
    "\n",
    "Finally, by using the optimal configuration, train a classifier (without using cross-validation) and report the classification accuracy on the training and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.24 ==========\n",
    "Is the classification accuracy on the validation set higher than in previous questions (1.22-1.23)? If not, can you explain why? Can you think of a way of further improving the performance of the classifier? You don't need to implement your suggestion at this stage. Would there be any associated problems with your suggested approach? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.25 ==========\n",
    "Now we want to evaluate the performance of an SVM classifier with polynomial kernel. Once again, we will tune the `degree` parameter by using cross-validation (similarly to what we did in Questions 1.21 and 1.22).\n",
    "\n",
    "By using the `K-fold` iterator from Question 1.11 and training data only, estimate the classification accuracy of polynomial SVM classifier, while you vary the `degree` parameter in the range `np.arange(1,8)`. \n",
    "\n",
    "Plot the mean cross-validated classification accuracy against the polynomial degree. Display the highest obtained mean accuracy score and the value of the `degree` parameter which yielded it. Label axes appropriately. \n",
    "\n",
    "Finally, train a classifier by using the optimal value for this parameter (without using cross-validation at this stage) and report the classification accuracy on the training and validation sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.26 ==========\n",
    "\n",
    "You might have noticted that so far, we have used cross-validation for optimising the various tuning parameters (e.g. regularisation parameter in logistic regression, SVM kernel parameters) rather than hold-out validation, although we did have access to a validation set. Why do you think this is a good/bad idea? Give one advantage and one disadvantage of the two different approaches. Which one would you trust more in practice?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.27 ==========\n",
    "\n",
    "Reload the full training and validation data that contain all indicator variables for all object categories. Remove the `imgId` attribute but keep all of the class indicator variables in the dataset this time. Your training features should include all attributes except `is_person` which should be your target variable. \n",
    "\n",
    "Once again, use a [StandardScaler](http://scikit-learn.org/0.17/modules/generated/sklearn.preprocessing.StandardScaler.html) to standardise your training and validation features. Then train a Random Forest Classifier by using the entropy `criterion`, 500 `n_estimators`, and also set the `random_state` to 31. Report the classification accuracy on the training and validation sets.\n",
    "\n",
    "Similarly to what we did in Question 1.18, order the features by decreasing importance and display the 50 most important features. \n",
    "\n",
    "Finally, answer the following questions:\n",
    "* What do you notice by looking at the list of the best 50 features?\n",
    "* How does the performance differ with respect to the case when the additional class indicator variables are not present (Question 1.16)? Relate your observations to the observed feature ranking.\n",
    "* Would it be easy to make use of the results in practice? Briey explain your reasoning.\n",
    "\n",
    "*(Hint: you might want to look at some of the [images](http://www.inf.ed.ac.uk/teaching/courses/iaml/2014/assts/asst3/images.html) to justify your explanations.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mini challenge [30%]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Important: You are allowed to write up to a maximum of 600 words in this part of the assignment. The thoroughness of the exploration and the quality of the resulting discussion is just as important as the final classification performance of your chosen method(s) and credit will be divided accordingly.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this final part of the assignment we will have a mini object-recognition challenge. Using the data provided you are asked to find the best classiffier for the person/no person classification task. You can apply any preprocessing steps to the data that you think fit and employ any classiffier you like (with the provison that you can explain what the classiffier/preprocessing steps are doing). You can also employ any lessons learnt during the course, either from previous Assignments, the Labs or the lecture material to try and squeeze out as much performance as you possibly can. The only restriction is that all steps must be performed in `Python` by using the `numpy`, `pandas` and `sklearn` packages. You can also make use of `matplotlib` and `seaborn` for visualisation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important:** The classification performance metric that we will use for this part is the *cross-entropy* or *logarithmic loss* (see Lab 4). You should familiarise yourself with the metric by reading the `sklearn` [user guide](http://scikit-learn.org/0.17/modules/model_evaluation.html#log-loss) and [documentation](http://scikit-learn.org/0.17/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss). To estimate this metric you will need to evaluate probability outputs, as opposed to discrete predictions which we have used so far to compute classification accuracies. Most models in `sklearn` implement a `predict_proba()` method which returns the posterior probabilities for each class. For instance, if your test set consists of `N` datapoints and there are `K` classes, the method will return a `N` x `K` matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We provide you with three new data sets: a training set (`train_images_partB.csv`), a validation set (`valid_images_partB.csv`), and a test set (`test_images_partB.csv`). You must use the former two for training and evaluating your models (as you see fit). Once you have chosen your favourite model (and pre-processing steps) you should apply it to the test set (for which no labels are provided). Estimate the posterior proabilities for the data points in the test set and submit your results as part of your answer. Your results will be evaluated in terms of the logarithmic loss metric. You also need to submit a brief description of the approaches you considered, your suggested final approach, and a short explanation of why you chose it. The thoroughness of the exploration and the quality of the resulting discussion is just as important as the final score of your chosen method(s) and credit will be divided accordingly.\n",
    "\n",
    "*Hint: Feature engineering, feature combination, model combination and model parameter optimization can significantly improve performance.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**How to submit your results**: Store the estimated posterior probabilities for the data points in the test set into a 2D numpy array. Then execute the provided cell at the end of this notebook which uses a provided `save_predictions` function to export your results into a `.txt` file (the function will return an error if the provided array has not the right shape). The `.txt` file will be saved where your notebook lives. You are then required to copy this file into your submission folder along with your notebook (see Mecanics section at the top of the notebook) and then submit the two files by using the `submit` command on DICE. You are only required to use the `submit` command once."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here (max. 600 words)*\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Execute this cell to export your results\n",
    "from numpy import savetxt\n",
    "def save_predictions(pred_proba):\n",
    "    if pred_proba.shape != (1114,2):\n",
    "        raise ValueError('Predicted probabilities array has not the right shape.')\n",
    "    \n",
    "    savetxt('assignment_3_predictions.txt', pred_proba)\n",
    "\n",
    "save_predictions(pred_proba) # You need to replace \"pred_proba\" with the name of the array\n",
    "                             # which contains the probability estimates for the data in \n",
    "                             # the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [iaml]",
   "language": "python",
   "name": "Python [iaml]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}