classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "jaxBhwggSEEM"
   },
   "source": [
    "# Logistic Regression and Classification problems\n",
    "\n",
    "This notebook will introduce you to the basics of logistic regression, guide you through a naive implementation of the algorithm, and teach you the basics of the SciKit-learn data analysis library via comparing our naive model with their implementation. At the end of the notebook we will look at a dataset and go through an initial qualitative analysis routine.\n",
    "\n",
    "### Assumptions\n",
    "\n",
    "\n",
    "* Assume a binary outcome of the dependent variable $y$ (1/0, yes/no, dead/alive, false/true)\n",
    "* Only non-collinear (mutually independent) variables $x_i$ (features) are included\n",
    "* Expect a linear dependence between the variables $x_i$ and the log odds\n",
    "* The training dataset should be sufficiently large if logistic regression is to be effective\n",
    "\n",
    "### Plan for today\n",
    "1. KNN Classifier\n",
    "2. Logistic Classifier\n",
    "3. Comparison with scikit implementation\n",
    "4. Short project\n",
    "\n",
    "### How to learn anything?\n",
    "\n",
    "* *Don't be afraid, make mistakes, and ask questions.*\n",
    "* *Practice makes perfect*. Initially you will be making mistakes, the more you practice the sooner you will become a pro.\n",
    "* If you don't understand something, first compose the question properly, think about it, and if you can't think of a solution ask someone!\n",
    "* Before you execute any cell, stop and try to predict what it will do! Ideally write your prediction on a piece of paper!\n",
    "* Experiment! Change the code in the cells, try various test cases to understand how the code behaves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "e-3yXQjU5zHx"
   },
   "source": [
    "## 1.  KNN classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j0OxZnK_5zHy"
   },
   "source": [
    "In K-nearest neighbour classification, the predicted label for an unknown datapoint is interpolated from its K nearest neighbours. Naive implementation of a precise k-nearest neighbour classifier is incredibly simple, let's look at it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R1kJ0RCl5zHy"
   },
   "source": [
    "Initially, we need to import some data. We will be using the glass type dataset. Below you can see the structure of five randomly sampled datapoints from the set - it is useful to get used to the pandas library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 901
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 854,
     "status": "error",
     "timestamp": 1541438713619,
     "user": {
      "displayName": "Edward Lim",
      "photoUrl": "",
      "userId": "05219823243799720278"
     },
     "user_tz": 0
    },
    "id": "uLb7tcP-5zH0",
    "outputId": "4d5f270d-a387-4678-cd27-d0482cd14293"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "dataset = pd.read_csv(\"glass.csv\") # importing the data from the .csv file\n",
    "dataset.sort_values('Al', inplace=True) # sort by ascending \"Al\" values\n",
    "dataset.head() # print the first 5 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting we will add another label to the dataset, if binary is equal to 0 the glass cannot be recycled. If binary equals 1 then it is possible to recycle the glass type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset['binary'] = dataset.Type.map({1:0, 2:0, 3:0, 5:1, 6:1, 7:1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I5bAbf_W5zH7"
   },
   "source": [
    "In KNN, it is trivial to implement a multidimensional classifier, that's why we will consider all features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sGkSeRHw5zH8"
   },
   "outputs": [],
   "source": [
    "knn_data = np.stack((dataset[\"RI\"],dataset[\"Na\"],dataset[\"Mg\"],dataset[\"Al\"],dataset[\"Si\"],dataset[\"K\"],dataset[\"Ca\"],dataset[\"Ba\"],dataset[\"Fe\"], dataset[\"binary\"]), axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6CvPxBGh5zH_"
   },
   "source": [
    "Write a function ```distance(p, data)``` that returns an array of distances between the point ```p``` and each point in the dataset ```data```. In other words, the distance is generalised as\n",
    "$$D^2 = \\sum_i x_i^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qBGqm7wB5zIA"
   },
   "outputs": [],
   "source": [
    "#CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g9nSl8Pd5zID"
   },
   "source": [
    "Use pandas to split the data into a training set and a test set at a ratio of 80:20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "T1ZHdrsS5zIF"
   },
   "outputs": [],
   "source": [
    "#CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pJwG6OT65zIH"
   },
   "source": [
    "Plot the \"Mg\" training values against the \"Si\" training values. Let points with binary value '1' be red, and points with binary value '0' be black. Also include the test points in blue colour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NYVHWRH05zIJ"
   },
   "outputs": [],
   "source": [
    "#CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hQymcFPI5zIK"
   },
   "source": [
    "Write a function ```knn_search(p, trainingset, k)``` that returns the indices of the ```k``` closest neighbours around the point ```p```, use the ```distance``` function implemented in previous task!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jXp6I74J5zIN"
   },
   "outputs": [],
   "source": [
    "#CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ke6jcmSw5zIQ"
   },
   "source": [
    "Write a code that looks at the labels of the closest neighbours, and chooses the most frequent label as its output. This is your predicted value for the point ```p```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mD_EKZDV5zIQ"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YfDVwA5X5zIT"
   },
   "source": [
    "Use the code to make a prediction about the labels in the testing set, calculate the accuracy, sensitivity, and specificity of this model. Don't forget that the data is not 2-dimensional, the above plot is only illustrative!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_gNqw_QI5zIU"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7cQXiKOH5zIW"
   },
   "source": [
    "Choose a metric you would like to optimise, and plot its value as a function of the ```k``` hyperparameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LhW5uL-75zIX"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the dataset carefully now - is this classifier biased? Can the bias of the classifier be decreased by transforming the dataset? (subtracting the mean / normalising the distribution)\n",
    "\n",
    "Try normalising the dataset and test its new accuracy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KyWmB3cFSEEO"
   },
   "source": [
    "## 2. Implementation of logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D_eAf4NCSEEP"
   },
   "source": [
    "In order to begin, use matplotlib and numpy to plot the logistic function $\\phi(z)$ over a sufficiently long range of $z$-values to showcase its behaviour. $$\\phi(z) = \\frac{1}{1 + exp(-z)}$$\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hHtKY0GISEEQ"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "paC2CAX5SEEU"
   },
   "source": [
    "Assuming the input data has a single independent variable $x$ (feature) and a binary outcome $y$, the maximum likelihood $L(\\alpha_0, \\alpha_1)$ becomes: \n",
    "\n",
    "$$L(\\alpha_0, \\alpha_1) = \\prod_{i} P(x_i)^{y_i}\\left(1-P(x_i)\\right)^{1-y_i},$$\n",
    "\n",
    "where the product runs over all the datapoints $i$, and $P(\\vec{x})$ is the logistic probability function. Log-likelihood can then be expressed as: \n",
    "\n",
    "$$l(\\alpha_0,\\alpha_1) = \\sum_{i=1}^n - \\log\\left(1 + \\exp({\\alpha_0 + \\alpha_1 x_i})\\right) + \\sum_{i=1}^n y_i(\\alpha_0 + \\alpha_1 x_i),$$\n",
    "\n",
    "and the partial derivatives of the log-likelihood:\n",
    "\n",
    "\n",
    "$$\\frac{\\partial l}{\\partial \\alpha_0} = -\\sum_{i=1}^n y_i - P(x_i; \\alpha_0, \\alpha_1)\\qquad\\text{and}\\qquad \\frac{\\partial l}{\\partial \\alpha_1} = -\\sum_{i=1}^n \\left(y_i - P(x_i; \\alpha_0, \\alpha_1)\\right)x_{i}$$\n",
    "\n",
    "As before, to find the optimal parameters $\\alpha_0, \\alpha_1$ we will code a function minimiser. In this lecture we will implement a simple gradient-descent, this is an important step in your journey. The more confident can google the Newton-Raphson minimiser and implement it instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Q8jrU0tgSEEV"
   },
   "outputs": [],
   "source": [
    "\"\"\" Code the functions beneath:\n",
    "    data -> a two-dimensional numpy array consisting of (x_i, y_i) pairs\n",
    "    weights -> a numpy array with the weights (alpha_0, alpha_1)\n",
    "\"\"\"\n",
    "\n",
    "def logit(z):\n",
    "    \"\"\" The logistic probability function \"\"\"\n",
    "    return ???\n",
    "\n",
    "def loglike(data, weights):\n",
    "    \"\"\" Log-likelihood function with one dependent variable\"\"\"\n",
    "    return ???\n",
    "\n",
    "def gradient_0(data, weights):\n",
    "    \"\"\" Partial derivative with respect to alpha 0 \"\"\"\n",
    "    return ???\n",
    "\n",
    "def gradient_1(data, weights):\n",
    "    \"\"\" Partial derivative with respect to alpha 1 \"\"\"\n",
    "    return ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oAcdTsEJSEEY"
   },
   "source": [
    "Numpy can be used to convert functions into their vectorised forms that act on whole arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1MAD7TT_SEEZ"
   },
   "outputs": [],
   "source": [
    "logit = np.vectorize(logit)\n",
    "loglike = np.vectorize(loglike)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CbG2P74HSEEc"
   },
   "source": [
    "The next step is to write the minimiser function, in multivariate gradient descent it is important that all directions are updated simultaneously, i.e. the updates should not be done sequentially. Calculate the steps required in each direction, store them in an array, and update the positions all at once! It increases the convergence speed of the algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SKIJWmXVSEEd"
   },
   "outputs": [],
   "source": [
    "def minimise(derivative_0, derivative_1, data, initVals, learning_rate, tolerance):\n",
    "    current_guess = initVals\n",
    "    adjustment = np.full_like(current_guess, 1)\n",
    "    \n",
    "    while :\n",
    "        adjustment[0] = ???????\n",
    "        adjustment[1] = ???????\n",
    "        current_guess = current_guess + adjustment\n",
    "    \n",
    "    return current_guess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8Wot3QpMSEEg"
   },
   "source": [
    "In order to see if our code works, we will be using the following dataset: https://www.kaggle.com/uciml/glass. Putting it all together:\n",
    "    - Import the dataset\n",
    "    - Guess the weights $\\alpha_0, \\alpha_1$\n",
    "    - Run the multivariate gradient descent on the cost function\n",
    "    - Use the weights to predict the outcome probabilities on the data set\n",
    "    - Evaluate the accuracy\n",
    "    \n",
    "We start by using pandas to import the dataset, and then have a quick look at a few entries to get the overall feel for the structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Qi8KRIjoSEEh"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "dataset = pd.read_csv(\"glass.csv\") # importing the data from the .csv file\n",
    "dataset.sort_values('Al', inplace=True) # sort by ascending \"Al\" values\n",
    "dataset.head() # print the first 5 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jpRL0eLJSEEm"
   },
   "source": [
    "As a data scientist, it is important to first do an \"exploratory analysis\" looking at the dataset and gaining qualitative understanding of the underlying rules before applying any quantitative models. Because we are only testing our code here, we will skip this part, and I will artificially claim that the only important feature is the aluminum content \"al\". Don't worry, we will do some full-fledged dataset analysis later on!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pLXobWa8SEEn"
   },
   "source": [
    "We can now plot the binary response vs the independent variable \"al\", uhm, I mean, you can! Plot the first column of the \"data\" array vs the second column. Don't forget to label the axes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YsAbnk0DSEEn"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "dataset['binary'] = dataset.Type.map({1:0, 2:0, 3:0, 5:1, 6:1, 7:1})\n",
    "data = np.stack((dataset[\"Al\"], dataset[\"binary\"]), axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QvTkIL7CSEEr"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1OxzC4gSSEEu"
   },
   "source": [
    "By now we have prepared everything necessary to employ our code for naive logistic regression. Use the functions you have implemented, find a good value for the learning rate, tolerance, and the initial weights in order to find the best fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XVt0fN5PSEEv"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "86e8VPysSEEy"
   },
   "source": [
    "Once you know the weights, use your knowledge of the logistic model to plot the data, along with the probability $P(x_i)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n1nOb5IJSEE0"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aH0v3XOfSEE3"
   },
   "source": [
    "It is important to remember the interpretation of the logit function, it is the probability that a feature with the associated Aluminum content has the binary outcome $1$. If we wanted to make a prediction of the binary outcome, we would set a threshold at $\\text{logit}(z) = 0.5$ and set anything larger to $1$, and anything smaller to $0$. Do just that, and plot the result alongside the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "413kj_XKSEE4"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "Ltr30IlqSEE9"
   },
   "source": [
    "While this may work here, it is not always the case that the threshold should be set to $0.5$. If the dataset you work with is unbalanced, i.e. it is much rarer to find a $1$ outcome than $0$, then it is reasonable to base your threshold value off of the ROC curve. It is not a good idea to use the accuracy as your metric of choice for unbalanced datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BMKzhWPUSEE9"
   },
   "source": [
    "## 3. Comparison to SciKit-learn implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "umLGNxulSEE-"
   },
   "source": [
    "We can now compare your code's prediction to the prediction made by the implementation of logistic regression in the scikit-learn library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "E6HOsqMsSEE_"
   },
   "outputs": [],
   "source": [
    "# fit a logistic regression model and store the class predictions\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logistic = LogisticRegression(C=1e9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HaOmv-tzSEFC"
   },
   "source": [
    "In the above, we set only a single parameter for the logistic classifier. C is the inverse of regularisation strength, thus we set the regularisation effectively to zero. This way we can compare the SciKit implementation to our naive implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "P-dDkOB4SEFD"
   },
   "outputs": [],
   "source": [
    "X = data[:,0].reshape(-1,1)\n",
    "y = data[:,1]\n",
    "logistic.fit(X, y)\n",
    "dataset['predicted'] = logistic.predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FHtq77M7SEFG"
   },
   "source": [
    "The features data[:,0] had to be reshaped, because the LogisticRegression.fit() function is built to accept models with multiple features. Therefore each value has to be converted into an array with a single element. Now we can plot the predicted values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "co051yG1SEFG"
   },
   "outputs": [],
   "source": [
    "plt.scatter(dataset.Al, dataset.binary)\n",
    "plt.plot(dataset.Al, dataset.predicted, color='red')\n",
    "plt.xlabel('Aluminum content')\n",
    "plt.ylabel('Binary outcome')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hIr770FbSEFK"
   },
   "source": [
    "Similarly, the sklearn can be used to predict the probabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FVX9BxgASEFK"
   },
   "outputs": [],
   "source": [
    "dataset[\"probability\"] = logistic.predict_proba(data[:,0].reshape(-1,1))[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mo_8eO1SSEFN"
   },
   "source": [
    "Plot the probabilities predicted by the SciKit library, and alongside them the probabilities predicted by your custom logistic regression implementation. Provide a quantitative description of how similar the two predictions are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R970uGYVSEFO"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rsHs4--x5zJZ"
   },
   "source": [
    "Compare the time it takes your implementation to run against the scikit implementation, for this you can use the timeit library in python. Search it up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d3NYwe3x5zJZ"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4_PguIJbSEFQ"
   },
   "source": [
    "*Advanced Problems:*\n",
    "    - Generalise your logistic regression code to accept multiple features with a binary outcome\n",
    "    - Implement a regularisation scheme, can you compare your regression with the SciKit library for different regularisation strengths?\n",
    "    - Google \"Softmax\" regression, can you implement the equivalent of logistic regression for multiple features *and* multiple outcomes?\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tdj0rPfQSEFR"
   },
   "source": [
    "## 4.  Problem solving (short project)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "7U664BVHSEFS"
   },
   "source": [
    "In this part we will look at a dataset with multiple variables using SciKit-Learn, the specific dataset in question is the Telco Customer churn dataset provided by IBM Sample Data Sets at https://www.ibm.com/communities/analytics/watson-analytics-blog/guide-to-sample-datasets/. Our goal is explained on the website, \"Predict behavior to retain customers. You can analyze all relevant customer data and develop focused customer retention programs.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gp7D9jfASEFS"
   },
   "source": [
    "Use pandas to import the dataset, look at a sample of the data to understand the structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cSa2_5icSEFU"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "z4IIpL0-SEFW"
   },
   "source": [
    "The dataset is rather extensive, but the overall structure of the information provided can be summarised as follows:\n",
    "    - Outcome: If the customer left last month (Yes / No) corresponding to the column header \"Churn\"\n",
    "    - Demographics: Gender, age (column SeniorCitizen), family status (columns Partner, Dependents)\n",
    "    - Account information: TotalCharges, PaymentMethod, MonthlyCharges, PaperlessBilling, Contract, Tenure\n",
    "    - Services: StreamingMovies, StreamingTV, TechSupport, DeviceProtection, OnlineSecurity, InternetService, MultipleLines, PhoneService\n",
    "    \n",
    "Now we will proceed to do some exploratory analysis. When working with a new dataset, it is always good to look for missing values in any of the columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Uq5nXRIXSEFX"
   },
   "source": [
    "Use pandas and matplotlib to plot a bar chart with the number of missing values per column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "X2hefAl7SEFY"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VyUO3oyjSEFc"
   },
   "source": [
    "There are only 11 values missing in the category \"TotalCharges\", in a dataset of 7043 datapoints we can easily discard the incomplete rows. If the dataset was smaller, we would have to replace these blank spaces with a statistic of the overall dataset. The statistic depends on whether or not the values or missing at random, completely at random, or not at random. Remove the missing rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EZmHZZUYSEFc"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9EfHIZusSEFh"
   },
   "source": [
    "Plot the churn, what percentage of the clients has cancelled their subscription over the last month?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "P5M50OhGSEFi"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gWea8J31SEFl"
   },
   "source": [
    "Now we can use the categorical variables, and see how churn depends on the category. To do this faster, we will use a function that will plot the relevant graphs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CTcA5js6SEFl"
   },
   "outputs": [],
   "source": [
    "def plot_bar(dataset, var):\n",
    "    \n",
    "    values = dataset[var].unique()\n",
    "    percentages = np.zeros(len(values))\n",
    "    \n",
    "    for i in range(len(values)):\n",
    "        percentages[i] = dataset.loc[dataset[var] == values[i]][\"Churn\"].eq(\"Yes\").sum()\n",
    "        \n",
    "    percentages = 100*percentages/(len(dataset))\n",
    "    \n",
    "    ax = plt.bar(np.arange(len(values)), percentages)\n",
    "    plt.xticks(np.arange(len(values)), values)\n",
    "    plt.ylabel(\"Churn per month [%]\")\n",
    "    \n",
    "    return ax "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zrzX0qcySEFn"
   },
   "source": [
    "### Gender dependence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "u9dklT7sSEFp",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plot_bar(dataset, 'gender')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9oP6eDxTSEFr"
   },
   "source": [
    "Plot the other graphs:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B_ax8Jx_SEFs"
   },
   "source": [
    "### Age dependence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zvW-oYy9SEFt"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jsAzrOtwSEFx"
   },
   "source": [
    "### Partner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pEdXhDlDSEFy"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8Q0VynKNSEF1"
   },
   "source": [
    "### Dependents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MG2OYdzmSEF2"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "e9Jzd19_SEF4"
   },
   "source": [
    "### Phone Service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bUX6hXwrSEF4"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aH1GgJ2SSEF-"
   },
   "source": [
    "### Multiple Lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zi8iDuDFSEF_"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "--FgKFnBSEGB"
   },
   "source": [
    "### Internet Service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hl5a_huFSEGD"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rIo_WLXsSEGF"
   },
   "source": [
    "### Online Security"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EzcMApdxSEGG"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ABaZxAvISEGK"
   },
   "source": [
    "### Device Protection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qPzfEkPdSEGK"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FUKZvaaESEGN"
   },
   "source": [
    "### Tech Support"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "X5gG7-6cSEGO"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_TAwkAsmSEGQ"
   },
   "source": [
    "### Streaming TV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VIGnClVoSEGQ"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6IaXhj8tSEGR"
   },
   "source": [
    "### Streaming Movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B-stgfxOSEGS"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X7qkzGV7SEGU"
   },
   "source": [
    "### Contract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3uMSYtH0SEGV"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TpXyXpc4SEGY"
   },
   "source": [
    "### Paperless Billing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2WGY8Fg2SEGY"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3jJIgoFESEGZ"
   },
   "source": [
    "### Payment Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fiWKwPosSEGa"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OkpmMs6oSEGc"
   },
   "source": [
    "Use the above graphs to perform a qualitative analysis of the dataset, which features seem to strongly relate to the churn rate? Which features seem unimportant?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "it1TZsM0SEGc"
   },
   "source": [
    "*WRITE YOUR FINIDINGS HERE*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ou74BBSESEGd"
   },
   "source": [
    "The next step is to visualise the remaining three continuous variables, this is usually done in one of two ways:\n",
    "    - Box plot\n",
    "    - Histogram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h-uIXxI3SEGd"
   },
   "source": [
    "### Tenure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XYyJIF68SEGg"
   },
   "outputs": [],
   "source": [
    "left = dataset.loc[dataset['Churn'] == \"Yes\"].tenure\n",
    "right = dataset.loc[dataset['Churn'] == \"No\"].tenure\n",
    "\n",
    "plt.boxplot([left.values, right.values], [1,2])\n",
    "plt.xticks([1,2], [\"Yes\", \"No\"])\n",
    "plt.ylabel(\"Churn per month [%]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2SikxNoQSEGh"
   },
   "source": [
    "We can also show this in a histogram!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WmbGCi_CSEGi",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plt.subplot(2,1,1)\n",
    "plt.hist(right, bins=10)\n",
    "plt.ylabel(\"Stayed [people]\")\n",
    "plt.xlabel(\"Tenure\")\n",
    "\n",
    "plt.subplot(2,1,2)\n",
    "plt.hist(left, bins=10)\n",
    "plt.ylabel(\"Left [people]\")\n",
    "plt.xlabel(\"Tenure\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Uh1x1wEbSEGk"
   },
   "source": [
    "### Monthly Charges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MV_Jas0cSEGk"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FTcztKItSEGm"
   },
   "source": [
    "### Total Charges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Yk797K0kSEGo"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D2l2WMSDSEGr"
   },
   "source": [
    "Deduce the findings from the above three plots:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eB6yUjcwSEGs"
   },
   "source": [
    "*WRITE YOUR FINIDINGS HERE*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SlIwAM_CSEGs"
   },
   "source": [
    "When working with continuous variables, it is also a good idea to look for correlations between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QtBu1HI2SEGs"
   },
   "outputs": [],
   "source": [
    "dataset[\"TotalCharges\"] = dataset[\"TotalCharges\"].astype(np.float) \n",
    "dataset[[\"TotalCharges\", \"MonthlyCharges\", \"tenure\"]].corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mNc_lHRYSEGu"
   },
   "source": [
    "We can see that the tenure strongly correlates with total charges, this is fairly obvious and intuitive. Similarly, we see that the tenure has little correlation with the monthly charges. It is often nice to visualise the correlations by colours:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p9MVgHJmSEGu"
   },
   "outputs": [],
   "source": [
    "plt.imshow(dataset[[\"MonthlyCharges\", \"TotalCharges\", \"tenure\"]].corr(), cmap=\"rainbow\")\n",
    "plt.xticks([0,1,2], [\"MonthlyCharges\", \"TotalCharges\", \"tenure\"])\n",
    "plt.yticks([0,1,2], [\"MonthlyCharges\", \"TotalCharges\", \"tenure\"])\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TUOgOnPRSEGw"
   },
   "source": [
    "### Preparing data for logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fVdgXBDZSEGw"
   },
   "source": [
    "Change all the \"Yes\" and \"No\" values in the dataset to $1$ and $0$ respectively. Some categories have more than two options, for example the StreamingMovies feature also has the option \"No internet service\". You can bundle those with the \"No\" values, is there any problem with giving that feature a new number, for example $2$? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eQmXlr8GSEGw"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UjPywkLHSEGy"
   },
   "source": [
    "Standardise the continuous variables by taking each, subtracting their mean, and dividing by the standard deviation. Doing this reduces the multicollinearity between the variables, and allows our model to predict its coefficients more precisely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lIOzHuIESEGy"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g_ZLRryhSEG0"
   },
   "source": [
    "It is also possible to derive new features from the current features, if you wish you can try creating some new that you think may be of interest. For example, only a few clients will have the exact same tenure, you can bin together all clients that have their tenure betwee 1-2 years, 3-4 years, etc. Similarly it may be interesting to do similar things for the monthly charges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0Veq3cCiSEG1"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AV-5Yo6dSEG3"
   },
   "source": [
    "### Logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "neMJE7bZSEG3"
   },
   "source": [
    "Now we can apply logistic regression to the prepared dataset, start by applying vanilla logistic regression from the sklearn library, with no regularisation. Build the model using all features of the dataset, train it on $80\\%$ of the rows. Print out a table of the coefficients for each feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uWUPNRkYSEG4"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "geU51AepSEG6"
   },
   "source": [
    "Set the threshold to $0.5$, and use the built model to predict outcomes of the remaining $20\\%$, plot the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GWOG_-DZSEG6"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UO2cr42USEG8"
   },
   "source": [
    "Because we are dealing with a balanced dataset, you can calculate the accuracy, specificity, and sensitivity for a number of thresholds between $0$ and $1$, plot the resulting values vs. threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Gno9qHvPSEG9"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R0UtmdkGSEG_"
   },
   "source": [
    "Can you use the above plot to estimate the best threshold value? Use it on the test set and print the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "as93JaTmSEHB"
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yFMqnHPjSEHC"
   },
   "source": [
    "*Advanced Problems*:\n",
    "    - It is ideal to include as few features as possible, look at AIC feature selection and include it in your model\n",
    "    - Use the ROC curve to choose your threshold value\\\n",
    "    - Cross-validate your results using the k-fold crossvalidation, make sure you are working with balanced subsets!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "lec4.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}