safe-loan-decision-tree-classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Identifying safe loans with decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [LendingClub](https://www.lendingclub.com/) is a peer-to-peer leading company that directly connects borrowers and potential lenders/investors. In this notebook, we build a classification model to predict whether or not a loan provided by LendingClub is likely to [default](https://en.wikipedia.org/wiki/Default_%28finance%29).\n",
    "\n",
    "In this notebook we will:\n",
    "\n",
    "* Use SFrames to do some feature engineering.\n",
    "* Train a decision-tree on the LendingClub dataset.\n",
    "* Visualize the tree.\n",
    "* Predict whether a loan will default along with prediction probabilities (on a validation set).\n",
    "* Train a complex tree model and compare it to simple tree model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fire up GraphLab Create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import graphlab\n",
    "graphlab.canvas.set_target('ipynb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load LendingClub dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using a dataset from the [LendingClub](https://www.lendingclub.com/). A parsed and cleaned form of the dataset is availiable [here](https://github.com/learnml/machine-learning-specialization-private).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] graphlab.cython.cy_server: GraphLab Create v2.1 started. Logging: /tmp/graphlab_server_1529869505.log\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This non-commercial license of GraphLab Create for academic use is assigned to nanlee_89@yahoo.com and will expire on December 07, 2018.\n"
     ]
    }
   ],
   "source": [
    "loans = graphlab.SFrame('lending-club-data.gl/')\n",
    "graphlab.canvas.set_target('ipynb')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring some features\n",
    "\n",
    "Quickly explore what the dataset looks like. First, print out the column names to see what features we have in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['id', 'member_id', 'loan_amnt', 'funded_amnt', 'funded_amnt_inv', 'term', 'int_rate', 'installment', 'grade', 'sub_grade', 'emp_title', 'emp_length', 'home_ownership', 'annual_inc', 'is_inc_v', 'issue_d', 'loan_status', 'pymnt_plan', 'url', 'desc', 'purpose', 'title', 'zip_code', 'addr_state', 'dti', 'delinq_2yrs', 'earliest_cr_line', 'inq_last_6mths', 'mths_since_last_delinq', 'mths_since_last_record', 'open_acc', 'pub_rec', 'revol_bal', 'revol_util', 'total_acc', 'initial_list_status', 'out_prncp', 'out_prncp_inv', 'total_pymnt', 'total_pymnt_inv', 'total_rec_prncp', 'total_rec_int', 'total_rec_late_fee', 'recoveries', 'collection_recovery_fee', 'last_pymnt_d', 'last_pymnt_amnt', 'next_pymnt_d', 'last_credit_pull_d', 'collections_12_mths_ex_med', 'mths_since_last_major_derog', 'policy_code', 'not_compliant', 'status', 'inactive_loans', 'bad_loans', 'emp_length_num', 'grade_num', 'sub_grade_num', 'delinq_2yrs_zero', 'pub_rec_zero', 'collections_12_mths_zero', 'short_emp', 'payment_inc_ratio', 'final_d', 'last_delinq_none', 'last_record_none', 'last_major_derog_none']\n"
     ]
    }
   ],
   "source": [
    "print loans.column_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we have some feature columns that have to do with grade of the loan, annual income, home ownership status, etc. Secondly, we take a look at the distribution of loan grades in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.1.0/css/font-awesome.min.css\"\n",
       "}));\n",
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"https://static.turi.com/products/graphlab-create/2.1/canvas/css/canvas.css\"\n",
       "}));\n",
       "\n",
       "            (function(){\n",
       "\n",
       "                var e = null;\n",
       "                if (typeof element == 'undefined') {\n",
       "                    var scripts = document.getElementsByTagName('script');\n",
       "                    var thisScriptTag = scripts[scripts.length-1];\n",
       "                    var parentDiv = thisScriptTag.parentNode;\n",
       "                    e = document.createElement('div');\n",
       "                    parentDiv.appendChild(e);\n",
       "                } else {\n",
       "                    e = element[0];\n",
       "                }\n",
       "\n",
       "                if (typeof requirejs !== 'undefined') {\n",
       "                    // disable load timeout; ipython_app.js is large and can take a while to load.\n",
       "                    requirejs.config({waitSeconds: 0});\n",
       "                }\n",
       "\n",
       "                require(['https://static.turi.com/products/graphlab-create/2.1/canvas/js/ipython_app.js'], function(IPythonApp){\n",
       "                    var app = new IPythonApp();\n",
       "                    app.attachView('sarray','Categorical', {\"ipython\": true, \"sketch\": {\"complete\": true, \"numeric\": false, \"num_unique\": 7, \"num_undefined\": 0, \"progress\": 1.0, \"frequent_items\": {\"A\": {\"frequency\": 22314, \"value\": \"A\"}, \"C\": {\"frequency\": 29950, \"value\": \"C\"}, \"B\": {\"frequency\": 37172, \"value\": \"B\"}, \"E\": {\"frequency\": 8990, \"value\": \"E\"}, \"D\": {\"frequency\": 19175, \"value\": \"D\"}, \"G\": {\"frequency\": 1074, \"value\": \"G\"}, \"F\": {\"frequency\": 3932, \"value\": \"F\"}}, \"size\": 122607}, \"selected_variable\": {\"name\": [\"<SArray>\"], \"dtype\": \"str\", \"view_component\": \"Categorical\", \"view_file\": \"sarray\", \"descriptives\": {\"rows\": 122607}, \"type\": \"SArray\", \"view_components\": [\"Categorical\"]}, \"histogram\": null}, e);\n",
       "                });\n",
       "            })();\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loans['grade'].show(view='Categorical')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that over half of the loan grades are assigned values `B` or `C`. Each loan is assigned one of these grades, along with a more finely discretized feature called `sub_grade` . These values depend on the loan application and credit report, and determine the interest rate of the loan. More information can be found [here](https://www.lendingclub.com/public/rates-and-fees.action).\n",
    "\n",
    "Take a look at a different feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.1.0/css/font-awesome.min.css\"\n",
       "}));\n",
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"https://static.turi.com/products/graphlab-create/2.1/canvas/css/canvas.css\"\n",
       "}));\n",
       "\n",
       "            (function(){\n",
       "\n",
       "                var e = null;\n",
       "                if (typeof element == 'undefined') {\n",
       "                    var scripts = document.getElementsByTagName('script');\n",
       "                    var thisScriptTag = scripts[scripts.length-1];\n",
       "                    var parentDiv = thisScriptTag.parentNode;\n",
       "                    e = document.createElement('div');\n",
       "                    parentDiv.appendChild(e);\n",
       "                } else {\n",
       "                    e = element[0];\n",
       "                }\n",
       "\n",
       "                if (typeof requirejs !== 'undefined') {\n",
       "                    // disable load timeout; ipython_app.js is large and can take a while to load.\n",
       "                    requirejs.config({waitSeconds: 0});\n",
       "                }\n",
       "\n",
       "                require(['https://static.turi.com/products/graphlab-create/2.1/canvas/js/ipython_app.js'], function(IPythonApp){\n",
       "                    var app = new IPythonApp();\n",
       "                    app.attachView('sarray','Categorical', {\"ipython\": true, \"sketch\": {\"complete\": true, \"numeric\": false, \"num_unique\": 4, \"num_undefined\": 0, \"progress\": 1.0, \"frequent_items\": {\"OWN\": {\"frequency\": 9943, \"value\": \"OWN\"}, \"OTHER\": {\"frequency\": 179, \"value\": \"OTHER\"}, \"RENT\": {\"frequency\": 53245, \"value\": \"RENT\"}, \"MORTGAGE\": {\"frequency\": 59240, \"value\": \"MORTGAGE\"}}, \"size\": 122607}, \"selected_variable\": {\"name\": [\"<SArray>\"], \"dtype\": \"str\", \"view_component\": \"Categorical\", \"view_file\": \"sarray\", \"descriptives\": {\"rows\": 122607}, \"type\": \"SArray\", \"view_components\": [\"Categorical\"]}, \"histogram\": null}, e);\n",
       "                });\n",
       "            })();\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loans['home_ownership'].show(view='Categorical')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This feature describes whether the loanee is mortaging, renting, or owns a home. We can see that a small percentage of the loanees own a home."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the target column\n",
    "\n",
    "The target column (label column) of the dataset that we are interested in is `bad_loans`. In this column **1** means a risky (bad) loan **0** means a safe  loan.\n",
    "\n",
    "In order to make this more intuitive and consistent with the lectures, we reassign the target to be:\n",
    "* **+1** as a safe  loan, \n",
    "* **-1** as a risky (bad) loan. \n",
    "\n",
    "We put this in a new column called `safe_loans`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# safe_loans =  1 => safe\n",
    "# safe_loans = -1 => risky\n",
    "loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)\n",
    "loans = loans.remove_column('bad_loans')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explore the distribution of the column `safe_loans`. This gives us a sense of how many safe and risky loans are present in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.1.0/css/font-awesome.min.css\"\n",
       "}));\n",
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"https://static.turi.com/products/graphlab-create/2.1/canvas/css/canvas.css\"\n",
       "}));\n",
       "\n",
       "            (function(){\n",
       "\n",
       "                var e = null;\n",
       "                if (typeof element == 'undefined') {\n",
       "                    var scripts = document.getElementsByTagName('script');\n",
       "                    var thisScriptTag = scripts[scripts.length-1];\n",
       "                    var parentDiv = thisScriptTag.parentNode;\n",
       "                    e = document.createElement('div');\n",
       "                    parentDiv.appendChild(e);\n",
       "                } else {\n",
       "                    e = element[0];\n",
       "                }\n",
       "\n",
       "                if (typeof requirejs !== 'undefined') {\n",
       "                    // disable load timeout; ipython_app.js is large and can take a while to load.\n",
       "                    requirejs.config({waitSeconds: 0});\n",
       "                }\n",
       "\n",
       "                require(['https://static.turi.com/products/graphlab-create/2.1/canvas/js/ipython_app.js'], function(IPythonApp){\n",
       "                    var app = new IPythonApp();\n",
       "                    app.attachView('sarray','Categorical', {\"ipython\": true, \"sketch\": {\"std\": 0.7827226562473063, \"complete\": true, \"min\": -1.0, \"max\": 1.0, \"quantile\": [-1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"median\": 1.0, \"numeric\": true, \"num_unique\": 2, \"num_undefined\": 0, \"var\": 0.6126547566028389, \"progress\": 1.0, \"size\": 122607, \"frequent_items\": {\"1\": {\"frequency\": 99457, \"value\": 1}, \"-1\": {\"frequency\": 23150, \"value\": -1}}, \"mean\": 0.6223706639914548}, \"selected_variable\": {\"name\": [\"<SArray>\"], \"dtype\": \"int\", \"view_component\": \"Categorical\", \"view_file\": \"sarray\", \"descriptives\": {\"rows\": 122607}, \"type\": \"SArray\", \"view_components\": [\"Numeric\", \"Categorical\"]}, \"histogram\": {\"progress\": 1.0, \"histogram\": {\"max\": 1.0, \"bins\": [23150, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 99457], \"min\": -1.0}, \"min\": -1, \"complete\": 1, \"max\": 1}}, e);\n",
       "                });\n",
       "            })();\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loans['safe_loans'].show(view = 'Categorical')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have:\n",
    "* Around 81% safe loans\n",
    "* Around 19% risky loans\n",
    "\n",
    "It looks like most of these loans are safe loans. But this does make our problem of identifying risky loans challenging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features for the classification algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will be using a subset of features (categorical and numeric). The features we will be using are **described in the code comments** below. Also, The [LendingClub](https://www.lendingclub.com/) website has a lot more details about these features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "features = ['grade',                     # grade of the loan\n",
    "            'sub_grade',                 # sub-grade of the loan\n",
    "            'short_emp',                 # one year or less of employment\n",
    "            'emp_length_num',            # number of years of employment\n",
    "            'home_ownership',            # home_ownership status: own, mortgage or rent\n",
    "            'dti',                       # debt to income ratio\n",
    "            'purpose',                   # the purpose of the loan\n",
    "            'term',                      # the term of the loan\n",
    "            'last_delinq_none',          # has borrower had a delinquincy\n",
    "            'last_major_derog_none',     # has borrower had 90 day or worse rating\n",
    "            'revol_util',                # percent of available credit being used\n",
    "            'total_rec_late_fee',        # total late fees received to day\n",
    "           ]\n",
    "\n",
    "target = 'safe_loans'                   # prediction target (y) (+1 means safe, -1 is risky)\n",
    "\n",
    "# Extract the feature columns and target column\n",
    "loans = loans[features + [target]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample data to balance classes\n",
    "\n",
    "As we explored above, our data is disproportionally full of safe loans.  In order to balance the classes, we first create two datasets: one with just the safe loans (`safe_loans_raw`) and one with just the risky loans (`risky_loans_raw`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of safe loans  : 99457\n",
      "Number of risky loans : 23150\n"
     ]
    }
   ],
   "source": [
    "safe_loans_raw = loans[loans[target] == +1]\n",
    "risky_loans_raw = loans[loans[target] == -1]\n",
    "print \"Number of safe loans  : %s\" % len(safe_loans_raw)\n",
    "print \"Number of risky loans : %s\" % len(risky_loans_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, compute the percentage of safe and risky loans in the dataset and validate these numbers against what was calculated earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of safe loans  : 0.811185331996\n",
      "Percentage of risky loans : 0.188814668004\n"
     ]
    }
   ],
   "source": [
    "print \"Percentage of safe loans  :\", float(len(safe_loans_raw))/len(loans)\n",
    "print \"Percentage of risky loans :\", float(len(risky_loans_raw))/len(loans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to combat class imbalance is to undersample the larger class until the class distribution is approximately half and half. Here, we will undersample the larger class (safe loans) in order to balance out our dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Since there are fewer risky loans than safe loans, find the ratio of the sizes\n",
    "# and use that percentage to undersample the safe loans.\n",
    "percentage = len(risky_loans_raw)/float(len(safe_loans_raw))\n",
    "\n",
    "risky_loans = risky_loans_raw\n",
    "safe_loans = safe_loans_raw.sample(percentage, seed=1)\n",
    "\n",
    "# Append the risky_loans with the downsampled version of safe_loans\n",
    "loans_data = risky_loans.append(safe_loans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that the resulting percentage of safe and risky loans are each nearly 50%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of safe loans                 : 0.502236174422\n",
      "Percentage of risky loans                : 0.497763825578\n",
      "Total number of loans in our new dataset : 46508\n"
     ]
    }
   ],
   "source": [
    "print \"Percentage of safe loans                 :\", len(safe_loans) / float(len(loans_data))\n",
    "print \"Percentage of risky loans                :\", len(risky_loans) / float(len(loans_data))\n",
    "print \"Total number of loans in our new dataset :\", len(loans_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split data into training and validation sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the data into training and validation sets using an 80/20 split and specifying `seed=1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_data, validation_data = loans_data.random_split(.8, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use decision tree to build a classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the built-in GraphLab Create decision tree learner to create a loan prediction model on the training data. Our feature columns and target column have already been decided above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decision_tree_model = graphlab.decision_tree_classifier.create(train_data, validation_set=None,\n",
    "                                target = target, features = features, verbose= False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing a learned model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As noted in the [documentation](https://dato.com/products/create/docs/generated/graphlab.boosted_trees_classifier.create.html#graphlab.boosted_trees_classifier.create), typically the max depth of the tree is capped at 6. However, such a tree can be hard to visualize graphically.  Here, we instead learn a smaller model with **max depth of 2** to gain some intuition by visualizing the learned tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "small_model = graphlab.decision_tree_classifier.create(train_data, validation_set=None,\n",
    "                   target = target, features = features, max_depth = 2, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** To better understand this visual:\n",
    "* The root node is represented using pink. \n",
    "* Intermediate nodes are in green. \n",
    "* Leaf nodes in blue and orange. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.1.0/css/font-awesome.min.css\"\n",
       "}));\n",
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"https://static.turi.com/products/graphlab-create/2.1/canvas/css/canvas.css\"\n",
       "}));\n",
       "\n",
       "            (function(){\n",
       "\n",
       "                var e = null;\n",
       "                if (typeof element == 'undefined') {\n",
       "                    var scripts = document.getElementsByTagName('script');\n",
       "                    var thisScriptTag = scripts[scripts.length-1];\n",
       "                    var parentDiv = thisScriptTag.parentNode;\n",
       "                    e = document.createElement('div');\n",
       "                    parentDiv.appendChild(e);\n",
       "                } else {\n",
       "                    e = element[0];\n",
       "                }\n",
       "\n",
       "                if (typeof requirejs !== 'undefined') {\n",
       "                    // disable load timeout; ipython_app.js is large and can take a while to load.\n",
       "                    requirejs.config({waitSeconds: 0});\n",
       "                }\n",
       "\n",
       "                require(['https://static.turi.com/products/graphlab-create/2.1/canvas/js/ipython_app.js'], function(IPythonApp){\n",
       "                    var app = new IPythonApp();\n",
       "                    app.attachView('sgraph','View', {\"edges_labels\": [\"yes\", \"no\", \"yes\", \"no\", \"no\", \"yes\"], \"selected_variable\": {\"name\": [\"<SGraph>\"], \"view_file\": \"sgraph\", \"view_component\": \"View\", \"view_params\": {\"elabel_hover\": false, \"vertex_positions\": null, \"h_offset\": 0.0, \"node_size\": 300, \"ecolor\": [0.37, 0.33, 0.33], \"elabel\": \"value\", \"arrows\": true, \"ewidth\": 1, \"vlabel\": \"__repr__\", \"highlight_color\": [0.69, 0.0, 0.498], \"vcolor\": [0.522, 0.741, 0.0], \"vlabel_hover\": false, \"highlight\": {\"0\": [0.69, 0.0, 0.48], \"3\": [0.039, 0.55, 0.77], \"4\": [1.0, 0.33, 0.0], \"5\": [1.0, 0.33, 0.0], \"6\": [0.039, 0.55, 0.77]}, \"v_offset\": 0.03}, \"view_components\": [\"View\"], \"type\": \"SGraph\", \"descriptives_links\": {\"edges\": \"edges\", \"vertices\": \"vertices\"}, \"descriptives\": {\"edges\": 6, \"vertices\": 7}}, \"positions\": null, \"error_type\": 0, \"vertices\": [5, 0, 2, 6, 3, 1, 4], \"vertices_labels\": [\"0.3223\", \"grade=A\", \"total_rec_late_fee<14.83015\", \"-0.273469\", \"-0.111044\", \"grade=B\", \"0.097203\"], \"edges\": [[0, 2], [0, 1], [2, 5], [2, 6], [1, 3], [1, 4]], \"ipython\": true, \"error_msg\": \"\"}, e);\n",
       "                });\n",
       "            })();\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "small_model.show(view=\"Tree\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making predictions\n",
    "\n",
    "Consider two positive and two negative examples **from the validation set** and see what the model predicts. We will do the following:\n",
    "* Predict whether or not a loan is safe.\n",
    "* Predict the probability that a loan is safe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sub_grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">short_emp</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">emp_length_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">home_ownership</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dti</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">purpose</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">term</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_delinq_none</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">OWN</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11.18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.85</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.97</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">other</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 60 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.33</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_major_derog_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">revol_util</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_late_fee</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">safe_loans</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">82.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">96.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">62.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[4 rows x 13 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tgrade\tstr\n",
       "\tsub_grade\tstr\n",
       "\tshort_emp\tint\n",
       "\temp_length_num\tint\n",
       "\thome_ownership\tstr\n",
       "\tdti\tfloat\n",
       "\tpurpose\tstr\n",
       "\tterm\tstr\n",
       "\tlast_delinq_none\tint\n",
       "\tlast_major_derog_none\tint\n",
       "\trevol_util\tfloat\n",
       "\ttotal_rec_late_fee\tfloat\n",
       "\tsafe_loans\tint\n",
       "\n",
       "Rows: 4\n",
       "\n",
       "Data:\n",
       "+-------+-----------+-----------+----------------+----------------+-------+\n",
       "| grade | sub_grade | short_emp | emp_length_num | home_ownership |  dti  |\n",
       "+-------+-----------+-----------+----------------+----------------+-------+\n",
       "|   B   |     B3    |     0     |       11       |      OWN       | 11.18 |\n",
       "|   D   |     D1    |     0     |       10       |      RENT      | 16.85 |\n",
       "|   D   |     D2    |     0     |       3        |      RENT      | 13.97 |\n",
       "|   A   |     A5    |     0     |       11       |    MORTGAGE    | 16.33 |\n",
       "+-------+-----------+-----------+----------------+----------------+-------+\n",
       "+--------------------+------------+------------------+-----------------------+\n",
       "|      purpose       |    term    | last_delinq_none | last_major_derog_none |\n",
       "+--------------------+------------+------------------+-----------------------+\n",
       "|    credit_card     |  36 months |        1         |           1           |\n",
       "| debt_consolidation |  36 months |        1         |           1           |\n",
       "|       other        |  60 months |        0         |           1           |\n",
       "| debt_consolidation |  36 months |        1         |           1           |\n",
       "+--------------------+------------+------------------+-----------------------+\n",
       "+------------+--------------------+------------+\n",
       "| revol_util | total_rec_late_fee | safe_loans |\n",
       "+------------+--------------------+------------+\n",
       "|    82.4    |        0.0         |     1      |\n",
       "|    96.4    |        0.0         |     1      |\n",
       "|    59.5    |        0.0         |     -1     |\n",
       "|    62.1    |        0.0         |     -1     |\n",
       "+------------+--------------------+------------+\n",
       "[4 rows x 13 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_safe_loans = validation_data[validation_data[target] == 1]\n",
    "validation_risky_loans = validation_data[validation_data[target] == -1]\n",
    "\n",
    "sample_validation_data_risky = validation_risky_loans[0:2]\n",
    "sample_validation_data_safe = validation_safe_loans[0:2]\n",
    "\n",
    "sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)\n",
    "sample_validation_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore label predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will use our model  to predict whether or not a loan is likely to default. For each row in the **sample_validation_data**, use the **decision_tree_model** to predict whether or not the loan is classified as a **safe loan**. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sample_validation_predictions = decision_tree_model.predict(sample_validation_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The percentage of the predictions on `sample_validation_data` did `decision_tree_model` get correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum([sample_validation_data['safe_loans']==sample_validation_predictions])/float(len(sample_validation_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore probability predictions\n",
    "\n",
    "For each row in the **sample_validation_data**, the probability (according **decision_tree_model**) of a loan being classified as **safe**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype: float\n",
       "Rows: 4\n",
       "[0.5473502278327942, 0.4891221821308136, 0.4559234082698822, 0.5864479541778564]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decision_tree_model.predict(sample_validation_data, output_type='probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tricky predictions!\n",
    "\n",
    "For each row in the **sample_validation_data**, the probability (according to **small_model**) of a loan being classified as **safe**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype: float\n",
       "Rows: 4\n",
       "[0.5242817997932434, 0.47226759791374207, 0.47226759791374207, 0.5798847675323486]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "small_model.predict(sample_validation_data, output_type='probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the prediction on a tree\n",
    "\n",
    "Consider the following point in the **sample_validation_data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'safe_loans': 1, 'dti': 16.85, 'emp_length_num': 10, 'revol_util': 96.4, 'grade': 'D', 'term': ' 36 months', 'sub_grade': 'D1', 'purpose': 'debt_consolidation', 'short_emp': 0, 'home_ownership': 'RENT', 'total_rec_late_fee': 0.0, 'last_major_derog_none': 1, 'last_delinq_none': 1}\n"
     ]
    }
   ],
   "source": [
    "print sample_validation_data[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the small tree here to do the traversing for this data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.1.0/css/font-awesome.min.css\"\n",
       "}));\n",
       "$(\"head\").append($(\"<link/>\").attr({\n",
       "  rel:  \"stylesheet\",\n",
       "  type: \"text/css\",\n",
       "  href: \"https://static.turi.com/products/graphlab-create/2.1/canvas/css/canvas.css\"\n",
       "}));\n",
       "\n",
       "            (function(){\n",
       "\n",
       "                var e = null;\n",
       "                if (typeof element == 'undefined') {\n",
       "                    var scripts = document.getElementsByTagName('script');\n",
       "                    var thisScriptTag = scripts[scripts.length-1];\n",
       "                    var parentDiv = thisScriptTag.parentNode;\n",
       "                    e = document.createElement('div');\n",
       "                    parentDiv.appendChild(e);\n",
       "                } else {\n",
       "                    e = element[0];\n",
       "                }\n",
       "\n",
       "                if (typeof requirejs !== 'undefined') {\n",
       "                    // disable load timeout; ipython_app.js is large and can take a while to load.\n",
       "                    requirejs.config({waitSeconds: 0});\n",
       "                }\n",
       "\n",
       "                require(['https://static.turi.com/products/graphlab-create/2.1/canvas/js/ipython_app.js'], function(IPythonApp){\n",
       "                    var app = new IPythonApp();\n",
       "                    app.attachView('sgraph','View', {\"edges_labels\": [\"yes\", \"no\", \"yes\", \"no\", \"no\", \"yes\"], \"selected_variable\": {\"name\": [\"<SGraph>\"], \"view_file\": \"sgraph\", \"view_component\": \"View\", \"view_params\": {\"elabel_hover\": false, \"vertex_positions\": null, \"h_offset\": 0.0, \"node_size\": 300, \"ecolor\": [0.37, 0.33, 0.33], \"elabel\": \"value\", \"arrows\": true, \"ewidth\": 1, \"vlabel\": \"__repr__\", \"highlight_color\": [0.69, 0.0, 0.498], \"vcolor\": [0.522, 0.741, 0.0], \"vlabel_hover\": false, \"highlight\": {\"0\": [0.69, 0.0, 0.48], \"3\": [0.039, 0.55, 0.77], \"4\": [1.0, 0.33, 0.0], \"5\": [1.0, 0.33, 0.0], \"6\": [0.039, 0.55, 0.77]}, \"v_offset\": 0.03}, \"view_components\": [\"View\"], \"type\": \"SGraph\", \"descriptives_links\": {\"edges\": \"edges\", \"vertices\": \"vertices\"}, \"descriptives\": {\"edges\": 6, \"vertices\": 7}}, \"positions\": null, \"error_type\": 0, \"vertices\": [5, 0, 2, 6, 3, 1, 4], \"vertices_labels\": [\"0.3223\", \"grade=A\", \"total_rec_late_fee<14.83015\", \"-0.273469\", \"-0.111044\", \"grade=B\", \"0.097203\"], \"edges\": [[0, 2], [0, 1], [2, 5], [2, 6], [1, 3], [1, 4]], \"ipython\": true, \"error_msg\": \"\"}, e);\n",
       "                });\n",
       "            })();\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "small_model.show(view=\"Tree\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** In the tree visualization above, the values at the leaf nodes are not class predictions but scores (a slightly advanced concept that is out of the scope of this course). You can read more about this [here](https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf).  If the score is $\\geq$ 0, the class +1 is predicted.  Otherwise, if the score < 0, we predict class -1.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating accuracy of the decision tree model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy is defined as follows:\n",
    "$$\n",
    "\\mbox{accuracy} = \\frac{\\mbox{# correctly classified examples}}{\\mbox{# total examples}}\n",
    "$$\n",
    "\n",
    "Get started by evaluating the accuracy of the `small_model` and `decision_tree_model` on the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.613502041694\n",
      "0.640581345369\n"
     ]
    }
   ],
   "source": [
    "print small_model.evaluate(train_data)['accuracy']\n",
    "print decision_tree_model.evaluate(train_data)['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** We can see that the **small_model** performs worse than the **decision_tree_model** on the training data.\n",
    "\n",
    "\n",
    "Now, let us evaluate the accuracy of the **small_model** and **decision_tree_model** on the entire **validation_data**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.636794485136\n",
      "0.619345109866\n"
     ]
    }
   ],
   "source": [
    "print decision_tree_model.evaluate(validation_data)['accuracy']\n",
    "print small_model.evaluate(validation_data)['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating accuracy of a complex decision tree model\n",
    "\n",
    "Here, we will train a large decision tree with `max_depth=10`. This will allow the learned tree to become very deep, and result in a very complex model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "big_model = graphlab.decision_tree_classifier.create(train_data, validation_set=None,\n",
    "                   target = target, features = features, max_depth = 10, verbose = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us evaluate **big_model** on the training set and validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.665538362347\n",
      "0.627423524343\n"
     ]
    }
   ],
   "source": [
    "print big_model.evaluate(train_data)['accuracy']\n",
    "print big_model.evaluate(validation_data)['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** We should see that **big_model** has even better performance on the training set than **decision_tree_model** did on the training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantifying the cost of mistakes\n",
    "\n",
    "Every mistake the model makes costs money. In this section, we will try and quantify the cost of each mistake made by the model.\n",
    "\n",
    "Assume the following:\n",
    "\n",
    "* **False negatives**: Loans that were actually safe but were predicted to be risky. This results in an oppurtunity cost of losing a loan that would have otherwise been accepted. \n",
    "* **False positives**: Loans that were actually risky but were predicted to be safe. These are much more expensive because it results in a risky loan being given. \n",
    "* **Correct predictions**: All correct predictions don't typically incur any cost.\n",
    "\n",
    "\n",
    "The following code can compute the cost of mistakes made by the model and complete the following 4 steps:\n",
    "1. First, compute the predictions made by the model.\n",
    "1. Second, compute the number of false positives.\n",
    "2. Third, compute the number of false negatives.\n",
    "3. Finally, compute the cost of mistakes made by the model by adding up the costs of true positives and false positives."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we make predictions on `validation_data` using the `decision_tree_model`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1, 1, -1, -1, 1, -1, 1, 1, -1, -1, -1, 1, 1, -1, 1, 1, 1, -1, -1, -1, 1, 1, -1, -1, -1, 1, -1, -1, 1, -1, -1, -1, -1, 1, 1, -1, 1, -1, 1, -1, 1, 1, 1, 1, 1, -1, 1, -1, 1, 1, -1, -1, -1, -1, -1, 1, 1, 1, -1, -1, 1, -1, -1, -1, 1, 1, -1, -1, -1, -1, -1, 1, -1, 1, -1, -1, -1, 1, -1, -1, -1, -1, 1, 1, -1, 1, 1, -1, -1, 1, -1, 1, -1, -1, -1, 1, -1, -1, -1, -1, ... ]\n"
     ]
    }
   ],
   "source": [
    "predictions = decision_tree_model.predict(validation_data)\n",
    "print predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**False positives** are predictions where the model predicts +1 but the true label is -1. The following code block for the number of false positives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1656\n"
     ]
    }
   ],
   "source": [
    "false_positives = np.sum([(predictions != validation_data['safe_loans']) & (predictions==+1)])\n",
    "print false_positives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**False negatives** are predictions where the model predicts -1 but the true label is +1. The following code block for the number of false negatives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1716\n"
     ]
    }
   ],
   "source": [
    "false_negatives = np.sum([(predictions != validation_data['safe_loans']) & (predictions==-1)])\n",
    "print false_negatives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us assume that each mistake costs money:\n",
    "* Assume a cost of \\$10,000 per false negative.\n",
    "* Assume a cost of \\$20,000 per false positive.\n",
    "\n",
    "The total cost of mistakes made by `decision_tree_model` on `validation_data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cost_of_mistakes = (false_negatives * 10000) + (false_positives * 20000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50280000"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cost_of_mistakes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}