msan621_ml1/MakingYourOwnTree.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import IPython\n",
    "import graphviz\n",
    "import re\n",
    "from IPython.display import display\n",
    "from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor, export_graphviz\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Decision Tree by Hand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For our baby case lets work through a tiny dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = pd.DataFrame({'feature':[1,2,3,4,5,6]})\n",
    "x_vec = np.array([1,2,3,4,5,6], dtype=int)\n",
    "y = np.array([1,1,0,1,0,0], dtype=int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  The Decision Tree Algorithm - in English\n",
    "\n",
    "1. Target: Looking for a division of the dataset where mostly `1`'s on one side and mostly `0`'s on other side\n",
    "2. Will check every element and separate the set into a left handed group and a right handed group and calculate a 'purity' score\n",
    "3. will loop and search for the **lowest** score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i=1, Split at 1st element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ True False False False False False] [False  True  True  True  True  True] [1] [1 0 1 0 0]\n"
     ]
    }
   ],
   "source": [
    "lhs_x = x_vec<=x_vec[0]\n",
    "rhs_x = x_vec>x_vec[0]\n",
    "lhs_y = y[lhs_x]\n",
    "rhs_y = y[rhs_x]\n",
    "\n",
    "print(lhs_x, rhs_x,lhs_y,rhs_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i=1, Calculate the purity score\n",
    "\n",
    "$$\\sigma_{ylhs}n_{xlhs} + \\sigma_{yrhs}n_{xrhs} $$\n",
    "\n",
    "The score is roughly the standard deviation of the y times the number of samples. Lets consider the extremes:\n",
    "\n",
    "1. If the split is perfect, one side will have all 1's and the other will have all 0's, the deviations will equal =0 and the score will be zero\n",
    "2. If the split is terrible, we will have a huge deviation times a large number of points that will give a very large score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.4494897427831783"
      ]
     },
     "execution_count": 265,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.std(lhs_y) * sum(lhs_x) + np.std(rhs_y) * sum(rhs_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i=2, Split at 2nd element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ True  True False False False False] [False False  True  True  True  True] [1 1] [0 1 0 0]\n"
     ]
    }
   ],
   "source": [
    "lhs_x = x_vec<=x_vec[1]\n",
    "rhs_x = x_vec>x_vec[1]\n",
    "lhs_y = y[lhs_x]\n",
    "rhs_y = y[rhs_x]\n",
    "\n",
    "print(lhs_x, rhs_x,lhs_y,rhs_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i=2, Calculate the score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.7320508075688772"
      ]
     },
     "execution_count": 267,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.std(lhs_y) * sum(lhs_x) + np.std(rhs_y) * sum(rhs_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i=3, Split at 3rd element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ True  True  True False False False] [False False False False  True  True] [1 1 0] [0 0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.4142135623730949"
      ]
     },
     "execution_count": 268,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lhs_x = x_vec<=x_vec[2]\n",
    "rhs_x = x_vec>x_vec[3]\n",
    "lhs_y = y[lhs_x]\n",
    "rhs_y = y[rhs_x]\n",
    "\n",
    "print(lhs_x, rhs_x,lhs_y,rhs_y)\n",
    "np.std(lhs_y) * sum(lhs_x) + np.std(rhs_y) * sum(rhs_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion: it seems that element 2~3 is a good split, with 2 samples going left, and 4 going right!\n",
    "\n",
    "### Let's compare to sklearn! We will use 1 decision tree with depth 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mse', max_depth=1, max_features=None,\n",
       "           max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "           min_impurity_split=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           presort=False, random_state=None, splitter='best')"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtr = DecisionTreeRegressor(max_depth=1)\n",
    "dtr.fit(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tree Drawing function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "def draw_tree(t, df, size=10, ratio=0.6, precision=0):\n",
    "    \"\"\" Draws a representation of a random forest in IPython.\n",
    "    Parameters:\n",
    "    -----------\n",
    "    t: The tree you wish to draw\n",
    "    df: The data used to train the tree. This is used to get the names of the features.\n",
    "    \"\"\"\n",
    "    s=export_graphviz(t, out_file=None, feature_names=df.columns, filled=True,\n",
    "                      special_characters=True, rotate=True, precision=precision)\n",
    "    IPython.display.display(graphviz.Source(re.sub('Tree {',\n",
    "       f'Tree {{ size={size}; ratio={ratio}', s)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"217pt\" height=\"133pt\"\n",
       " viewBox=\"0.00 0.00 216.75 132.80\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 128.8)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-128.8 212.752,-128.8 212.752,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\n",
       "<polygon fill=\"#e58139\" fill-opacity=\"0.333333\" stroke=\"black\" points=\"87.291,-94.4407 0.212882,-94.4407 0.212882,-30.4407 87.291,-30.4407 87.291,-94.4407\"/>\n",
       "<text text-anchor=\"start\" x=\"8.23242\" y=\"-79.2407\" font-family=\"Times,serif\" font-size=\"14.00\">feature ≤ 2.5</text>\n",
       "<text text-anchor=\"start\" x=\"12.7783\" y=\"-65.2407\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 0.25</text>\n",
       "<text text-anchor=\"start\" x=\"10.2524\" y=\"-51.2407\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 6</text>\n",
       "<text text-anchor=\"start\" x=\"12.3955\" y=\"-37.2407\" font-family=\"Times,serif\" font-size=\"14.00\">value = 0.5</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\n",
       "<polygon fill=\"#e58139\" stroke=\"black\" points=\"207.251,-123.441 124.253,-123.441 124.253,-73.4407 207.251,-73.4407 207.251,-123.441\"/>\n",
       "<text text-anchor=\"start\" x=\"138.278\" y=\"-108.241\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 0.0</text>\n",
       "<text text-anchor=\"start\" x=\"132.252\" y=\"-94.2407\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 2</text>\n",
       "<text text-anchor=\"start\" x=\"134.396\" y=\"-80.2407\" font-family=\"Times,serif\" font-size=\"14.00\">value = 1.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M87.3973,-75.2343C96.0118,-77.8186 105.148,-80.5596 114.009,-83.2178\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"113.269,-86.6499 123.853,-86.1711 115.281,-79.9451 113.269,-86.6499\"/>\n",
       "<text text-anchor=\"middle\" x=\"101.841\" y=\"-93.8236\" font-family=\"Times,serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"208.465,-51.4407 123.039,-51.4407 123.039,-1.44068 208.465,-1.44068 208.465,-51.4407\"/>\n",
       "<text text-anchor=\"start\" x=\"131.278\" y=\"-36.2407\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 0.188</text>\n",
       "<text text-anchor=\"start\" x=\"132.252\" y=\"-22.2407\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 4</text>\n",
       "<text text-anchor=\"start\" x=\"130.896\" y=\"-8.24068\" font-family=\"Times,serif\" font-size=\"14.00\">value = 0.25</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M87.3973,-49.6471C95.6286,-47.1777 104.337,-44.5652 112.825,-42.0187\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"114.075,-45.298 122.647,-39.072 112.063,-38.5932 114.075,-45.298\"/>\n",
       "<text text-anchor=\"middle\" x=\"100.636\" y=\"-23.0195\" font-family=\"Times,serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1a1bbcc630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw_tree(dtr, x, precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We see we are consistent with sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# On to the real Data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('./train (1).csv')\n",
    "df_test = pd.read_csv('./test (1).csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For simplicity lets choose only a few features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Id', 'MSSubClass', 'MSZoning', 'LotFrontage', 'LotArea', 'Street',\n",
       "       'Alley', 'LotShape', 'LandContour', 'Utilities', 'LotConfig',\n",
       "       'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType',\n",
       "       'HouseStyle', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd',\n",
       "       'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType',\n",
       "       'MasVnrArea', 'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual',\n",
       "       'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinSF1',\n",
       "       'BsmtFinType2', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'Heating',\n",
       "       'HeatingQC', 'CentralAir', 'Electrical', '1stFlrSF', '2ndFlrSF',\n",
       "       'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath',\n",
       "       'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'KitchenQual',\n",
       "       'TotRmsAbvGrd', 'Functional', 'Fireplaces', 'FireplaceQu', 'GarageType',\n",
       "       'GarageYrBlt', 'GarageFinish', 'GarageCars', 'GarageArea', 'GarageQual',\n",
       "       'GarageCond', 'PavedDrive', 'WoodDeckSF', 'OpenPorchSF',\n",
       "       'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'PoolQC',\n",
       "       'Fence', 'MiscFeature', 'MiscVal', 'MoSold', 'YrSold', 'SaleType',\n",
       "       'SaleCondition', 'SalePrice'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's select YrSold, General Living Area, Total Rooms Above Ground"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>208500</td>\n",
       "      <td>2008</td>\n",
       "      <td>1710</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>181500</td>\n",
       "      <td>2007</td>\n",
       "      <td>1262</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>223500</td>\n",
       "      <td>2008</td>\n",
       "      <td>1786</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>140000</td>\n",
       "      <td>2006</td>\n",
       "      <td>1717</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>250000</td>\n",
       "      <td>2008</td>\n",
       "      <td>2198</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   SalePrice  YrSold  GrLivArea  TotRmsAbvGrd\n",
       "0     208500    2008       1710             8\n",
       "1     181500    2007       1262             6\n",
       "2     223500    2008       1786             6\n",
       "3     140000    2006       1717             7\n",
       "4     250000    2008       2198             9"
      ]
     },
     "execution_count": 229,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_df = train_df[['SalePrice','YrSold','GrLivArea','TotRmsAbvGrd']].copy()\n",
    "sample_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split into X and y variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = sample_df['SalePrice']\n",
    "X_train = sample_df[[x for x in sample_df.columns if x != 'SalePrice']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1460,) (1460, 3)\n"
     ]
    }
   ],
   "source": [
    "print(y_train.shape, X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make a Decision Tree with Depth 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mse', max_depth=1, max_features=None,\n",
       "           max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "           min_impurity_split=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           presort=False, random_state=None, splitter='best')"
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtr = DecisionTreeRegressor(max_depth=1)\n",
    "dtr.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets see what the Sklearn Decided"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"326pt\" height=\"198pt\"\n",
       " viewBox=\"0.00 0.00 325.89 198.20\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 194.2)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-194.2 321.895,-194.2 321.895,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\n",
       "<polygon fill=\"#e58139\" fill-opacity=\"0.474510\" stroke=\"black\" points=\"141.342,-127.297 0.447256,-127.297 0.447256,-63.2966 141.342,-63.2966 141.342,-127.297\"/>\n",
       "<text text-anchor=\"start\" x=\"13.5957\" y=\"-112.097\" font-family=\"Times,serif\" font-size=\"14.00\">GrLivArea ≤ 1488.0</text>\n",
       "<text text-anchor=\"start\" x=\"8.4209\" y=\"-98.0966\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 6306788585.35</text>\n",
       "<text text-anchor=\"start\" x=\"26.895\" y=\"-84.0966\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 1460</text>\n",
       "<text text-anchor=\"start\" x=\"22.0381\" y=\"-70.0966\" font-family=\"Times,serif\" font-size=\"14.00\">value = 180921.2</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"318.342,-175.297 177.447,-175.297 177.447,-125.297 318.342,-125.297 318.342,-175.297\"/>\n",
       "<text text-anchor=\"start\" x=\"185.421\" y=\"-160.097\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 1408525823.77</text>\n",
       "<text text-anchor=\"start\" x=\"207.395\" y=\"-146.097\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 767</text>\n",
       "<text text-anchor=\"start\" x=\"195.538\" y=\"-132.097\" font-family=\"Times,serif\" font-size=\"14.00\">value = 137687.23</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141.562,-117.192C150.08,-119.869 158.834,-122.62 167.481,-125.338\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"166.592,-128.727 177.181,-128.387 168.69,-122.049 166.592,-128.727\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.016\" y=\"-135.751\" font-family=\"Times,serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\n",
       "<polygon fill=\"#e58139\" stroke=\"black\" points=\"314.842,-65.2966 180.947,-65.2966 180.947,-15.2966 314.842,-15.2966 314.842,-65.2966\"/>\n",
       "<text text-anchor=\"start\" x=\"188.921\" y=\"-50.0966\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 7369649464.1</text>\n",
       "<text text-anchor=\"start\" x=\"207.395\" y=\"-36.0966\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 693</text>\n",
       "<text text-anchor=\"start\" x=\"195.538\" y=\"-22.0966\" font-family=\"Times,serif\" font-size=\"14.00\">value = 228771.78</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141.562,-73.4011C151.284,-70.3456 161.313,-67.1936 171.141,-64.1049\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"172.392,-67.3805 180.883,-61.0432 170.293,-60.7026 172.392,-67.3805\"/>\n",
       "<text text-anchor=\"middle\" x=\"158.718\" y=\"-45.2791\" font-family=\"Times,serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1a1b5c15f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw_tree(dtr, X_train, precision=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting Algorithm : First for one feature\n",
    "\n",
    "####  English\n",
    "1. Target: Looking for a division of the dataset where mostly 1's on one side and mostly 0's on other side\n",
    "2. Will check every element and separate the set into a left handed group and a right handed group and calculate a 'purity' score\n",
    "3. will loop and search for the lowest score\n",
    "\n",
    "#### Score Calculation:\n",
    "\n",
    "    - left_grp_Ytrain_deviation * left_Xtrain_sum + right_grp_Ytrain_deviation * right_Xtrain_sum\n",
    "\n",
    "#### Python Explicit Code: Setup\n",
    "- `feature_idx =1` : we can choose which feature we want to look at \n",
    "- `stored_score = float('inf')` : choose a very high starter score\n",
    "- `stored_split_feature = 0` this is redundant right now, but it would be the idea feature to split on\n",
    "- `stored_split_value = 0` - this is the actual data-frame value to split on, for us `GrLivArea` is in ft\n",
    "- `all_indexes = [x for x in X_train.index]` : makes list of all possible indexes\n",
    "- `feature_list = X_train.columns` - feature list of column names, for labeling\n",
    "\n",
    "#### Datasetup\n",
    "\n",
    "- `x,y = X_train.iloc[all_indexes,feature_idx],y_train.values[all_indexes]` pulls out the actual values\n",
    "\n",
    "#### Split logic: for a given index\n",
    "\n",
    "##### X values split\n",
    "- `lhs_x = x<=x[split_index]` : for whatever split value you are at, how many values are less\n",
    "- `rhs_x = x>x[split_index]` : how many values are above the selected value\n",
    "\n",
    "##### y values split\n",
    "- `lhs_y_std = y[lhs_x].std()` : the stdev of the left hand y values\n",
    "- `rhs_x_std = y[rhs_x].std()` : the stdev of the right hand y values\n",
    "\n",
    "##### Calc the score\n",
    "- `curr_score = lhs_y_std*lhs_x.sum() + rhs_x_std*rhs_x.sum()` \n",
    "\n",
    "##### If score is better, store the data\n",
    "\n",
    "```python\n",
    " if curr_score<stored_score: \n",
    "            print('split index :%d lhs ct: %d | rhs ct: %d| %f' % (split_index, \n",
    "                                                                   len(lhs_x),\n",
    "                                                                   len(rhs_x), \n",
    "                                                                   curr_score))\n",
    "            stored_split_feature = feature_idx\n",
    "            stored_score = curr_score\n",
    "            stored_split_value = x[split_index]\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A manual loop "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start loop\n",
      "split index :1 lhs ct: 512 | rhs ct: 948| score: 93281398.205757\n",
      "split index :2 lhs ct: 1101 | rhs ct: 359| score: 91804469.340026\n",
      "split index :3 lhs ct: 1037 | rhs ct: 423| score: 91487146.514272\n",
      "split index :5 lhs ct: 618 | rhs ct: 842| score: 90970171.513650\n",
      "split index :6 lhs ct: 1008 | rhs ct: 452| score: 90944631.172519\n",
      "split index :13 lhs ct: 778 | rhs ct: 682| score: 88899549.318965\n",
      "split index :67 lhs ct: 755 | rhs ct: 705| score: 88506980.330170\n",
      "split index :239 lhs ct: 767 | rhs ct: 693| score: 88277501.801643\n",
      "88277501.8016 GrLivArea 1487\n"
     ]
    }
   ],
   "source": [
    "feature_idx =1\n",
    "stored_score = float('inf')\n",
    "stored_split_feature = 0\n",
    "stored_split_value = 0\n",
    "\n",
    "all_indexes = [x for x in X_train.index]\n",
    "\n",
    "feature_list = X_train.columns\n",
    "x,y = X_train.iloc[all_indexes,feature_idx],y_train.values[all_indexes]\n",
    "\n",
    "print('start loop')\n",
    "for split_index in range(1, x.shape[0]-1):\n",
    "    lhs_x = x<=x[split_index]\n",
    "    rhs_x = x>x[split_index]\n",
    "\n",
    "    if rhs_x.sum()==0:\n",
    "        continue\n",
    "    else:\n",
    "        lhs_y_std = y[lhs_x].std()\n",
    "        rhs_y_std = y[rhs_x].std()\n",
    "\n",
    "        curr_score = lhs_y_std*lhs_x.sum() + rhs_y_std*rhs_x.sum()\n",
    "        \n",
    "        if curr_score<stored_score: \n",
    "            print('split index :%d lhs ct: %d | rhs ct: %d| score: %f' % (split_index, sum(lhs_x),sum(rhs_x), curr_score))\n",
    "            stored_split_feature = feature_idx\n",
    "            stored_score = curr_score\n",
    "            stored_split_value = x[split_index]\n",
    "\n",
    "\n",
    "print(stored_score, feature_list[stored_split_feature], stored_split_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lets turn it into a class with a single function\n",
    "\n",
    "- **Tricky hard to change `variables` to `self.variables`:** be sure to keep track of these, its hard to find and replace all of them. I highly recommend restartting your kernel to make sure old variable values don't remain from a previous calculation and then fail to throw an error when prototyping your tree \n",
    "\n",
    "- **Dont repeat yourself** try to streamline as much as possible\n",
    "\n",
    "- **Don't send data to functions in the same class**: when acting on the data, always use the `self.` versions. The persistence between functions (all functions can access the same data) will help avoid version issues and naming problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class myIndecisionTree():\n",
    "    def __init__(self,x,y):\n",
    "        self.indexes = x.index\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        self.feature_idx =1\n",
    "        self.stored_score = float('inf')\n",
    "        self.stored_split_feature = 0\n",
    "        self.stored_split_value = 0\n",
    "        self.feature_list = x.columns\n",
    "    \n",
    "    def find_split_in_single_feature(self, feature_idx):\n",
    "        X_train = self.x\n",
    "        y_train = self.y\n",
    "\n",
    "        x,y = X_train.iloc[self.indexes,feature_idx],y_train.values[self.indexes]\n",
    "        print('start loop')\n",
    "        for split_index in range(1, self.x.shape[0]-1):\n",
    "            lhs_x = x<=x[split_index]\n",
    "            rhs_x = x>x[split_index]\n",
    "\n",
    "            if rhs_x.sum()==0:\n",
    "                print(self.stored_score, feature_list[self.stored_split_feature], self.stored_split_value)\n",
    "            else:\n",
    "                lhs_y_std = y[lhs_x].std()\n",
    "                rhs_x_std = y[rhs_x].std()\n",
    "\n",
    "                curr_score = lhs_y_std*lhs_x.sum() + rhs_x_std*rhs_x.sum()\n",
    "                if curr_score<self.stored_score: \n",
    "                    print('split index :%d lhs ct: %d | rhs ct: %d| score: %f' % (split_index, sum(lhs_x),sum(rhs_x), curr_score))\n",
    "                    self.stored_split_feature = feature_idx\n",
    "                    self.stored_score = curr_score\n",
    "                    self.stored_split_value = x[split_index]\n",
    "\n",
    "        print(self.stored_score, self.feature_list[self.stored_split_feature], self.stored_split_value)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "93281398.2058 < float('inf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start loop\n",
      "split index :1 lhs ct: 512 | rhs ct: 948| score: 93281398.205757\n",
      "split index :2 lhs ct: 1101 | rhs ct: 359| score: 91804469.340026\n",
      "split index :3 lhs ct: 1037 | rhs ct: 423| score: 91487146.514272\n",
      "split index :5 lhs ct: 618 | rhs ct: 842| score: 90970171.513650\n",
      "split index :6 lhs ct: 1008 | rhs ct: 452| score: 90944631.172519\n",
      "split index :13 lhs ct: 778 | rhs ct: 682| score: 88899549.318965\n",
      "split index :67 lhs ct: 755 | rhs ct: 705| score: 88506980.330170\n",
      "split index :239 lhs ct: 767 | rhs ct: 693| score: 88277501.801643\n",
      "88277501.8016 GrLivArea 1487\n",
      "88277501.8016 GrLivArea 1487\n"
     ]
    }
   ],
   "source": [
    "myTree = myIndecisionTree(X_train,y_train)\n",
    "myTree.find_split_in_single_feature(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make a function for all features\n",
    "\n",
    "Note that when we call this function we don't have to keep passing around updated scores or split values. That is taken care of by using the `self.` notation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {},
   "outputs": [],
   "source": [
    "class myIndecisionTree():\n",
    "    def __init__(self,x,y):\n",
    "        self.indexes = x.index\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        # self.feature_idx =1\n",
    "        self.stored_score = float('inf')\n",
    "        self.stored_split_feature = 0\n",
    "        self.stored_split_value = 0\n",
    "        self.feature_list = x.columns\n",
    "\n",
    "    \n",
    "    # ======== NEW FUNCTION =========================\n",
    "    def check_all_features(self):\n",
    "        for i in range(len(feature_list)-1):\n",
    "            self.find_split_in_single_feature(i)  \n",
    "            \n",
    "        print(self.stored_score, self.feature_list[self.stored_split_feature], self.stored_split_value)\n",
    "    # ======== NEW FUNCTION =========================    \n",
    "    \n",
    "    def find_split_in_single_feature(self, feature_idx):\n",
    "        X_train = self.x\n",
    "        y_train = self.y\n",
    "\n",
    "        x,y = X_train.iloc[self.indexes,feature_idx],y_train.values[self.indexes]\n",
    "        print('start loop')\n",
    "        for split_index in range(1, self.x.shape[0]-1):\n",
    "            lhs_x = x<=x[split_index]\n",
    "            rhs_x = x>x[split_index]\n",
    "\n",
    "            if rhs_x.sum()==0:\n",
    "                continue\n",
    "            else:\n",
    "                lhs_y_std = y[lhs_x].std()\n",
    "                rhs_x_std = y[rhs_x].std()\n",
    "\n",
    "                curr_score = lhs_y_std*lhs_x.sum() + rhs_x_std*rhs_x.sum()\n",
    "                if curr_score<self.stored_score: \n",
    "                    print('split index :%d lhs ct: %d | rhs ct: %d| score: %f' % (split_index, sum(lhs_x),sum(rhs_x), curr_score))\n",
    "                    self.stored_split_feature = feature_idx\n",
    "                    self.stored_score = curr_score\n",
    "                    self.stored_split_value = x[split_index]\n",
    "\n",
    "        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test it with a different data set - Single Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = sample_df['SalePrice']\n",
    "#X_train = sample_df[['GrLivArea']]\n",
    "X_train = sample_df[[x for x in sample_df.columns if x not in ['SalePrice','GrLivArea']]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mse', max_depth=1, max_features=None,\n",
       "           max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "           min_impurity_split=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           presort=False, random_state=None, splitter='best')"
      ]
     },
     "execution_count": 260,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtr = DecisionTreeRegressor(max_depth=1)\n",
    "dtr.fit(X_train,y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"326pt\" height=\"198pt\"\n",
       " viewBox=\"0.00 0.00 325.89 198.20\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 194.2)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-194.2 321.895,-194.2 321.895,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\n",
       "<polygon fill=\"#e58139\" fill-opacity=\"0.458824\" stroke=\"black\" points=\"141.342,-127.297 0.447256,-127.297 0.447256,-63.2966 141.342,-63.2966 141.342,-127.297\"/>\n",
       "<text text-anchor=\"start\" x=\"9.7915\" y=\"-112.097\" font-family=\"Times,serif\" font-size=\"14.00\">TotRmsAbvGrd ≤ 6.5</text>\n",
       "<text text-anchor=\"start\" x=\"8.4209\" y=\"-98.0966\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 6306788585.35</text>\n",
       "<text text-anchor=\"start\" x=\"26.895\" y=\"-84.0966\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 1460</text>\n",
       "<text text-anchor=\"start\" x=\"22.0381\" y=\"-70.0966\" font-family=\"Times,serif\" font-size=\"14.00\">value = 180921.2</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"318.342,-175.297 177.447,-175.297 177.447,-125.297 318.342,-125.297 318.342,-175.297\"/>\n",
       "<text text-anchor=\"start\" x=\"185.421\" y=\"-160.097\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 2512260840.62</text>\n",
       "<text text-anchor=\"start\" x=\"207.395\" y=\"-146.097\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 792</text>\n",
       "<text text-anchor=\"start\" x=\"195.538\" y=\"-132.097\" font-family=\"Times,serif\" font-size=\"14.00\">value = 148505.44</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141.562,-117.192C150.08,-119.869 158.834,-122.62 167.481,-125.338\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"166.592,-128.727 177.181,-128.387 168.69,-122.049 166.592,-128.727\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.016\" y=\"-135.751\" font-family=\"Times,serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\n",
       "<polygon fill=\"#e58139\" stroke=\"black\" points=\"318.342,-65.2966 177.447,-65.2966 177.447,-15.2966 318.342,-15.2966 318.342,-65.2966\"/>\n",
       "<text text-anchor=\"start\" x=\"185.421\" y=\"-50.0966\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 8082754159.99</text>\n",
       "<text text-anchor=\"start\" x=\"207.395\" y=\"-36.0966\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 668</text>\n",
       "<text text-anchor=\"start\" x=\"195.538\" y=\"-22.0966\" font-family=\"Times,serif\" font-size=\"14.00\">value = 219354.25</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141.562,-73.4011C150.08,-70.724 158.834,-67.9729 167.481,-65.2551\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"168.69,-68.5439 177.181,-62.2066 166.592,-61.866 168.69,-68.5439\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.016\" y=\"-46.4425\" font-family=\"Times,serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1a1bbf7278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw_tree(dtr, X_train, precision=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start loop\n",
      "split index :1 lhs ct: 643 | rhs ct: 817| score: 115781552.380532\n",
      "start loop\n",
      "split index :1 lhs ct: 792 | rhs ct: 668| score: 99752951.757499\n",
      "99752951.7575 TotRmsAbvGrd 6\n"
     ]
    }
   ],
   "source": [
    "myTree = myIndecisionTree(X_train,y_train)\n",
    "myTree.check_all_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's make it nestable\n",
    "\n",
    "And make provisions for indexes to be passed through"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [],
   "source": [
    "class myIndecisionTree():\n",
    "    def __init__(self,x,y, indexes==None, min_leaf=5):\n",
    "        if indexes == None:\n",
    "            self.indexes = x.index\n",
    "        else:\n",
    "            self.indexes = indexes\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "\n",
    "        self.stored_score = float('inf')\n",
    "        self.stored_split_feature = 0\n",
    "        self.stored_split_value = 0\n",
    "        self.feature_list = x.columns\n",
    "\n",
    "    \n",
    "    def check_all_features(self):\n",
    "        for i in range(len(feature_list)-1):\n",
    "            self.find_split_in_single_feature(i)  \n",
    "            \n",
    "        print(self.stored_score, self.feature_list[self.stored_split_feature], self.stored_split_value)\n",
    "    \n",
    "    def find_split_in_single_feature(self, feature_idx):\n",
    "        X_train = self.x\n",
    "        y_train = self.y\n",
    "\n",
    "        x,y = X_train.iloc[self.indexes,feature_idx],y_train.values[self.indexes]\n",
    "        print('start loop')\n",
    "        for split_index in range(1, self.x.shape[0]-1):\n",
    "            lhs_x = x<=x[split_index]\n",
    "            rhs_x = x>x[split_index]\n",
    "\n",
    "            if rhs_x.sum()==0:\n",
    "                continue\n",
    "            else:\n",
    "                lhs_y_std = y[lhs_x].std()\n",
    "                rhs_x_std = y[rhs_x].std()\n",
    "\n",
    "                curr_score = lhs_y_std*lhs_x.sum() + rhs_x_std*rhs_x.sum()\n",
    "                if curr_score<self.stored_score: \n",
    "                    print('split index :%d lhs ct: %d | rhs ct: %d| score: %f' % (split_index, sum(lhs_x),sum(rhs_x), curr_score))\n",
    "                    self.stored_split_feature = feature_idx\n",
    "                    self.stored_score = curr_score\n",
    "                    self.stored_split_value = x[split_index]\n",
    "\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 384,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---YrSold---\n",
      "split index :1 lhs ct: 1460 | rhs ct: 1460| score: 115781552.380532\n",
      "115781552.381 YrSold 2007\n",
      "---GrLivArea---\n",
      "split index :1 lhs ct: 1460 | rhs ct: 1460| score: 93281398.205757\n",
      "split index :2 lhs ct: 1460 | rhs ct: 1460| score: 91804469.340026\n",
      "split index :3 lhs ct: 1460 | rhs ct: 1460| score: 91487146.514272\n",
      "split index :5 lhs ct: 1460 | rhs ct: 1460| score: 90970171.513650\n",
      "split index :6 lhs ct: 1460 | rhs ct: 1460| score: 90944631.172519\n",
      "split index :13 lhs ct: 1460 | rhs ct: 1460| score: 88899549.318965\n",
      "split index :67 lhs ct: 1460 | rhs ct: 1460| score: 88506980.330170\n",
      "split index :239 lhs ct: 1460 | rhs ct: 1460| score: 88277501.801643\n",
      "88277501.8016 GrLivArea 1487\n",
      "---MSSubClass---\n",
      "88277501.8016 GrLivArea 1487\n",
      "---TotRmsAbvGrd---\n",
      "88277501.8016 GrLivArea 1487\n",
      "---OverallCond---\n",
      "88277501.8016 GrLivArea 1487\n",
      "88277501.8016 1 1487\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<__main__.myDecTree at 0x1a22752828>"
      ]
     },
     "execution_count": 384,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myDecTree(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 385,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---YrSold---\n",
      "split index :1 lhs ct: 100 | rhs ct: 100| score: 6776738.003673\n",
      "split index :3 lhs ct: 100 | rhs ct: 100| score: 6728372.189336\n",
      "6728372.18934 YrSold 2006\n",
      "---GrLivArea---\n",
      "split index :1 lhs ct: 100 | rhs ct: 100| score: 5544749.917601\n",
      "split index :5 lhs ct: 100 | rhs ct: 100| score: 4995598.238723\n",
      "split index :48 lhs ct: 100 | rhs ct: 100| score: 4965895.299135\n",
      "4965895.29913 GrLivArea 1452\n",
      "---MSSubClass---\n",
      "4965895.29913 GrLivArea 1452\n",
      "---TotRmsAbvGrd---\n",
      "4965895.29913 GrLivArea 1452\n",
      "---OverallCond---\n",
      "4965895.29913 GrLivArea 1452\n",
      "4965895.29913 1 1452\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<__main__.myDecTree at 0x1159c7080>"
      ]
     },
     "execution_count": 385,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myDecTree(X_train, y_train, [x for x in range(100)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's make it a multi-level tree class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 473,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = np.arange(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 477,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])"
      ]
     },
     "execution_count": 477,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 480,
   "metadata": {},
   "outputs": [],
   "source": [
    "z=np.nonzero(b>5)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 481,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6, 7, 8, 9])"
      ]
     },
     "execution_count": 481,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 476,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3, 5])"
      ]
     },
     "execution_count": 476,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b[[1,2,3,5]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 492,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6, 7, 8, 9])"
      ]
     },
     "execution_count": 492,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b[z]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 487,
   "metadata": {},
   "outputs": [],
   "source": [
    "class myNestedDecTree():\n",
    "    def __init__(self, x, y, idxs=None, min_leaf=5):\n",
    "        if idxs is None:\n",
    "            idxs = np.arange(len(y))\n",
    "        \n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        self.idxs = idxs\n",
    "        self.min_leaf = min_leaf\n",
    "        self.n = len(idxs)\n",
    "        self.column_ct = x.shape[1]\n",
    "        self.mean_value = np.mean(y[idxs])\n",
    "        self.stored_score = float('inf')\n",
    "        self.stored_score, self.split_feature_idx, self.split_value = self.find_split(x.iloc[idxs],y.iloc[idxs])\n",
    "        \n",
    "    @property\n",
    "    def get_feature_values(self): \n",
    "        return self.x.values[self.idxs,self.split_feature_idx]\n",
    "    \n",
    "    @property\n",
    "    def get_feature_names(self): return self.x.columns[self.var_idx]\n",
    "\n",
    "    @property\n",
    "    def is_leaf(self): return self.stored_score == float('inf')\n",
    "    \n",
    "    \n",
    "    def find_split(self, X_train, y_train):\n",
    "        stored_score = float('inf')\n",
    "        stored_split_feature = None\n",
    "        stored_split_value = None\n",
    "        feature_list = X_train.columns\n",
    "\n",
    "        for k in range(len(feature_list)):\n",
    "            print('---%s---' %feature_list[k],)\n",
    "            stored_score, stored_split_feature, stored_split_value = self.find_split_in_single_feature(X_train, y_train, stored_score, stored_split_feature, stored_split_value, k)\n",
    "        \n",
    "        print('now')\n",
    "        # =========== NEW CODE ============\n",
    "        if stored_score == float('inf'): \n",
    "            return\n",
    "        else:\n",
    "            x = self.x.values[self.idxs, stored_split_feature]\n",
    "            print(stored_split_value)\n",
    "            lhs_indexes = np.nonzero(x<=stored_split_value)[0]\n",
    "            rhs_indexes = np.nonzero(x>stored_split_value)[0]\n",
    "\n",
    "            ### recursive call\n",
    "            print('hi left')\n",
    "            print(lhs_indexes, type(lhs_indexes))\n",
    "            print(self.idxs[lhs_indexes])\n",
    "            self.lhs_tree = myNestedDecTree(self.x, self.y, self.idxs[lhs_indexes])\n",
    "            print('hi right')\n",
    "            self.rhs_tree = myNestedDecTree(self.x, self.y, self.idxs[rhs_indexes])\n",
    "            # =========== END =================\n",
    "\n",
    "            print(stored_score, stored_split_feature, stored_split_value)\n",
    "            return(stored_score, stored_split_feature, stored_split_value)\n",
    "\n",
    "    \n",
    "    def predict_row(self, xi):\n",
    "        if self.is_leaf: \n",
    "            return self.mean_value\n",
    "        else:\n",
    "            t = self.lhs if xi[self.var_idx]<=self.split else self.rhs\n",
    "            return t.predict_row(xi)\n",
    "    \n",
    "    \n",
    "    def find_split_in_single_feature(self,\n",
    "                                     X_train, \n",
    "                                     y_train, \n",
    "                                     stored_score, \n",
    "                                     stored_split_feature, \n",
    "                                     stored_split_value,\n",
    "                                     feature_idx):\n",
    "        all_indexes = [x for x in X_train.index]\n",
    "\n",
    "        feature_list = X_train.columns\n",
    "        x,y = X_train.iloc[all_indexes,feature_idx],y_train.values[all_indexes]\n",
    "\n",
    "        for split_index in range(1, n-1):\n",
    "            lhs_x = x<=x[split_index]\n",
    "            rhs_x = x>x[split_index]\n",
    "\n",
    "            if rhs_x.sum()==0:\n",
    "                print(stored_score, feature_list[stored_split_feature], stored_split_value)\n",
    "                return stored_score, stored_split_feature, stored_split_value\n",
    "            else:\n",
    "                lhs_y_std = y[lhs_x].std()\n",
    "                rhs_x_std = y[rhs_x].std()\n",
    "\n",
    "                curr_score = lhs_y_std*lhs_x.sum() + rhs_x_std*rhs_x.sum()\n",
    "\n",
    "                if curr_score<stored_score: \n",
    "                    print('split index :%d lhs ct: %d | rhs ct: %d| score: %f' % (split_index, len(lhs_x),len(rhs_x), curr_score))\n",
    "                    stored_split_feature = feature_idx\n",
    "                    stored_score = curr_score\n",
    "                    stored_split_value = x[split_index]\n",
    "\n",
    "        print(stored_score, feature_list[stored_split_feature], stored_split_value)\n",
    "        return stored_score, stored_split_feature, stored_split_value\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 488,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = sample_df['SalePrice']\n",
    "X_train = sample_df[[x for x in sample_df.columns if x not in ['SalePrice']]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 489,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,\n",
       "           max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "           min_impurity_split=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           presort=False, random_state=None, splitter='best')"
      ]
     },
     "execution_count": 489,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtr = DecisionTreeRegressor(max_depth=2)\n",
    "dtr.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 490,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"503pt\" height=\"304pt\"\n",
       " viewBox=\"0.00 0.00 502.84 304.40\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 300.4)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-300.4 498.842,-300.4 498.842,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\n",
       "<polygon fill=\"#e58139\" fill-opacity=\"0.294118\" stroke=\"black\" points=\"141.289,-181.833 0.394521,-181.833 0.394521,-117.833 141.289,-117.833 141.289,-181.833\"/>\n",
       "<text text-anchor=\"start\" x=\"13.543\" y=\"-166.633\" font-family=\"Times,serif\" font-size=\"14.00\">GrLivArea ≤ 1488.0</text>\n",
       "<text text-anchor=\"start\" x=\"8.36816\" y=\"-152.633\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 6306788585.35</text>\n",
       "<text text-anchor=\"start\" x=\"26.8423\" y=\"-138.633\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 1460</text>\n",
       "<text text-anchor=\"start\" x=\"21.9854\" y=\"-124.633\" font-family=\"Times,serif\" font-size=\"14.00\">value = 180921.2</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\n",
       "<polygon fill=\"#e58139\" fill-opacity=\"0.082353\" stroke=\"black\" points=\"318.289,-228.833 177.395,-228.833 177.395,-164.833 318.289,-164.833 318.289,-228.833\"/>\n",
       "<text text-anchor=\"start\" x=\"190.799\" y=\"-213.633\" font-family=\"Times,serif\" font-size=\"14.00\">GrLivArea ≤ 1198.5</text>\n",
       "<text text-anchor=\"start\" x=\"185.368\" y=\"-199.633\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 1408525823.77</text>\n",
       "<text text-anchor=\"start\" x=\"207.342\" y=\"-185.633\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 767</text>\n",
       "<text text-anchor=\"start\" x=\"195.485\" y=\"-171.633\" font-family=\"Times,serif\" font-size=\"14.00\">value = 137687.23</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141.509,-168.543C150.027,-170.831 158.781,-173.182 167.428,-175.505\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"166.563,-178.896 177.128,-178.11 168.378,-172.136 166.563,-178.896\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.47\" y=\"-186.397\" font-family=\"Times,serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\"><title>4</title>\n",
       "<polygon fill=\"#e58139\" fill-opacity=\"0.529412\" stroke=\"black\" points=\"314.789,-134.833 180.895,-134.833 180.895,-70.8327 314.789,-70.8327 314.789,-134.833\"/>\n",
       "<text text-anchor=\"start\" x=\"190.543\" y=\"-119.633\" font-family=\"Times,serif\" font-size=\"14.00\">GrLivArea ≤ 2361.0</text>\n",
       "<text text-anchor=\"start\" x=\"188.868\" y=\"-105.633\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 7369649464.1</text>\n",
       "<text text-anchor=\"start\" x=\"207.342\" y=\"-91.6327\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 693</text>\n",
       "<text text-anchor=\"start\" x=\"195.485\" y=\"-77.6327\" font-family=\"Times,serif\" font-size=\"14.00\">value = 228771.78</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141.509,-131.122C151.232,-128.511 161.261,-125.817 171.088,-123.178\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"172.08,-126.536 180.83,-120.562 170.264,-119.775 172.08,-126.536\"/>\n",
       "<text text-anchor=\"middle\" x=\"159.172\" y=\"-103.874\" font-family=\"Times,serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"488.289,-292.833 361.395,-292.833 361.395,-242.833 488.289,-242.833 488.289,-292.833\"/>\n",
       "<text text-anchor=\"start\" x=\"369.368\" y=\"-277.633\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 770355693.8</text>\n",
       "<text text-anchor=\"start\" x=\"384.342\" y=\"-263.633\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 425</text>\n",
       "<text text-anchor=\"start\" x=\"375.985\" y=\"-249.633\" font-family=\"Times,serif\" font-size=\"14.00\">value = 120989.6</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M318.509,-225.098C329.754,-229.66 341.408,-234.388 352.687,-238.964\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"351.474,-242.249 362.056,-242.765 354.105,-235.762 351.474,-242.249\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\"><title>3</title>\n",
       "<polygon fill=\"#e58139\" fill-opacity=\"0.184314\" stroke=\"black\" points=\"495.289,-213.833 354.395,-213.833 354.395,-163.833 495.289,-163.833 495.289,-213.833\"/>\n",
       "<text text-anchor=\"start\" x=\"362.368\" y=\"-198.633\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 1424537020.21</text>\n",
       "<text text-anchor=\"start\" x=\"384.342\" y=\"-184.633\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 342</text>\n",
       "<text text-anchor=\"start\" x=\"375.985\" y=\"-170.633\" font-family=\"Times,serif\" font-size=\"14.00\">value = 158437.2</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M318.509,-193.648C326.844,-193.267 335.405,-192.876 343.87,-192.489\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"344.298,-195.973 354.128,-192.02 343.979,-188.98 344.298,-195.973\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\"><title>5</title>\n",
       "<polygon fill=\"#e58139\" fill-opacity=\"0.454902\" stroke=\"black\" points=\"495.289,-131.833 354.395,-131.833 354.395,-81.8327 495.289,-81.8327 495.289,-131.833\"/>\n",
       "<text text-anchor=\"start\" x=\"362.368\" y=\"-116.633\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 4454528956.82</text>\n",
       "<text text-anchor=\"start\" x=\"384.342\" y=\"-102.633\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 599</text>\n",
       "<text text-anchor=\"start\" x=\"372.485\" y=\"-88.6327\" font-family=\"Times,serif\" font-size=\"14.00\">value = 213676.73</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M315.002,-104.345C324.556,-104.563 334.472,-104.79 344.255,-105.014\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"344.303,-108.515 354.381,-105.245 344.463,-101.517 344.303,-108.515\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\"><title>6</title>\n",
       "<polygon fill=\"#e58139\" stroke=\"black\" points=\"495.277,-53.8327 354.407,-53.8327 354.407,-3.83268 495.277,-3.83268 495.277,-53.8327\"/>\n",
       "<text text-anchor=\"start\" x=\"362.625\" y=\"-38.6327\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 15241122577.6</text>\n",
       "<text text-anchor=\"start\" x=\"387.842\" y=\"-24.6327\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 94</text>\n",
       "<text text-anchor=\"start\" x=\"372.485\" y=\"-10.6327\" font-family=\"Times,serif\" font-size=\"14.00\">value = 324962.54</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M315.002,-74.8565C328.087,-69.3234 341.85,-63.5036 355.02,-57.9343\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"356.641,-61.0493 364.488,-53.9309 353.914,-54.602 356.641,-61.0493\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1156fa978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw_tree(dtr, X_train, precision=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 491,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---YrSold---\n",
      "split index :1 lhs ct: 100 | rhs ct: 100| score: 6776738.003673\n",
      "split index :3 lhs ct: 100 | rhs ct: 100| score: 6728372.189336\n",
      "6728372.18934 YrSold 2006\n",
      "---GrLivArea---\n",
      "split index :1 lhs ct: 100 | rhs ct: 100| score: 5544749.917601\n",
      "split index :5 lhs ct: 100 | rhs ct: 100| score: 4995598.238723\n",
      "split index :48 lhs ct: 100 | rhs ct: 100| score: 4965895.299135\n",
      "4965895.29913 GrLivArea 1452\n",
      "---MSSubClass---\n",
      "4965895.29913 GrLivArea 1452\n",
      "---TotRmsAbvGrd---\n",
      "4965895.29913 GrLivArea 1452\n",
      "---OverallCond---\n",
      "4965895.29913 GrLivArea 1452\n",
      "now\n",
      "1452\n",
      "hi left\n",
      "[ 1  5  9 10 12 14 15 16 17 18 19 21 23 24 26 29 30 31 32 36 37 38 39 40 41\n",
      " 42 43 44 48 49 51 52 54 55 59 60 61 62 68 71 73 75 76 77 79 81 83 87 89 90\n",
      " 91 92 97 98 99] <class 'numpy.ndarray'>\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "only integer scalar arrays can be converted to a scalar index",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-491-8fd3799d58b6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmdt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmyNestedDecTree\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-487-9df5dbab1cc9>\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, x, y, idxs, min_leaf)\u001b[0m\n\u001b[1;32m     12\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean_value\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstored_score\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'inf'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstored_score\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit_feature_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit_value\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-487-9df5dbab1cc9>\u001b[0m in \u001b[0;36mfind_split\u001b[0;34m(self, X_train, y_train)\u001b[0m\n\u001b[1;32m     48\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'hi left'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     49\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlhs_indexes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlhs_indexes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 50\u001b[0;31m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlhs_indexes\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     51\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlhs_tree\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmyNestedDecTree\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlhs_indexes\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     52\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'hi right'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: only integer scalar arrays can be converted to a scalar index"
     ]
    }
   ],
   "source": [
    "mdt = myNestedDecTree(X_train, y_train, [x for x in range(100)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's make it a multi-level tree class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 471,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "<__main__.myDecTree object at 0x1a22678a20> is not an estimator instance.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-471-6a356f6064a5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdraw_tree\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmdt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-85-6153ce29706f>\u001b[0m in \u001b[0;36mdraw_tree\u001b[0;34m(t, df, size, ratio, precision)\u001b[0m\n\u001b[1;32m      7\u001b[0m     \"\"\"\n\u001b[1;32m      8\u001b[0m     s=export_graphviz(t, out_file=None, feature_names=df.columns, filled=True,\n\u001b[0;32m----> 9\u001b[0;31m                       special_characters=True, rotate=True, precision=precision)\n\u001b[0m\u001b[1;32m     10\u001b[0m     IPython.display.display(graphviz.Source(re.sub('Tree {',\n\u001b[1;32m     11\u001b[0m        f'Tree {{ size={size}; ratio={ratio}', s)))\n",
      "\u001b[0;32m~/anaconda/envs/fastai/lib/python3.6/site-packages/sklearn/tree/export.py\u001b[0m in \u001b[0;36mexport_graphviz\u001b[0;34m(decision_tree, out_file, max_depth, feature_names, class_names, label, filled, leaves_parallel, impurity, node_ids, proportion, rotate, rounded, special_characters, precision)\u001b[0m\n\u001b[1;32m    390\u001b[0m                 \u001b[0mout_file\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'%d -> %d ;\\n'\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mparent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnode_id\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    391\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 392\u001b[0;31m     \u001b[0mcheck_is_fitted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdecision_tree\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'tree_'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    393\u001b[0m     \u001b[0mown_file\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    394\u001b[0m     \u001b[0mreturn_string\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/fastai/lib/python3.6/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36mcheck_is_fitted\u001b[0;34m(estimator, attributes, msg, all_or_any)\u001b[0m\n\u001b[1;32m    760\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    761\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mestimator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'fit'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 762\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"%s is not an estimator instance.\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mestimator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    763\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    764\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mattributes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: <__main__.myDecTree object at 0x1a22678a20> is not an estimator instance."
     ]
    }
   ],
   "source": [
    "draw_tree(mdt, X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now an ensemble of trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class myTreeEnsemble():\n",
    "    def __init__(self, x, y, n_trees, sample_sz, min_leaf=5):\n",
    "        np.random.seed(42)\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        self.sample_sz = sample_sz\n",
    "        self.min_leaf = min_leaf\n",
    "        self.trees = [self.create_tree() for i in range(n_trees)]\n",
    "    \n",
    "    def create_tree(self):\n",
    "        # get random indexes\n",
    "        rnd_idxs = np.random.permutation(len(self.y))[:self.sample_sz]\n",
    "        \n",
    "        # then return a decision tree based on a random dataset\n",
    "        return DecisionTree(self.x.iloc[rnd_idxs], self.y[rnd_idxs], min_leaf=self.min_leaf)\n",
    "        \n",
    "    def predict(self, x):\n",
    "        \n",
    "        # currently returning the mean of the prediction of all the \n",
    "        # trees\n",
    "        return np.mean([t.predict(x) for t in self.trees], axis=0)\n",
    "1.2 Decision Tree Version 1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}