msan621_ml1/PandasSklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice with Pandas + Sklearn: Features and How-Tos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Will Cover the following topics:\n",
    "1. Basic Data Cleaning\n",
    "2. Helpful Pandas data transformation tools\n",
    "3. Sklearn preprocessing toolkits\n",
    "4. Sklearn different models\n",
    "5. Basic Metric evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.kaggle.com/c/house-prices-advanced-regression-techniques\n",
    "\n",
    "#### How much does a house cost?\n",
    "![](http://www.hubbellhomes.com/images/community/368/tb/Pemberley-Hills-thumbNEW.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Beginners Tips\n",
    "\n",
    "<img src='http://www.allwebdesigninfo.com/wp-content/uploads/2014/03/Beginner-Programming-Mistakes.jpg'  width='400px' />\n",
    "- **Keywords** For long analysis, becareful how you name your variables. With really complicated notebooks, a simple rewrite can ruin the rest of your programming setup. especially becareful of these keywords:\n",
    "\n",
    "` X, y, n, df, data, dataframe, train, test, results, final_results, predict` \n",
    "\n",
    "- **Self-Assignment** As you are going on becare of reusing names and looping coding blocks, not how you divide your code blocks. A standalone block where you do a self-assignment, or a variable name is reused, can lead to a recursive like effects.\n",
    "\n",
    "```python\n",
    "# this is fine because the data is reloaded each time this block is run\n",
    "mydf = load_data()\n",
    "mydf = replace_any_zeroes(df)\n",
    "```\n",
    "\n",
    "```python\n",
    "# this is dangerous. if you keep re-running this block, it will \n",
    "# continously append new columns over and over with no end\n",
    "mydf = mydf.left_join(new_dataset)\n",
    "```\n",
    "\n",
    "- **Common Datascience vocab; choose other words as vars** Just know up front that you will most likely be using `X`, `y`, `model`, `train`, `test`, `pred`, and result keywords. Becareful of using these variables elsewhere in coding loops etc.\n",
    "\n",
    "```python\n",
    "# currently y is a matrices\n",
    "y,X = train_test_split(mydf)\n",
    "\n",
    "## if you start \n",
    "for y in list_of_columns:\n",
    "    print('column named' + y)\n",
    "\n",
    "## what is in 'y' now?\n",
    "model.fit(X,y)\n",
    "```\n",
    "\n",
    "- **Pandas Warning on DataFrame copying** Pandas dataframes as they get manipulated for a majority of the time are still referencing the same memory or dataframe:\n",
    "\n",
    "```python\n",
    "\n",
    "# IS STILL POINTING TO ORIGINAL, EDITS TO NEW WILL ALSO BE APPLIED TO ORIGINAL\n",
    "new_df = df \n",
    "\n",
    "# makes a new copy\n",
    "\n",
    "new_df = df.copy()\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load Your Data \n",
    "\n",
    "<img src='https://ak8.picdn.net/shutterstock/videos/17983978/thumb/7.jpg' style='width:300px'/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import itertools\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "#prep\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder, StandardScaler, MaxAbsScaler, QuantileTransformer\n",
    "\n",
    "#models\n",
    "from sklearn.linear_model import LogisticRegression, LogisticRegressionCV, LinearRegression, Ridge, RidgeCV\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "#validation libraries\n",
    "from sklearn.cross_validation import KFold, StratifiedKFold\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from IPython.display import display\n",
    "from sklearn import metrics\n",
    "\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Navigating your data\n",
    "\n",
    "<img src='https://www.eurisy.org/data_files/mce_images/Blog_images/Human_hand_with_a_laptop_surrounded_by_iconsellagrin.jpg' style='width:400px' />\n",
    "#### Dataset size\n",
    "\n",
    "- **Train**: 1460\n",
    "- **Test**: 1458\n",
    "\n",
    "#### read in CSV's from a file path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('./train (1).csv')\n",
    "df_test = pd.read_csv('./test (1).csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### whats the size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1460, 81)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### whats the description?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>...</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1201.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1452.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>730.500000</td>\n",
       "      <td>56.897260</td>\n",
       "      <td>70.049958</td>\n",
       "      <td>10516.828082</td>\n",
       "      <td>6.099315</td>\n",
       "      <td>5.575342</td>\n",
       "      <td>1971.267808</td>\n",
       "      <td>1984.865753</td>\n",
       "      <td>103.685262</td>\n",
       "      <td>443.639726</td>\n",
       "      <td>...</td>\n",
       "      <td>94.244521</td>\n",
       "      <td>46.660274</td>\n",
       "      <td>21.954110</td>\n",
       "      <td>3.409589</td>\n",
       "      <td>15.060959</td>\n",
       "      <td>2.758904</td>\n",
       "      <td>43.489041</td>\n",
       "      <td>6.321918</td>\n",
       "      <td>2007.815753</td>\n",
       "      <td>180921.195890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>421.610009</td>\n",
       "      <td>42.300571</td>\n",
       "      <td>24.284752</td>\n",
       "      <td>9981.264932</td>\n",
       "      <td>1.382997</td>\n",
       "      <td>1.112799</td>\n",
       "      <td>30.202904</td>\n",
       "      <td>20.645407</td>\n",
       "      <td>181.066207</td>\n",
       "      <td>456.098091</td>\n",
       "      <td>...</td>\n",
       "      <td>125.338794</td>\n",
       "      <td>66.256028</td>\n",
       "      <td>61.119149</td>\n",
       "      <td>29.317331</td>\n",
       "      <td>55.757415</td>\n",
       "      <td>40.177307</td>\n",
       "      <td>496.123024</td>\n",
       "      <td>2.703626</td>\n",
       "      <td>1.328095</td>\n",
       "      <td>79442.502883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>1300.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1872.000000</td>\n",
       "      <td>1950.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2006.000000</td>\n",
       "      <td>34900.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>365.750000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>7553.500000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1954.000000</td>\n",
       "      <td>1967.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>2007.000000</td>\n",
       "      <td>129975.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>730.500000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>69.000000</td>\n",
       "      <td>9478.500000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1973.000000</td>\n",
       "      <td>1994.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>383.500000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>2008.000000</td>\n",
       "      <td>163000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1095.250000</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>11601.500000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>2000.000000</td>\n",
       "      <td>2004.000000</td>\n",
       "      <td>166.000000</td>\n",
       "      <td>712.250000</td>\n",
       "      <td>...</td>\n",
       "      <td>168.000000</td>\n",
       "      <td>68.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>2009.000000</td>\n",
       "      <td>214000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1460.000000</td>\n",
       "      <td>190.000000</td>\n",
       "      <td>313.000000</td>\n",
       "      <td>215245.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>2010.000000</td>\n",
       "      <td>2010.000000</td>\n",
       "      <td>1600.000000</td>\n",
       "      <td>5644.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>857.000000</td>\n",
       "      <td>547.000000</td>\n",
       "      <td>552.000000</td>\n",
       "      <td>508.000000</td>\n",
       "      <td>480.000000</td>\n",
       "      <td>738.000000</td>\n",
       "      <td>15500.000000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>2010.000000</td>\n",
       "      <td>755000.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Id   MSSubClass  LotFrontage        LotArea  OverallQual  \\\n",
       "count  1460.000000  1460.000000  1201.000000    1460.000000  1460.000000   \n",
       "mean    730.500000    56.897260    70.049958   10516.828082     6.099315   \n",
       "std     421.610009    42.300571    24.284752    9981.264932     1.382997   \n",
       "min       1.000000    20.000000    21.000000    1300.000000     1.000000   \n",
       "25%     365.750000    20.000000    59.000000    7553.500000     5.000000   \n",
       "50%     730.500000    50.000000    69.000000    9478.500000     6.000000   \n",
       "75%    1095.250000    70.000000    80.000000   11601.500000     7.000000   \n",
       "max    1460.000000   190.000000   313.000000  215245.000000    10.000000   \n",
       "\n",
       "       OverallCond    YearBuilt  YearRemodAdd   MasVnrArea   BsmtFinSF1  \\\n",
       "count  1460.000000  1460.000000   1460.000000  1452.000000  1460.000000   \n",
       "mean      5.575342  1971.267808   1984.865753   103.685262   443.639726   \n",
       "std       1.112799    30.202904     20.645407   181.066207   456.098091   \n",
       "min       1.000000  1872.000000   1950.000000     0.000000     0.000000   \n",
       "25%       5.000000  1954.000000   1967.000000     0.000000     0.000000   \n",
       "50%       5.000000  1973.000000   1994.000000     0.000000   383.500000   \n",
       "75%       6.000000  2000.000000   2004.000000   166.000000   712.250000   \n",
       "max       9.000000  2010.000000   2010.000000  1600.000000  5644.000000   \n",
       "\n",
       "           ...         WoodDeckSF  OpenPorchSF  EnclosedPorch    3SsnPorch  \\\n",
       "count      ...        1460.000000  1460.000000    1460.000000  1460.000000   \n",
       "mean       ...          94.244521    46.660274      21.954110     3.409589   \n",
       "std        ...         125.338794    66.256028      61.119149    29.317331   \n",
       "min        ...           0.000000     0.000000       0.000000     0.000000   \n",
       "25%        ...           0.000000     0.000000       0.000000     0.000000   \n",
       "50%        ...           0.000000    25.000000       0.000000     0.000000   \n",
       "75%        ...         168.000000    68.000000       0.000000     0.000000   \n",
       "max        ...         857.000000   547.000000     552.000000   508.000000   \n",
       "\n",
       "       ScreenPorch     PoolArea       MiscVal       MoSold       YrSold  \\\n",
       "count  1460.000000  1460.000000   1460.000000  1460.000000  1460.000000   \n",
       "mean     15.060959     2.758904     43.489041     6.321918  2007.815753   \n",
       "std      55.757415    40.177307    496.123024     2.703626     1.328095   \n",
       "min       0.000000     0.000000      0.000000     1.000000  2006.000000   \n",
       "25%       0.000000     0.000000      0.000000     5.000000  2007.000000   \n",
       "50%       0.000000     0.000000      0.000000     6.000000  2008.000000   \n",
       "75%       0.000000     0.000000      0.000000     8.000000  2009.000000   \n",
       "max     480.000000   738.000000  15500.000000    12.000000  2010.000000   \n",
       "\n",
       "           SalePrice  \n",
       "count    1460.000000  \n",
       "mean   180921.195890  \n",
       "std     79442.502883  \n",
       "min     34900.000000  \n",
       "25%    129975.000000  \n",
       "50%    163000.000000  \n",
       "75%    214000.000000  \n",
       "max    755000.000000  \n",
       "\n",
       "[8 rows x 38 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1460 entries, 0 to 1459\n",
      "Data columns (total 81 columns):\n",
      "Id               1460 non-null int64\n",
      "MSSubClass       1460 non-null int64\n",
      "MSZoning         1460 non-null object\n",
      "LotFrontage      1201 non-null float64\n",
      "LotArea          1460 non-null int64\n",
      "Street           1460 non-null object\n",
      "Alley            91 non-null object\n",
      "LotShape         1460 non-null object\n",
      "LandContour      1460 non-null object\n",
      "Utilities        1460 non-null object\n",
      "LotConfig        1460 non-null object\n",
      "LandSlope        1460 non-null object\n",
      "Neighborhood     1460 non-null object\n",
      "Condition1       1460 non-null object\n",
      "Condition2       1460 non-null object\n",
      "BldgType         1460 non-null object\n",
      "HouseStyle       1460 non-null object\n",
      "OverallQual      1460 non-null int64\n",
      "OverallCond      1460 non-null int64\n",
      "YearBuilt        1460 non-null int64\n",
      "YearRemodAdd     1460 non-null int64\n",
      "RoofStyle        1460 non-null object\n",
      "RoofMatl         1460 non-null object\n",
      "Exterior1st      1460 non-null object\n",
      "Exterior2nd      1460 non-null object\n",
      "MasVnrType       1452 non-null object\n",
      "MasVnrArea       1452 non-null float64\n",
      "ExterQual        1460 non-null object\n",
      "ExterCond        1460 non-null object\n",
      "Foundation       1460 non-null object\n",
      "BsmtQual         1423 non-null object\n",
      "BsmtCond         1423 non-null object\n",
      "BsmtExposure     1422 non-null object\n",
      "BsmtFinType1     1423 non-null object\n",
      "BsmtFinSF1       1460 non-null int64\n",
      "BsmtFinType2     1422 non-null object\n",
      "BsmtFinSF2       1460 non-null int64\n",
      "BsmtUnfSF        1460 non-null int64\n",
      "TotalBsmtSF      1460 non-null int64\n",
      "Heating          1460 non-null object\n",
      "HeatingQC        1460 non-null object\n",
      "CentralAir       1460 non-null object\n",
      "Electrical       1459 non-null object\n",
      "1stFlrSF         1460 non-null int64\n",
      "2ndFlrSF         1460 non-null int64\n",
      "LowQualFinSF     1460 non-null int64\n",
      "GrLivArea        1460 non-null int64\n",
      "BsmtFullBath     1460 non-null int64\n",
      "BsmtHalfBath     1460 non-null int64\n",
      "FullBath         1460 non-null int64\n",
      "HalfBath         1460 non-null int64\n",
      "BedroomAbvGr     1460 non-null int64\n",
      "KitchenAbvGr     1460 non-null int64\n",
      "KitchenQual      1460 non-null object\n",
      "TotRmsAbvGrd     1460 non-null int64\n",
      "Functional       1460 non-null object\n",
      "Fireplaces       1460 non-null int64\n",
      "FireplaceQu      770 non-null object\n",
      "GarageType       1379 non-null object\n",
      "GarageYrBlt      1379 non-null float64\n",
      "GarageFinish     1379 non-null object\n",
      "GarageCars       1460 non-null int64\n",
      "GarageArea       1460 non-null int64\n",
      "GarageQual       1379 non-null object\n",
      "GarageCond       1379 non-null object\n",
      "PavedDrive       1460 non-null object\n",
      "WoodDeckSF       1460 non-null int64\n",
      "OpenPorchSF      1460 non-null int64\n",
      "EnclosedPorch    1460 non-null int64\n",
      "3SsnPorch        1460 non-null int64\n",
      "ScreenPorch      1460 non-null int64\n",
      "PoolArea         1460 non-null int64\n",
      "PoolQC           7 non-null object\n",
      "Fence            281 non-null object\n",
      "MiscFeature      54 non-null object\n",
      "MiscVal          1460 non-null int64\n",
      "MoSold           1460 non-null int64\n",
      "YrSold           1460 non-null int64\n",
      "SaleType         1460 non-null object\n",
      "SaleCondition    1460 non-null object\n",
      "SalePrice        1460 non-null int64\n",
      "dtypes: float64(3), int64(35), object(43)\n",
      "memory usage: 924.0+ KB\n"
     ]
    }
   ],
   "source": [
    "train_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering and Peering inside\n",
    "\n",
    "<img src='http://www.blastam.com/wp-content/uploads/ecommerce-filter-feature.jpg' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### take a peek at the first few rows,  note what may be categorical , what may be numeric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "\n",
       "  LandContour Utilities    ...     PoolArea PoolQC Fence MiscFeature MiscVal  \\\n",
       "0         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "1         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "\n",
       "  MoSold YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0      2   2008        WD         Normal     208500  \n",
       "1      5   2007        WD         Normal     181500  \n",
       "\n",
       "[2 rows x 81 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### When is it an array? When is it a Dataframe?, note the difference `[ ]` vs `[[ ]]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    208500\n",
       "1    181500\n",
       "2    223500\n",
       "3    140000\n",
       "4    250000\n",
       "Name: SalePrice, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['SalePrice'].head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   SalePrice\n",
       "0     208500\n",
       "1     181500\n",
       "2     223500\n",
       "3     140000\n",
       "4     250000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[['SalePrice']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    208500\n",
       "1    181500\n",
       "2    223500\n",
       "3    140000\n",
       "Name: SalePrice, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['SalePrice'].head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>LotShape</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>208500</td>\n",
       "      <td>Reg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>181500</td>\n",
       "      <td>Reg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>223500</td>\n",
       "      <td>IR1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>140000</td>\n",
       "      <td>IR1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   SalePrice LotShape\n",
       "0     208500      Reg\n",
       "1     181500      Reg\n",
       "2     223500      IR1\n",
       "3     140000      IR1"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[['SalePrice','LotShape']].head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities    ...     PoolArea PoolQC Fence MiscFeature MiscVal  \\\n",
       "0         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "1         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "2         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "\n",
       "  MoSold YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0      2   2008        WD         Normal     208500  \n",
       "1      5   2007        WD         Normal     181500  \n",
       "2      9   2008        WD         Normal     223500  \n",
       "\n",
       "[3 rows x 81 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.iloc[range(3),]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "4   5          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities    ...     PoolArea PoolQC Fence MiscFeature MiscVal  \\\n",
       "0         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "2         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "4         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "\n",
       "  MoSold YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0      2   2008        WD         Normal     208500  \n",
       "2      9   2008        WD         Normal     223500  \n",
       "4     12   2008        WD         Normal     250000  \n",
       "\n",
       "[3 rows x 81 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['SalePrice']>200000].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities    ...     PoolArea PoolQC Fence MiscFeature MiscVal  \\\n",
       "0         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "1         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "2         Lvl    AllPub    ...            0    NaN   NaN         NaN       0   \n",
       "\n",
       "  MoSold YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0      2   2008        WD         Normal     208500  \n",
       "1      5   2007        WD         Normal     181500  \n",
       "2      9   2008        WD         Normal     223500  \n",
       "\n",
       "[3 rows x 81 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['LotShape'].isin(['Reg','IR1'])].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Take a look at the column names of all the fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this many columns:81 \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Index(['Id', 'MSSubClass', 'MSZoning', 'LotFrontage', 'LotArea', 'Street',\n",
       "       'Alley', 'LotShape', 'LandContour', 'Utilities', 'LotConfig',\n",
       "       'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType',\n",
       "       'HouseStyle', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd',\n",
       "       'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType',\n",
       "       'MasVnrArea', 'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual',\n",
       "       'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinSF1',\n",
       "       'BsmtFinType2', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'Heating',\n",
       "       'HeatingQC', 'CentralAir', 'Electrical', '1stFlrSF', '2ndFlrSF',\n",
       "       'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath',\n",
       "       'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'KitchenQual',\n",
       "       'TotRmsAbvGrd', 'Functional', 'Fireplaces', 'FireplaceQu', 'GarageType',\n",
       "       'GarageYrBlt', 'GarageFinish', 'GarageCars', 'GarageArea', 'GarageQual',\n",
       "       'GarageCond', 'PavedDrive', 'WoodDeckSF', 'OpenPorchSF',\n",
       "       'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'PoolQC',\n",
       "       'Fence', 'MiscFeature', 'MiscVal', 'MoSold', 'YrSold', 'SaleType',\n",
       "       'SaleCondition', 'SalePrice'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('this many columns:%d ' % len(train_df.columns))\n",
    "train_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Changing the Column names starting with numbers "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.columns = ['Id', 'MSSubClass', 'MSZoning', 'LotFrontage', 'LotArea', 'Street',\n",
    "       'Alley', 'LotShape', 'LandContour', 'Utilities', 'LotConfig',\n",
    "       'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType',\n",
    "       'HouseStyle', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd',\n",
    "       'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType',\n",
    "       'MasVnrArea', 'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual',\n",
    "       'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinSF1',\n",
    "       'BsmtFinType2', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'Heating',\n",
    "       'HeatingQC', 'CentralAir', 'Electrical', 'FirsstFlrSF', 'SecondFlrSF',\n",
    "       'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath',\n",
    "       'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'KitchenQual',\n",
    "       'TotRmsAbvGrd', 'Functional', 'Fireplaces', 'FireplaceQu', 'GarageType',\n",
    "       'GarageYrBlt', 'GarageFinish', 'GarageCars', 'GarageArea', 'GarageQual',\n",
    "       'GarageCond', 'PavedDrive', 'WoodDeckSF', 'OpenPorchSF',\n",
    "       'EnclosedPorch', 'ThreeSsnPorch', 'ScreenPorch', 'PoolArea', 'PoolQC',\n",
    "       'Fence', 'MiscFeature', 'MiscVal', 'MoSold', 'YrSold', 'SaleType',\n",
    "       'SaleCondition', 'SalePrice']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Clean your Data\n",
    "<img src='https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAALAAAAAJGU3YjAwY2I5LTg1ODMtNDg5Ny04MTg2LTNhMWUwYTdkZDk2Zg.png'  width='300px'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check for Blanks or NA's!\n",
    "\n",
    "Note that there is a difference between **`NaN`, `''`, `None`** so be aware when looking at your data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### check for NA's in a column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>50</td>\n",
       "      <td>RL</td>\n",
       "      <td>85.0</td>\n",
       "      <td>14115</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>Shed</td>\n",
       "      <td>700</td>\n",
       "      <td>10</td>\n",
       "      <td>2009</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>143000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>75.0</td>\n",
       "      <td>10084</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>307000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10382</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Shed</td>\n",
       "      <td>350</td>\n",
       "      <td>11</td>\n",
       "      <td>2009</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>50</td>\n",
       "      <td>RM</td>\n",
       "      <td>51.0</td>\n",
       "      <td>6120</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>129900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>190</td>\n",
       "      <td>RL</td>\n",
       "      <td>50.0</td>\n",
       "      <td>7420</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>118000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "3   4          70       RL         60.0     9550   Pave   NaN      IR1   \n",
       "4   5          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "5   6          50       RL         85.0    14115   Pave   NaN      IR1   \n",
       "6   7          20       RL         75.0    10084   Pave   NaN      Reg   \n",
       "7   8          60       RL          NaN    10382   Pave   NaN      IR1   \n",
       "8   9          50       RM         51.0     6120   Pave   NaN      Reg   \n",
       "9  10         190       RL         50.0     7420   Pave   NaN      Reg   \n",
       "\n",
       "  LandContour Utilities    ...     PoolArea PoolQC  Fence MiscFeature MiscVal  \\\n",
       "0         Lvl    AllPub    ...            0    NaN    NaN         NaN       0   \n",
       "1         Lvl    AllPub    ...            0    NaN    NaN         NaN       0   \n",
       "2         Lvl    AllPub    ...            0    NaN    NaN         NaN       0   \n",
       "3         Lvl    AllPub    ...            0    NaN    NaN         NaN       0   \n",
       "4         Lvl    AllPub    ...            0    NaN    NaN         NaN       0   \n",
       "5         Lvl    AllPub    ...            0    NaN  MnPrv        Shed     700   \n",
       "6         Lvl    AllPub    ...            0    NaN    NaN         NaN       0   \n",
       "7         Lvl    AllPub    ...            0    NaN    NaN        Shed     350   \n",
       "8         Lvl    AllPub    ...            0    NaN    NaN         NaN       0   \n",
       "9         Lvl    AllPub    ...            0    NaN    NaN         NaN       0   \n",
       "\n",
       "  MoSold YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0      2   2008        WD         Normal     208500  \n",
       "1      5   2007        WD         Normal     181500  \n",
       "2      9   2008        WD         Normal     223500  \n",
       "3      2   2006        WD        Abnorml     140000  \n",
       "4     12   2008        WD         Normal     250000  \n",
       "5     10   2009        WD         Normal     143000  \n",
       "6      8   2007        WD         Normal     307000  \n",
       "7     11   2009        WD         Normal     200000  \n",
       "8      4   2008        WD        Abnorml     129900  \n",
       "9      1   2008        WD         Normal     118000  \n",
       "\n",
       "[10 rows x 81 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['Alley'].isna()].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fill in NA's\n",
    "\n",
    "<img src='http://csharpcorner.mindcrackerinc.netdna-cdn.com/UploadFile/219d4d/different-ways-to-replace-null-in-sql-server/Images/records.jpg'/>\n",
    "\n",
    "The `fillna` function is much handier to fill in NA's that trying to the boolean. If needed you can do a comparison with `numpy.NaN` or in our case `==np.nan`, `np.isnan(some number )`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['Alley'].fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### check the whole array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PoolQC          1453\n",
       "MiscFeature     1406\n",
       "Fence           1179\n",
       "FireplaceQu      690\n",
       "LotFrontage      259\n",
       "GarageCond        81\n",
       "GarageType        81\n",
       "GarageYrBlt       81\n",
       "GarageFinish      81\n",
       "GarageQual        81\n",
       "BsmtExposure      38\n",
       "BsmtFinType2      38\n",
       "BsmtFinType1      37\n",
       "BsmtCond          37\n",
       "BsmtQual          37\n",
       "MasVnrArea         8\n",
       "MasVnrType         8\n",
       "Electrical         1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "na_totals = train_df.isna().sum().sort_values(ascending=False)\n",
    "na_totals[na_totals>0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fill in NA's for all "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assign numerical features\n",
    "\n",
    "Identified square area as numeric. Identified any field that had that aspect to them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LotArea',\n",
       " 'MasVnrArea',\n",
       " 'BsmtFinSF1',\n",
       " 'BsmtFinSF2',\n",
       " 'BsmtUnfSF',\n",
       " 'TotalBsmtSF',\n",
       " 'FirsstFlrSF',\n",
       " 'SecondFlrSF',\n",
       " 'LowQualFinSF',\n",
       " 'GrLivArea',\n",
       " 'GarageArea',\n",
       " 'WoodDeckSF',\n",
       " 'OpenPorchSF',\n",
       " 'PoolArea',\n",
       " 'SalePrice',\n",
       " 'LotFrontage',\n",
       " 'MiscVal',\n",
       " 'EnclosedPorch',\n",
       " 'ThreeSsnPorch',\n",
       " 'ScreenPorch',\n",
       " 'OverallQual',\n",
       " 'OverallCond',\n",
       " 'YearBuilt']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numeric_cols = [x for x in train_df.columns if ('Area' in x) | ('SF' in x)] + ['SalePrice','LotFrontage','MiscVal','EnclosedPorch','ThreeSsnPorch','ScreenPorch','OverallQual','OverallCond','YearBuilt']\n",
    "\n",
    "for col in numeric_cols:\n",
    "    train_df[col] = train_df[col].astype(float)\n",
    "numeric_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Turn the rest into categorical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_cols = [x for x in train_df.columns if x not in numeric_cols]\n",
    "\n",
    "for col in categorical_cols:\n",
    "    train_df[col] = train_df[col].astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Double check the headings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#train_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Feature engineering\n",
    "\n",
    "<img src='https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAiuAAAAJGRkNGQzNjU5LTQzOTAtNDRlZC05NTZkLTcxZWM5YmYwZmRkNA.png' style='width:300px'>\n",
    "\n",
    "### Pandas: How to transform a column / apply a function against it\n",
    "In this case we will be looking at Log sale price. Here's how to transform a column. Both ways are valid, but the 2nd method only works because `np.log` was designed for handling vectors. If you tried \n",
    "\n",
    "`df['full name'] = df['first name'] + df['last name']` it will not work with this approach because there is no + operator that concats two vectors elementwise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create calculated fields (2 ways outlined below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['LogSalePrice'] = train_df['SalePrice'].map(lambda x : np.log(x)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['LogSalePrice'] = np.log(train_df['SalePrice'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10eca0dd8>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFpFJREFUeJzt3X+M3PWd3/HnuzgQwiS2+bVybesM\nOis9dG4IXnGOqKJduMthc4r5I0ggdBjqk6uWRolCxZlGuuqkVpBKudyhq7hYS3KmyuVCuVBbmLsc\nMmwrKkEOB4IhhHpNXVjs2EcCzm3I9er23T/mYzO37O7M2jOemU+fD2k03+9nPvP9vnZnee3X3/3O\nEJmJJKle/6DfASRJvWXRS1LlLHpJqpxFL0mVs+glqXIWvSRVzqKXpMpZ9JJUOYtekiq3pN8BAC6+\n+OK85JJLuOCCC/odpa2f/exn5uyiYckJw5PVnN01yDn37dv3VmZe0nZiZvb9tn79+nzqqadyGJiz\nu4YlZ+bwZDVndw1yTuC57KBjPXUjSZWz6CWpcha9JFXOopekyln0klQ5i16SKmfRS1LlLHpJqpxF\nL0mVG4iPQBhWa7bv6ct+D913Q1/2K2k4eUQvSZWz6CWpcm2LPiI+GhEvtNx+GhGfj4gLI+KJiDhQ\n7peX+RER90fEVES8GBFX9f7LkCTNp23RZ+armXllZl4JrAfeBR4FtgN7M3MtsLesA2wE1pbbNuCB\nXgSXJHVmsadurgMOZub/BDYDO8v4TuDGsrwZeKh8iuYzwLKIWNGVtJKkRYvmRxp3ODnia8D3MvMP\nI+KdzFzW8tjbmbk8Ih4D7svMp8v4XuC3M/O5WdvaRvOIn5GRkfUTExM0Go0ufEm9NTMzcyrn/jeP\n9yXDupVL285pzTnIhiUnDE9Wc3bXIOccHx/fl5mj7eZ1fHllRJwLfBq4p93UOcbe99skM3cAOwBG\nR0ez0WgwNjbWaZy+mZycPJXz9n5dXnnrWNs5rTkH2bDkhOHJas7uGpacC1nMqZuNNI/mj5b1oydP\nyZT7Y2V8Gljd8rxVwOEzDSpJOj2LKfpbgG+2rO8GtpTlLcCulvHbytU3G4DjmXnkjJNKkk5LR6du\nIuJDwK8B/6xl+D7g4YjYCrwO3FTGHwc2AVM0r9C5o2tpJUmL1lHRZ+a7wEWzxn5M8yqc2XMTuLMr\n6SRJZ8x3xkpS5Sx6SaqcRS9JlbPoJalyFr0kVc6il6TKWfSSVDmLXpIqZ9FLUuUsekmqnEUvSZWz\n6CWpcha9JFXOopekyln0klQ5i16SKmfRS1LlLHpJqpxFL0mV66joI2JZRDwSET+MiFci4hMRcWFE\nPBERB8r98jI3IuL+iJiKiBcj4qrefgmSpIV0ekT/B8BfZOY/Aj4GvAJsB/Zm5lpgb1kH2AisLbdt\nwANdTSxJWpS2RR8RHwE+CTwIkJl/l5nvAJuBnWXaTuDGsrwZeCibngGWRcSKrieXJHWkkyP6y4G/\nBr4eEc9HxEREXACMZOYRgHJ/aZm/Enij5fnTZUyS1AeRmQtPiBgFngGuycxnI+IPgJ8Cn83MZS3z\n3s7M5RGxB7g3M58u43uBuzNz36ztbqN5aoeRkZH1ExMTNBqNbn5tPTEzM3Mq5/43j/clw7qVS9vO\nac05yIYlJwxPVnN21yDnHB8f35eZo+3mLelgW9PAdGY+W9YfoXk+/mhErMjMI+XUzLGW+atbnr8K\nODx7o5m5A9gBMDo6mo1Gg7GxsQ7i9Nfk5OSpnLdv39OXDIduHWs7pzXnIBuWnDA8Wc3ZXcOScyFt\nT91k5o+ANyLio2XoOuAHwG5gSxnbAuwqy7uB28rVNxuA4ydP8UiSzr5OjugBPgt8IyLOBV4D7qD5\nS+LhiNgKvA7cVOY+DmwCpoB3y1xJUp90VPSZ+QIw13mg6+aYm8CdZ5hLktQlvjNWkipn0UtS5Sx6\nSaqcRS9JlbPoJalyFr0kVc6il6TKWfSSVDmLXpIqZ9FLUuUsekmqnEUvSZWz6CWpcha9JFXOopek\nyln0klQ5i16SKmfRS1LlLHpJqpxFL0mV66joI+JQROyPiBci4rkydmFEPBERB8r98jIeEXF/RExF\nxIsRcVUvvwBJ0sIWc0Q/nplXZuZoWd8O7M3MtcDesg6wEVhbbtuAB7oVVpK0eGdy6mYzsLMs7wRu\nbBl/KJueAZZFxIoz2I8k6Qx0WvQJ/GVE7IuIbWVsJDOPAJT7S8v4SuCNludOlzFJUh9EZrafFPEP\nM/NwRFwKPAF8Ftidmcta5rydmcsjYg9wb2Y+Xcb3Andn5r5Z29xG89QOIyMj6ycmJmg0Gl37wnpl\nZmbmVM79bx7vS4Z1K5e2ndOac5ANS04Ynqzm7K5Bzjk+Pr6v5XT6vJZ0srHMPFzuj0XEo8DVwNGI\nWJGZR8qpmWNl+jSwuuXpq4DDc2xzB7ADYHR0NBuNBmNjY53E6avJyclTOW/fvqcvGQ7dOtZ2TmvO\nQTYsOWF4spqzu4Yl50LanrqJiAsi4sMnl4FPAS8Bu4EtZdoWYFdZ3g3cVq6+2QAcP3mKR5J09nVy\nRD8CPBoRJ+f/SWb+RUT8FfBwRGwFXgduKvMfBzYBU8C7wB1dTy1J6ljbos/M14CPzTH+Y+C6OcYT\nuLMr6SRJZ8x3xkpS5Sx6SaqcRS9JlbPoJalyFr0kVc6il6TKWfSSVDmLXpIqZ9FLUuUsekmqnEUv\nSZWz6CWpcha9JFXOopekyln0klQ5i16SKmfRS1LlLHpJqpxFL0mVs+glqXIdF31EnBMRz0fEY2X9\nsoh4NiIORMS3IuLcMn5eWZ8qj6/pTXRJUicWc0T/OeCVlvUvAV/JzLXA28DWMr4VeDszfxH4Spkn\nSeqTjoo+IlYBNwATZT2Aa4FHypSdwI1leXNZpzx+XZkvSeqDyMz2kyIeAe4FPgz8K+B24Jly1E5E\nrAb+PDN/OSJeAq7PzOny2EHgVzLzrVnb3AZsAxgZGVk/MTFBo9Ho2hfWKzMzM6dy7n/zeF8yrFu5\ntO2c1pyDbFhywvBkNWd3DXLO8fHxfZk52m7eknYTIuI3gGOZuS8ixk4OzzE1O3jsvYHMHcAOgNHR\n0Ww0GoyNjc2eNnAmJydP5bx9+56+ZDh061jbOa05B9mw5IThyWrO7hqWnAtpW/TANcCnI2IT8EHg\nI8DvA8siYklmngBWAYfL/GlgNTAdEUuApcBPup5cktSRtufoM/OezFyVmWuAm4EnM/NW4CngM2Xa\nFmBXWd5d1imPP5mdnB+SJPXEmVxH/9vAFyJiCrgIeLCMPwhcVMa/AGw/s4iSpDPRyambUzJzEpgs\ny68BV88x52+Bm7qQTZLUBb4zVpIqZ9FLUuUWdepGg2FNB5d13rXuRE8u/zx03w1d36ak3vKIXpIq\nZ9FLUuUsekmqnEUvSZWz6CWpcha9JFXOopekyln0klQ5i16SKmfRS1LlLHpJqpxFL0mVs+glqXIW\nvSRVzqKXpMpZ9JJUubZFHxEfjIjvRsT3I+LliPjdMn5ZRDwbEQci4lsRcW4ZP6+sT5XH1/T2S5Ak\nLaSTI/r/BVybmR8DrgSuj4gNwJeAr2TmWuBtYGuZvxV4OzN/EfhKmSdJ6pO2RZ9NM2X1A+WWwLXA\nI2V8J3BjWd5c1imPXxcR0bXEkqRF6egcfUScExEvAMeAJ4CDwDuZeaJMmQZWluWVwBsA5fHjwEXd\nDC1J6lxkZueTI5YBjwK/A3y9nJ4hIlYDj2fmuoh4Gfj1zJwujx0Ers7MH8/a1jZgG8DIyMj6iYkJ\nGo1GN76mnpqZmTmVc/+bx/ucZn4j58PRn3d/u+tWLu3q9lq/n4NuWLKas7sGOef4+Pi+zBxtN2/J\nYjaame9ExCSwAVgWEUvKUfsq4HCZNg2sBqYjYgmwFPjJHNvaAewAGB0dzUajwdjY2GLi9MXk5OSp\nnLdv39PfMAu4a90Jvrx/US9vRw7dOtbV7bV+PwfdsGQ1Z3cNS86FdHLVzSXlSJ6IOB/4VeAV4Cng\nM2XaFmBXWd5d1imPP5mL+WeDJKmrOjnkWwHsjIhzaP5ieDgzH4uIHwB/GhH/FngeeLDMfxD4jxEx\nRfNI/uYe5JYkdaht0Wfmi8DH5xh/Dbh6jvG/BW7qSjpJ0hnznbGSVDmLXpIqZ9FLUuUsekmqnEUv\nSZWz6CWpcha9JFXOopekyln0klQ5i16SKmfRS1LlLHpJqpxFL0mVs+glqXIWvSRVzqKXpMpZ9JJU\nOYtekipn0UtS5Sx6Sapc26KPiNUR8VREvBIRL0fE58r4hRHxREQcKPfLy3hExP0RMRURL0bEVb3+\nIiRJ8+vkiP4EcFdm/hKwAbgzIq4AtgN7M3MtsLesA2wE1pbbNuCBrqeWJHWsbdFn5pHM/F5Z/hvg\nFWAlsBnYWabtBG4sy5uBh7LpGWBZRKzoenJJUkcWdY4+ItYAHweeBUYy8wg0fxkAl5ZpK4E3Wp42\nXcYkSX0QmdnZxIgG8F+Af5eZ346IdzJzWcvjb2fm8ojYA9ybmU+X8b3A3Zm5b9b2ttE8tcPIyMj6\niYkJGo1Gd76qHpqZmTmVc/+bx/ucZn4j58PRn3d/u+tWLu3q9lq/n4NuWLKas7sGOef4+Pi+zBxt\nN29JJxuLiA8AfwZ8IzO/XYaPRsSKzDxSTs0cK+PTwOqWp68CDs/eZmbuAHYAjI6OZqPRYGxsrJM4\nfTU5OXkq5+3b9/Q3zALuWneCL+/v6OVdlEO3jnV1e63fz0E3LFnN2V3DknMhnVx1E8CDwCuZ+Xst\nD+0GtpTlLcCulvHbytU3G4DjJ0/xSJLOvk4O+a4BfhPYHxEvlLF/DdwHPBwRW4HXgZvKY48Dm4Ap\n4F3gjq4mliQtStuiL+faY56Hr5tjfgJ3nmEuSVKX+M5YSaqcRS9JlbPoJalyFr0kVc6il6TKWfSS\nVDmLXpIqZ9FLUuUsekmqXPc/9eosW3OWP1jsrnUnBvrDzCRpNo/oJalyFr0kVc6il6TKDf05ep1d\n3f6bSKd/8zh03w1d3a/0/xOP6CWpcha9JFXOopekyln0klQ5i16SKte26CPiaxFxLCJeahm7MCKe\niIgD5X55GY+IuD8ipiLixYi4qpfhJUntdXJE/8fA9bPGtgN7M3MtsLesA2wE1pbbNuCB7sSUJJ2u\ntkWfmf8V+Mms4c3AzrK8E7ixZfyhbHoGWBYRK7oVVpK0eKd7jn4kM48AlPtLy/hK4I2WedNlTJLU\nJ5GZ7SdFrAEey8xfLuvvZOaylsffzszlEbEHuDczny7je4G7M3PfHNvcRvP0DiMjI+snJiZoNBqL\n/gL2v3l80c85EyPnw9Gfn9Vdnpbacq5bubT3YdqYmZk5rZ/Rs82c3TXIOcfHx/dl5mi7eaf7EQhH\nI2JFZh4pp2aOlfFpYHXLvFXA4bk2kJk7gB0Ao6Oj2Wg0GBsbW3SQs/2RwXetO8GX9w/+J0fUlvPQ\nrWO9D9PG5OTkaf2Mnm3m7K5hybmQ0z11sxvYUpa3ALtaxm8rV99sAI6fPMUjSeqPtodSEfFNYAy4\nOCKmgX8D3Ac8HBFbgdeBm8r0x4FNwBTwLnBHDzJLkhahbdFn5i3zPHTdHHMTuPNMQ0mSusd3xkpS\n5Sx6SaqcRS9JlbPoJalyFr0kVc6il6TKWfSSVDmLXpIqZ9FLUuUG/1OvJGDNWf7wupMO3XdDX/Yr\ndZNH9JJUOYtekipn0UtS5Sx6SaqcRS9JlbPoJalyFr0kVc6il6TK+YYpaQGtb9S6a90Jbj+Lb9zy\nzVrqFo/oJalyPSn6iLg+Il6NiKmI2N6LfUiSOtP1oo+Ic4D/AGwErgBuiYgrur0fSVJnenFEfzUw\nlZmvZebfAX8KbO7BfiRJHejFH2NXAm+0rE8Dv9KD/UhVO91P7DzbfzQ+XeZsOht/dI/M7O4GI24C\nfj0zf6us/yZwdWZ+dta8bcC2svpR4MfAW10N0xsXY85uGpacMDxZzdldg5zzFzLzknaTenFEPw2s\nbllfBRyePSkzdwA7Tq5HxHOZOdqDPF1lzu4alpwwPFnN2V3DknMhvThH/1fA2oi4LCLOBW4Gdvdg\nP5KkDnT9iD4zT0TEvwS+A5wDfC0zX+72fiRJnenJO2Mz83Hg8UU+bUf7KQPBnN01LDlheLKas7uG\nJee8uv7HWEnSYPEjECSpdpnZ1xtwPfAqMAVs7+F+vgYcA15qGbsQeAI4UO6Xl/EA7i+ZXgSuannO\nljL/ALClZXw9sL88537e+9fSnPtYIOdq4CngFeBl4HODmBX4IPBd4Psl5++W8cuAZ8s2vgWcW8bP\nK+tT5fE1Ldu6p4y/SvPS3AV/NubbR5vv6znA88BjA57zUHltXgCeG8TXvsxfBjwC/JDmz+onBi0n\nzcu2X2i5/RT4/KDlPBu3vu245T++g8DlwLk0S+OKHu3rk8BV/P2i//cn/8MEtgNfKsubgD8vL/wG\n4NmWF++1cr+8LJ/8Iflu+WGP8tyNC+1jgZwrTv6AAR8G/jvNj5IYqKzluY2y/AGahbYBeBi4uYz/\nEfDPy/K/AP6oLN8MfKssX1Fe9/NoFuPB8nMx78/GfPto8339AvAnvFf0g5rzEHDxrLGBeu3LnJ3A\nb5Xlc2kW/8DlnNU1PwJ+YZBz9urW76L/BPCdlvV7gHt6uL81/P2ifxVYUZZXAK+W5a8Ct8yeB9wC\nfLVl/KtlbAXww5bxU/Pm28ciMu8Cfm2QswIfAr5H8x3QbwFLZr++NK/C+kRZXlLmxezX/OS8+X42\nynPm3McC+VYBe4FrgccW2kY/c5Z5h3h/0Q/Uaw98BPgflKPXQc05K9ungP826Dl7dev3Ofq5Pi5h\n5Vnc/0hmHgEo95e2ybXQ+PQc4wvto62IWAN8nObR8sBljYhzIuIFmqfEnqB5ZPtOZp6YY9un8pTH\njwMXnUb+ixbYx3x+H7gb+L9lfaFt9DMnQAJ/GRH7yrvHYfBe+8uBvwa+HhHPR8RERFwwgDlb3Qx8\ns802BiFnT/S76GOOsTzrKd5vvlyLHT/9ABEN4M+Az2fmTxeaushMXcuamf8nM6+kecR8NfBLC2y7\nWzkXlT8ifgM4lpn7WocHLWeLazLzKpqf/npnRHxygbn9eu2X0DwN+kBmfhz4Gc3TE/Pp639P5Y2b\nnwb+U7upi8wzqP31Pv0u+o4+LqGHjkbECoByf6xNroXGV80xvtA+5hURH6BZ8t/IzG8PclaAzHwH\nmKR5XnNZRJx8f0brtk/lKY8vBX5yGvnfWmAfc7kG+HREHKL5SarX0jzCH7ScAGTm4XJ/DHiU5i/Q\nQXvtp4HpzHy2rD9Cs/gHLedJG4HvZebRNtvod86e6XfR9/vjEnbT/Gs65X5Xy/ht0bQBOF7++fUd\n4FMRsTwiltM87/ed8tjfRMSGiAjgtlnbmmsfcyrPfxB4JTN/b1CzRsQlEbGsLJ8P/CrNqy+eAj4z\nT86T2/4M8GQ2T2DuBm6OiPMi4jJgLc0/cM35s1GeM98+3icz78nMVZm5pmzjycy8ddBylu/jBRHx\n4ZPLNF+zlxiw1z4zfwS8EREfLUPXAT8YtJwtbuG90zYLbaPfOXunn38gKH+o2ETzypKDwBd7uJ9v\nAkeA/03zN/FWmudR99K8BGovcGGZGzT/5ykHaV46NdqynX9K81KqKeCOlvFRmv9RHgT+kPcus5pz\nHwvk/Cc0//n3Iu9dFrZp0LIC/5jm5Yovlm39Thm/nGYBTtH8p/J5ZfyDZX2qPH55y7a+WLK8Srlq\nYaGfjfn20cHPwBjvXXUzcDnL/O/z3iWrX1zodenXa1/mXwk8V17//0zzapRBzPkhmp+Mu7RlbOBy\n9vrmO2MlqXL9PnUjSeoxi16SKmfRS1LlLHpJqpxFL0mVs+glqXIWvSRVzqKXpMr9Pw+dRXw0UJnS\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ddf7e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_df['SalePrice'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10ea43ac8>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEFZJREFUeJzt3X9sXWd9x/H3dw2FqhlN2oBVkmxh\nI5pAizZaq6pWCTl0o6VFpNOoxBZBWjLlj5UNiUyQDYlp2hDtpq4IaQJFlC2bGAbxQ81IplIFPIS0\nMppRmobCklYZdRslgrbZDIUt47s//BQZz+69ts/1tb+8X5J1z3nOc5/7fHV8Pz557o9EZiJJqutn\nhj0BSdJgGfSSVJxBL0nFGfSSVJxBL0nFGfSSVJxBL0nFGfSSVJxBL0nFrRn2BAA2bNiQW7Zs6WSs\n733ve1x88cWdjDVMFeqoUAPUqKNCDVCjji5rOHr06Hcy8yW9+q2IoN+yZQsPPPBAJ2NNTEwwNjbW\nyVjDVKGOCjVAjToq1AA16uiyhoj4j376uXQjScUZ9JJUnEEvScUZ9JJUnEEvScUZ9JJUnEEvScUZ\n9JJUnEEvScWtiE/GSr1s2XdoKI976vYbh/K4Upe8opek4gx6SSrOoJek4gx6SSrOoJek4gx6SSrO\noJek4gx6SSrOoJek4gx6SSrOoJek4gx6SSrOoJek4gx6SSrOoJek4gx6SSrOoJek4gx6SSrOoJek\n4gx6SSrOoJek4gx6SSrOoJek4gx6SSrOoJek4gx6SSqu76CPiAsi4msR8bm2//KI+EpEnIiIT0TE\nha39hW3/ZDu+ZTBTlyT1YyFX9O8AHpmxfwdwV2ZuBZ4Gdrf23cDTmfkK4K7WT5I0JH0FfURsAm4E\nPtL2A3gt8KnW5QBwU9ve0fZpx69t/SVJQ9DvFf0HgHcBP2r7lwHPZOb5tj8JbGzbG4HHAdrxc62/\nJGkIIjOfv0PEG4AbMvP3ImIM+EPgVuBf2vIMEbEZOJyZ2yLiOHBdZk62Y48CV2Xmd2eNuwfYAzAy\nMnLl+Ph4JwVNTU2xdu3aTsYapgp1dFnDsSfOdTLOYoxcBGeeXf7H3bbxks7GqvD7BDXq6LKG7du3\nH83M0V791vQx1jXAGyPiBuBFwIuZvsJfFxFr2lX7JuDJ1n8S2AxMRsQa4BLgqdmDZuZ+YD/A6Oho\njo2N9TGV3iYmJuhqrGGqUEeXNdyy71An4yzG3m3nufNYP0+Vbp3aOdbZWBV+n6BGHcOooefSTWb+\nUWZuyswtwJuBL2TmTuCLwJtat13APW37YNunHf9C9vpngyRpYJbyPvp3A++MiJNMr8Hf3drvBi5r\n7e8E9i1tipKkpVjQv0czcwKYaNuPAVfN0ecHwM0dzE2S1AE/GStJxRn0klScQS9JxRn0klScQS9J\nxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0\nklScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klSc\nQS9JxRn0klScQS9JxRn0klScQS9JxfUM+oh4UUT8a0R8PSKOR8SftvaXR8RXIuJERHwiIi5s7S9s\n+yfb8S2DLUGS9Hz6uaL/IfDazPwV4FeB6yPiauAO4K7M3Ao8Dexu/XcDT2fmK4C7Wj9J0pD0DPqc\nNtV2X9B+Engt8KnWfgC4qW3vaPu049dGRHQ2Y0nSgvS1Rh8RF0TEg8BZ4D7gUeCZzDzfukwCG9v2\nRuBxgHb8HHBZl5OWJPUvMrP/zhHrgM8C7wX+pi3PEBGbgcOZuS0ijgPXZeZkO/YocFVmfnfWWHuA\nPQAjIyNXjo+Pd1EPU1NTrF27tpOxhqlCHV3WcOyJc52MsxgjF8GZZ5f/cbdtvKSzsSr8PkGNOrqs\nYfv27Uczc7RXvzULGTQzn4mICeBqYF1ErGlX7ZuAJ1u3SWAzMBkRa4BLgKfmGGs/sB9gdHQ0x8bG\nFjKVeU1MTNDVWMNUoY4ua7hl36FOxlmMvdvOc+exBT1VOnFq51hnY1X4fYIadQyjhn7edfOSdiVP\nRFwE/DrwCPBF4E2t2y7gnrZ9sO3Tjn8hF/LPBklSp/q5TLkcOBARFzD9h+GTmfm5iPgGMB4Rfw58\nDbi79b8b+PuIOMn0lfybBzBvSVKfegZ9Zj4EvHqO9seAq+Zo/wFwcyezkyQtmZ+MlaTiDHpJKs6g\nl6TiDHpJKs6gl6TiDHpJKs6gl6TiDHpJKs6gl6TiDHpJKs6gl6TiDHpJKs6gl6TiDHpJKs6gl6Ti\nDHpJKs6gl6TiDHpJKs6gl6TiDHpJKs6gl6TiDHpJKs6gl6TiDHpJKs6gl6TiDHpJKs6gl6TiDHpJ\nKs6gl6TiDHpJKm7NsCcgaW5b9h3qbKy9285zS5/jnbr9xs4eVyuDV/SSVJxBL0nFGfSSVJxBL0nF\n9Qz6iNgcEV+MiEci4nhEvKO1XxoR90XEiXa7vrVHRHwwIk5GxEMRccWgi5Akza+fK/rzwN7MfCVw\nNXBbRLwK2AccycytwJG2D/B6YGv72QN8qPNZS5L61jPoM/N0Zv5b2/4v4BFgI7ADONC6HQBuats7\ngL/LafcD6yLi8s5nLknqy4LW6CNiC/Bq4CvASGaehuk/BsBLW7eNwOMz7jbZ2iRJQxCZ2V/HiLXA\nPwPvy8zPRMQzmbluxvGnM3N9RBwC3p+ZX27tR4B3ZebRWePtYXpph5GRkSvHx8c7KWhqaoq1a9d2\nMtYwVaijyxqOPXGuk3EWY+QiOPPs0B6+EwupYdvGSwY7mSXwefGTtm/ffjQzR3v16+uTsRHxAuDT\nwMcy8zOt+UxEXJ6Zp9vSzNnWPglsnnH3TcCTs8fMzP3AfoDR0dEcGxvrZyo9TUxM0NVYw1Shji5r\n6PdTnYOwd9t57jy2uj9EvpAaTu0cG+xklsDnxeL0866bAO4GHsnMv5px6CCwq23vAu6Z0f7W9u6b\nq4Fzzy3xSJKWXz9/4q8B3gIci4gHW9sfA7cDn4yI3cC3gZvbscPADcBJ4PvArZ3OWJK0ID2Dvq21\nxzyHr52jfwK3LXFekqSO+MlYSSrOoJek4lb3Wwm07BbyHekL+Q50SYPjFb0kFWfQS1JxBr0kFWfQ\nS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1Jx\nBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0k\nFWfQS1JxBr0kFWfQS1JxPYM+Ij4aEWcj4uEZbZdGxH0RcaLdrm/tEREfjIiTEfFQRFwxyMlLknrr\n54r+b4HrZ7XtA45k5lbgSNsHeD2wtf3sAT7UzTQlSYvVM+gz80vAU7OadwAH2vYB4KYZ7X+X0+4H\n1kXE5V1NVpK0cItdox/JzNMA7falrX0j8PiMfpOtTZI0JJGZvTtFbAE+l5m/3Pafycx1M44/nZnr\nI+IQ8P7M/HJrPwK8KzOPzjHmHqaXdxgZGblyfHy8g3JgamqKtWvXdjLWMK3UOo49ca7vviMXwZln\nBziZZVKhjoXUsG3jJYOdzBKs1OfFQnRZw/bt249m5mivfmsWOf6ZiLg8M0+3pZmzrX0S2Dyj3ybg\nybkGyMz9wH6A0dHRHBsbW+RUftLExARdjTVMK7WOW/Yd6rvv3m3nufPYYn/FVo4KdSykhlM7xwY7\nmSVYqc+LhRhGDYtdujkI7Grbu4B7ZrS/tb375mrg3HNLPJKk4ej5Jz4iPg6MARsiYhL4E+B24JMR\nsRv4NnBz634YuAE4CXwfuHUAc5YkLUDPoM/M357n0LVz9E3gtqVOStLwbFnA8lzXTt1+49AeuzI/\nGStJxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9JxRn0klScQS9J\nxRn0klScQS9Jxa3u/x/tp9Qwvy9c0urjFb0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQ\nS1JxBr0kFWfQS1JxBr0kFWfQS1JxBr0kFWfQS1Jxfk2xpBWj11dw7912nlsG8DXdp26/sfMxVxKv\n6CWpOINekooz6CWpONfol2CQ/6XfoNYiJf30GcgVfURcHxHfioiTEbFvEI8hSepP51f0EXEB8NfA\nbwCTwFcj4mBmfqPrx5KkLgzyX+ezzf7X+nK842cQSzdXAScz8zGAiBgHdgADCfrZJ8glD0n6SYNY\nutkIPD5jf7K1SZKGIDKz2wEjbgauy8zfbftvAa7KzN+f1W8PsKft/hLwrY6msAH4TkdjDVOFOirU\nADXqqFAD1Kijyxp+PjNf0qvTIJZuJoHNM/Y3AU/O7pSZ+4H9XT94RDyQmaNdj7vcKtRRoQaoUUeF\nGqBGHcOoYRBLN18FtkbEyyPiQuDNwMEBPI4kqQ+dX9Fn5vmIeDtwL3AB8NHMPN7140iS+jOQD0xl\n5mHg8CDG7kPny0FDUqGOCjVAjToq1AA16lj2Gjp/MVaStLL4XTeSVNyqCfqI+GhEnI2Ih2e0XRoR\n90XEiXa7fp77/m9EPNh+hvrC8Dx13BwRxyPiRxEx76vxK+WrJZZYw6mIONbOxQPLM+N55zJXHX8Z\nEd+MiIci4rMRsW6e+67kc9FvDSv9XPxZq+HBiPh8RLxsnvvuahlwIiJ2Ld+s/988llLDYDMqM1fF\nD/Aa4Arg4RltfwHsa9v7gDvmue/UsOffo45XMv1ZgglgdJ77XQA8CvwCcCHwdeBVq6mG1u8UsGHY\n5+F56ngdsKZt3zHX79QqOBc9a1gl5+LFM7b/APjwHPe7FHis3a5v2+tXUw3t2EAzatVc0Wfml4Cn\nZjXvAA607QPATcs6qUWYq47MfCQze31g7MdfLZGZ/w0899USy24JNawo89Tx+cw833bvZ/pzILOt\n9HPRTw0ryjx1/OeM3YuBuV5QvA64LzOfysyngfuA6wc20eexhBoGbtUE/TxGMvM0QLt96Tz9XhQR\nD0TE/RGx4v8YzKPKV0sk8PmIONo+Hb2SvQ34pznaV9O5mK8GWAXnIiLeFxGPAzuB987RZcWfiz5q\ngAFn1GoP+n79XE5/Eu13gA9ExC8Oe0KLEHO0rca3TF2TmVcArwdui4jXDHtCc4mI9wDngY/NdXiO\nthV3LnrUAKvgXGTmezJzM9M1vH2OLiv+XPRRAww4o1Z70J+JiMsB2u3ZuTpl5pPt9jGm15BfvVwT\n7FBfXy2x0s04F2eBzzK9DLKitBf03gDszLaAOsuKPxd91LAqzsUM/wD81hztK/5czDBfDQPPqNUe\n9AeB515l3wXcM7tDRKyPiBe27Q3ANQzoK5MHbNV/tUREXBwRP/vcNtMvGj78/PdaXhFxPfBu4I2Z\n+f15uq3oc9FPDavkXGydsftG4JtzdLsXeF17nq9nuo57l2N+/einhmXJqGG8Or3IV7Q/DpwG/ofp\nv+K7gcuAI8CJdntp6zsKfKRt/xpwjOl3RhwDdq/AOn6zbf8QOAPc2/q+DDg84743AP/O9Ds+3rPa\namD6XSpfbz/Hh1nD89Rxkuk13wfbz4dX4bnoWcMqORefZvqPz0PAPwIbW98fP7/b/ttazSeBW1db\nDcuRUX4yVpKKW+1LN5KkHgx6SSrOoJek4gx6SSrOoJek4gx6SSrOoJek4gx6SSru/wAErszkbMqY\ngAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ea59668>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_df['LogSalePrice'].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What if you wanted to make a categorical? (2 ways)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['above_200k'] = train_df['SalePrice'].map(lambda x : 1 if x > 200000 else 0) \n",
    "train_df['above_200k'] = train_df['above_200k'].astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.loc[train_df['SalePrice']>200000,'above_200k'] = 1\n",
    "train_df.loc[train_df['SalePrice']<=200000,'above_200k'] = 0\n",
    "train_df['above_200k'] = train_df['above_200k'].astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas: how to add columns together (numeric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LivArea_Total</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>PoolArea</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2258.0</td>\n",
       "      <td>1710.0</td>\n",
       "      <td>548.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1722.0</td>\n",
       "      <td>1262.0</td>\n",
       "      <td>460.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2394.0</td>\n",
       "      <td>1786.0</td>\n",
       "      <td>608.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2359.0</td>\n",
       "      <td>1717.0</td>\n",
       "      <td>642.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3034.0</td>\n",
       "      <td>2198.0</td>\n",
       "      <td>836.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LivArea_Total  GrLivArea  GarageArea  PoolArea\n",
       "0         2258.0     1710.0       548.0       0.0\n",
       "1         1722.0     1262.0       460.0       0.0\n",
       "2         2394.0     1786.0       608.0       0.0\n",
       "3         2359.0     1717.0       642.0       0.0\n",
       "4         3034.0     2198.0       836.0       0.0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['LivArea_Total'] = train_df['GrLivArea'] + train_df['GarageArea'] + train_df['PoolArea']\n",
    "train_df[['LivArea_Total','GrLivArea','GarageArea','PoolArea']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas: how to apply a function rowwise (example is adding strings)\n",
    "\n",
    "- `axis=0` - pass a **column** of data\n",
    "- `axis=1` - pass a **row** of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Lot_desc</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotShape</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RLReg</td>\n",
       "      <td>RL</td>\n",
       "      <td>Reg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RLReg</td>\n",
       "      <td>RL</td>\n",
       "      <td>Reg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RLIR1</td>\n",
       "      <td>RL</td>\n",
       "      <td>IR1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RLIR1</td>\n",
       "      <td>RL</td>\n",
       "      <td>IR1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RLIR1</td>\n",
       "      <td>RL</td>\n",
       "      <td>IR1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Lot_desc MSZoning LotShape\n",
       "0    RLReg       RL      Reg\n",
       "1    RLReg       RL      Reg\n",
       "2    RLIR1       RL      IR1\n",
       "3    RLIR1       RL      IR1\n",
       "4    RLIR1       RL      IR1"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['Lot_desc'] = train_df.apply(lambda val : val['MSZoning'] + val['LotShape'], axis=1)\n",
    "train_df[['Lot_desc','MSZoning','LotShape']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sklearn: How to scale fields\n",
    "\n",
    "<img src='http://images.slideplayer.com/38/10783489/slides/slide_36.jpg' style='width:500px'/>\n",
    "\n",
    "- `StandardScaler` - subtract the mean and divide by std\n",
    "- `MaxAbsScaler` - transform down to [-1, 1] bounds\n",
    "- `QuantileTransformer` - transform down to [0 1] bounds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotArea_norm</th>\n",
       "      <th>LotArea_mas</th>\n",
       "      <th>LotArea_qs</th>\n",
       "      <th>LotArea</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.207142</td>\n",
       "      <td>0.039258</td>\n",
       "      <td>0.349349</td>\n",
       "      <td>8450.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.091886</td>\n",
       "      <td>0.044600</td>\n",
       "      <td>0.525025</td>\n",
       "      <td>9600.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.073480</td>\n",
       "      <td>0.052266</td>\n",
       "      <td>0.720220</td>\n",
       "      <td>11250.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.096897</td>\n",
       "      <td>0.044368</td>\n",
       "      <td>0.509667</td>\n",
       "      <td>9550.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.375148</td>\n",
       "      <td>0.066250</td>\n",
       "      <td>0.897123</td>\n",
       "      <td>14260.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LotArea_norm  LotArea_mas  LotArea_qs  LotArea\n",
       "0     -0.207142     0.039258    0.349349   8450.0\n",
       "1     -0.091886     0.044600    0.525025   9600.0\n",
       "2      0.073480     0.052266    0.720220  11250.0\n",
       "3     -0.096897     0.044368    0.509667   9550.0\n",
       "4      0.375148     0.066250    0.897123  14260.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['LotArea_norm'] = train_df['LotArea']\n",
    "\n",
    "ss = StandardScaler()\n",
    "mas = MaxAbsScaler()\n",
    "qs = QuantileTransformer()\n",
    "train_df['LotArea_norm'] = ss.fit_transform(train_df[['LotArea']])\n",
    "train_df['LotArea_mas'] = mas.fit_transform(train_df[['LotArea']])\n",
    "train_df['LotArea_qs'] = qs.fit_transform(train_df[['LotArea']])\n",
    "\n",
    "\n",
    "train_df[['LotArea_norm','LotArea_mas','LotArea_qs', 'LotArea']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sklearn: 1 hot encoding and Label Encoding\n",
    "\n",
    "<img src='https://cdn-images-1.medium.com/max/1600/1*ZsYkXEa1qrKeGiplNnFHyQ.jpeg' width='500px' />\n",
    "\n",
    "### Sklearn: 1 hot encoding \n",
    "We know that MSSubclass is a categorical, lets turn it into 1 hot encoding. Note if you pass your entire dataframe, it will convert all if it AUTOMATICALLY. \n",
    "\n",
    "#### If you want to do this piece meal, isolate the single target categorical column, transform it and join it back together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RL</td>\n",
       "      <td>208500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RL</td>\n",
       "      <td>181500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RL</td>\n",
       "      <td>223500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RL</td>\n",
       "      <td>140000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RL</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  MSZoning  SalePrice\n",
       "0       RL   208500.0\n",
       "1       RL   181500.0\n",
       "2       RL   223500.0\n",
       "3       RL   140000.0\n",
       "4       RL   250000.0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "small_df = train_df[['MSZoning','SalePrice']].copy()\n",
    "small_df['MSZoning'] = small_df['MSZoning'].astype('category')\n",
    "small_df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>MSZoning_C (all)</th>\n",
       "      <th>MSZoning_FV</th>\n",
       "      <th>MSZoning_RH</th>\n",
       "      <th>MSZoning_RL</th>\n",
       "      <th>MSZoning_RM</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>208500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>181500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>223500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>140000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>250000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   SalePrice  MSZoning_C (all)  MSZoning_FV  MSZoning_RH  MSZoning_RL  \\\n",
       "0   208500.0                 0            0            0            1   \n",
       "1   181500.0                 0            0            0            1   \n",
       "2   223500.0                 0            0            0            1   \n",
       "3   140000.0                 0            0            0            1   \n",
       "4   250000.0                 0            0            0            1   \n",
       "\n",
       "   MSZoning_RM  \n",
       "0            0  \n",
       "1            0  \n",
       "2            0  \n",
       "3            0  \n",
       "4            0  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.get_dummies(small_df).head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sklearn: Label Encoding\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>208500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>181500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>223500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70</td>\n",
       "      <td>140000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  MSSubClass  SalePrice\n",
       "0         60   208500.0\n",
       "1         20   181500.0\n",
       "2         60   223500.0\n",
       "3         70   140000.0\n",
       "4         60   250000.0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "small_df = train_df[['MSSubClass','SalePrice']].copy()\n",
    "small_df['MSSubClass'] = small_df['MSSubClass'].astype('category')\n",
    "small_df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 0, 5, ..., 6, 0, 0])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "le = LabelEncoder()\n",
    "trf_MSSubClass = le.fit_transform(small_df['MSSubClass'])\n",
    "trf_MSSubClass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 20,  30,  40,  45,  50,  60,  70,  75,  80,  85,  90, 120, 160,\n",
       "       180, 190])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "le.classes_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([60, 20, 60, ..., 70, 20, 20])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "le.inverse_transform(trf_MSSubClass)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Prep! making y = X , 2 methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='http://okfnlabs.org/img/posts/olap-slice_and_dice-overview.png' style='width:400px' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols = [col for col in train_df.columns if 'Price' not in col]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 1 manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    12.247694\n",
      "1    12.109011\n",
      "Name: LogSalePrice, dtype: float64 \n",
      "\n",
      "   Id MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
      "0  1         60       RL         65.0   8450.0   Pave     0      Reg   \n",
      "1  2         20       RL         80.0   9600.0   Pave     0      Reg   \n",
      "\n",
      "  LandContour Utilities     ...     MoSold YrSold SaleType SaleCondition  \\\n",
      "0         Lvl    AllPub     ...          2   2008       WD        Normal   \n",
      "1         Lvl    AllPub     ...          5   2007       WD        Normal   \n",
      "\n",
      "  above_200k LivArea_Total Lot_desc  LotArea_norm  LotArea_mas  LotArea_qs  \n",
      "0          1        2258.0    RLReg     -0.207142     0.039258    0.349349  \n",
      "1          0        1722.0    RLReg     -0.091886     0.044600    0.525025  \n",
      "\n",
      "[2 rows x 86 columns]\n"
     ]
    }
   ],
   "source": [
    "y = train_df['LogSalePrice']\n",
    "X = train_df[feature_cols]\n",
    "print(y.head(2),'\\n\\n', X.head(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 1b: Quick and Dirty way to get 100% numerical matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>...</th>\n",
       "      <th>Lot_desc_RHIR2</th>\n",
       "      <th>Lot_desc_RHReg</th>\n",
       "      <th>Lot_desc_RLIR1</th>\n",
       "      <th>Lot_desc_RLIR2</th>\n",
       "      <th>Lot_desc_RLIR3</th>\n",
       "      <th>Lot_desc_RLReg</th>\n",
       "      <th>Lot_desc_RMIR1</th>\n",
       "      <th>Lot_desc_RMIR2</th>\n",
       "      <th>Lot_desc_RMIR3</th>\n",
       "      <th>Lot_desc_RMReg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65.0</td>\n",
       "      <td>8450.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>706.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>856.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80.0</td>\n",
       "      <td>9600.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>978.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>284.0</td>\n",
       "      <td>1262.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68.0</td>\n",
       "      <td>11250.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>486.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>434.0</td>\n",
       "      <td>920.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60.0</td>\n",
       "      <td>9550.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1915.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>216.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>540.0</td>\n",
       "      <td>756.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84.0</td>\n",
       "      <td>14260.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>655.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>490.0</td>\n",
       "      <td>1145.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 2010 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  MasVnrArea  \\\n",
       "0         65.0   8450.0          7.0          5.0     2003.0       196.0   \n",
       "1         80.0   9600.0          6.0          8.0     1976.0         0.0   \n",
       "2         68.0  11250.0          7.0          5.0     2001.0       162.0   \n",
       "3         60.0   9550.0          7.0          5.0     1915.0         0.0   \n",
       "4         84.0  14260.0          8.0          5.0     2000.0       350.0   \n",
       "\n",
       "   BsmtFinSF1  BsmtFinSF2  BsmtUnfSF  TotalBsmtSF       ...        \\\n",
       "0       706.0         0.0      150.0        856.0       ...         \n",
       "1       978.0         0.0      284.0       1262.0       ...         \n",
       "2       486.0         0.0      434.0        920.0       ...         \n",
       "3       216.0         0.0      540.0        756.0       ...         \n",
       "4       655.0         0.0      490.0       1145.0       ...         \n",
       "\n",
       "   Lot_desc_RHIR2  Lot_desc_RHReg  Lot_desc_RLIR1  Lot_desc_RLIR2  \\\n",
       "0               0               0               0               0   \n",
       "1               0               0               0               0   \n",
       "2               0               0               1               0   \n",
       "3               0               0               1               0   \n",
       "4               0               0               1               0   \n",
       "\n",
       "   Lot_desc_RLIR3  Lot_desc_RLReg  Lot_desc_RMIR1  Lot_desc_RMIR2  \\\n",
       "0               0               1               0               0   \n",
       "1               0               1               0               0   \n",
       "2               0               0               0               0   \n",
       "3               0               0               0               0   \n",
       "4               0               0               0               0   \n",
       "\n",
       "   Lot_desc_RMIR3  Lot_desc_RMReg  \n",
       "0               0               0  \n",
       "1               0               0  \n",
       "2               0               0  \n",
       "3               0               0  \n",
       "4               0               0  \n",
       "\n",
       "[5 rows x 2010 columns]"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_numerical = pd.get_dummies(X)\n",
    "X_numerical.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With Patsy\n",
    "\n",
    "https://patsy.readthedocs.io/en/latest/formulas.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   LogSalePrice\n",
      "0     12.247694\n",
      "1     12.109011 \n",
      "\n",
      "    Intercept  Id[T.2]  Id[T.3]  Id[T.4]  Id[T.5]  Id[T.6]  Id[T.7]  Id[T.8]  \\\n",
      "0        1.0      0.0      0.0      0.0      0.0      0.0      0.0      0.0   \n",
      "1        1.0      1.0      0.0      0.0      0.0      0.0      0.0      0.0   \n",
      "\n",
      "   Id[T.9]  Id[T.10]     ...      OpenPorchSF  EnclosedPorch  ThreeSsnPorch  \\\n",
      "0      0.0       0.0     ...             61.0            0.0            0.0   \n",
      "1      0.0       0.0     ...              0.0            0.0            0.0   \n",
      "\n",
      "   ScreenPorch  PoolArea  MiscVal  LivArea_Total  LotArea_norm  LotArea_mas  \\\n",
      "0          0.0       0.0      0.0         2258.0     -0.207142     0.039258   \n",
      "1          0.0       0.0      0.0         1722.0     -0.091886     0.044600   \n",
      "\n",
      "   LotArea_qs  \n",
      "0    0.349349  \n",
      "1    0.525025  \n",
      "\n",
      "[2 rows x 1951 columns]\n"
     ]
    }
   ],
   "source": [
    "import patsy\n",
    "formula = 'LogSalePrice ~ %s' % (' + '.join(feature_cols)) \n",
    "y, X = patsy.dmatrices(formula, train_df, return_type='dataframe')\n",
    "print(y.head(2),'\\n\\n', X.head(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note with out fast.ai train test split, patsy is great to make a numerical matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Test Split\n",
    "\n",
    "<img src='https://cdn-images-1.medium.com/max/948/1*4G__SV580CxFj78o9yUXuQ.png' width='400px'/>\n",
    "#### Method 1 - manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1290, 1951) (170, 1951) (1290, 1) (170, 1)\n"
     ]
    }
   ],
   "source": [
    "def split_vals(a,n): return a[:n], a[n:]\n",
    "n_valid = 170\n",
    "n_trn = len(y)-n_valid\n",
    "X_train, X_valid = split_vals(X, n_trn)\n",
    "y_train, y_valid = split_vals(y, n_trn)\n",
    "\n",
    "print(X_train.shape, X_valid.shape, y_train.shape, y_valid.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 2 - train_test split\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1168, 1951) (292, 1951) (1168, 1) (292, 1)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X,y, test_size=0.2)\n",
    "print(X_train.shape, X_valid.shape, y_train.shape, y_valid.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://www.lintao-dashboards.com/wp-content/uploads/2016/02/ldm930.jpg' style='width:400px' />\n",
    "### Fitting a Linear Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm = LinearRegression()\n",
    "lm.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.score(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8926791758475684"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.score(X_valid,y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13557600858382415"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = lm.predict(X_valid)\n",
    "rmse = np.sqrt(metrics.mean_squared_error(y_pred, y_valid))\n",
    "rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting a regularized Model (with k Folds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RidgeCV(alphas=[0.01, 0.1, 1, 10, 100, 1000], cv=5, fit_intercept=True,\n",
       "    gcv_mode=None, normalize=False, scoring=None, store_cv_values=False)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdgCV = RidgeCV(alphas=[0.01,0.1,1,10,100,1000], cv=5)\n",
    "rdgCV.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdgCV.alpha_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91738970120679553"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdg = Ridge(alpha=10)\n",
    "rdg.fit(X_train, y_train)\n",
    "rdg.score(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11894821820610976"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = rdg.predict(X_valid)\n",
    "rmse = np.sqrt(metrics.mean_squared_error(y_pred, y_valid))\n",
    "rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting a RandomForest Regressor\n",
    "\n",
    "\n",
    "#### Parameters:\n",
    "**`df`**: The data frame you wish to process.\n",
    "\n",
    "**`y_fld`**: The name of the response variable\n",
    "\n",
    "**`skip_flds`**: A list of fields that dropped from df.\n",
    "**`do_scale`**: Standardizes each column in df.\n",
    "\n",
    "**`na_dict`**: a dictionary of na columns to add. Na columns are also added if there\n",
    "    are any missing values.\n",
    "\n",
    "**`preproc_fn`**: A function that gets applied to df.\n",
    "**`max_n_cat`**: The maximum number of categories to break into dummy values, instead\n",
    "    of integer codes.\n",
    "\n",
    "subset: Takes a random subset of size subset from df.\n",
    "\n",
    "#### Returns:\n",
    "**`[x, y, nas]`**:\n",
    "    **`x`**: x is the transformed version of df. x will not have the response variable\n",
    "        and is entirely numeric.\n",
    "\n",
    "    **`y`**: y is the response variable\n",
    "\n",
    "    nas: returns a dictionary of which nas it created, and the associated median.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/tlee010/anaconda/envs/fastai/lib/python3.6/site-packages/ipykernel_launcher.py:2: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
       "           max_features='auto', max_leaf_nodes=None,\n",
       "           min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "           min_samples_leaf=1, min_samples_split=2,\n",
       "           min_weight_fraction_leaf=0.0, n_estimators=100, n_jobs=-1,\n",
       "           oob_score=False, random_state=None, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rfr = RandomForestRegressor(n_jobs=-1, n_estimators=100)\n",
    "rfr.fit(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.98662586177542244"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rfr.score(X_valid,y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.047860116133087299"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = rfr.predict(X_valid)\n",
    "rmse = np.sqrt(metrics.mean_squared_error(y_pred, y_valid))\n",
    "rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting a Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Resplitting data for categorical prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   above_200k\n",
      "0         1.0\n",
      "1         0.0 \n",
      "\n",
      "    Intercept  Id[T.2]  Id[T.3]  Id[T.4]  Id[T.5]  Id[T.6]  Id[T.7]  Id[T.8]  \\\n",
      "0        1.0      0.0      0.0      0.0      0.0      0.0      0.0      0.0   \n",
      "1        1.0      1.0      0.0      0.0      0.0      0.0      0.0      0.0   \n",
      "\n",
      "   Id[T.9]  Id[T.10]     ...      EnclosedPorch  ThreeSsnPorch  ScreenPorch  \\\n",
      "0      0.0       0.0     ...                0.0            0.0          0.0   \n",
      "1      0.0       0.0     ...                0.0            0.0          0.0   \n",
      "\n",
      "   PoolArea  MiscVal  above_200k  LivArea_Total  LotArea_norm  LotArea_mas  \\\n",
      "0       0.0      0.0         1.0         2258.0     -0.207142     0.039258   \n",
      "1       0.0      0.0         0.0         1722.0     -0.091886     0.044600   \n",
      "\n",
      "   LotArea_qs  \n",
      "0    0.349349  \n",
      "1    0.525025  \n",
      "\n",
      "[2 rows x 1951 columns]\n",
      "(1168, 1951) (292, 1951) (1168, 1) (292, 1)\n"
     ]
    }
   ],
   "source": [
    "import patsy\n",
    "\n",
    "train_df['above_200k'] = train_df['above_200k'].astype(float)\n",
    "formula = 'above_200k ~ %s' % (' + '.join(feature_cols)) \n",
    "y, X = patsy.dmatrices(formula, train_df, return_type='dataframe')\n",
    "print(y.head(2),'\\n\\n', X.head(2))\n",
    "\n",
    "\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X,y, test_size=0.2)\n",
    "print(X_train.shape, X_valid.shape, y_train.shape, y_valid.shape)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/tlee010/anaconda/envs/fastai/lib/python3.6/site-packages/sklearn/utils/validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgm = LogisticRegression()\n",
    "lgm.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgm.score(X_valid,y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = lgm.predict(X_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[201   0]\n",
      " [  0  91]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVgAAAEmCAYAAAAnRIjxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XecVNX9//HXGxBFRVFBoyAiij2K\n3cRYEht2TWLX2CKaWOIvzRJjiSnGXrBEv9aooMbeJSbWiAXFghUVlaKCFRsKfn5/nDM4rLs7s8vM\nzs7u+8ljHsw99869Z2Z2PnPmc+85RxGBmZlVXpdaV8DMrKNygDUzqxIHWDOzKnGANTOrEgdYM7Mq\ncYA1M6sSB9gqk9RD0q2SPpJ03RzsZw9J91SybrUiaQNJL7WX40kaICkkdWurOtULSeMlbZrvHy3p\n/6pwjAsk/bHS+20P5OtgE0m7A78GVgCmAWOAv0TEQ3O4372AQ4HvR8SMOa5oOycpgEERMa7WdWmK\npPHAzyPi33l5APA6MFel3yNJlwETIuKYSu63rTR8rSqwv33y/n5Qif21d27BApJ+DZwJ/BVYDOgP\nnAdsX4HdLwW83BmCazncSqwev7btUER06huwIPAJsFMz28xNCsCT8u1MYO68bmNgAvAb4F1gMrBv\nXncC8CXwVT7G/sDxwJVF+x4ABNAtL+8DvEZqRb8O7FFU/lDR474PPA58lP//ftG6+4ATgYfzfu4B\nejfx3Ar1/31R/XcAtgJeBt4Hji7afh3gEeDDvO0woHte90B+Lp/m57tL0f6PAN4G/lkoy49ZJh9j\njby8BDAV2LiM9+5y4Df5ft987F/m5WXzftXgeP8EvgY+z3X8fdF7sDfwZj7+H8p8/2d7X3JZ5OMP\nze/9l/lYtzbxPAI4CHgF+AA4l29+XXYBjgHeyO/PFcCCDf529s/1fqCobF/grby/g4C1gWfy+zas\n6NjLAP8B3svP+yqgV9H68cCm+f7x5L/d/L5/UnSbARyf1x0JvEr623se2DGXrwh8AczMj/kwl18G\n/LnomAcA4/L7dwuwRDmvVXu81bwCtb4BQ/IfR7dmtvkTMApYFOgD/A84Ma/bOD/+T8BcpMD0GbBQ\nwz/KJpYLH4huwHzAx8Dyed3iwMr5/j7kDzKwcP7j2is/bre8vEhef1/+A18O6JGXT2riuRXqf2yu\n/wHAFOBqoCewcv5QDMzbrwmsl487AHgBOLzBB2DZRvb/d1Kg6kFRwMvbHJD3My9wN3Bqme/dfuSg\nBeyen/M1RetuLqpD8fHGk4NGg/fgoly/1YDpwIplvP+z3pfGXgMaBI8mnkcAtwG9SL+epgBDip7H\nOGAgMD9wA/DPBvW+gvS306Oo7AJgHmDz/P7dlOvflxSoN8r7WBbYLL83fUhB+szGXisa/O0WbTM4\n13n1vLwT6YuyC+lL9lNg8WZer1mvEfAjUqBfI9fpHOCBcl6r9nhzigAWAaZG8z/h9wD+FBHvRsQU\nUst0r6L1X+X1X0XEHaRv5+VbWZ+vgVUk9YiIyRExtpFttgZeiYh/RsSMiBgOvAhsW7TNpRHxckR8\nDlxL+hA05StSvvkrYATQGzgrIqbl448FVgWIiNERMSofdzzwD2CjMp7TcRExPddnNhFxEalF8ijp\nS+UPJfZXcD+wgaQuwIbAycD6ed1GeX1LnBARn0fE08DTpEALpd//SjgpIj6MiDeB//LN+7UHcHpE\nvBYRnwBHAbs2SAccHxGfNnhtT4yILyLiHlKAG57rPxF4EFgdICLGRcTI/N5MAU6n9Ps5i6Q+pOB9\naEQ8lfd5XURMioivI+Ia0nu7Tpm73AO4JCKejIjp+fl+L+fJC5p6rdodB9j006h3ifzVEqSfaAVv\n5LJZ+2gQoD8jtTZaJCI+JX3jHwRMlnS7pBXKqE+hTn2Llt9uQX3ei4iZ+X7hQ/pO0frPC4+XtJyk\n2yS9LeljUt66dzP7BpgSEV+U2OYiYBXgnPzBKikiXiV9mQ0GNiC1bCZJWp7WBdimXrNS738ltOTY\n3UjnCgreamR/Dd+/pt7PRSWNkDQxv59XUvr9JD92LuBfwNURMaKo/GeSxkj6UNKHpPe1rH3S4Pnm\nL5X3aP3fdk05wKZ84hekvGNTJpFOVhX0z2Wt8Snpp3DBd4pXRsTdEbEZqSX3IinwlKpPoU4TW1mn\nljifVK9BEbEAcDQpz9mcZi9VkTQ/Ka95MXC8pIVbUJ/7gZ+S8sAT8/LPgIVIV4K0uD6NaO79n+39\nlDTb+9mKY5Vz7BnMHjDn5Bh/y49fNb+fe1L6/Sw4h5RnnXWFhKSlSH+zh5BSVr2A54r2Waqusz1f\nSfORfmW2xd92xXX6ABsRH5Hyj+dK2kHSvJLmkrSlpJPzZsOBYyT1kdQ7b39lKw85BthQUn9JC5J+\nAgEgaTFJ2+U/qumk1tnMRvZxB7CcpN0ldZO0C7ASqQVXbT1JeeJPcuv6Fw3Wv0PKF7bEWcDoiPg5\ncDspfwiApOMl3dfMY+8nfZgfyMv3kS6Le6ioVd5QS+vY3Pv/NLCypMGS5iHlKefkWI0d+/9JWjp/\nEf2VlGeu1FUpPcknnCT1BX5XzoMkHUj6lbB7RHxdtGo+UhCdkrfbl9SCLXgH6CepexO7vhrYN7+e\nc5Oe76M5HVV3On2ABYiI00nXwB5D+sN4i/ShvSlv8mfgCdJZ2GeBJ3NZa441Ergm72s0swfFLqSr\nESaRzqBuBPyykX28B2yTt32PdCZ8m4iY2po6tdBvSSeUppFaKtc0WH88cHn+ebhzqZ1J2p50ovGg\nXPRrYA1Je+TlJUlXQzTlflKQKATYh0gtygeafERqtR2T6/jbUnWkmfc/Il4mnQT7NynX2PC66YuB\nlfKxbqLlLiFd+fAA6aqSL0hfIJVyAumE0kekL7cbynzcbqQvjkmSPsm3oyPieeA00i/Dd4DvMvv7\n9x9STv9tSd/6e42Ie4E/AteTrlJZBti1NU+sPXBHA2vXJI0BNslfKmZ1xQHWzKxKnCIwM6sSB1gz\n65QkLSnpv5JekDRW0q9y+cKSRkp6Jf+/UC6XpLMljZP0jKQ1Sh3DAdbMOqsZpK7WK5J6Jx4saSVS\nV997I2IQcG9eBtgSGJRvQ0mXLDbLg0NUmbr1CHXvWetqdGqrr9i/1lXo1N54YzxTp04t99rakrou\nsFTEjG91CPyW+HzK3RExpMn1EZNJVyoQEdMkvUDq0LA9qXs1pPEu7iONpbE9cEWkE1ejJPWStHje\nT6McYKtM3Xsy9/Ilr1ayKnr40WG1rkKntv66a1V0fzHj87I+U1+MOXcFSU8UFV0YERc2tm3uirs6\nqbv2YoWgGRGTJS2aN+vL7L3mJuQyB1gz6yAk6NK1nC2nRkTJ6J47cFxPGrToY6nJxnZjK5q9DMs5\nWDOrP+pS+lbObtJ4CtcDV0VEoZPFO5IWz+sXJ40+BqnFumTRw/tRosu8A6yZ1R+p9K3kLiRST7sX\ncm/OgltIYwOT/7+5qPxn+WqC9YCPmsu/glMEZlZ3yk4RlLI+adjJZ3OPQUiDF50EXCupMJD5Tnnd\nHaTxnseRRvHat9QBHGDNrL6IslMAzYk0315TTd1NGtk+gINbcgwHWDOrM+WlANoDB1gzqz+VSRFU\nnQOsmdUZVSRF0BYcYM2svginCMzMqkPQpT5CV33U0sysWBe3YM3MKq9Cl2m1BQdYM6szFetoUHUO\nsGZWf3ySy8ysSpwiMDOrgjIHc2kPHGDNrP44B2tmVg3uyWVmVj1OEZiZVYHqpydXfbSzzcyKVWZG\ng0skvSvpuaKyaySNybfxhYG4JQ2Q9HnRugvKqWZ9fA2YmRWrTA72MmAYcEWhICJ2mXUI6TTgo6Lt\nX42IwS05gAOsmdWX8meVbVZEPJCn627kEBKwM/CjOTmGUwRmVn/KSxH0lvRE0W1oC46wAfBORLxS\nVLa0pKck3S9pg3J24hasmdUdlXcVwdSIWKuVh9gNGF60PBnoHxHvSVoTuEnSyhHxcXM7cYA1s7oi\ngao4XKGkbsCPgTULZRExHZie74+W9CqwHPBEc/tygDWzOqNyW7CttSnwYkRMmHVEqQ/wfkTMlDQQ\nGAS8VmpHzsGaWd2RVPJWxj6GA48Ay0uaIGn/vGpXZk8PAGwIPCPpaeBfwEER8X6pY7gFa2Z1p0uX\nOW8bRsRuTZTv00jZ9cD1LT2GA6yZ1RflWx1wgDWzuqLq52ArxgHWzOpOJVIEbcEB1szqjluwZmbV\n4BysmVl1CDlFYGZWLU4RmJlVS33EVwdYM6sz8lUEZmZV4xSBmVkV1FNHg/poZ1vV9FusF3ddeBhP\nXX8Mo//1Bw7ebWMAFlpgXm47/xCevflYbjv/EHr17AHAcgMW477Lf8OHj57B4XttUsOadw733H0X\nq668PCuvsCynnHxSravTPuThCkvd2gMH2E5uxsyvOfL0G1j9J39mo5+dyoG7bMgKA7/Db/fdjPse\ne4nvbv8n7nvsJX677+YAfPDRp/zm79dx5hX/qXHNO76ZM2dy+GEHc/Otd/LUM89z3YjhvPD887Wu\nVrtQidG02oIDbCf39tSPGfNiGvbyk8+m8+Lrb7NEn15ss/GqXHnrowBceeujbPvDVQGY8sEnjH7+\nTb6aMbNmde4sHn/sMZZZZlmWHjiQ7t27s9Muu3LbrTfXulrtggOs1Z3+iy/M4OX78fhz41l0kZ68\nPTXNhvH21I/ps3DPGteu85k0aSL9+i05a7lv335MnDixhjVqP5wiaIKkT2pwzFMkvSjpGUk3SupV\ntO4oSeMkvSRpi6LyIblsnKQji8rHS+rd1s+h2ubr0Z3hp/6c3516PdM+/aLW1TEgIr5V1l5aZrVU\nTuu1zAG3L5H0rqTnisqOlzRR0ph826poXaOxojmdpQU7ElglIlYFXgaOApC0Emn08pWBIcB5krpK\n6gqcC2wJrATslrftkLp168LwUw/gmjuf4Ob/PA3Au+9N4zu9FwDgO70XYMr702pZxU6pb99+TJjw\n1qzliRMnsMQSS9SwRu1HhVIEl5E+9w2dERGD8+2OfLxGY0WpA1Q1wEq6SdJoSWOLp8yVdJqkJyXd\nm+e6QdJgSaOKWpkLSVpR0mNFjxsg6Zl8f808fe5oSXdLWrypekTEPRExIy+OAvrl+9sDIyJiekS8\nDowD1sm3cRHxWkR8CYzI2xY/tx6S7pJ0wJy+TrV2wXF78NLrb3P2ld+cuLr9/mfZc9t1Adhz23W5\n7b5nalW9Tmuttddm3LhXGP/663z55Zdcd80Itt5mu1pXq12oRIogIh4ASk77kjUVK5pV7RbsfhGx\nJrAWcJikRYD5gCcjYg3gfuC4vO0VwBG5lfkscFxEvAB0z5OMAewCXCtpLuAc4Kd5/5cAfym3TsCd\n+X5f4K2idRNyWVPlBfMDtwJXR8RFDQ8gaajyXOwx4/Myq1Ub3x88kD22WZeN1l6OUSOOZNSII9ni\nBytx6qUj+dG6K/Dszcfyo3VX4NRLRwKw2CI9GXfXiRy25w854oAtGHfXifScb54aP4uOqVu3bpxx\n1jC23XoLBn93RX6y086stPLKta5Wu1BmC7Z34XOYb0NL7Tc7JDf0LpG0UC4rFRMaVe2OBodJ2jHf\nX5I0E+PXwDW57ErgBkkLAr0i4v5cfjlwXb5/LbAzcBIpwO4CLA+sAozML2RX0rzlzZL0B2AGcFWh\nqJHNgsa/eIoTYjcDJ0fEVY1sR0RcCFwI0GXeRb+dSGtH/jfmNXqsfkij67Y66Jxvlb3z3jSWHfLH\nalfLsiFbbsWQLbcqvWFnorJz0VMjYq0W7v184ETS5/1E4DRSo6ypWNGsqgVYSRuTpr/9XkR8Juk+\noLGmTqlKXgNcJ+kGICLiFUnfBcZGxPdaUJ+9gW2ATeKbswcTSIG/oB8wKd9vqhzgYWBLSVdHY2ci\nzKxq0nCF1TnZFxHvzDqOdBFwW15sLlY0qZopggWBD3JwXQFYr+iYP833dwceioiPgA8kbZDL9yKl\nD4iIV4GZwB/5puX7EtBH0vcAJM0lqcnfTpKGAEcA20XEZ0WrbgF2lTS3pKVJLezHgMeBQZKWltSd\nlNy+pehxxwLvAee16BUxs4qQSt9at9/ZzuXsCBSuMGgqVjSrmimCu4CD8kmpl0gnlwA+BVaWNBr4\niPSTH2Bv4AJJ8wKvAfsW7esa4BRgaYCI+FLST4Gzc3qhG3AmMLaJugwD5uablMKoiDgoIsZKuhZ4\nnpQ6ODgiZgJIOgS4m5R+uCQiGu77cOASSSdHxO9b+NqY2RyoxOVqkoYDG5NytRNI54M2ljSY9Mt6\nPHAgQHOxotlj+BdudXWZd9GYe/mda12NTu2Dx4fVugqd2vrrrsXo0U9U7Dd9j8WXi6X3Lf2evvC3\nLUa3IgdbUR5Ny8zqTr30t+hQAVbSucD6DYrPiohLa1EfM6uOeunR1qECbEQcXOs6mFmVzcFJrLbW\noQKsmXV8nlXWzKyK3II1M6sS52DNzKpAomo9uSrNAdbM6k6dNGAdYM2s/jhFYGZWDU4RmJlVh3CK\nwMysStrPrLGlOMCaWd1xisDMrBrcVdbMrDpSDrY+Imx9dOg1MyvSpYtK3krJkxq+K+m5orJTJL1Y\nNLt1r1w+QNLnksbk2wVl1bPVz9DMrEbKnFW2lMuAIQ3KRgKr5NmtXwaOKlr3akQMzreDyjmAA6yZ\n1Zcy5uMqJ75GxAPA+w3K7omIGXlxFGlyw1ZzgDWzulKYVbaMFEFvSU8U3Ya28FD7AXcWLS8t6SlJ\n9xdN0NqsJk9ySVqguQdGxMfl1dHMrLK6lJcCmNraObkk/YE0ueFVuWgy0D8i3pO0JnCTpJVLxcHm\nriIYS5pZsfiZFJYD6N+aipuZzalqXkQgaW9gG2CTyLPCRsR0YHq+P1rSq8BywBPN7avJABsRS1as\nxmZmFSJB1yp1NJA0BDgC2CgiPisq7wO8HxEzJQ0EBgGvldpfWTlYSbtKOjrf75ebyGZmNVGJqwgk\nDQceAZaXNEHS/sAwoCcwssHlWBsCz0h6GvgXcFBEvN/ojouU7GggaRgwVz7AX4HPgAuAtUs+AzOz\nKqhEiiAidmuk+OImtr0euL6lxyinJ9f3I2INSU/lA70vqXtLD2RmVgkCutZJT65yAuxXkrqQTmwh\naRHg66rWysysKeV3JKi5cnKw55Kaxn0knQA8BPy9qrUyM2tGJToatIWSLdiIuELSaGDTXLRTRDzX\n3GPMzKpFVO8qgkordzStrsBXpDSBe3+ZWU11mBRB7tEwHFiC1C/3aklHNf8oM7PqKCc90F7ibzkt\n2D2BNQsX3Ur6CzAa+Fs1K2Zm1pSOdBXBGw2260YZPRjMzKqlXlIEzQ32cgYp5/oZMFbS3Xl5c9KV\nBGZmbU5AnZzjarYFW7hSYCxwe1H5qOpVx8ysBJU3Y0F70NxgL412GTMzq7W6TxEUSFoG+AuwEjBP\noTwilqtivczMGlVPKYJyrmm9DLiU9Ly2BK4FRlSxTmZmzeoilby1B+UE2Hkj4m6AiHg1Io4Bfljd\napmZNU6qnwBbzmVa05USHq9KOgiYCCxa3WqZmTWtncTPksppwf4/YH7gMGB94ADSZGBmZjVRoQG3\nL5H0rqTnisoWljRS0iv5/4VyuSSdLWmcpGckrVFOPUsG2Ih4NCKmRcSbEbFXRGwXEQ+Xs3Mzs0oT\nomuX0rcyXAYMaVB2JHBvRAwC7s3LkM4/Dcq3ocD55RyguY4GN5LHgG1MRPy4nAOYmVVUhcYaiIgH\nJA1oULw9sHG+fzlwH2mOru2BK/IkiKMk9ZK0eERMbu4YzeVgh7WiztbA6iv25+FH/VLW0kn3vlLr\nKnRqk6dNr/g+q3gd7GKFoBkRkyUVzjf1Bd4q2m5CLmtdgI2Ie+ewomZmFdeCKWN6SyqeVvvCiLhw\nDg7bUJO/8AvKHQ/WzKzdKLOjwdSIWKuFu36n8NNf0uLAu7l8ArBk0Xb9gEkl69nCg5uZ1VwXlb61\n0i3A3vn+3sDNReU/y1cTrAd8VCr/Ci1owUqaOyIqn0wxM2sBqTJTxkgaTjqh1VvSBOA44CTgWkn7\nA28CO+XN7wC2AsaRRhjct5xjlDMWwTqkucIXBPpLWg34eUQc2qJnY2ZWIRW6imC3JlZt0si2ARzc\n0mOUkyI4G9gGeC8f6GncVdbMaiQN9tJxusp2iYg3GlwWMbNK9TEzK6lr+4ifJZUTYN/KaYKQ1BU4\nFHi5utUyM2uc2lELtZRyAuwvSGmC/sA7wL9zmZlZTdRJfC0dYCPiXWDXNqiLmVlJArrVyYjb5VxF\ncBGN9FiIiKFVqZGZWQkdpgVLSgkUzAPsyOx9cs3M2s6cdSRoU+WkCK4pXpb0T2Bk1WpkZtaMFoxF\nUHOtGYtgaWCpSlfEzKxcHaYFK+kDvsnBdgHe55tBaM3M2lyHmLY7z8W1GmkeLoCvc5cxM7OaSGMR\n1LoW5Wm2mjmY3hgRM/PNwdXMaq5eusqW8z3wWLkTfJmZVVsai6BqwxVWVHNzcnWLiBnAD4ADJL0K\nfEp6fhERDrpmVgPqEFcRPAasAezQRnUxMytJdIyOBgKIiFfbqC5mZqVVKAUgaXmg+Dr/gcCxQC/g\nAGBKLj86Iu5ozTGaC7B9JP26qZURcXprDmhmNidEZWY0iIiXgMEAeaTAicCNpNkKzoiIU+f0GM0F\n2K7A/DQ+m6KZWc1U4SqBTYBXGxn7eo40F2AnR8SfKnYkM7MKqUIOdldgeNHyIZJ+BjwB/CYiPmjN\nTpu7TMstVzNrd6Q0FkGpG2kywyeKbo2OACipO7AdcF0uOh9YhpQ+mAyc1tq6NteC/dbEX2Zm7UGZ\nrb+pEbFWGdttCTwZEe8AFP6HWcO13taKKgLNBNiIeL+1OzUzq5bCpIcVtBtF6QFJi0fE5Ly4I/Bc\na3fcmtG0zMxqqlI9tSTNC2wGHFhUfLKkwaRBrsY3WNciDrBmVmdUsdG0IuIzYJEGZXtVZOc4wJpZ\nnRHlDaLSHjjAmlndaS+jZZXiAGtm9UUdZMBtM7P2xikCM7MqcgvWzKxK2suA2qU4wJpZXUkpgvqI\nsA6wZlZ36iRD4ABrZvWm/UxqWIoDrJnVFacIzMyqRfWTIqiXy8msBu65+y5WXXl5Vl5hWU45+aRa\nV6dTGHXT5Zx34NacN3QrRt14GQBjH7iT84ZuxQlbLs+kl5+tbQXbiS5SyVt74ABrjZo5cyaHH3Yw\nN996J0898zzXjRjOC88/X+tqdWjvjn+ZJ++8lgPO+hcHnX8LLz/6X96bOJ5FBwxi5z8OY6lV1q51\nFduFNFxh6Vt74ABrjXr8scdYZpllWXrgQLp3785Ou+zKbbfeXOtqdWhT3nyVfiusxlzz9KBL124s\n9d11ePF/I+nTf1l6Lzmw1tVrV1TGv/bAAdYaNWnSRPr1W3LWct++/Zg4cWINa9TxLTpgEG889wSf\nffwBX33xOeMev5+Ppkwu/cBOqF5SBD7JZY2KiG+V1Uv3xHrVp/+yrL/TAfzzqH3p3mNeFhu4Al26\n+iPaUCFFUJF9SeOBacBMYEZErCVpYeAaYABpwO2dqzHp4RyTNEBS2dMtSLpM0k+rUI9TJL0o6RlJ\nN0rqVbTuKEnjJL0kaYui8iG5bJykI4vKx0vqXek6tjd9+/ZjwoS3Zi1PnDiBJZZYooY16hzWGLIT\nB557E/ueejU9ei7IIkssVesqtUPlJAhaFIF/GBGDi+bvOhK4NyIGAffm5VbpLCmCkcAqEbEq8DJw\nFICklUjT9a4MDAHOk9RVUlfgXNJkaCsBu+VtO4211l6bceNeYfzrr/Pll19y3TUj2Hqb7WpdrQ7v\n0w/fA+CjdyfxwsP3sMrG29S4Ru1QGSe45rCFuz1web5/ObBDa3fUFr8/ukm6HFidFNx+BqwInA7M\nD0wF9imaZAwASZsAp+Y6Pg78AlgNODIifixpe2AEsCDpi+L5iGj0TEBE3FO0OAootJK3B0ZExHTg\ndUnjgHXyunER8Vquy4i87azT6JJ6ADcC10fERQ3qPhQYCrBk//7lvEbtTrdu3TjjrGFsu/UWzJw5\nk7332Y+VVl651tXq8K498RA+m/YhXbt2Y6uDj6NHzwV54eF7uPP8E/nso/e5+tihfGfgiuz510tq\nXdWaqfCkhwHcIymAf0TEhcBihXgUEZMlLdranbdFgF0e2D8iHpZ0CXAwaabG7SNiiqRdgL8A+xUe\nIGke4DJgk4h4WdIVpAA7jBSoATYgzfa4dn4ej5ZZn/1I+RWAvqSAWzAhlwG81aB83aLl+UnB/YqI\nuKLhAfKbdCHAmmuu9e1kZp0YsuVWDNlyq1pXo1PZ97Th3ypbcf3NWXH9zWtQm/arzPDaW9ITRcsX\n5s9msfUjYlIOoiMlvVihKgJtE2DfioiH8/0rgaOBVUhPBqAr0PBU6fLA6xHxcl6+HDg4Is7MOdEV\nSS3N04EN8z4eLFURSX8AZgBXFYoa2SxoPHVSHChvBk6OiKsa2c7MqqzME65Ti/KqjYqISfn/dyXd\nSIor7xSm7pa0OPBua+vZFjnYhi24acDYnFQeHBHfjYiGX8/NvXoPknKjXwH/Bn6Qbw80VwlJewPb\nAHvEN6fIJwBLFm3WD5jUTHnBw8CW8ml1s5qQSt9K70PzSepZuA9sTvpVfAuwd95sb1KDqlXaIsD2\nl/S9fH830k/yPoUySXNJapjcexEYIGnZvLwXcH++/wBwOPBIREwhTbm7AjC2qQpIGgIcAWyXp+kt\nuAXYVdLckpYGBgGPkXK+gyQtLak76UTYLUWPOxZ4Dziv3BfBzCpHZdzKsBjwkKSnSZ/72yPiLuAk\nYDNJrwCb5eVWaYsUwQvA3pL+AbwCnAPcDZwtacFchzMpCpAR8YWkfYHrJBVOcl2QVz9KemEKLdZn\ngHeLWqWNGQbMzTdpiVERcVBEjJV0Lenk1QxSGmImgKRDcj27ApdERMMAfjhwiaSTI+L3LX5VzKxV\nRGWuyc4nsVdrpPw9YJM5PgBVDrARMZ50mVNDY0i504bb71N0/16+OaFVvM3npGBZWB5aRj2WbWbd\nX0gn2RqW3wHc0Uj5gKLFfUsd28wqrI5G03I3ETOrO3USXztWgJV0LrB+g+KzIuLSWtTHzKpBddNt\nu0MF2Ig4uNZ1MLPqq5P42rGn44nEAAAQi0lEQVQCrJl1fC24SqDmHGDNrO44RWBmViV1El8dYM2s\n/tRJfHWANbM6I6cIzMyqIvXkqnUtyuMAa2Z1p07iqwOsmdUfpwjMzKqkTuKrA6yZ1Z86ia8OsGZW\nXyo1XGFb6CyzyppZR1HGbAZlzmiwpKT/SnpB0lhJv8rlx0uaKGlMvrV6Yjq3YM2s7lSo/ToD+E1E\nPJmnjhktaWRed0ZEnDqnB3CANbM6U5nhCvPU3IXpuadJeoFvZpWuCKcIzKzuVCJFMPv+NIA0g8qj\nuegQSc9IukTSQq2tpwOsmdWVciY8zPG1t6Qnim6NTi8laX7geuDwiPgYOB9YBhhMauGe1tq6OkVg\nZvWnvBbq1IhYq9ndSHORgutVEXEDQES8U7T+IuC21lbTAdbM6k6XCuRglRK5FwMvRMTpReWL5/ws\nwI7Ac609hgOsmdWdCl1FsD6wF/CspDG57GhgN0mDgQDGAwe29gAOsGZWXyo0bXdEPETjsfqOOd97\n4gBrZnWlnnpyOcCaWd2pj/DqAGtmdahOGrAOsGZWf5wiMDOrkvoIrw6wZlZnWtMVtlYcYM2s7jhF\nYGZWJfURXh1gzawO1UkD1gHWzOqLUEXGImgLHq7QzKxK3II1s7pTJw1YB1gzqzOqzHCFbcEB1szq\nStGMBe2eA6yZ1Z86ibA+yWVmdaeLVPJWiqQhkl6SNE7SkVWpZzV2amZWTWVOetj046WuwLnAlsBK\npFkMVqp0PR1gzaz+zGmEhXWAcRHxWkR8CYwAtq90NZ2DNbO6IipyFUFf4K2i5QnAunO604YcYKvs\nySdHT+0xl96odT3mUG9gaq0r0YnV++u/VCV39uSTo+/uMZd6l7HpPJKeKFq+MCIuzPcbi9Ax57Wb\nnQNslUVEn1rXYU5JeqLU/PJWPX79ZxcRQyqwmwnAkkXL/YBJFdjvbJyDNbPO6HFgkKSlJXUHdgVu\nqfRB3II1s04nImZIOgS4G+gKXBIRYyt9HAdYK8eFpTexKvLrXwURcQdwRzWPoYiK53XNzAznYM3M\nqsYB1sysShxgzcyqxAHWzKxKHGCt6lQvcyx3UIXXX1LPWtels3GAtaqLiJC0nqQta12XziSPGFV4\n/b8PHCGpm7/w2o4DrFVF7iGzV1HRgcA8tapPZyOpD3CXpMI4AL2BryJiBv7ctxm/0FYtvYGzJR2U\nl7sCnwNImquwkVtT1RERU4DXgWslLQnMT3oPiIiZtaxbZ+KOBlZRkuYl/V19KmkN4Brgd8Ag4CXg\nfmAu0shFn0bEFzWrbAckaVFgqYh4PC+fAPwQuAlYFHgWeA+YSWrR3lejqnYK7iprFSNpVeBPpGHi\nriN1Q9wBuIE0ZN31wP6kllSQAu+LtaltxyPpu6SBoydK+gS4KyKOy/f/DtwKfE0KtPMDZ9Sssp2E\nW7BWEfkM9YPAKcAHpMGLlwDOycv3AJcCZ5JasH0iYnxNKtsB5VTLucCYiLhQ0s7A2sAXwHHAYcDu\nwOYR8aGk7nkkf6si52CtUroDEyLiqjyIxuXAk8ChpF9KuwK/BX4eEZ8C9T4IeXvTBZgXKFyKdR0p\nLTAPcEhEnAk8Bfw3p3G+rkktOxkHWKuIiHgP+ETSsLz8GnAX8BqwdUQ8DewIPJ/X+6dTBUjqIqlb\nPnF1PrCppC0ieZj0q+IHkrpGxIHA7hHxWb6awKrMAdZaTdJykvaUtH8uOhGYKekIgIh4HXiUNGPn\nQhHxcETc5ysHKiPnvC8FrpS0FfAm6aTiTyRtDhARtwALAD/KD3POuw05wFqrSBpAuiJgGWCopLOB\nxYB7gSUknZ43nQpMJ+VdAbdeK0FSb+AK0ntwOynHugNpIr9HgV9KOjh3MFgGmAx+7duaT3JZq0ja\nG1grIg6VNDfpREo30sms94EjSNfC9gP+GBHX1qyyHZCklYDzImLjvPw9YF9SnvUu0nxTR5FOMN4Y\nEdfVqKqdmgOstUpuGf0dGBoRL+R5jY4F5o+Iw/M2A0jXWk6UJLeeKkvSFaRfDFdHxFc5yB5Jmv7k\n5vyeRF7n178GnCKwsklaIP8/b0T8j/RTdH1Ji+dLfk4E1pT0O4CIGB8RE/N9f7jnkKS+klaXtEou\nGgmsRjqJ1S0iHgGuAn4lqUdEfBkRX4Ff/1pxRwMri6QVgGGSxgNfSPojqSPB7kBIejAiXpZ0MbBg\nDavaIeWUwCWk6aZD0p2kHOyywDZAH+BaUvfYj/BlWO2CW7BWkqRlSL2xrgAuAz4BdoqI/5CC7ErA\nmZKOA04mdYm1CpG0BOm1PxXYGfgnsE6+1OoU0qVwQyTdn7cbHhHTa1Vf+4ZzsNasfEnVocA8EXFy\nLjsQ2DAi9sjLfYEVgcHA4xFxv3N+lSNpMLBlRPwtL/cFbgT2iYjnJRUaSusB70fEi3792wenCKxZ\neSzRS0ndXgv+A2xdtPxuPpF1rz/UlRcRYyRNgjQSWX6tx5O6wQIsGhFvA/8reozfh3bAKQJrVm4J\nTYuIhj/7++f1GwAn5ku1ZnUg8Ae8siLi3fz/V7moO9BN0vrAJfmKDWtnHGCtWU0Eyo+B5/JlQWcD\n/42I6RHhEytt523gAOAs4FwPnNM+OUVgrfEBKd+3HvCriLi7xvXpcMrMoe4H7BoRI9uiTtZybsHa\nt5QxVkBXUtfLwyLi9jaoUmc0P6TBXJpY/z/SSa6RHtuh/fJVBDZLodUkae7iy3yKW1NF2/SIiMIU\nMF2cHqgMSd2A7wD/BbaNCA/OUsccYG02koYAQ4HngEkRcUEj23SLiBl5bq0uvuZyzjVMCUj6PWl8\n19Mj4uMG2xZe/x5A/0ZOQFo74RSBzSLph8DfSBevLwrskK8OKN6ma/5w9wKG4V5bFZF/FXxP0nOS\nVgPGkubNWhK+mYK7wet/a+1qbOVwgO3kGuTvliBNr90DWJ00kMt0SQPztl0iYqakBUk9u4YXLh+y\niviU1Gr9KbAmMIQ0Qhb5de+a/+9F6hZ7oluv7ZsDbCeXW04/UJra+XPSxIR/JfUcelPSZsDekuaP\niK/zh/tm4NjwjKQVIWkVSYdFxDPAv0jTvFxD6kjwa0m/hVlBdgHSVDAnRsT9Nau0lcUB1gA2Jc2X\ndStp+LvJEfG+pE1I11k+GhGf5BMwp5LGd32odtXtcOYFfirpNNIX3DrAfMA+wD+AB4q23RM4KiIe\nbOtKWsv5JFcn1MgJle8DewOHkz7YpwF9SV/ApxVfiiWpd0RMbeMqdyhFV2KsRroca3JEvCbpJOAz\n0vXFb0TEL5Rnfy16jK/YqCMOsJ2UpHWB5SLin3l5BDAlIg7NywsARMTHOU8rf7ArR9J2wPGk4QVn\nAM9HxAmSVid1IPgFsFRhPF2rTw6wnZSkrUk/928mDZw9ijTty7CIeK6WdeuIJPUEukXEB0rTZl9H\nymOPzgNoHw48FhEX5u2XjjRppNUx52A7icLVApJWy8PfPUsaXvBpYCdS7m9d0s9Tq6AcXP8O7Chp\nYeBLoBcp9wowjvQlt0zRw8bnx7qXVh1zgO0kcv5ua9KA2b8GLgYOj4jhEbE7MJzU/XVK7WrZMUXE\nNOAxYAPS1RkzgMuBoyWtHBFfAO8Ay0iatzhH7lHJ6psHe+kkcivq98ChEfGQpGWBiyV9EhHnRsQ5\nkv4vIj73YM2VUzgpFRGXSZoObJtbpf8mTWV+u9I0O/sAB0fEZzWsrlWYA2wH1UiQ/Jo01csEgIgY\nJ+kCoDCBHoWxBRxcKyO/B1/nL7PJpGtbp5Ku2PiaNMfWWFJvuL0iTSRpHYgDbAdVNDhLf2BqRHwq\naRRwjaSN8s/SAAYpTe/8lQNrZeW0zJak2XbvIPXO+gmwCKmX1tzA9Q3HGrCOwwG2g1GaoHDbiDhT\n0uakAbFH577svyfNOvBU/ln6c1Ie9sva1bjjUpoJ9i+krq8/JgXWuSJihKQAdgTuJg1gbh2QL9Pq\nYPK4AeOA03PRjaS0wC7ADsAW+Qbpuld3t6ygwngB+f4ypNf6deAEYLeIeFXS+hHxsKTFIuKdWtbX\nqssBtgMpGsZuIKnL6wcRsUbhRIukc4AnIuLyGle1w5HUM18tgKQfAAOB6aQRx6YCa+fuxhuSfkkM\njYhJNauwtQlfptVB5BMqMyQtD0wDNgQGSDqqqAfWB8BSNatkB5U7Dtwu6SeSlgPOAzYD1gLeJPXU\n+omknUljO1zk4No5OAfbQeQTKtsCfyZdpP4y6Wz1cEmDSMPbDQH+WLNKdlAR8ZmkM4AjSV9uQyNi\nVE4RTAC+R8q3jgOOjog7fSlc5+AA20FIWg84ltRy2gy4kDRwyK6kIfAGAPtFxHP+cFdeRNwo6RPS\na70pqevxm6T8a7+I+F1hW7/+nYcDbMcxAfglqfvrr4DVSEPdLQPsBnxUGGPAH+7qiDQB4T7AKZJe\njYjhkj4ENpK0GPBuZLWtqbUVn+TqYCT9hfRBPkvSXqRg++M8eLZbTm0gp2ouB+4DPgRuiIjbalop\nqwm3YDueZ4ED8+DY25K6xr4Jbrm2lYi4VdLPScMR7h8Rj/vLrXNyC7aDyeO47ghsB1wcEXfUuEqd\nlqSFI+L9WtfDascBtoMquibWLSezGvF1sB3XTHBawKyW3II1M6sSt2DNzKrEAdbMrEocYM3MqsQB\n1ipK0kxJYyQ9J+m6PBBKa/e1saTb8v3tJB3ZzLa9JP2yFcc4XtJvyy1vsM1lkn7agmMNkOQZezsR\nB1irtM8jYnBErEKaPfWg4pVKWvx3FxG3RMRJzWzSi9RV2KzdcIC1anoQWDa33F6QdB7wJLCkpM0l\nPSLpydzSnR9A0hBJL0p6iDQLALl8H0nD8v3FJN0o6el8+z5wEmlW1jGSTsnb/U7S45KekXRC0b7+\nIOklSf8Gli/1JCQdkPfztKTrG7TKN5X0oKSXJW2Tt+8q6ZSiYx84py+k1ScHWKuK3FV3S1LXXUiB\n7IqIWB34FDgG2DQi1gCeAH4taR7gIlIX3w2A7zSx+7OB+yNiNWAN0sSBRwKv5tbz7/J0OYOAdUgD\n4KwpaUNJa5JGGFudFMDXLuPp3BARa+fjvQDsX7RuALARsDVwQX4O+5MG11k77/8ASUuXcRzrYDwW\ngVVaD0lj8v0HgYuBJYA3ImJULl8PWAl4WBJAd+ARYAXg9Yh4BUDSlcDQRo7xI+BnAHl6lo8kLdRg\nm83z7am8PD8p4PYEbixMjy3pljKe0yqS/kxKQ8xPmker4No8oPkrkl7Lz2FzYNWi/OyC+dgvl3Es\n60AcYK3SPo+IwcUFOYh+WlwEjIyI3RpsN5g0020lCPhbRPyjwTEOb8UxLgN2iIin83CEGxeta7iv\nyMc+NCKKAzGSBrTwuFbnnCKwWhgFrC9pWUhTruSpVl4Els4zAUAax7Yx9wK/yI/tmge4mUZqnRbc\nDexXlNvtK2lR4AFgR0k9JPUkpSNK6QlMljQXsEeDdTtJ6pLrPBB4KR/7F3l7JC0nab4yjmMdjFuw\n1uYiYkpuCQ6XNHcuPiYiXpY0lDS/1VTgIWCVRnbxK+BCSfuTxlz4RUQ8IunhfBnUnTkPuyLwSG5B\nfwLsGRFPSroGGAO8QUpjlPJH4NG8/bPMHshfAu4HFgMOiogvJP0fKTf7pNLBp5Bm9LVOxmMRmJlV\niVMEZmZV4gBrZlYlDrBmZlXiAGtmViUOsGZmVeIAa2ZWJQ6wZmZV8v8BLQ8XvlONk0IAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a1d77c860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cnf_matrix = metrics.confusion_matrix(y_valid, y_pred)\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "# Plot non-normalized confusion matrix\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cnf_matrix, classes=['above_200k', 'below_200k'],\n",
    "                      title='Confusion matrix, without normalization')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}