multiple-linear-regression-and-feature-engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Multiple Regression and Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fire up graphlab create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import graphlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load in house sales data\n",
    "\n",
    "Dataset is from house sales in King County, the region where the city of Seattle, WA is located."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] graphlab.cython.cy_server: GraphLab Create v2.1 started. Logging: /tmp/graphlab_server_1529352599.log\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This non-commercial license of GraphLab Create for academic use is assigned to nanlee_89@yahoo.com and will expire on December 07, 2018.\n"
     ]
    }
   ],
   "source": [
    "sales = graphlab.SFrame('kc_house_data.gl/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split data into training and testing.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data,test_data = sales.random_split(.8,seed=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn a multiple regression model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "example_features = ['sqft_living', 'bedrooms', 'bathrooms']\n",
    "example_model = graphlab.linear_regression.create(train_data, target = 'price', \n",
    "                                                  features = example_features, \n",
    "                                                  verbose = False,\n",
    "                                                  validation_set = None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the regression weights (coefficients)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+-------+----------------+---------------+\n",
      "|     name    | index |     value      |     stderr    |\n",
      "+-------------+-------+----------------+---------------+\n",
      "| (intercept) |  None | 87910.0724924  |  7873.3381434 |\n",
      "| sqft_living |  None | 315.403440552  | 3.45570032585 |\n",
      "|   bedrooms  |  None | -65080.2155528 | 2717.45685442 |\n",
      "|  bathrooms  |  None | 6944.02019265  | 3923.11493144 |\n",
      "+-------------+-------+----------------+---------------+\n",
      "[4 rows x 4 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "example_weight_summary = example_model.get(\"coefficients\")\n",
    "print example_weight_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Predictions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "271789.505878\n"
     ]
    }
   ],
   "source": [
    "example_predictions = example_model.predict(train_data)\n",
    "print example_predictions[0] # should be 271789.505878"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compute RSS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_residual_sum_of_squares(model, data, outcome):\n",
    "    # First get the predictions\n",
    "    predictions = model.predict(data)\n",
    "\n",
    "    # Then compute the residuals/errors\n",
    "    residuals = outcome - predictions\n",
    "\n",
    "    # Then square and add them up\n",
    "    squared_residuals = residuals*residuals\n",
    "    \n",
    "    RSS = squared_residuals.sum()\n",
    "\n",
    "    return(RSS)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create some new features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, squarefeet, and # of bathrooms) but we can also consider transformations of existing features e.g. the log of the squarefeet or even \"interaction\" features such as the product of bedrooms and bathrooms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from math import log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next create the following 4 new features as column in both TEST and TRAIN data:\n",
    "* bedrooms_squared = bedrooms\\*bedrooms\n",
    "* bed_bath_rooms = bedrooms\\*bathrooms\n",
    "* log_sqft_living = log(sqft_living)\n",
    "* lat_plus_long = lat + long \n",
    "As an example here's the first one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data['bedrooms_squared'] = train_data['bedrooms'].apply(lambda x: x**2)\n",
    "test_data['bedrooms_squared'] = test_data['bedrooms'].apply(lambda x: x**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create the remaining 3 features in both TEST and TRAIN data\n",
    "\n",
    "train_data['bed_bath_rooms'] = train_data['bedrooms']*train_data['bathrooms']\n",
    "test_data['bed_bath_rooms'] = test_data['bedrooms']*test_data['bathrooms']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data['log_sqft_living'] = train_data['sqft_living'].apply(lambda x: log(x))\n",
    "test_data['log_sqft_living'] = test_data['sqft_living'].apply(lambda x: log(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data['lat_plus_long'] = train_data['lat'] + train_data['long']\n",
    "test_data['lat_plus_long'] = test_data['lat']+ test_data['long']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this feature will mostly affect houses with many bedrooms.\n",
    "* bedrooms times bathrooms gives what's called an \"interaction\" feature. It is large when *both* of them are large.\n",
    "* Taking the log of squarefeet has the effect of bringing large values closer together and spreading out small values.\n",
    "* Adding latitude to longitude is totally non-sensical but we will do it anyway (you'll see why)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mean (arithmetic average) value of  4 new features on TEST data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Then mean value of bedroom_squared on TEST data:  12.4466777016\n",
      "Then mean value of bed_bath_rooms on TEST data:  7.50390163159\n",
      "Then mean value of log_sqft_living on TEST data:  7.55027467965\n",
      "Then mean value of lat_plus_long on TEST data:  -74.6533349722\n"
     ]
    }
   ],
   "source": [
    "print \"Then mean value of bedroom_squared on TEST data: \", test_data['bedrooms_squared'].mean()\n",
    "print \"Then mean value of bed_bath_rooms on TEST data: \", test_data['bed_bath_rooms'].mean()\n",
    "print \"Then mean value of log_sqft_living on TEST data: \", test_data['log_sqft_living'].mean()\n",
    "print \"Then mean value of lat_plus_long on TEST data: \", test_data['lat_plus_long'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Multiple Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will learn the weights for three (nested) models for predicting house prices. The first model will have the fewest features the second model will add one more feature and the third will add a few more:\n",
    "* Model 1: squarefeet, # bedrooms, # bathrooms, latitude & longitude\n",
    "* Model 2: add bedrooms\\*bathrooms\n",
    "* Model 3: Add log squarefeet, bedrooms squared, and the (nonsensical) latitude + longitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_1_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat', 'long']\n",
    "model_2_features = model_1_features + ['bed_bath_rooms']\n",
    "model_3_features = model_2_features + ['bedrooms_squared', 'log_sqft_living', 'lat_plus_long']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have the features, learn the weights for the three different models for predicting target = 'price' using graphlab.linear_regression.create() and look at the value of the weights/coefficients:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learn the three models: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_1 = graphlab.linear_regression.create(train_data, target = 'price', features = model_1_features,\n",
    "                                            verbose=False, validation_set = None)\n",
    "model_2 = graphlab.linear_regression.create(train_data, target = 'price', features = model_2_features, \n",
    "                                            verbose=False, validation_set = None)\n",
    "model_3 = graphlab.linear_regression.create(train_data, target = 'price', features = model_3_features, \n",
    "                                            verbose=False, validation_set = None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examine/extract each model's coefficients:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model 1 coefficients \n",
      "+-------------+-------+----------------+---------------+\n",
      "|     name    | index |     value      |     stderr    |\n",
      "+-------------+-------+----------------+---------------+\n",
      "| (intercept) |  None | -56140675.7444 | 1649985.42028 |\n",
      "| sqft_living |  None | 310.263325778  | 3.18882960408 |\n",
      "|   bedrooms  |  None | -59577.1160682 | 2487.27977322 |\n",
      "|  bathrooms  |  None | 13811.8405418  | 3593.54213297 |\n",
      "|     lat     |  None | 629865.789485  | 13120.7100323 |\n",
      "|     long    |  None | -214790.285186 | 13284.2851607 |\n",
      "+-------------+-------+----------------+---------------+\n",
      "[6 rows x 4 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_1_weight_summary = model_1.get(\"coefficients\")\n",
    "print \"Model 1 coefficients \\n\", model_1_weight_summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model 2 coefficients \n",
      "+----------------+-------+----------------+---------------+\n",
      "|      name      | index |     value      |     stderr    |\n",
      "+----------------+-------+----------------+---------------+\n",
      "|  (intercept)   |  None | -54410676.1152 | 1650405.16541 |\n",
      "|  sqft_living   |  None | 304.449298057  | 3.20217535637 |\n",
      "|    bedrooms    |  None | -116366.043231 | 4805.54966546 |\n",
      "|   bathrooms    |  None | -77972.3305135 | 7565.05991091 |\n",
      "|      lat       |  None | 625433.834953  | 13058.3530972 |\n",
      "|      long      |  None | -203958.60296  | 13268.1283711 |\n",
      "| bed_bath_rooms |  None | 26961.6249092  | 1956.36561555 |\n",
      "+----------------+-------+----------------+---------------+\n",
      "[7 rows x 4 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_2_weight_summary = model_2.get(\"coefficients\")\n",
    "print \"Model 2 coefficients \\n\", model_2_weight_summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model 3 coefficients \n",
      "+------------------+-------+----------------+---------------+\n",
      "|       name       | index |     value      |     stderr    |\n",
      "+------------------+-------+----------------+---------------+\n",
      "|   (intercept)    |  None | -52974974.0602 |  1615194.9439 |\n",
      "|   sqft_living    |  None | 529.196420564  | 7.69913498511 |\n",
      "|     bedrooms     |  None | 28948.5277313  | 9395.72889106 |\n",
      "|    bathrooms     |  None |  65661.207231  | 10795.3380703 |\n",
      "|       lat        |  None | 704762.148408  | 1292011141.66 |\n",
      "|       long       |  None | -137780.01994  | 1292011141.57 |\n",
      "|  bed_bath_rooms  |  None | -8478.36410518 | 2858.95391257 |\n",
      "| bedrooms_squared |  None | -6072.38466067 | 1494.97042777 |\n",
      "| log_sqft_living  |  None | -563467.784269 | 17567.8230814 |\n",
      "|  lat_plus_long   |  None | -83217.1979248 | 1292011141.58 |\n",
      "+------------------+-------+----------------+---------------+\n",
      "[10 rows x 4 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_3_weight_summary = model_3.get(\"coefficients\")\n",
    "print \"Model 3 coefficients \\n\", model_3_weight_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing multiple models\n",
    "\n",
    "Now that we've learned three models and extracted the model weights we want to evaluate which model is best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the RSS on TRAINING data for each of the three models and record the values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RSS of model 1: 9.71328233544e+14\n",
      "RSS of model 2: 9.61592067856e+14\n",
      "RSS of model 3: 9.05276314555e+14\n"
     ]
    }
   ],
   "source": [
    "rss_model_1_train = get_residual_sum_of_squares(model_1, train_data, train_data['price'])\n",
    "print \"RSS of model 1:\", rss_model_1_train\n",
    "\n",
    "rss_model_2_train = get_residual_sum_of_squares(model_2, train_data, train_data['price'])\n",
    "print \"RSS of model 2:\", rss_model_2_train\n",
    "\n",
    "rss_model_3_train = get_residual_sum_of_squares(model_3, train_data, train_data['price'])\n",
    "print \"RSS of model 3:\", rss_model_3_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the RSS on on TEST data for each of the three models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RSS of model 1: 2.26568089093e+14\n",
      "RSS of model 2: 2.24368799994e+14\n",
      "RSS of model 3: 2.51829318952e+14\n"
     ]
    }
   ],
   "source": [
    "# Compute the RSS on TESTING data for each of the three models and record the values:\n",
    "rss_model_1_test = get_residual_sum_of_squares(model_1, test_data, test_data['price'])\n",
    "print \"RSS of model 1:\", rss_model_1_test\n",
    "\n",
    "rss_model_2_test = get_residual_sum_of_squares(model_2, test_data, test_data['price'])\n",
    "print \"RSS of model 2:\", rss_model_2_test\n",
    "\n",
    "rss_model_3_test = get_residual_sum_of_squares(model_3, test_data, test_data['price'])\n",
    "print \"RSS of model 3:\", rss_model_3_test\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}