linear-classifier-feature-engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting sentiment from product reviews\n",
    "\n",
    "In this notebook we will use product review data from Amazon.com to predict whether the sentiments about a product (from its reviews) are positive or negative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    \n",
    "## Fire up GraphLab Create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "import graphlab\n",
    "import math\n",
    "import string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data preparation\n",
    "\n",
    "We will use a dataset consisting of baby product reviews on Amazon.com."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] graphlab.cython.cy_server: GraphLab Create v2.1 started. Logging: C:\\Users\\Nan\\AppData\\Local\\Temp\\graphlab_server_1530239354.log.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This non-commercial license of GraphLab Create for academic use is assigned to nanlee_89@yahoo.com and will expire on December 07, 2018.\n"
     ]
    }
   ],
   "source": [
    "products = graphlab.SFrame('amazon_baby.gl/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a preview of what the dataset looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">review</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rating</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Planetwise Flannel Wipes</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">These flannel wipes are<br>OK, but in my opinion ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Planetwise Wipe Pouch</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">it came early and was not<br>disappointed. i love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Annas Dream Full Quilt<br>with 2 Shams ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Very soft and comfortable<br>and warmer than it ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Stop Pacifier Sucking<br>without tears with ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">This is a product well<br>worth the purchase.  I ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Stop Pacifier Sucking<br>without tears with ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">All of my kids have cried<br>non-stop when I tried to ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Stop Pacifier Sucking<br>without tears with ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">When the Binky Fairy came<br>to our house, we didn't ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A Tale of Baby's Days<br>with Peter Rabbit ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Lovely book, it's bound<br>tightly so you may no ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Baby Tracker&amp;reg; - Daily<br>Childcare Journal, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Perfect for new parents.<br>We were able to keep ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Baby Tracker&amp;reg; - Daily<br>Childcare Journal, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A friend of mine pinned<br>this product on Pinte ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Baby Tracker&amp;reg; - Daily<br>Childcare Journal, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">This has been an easy way<br>for my nanny to record ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4.0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[183531 rows x 3 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tname\tstr\n",
       "\treview\tstr\n",
       "\trating\tfloat\n",
       "\n",
       "Rows: 183531\n",
       "\n",
       "Data:\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "|              name             |             review            | rating |\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "|    Planetwise Flannel Wipes   | These flannel wipes are OK... |  3.0   |\n",
       "|     Planetwise Wipe Pouch     | it came early and was not ... |  5.0   |\n",
       "| Annas Dream Full Quilt wit... | Very soft and comfortable ... |  5.0   |\n",
       "| Stop Pacifier Sucking with... | This is a product well wor... |  5.0   |\n",
       "| Stop Pacifier Sucking with... | All of my kids have cried ... |  5.0   |\n",
       "| Stop Pacifier Sucking with... | When the Binky Fairy came ... |  5.0   |\n",
       "| A Tale of Baby's Days with... | Lovely book, it's bound ti... |  4.0   |\n",
       "| Baby Tracker&reg; - Daily ... | Perfect for new parents. W... |  5.0   |\n",
       "| Baby Tracker&reg; - Daily ... | A friend of mine pinned th... |  5.0   |\n",
       "| Baby Tracker&reg; - Daily ... | This has been an easy way ... |  4.0   |\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "[183531 rows x 3 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the word count vector for each review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def remove_punctuation(text):\n",
    "    import string\n",
    "    return text.translate(None, string.punctuation) \n",
    "\n",
    "review_without_punctuation = products['review'].apply(remove_punctuation)\n",
    "products['word_count'] = graphlab.text_analytics.count_words(review_without_punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract sentiments\n",
    "\n",
    "We will **ignore** all reviews with *rating = 3*, since they tend to have a neutral sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "166752"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "products = products[products['rating'] != 3]\n",
    "len(products)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will assign reviews with a rating of 4 or higher to be *positive* reviews, while the ones with rating of 2 or lower are *negative*. For the sentiment column, we use +1 for the positive class label and -1 for the negative class label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">review</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">word_count</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sentiment</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Planetwise Wipe Pouch</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">it came early and was not<br>disappointed. i love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 3L, 'love': 1L,<br>'it': 3L, 'highly': 1L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Annas Dream Full Quilt<br>with 2 Shams ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Very soft and comfortable<br>and warmer than it ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 2L, 'quilt': 1L,<br>'it': 1L, 'comfortable': ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Stop Pacifier Sucking<br>without tears with ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">This is a product well<br>worth the purchase.  I ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 3L, 'ingenious':<br>1L, 'love': 2L, 'is': ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Stop Pacifier Sucking<br>without tears with ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">All of my kids have cried<br>non-stop when I tried to ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 2L, 'all': 2L,<br>'help': 1L, 'cried': 1L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Stop Pacifier Sucking<br>without tears with ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">When the Binky Fairy came<br>to our house, we didn't ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 2L, 'cute': 1L,<br>'help': 2L, 'habit': 1L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A Tale of Baby's Days<br>with Peter Rabbit ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Lovely book, it's bound<br>tightly so you may no ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'shop': 1L, 'be': 1L,<br>'is': 1L, 'bound': 1L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Baby Tracker&amp;reg; - Daily<br>Childcare Journal, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Perfect for new parents.<br>We were able to keep ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 2L, 'all': 1L,<br>'right': 1L, 'able': 1L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Baby Tracker&amp;reg; - Daily<br>Childcare Journal, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A friend of mine pinned<br>this product on Pinte ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 1L, 'fantastic':<br>1L, 'help': 1L, 'give': ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Baby Tracker&amp;reg; - Daily<br>Childcare Journal, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">This has been an easy way<br>for my nanny to record ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'all': 1L, 'standarad':<br>1L, 'another': 1L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Baby Tracker&amp;reg; - Daily<br>Childcare Journal, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">I love this journal and<br>our nanny uses it ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'all': 2L, 'nannys': 1L,<br>'just': 1L, 'sleep': 2L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[166752 rows x 5 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tname\tstr\n",
       "\treview\tstr\n",
       "\trating\tfloat\n",
       "\tword_count\tdict\n",
       "\tsentiment\tint\n",
       "\n",
       "Rows: 166752\n",
       "\n",
       "Data:\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "|              name             |             review            | rating |\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "|     Planetwise Wipe Pouch     | it came early and was not ... |  5.0   |\n",
       "| Annas Dream Full Quilt wit... | Very soft and comfortable ... |  5.0   |\n",
       "| Stop Pacifier Sucking with... | This is a product well wor... |  5.0   |\n",
       "| Stop Pacifier Sucking with... | All of my kids have cried ... |  5.0   |\n",
       "| Stop Pacifier Sucking with... | When the Binky Fairy came ... |  5.0   |\n",
       "| A Tale of Baby's Days with... | Lovely book, it's bound ti... |  4.0   |\n",
       "| Baby Tracker&reg; - Daily ... | Perfect for new parents. W... |  5.0   |\n",
       "| Baby Tracker&reg; - Daily ... | A friend of mine pinned th... |  5.0   |\n",
       "| Baby Tracker&reg; - Daily ... | This has been an easy way ... |  4.0   |\n",
       "| Baby Tracker&reg; - Daily ... | I love this journal and ou... |  4.0   |\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "+-------------------------------+-----------+\n",
       "|           word_count          | sentiment |\n",
       "+-------------------------------+-----------+\n",
       "| {'and': 3L, 'love': 1L, 'i... |     1     |\n",
       "| {'and': 2L, 'quilt': 1L, '... |     1     |\n",
       "| {'and': 3L, 'ingenious': 1... |     1     |\n",
       "| {'and': 2L, 'all': 2L, 'he... |     1     |\n",
       "| {'and': 2L, 'cute': 1L, 'h... |     1     |\n",
       "| {'shop': 1L, 'be': 1L, 'is... |     1     |\n",
       "| {'and': 2L, 'all': 1L, 'ri... |     1     |\n",
       "| {'and': 1L, 'fantastic': 1... |     1     |\n",
       "| {'all': 1L, 'standarad': 1... |     1     |\n",
       "| {'all': 2L, 'nannys': 1L, ... |     1     |\n",
       "+-------------------------------+-----------+\n",
       "[166752 rows x 5 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "products['sentiment'] = products['rating'].apply(lambda rating : +1 if rating > 3 else -1)\n",
    "products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can see that the dataset contains an extra column called **sentiment** which is either positive (+1) or negative (-1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split data into training and test sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's perform a train/test split with 80% of the data in the training set and 20% of the data in the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "133416\n",
      "33336\n"
     ]
    }
   ],
   "source": [
    "train_data, test_data = products.random_split(.8, seed=1)\n",
    "print len(train_data)\n",
    "print len(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train a sentiment classifier with logistic regression\n",
    "\n",
    "We will use logistic regression to create a sentiment classifier on the training data. This model will use the column **word_count** as a feature and the column **sentiment** as the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sentiment_model = graphlab.logistic_classifier.create(train_data,\n",
    "                                                      target = 'sentiment',\n",
    "                                                      features=['word_count'],\n",
    "                                                      verbose = False,\n",
    "                                                      validation_set=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Class                          : LogisticClassifier\n",
       "\n",
       "Schema\n",
       "------\n",
       "Number of coefficients         : 121713\n",
       "Number of examples             : 133416\n",
       "Number of classes              : 2\n",
       "Number of feature columns      : 1\n",
       "Number of unpacked features    : 121712\n",
       "\n",
       "Hyperparameters\n",
       "---------------\n",
       "L1 penalty                     : 0.0\n",
       "L2 penalty                     : 0.01\n",
       "\n",
       "Training Summary\n",
       "----------------\n",
       "Solver                         : lbfgs\n",
       "Solver iterations              : 6\n",
       "Solver status                  : TERMINATED: Terminated due to numerical difficulties.\n",
       "Training time (sec)            : 4.5231\n",
       "\n",
       "Settings\n",
       "--------\n",
       "Log-likelihood                 : inf\n",
       "\n",
       "Highest Positive Coefficients\n",
       "-----------------------------\n",
       "word_count[mobileupdate]       : 41.9847\n",
       "word_count[placeid]            : 41.7354\n",
       "word_count[labelbox]           : 41.151\n",
       "word_count[httpwwwamazoncomreviewrhgg6qp7tdnhbrefcmcrprcmtieutf8asinb00318cla0nodeid] : 40.0454\n",
       "word_count[knobskeeping]       : 36.2091\n",
       "\n",
       "Lowest Negative Coefficients\n",
       "----------------------------\n",
       "word_count[probelm]            : -44.9283\n",
       "word_count[impulsejeep]        : -43.081\n",
       "word_count[infantsyoung]       : -39.5945\n",
       "word_count[cutereditafter]     : -35.6875\n",
       "word_count[avacado]            : -35.0542"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentiment_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have fitted the model, we can extract the weights (coefficients) as an SFrame as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['name', 'index', 'class', 'value', 'stderr']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights = sentiment_model.coefficients\n",
    "weights.column_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a total of `121713` coefficients in the model. Positive weights $w_j$ correspond to weights that cause positive sentiment, while negative weights correspond to negative sentiment. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of positive weights: 68419 \n",
      "Number of negative weights: 53294 \n"
     ]
    }
   ],
   "source": [
    "num_positive_weights = len(weights[weights['value']>=0])\n",
    "num_negative_weights = len(weights[weights['value']<0])\n",
    "\n",
    "print \"Number of positive weights: %s \" % num_positive_weights\n",
    "print \"Number of negative weights: %s \" % num_negative_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions with logistic regression\n",
    "\n",
    "Now that a model is trained, we can make predictions on the **test data**. In this section, we will explore this in the context of 3 examples in the test dataset.  We refer to this set of 3 examples as the **sample_test_data**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5.0, 2.0, 1.0]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">review</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">word_count</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sentiment</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Our Baby Girl Memory Book</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Absolutely love it and<br>all of the Scripture in ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 2L, 'all': 1L,<br>'love': 1L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Wall Decor Removable<br>Decal Sticker - Colorful ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Would not purchase again<br>or recommend. The decals ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'and': 1L, 'wall': 1L,<br>'them': 1L, 'decals': ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">New Style Trailing Cherry<br>Blossom Tree Decal ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Was so excited to get<br>this product for my baby ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">{'all': 1L, 'money': 1L,<br>'into': 1L, 'it': 3L, ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[3 rows x 5 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tname\tstr\n",
       "\treview\tstr\n",
       "\trating\tfloat\n",
       "\tword_count\tdict\n",
       "\tsentiment\tint\n",
       "\n",
       "Rows: 3\n",
       "\n",
       "Data:\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "|              name             |             review            | rating |\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "|   Our Baby Girl Memory Book   | Absolutely love it and all... |  5.0   |\n",
       "| Wall Decor Removable Decal... | Would not purchase again o... |  2.0   |\n",
       "| New Style Trailing Cherry ... | Was so excited to get this... |  1.0   |\n",
       "+-------------------------------+-------------------------------+--------+\n",
       "+-------------------------------+-----------+\n",
       "|           word_count          | sentiment |\n",
       "+-------------------------------+-----------+\n",
       "| {'and': 2L, 'all': 1L, 'lo... |     1     |\n",
       "| {'and': 1L, 'wall': 1L, 't... |     -1    |\n",
       "| {'all': 1L, 'money': 1L, '... |     -1    |\n",
       "+-------------------------------+-----------+\n",
       "[3 rows x 5 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_test_data = test_data[10:13]\n",
    "print sample_test_data['rating']\n",
    "sample_test_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's dig deeper into the first row of the **sample_test_data**. Here's the full review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Absolutely love it and all of the Scripture in it.  I purchased the Baby Boy version for my grandson when he was born and my daughter-in-law was thrilled to receive the same book again.'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_test_data[0]['review']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That review seems pretty positive.\n",
    "\n",
    "Now, let's see what the next row of the **sample_test_data** looks like. As we could guess from the sentiment (-1), the review is quite negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Would not purchase again or recommend. The decals were thick almost plastic like and were coming off the wall as I was applying them! The would NOT stick! Literally stayed stuck for about 5 minutes then started peeling off.'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_test_data[1]['review']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now make a **class** prediction for the **sample_test_data**. The `sentiment_model` should predict **+1** if the sentiment is positive and **-1** if the sentiment is negative. The **score** (sometimes called **margin**) for the logistic regression model  is defined as:\n",
    "\n",
    "$$\n",
    "\\mbox{score}_i = \\mathbf{w}^T h(\\mathbf{x}_i)\n",
    "$$ \n",
    "\n",
    "where $h(\\mathbf{x}_i)$ represents the features for example $i$.  We will write some code to obtain the **scores** using GraphLab Create. For each row, the **score** (or margin) is a number in the range **[-inf, inf]**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6.734619727059137, -5.734130996760246, -14.668460404468481]\n"
     ]
    }
   ],
   "source": [
    "scores = sentiment_model.predict(sample_test_data, output_type='margin')\n",
    "print scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predicting sentiment\n",
    "\n",
    "These scores can be used to make class predictions as follows:\n",
    "\n",
    "$$\n",
    "\\hat{y} = \n",
    "\\left\\{\n",
    "\\begin{array}{ll}\n",
    "      +1 & \\mathbf{w}^T h(\\mathbf{x}_i) > 0 \\\\\n",
    "      -1 & \\mathbf{w}^T h(\\mathbf{x}_i) \\leq 0 \\\\\n",
    "\\end{array} \n",
    "\\right.\n",
    "$$\n",
    "\n",
    "Using scores, write code to calculate $\\hat{y}$, the class predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sample_test_data['predictions'] = scores.apply(lambda x: 1 if x>0 else -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code verifies that the class predictions obtained by your calculations are the same as that obtained from GraphLab Create."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class predictions according to GraphLab Create:\n",
      "[1L, -1L, -1L]\n"
     ]
    }
   ],
   "source": [
    "print \"Class predictions according to GraphLab Create:\" \n",
    "print sentiment_model.predict(sample_test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Probability predictions\n",
    "\n",
    "The probability predictions could be computed from the scores using:\n",
    "$$\n",
    "P(y_i = +1 | \\mathbf{x}_i,\\mathbf{w}) = \\frac{1}{1 + \\exp(-\\mathbf{w}^T h(\\mathbf{x}_i))}.\n",
    "$$\n",
    "\n",
    "For each row, the probabilities should be a number in the range **[0, 1]**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sample_test_data['probability']=scores.apply(lambda x: 1/(1+math.exp(-x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checkpoint**: Make sure probability predictions match the ones obtained from GraphLab Create."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class predictions according to GraphLab Create:\n",
      "[0.9988123848377196, 0.003223268181800717, 4.261557996655577e-07]\n"
     ]
    }
   ],
   "source": [
    "print \"Class predictions according to GraphLab Create:\" \n",
    "print sentiment_model.predict(sample_test_data, output_type='probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find the most positive (and negative) review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now turn to examining the full test dataset, **test_data**, and use GraphLab Create to form predictions on all of the test data points for faster performance.\n",
    "\n",
    "Using the `sentiment_model`, find the 20 reviews in the entire **test_data** with the **highest probability** of being classified as a **positive review**. We refer to these as the \"most positive reviews.\"\n",
    "\n",
    "To calculate these top-20 reviews, use the following steps:\n",
    "1.  Make probability predictions on **test_data** using the `sentiment_model`. \n",
    "2.  Sort the data according to those predictions and pick the top 20. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_data['probability']= sentiment_model.predict(test_data, output_type='probability')\n",
    "most_positive_reviews = test_data.topk('probability', k=20)['name']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us find the \"most negative reviews.\" Use the prediction probabilities to find the  20 reviews in the **test_data** with the **lowest probability** of being classified as a **positive review**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "most_negative_reviews = test_data.topk('probability', k=20, reverse=True)['name']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute accuracy of the classifier\n",
    "\n",
    "We will now evaluate the accuracy of the trained classifier. Recall that the accuracy is given by\n",
    "\n",
    "\n",
    "$$\n",
    "\\mbox{accuracy} = \\frac{\\mbox{# correctly classified examples}}{\\mbox{# total examples}}\n",
    "$$\n",
    "\n",
    "This can be computed as follows:\n",
    "\n",
    "* **Step 1:** Use the trained model to compute class predictions (**Hint:** Use the `predict` method)\n",
    "* **Step 2:** Count the number of data points when the predicted class labels match the ground truth labels (called `true_labels` below).\n",
    "* **Step 3:** Divide the total number of correct predictions by the total number of data points in the dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def get_classification_accuracy(model, data, true_labels):\n",
    "    # First get the predictions\n",
    "    predictions = model.predict(data)\n",
    "    \n",
    "    # Compute the number of correctly classified examples\n",
    "   \n",
    "    num_correct = [predictions==true_labels]\n",
    "    # print np.sum(num_correct)\n",
    "    # Then compute accuracy by dividing num_correct by total number of examples\n",
    "    accuracy = np.sum(num_correct)/len(data)\n",
    "    \n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's compute the classification accuracy of the **sentiment_model** on the **test_data**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91453683705303579"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_classification_accuracy(sentiment_model, test_data, test_data['sentiment'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn another classifier with fewer words\n",
    "\n",
    "There were a lot of words in the model we trained above. We will now train a simpler logistic regression model using only a subset of words that occur in the reviews. In this section, we selected a 20 words to work with. These are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "significant_words = ['love', 'great', 'easy', 'old', 'little', 'perfect', 'loves', \n",
    "      'well', 'able', 'car', 'broke', 'less', 'even', 'waste', 'disappointed', \n",
    "      'work', 'product', 'money', 'would', 'return']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(significant_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each review, we will use the **word_count** column and trim out all words that are **not** in the **significant_words** list above. We will use the [SArray dictionary trim by keys functionality]( https://dato.com/products/create/docs/generated/graphlab.SArray.dict_trim_by_keys.html). Note that we are performing this on both the training and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_data['word_count_subset'] = train_data['word_count'].dict_trim_by_keys(significant_words, exclude=False)\n",
    "test_data['word_count_subset'] = test_data['word_count'].dict_trim_by_keys(significant_words, exclude=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the first example of the dataset looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'it came early and was not disappointed. i love planet wise bags and now my wipe holder. it keps my osocozy wipes moist and does not leak. highly recommend it.'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]['review']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **word_count** column had been working with before looks like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'and': 3L, 'love': 1L, 'it': 3L, 'highly': 1L, 'osocozy': 1L, 'bags': 1L, 'leak': 1L, 'moist': 1L, 'does': 1L, 'recommend': 1L, 'was': 1L, 'wipes': 1L, 'disappointed': 1L, 'early': 1L, 'not': 2L, 'now': 1L, 'holder': 1L, 'wipe': 1L, 'keps': 1L, 'wise': 1L, 'i': 1L, 'planet': 1L, 'my': 2L, 'came': 1L}\n"
     ]
    }
   ],
   "source": [
    "print train_data[0]['word_count']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are only working with a subset of these words, the column **word_count_subset** is a subset of the above dictionary. In this example, only 2 `significant words` are present in this review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'love': 1L, 'disappointed': 1L}\n"
     ]
    }
   ],
   "source": [
    "print train_data[0]['word_count_subset']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a logistic regression model on a subset of data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now build a classifier with **word_count_subset** as the feature and **sentiment** as the target. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Class                          : LogisticClassifier\n",
       "\n",
       "Schema\n",
       "------\n",
       "Number of coefficients         : 21\n",
       "Number of examples             : 133416\n",
       "Number of classes              : 2\n",
       "Number of feature columns      : 1\n",
       "Number of unpacked features    : 20\n",
       "\n",
       "Hyperparameters\n",
       "---------------\n",
       "L1 penalty                     : 0.0\n",
       "L2 penalty                     : 0.01\n",
       "\n",
       "Training Summary\n",
       "----------------\n",
       "Solver                         : newton\n",
       "Solver iterations              : 6\n",
       "Solver status                  : SUCCESS: Optimal solution found.\n",
       "Training time (sec)            : 0.6126\n",
       "\n",
       "Settings\n",
       "--------\n",
       "Log-likelihood                 : 44323.7254\n",
       "\n",
       "Highest Positive Coefficients\n",
       "-----------------------------\n",
       "word_count_subset[loves]       : 1.6773\n",
       "word_count_subset[perfect]     : 1.5145\n",
       "word_count_subset[love]        : 1.3654\n",
       "(intercept)                    : 1.2995\n",
       "word_count_subset[easy]        : 1.1937\n",
       "\n",
       "Lowest Negative Coefficients\n",
       "----------------------------\n",
       "word_count_subset[disappointed] : -2.3551\n",
       "word_count_subset[return]      : -2.1173\n",
       "word_count_subset[waste]       : -2.0428\n",
       "word_count_subset[broke]       : -1.658\n",
       "word_count_subset[money]       : -0.8979"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_model = graphlab.logistic_classifier.create(train_data,\n",
    "                                                   target = 'sentiment',\n",
    "                                                   features=['word_count_subset'],\n",
    "                                                   verbose = False,\n",
    "                                                   validation_set=None)\n",
    "simple_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute the classification accuracy using the `get_classification_accuracy` function we implemented earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8693004559635229"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_classification_accuracy(simple_model, test_data, test_data['sentiment'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will inspect the weights (coefficients) of the **simple_model**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">index</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">class</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">value</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">stderr</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">(intercept)</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.2995449552</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0120888541331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">disappointed</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-2.35509250061</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0504149888557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">love</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.36543549368</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0303546295109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">well</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.504256746398</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.021381300631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">product</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-0.320555492996</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0154311321362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">loves</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.67727145556</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0482328275384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">little</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.520628636025</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0214691475665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">work</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-0.621700012425</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0230330597946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">easy</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.19366189833</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.029288869202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">word_count_subset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">great</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.94469126948</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0209509926591</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[21 rows x 5 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tname\tstr\n",
       "\tindex\tstr\n",
       "\tclass\tint\n",
       "\tvalue\tfloat\n",
       "\tstderr\tfloat\n",
       "\n",
       "Rows: 21\n",
       "\n",
       "Data:\n",
       "+-------------------+--------------+-------+-----------------+-----------------+\n",
       "|        name       |    index     | class |      value      |      stderr     |\n",
       "+-------------------+--------------+-------+-----------------+-----------------+\n",
       "|    (intercept)    |     None     |   1   |   1.2995449552  | 0.0120888541331 |\n",
       "| word_count_subset | disappointed |   1   |  -2.35509250061 | 0.0504149888557 |\n",
       "| word_count_subset |     love     |   1   |  1.36543549368  | 0.0303546295109 |\n",
       "| word_count_subset |     well     |   1   |  0.504256746398 |  0.021381300631 |\n",
       "| word_count_subset |   product    |   1   | -0.320555492996 | 0.0154311321362 |\n",
       "| word_count_subset |    loves     |   1   |  1.67727145556  | 0.0482328275384 |\n",
       "| word_count_subset |    little    |   1   |  0.520628636025 | 0.0214691475665 |\n",
       "| word_count_subset |     work     |   1   | -0.621700012425 | 0.0230330597946 |\n",
       "| word_count_subset |     easy     |   1   |  1.19366189833  |  0.029288869202 |\n",
       "| word_count_subset |    great     |   1   |  0.94469126948  | 0.0209509926591 |\n",
       "+-------------------+--------------+-------+-----------------+-----------------+\n",
       "[21 rows x 5 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_model.coefficients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's sort the coefficients (in descending order) by the **value** to obtain the coefficients with the most positive effect on the sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+--------------+-------+-----------------+-----------------+\n",
      "|        name       |    index     | class |      value      |      stderr     |\n",
      "+-------------------+--------------+-------+-----------------+-----------------+\n",
      "| word_count_subset |    loves     |   1   |  1.67727145556  | 0.0482328275384 |\n",
      "| word_count_subset |   perfect    |   1   |  1.51448626703  |  0.049861952294 |\n",
      "| word_count_subset |     love     |   1   |  1.36543549368  | 0.0303546295109 |\n",
      "|    (intercept)    |     None     |   1   |   1.2995449552  | 0.0120888541331 |\n",
      "| word_count_subset |     easy     |   1   |  1.19366189833  |  0.029288869202 |\n",
      "| word_count_subset |    great     |   1   |  0.94469126948  | 0.0209509926591 |\n",
      "| word_count_subset |    little    |   1   |  0.520628636025 | 0.0214691475665 |\n",
      "| word_count_subset |     well     |   1   |  0.504256746398 |  0.021381300631 |\n",
      "| word_count_subset |     able     |   1   |  0.191438302295 | 0.0337581955697 |\n",
      "| word_count_subset |     old      |   1   | 0.0853961886678 | 0.0200863423025 |\n",
      "| word_count_subset |     car      |   1   |  0.058834990068 | 0.0168291532091 |\n",
      "| word_count_subset |     less     |   1   | -0.209709815216 |  0.040505735954 |\n",
      "| word_count_subset |   product    |   1   | -0.320555492996 | 0.0154311321362 |\n",
      "| word_count_subset |    would     |   1   | -0.362308947711 | 0.0127544751985 |\n",
      "| word_count_subset |     even     |   1   |  -0.51173855127 | 0.0199612760261 |\n",
      "| word_count_subset |     work     |   1   | -0.621700012425 | 0.0230330597946 |\n",
      "| word_count_subset |    money     |   1   | -0.897884155776 | 0.0339936732836 |\n",
      "| word_count_subset |    broke     |   1   |  -1.65796447838 | 0.0580878907166 |\n",
      "| word_count_subset |    waste     |   1   |   -2.042773611  | 0.0644702932444 |\n",
      "| word_count_subset |    return    |   1   |  -2.11729659718 | 0.0578650807241 |\n",
      "| word_count_subset | disappointed |   1   |  -2.35509250061 | 0.0504149888557 |\n",
      "+-------------------+--------------+-------+-----------------+-----------------+\n",
      "[21 rows x 5 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "simple_model.coefficients.sort('value', ascending=False).print_rows(num_rows=21)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the coefficients of **simple_model**. There should be 21 of them, an intercept term + one for each word in **significant_words**. How many of the 20 coefficients (corresponding to the 20 **significant_words** and *excluding the intercept term*) are positive for the `simple_model`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum([simple_model['coefficients']['value']>0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are the positive words in the **simple_model** (let us call them `positive_significant_words`) also positive words in the **sentiment_model**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype: str\n",
       "Rows: ?\n",
       "[None, 'love', 'well', 'loves', 'little', 'easy', 'great', 'able', 'perfect', 'old', 'car', ... ]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "positive_significant_words = simple_model.coefficients['index'][simple_model['coefficients']['value']>0]\n",
    "positive_significant_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# positive significant words in sentiment model \n",
    "\n",
    "sentiment_model_significant_word = sentiment_model.coefficients[sentiment_model.coefficients['index'].is_in(positive_significant_words)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n"
     ]
    }
   ],
   "source": [
    "print len(sentiment_model_significant_word['value']>0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now compare the accuracy of the **sentiment_model** and the **simple_model** using the `get_classification_accuracy` method we implemented above.\n",
    "\n",
    "First, compute the classification accuracy of the **sentiment_model** on the **train_data**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.97944024704683097"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_classification_accuracy(sentiment_model, train_data, train_data['sentiment'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, compute the classification accuracy of the **simple_model** on the **train_data**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.86681507465371466"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_classification_accuracy(simple_model, train_data, train_data['sentiment'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, **sentiment_model** has higher accuracy on the TRAINING set than **simple_model**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will repeat this exercise on the **test_data**. Start by computing the classification accuracy of the **sentiment_model** on the **test_data**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91453683705303579"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_classification_accuracy(sentiment_model, test_data, test_data['sentiment'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will compute the classification accuracy of the **simple_model** on the **test_data**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8693004559635229"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_classification_accuracy(simple_model, test_data, test_data['sentiment'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the **sentiment_model** has higher accuracy on the TEST set than **simple_model**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline: Majority class prediction\n",
    "\n",
    "It is quite common to use the **majority class classifier** as the a baseline (or reference) model for comparison with the learned classifier model. The majority classifier model predicts the majority class for all data points. At the very least, we should healthily beat the majority class classifier, otherwise, the model is (usually) pointless.\n",
    "\n",
    "What is the majority class in the **train_data**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "112164\n",
      "21252\n"
     ]
    }
   ],
   "source": [
    "num_positive  = (train_data['sentiment'] == +1).sum()\n",
    "num_negative = (train_data['sentiment'] == -1).sum()\n",
    "print num_positive\n",
    "print num_negative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a classifier that always picks the most frequently occurring class (the \"majority class\"). For example, consider spam classification with two classes: positive/\"it's spam\" and negative/\"it's not spam\". Around 90% of all email is spam, so in the case of spam classification, the positive class is the majority class. The majority class spam classifier always guesses that an email is spam. Despite the simplicity, this classifier will have around 90% accuracy!\n",
    "\n",
    "A majority class classifier is \"trained\" to guess the majority class of the training set. For example, if the majority of reviews in the training set are positive, then the majority class will always guess that a review is positive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now compute the accuracy of the majority class classifier on **test_data**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "28095\n",
      "5241\n"
     ]
    }
   ],
   "source": [
    "print (test_data['sentiment'] == +1).sum()\n",
    "print (test_data['sentiment'] == -1).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.842782577394\n"
     ]
    }
   ],
   "source": [
    "print (test_data['sentiment'] == +1).sum()/len(test_data['sentiment'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}