Training Part 1 With Output.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**\\*Remember to activate the *far_nlp* environment before starting jupyter notebook from command line and running this notebook:**\n",
    "\n",
    "**OS X, Linux:** `$ source activate far_nlp`\n",
    "\n",
    "**Windows:** `$ activate far_nlp`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/Justin/anaconda/envs/far_nlp/bin/python\n"
     ]
    }
   ],
   "source": [
    "# Verify the right environment is enabled by checking the python path\n",
    "import sys\n",
    "print(sys.executable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How Can We Represent Text for Use in Machine Learning?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classic way to represent text for machine learning is with a *term frequency matrix* also sometimes referred to as a *Bag of Words representation*. In its simplest form this means counting how many times a word appears in a document, and we do this count for each document in our collection of documents (aka our corpus). \n",
    "\n",
    "In the example below our corpus is *eagle_powers* and each sentence is considered a separate document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "format": "row"
   },
   "outputs": [],
   "source": [
    "eagle_powers = [\"He knows where to find the eagle eggs.\",\n",
    "        \"His eagle eggs possess magical powers.\",\n",
    "        \"Summon your eagle powers.\"\n",
    "        ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is a custom script to show how vectorization works easily\n",
    "from utilities import SimpleVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false,
    "solution": "hidden"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>He</th>\n",
       "      <th>His</th>\n",
       "      <th>Summon</th>\n",
       "      <th>eagle</th>\n",
       "      <th>eggs</th>\n",
       "      <th>find</th>\n",
       "      <th>knows</th>\n",
       "      <th>magical</th>\n",
       "      <th>possess</th>\n",
       "      <th>powers</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>where</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        He  His  Summon  eagle  eggs  find  \\\n",
       "He knows where to find the eagle eggs.   1    0       0      1     1     1   \n",
       "His eagle eggs possess magical powers.   0    1       0      1     1     0   \n",
       "Summon your eagle powers.                0    0       1      1     0     0   \n",
       "\n",
       "                                        knows  magical  possess  powers  the  \\\n",
       "He knows where to find the eagle eggs.      1        0        0       0    1   \n",
       "His eagle eggs possess magical powers.      0        1        1       1    0   \n",
       "Summon your eagle powers.                   0        0        0       1    0   \n",
       "\n",
       "                                        to  where  your  \n",
       "He knows where to find the eagle eggs.   1      1     0  \n",
       "His eagle eggs possess magical powers.   0      0     0  \n",
       "Summon your eagle powers.                0      0     1  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SimpleVectorizer(eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out\n",
    "Make your own test corpus by creating a new variable that contains a list of sentences. How does the vectorizer handle your corpus, does it behave as expected?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Make a new list variable that contains a few sentences.\n",
    "# Pass your documents into the simpleVectorizer function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>This</th>\n",
       "      <th>another</th>\n",
       "      <th>is</th>\n",
       "      <th>sentance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>This is a sentance.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>This is another sentance.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           This  another  is  sentance\n",
       "This is a sentance.           1        0   1         1\n",
       "This is another sentance.     1        1   1         1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_corpus = [\"This is a sentance.\", \"This is another sentance.\"]\n",
    "SimpleVectorizer(my_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How does the SimpleVectorizer fall short?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modify your previous list and try to explore what some of the limitations of the SimpleVectorizer might be. For example, how does it handle capitalization? If the goal of vectorizing is capturing the meaning (or similarity of like documents), how does it fall short? How does it perform well? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Try to find shortcomings of the SimpleVectorizer by modifying your document corpus.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>He</th>\n",
       "      <th>Run</th>\n",
       "      <th>Running</th>\n",
       "      <th>The</th>\n",
       "      <th>east</th>\n",
       "      <th>hard</th>\n",
       "      <th>is</th>\n",
       "      <th>meeting</th>\n",
       "      <th>over</th>\n",
       "      <th>ran</th>\n",
       "      <th>river</th>\n",
       "      <th>runs</th>\n",
       "      <th>there</th>\n",
       "      <th>yesterday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He ran yesterday.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Run over there!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Running a meeting is hard.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The river runs east.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            He  Run  Running  The  east  hard  is  meeting  \\\n",
       "He ran yesterday.            1    0        0    0     0     0   0        0   \n",
       "Run over there!              0    1        0    0     0     0   0        0   \n",
       "Running a meeting is hard.   0    0        1    0     0     1   1        1   \n",
       "The river runs east.         0    0        0    1     1     0   0        0   \n",
       "\n",
       "                            over  ran  river  runs  there  yesterday  \n",
       "He ran yesterday.              0    1      0     0      0          1  \n",
       "Run over there!                1    0      0     0      1          0  \n",
       "Running a meeting is hard.     0    0      0     0      0          0  \n",
       "The river runs east.           0    0      1     1      0          0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_corpus = [\"He ran yesterday.\", \"Run over there!\", \"Running a meeting is hard.\", \"The river runs east.\"]\n",
    "SimpleVectorizer(my_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "What did we find?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Under the Hood of the SimpleVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we move on and see options for creating a a term frequency matrix, let's look under the hood of our SimpleVectorizer function and see how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Run this cell to see the components of the SimpleVectorizer function\n",
    "??SimpleVectorizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The SimpleVectorizer function is using a function called CountVectorizer from sci-kit learn to build the term frequency matrix. Let's see how to use CountVectorizer on its own.\n",
    "\n",
    "To use CountVectorizer we'll have to:\n",
    "1. Import it from sci-kit learn\n",
    "2. Instantiate it\n",
    "3. And call ```fit_transform``` on a set of documents.\n",
    "\n",
    "See the sci-kit learn docs on [CountVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import CountVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Instantiate CountVectorizer, this makes an instance of CV\n",
    "# with whatever options you specify.\n",
    "CV = CountVectorizer(lowercase=False, min_df=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3x14 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 18 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit CountVectoroizer instance on our document corpus\n",
    "# The result is a sparse matrix\n",
    "results = CV.fit_transform(eagle_powers)\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 4)\t1\n",
      "  (0, 3)\t1\n",
      "  (0, 10)\t1\n",
      "  (0, 5)\t1\n",
      "  (0, 11)\t1\n",
      "  (0, 12)\t1\n",
      "  (0, 6)\t1\n",
      "  (0, 0)\t1\n",
      "  (1, 9)\t1\n",
      "  (1, 7)\t1\n",
      "  (1, 8)\t1\n",
      "  (1, 1)\t1\n",
      "  (1, 4)\t1\n",
      "  (1, 3)\t1\n",
      "  (2, 13)\t1\n",
      "  (2, 2)\t1\n",
      "  (2, 9)\t1\n",
      "  (2, 3)\t1\n"
     ]
    }
   ],
   "source": [
    "# Here is what the sparse representation looks like\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0],\n",
       "        [0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0],\n",
       "        [0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1]], dtype=int64)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To see what this looks like as a regular matrix we need to use .todense()\n",
    "results.todense()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import the displayTFMatrix function to view the matrix as we have been.\n",
    "# For this function, pass in the CountVectorizer instance and the documents\n",
    "# that will be vectorized.\n",
    "from utilities import DisplayTFMatrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>He</th>\n",
       "      <th>His</th>\n",
       "      <th>Summon</th>\n",
       "      <th>eagle</th>\n",
       "      <th>eggs</th>\n",
       "      <th>find</th>\n",
       "      <th>knows</th>\n",
       "      <th>magical</th>\n",
       "      <th>possess</th>\n",
       "      <th>powers</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>where</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        He  His  Summon  eagle  eggs  find  \\\n",
       "He knows where to find the eagle eggs.   1    0       0      1     1     1   \n",
       "His eagle eggs possess magical powers.   0    1       0      1     1     0   \n",
       "Summon your eagle powers.                0    0       1      1     0     0   \n",
       "\n",
       "                                        knows  magical  possess  powers  the  \\\n",
       "He knows where to find the eagle eggs.      1        0        0       0    1   \n",
       "His eagle eggs possess magical powers.      0        1        1       1    0   \n",
       "Summon your eagle powers.                   0        0        0       1    0   \n",
       "\n",
       "                                        to  where  your  \n",
       "He knows where to find the eagle eggs.   1      1     0  \n",
       "His eagle eggs possess magical powers.   0      0     0  \n",
       "Summon your eagle powers.                0      0     1  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DisplayTFMatrix(CV, eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out:\n",
    "Repeat the steps above using your own document corpus to:\n",
    "1. **Instantiate CountVectorizer:** Make a new instance of CountVectorizer, you can set it equal to *CV* or anything else that makes sense to you.\n",
    "2. **Transform your Corpus:** Call the ```fit_transform``` method on the new CountVectorizer instance and pass in your corpus. Set this to a variable to capture the resulting sparse matrix.\n",
    "3. **View Sparse Representation of the Matrix:** View the sparse version of your term frequency matrix using ```print()```.\n",
    "4. **View Dense Representation of the Matrix:** View the dense representation of your term frequency matrix by calling `.todense()` on variable containing your term frequency matrix.\n",
    "5. **View the DataFrame:** View the data frame of your term frequency matrix using ```DisplayTFMatrix(CVInstance, corpus)```. Make sure to pass in the CountVectorizer instance and your corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Improving on Term Frequency Matricies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two of the main issues with Term Frequency Matrices are:\n",
    "- **Size:** They grow logarithmically to the number of documents in the corpus and get large very fast. This can lead to high computing costs, it can take a long time to vectorize all of the documents in a corpus.\n",
    "- **Noise:** Related to the first problem is that these representations of documents also contain a lot of noise. There is a lot of extra information that may not be helpful for our machine learning task. For example, is the word *the* indicative of document content? Is it useful to have a count for this word?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stop Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop words are words that we want to ommit from being counted. This could be because we don't think they convey any important meaning or we may want to omit them for other reasons.\n",
    "\n",
    "We can set the stop words by passing in `stopwords=\"english\"` as a parameter in count vectorizer. We can also pass in a list of our own custom words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>He</th>\n",
       "      <th>His</th>\n",
       "      <th>Summon</th>\n",
       "      <th>eagle</th>\n",
       "      <th>eggs</th>\n",
       "      <th>knows</th>\n",
       "      <th>magical</th>\n",
       "      <th>possess</th>\n",
       "      <th>powers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        He  His  Summon  eagle  eggs  knows  \\\n",
       "He knows where to find the eagle eggs.   1    0       0      1     1      1   \n",
       "His eagle eggs possess magical powers.   0    1       0      1     1      0   \n",
       "Summon your eagle powers.                0    0       1      1     0      0   \n",
       "\n",
       "                                        magical  possess  powers  \n",
       "He knows where to find the eagle eggs.        0        0       0  \n",
       "His eagle eggs possess magical powers.        1        1       1  \n",
       "Summon your eagle powers.                     0        0       1  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See what happens when we use the list of stopwords provided \n",
    "# with sci-kit learn.\n",
    "\n",
    "CV = CountVectorizer(stop_words=\"english\", lowercase=False)\n",
    "DisplayTFMatrix(CV, eagle_powers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "frozenset({'a',\n",
       "           'about',\n",
       "           'above',\n",
       "           'across',\n",
       "           'after',\n",
       "           'afterwards',\n",
       "           'again',\n",
       "           'against',\n",
       "           'all',\n",
       "           'almost',\n",
       "           'alone',\n",
       "           'along',\n",
       "           'already',\n",
       "           'also',\n",
       "           'although',\n",
       "           'always',\n",
       "           'am',\n",
       "           'among',\n",
       "           'amongst',\n",
       "           'amoungst',\n",
       "           'amount',\n",
       "           'an',\n",
       "           'and',\n",
       "           'another',\n",
       "           'any',\n",
       "           'anyhow',\n",
       "           'anyone',\n",
       "           'anything',\n",
       "           'anyway',\n",
       "           'anywhere',\n",
       "           'are',\n",
       "           'around',\n",
       "           'as',\n",
       "           'at',\n",
       "           'back',\n",
       "           'be',\n",
       "           'became',\n",
       "           'because',\n",
       "           'become',\n",
       "           'becomes',\n",
       "           'becoming',\n",
       "           'been',\n",
       "           'before',\n",
       "           'beforehand',\n",
       "           'behind',\n",
       "           'being',\n",
       "           'below',\n",
       "           'beside',\n",
       "           'besides',\n",
       "           'between',\n",
       "           'beyond',\n",
       "           'bill',\n",
       "           'both',\n",
       "           'bottom',\n",
       "           'but',\n",
       "           'by',\n",
       "           'call',\n",
       "           'can',\n",
       "           'cannot',\n",
       "           'cant',\n",
       "           'co',\n",
       "           'con',\n",
       "           'could',\n",
       "           'couldnt',\n",
       "           'cry',\n",
       "           'de',\n",
       "           'describe',\n",
       "           'detail',\n",
       "           'do',\n",
       "           'done',\n",
       "           'down',\n",
       "           'due',\n",
       "           'during',\n",
       "           'each',\n",
       "           'eg',\n",
       "           'eight',\n",
       "           'either',\n",
       "           'eleven',\n",
       "           'else',\n",
       "           'elsewhere',\n",
       "           'empty',\n",
       "           'enough',\n",
       "           'etc',\n",
       "           'even',\n",
       "           'ever',\n",
       "           'every',\n",
       "           'everyone',\n",
       "           'everything',\n",
       "           'everywhere',\n",
       "           'except',\n",
       "           'few',\n",
       "           'fifteen',\n",
       "           'fifty',\n",
       "           'fill',\n",
       "           'find',\n",
       "           'fire',\n",
       "           'first',\n",
       "           'five',\n",
       "           'for',\n",
       "           'former',\n",
       "           'formerly',\n",
       "           'forty',\n",
       "           'found',\n",
       "           'four',\n",
       "           'from',\n",
       "           'front',\n",
       "           'full',\n",
       "           'further',\n",
       "           'get',\n",
       "           'give',\n",
       "           'go',\n",
       "           'had',\n",
       "           'has',\n",
       "           'hasnt',\n",
       "           'have',\n",
       "           'he',\n",
       "           'hence',\n",
       "           'her',\n",
       "           'here',\n",
       "           'hereafter',\n",
       "           'hereby',\n",
       "           'herein',\n",
       "           'hereupon',\n",
       "           'hers',\n",
       "           'herself',\n",
       "           'him',\n",
       "           'himself',\n",
       "           'his',\n",
       "           'how',\n",
       "           'however',\n",
       "           'hundred',\n",
       "           'i',\n",
       "           'ie',\n",
       "           'if',\n",
       "           'in',\n",
       "           'inc',\n",
       "           'indeed',\n",
       "           'interest',\n",
       "           'into',\n",
       "           'is',\n",
       "           'it',\n",
       "           'its',\n",
       "           'itself',\n",
       "           'keep',\n",
       "           'last',\n",
       "           'latter',\n",
       "           'latterly',\n",
       "           'least',\n",
       "           'less',\n",
       "           'ltd',\n",
       "           'made',\n",
       "           'many',\n",
       "           'may',\n",
       "           'me',\n",
       "           'meanwhile',\n",
       "           'might',\n",
       "           'mill',\n",
       "           'mine',\n",
       "           'more',\n",
       "           'moreover',\n",
       "           'most',\n",
       "           'mostly',\n",
       "           'move',\n",
       "           'much',\n",
       "           'must',\n",
       "           'my',\n",
       "           'myself',\n",
       "           'name',\n",
       "           'namely',\n",
       "           'neither',\n",
       "           'never',\n",
       "           'nevertheless',\n",
       "           'next',\n",
       "           'nine',\n",
       "           'no',\n",
       "           'nobody',\n",
       "           'none',\n",
       "           'noone',\n",
       "           'nor',\n",
       "           'not',\n",
       "           'nothing',\n",
       "           'now',\n",
       "           'nowhere',\n",
       "           'of',\n",
       "           'off',\n",
       "           'often',\n",
       "           'on',\n",
       "           'once',\n",
       "           'one',\n",
       "           'only',\n",
       "           'onto',\n",
       "           'or',\n",
       "           'other',\n",
       "           'others',\n",
       "           'otherwise',\n",
       "           'our',\n",
       "           'ours',\n",
       "           'ourselves',\n",
       "           'out',\n",
       "           'over',\n",
       "           'own',\n",
       "           'part',\n",
       "           'per',\n",
       "           'perhaps',\n",
       "           'please',\n",
       "           'put',\n",
       "           'rather',\n",
       "           're',\n",
       "           'same',\n",
       "           'see',\n",
       "           'seem',\n",
       "           'seemed',\n",
       "           'seeming',\n",
       "           'seems',\n",
       "           'serious',\n",
       "           'several',\n",
       "           'she',\n",
       "           'should',\n",
       "           'show',\n",
       "           'side',\n",
       "           'since',\n",
       "           'sincere',\n",
       "           'six',\n",
       "           'sixty',\n",
       "           'so',\n",
       "           'some',\n",
       "           'somehow',\n",
       "           'someone',\n",
       "           'something',\n",
       "           'sometime',\n",
       "           'sometimes',\n",
       "           'somewhere',\n",
       "           'still',\n",
       "           'such',\n",
       "           'system',\n",
       "           'take',\n",
       "           'ten',\n",
       "           'than',\n",
       "           'that',\n",
       "           'the',\n",
       "           'their',\n",
       "           'them',\n",
       "           'themselves',\n",
       "           'then',\n",
       "           'thence',\n",
       "           'there',\n",
       "           'thereafter',\n",
       "           'thereby',\n",
       "           'therefore',\n",
       "           'therein',\n",
       "           'thereupon',\n",
       "           'these',\n",
       "           'they',\n",
       "           'thick',\n",
       "           'thin',\n",
       "           'third',\n",
       "           'this',\n",
       "           'those',\n",
       "           'though',\n",
       "           'three',\n",
       "           'through',\n",
       "           'throughout',\n",
       "           'thru',\n",
       "           'thus',\n",
       "           'to',\n",
       "           'together',\n",
       "           'too',\n",
       "           'top',\n",
       "           'toward',\n",
       "           'towards',\n",
       "           'twelve',\n",
       "           'twenty',\n",
       "           'two',\n",
       "           'un',\n",
       "           'under',\n",
       "           'until',\n",
       "           'up',\n",
       "           'upon',\n",
       "           'us',\n",
       "           'very',\n",
       "           'via',\n",
       "           'was',\n",
       "           'we',\n",
       "           'well',\n",
       "           'were',\n",
       "           'what',\n",
       "           'whatever',\n",
       "           'when',\n",
       "           'whence',\n",
       "           'whenever',\n",
       "           'where',\n",
       "           'whereafter',\n",
       "           'whereas',\n",
       "           'whereby',\n",
       "           'wherein',\n",
       "           'whereupon',\n",
       "           'wherever',\n",
       "           'whether',\n",
       "           'which',\n",
       "           'while',\n",
       "           'whither',\n",
       "           'who',\n",
       "           'whoever',\n",
       "           'whole',\n",
       "           'whom',\n",
       "           'whose',\n",
       "           'why',\n",
       "           'will',\n",
       "           'with',\n",
       "           'within',\n",
       "           'without',\n",
       "           'would',\n",
       "           'yet',\n",
       "           'you',\n",
       "           'your',\n",
       "           'yours',\n",
       "           'yourself',\n",
       "           'yourselves'})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check out the default list of stop words from sk learn\n",
    "from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS\n",
    "ENGLISH_STOP_WORDS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out:\n",
    "Make a new list variable that will contain a list of custom stopwords. Use the default stopwords from sci-kit learn as the starting point for the new list. Then append your own stopwords to the new stopwords list. Use ```DisplayTFMatrix()``` to see the effects of using the new stopwords list on your custom corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>He</th>\n",
       "      <th>His</th>\n",
       "      <th>Summon</th>\n",
       "      <th>eggs</th>\n",
       "      <th>knows</th>\n",
       "      <th>possess</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        He  His  Summon  eggs  knows  possess\n",
       "He knows where to find the eagle eggs.   1    0       0     1      1        0\n",
       "His eagle eggs possess magical powers.   0    1       0     1      0        1\n",
       "Summon your eagle powers.                0    0       1     0      0        0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the defualt stopwords to a new list variable. You need\n",
    "# to change it to a list because the original list is a frozen set and will\n",
    "# not allow you to add on other words\n",
    "custom_stop_words = list(ENGLISH_STOP_WORDS)\n",
    "\n",
    "# My custom Words\n",
    "new_stop_words = [\"eagle\",\"magical\",\"powers\"]\n",
    "\n",
    "# Add my words to the default stop words\n",
    "custom_stop_words.extend(new_stop_words)\n",
    "\n",
    "# Make a TF Matrix using my custom stop words.\n",
    "CV = CountVectorizer(stop_words=custom_stop_words, lowercase=False)\n",
    "DisplayTFMatrix(CV, eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimum and Maximum Limits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# To see how minimum and maxim limits work, it will help to\n",
    "# have a larger corpus with even more eagle powers.\n",
    "more_eagle_powers = [\"He knows where to find the eagle eggs.\",\n",
    "                     \"Eagle eggs?\",\n",
    "                     \"His eagle eggs possess magical powers.\",\n",
    "                     \"For you to become empowered by the eagle you must climb that cliff\",\n",
    "                     \"Find the egg and drink it.\",\n",
    "                     \"Summon your eagle powers.\",\n",
    "                     \"Eagle powers, come to me! Please!\"\n",
    "                    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: left;\" src=\"img/eagle_powers.gif\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Eagle</th>\n",
       "      <th>eagle</th>\n",
       "      <th>eggs</th>\n",
       "      <th>powers</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle eggs?</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>For you to become empowered by the eagle you must climb that cliff</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Find the egg and drink it.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle powers, come to me! Please!</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    Eagle  eagle  eggs  \\\n",
       "He knows where to find the eagle eggs.                  0      1     1   \n",
       "Eagle eggs?                                             1      0     1   \n",
       "His eagle eggs possess magical powers.                  0      1     1   \n",
       "For you to become empowered by the eagle you mu...      0      1     0   \n",
       "Find the egg and drink it.                              0      0     0   \n",
       "Summon your eagle powers.                               0      1     0   \n",
       "Eagle powers, come to me! Please!                       1      0     0   \n",
       "\n",
       "                                                    powers  the  to  \n",
       "He knows where to find the eagle eggs.                   0    1   1  \n",
       "Eagle eggs?                                              0    0   0  \n",
       "His eagle eggs possess magical powers.                   1    0   0  \n",
       "For you to become empowered by the eagle you mu...       0    1   1  \n",
       "Find the egg and drink it.                               0    1   0  \n",
       "Summon your eagle powers.                                1    0   0  \n",
       "Eagle powers, come to me! Please!                        1    0   1  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also get rid of the most and least frequent\n",
    "# words by setting minimum and maximum limits.\n",
    "# This can also reduce the matrix size and reduce noise.\n",
    "\n",
    "# CV = CountVectorizer(lowercase=False)\n",
    "CV = CountVectorizer(min_df=2, max_df=6, lowercase=False)\n",
    "DisplayTFMatrix(CV, more_eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out:\n",
    "\n",
    "Use the example above to do the following:\n",
    "\n",
    "- Add a few more sentences to your document corpus. Make sure to have some words that occur frequently throughout documents in the corpus so you can see the effects of setting a maximum term frequency.\n",
    "- Vectorize your corpus with CountVectorizer using upper and lower limits to see how it affects the final matrix.\n",
    "- Also, notice that we've been seeing a lowercase option for CountVectorizer. Set `lowercase=True` to have all the letters converted to lowercase notice how this effects the outcome in the final term frequency matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NGrams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of tokenizing a single word we can tokeninze several words at a time to get a better sense of meaning and nuance in the text. However, we also need to keep in mind this can greatly increase the the size of our matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eagle</th>\n",
       "      <th>eggs</th>\n",
       "      <th>find</th>\n",
       "      <th>he</th>\n",
       "      <th>his</th>\n",
       "      <th>knows</th>\n",
       "      <th>magical</th>\n",
       "      <th>possess</th>\n",
       "      <th>powers</th>\n",
       "      <th>summon</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>where</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        eagle  eggs  find  he  his  knows  \\\n",
       "He knows where to find the eagle eggs.      1     1     1   1    0      1   \n",
       "His eagle eggs possess magical powers.      1     1     0   0    1      0   \n",
       "Summon your eagle powers.                   1     0     0   0    0      0   \n",
       "\n",
       "                                        magical  possess  powers  summon  the  \\\n",
       "He knows where to find the eagle eggs.        0        0       0       0    1   \n",
       "His eagle eggs possess magical powers.        1        1       1       0    0   \n",
       "Summon your eagle powers.                     0        0       1       1    0   \n",
       "\n",
       "                                        to  where  your  \n",
       "He knows where to find the eagle eggs.   1      1     0  \n",
       "His eagle eggs possess magical powers.   0      0     0  \n",
       "Summon your eagle powers.                0      0     1  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is what our TF matrix looks like without using ngrams.\n",
    "# Or in other words using an ngram of 1, or a unigram\n",
    "# Note the size: 3 rows, 14 columns\n",
    "CV = CountVectorizer()\n",
    "DisplayTFMatrix(CV, eagle_powers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eagle eggs</th>\n",
       "      <th>eagle eggs possess</th>\n",
       "      <th>eagle powers</th>\n",
       "      <th>eggs possess</th>\n",
       "      <th>eggs possess magical</th>\n",
       "      <th>find the</th>\n",
       "      <th>find the eagle</th>\n",
       "      <th>he knows</th>\n",
       "      <th>he knows where</th>\n",
       "      <th>his eagle</th>\n",
       "      <th>...</th>\n",
       "      <th>summon your</th>\n",
       "      <th>summon your eagle</th>\n",
       "      <th>the eagle</th>\n",
       "      <th>the eagle eggs</th>\n",
       "      <th>to find</th>\n",
       "      <th>to find the</th>\n",
       "      <th>where to</th>\n",
       "      <th>where to find</th>\n",
       "      <th>your eagle</th>\n",
       "      <th>your eagle powers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        eagle eggs  eagle eggs possess  \\\n",
       "He knows where to find the eagle eggs.           1                   0   \n",
       "His eagle eggs possess magical powers.           1                   1   \n",
       "Summon your eagle powers.                        0                   0   \n",
       "\n",
       "                                        eagle powers  eggs possess  \\\n",
       "He knows where to find the eagle eggs.             0             0   \n",
       "His eagle eggs possess magical powers.             0             1   \n",
       "Summon your eagle powers.                          1             0   \n",
       "\n",
       "                                        eggs possess magical  find the  \\\n",
       "He knows where to find the eagle eggs.                     0         1   \n",
       "His eagle eggs possess magical powers.                     1         0   \n",
       "Summon your eagle powers.                                  0         0   \n",
       "\n",
       "                                        find the eagle  he knows  \\\n",
       "He knows where to find the eagle eggs.               1         1   \n",
       "His eagle eggs possess magical powers.               0         0   \n",
       "Summon your eagle powers.                            0         0   \n",
       "\n",
       "                                        he knows where  his eagle  \\\n",
       "He knows where to find the eagle eggs.               1          0   \n",
       "His eagle eggs possess magical powers.               0          1   \n",
       "Summon your eagle powers.                            0          0   \n",
       "\n",
       "                                              ...          summon your  \\\n",
       "He knows where to find the eagle eggs.        ...                    0   \n",
       "His eagle eggs possess magical powers.        ...                    0   \n",
       "Summon your eagle powers.                     ...                    1   \n",
       "\n",
       "                                        summon your eagle  the eagle  \\\n",
       "He knows where to find the eagle eggs.                  0          1   \n",
       "His eagle eggs possess magical powers.                  0          0   \n",
       "Summon your eagle powers.                               1          0   \n",
       "\n",
       "                                        the eagle eggs  to find  to find the  \\\n",
       "He knows where to find the eagle eggs.               1        1            1   \n",
       "His eagle eggs possess magical powers.               0        0            0   \n",
       "Summon your eagle powers.                            0        0            0   \n",
       "\n",
       "                                        where to  where to find  your eagle  \\\n",
       "He knows where to find the eagle eggs.         1              1           0   \n",
       "His eagle eggs possess magical powers.         0              0           0   \n",
       "Summon your eagle powers.                      0              0           1   \n",
       "\n",
       "                                        your eagle powers  \n",
       "He knows where to find the eagle eggs.                  0  \n",
       "His eagle eggs possess magical powers.                  0  \n",
       "Summon your eagle powers.                               1  \n",
       "\n",
       "[3 rows x 26 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now let's see what it looks like using an ngram range of 2-3.\n",
    "# This means we'll have only 2 and 3 word strings.\n",
    "# Notice the new size of the dataFrame\n",
    "\n",
    "CV = CountVectorizer(ngram_range=(2,3))\n",
    "DisplayTFMatrix(CV, eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a Term Frequency matrix using `ngram_range`. Experiment with different ranges to see how it affects the final matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your Code Here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eagle</th>\n",
       "      <th>eagle eggs</th>\n",
       "      <th>eagle eggs possess</th>\n",
       "      <th>eagle powers</th>\n",
       "      <th>eggs</th>\n",
       "      <th>eggs possess</th>\n",
       "      <th>eggs possess magical</th>\n",
       "      <th>find</th>\n",
       "      <th>find the</th>\n",
       "      <th>find the eagle</th>\n",
       "      <th>...</th>\n",
       "      <th>the eagle eggs</th>\n",
       "      <th>to</th>\n",
       "      <th>to find</th>\n",
       "      <th>to find the</th>\n",
       "      <th>where</th>\n",
       "      <th>where to</th>\n",
       "      <th>where to find</th>\n",
       "      <th>your</th>\n",
       "      <th>your eagle</th>\n",
       "      <th>your eagle powers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        eagle  eagle eggs  eagle eggs possess  \\\n",
       "He knows where to find the eagle eggs.      1           1                   0   \n",
       "His eagle eggs possess magical powers.      1           1                   1   \n",
       "Summon your eagle powers.                   1           0                   0   \n",
       "\n",
       "                                        eagle powers  eggs  eggs possess  \\\n",
       "He knows where to find the eagle eggs.             0     1             0   \n",
       "His eagle eggs possess magical powers.             0     1             1   \n",
       "Summon your eagle powers.                          1     0             0   \n",
       "\n",
       "                                        eggs possess magical  find  find the  \\\n",
       "He knows where to find the eagle eggs.                     0     1         1   \n",
       "His eagle eggs possess magical powers.                     1     0         0   \n",
       "Summon your eagle powers.                                  0     0         0   \n",
       "\n",
       "                                        find the eagle        ...          \\\n",
       "He knows where to find the eagle eggs.               1        ...           \n",
       "His eagle eggs possess magical powers.               0        ...           \n",
       "Summon your eagle powers.                            0        ...           \n",
       "\n",
       "                                        the eagle eggs  to  to find  \\\n",
       "He knows where to find the eagle eggs.               1   1        1   \n",
       "His eagle eggs possess magical powers.               0   0        0   \n",
       "Summon your eagle powers.                            0   0        0   \n",
       "\n",
       "                                        to find the  where  where to  \\\n",
       "He knows where to find the eagle eggs.            1      1         1   \n",
       "His eagle eggs possess magical powers.            0      0         0   \n",
       "Summon your eagle powers.                         0      0         0   \n",
       "\n",
       "                                        where to find  your  your eagle  \\\n",
       "He knows where to find the eagle eggs.              1     0           0   \n",
       "His eagle eggs possess magical powers.              0     0           0   \n",
       "Summon your eagle powers.                           0     1           1   \n",
       "\n",
       "                                        your eagle powers  \n",
       "He knows where to find the eagle eggs.                  0  \n",
       "His eagle eggs possess magical powers.                  0  \n",
       "Summon your eagle powers.                               1  \n",
       "\n",
       "[3 rows x 40 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CV = CountVectorizer(ngram_range=(1,3))\n",
    "DisplayTFMatrix(CV, eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Truncated SVD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we've seen with ngrams, it is easy for a vectorized corpus to become very wide and have a large number of features/columns. Truncated SVD allows us to reduce the dimensionality of our data (the number of features/columns) while maintaining a large degree of the variance from the original data. Truncated SVD works efficiently with data in sparse form, so it works well with the output from CountVectorizer.\n",
    "\n",
    "Scikit learn reference: [TruncatedSVD](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html)\n",
    "\n",
    "More info on TFIDF and how it works:\n",
    "- [https://www.quora.com/What-is-an-intuitive-explanation-of-singular-value-decomposition-SVD/answer/Jason-Liu-21?srid=nGs9](https://www.quora.com/What-is-an-intuitive-explanation-of-singular-value-decomposition-SVD/answer/Jason-Liu-21?srid=nGs9)\n",
    "- [https://stats.stackexchange.com/questions/134282/relationship-between-svd-and-pca-how-to-use-svd-to-perform-pca](https://stats.stackexchange.com/questions/134282/relationship-between-svd-and-pca-how-to-use-svd-to-perform-pca)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>and</th>\n",
       "      <th>and drink</th>\n",
       "      <th>and drink it</th>\n",
       "      <th>become</th>\n",
       "      <th>become empowered</th>\n",
       "      <th>become empowered by</th>\n",
       "      <th>by</th>\n",
       "      <th>by the</th>\n",
       "      <th>by the eagle</th>\n",
       "      <th>cliff</th>\n",
       "      <th>...</th>\n",
       "      <th>where to</th>\n",
       "      <th>where to find</th>\n",
       "      <th>you</th>\n",
       "      <th>you must</th>\n",
       "      <th>you must climb</th>\n",
       "      <th>you to</th>\n",
       "      <th>you to become</th>\n",
       "      <th>your</th>\n",
       "      <th>your eagle</th>\n",
       "      <th>your eagle powers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle eggs?</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>For you to become empowered by the eagle you must climb that cliff</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Find the egg and drink it.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle powers, come to me! Please!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7 rows × 94 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    and  and drink  \\\n",
       "He knows where to find the eagle eggs.                0          0   \n",
       "Eagle eggs?                                           0          0   \n",
       "His eagle eggs possess magical powers.                0          0   \n",
       "For you to become empowered by the eagle you mu...    0          0   \n",
       "Find the egg and drink it.                            1          1   \n",
       "Summon your eagle powers.                             0          0   \n",
       "Eagle powers, come to me! Please!                     0          0   \n",
       "\n",
       "                                                    and drink it  become  \\\n",
       "He knows where to find the eagle eggs.                         0       0   \n",
       "Eagle eggs?                                                    0       0   \n",
       "His eagle eggs possess magical powers.                         0       0   \n",
       "For you to become empowered by the eagle you mu...             0       1   \n",
       "Find the egg and drink it.                                     1       0   \n",
       "Summon your eagle powers.                                      0       0   \n",
       "Eagle powers, come to me! Please!                              0       0   \n",
       "\n",
       "                                                    become empowered  \\\n",
       "He knows where to find the eagle eggs.                             0   \n",
       "Eagle eggs?                                                        0   \n",
       "His eagle eggs possess magical powers.                             0   \n",
       "For you to become empowered by the eagle you mu...                 1   \n",
       "Find the egg and drink it.                                         0   \n",
       "Summon your eagle powers.                                          0   \n",
       "Eagle powers, come to me! Please!                                  0   \n",
       "\n",
       "                                                    become empowered by  by  \\\n",
       "He knows where to find the eagle eggs.                                0   0   \n",
       "Eagle eggs?                                                           0   0   \n",
       "His eagle eggs possess magical powers.                                0   0   \n",
       "For you to become empowered by the eagle you mu...                    1   1   \n",
       "Find the egg and drink it.                                            0   0   \n",
       "Summon your eagle powers.                                             0   0   \n",
       "Eagle powers, come to me! Please!                                     0   0   \n",
       "\n",
       "                                                    by the  by the eagle  \\\n",
       "He knows where to find the eagle eggs.                   0             0   \n",
       "Eagle eggs?                                              0             0   \n",
       "His eagle eggs possess magical powers.                   0             0   \n",
       "For you to become empowered by the eagle you mu...       1             1   \n",
       "Find the egg and drink it.                               0             0   \n",
       "Summon your eagle powers.                                0             0   \n",
       "Eagle powers, come to me! Please!                        0             0   \n",
       "\n",
       "                                                    cliff        ...          \\\n",
       "He knows where to find the eagle eggs.                  0        ...           \n",
       "Eagle eggs?                                             0        ...           \n",
       "His eagle eggs possess magical powers.                  0        ...           \n",
       "For you to become empowered by the eagle you mu...      1        ...           \n",
       "Find the egg and drink it.                              0        ...           \n",
       "Summon your eagle powers.                               0        ...           \n",
       "Eagle powers, come to me! Please!                       0        ...           \n",
       "\n",
       "                                                    where to  where to find  \\\n",
       "He knows where to find the eagle eggs.                     1              1   \n",
       "Eagle eggs?                                                0              0   \n",
       "His eagle eggs possess magical powers.                     0              0   \n",
       "For you to become empowered by the eagle you mu...         0              0   \n",
       "Find the egg and drink it.                                 0              0   \n",
       "Summon your eagle powers.                                  0              0   \n",
       "Eagle powers, come to me! Please!                          0              0   \n",
       "\n",
       "                                                    you  you must  \\\n",
       "He knows where to find the eagle eggs.                0         0   \n",
       "Eagle eggs?                                           0         0   \n",
       "His eagle eggs possess magical powers.                0         0   \n",
       "For you to become empowered by the eagle you mu...    2         1   \n",
       "Find the egg and drink it.                            0         0   \n",
       "Summon your eagle powers.                             0         0   \n",
       "Eagle powers, come to me! Please!                     0         0   \n",
       "\n",
       "                                                    you must climb  you to  \\\n",
       "He knows where to find the eagle eggs.                           0       0   \n",
       "Eagle eggs?                                                      0       0   \n",
       "His eagle eggs possess magical powers.                           0       0   \n",
       "For you to become empowered by the eagle you mu...               1       1   \n",
       "Find the egg and drink it.                                       0       0   \n",
       "Summon your eagle powers.                                        0       0   \n",
       "Eagle powers, come to me! Please!                                0       0   \n",
       "\n",
       "                                                    you to become  your  \\\n",
       "He knows where to find the eagle eggs.                          0     0   \n",
       "Eagle eggs?                                                     0     0   \n",
       "His eagle eggs possess magical powers.                          0     0   \n",
       "For you to become empowered by the eagle you mu...              1     0   \n",
       "Find the egg and drink it.                                      0     0   \n",
       "Summon your eagle powers.                                       0     1   \n",
       "Eagle powers, come to me! Please!                               0     0   \n",
       "\n",
       "                                                    your eagle  \\\n",
       "He knows where to find the eagle eggs.                       0   \n",
       "Eagle eggs?                                                  0   \n",
       "His eagle eggs possess magical powers.                       0   \n",
       "For you to become empowered by the eagle you mu...           0   \n",
       "Find the egg and drink it.                                   0   \n",
       "Summon your eagle powers.                                    1   \n",
       "Eagle powers, come to me! Please!                            0   \n",
       "\n",
       "                                                    your eagle powers  \n",
       "He knows where to find the eagle eggs.                              0  \n",
       "Eagle eggs?                                                         0  \n",
       "His eagle eggs possess magical powers.                              0  \n",
       "For you to become empowered by the eagle you mu...                  0  \n",
       "Find the egg and drink it.                                          0  \n",
       "Summon your eagle powers.                                           1  \n",
       "Eagle powers, come to me! Please!                                   0  \n",
       "\n",
       "[7 rows x 94 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's generate a wide term frequency matrix for comparison\n",
    "CV = CountVectorizer(ngram_range=(1,3))\n",
    "\n",
    "# The original matrix has 94 columns for this \n",
    "DisplayTFMatrix(CV, more_eagle_powers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now Let's use TSVD\n",
    "\n",
    "# Vectorize the documents with CV\n",
    "CV = CountVectorizer(ngram_range=(1,3))\n",
    "cv_results = CV.fit_transform(more_eagle_powers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Like other methods we need to import TSVD before use\n",
    "from sklearn.decomposition import TruncatedSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Then instantiate TSVD and set parameters\n",
    "TSVD = TruncatedSVD(n_components=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.60494085,  3.649311  ],\n",
       "       [ 0.37147293,  0.84195194],\n",
       "       [ 0.57243547,  1.9041164 ],\n",
       "       [ 5.97640789, -1.50231841],\n",
       "       [ 0.4385091 ,  1.11686785],\n",
       "       [ 0.36804936,  0.74440844],\n",
       "       [ 0.72263218,  1.32163696]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit TSVD on our term matrix, note it is not sparse\n",
    "tsvd_result = TSVD.fit_transform(cv_results)\n",
    "tsvd_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1.604941</td>\n",
       "      <td>3.649311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle eggs?</th>\n",
       "      <td>0.371473</td>\n",
       "      <td>0.841952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0.572435</td>\n",
       "      <td>1.904116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>For you to become empowered by the eagle you must climb that cliff</th>\n",
       "      <td>5.976408</td>\n",
       "      <td>-1.502318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Find the egg and drink it.</th>\n",
       "      <td>0.438509</td>\n",
       "      <td>1.116868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0.368049</td>\n",
       "      <td>0.744408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle powers, come to me! Please!</th>\n",
       "      <td>0.722632</td>\n",
       "      <td>1.321637</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                           0         1\n",
       "He knows where to find the eagle eggs.              1.604941  3.649311\n",
       "Eagle eggs?                                         0.371473  0.841952\n",
       "His eagle eggs possess magical powers.              0.572435  1.904116\n",
       "For you to become empowered by the eagle you mu...  5.976408 -1.502318\n",
       "Find the egg and drink it.                          0.438509  1.116868\n",
       "Summon your eagle powers.                           0.368049  0.744408\n",
       "Eagle powers, come to me! Please!                   0.722632  1.321637"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dispaly the TSVD results\n",
    "pd.DataFrame(tsvd_result, index=more_eagle_powers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.28052618,  0.15761034])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can see how much of the variance from the original data\n",
    "# is retained by using explained_variance_ratio_\n",
    "# With only two columns of data we're capturing 43% of the variance\n",
    "\n",
    "TSVD.explained_variance_ratio_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Make a wide term frequency matrix (use something like an `ngram_range(1,3)` or more) and set the results to a variable.\n",
    "2. Use Truncated SVD to reduce the columns of this data set to 2 columns.\n",
    "3. Display the results of the new matrix.\n",
    "4. Use `.explained_variance_ratio_` to see how much of the variance of the original data that each column captures.\n",
    "5. Adjust the `n_components` setting in Truncated SVD so that you are capturing 70%-80% of the variation of the original data. How many columns does this end up being? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1.604941</td>\n",
       "      <td>3.649311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle eggs?</th>\n",
       "      <td>0.371473</td>\n",
       "      <td>0.841952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0.572435</td>\n",
       "      <td>1.904116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>For you to become empowered by the eagle you must climb that cliff</th>\n",
       "      <td>5.976408</td>\n",
       "      <td>-1.502318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Find the egg and drink it.</th>\n",
       "      <td>0.438509</td>\n",
       "      <td>1.116868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0.368049</td>\n",
       "      <td>0.744408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle powers, come to me! Please!</th>\n",
       "      <td>0.722632</td>\n",
       "      <td>1.321637</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                           0         1\n",
       "He knows where to find the eagle eggs.              1.604941  3.649311\n",
       "Eagle eggs?                                         0.371473  0.841952\n",
       "His eagle eggs possess magical powers.              0.572435  1.904116\n",
       "For you to become empowered by the eagle you mu...  5.976408 -1.502318\n",
       "Find the egg and drink it.                          0.438509  1.116868\n",
       "Summon your eagle powers.                           0.368049  0.744408\n",
       "Eagle powers, come to me! Please!                   0.722632  1.321637"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array([ 0.33575538,  0.157374  ])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>1.604941</td>\n",
       "      <td>3.649311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle eggs?</th>\n",
       "      <td>0.371473</td>\n",
       "      <td>0.841952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0.572435</td>\n",
       "      <td>1.904116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>For you to become empowered by the eagle you must climb that cliff</th>\n",
       "      <td>5.976408</td>\n",
       "      <td>-1.502318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Find the egg and drink it.</th>\n",
       "      <td>0.438509</td>\n",
       "      <td>1.116868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0.368049</td>\n",
       "      <td>0.744408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle powers, come to me! Please!</th>\n",
       "      <td>0.722632</td>\n",
       "      <td>1.321637</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                           0         1\n",
       "He knows where to find the eagle eggs.              1.604941  3.649311\n",
       "Eagle eggs?                                         0.371473  0.841952\n",
       "His eagle eggs possess magical powers.              0.572435  1.904116\n",
       "For you to become empowered by the eagle you mu...  5.976408 -1.502318\n",
       "Find the egg and drink it.                          0.438509  1.116868\n",
       "Summon your eagle powers.                           0.368049  0.744408\n",
       "Eagle powers, come to me! Please!                   0.722632  1.321637"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array([ 0.33575538,  0.157374  ,  0.15684521,  0.12789046])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Vectorize the corpus\n",
    "CV = CountVectorizer(ngram_range=(1,5))\n",
    "cv_results = CV.fit_transform(more_eagle_powers)\n",
    "\n",
    "# Use TSVD on the Term Frequency Matrix\n",
    "TSVD = TruncatedSVD(n_components=2)\n",
    "tsvd_results = TSVD.fit_transform(cv_results)\n",
    "\n",
    "# Display the resulting reduced-feature Matrix \n",
    "display(pd.DataFrame(tsvd_result, index=more_eagle_powers))\n",
    "# Display how much variance was captured in each column\n",
    "display(TSVD.explained_variance_ratio_)\n",
    "\n",
    "\n",
    "# Adjust n_components to caputre 70% of the variance\n",
    "TSVD = TruncatedSVD(n_components=4)\n",
    "tsvd_results = TSVD.fit_transform(cv_results)\n",
    "\n",
    "# Display matrix and variance\n",
    "display(pd.DataFrame(tsvd_result, index=more_eagle_powers))\n",
    "display(TSVD.explained_variance_ratio_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Term frequency-inverse document frequency (TF-IDF) is a way of weighting words that gives the highest value to words which occur frequently in a specific document but occurs less frequently in the overall corpus.\n",
    "\n",
    "1. Highest when a term occurs many times within a small number of documents (thus lending high discriminating power to those documents);\n",
    "\n",
    "2. Lower when a term occurs fewer times in a document, or occurs in many documents (thus offering a less pronounced relevance signal);\n",
    "\n",
    "3. Lowest when a term occurs in virtually all documents.\n",
    "\n",
    "**Above taken from second resource linked to below.*\n",
    "\n",
    "Resources for learning more about TF-IDF:\n",
    "- Good introduction to the idea of TF-IDF: [http://www.tfidf.com/](http://www.tfidf.com/)\n",
    "- [Introduction to Information Retrieval: TF-idf Stanford](https://nlp.stanford.edu/IR-book/html/htmledition/tf-idf-weighting-1.html)\n",
    "- Details on Sci-kit learn's specific implementation of TF-IDF: [Tf–idf term weighting](http://scikit-learn.org/stable/modules/feature_extraction.html#tfidf-term-weighting)\n",
    "- Documentatio for using Tf-Idf in sci-kit learn: [sklearn.feature_extraction.text.TfidfVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the basic concept of the math behind TF-IDF. However, there are several different implementations and options that can be added to TF-IDF so you may see slightly different formulas out in the wild.\n",
    "\n",
    "$\\text{tfidf}_t = \\frac{\\text{number of times a word appears in a document}}{\\text{total number of words in a document}} \\cdot log( \\frac{\\text{total number of documents in corpus}}{\\text{number of documents in corpus where a word appears at least once}} )$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A quick TFIDF example (taken from [tfidf.com](http://www.tfidf.com/))**\n",
    "\n",
    "Our goal is to calculate the TFIDF value for the word *cat* in the given document below:\n",
    "- Our document has a total of 100 words.\n",
    "- The word *cat* occurs 3 times in the document.\n",
    "- Our corpus has 10,000,000 documents total.\n",
    "- The word cat occurs at least once in 1,000 documents in our corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The normalized term frequency is:  0.03\n",
      "The inverse document frequency is:  4.0\n",
      "The TF-IDF value for the word cat in our document is:  0.12\n"
     ]
    }
   ],
   "source": [
    "from math import log\n",
    "\n",
    "normalized_term_frequency = 3/100\n",
    "inverse_document_frequency = log(10000000/1000, 10)\n",
    "tfidf_value = normalized_term_frequency*inverse_document_frequency\n",
    "\n",
    "print(\"The normalized term frequency is: \", normalized_term_frequency)\n",
    "print(\"The inverse document frequency is: \", inverse_document_frequency)\n",
    "print(\"The TF-IDF value for the word cat in our document is: \", tfidf_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Similar to CountVectorizer we need to import TFIDF Vectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eagle</th>\n",
       "      <th>eggs</th>\n",
       "      <th>find</th>\n",
       "      <th>he</th>\n",
       "      <th>his</th>\n",
       "      <th>knows</th>\n",
       "      <th>magical</th>\n",
       "      <th>possess</th>\n",
       "      <th>powers</th>\n",
       "      <th>summon</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>where</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>0.224401</td>\n",
       "      <td>0.288958</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0.278245</td>\n",
       "      <td>0.358291</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.47111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.47111</td>\n",
       "      <td>0.47111</td>\n",
       "      <td>0.358291</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0.345205</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.444514</td>\n",
       "      <td>0.584483</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.584483</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           eagle      eggs      find  \\\n",
       "He knows where to find the eagle eggs.  0.224401  0.288958  0.379945   \n",
       "His eagle eggs possess magical powers.  0.278245  0.358291  0.000000   \n",
       "Summon your eagle powers.               0.345205  0.000000  0.000000   \n",
       "\n",
       "                                              he      his     knows  magical  \\\n",
       "He knows where to find the eagle eggs.  0.379945  0.00000  0.379945  0.00000   \n",
       "His eagle eggs possess magical powers.  0.000000  0.47111  0.000000  0.47111   \n",
       "Summon your eagle powers.               0.000000  0.00000  0.000000  0.00000   \n",
       "\n",
       "                                        possess    powers    summon       the  \\\n",
       "He knows where to find the eagle eggs.  0.00000  0.000000  0.000000  0.379945   \n",
       "His eagle eggs possess magical powers.  0.47111  0.358291  0.000000  0.000000   \n",
       "Summon your eagle powers.               0.00000  0.444514  0.584483  0.000000   \n",
       "\n",
       "                                              to     where      your  \n",
       "He knows where to find the eagle eggs.  0.379945  0.379945  0.000000  \n",
       "His eagle eggs possess magical powers.  0.000000  0.000000  0.000000  \n",
       "Summon your eagle powers.               0.000000  0.000000  0.584483  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Instantiate TFIDF\n",
    "TFIDF = TfidfVectorizer()\n",
    "\n",
    "# Display TFIDF\n",
    "DisplayTFMatrix(TFIDF, eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look over the matrix above to get a feel for what TFIDF is doing.\n",
    "\n",
    "- What words have the highest value, and the lowest?\n",
    "- For words that occur in all documents, why are values different or the same across documents?\n",
    "\n",
    "**Some may notice that the values are not the same as if we calculated them by hand using the formula above. I'm guessing this is due to Sci-kit learn's specific way of implementing TF-IDF.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `TfidfVectorizer()` to vectorize your corpus and then use `DisplayTFMatrix()` to display it. Examine the results and answer the same questions as above:\n",
    "- What words have the highest value, and the lowest?\n",
    "- For words that occur in all documents, why are values different or the same across documents?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your Code Here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eagle</th>\n",
       "      <th>eggs</th>\n",
       "      <th>find</th>\n",
       "      <th>he</th>\n",
       "      <th>his</th>\n",
       "      <th>knows</th>\n",
       "      <th>magical</th>\n",
       "      <th>possess</th>\n",
       "      <th>powers</th>\n",
       "      <th>summon</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>where</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>0.224401</td>\n",
       "      <td>0.288958</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.379945</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0.278245</td>\n",
       "      <td>0.358291</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.47111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.47111</td>\n",
       "      <td>0.47111</td>\n",
       "      <td>0.358291</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0.345205</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.444514</td>\n",
       "      <td>0.584483</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.584483</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           eagle      eggs      find  \\\n",
       "He knows where to find the eagle eggs.  0.224401  0.288958  0.379945   \n",
       "His eagle eggs possess magical powers.  0.278245  0.358291  0.000000   \n",
       "Summon your eagle powers.               0.345205  0.000000  0.000000   \n",
       "\n",
       "                                              he      his     knows  magical  \\\n",
       "He knows where to find the eagle eggs.  0.379945  0.00000  0.379945  0.00000   \n",
       "His eagle eggs possess magical powers.  0.000000  0.47111  0.000000  0.47111   \n",
       "Summon your eagle powers.               0.000000  0.00000  0.000000  0.00000   \n",
       "\n",
       "                                        possess    powers    summon       the  \\\n",
       "He knows where to find the eagle eggs.  0.00000  0.000000  0.000000  0.379945   \n",
       "His eagle eggs possess magical powers.  0.47111  0.358291  0.000000  0.000000   \n",
       "Summon your eagle powers.               0.00000  0.444514  0.584483  0.000000   \n",
       "\n",
       "                                              to     where      your  \n",
       "He knows where to find the eagle eggs.  0.379945  0.379945  0.000000  \n",
       "His eagle eggs possess magical powers.  0.000000  0.000000  0.000000  \n",
       "Summon your eagle powers.               0.000000  0.000000  0.584483  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TFIDF = TfidfVectorizer()\n",
    "DisplayTFMatrix(TFIDF, eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Lemmatization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lemmatization is getting the base form of a word which we will then use to make our term frequency matrix. In this way, different variations of a word can be mapped to the root word. This can further reduce the size and (potentially) the noise in our Term Frequency Matrix. We'll use a NLP library called spaCy to tokenzie our text and get the lemmas. We'll pass the results from spaCy to CountVectorizer to generate our term frequency matrix.\n",
    "\n",
    "SpaCy documentation: [https://spacy.io/docs/usage/](https://spacy.io/docs/usage/)\n",
    "SpaCy can do a LOT more than what we are using it for here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is how we load spaCy\n",
    "import spacy              # First import the package\n",
    "nlp = spacy.load('en')    # Then load the langauge model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: He  | Lemma: -PRON-\n",
      "Original: knows  | Lemma: know\n",
      "Original: where  | Lemma: where\n",
      "Original: to  | Lemma: to\n",
      "Original: find  | Lemma: find\n",
      "Original: the  | Lemma: the\n",
      "Original: eagle  | Lemma: eagle\n",
      "Original: eggs  | Lemma: egg\n",
      "Original: .  | Lemma: .\n",
      "Original: His  | Lemma: -PRON-\n",
      "Original: eagle  | Lemma: eagle\n",
      "Original: eggs  | Lemma: egg\n",
      "Original: possess  | Lemma: posses\n",
      "Original: magical  | Lemma: magical\n",
      "Original: powers  | Lemma: power\n",
      "Original: .  | Lemma: .\n",
      "Original: Summon  | Lemma: summon\n",
      "Original: your  | Lemma: -PRON-\n",
      "Original: eagle  | Lemma: eagle\n",
      "Original: powers  | Lemma: power\n",
      "Original: .  | Lemma: .\n"
     ]
    }
   ],
   "source": [
    "# let's see see how spacy works and how the lemmas compare to \n",
    "# the original text\n",
    "\n",
    "for doc in eagle_powers:\n",
    "    spacy_doc = nlp(doc)\n",
    "    for tok in spacy_doc:\n",
    "        print(\"Original:\", tok.orth_, \" | Lemma:\", tok.lemma_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#  Now let's bring together tokenization with vectorization.\n",
    "\n",
    "# Setup an empty list that will hold our tokenized (lemma) documents\n",
    "eagle_power_lemmas = []\n",
    "\n",
    "# Iterate Through each document\n",
    "for doc in more_eagle_powers: \n",
    "    \n",
    "    # Process the document with spaCy\n",
    "    spacy_doc = nlp(doc)\n",
    "    \n",
    "    # Grab the lemma for each token\n",
    "    lemmas = [token.lemma_ for token in spacy_doc] \n",
    "    \n",
    "    # Make the list of lemmas into a string\n",
    "    lemmas = \" \".join(lemmas)\n",
    "    \n",
    "    # Add the lemma string to the eagle_power_lemmas list\n",
    "    eagle_power_lemmas.append(lemmas) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['-PRON- know where to find the eagle egg .',\n",
       " 'eagle egg ?',\n",
       " '-PRON- eagle egg posses magical power .',\n",
       " 'for -PRON- to become empower by the eagle -PRON- must climb that cliff',\n",
       " 'find the egg and drink -PRON- .',\n",
       " 'summon -PRON- eagle power .',\n",
       " 'eagle power , come to -PRON- ! please !']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's see what our string of lemma's looks like\n",
    "eagle_power_lemmas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>and</th>\n",
       "      <th>become</th>\n",
       "      <th>by</th>\n",
       "      <th>cliff</th>\n",
       "      <th>climb</th>\n",
       "      <th>come</th>\n",
       "      <th>drink</th>\n",
       "      <th>eagle</th>\n",
       "      <th>egg</th>\n",
       "      <th>empower</th>\n",
       "      <th>...</th>\n",
       "      <th>must</th>\n",
       "      <th>please</th>\n",
       "      <th>posses</th>\n",
       "      <th>power</th>\n",
       "      <th>pron</th>\n",
       "      <th>summon</th>\n",
       "      <th>that</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>where</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>-PRON- know where to find the eagle egg .</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eagle egg ?</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-PRON- eagle egg posses magical power .</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>for -PRON- to become empower by the eagle -PRON- must climb that cliff</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>find the egg and drink -PRON- .</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>summon -PRON- eagle power .</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eagle power , come to -PRON- ! please !</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    and  become  by  cliff  \\\n",
       "-PRON- know where to find the eagle egg .             0       0   0      0   \n",
       "eagle egg ?                                           0       0   0      0   \n",
       "-PRON- eagle egg posses magical power .               0       0   0      0   \n",
       "for -PRON- to become empower by the eagle -PRON...    0       1   1      1   \n",
       "find the egg and drink -PRON- .                       1       0   0      0   \n",
       "summon -PRON- eagle power .                           0       0   0      0   \n",
       "eagle power , come to -PRON- ! please !               0       0   0      0   \n",
       "\n",
       "                                                    climb  come  drink  eagle  \\\n",
       "-PRON- know where to find the eagle egg .               0     0      0      1   \n",
       "eagle egg ?                                             0     0      0      1   \n",
       "-PRON- eagle egg posses magical power .                 0     0      0      1   \n",
       "for -PRON- to become empower by the eagle -PRON...      1     0      0      1   \n",
       "find the egg and drink -PRON- .                         0     0      1      0   \n",
       "summon -PRON- eagle power .                             0     0      0      1   \n",
       "eagle power , come to -PRON- ! please !                 0     1      0      1   \n",
       "\n",
       "                                                    egg  empower  ...    must  \\\n",
       "-PRON- know where to find the eagle egg .             1        0  ...       0   \n",
       "eagle egg ?                                           1        0  ...       0   \n",
       "-PRON- eagle egg posses magical power .               1        0  ...       0   \n",
       "for -PRON- to become empower by the eagle -PRON...    0        1  ...       1   \n",
       "find the egg and drink -PRON- .                       1        0  ...       0   \n",
       "summon -PRON- eagle power .                           0        0  ...       0   \n",
       "eagle power , come to -PRON- ! please !               0        0  ...       0   \n",
       "\n",
       "                                                    please  posses  power  \\\n",
       "-PRON- know where to find the eagle egg .                0       0      0   \n",
       "eagle egg ?                                              0       0      0   \n",
       "-PRON- eagle egg posses magical power .                  0       1      1   \n",
       "for -PRON- to become empower by the eagle -PRON...       0       0      0   \n",
       "find the egg and drink -PRON- .                          0       0      0   \n",
       "summon -PRON- eagle power .                              0       0      1   \n",
       "eagle power , come to -PRON- ! please !                  1       0      1   \n",
       "\n",
       "                                                    pron  summon  that  the  \\\n",
       "-PRON- know where to find the eagle egg .              1       0     0    1   \n",
       "eagle egg ?                                            0       0     0    0   \n",
       "-PRON- eagle egg posses magical power .                1       0     0    0   \n",
       "for -PRON- to become empower by the eagle -PRON...     2       0     1    1   \n",
       "find the egg and drink -PRON- .                        1       0     0    1   \n",
       "summon -PRON- eagle power .                            1       1     0    0   \n",
       "eagle power , come to -PRON- ! please !                1       0     0    0   \n",
       "\n",
       "                                                    to  where  \n",
       "-PRON- know where to find the eagle egg .            1      1  \n",
       "eagle egg ?                                          0      0  \n",
       "-PRON- eagle egg posses magical power .              0      0  \n",
       "for -PRON- to become empower by the eagle -PRON...   1      0  \n",
       "find the egg and drink -PRON- .                      0      0  \n",
       "summon -PRON- eagle power .                          0      0  \n",
       "eagle power , come to -PRON- ! please !              1      0  \n",
       "\n",
       "[7 rows x 24 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CV = CountVectorizer()\n",
    "DisplayTFMatrix(CV, eagle_power_lemmas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drawing on what we've seen above, build a function named *spacy_tokenizer* that does the following:\n",
    "- Take in a single string (document) as an argument — not a corpus or collection of docs/\n",
    "- Process the string and create a spaCy document.\n",
    "- Iterates tokens in the spaCy document and returns the lemma for each token as a list.\n",
    "- Test out the function with a single string (document)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your Code Here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def spacy_tokenizer(doc_as_string):\n",
    "    spacy_doc = nlp(doc_as_string)\n",
    "    tokens = [tok.lemma_ for tok in spacy_doc]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['test', 'some', 'string', 'here', '.']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy_tokenizer(\"Test some string here.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it Out 2:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now make a new instance of CountVectorizer and add the following parameter: `tokenizer=spacy_tokenizer`. Feel free to add in any of the other parameters that we've learned.\n",
    "\n",
    "Use `DisplayTFMatrix` to display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>!</th>\n",
       "      <th>,</th>\n",
       "      <th>-PRON-</th>\n",
       "      <th>.</th>\n",
       "      <th>?</th>\n",
       "      <th>cliff</th>\n",
       "      <th>climb</th>\n",
       "      <th>come</th>\n",
       "      <th>drink</th>\n",
       "      <th>eagle</th>\n",
       "      <th>egg</th>\n",
       "      <th>empower</th>\n",
       "      <th>know</th>\n",
       "      <th>magical</th>\n",
       "      <th>posses</th>\n",
       "      <th>power</th>\n",
       "      <th>summon</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>He knows where to find the eagle eggs.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle eggs?</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>His eagle eggs possess magical powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>For you to become empowered by the eagle you must climb that cliff</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Find the egg and drink it.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Summon your eagle powers.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eagle powers, come to me! Please!</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    !  ,  -PRON-  .  ?  cliff  \\\n",
       "He knows where to find the eagle eggs.              0  0       1  1  0      0   \n",
       "Eagle eggs?                                         0  0       0  0  1      0   \n",
       "His eagle eggs possess magical powers.              0  0       1  1  0      0   \n",
       "For you to become empowered by the eagle you mu...  0  0       2  0  0      1   \n",
       "Find the egg and drink it.                          0  0       1  1  0      0   \n",
       "Summon your eagle powers.                           0  0       1  1  0      0   \n",
       "Eagle powers, come to me! Please!                   2  1       1  0  0      0   \n",
       "\n",
       "                                                    climb  come  drink  eagle  \\\n",
       "He knows where to find the eagle eggs.                  0     0      0      1   \n",
       "Eagle eggs?                                             0     0      0      1   \n",
       "His eagle eggs possess magical powers.                  0     0      0      1   \n",
       "For you to become empowered by the eagle you mu...      1     0      0      1   \n",
       "Find the egg and drink it.                              0     0      1      0   \n",
       "Summon your eagle powers.                               0     0      0      1   \n",
       "Eagle powers, come to me! Please!                       0     1      0      1   \n",
       "\n",
       "                                                    egg  empower  know  \\\n",
       "He knows where to find the eagle eggs.                1        0     1   \n",
       "Eagle eggs?                                           1        0     0   \n",
       "His eagle eggs possess magical powers.                1        0     0   \n",
       "For you to become empowered by the eagle you mu...    0        1     0   \n",
       "Find the egg and drink it.                            1        0     0   \n",
       "Summon your eagle powers.                             0        0     0   \n",
       "Eagle powers, come to me! Please!                     0        0     0   \n",
       "\n",
       "                                                    magical  posses  power  \\\n",
       "He knows where to find the eagle eggs.                    0       0      0   \n",
       "Eagle eggs?                                               0       0      0   \n",
       "His eagle eggs possess magical powers.                    1       1      1   \n",
       "For you to become empowered by the eagle you mu...        0       0      0   \n",
       "Find the egg and drink it.                                0       0      0   \n",
       "Summon your eagle powers.                                 0       0      1   \n",
       "Eagle powers, come to me! Please!                         0       0      1   \n",
       "\n",
       "                                                    summon  \n",
       "He knows where to find the eagle eggs.                   0  \n",
       "Eagle eggs?                                              0  \n",
       "His eagle eggs possess magical powers.                   0  \n",
       "For you to become empowered by the eagle you mu...       0  \n",
       "Find the egg and drink it.                               0  \n",
       "Summon your eagle powers.                                1  \n",
       "Eagle powers, come to me! Please!                        0  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CV = CountVectorizer(stop_words='english', tokenizer=spacy_tokenizer)\n",
    "DisplayTFMatrix(CV, more_eagle_powers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# End of Part 1"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}