# Unfolding Naïve Bayes from Scratch! Take-2 🎬.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unfolding Naïve Bayes from Scratch! Take-2 🎬\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cdn-images-1.medium.com/max/1000/1*sjet9qSO4O8fX2-FXvxflw.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation of Naïve Bayes from Scratch using Python ONLY - No Fancy Frameworks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](using.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Outcome of this Tutorial - A Hands-On Pythonic Implementation of NB\n",
    "So in my previous blog post of [Unfolding Naïve Bayes from Scratch! Take-1 🎬](https://towardsdatascience.com/unfolding-na%C3%AFve-bayes-from-scratch-2e86dcae4b01) , I tried to decode the rocket science behind the working of The Naïve Bayes (NB) ML algorithm, and after going through it's algorithmic insights, you too must have realized that it's quite a painless algorithm and once we walk-through it's complete step by step pythonic implementation ( no API and by using basic python  only) , it will be quite evident that how easy it is to code NB from scratch and that NB is not that Naïve at classifying !\n",
    "Oh and by the way, once you reach the end of this blog post, you will be done with complete 80% of understanding NB and only 20% will be remaining to master it! Only 20% remaining to go from zero to hero 🎓💯🥇"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin writing code for Naive Bayes, I assume you are familiar with:\n",
    "1.  Python Lists\n",
    "2.  Numpy & a lil bit of writing vectorized codes\n",
    "3.  Dictionaries\n",
    "4.  Regex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin with a few imports that we would need while implementing Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "from collections import defaultdict\n",
    "import re "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is much much more easier to organize and reuse the code if we define a class of NaiveBayes rather than using the traditional functional programming approach. So we will be defining a NaiveBayes class and write all relevant functions inside this class.\n",
    "However, funcions that are not relevant to NaiveBayes class will be defined separately"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh by the way - We will be writing a a fully generic code for the NaiveBayes Classifier! <br> No matter how many classes come into the training dataset - it will be able to train a fully working model  👏 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets first write a handy text preprocessing function which is not part of the NaiveBayes class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def preprocess_string(str_arg):\n",
    "    \n",
    "    \"\"\"\"\n",
    "        Parameters:\n",
    "        ----------\n",
    "        str_arg: example string to be preprocessed\n",
    "        \n",
    "        What the function does?\n",
    "        -----------------------\n",
    "        Preprocess the string argument - str_arg - such that :\n",
    "        1. everything apart from letters is excluded\n",
    "        2. multiple spaces are replaced by single space\n",
    "        3. str_arg is converted to lower case \n",
    "        \n",
    "        Example:\n",
    "        --------\n",
    "        Input :  Menu is absolutely perfect,loved it!\n",
    "        Output:  ['menu', 'is', 'absolutely', 'perfect', 'loved', 'it']\n",
    "        \n",
    "\n",
    "        Returns:\n",
    "        ---------\n",
    "        Preprocessed string \n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    cleaned_str=re.sub('[^a-z\\s]+',' ',str_arg,flags=re.IGNORECASE) #every char except alphabets is replaced\n",
    "    cleaned_str=re.sub('(\\s+)',' ',cleaned_str) #multiple spaces are replaced by single space\n",
    "    cleaned_str=cleaned_str.lower() #converting the cleaned string to lower case\n",
    "    \n",
    "    return cleaned_str # eturning the preprocessed string in tokenized form"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class NaiveBayes:\n",
    "    \n",
    "    def __init__(self,unique_classes):\n",
    "        \n",
    "        self.classes=unique_classes # Constructor is sinply passed with unique number of classes of the training set\n",
    "        \n",
    "\n",
    "    def addToBow(self,example,dict_index):\n",
    "        \n",
    "        '''\n",
    "            Parameters:\n",
    "            1. example \n",
    "            2. dict_index - implies to which BoW category this example belongs to\n",
    "\n",
    "            What the function does?\n",
    "            -----------------------\n",
    "            It simply splits the example on the basis of space as a tokenizer and adds every tokenized word to\n",
    "            its corresponding dictionary/BoW\n",
    "\n",
    "            Returns:\n",
    "            ---------\n",
    "            Nothing\n",
    "        \n",
    "       '''\n",
    "        \n",
    "        if isinstance(example,np.ndarray): example=example[0]\n",
    "     \n",
    "        for token_word in example.split(): #for every word in preprocessed example\n",
    "          \n",
    "            self.bow_dicts[dict_index][token_word]+=1 #increment in its count\n",
    "            \n",
    "    def train(self,dataset,labels):\n",
    "        \n",
    "        '''\n",
    "            Parameters:\n",
    "            1. dataset - shape = (m X d)\n",
    "            2. labels - shape = (m,)\n",
    "\n",
    "            What the function does?\n",
    "            -----------------------\n",
    "            This is the training function which will train the Naive Bayes Model i.e compute a BoW for each\n",
    "            category/class. \n",
    "\n",
    "            Returns:\n",
    "            ---------\n",
    "            Nothing\n",
    "        \n",
    "        '''\n",
    "    \n",
    "        self.examples=dataset\n",
    "        self.labels=labels\n",
    "        self.bow_dicts=np.array([defaultdict(lambda:0) for index in range(self.classes.shape[0])])\n",
    "        \n",
    "        #only convert to numpy arrays if initially not passed as numpy arrays - else its a useless recomputation\n",
    "        \n",
    "        if not isinstance(self.examples,np.ndarray): self.examples=np.array(self.examples)\n",
    "        if not isinstance(self.labels,np.ndarray): self.labels=np.array(self.labels)\n",
    "            \n",
    "        #constructing BoW for each category\n",
    "        for cat_index,cat in enumerate(self.classes):\n",
    "          \n",
    "            all_cat_examples=self.examples[self.labels==cat] #filter all examples of category == cat\n",
    "            \n",
    "            #get examples preprocessed\n",
    "            \n",
    "            cleaned_examples=[preprocess_string(cat_example) for cat_example in all_cat_examples]\n",
    "            \n",
    "            cleaned_examples=pd.DataFrame(data=cleaned_examples)\n",
    "            \n",
    "            #now costruct BoW of this particular category\n",
    "            np.apply_along_axis(self.addToBow,1,cleaned_examples,cat_index)\n",
    "            \n",
    "                \n",
    "        ###################################################################################################\n",
    "        \n",
    "        '''\n",
    "            Although we are done with the training of Naive Bayes Model BUT!!!!!!\n",
    "            ------------------------------------------------------------------------------------\n",
    "            Remember The Test Time Forumla ? : {for each word w [ count(w|c)+1 ] / [ count(c) + |V| + 1 ] } * p(c)\n",
    "            ------------------------------------------------------------------------------------\n",
    "            \n",
    "            We are done with constructing of BoW for each category. But we need to precompute a few \n",
    "            other calculations at training time too:\n",
    "            1. prior probability of each class - p(c)\n",
    "            2. vocabulary |V| \n",
    "            3. denominator value of each class - [ count(c) + |V| + 1 ] \n",
    "            \n",
    "            Reason for doing this precomputing calculations stuff ???\n",
    "            ---------------------\n",
    "            We can do all these 3 calculations at test time too BUT doing so means to re-compute these \n",
    "            again and again every time the test function will be called - this would significantly\n",
    "            increase the computation time especially when we have a lot of test examples to classify!!!).  \n",
    "            And moreover, it doensot make sense to repeatedly compute the same thing - \n",
    "            why do extra computations ???\n",
    "            So we will precompute all of them & use them during test time to speed up predictions.\n",
    "            \n",
    "        '''\n",
    "        \n",
    "        ###################################################################################################\n",
    "      \n",
    "        prob_classes=np.empty(self.classes.shape[0])\n",
    "        all_words=[]\n",
    "        cat_word_counts=np.empty(self.classes.shape[0])\n",
    "        for cat_index,cat in enumerate(self.classes):\n",
    "           \n",
    "            #Calculating prior probability p(c) for each class\n",
    "            prob_classes[cat_index]=np.sum(self.labels==cat)/float(self.labels.shape[0]) \n",
    "            \n",
    "            #Calculating total counts of all the words of each class \n",
    "            count=list(self.bow_dicts[cat_index].values())\n",
    "            cat_word_counts[cat_index]=np.sum(np.array(list(self.bow_dicts[cat_index].values())))+1 # |v| is remaining to be added\n",
    "            \n",
    "            #get all words of this category                                \n",
    "            all_words+=self.bow_dicts[cat_index].keys()\n",
    "                                                     \n",
    "        \n",
    "        #combine all words of every category & make them unique to get vocabulary -V- of entire training set\n",
    "        \n",
    "        self.vocab=np.unique(np.array(all_words))\n",
    "        self.vocab_length=self.vocab.shape[0]\n",
    "                                  \n",
    "        #computing denominator value                                      \n",
    "        denoms=np.array([cat_word_counts[cat_index]+self.vocab_length+1 for cat_index,cat in enumerate(self.classes)])                                                                          \n",
    "      \n",
    "        '''\n",
    "            Now that we have everything precomputed as well, its better to organize everything in a tuple \n",
    "            rather than to have a separate list for every thing.\n",
    "            \n",
    "            Every element of self.cats_info has a tuple of values\n",
    "            Each tuple has a dict at index 0, prior probability at index 1, denominator value at index 2\n",
    "        '''\n",
    "        \n",
    "        self.cats_info=[(self.bow_dicts[cat_index],prob_classes[cat_index],denoms[cat_index]) for cat_index,cat in enumerate(self.classes)]                               \n",
    "        self.cats_info=np.array(self.cats_info)                                 \n",
    "                                              \n",
    "                                              \n",
    "    def getExampleProb(self,test_example):                                \n",
    "        \n",
    "        '''\n",
    "            Parameters:\n",
    "            -----------\n",
    "            1. a single test example \n",
    "\n",
    "            What the function does?\n",
    "            -----------------------\n",
    "            Function that estimates posterior probability of the given test example\n",
    "\n",
    "            Returns:\n",
    "            ---------\n",
    "            probability of test example in ALL CLASSES\n",
    "        '''                                      \n",
    "                                              \n",
    "        likelihood_prob=np.zeros(self.classes.shape[0]) #to store probability w.r.t each class\n",
    "        \n",
    "        #finding probability w.r.t each class of the given test example\n",
    "        for cat_index,cat in enumerate(self.classes): \n",
    "                             \n",
    "            for test_token in test_example.split(): #split the test example and get p of each test word\n",
    "                \n",
    "                ####################################################################################\n",
    "                                              \n",
    "                #This loop computes : for each word w [ count(w|c)+1 ] / [ count(c) + |V| + 1 ]                               \n",
    "                                              \n",
    "                ####################################################################################                              \n",
    "                \n",
    "                #get total count of this test token from it's respective training dict to get numerator value                           \n",
    "                test_token_counts=self.cats_info[cat_index][0].get(test_token,0)+1\n",
    "                \n",
    "                #now get likelihood of this test_token word                              \n",
    "                test_token_prob=test_token_counts/float(self.cats_info[cat_index][2])                              \n",
    "                \n",
    "                #remember why taking log? To prevent underflow!\n",
    "                likelihood_prob[cat_index]+=np.log(test_token_prob)\n",
    "                                              \n",
    "        # we have likelihood estimate of the given example against every class but we need posterior probility\n",
    "        post_prob=np.empty(self.classes.shape[0])\n",
    "        for cat_index,cat in enumerate(self.classes):\n",
    "            post_prob[cat_index]=likelihood_prob[cat_index]+np.log(self.cats_info[cat_index][1])                                  \n",
    "      \n",
    "        return post_prob\n",
    "    \n",
    "   \n",
    "    def test(self,test_set):\n",
    "      \n",
    "        '''\n",
    "            Parameters:\n",
    "            -----------\n",
    "            1. A complete test set of shape (m,)\n",
    "            \n",
    "\n",
    "            What the function does?\n",
    "            -----------------------\n",
    "            Determines probability of each test example against all classes and predicts the label\n",
    "            against which the class probability is maximum\n",
    "\n",
    "            Returns:\n",
    "            ---------\n",
    "            Predictions of test examples - A single prediction against every test example\n",
    "        '''       \n",
    "       \n",
    "        predictions=[] #to store prediction of each test example\n",
    "        for example in test_set: \n",
    "                                              \n",
    "            #preprocess the test example the same way we did for training set exampels                                  \n",
    "            cleaned_example=preprocess_string(example) \n",
    "             \n",
    "            #simply get the posterior probability of every example                                  \n",
    "            post_prob=self.getExampleProb(cleaned_example) #get prob of this example for both classes\n",
    "            \n",
    "            #simply pick the max value and map against self.classes!\n",
    "            predictions.append(self.classes[np.argmax(post_prob)])\n",
    "                \n",
    "        return np.array(predictions) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# That's it!!! Let's Move to Training! ⛸⛸⛸"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will just load a dataset from sklearn - but we are still coding NB from scratch!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Number of Training Examples:  2257\n",
      "Total Number of Training Labels:  2257\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\"\"\" \n",
    "just so you know - fetch_20newsgroups is a dataset that has 20 categories but we will restrict the categories\n",
    "to 4 for the time being \n",
    "\"\"\"\n",
    "categories=['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med'] \n",
    "newsgroups_train=fetch_20newsgroups(subset='train',categories=categories)\n",
    "\n",
    "\"\"\"\n",
    "    It's not a problem at all if you didnt understand this block of code - You should just know that some\n",
    "    training data is being loaded where training examples are saved in train_data and train labels are \n",
    "    saved in train_labels\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "train_data=newsgroups_train.data #getting all trainign examples\n",
    "train_labels=newsgroups_train.target #getting training labels\n",
    "print (\"Total Number of Training Examples: \",len(train_data))\n",
    "print (\"Total Number of Training Labels: \",len(train_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------- Dataset Categories -------------- \n",
      "['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian']\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "print (\"------------------- Dataset Categories -------------- \") \n",
    "pprint(list(newsgroups_train.target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# If you are curious to know what training data actually looks like .....  🤔 \n",
    "<br> Training Examples : <br>\n",
    "    The 20 newsgroups dataset comprises around 18000 newsgroups posts on 20 topics \n",
    "    \n",
    "<br> Training Labels : <br>\n",
    "    Training Labels ['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian'] are changed to their\n",
    "    corresponding integer forms. If you don't get this - just accept that Labels are in integer form"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Training Examples</th>\n",
       "      <th>Training Labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier)\\nSubject: Converting images to HP LaserJet III?\\nNntp-Posting-Host: hampton\\nOrganization: The City University\\nLines: 14\\n\\nDoes anyone know of a good way (standard PC application/PD utility) to\\nconvert ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: ani@ms.uky.edu (Aniruddha B. Deglurkar)\\nSubject: help: Splitting a trimming region along a mesh \\nOrganization: University Of Kentucky, Dept. of Math Sciences\\nLines: 28\\n\\n\\n\\n\\tHi,\\n\\n\\tI have a problem, I hope some of the 'gurus' can he...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: djohnson@cs.ucsd.edu (Darin Johnson)\\nSubject: Re: harrassed at work, could use some prayers\\nOrganization: =CSE Dept., U.C. San Diego\\nLines: 63\\n\\n(Well, I'll email also, but this may apply to other people, so\\nI'll post also.)\\n\\n&gt;I've b...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: s0612596@let.rug.nl (M.M. Zwart)\\nSubject: catholic church poland\\nOrganization: Faculteit der Letteren, Rijksuniversiteit Groningen, NL\\nLines: 10\\n\\nHello,\\n\\nI'm writing a paper on the role of the catholic church in Poland after 1989. \\n...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From: stanly@grok11.columbiasc.ncr.com (stanly)\\nSubject: Re: Elder Brother\\nOrganization: NCR Corp., Columbia SC\\nLines: 15\\n\\nIn article &lt;Apr.8.00.57.41.1993.28246@athos.rutgers.edu&gt; REXLEX@fnal.gov writes:\\n&gt;In article &lt;Apr.7.01.56.56.1993.228...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                           Training Examples  \\\n",
       "0  From: sd345@city.ac.uk (Michael Collier)\\nSubject: Converting images to HP LaserJet III?\\nNntp-Posting-Host: hampton\\nOrganization: The City University\\nLines: 14\\n\\nDoes anyone know of a good way (standard PC application/PD utility) to\\nconvert ...   \n",
       "1  From: ani@ms.uky.edu (Aniruddha B. Deglurkar)\\nSubject: help: Splitting a trimming region along a mesh \\nOrganization: University Of Kentucky, Dept. of Math Sciences\\nLines: 28\\n\\n\\n\\n\\tHi,\\n\\n\\tI have a problem, I hope some of the 'gurus' can he...   \n",
       "2  From: djohnson@cs.ucsd.edu (Darin Johnson)\\nSubject: Re: harrassed at work, could use some prayers\\nOrganization: =CSE Dept., U.C. San Diego\\nLines: 63\\n\\n(Well, I'll email also, but this may apply to other people, so\\nI'll post also.)\\n\\n>I've b...   \n",
       "3  From: s0612596@let.rug.nl (M.M. Zwart)\\nSubject: catholic church poland\\nOrganization: Faculteit der Letteren, Rijksuniversiteit Groningen, NL\\nLines: 10\\n\\nHello,\\n\\nI'm writing a paper on the role of the catholic church in Poland after 1989. \\n...   \n",
       "4  From: stanly@grok11.columbiasc.ncr.com (stanly)\\nSubject: Re: Elder Brother\\nOrganization: NCR Corp., Columbia SC\\nLines: 15\\n\\nIn article <Apr.8.00.57.41.1993.28246@athos.rutgers.edu> REXLEX@fnal.gov writes:\\n>In article <Apr.7.01.56.56.1993.228...   \n",
       "\n",
       "  Training Labels  \n",
       "0               1  \n",
       "1               1  \n",
       "2               3  \n",
       "3               3  \n",
       "4               3  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.options.display.max_colwidth=250\n",
    "pd.DataFrame(data=np.column_stack([train_data,train_labels]),columns=[\"Training Examples\",\"Training Labels\"]).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Woohoo! Let's actually begin Training .... 🏋🏋🏋"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------- Training In Progress --------------------\n",
      "----------------- Training Completed ---------------------\n"
     ]
    }
   ],
   "source": [
    "nb=NaiveBayes(np.unique(train_labels)) #instantiate a NB class object\n",
    "\n",
    "print (\"---------------- Training In Progress --------------------\")\n",
    " \n",
    "nb.train(train_data,train_labels) #start tarining by calling the train function\n",
    "\n",
    "print ('----------------- Training Completed ---------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So Now That We Have Trained NB Model - Let's Move to Testing! 🏄🏽🏄🏽🏄🏽"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Test Examples:  1502\n",
      "Number of Test Labels:  1502\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "    Again - it's not a problem at all if you didnt understand this block of code - You should just know that some\n",
    "    test data is being loaded where test examples are saved in test_data and test labels are saved in test_labels\n",
    "\n",
    "\"\"\"\n",
    "newsgroups_test=fetch_20newsgroups(subset='test',categories=categories) #loading test data\n",
    "test_data=newsgroups_test.data #get test set examples\n",
    "test_labels=newsgroups_test.target #get test set labels\n",
    "print (\"Number of Test Examples: \",len(test_data))\n",
    "print (\"Number of Test Labels: \",len(test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's Test on Above Loaded Test Examples Using the Trained NB Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Set Examples:  1502\n",
      "Test Set Accuracy:  93.8748335553 %\n"
     ]
    }
   ],
   "source": [
    "pclasses=nb.test(test_data) #get predcitions for test set\n",
    "\n",
    "#check how many predcitions actually match original test labels\n",
    "test_acc=np.sum(pclasses==test_labels)/float(test_labels.shape[0]) \n",
    "\n",
    "print (\"Test Set Examples: \",test_labels.shape[0])\n",
    "print (\"Test Set Accuracy: \",test_acc*100,\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wow! Pretty Good Accuracy of ~ 93% \n",
    "### See now you realise NB is not soooooo Naïve 👍👍👍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plus, as I said, the code we have written is generic! <br> So let's use the same code on a different dataset and with different class labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have taken this dataset from Kaggle : https://www.kaggle.com/c/word2vec-nlp-tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "training_set=pd.read_csv('./data/labeledTrainData.tsv',sep='\\t') # reading the training data-set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's see what the dataset looks like? 🤔 🤔 🤔 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It has movie reviews and their corresponding sentiment labels...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5814_8</td>\n",
       "      <td>1</td>\n",
       "      <td>With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2381_9</td>\n",
       "      <td>1</td>\n",
       "      <td>\\The Classic War of the Worlds\\\" by Timothy Hines is a very entertaining film that obviously goes to great effort and lengths to faithfully recreate H. G. Wells' classic book. Mr. Hines succeeds in doing so. I, and those who watched his film with...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7759_3</td>\n",
       "      <td>0</td>\n",
       "      <td>The film starts with a manager (Nicholas Bell) giving welcome investors (Robert Carradine) to Primal Park . A secret project mutating a primal animal using fossilized DNA, like ¨Jurassik Park¨, and some scientists resurrect one of nature's most f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3630_4</td>\n",
       "      <td>0</td>\n",
       "      <td>It must be assumed that those who praised this film (\\the greatest filmed opera ever,\\\" didn't I read somewhere?) either don't care for opera, don't care for Wagner, or don't care about anything except their desire to appear Cultured. Either as a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9495_8</td>\n",
       "      <td>1</td>\n",
       "      <td>Superbly trashy and wondrously unpretentious 80's exploitation, hooray! The pre-credits opening sequences somewhat give the false impression that we're dealing with a serious and harrowing drama, but you need not fear because barely ten minutes l...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id  sentiment  \\\n",
       "0  5814_8          1   \n",
       "1  2381_9          1   \n",
       "2  7759_3          0   \n",
       "3  3630_4          0   \n",
       "4  9495_8          1   \n",
       "\n",
       "                                                                                                                                                                                                                                                      review  \n",
       "0  With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i th...  \n",
       "1  \\The Classic War of the Worlds\\\" by Timothy Hines is a very entertaining film that obviously goes to great effort and lengths to faithfully recreate H. G. Wells' classic book. Mr. Hines succeeds in doing so. I, and those who watched his film with...  \n",
       "2  The film starts with a manager (Nicholas Bell) giving welcome investors (Robert Carradine) to Primal Park . A secret project mutating a primal animal using fossilized DNA, like ¨Jurassik Park¨, and some scientists resurrect one of nature's most f...  \n",
       "3  It must be assumed that those who praised this film (\\the greatest filmed opera ever,\\\" didn't I read somewhere?) either don't care for opera, don't care for Wagner, or don't care about anything except their desire to appear Cultured. Either as a...  \n",
       "4  Superbly trashy and wondrously unpretentious 80's exploitation, hooray! The pre-credits opening sequences somewhat give the false impression that we're dealing with a serious and harrowing drama, but you need not fear because barely ten minutes l...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_set.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique Classes:  [0 1]\n",
      "Total Number of Training Examples:  (25000,)\n"
     ]
    }
   ],
   "source": [
    "#getting training set examples labels\n",
    "y_train=training_set['sentiment'].values\n",
    "x_train=training_set['review'].values\n",
    "print (\"Unique Classes: \",np.unique(y_train))\n",
    "print (\"Total Number of Training Examples: \",x_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "\"\"\"\n",
    "    Again - it's not a problem at all if you didnt understand this block of code - You should just know that some\n",
    "    train & test data is being loaded and saved in their corresponding variables\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "train_data,test_data,train_labels,test_labels=train_test_split(x_train,y_train,shuffle=True,test_size=0.25,random_state=42,stratify=y_train)\n",
    "classes=np.unique(train_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's train a NB classifier on this dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------Training In Progress------------------------\n",
      "Training Examples:  (18750,)\n",
      "------------------------Training Completed!\n",
      "Test Set Examples:  6250\n",
      "Test Set Accuracy:  0.84192\n"
     ]
    }
   ],
   "source": [
    "# Training phase....\n",
    "\n",
    "nb=NaiveBayes(classes)\n",
    "print (\"------------------Training In Progress------------------------\")\n",
    "print (\"Training Examples: \",train_data.shape)\n",
    "nb.train(train_data,train_labels)\n",
    "print ('------------------------Training Completed!')\n",
    "\n",
    "# Testing phase \n",
    "\n",
    "pclasses=nb.test(test_data)\n",
    "test_acc=np.sum(pclasses==test_labels)/float(test_labels.shape[0])\n",
    "print (\"Test Set Examples: \",test_labels.shape[0])\n",
    "print (\"Test Set Accuracy: \",test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's test on Kaggle test set and uplaoding our predictions on kaggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predcitions Generated and saved to naive_bayes_model_take1.csv\n"
     ]
    }
   ],
   "source": [
    "# Loading the kaggle test dataset\n",
    "test=pd.read_csv('./data/testData.tsv',sep='\\t')\n",
    "Xtest=test.review.values\n",
    "\n",
    "#generating predictions....\n",
    "pclasses=nb.test(Xtest) \n",
    "\n",
    "#writing results to csv to uplaoding on kaggle!\n",
    "kaggle_df=pd.DataFrame(data=np.column_stack([test[\"id\"].values,pclasses]),columns=[\"id\",\"sentiment\"])\n",
    "kaggle_df.to_csv(\"./naive_bayes_model_take1.csv\",index=False)\n",
    "print ('Predcitions Generated and saved to naive_bayes_model_take1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A screen shot of kaggle results "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "![](take1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}