NativeBayes_Tf-Idf.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run Import_Library.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run text_functions.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run evaluation_metrics.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Introduction**\n",
    "\n",
    "- Term Frequency–Inverse Document Frequency (TF-IDF): Machine doesn't understand the text. We have to transform text into sparse matrix or term-document matrix. The term-document matrix then is a two-dimensional matrix whose rows are the terms and columns are the documents, so each entry (i, j) represents the frequency of term i in document j.\n",
    "\n",
    "- For each entry in the matrix, the term frequency measures the number of times that term i appears in document j, and the inverse document frequency measures the number of documents in the corpus which contain term i. \n",
    "\n",
    "- The tf-idf score is the product of these two metrics (tf*idf). So an entry's tf-idf score increases when term i appears frequently in document j, but decreases as the term appears in other documents. In another word, idf is a cross-document normalization, that puts less weight on common terms, and more weight on rare terms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data Preparation**\n",
    "- Convert values in predicted column to binary\n",
    "- Feature selection: len_text, digits, non_alpha_char, processed_text\n",
    "- Shuffle randomly the data with selected features in order to reduce the bias.\n",
    "- Split dataset into train and test subsets with the ratio 80:20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>len_text</th>\n",
       "      <th>digits</th>\n",
       "      <th>non_alpha_char</th>\n",
       "      <th>processed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "      <td>111</td>\n",
       "      <td>0</td>\n",
       "      <td>28</td>\n",
       "      <td>go jurong point crazy available bugis n great ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>ok lar joking wif u oni</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "      <td>155</td>\n",
       "      <td>25</td>\n",
       "      <td>33</td>\n",
       "      <td>free entry 2 wkly comp win fa cup final tkts 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>u dun say early hor u c already say</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>nah dont think go usf life around though</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                               text  len_text  digits  \\\n",
       "0   ham  Go until jurong point, crazy.. Available only ...       111       0   \n",
       "1   ham                      Ok lar... Joking wif u oni...        29       0   \n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...       155      25   \n",
       "3   ham  U dun say so early hor... U c already then say...        49       0   \n",
       "4   ham  Nah I don't think he goes to usf, he lives aro...        61       0   \n",
       "\n",
       "   non_alpha_char                                     processed_text  \n",
       "0              28  go jurong point crazy available bugis n great ...  \n",
       "1              11                            ok lar joking wif u oni  \n",
       "2              33  free entry 2 wkly comp win fa cup final tkts 2...  \n",
       "3              16                u dun say early hor u c already say  \n",
       "4              14           nah dont think go usf life around though  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from joblib import load\n",
    "\n",
    "data = load('data.lib')\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\nDense matrices store every entry in the matrix. Sparse matrices only store the nonzero entries. Sparse matrices don't have a lot of extra features, and some algorithms may not work for them. You use them when you need to work with matrices that would be too big for the computer to handle them, but they are mostly zero, so they compress easily.\\n\""
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Dense matrices store every entry in the matrix. Sparse matrices only store the nonzero entries. Sparse matrices don't have a lot of extra features, and some algorithms may not work for them. You use them when you need to work with matrices that would be too big for the computer to handle them, but they are mostly zero, so they compress easily.\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 123\n",
    "# Convert values in label column to binary: spam is 1 and ham is 0\n",
    "#data['label'] = np.where(data['label']=='spam', 1, 0)\n",
    "# Select relevant features for training the model\n",
    "features = ['len_text', 'digits', 'non_alpha_char', 'processed_text']\n",
    "# Feature variables\n",
    "X_text = data['processed_text'].values\n",
    "\n",
    "X_len_text = data['len_text'].values\n",
    "\n",
    "X_digits = data['digits'].values\n",
    "\n",
    "X_non_alpha_char = data['non_alpha_char'].values\n",
    "# Target variable\n",
    "Y = data['label'].values\n",
    "# Shuffle the training data and labels.\n",
    "myshuffle(seed, X_text)\n",
    "\n",
    "myshuffle(seed, X_len_text)\n",
    "\n",
    "myshuffle(seed, X_digits)\n",
    "\n",
    "myshuffle(seed, X_non_alpha_char)\n",
    "\n",
    "myshuffle(seed, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "print(type(X_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['len_text', 'digits', 'non_alpha_char', 'processed_text']\n",
    "target = 'label'\n",
    "\n",
    "x = data[features]\n",
    "y = data[target]\n",
    "\n",
    "x_train, x_val, y_train, y_val = train_test_split(x\n",
    "                                                  , y\n",
    "                                                  , shuffle = True\n",
    "                                                  , test_size = 0.2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>len_text</th>\n",
       "      <th>digits</th>\n",
       "      <th>non_alpha_char</th>\n",
       "      <th>processed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>4437</td>\n",
       "      <td>148</td>\n",
       "      <td>40</td>\n",
       "      <td>29</td>\n",
       "      <td>private 2003 account statement 07753741225 sho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4182</td>\n",
       "      <td>83</td>\n",
       "      <td>1</td>\n",
       "      <td>26</td>\n",
       "      <td>err cud im go 8pm havent get way contact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4485</td>\n",
       "      <td>146</td>\n",
       "      <td>24</td>\n",
       "      <td>27</td>\n",
       "      <td>win urgent mobile number award 2000 prize guar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3664</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>dad back ph</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5210</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>great office today</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      len_text  digits  non_alpha_char  \\\n",
       "4437       148      40              29   \n",
       "4182        83       1              26   \n",
       "4485       146      24              27   \n",
       "3664        23       0               6   \n",
       "5210        31       0               7   \n",
       "\n",
       "                                         processed_text  \n",
       "4437  private 2003 account statement 07753741225 sho...  \n",
       "4182           err cud im go 8pm havent get way contact  \n",
       "4485  win urgent mobile number award 2000 prize guar...  \n",
       "3664                                        dad back ph  \n",
       "5210                                 great office today  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4437    spam\n",
       "4182     ham\n",
       "4485    spam\n",
       "3664     ham\n",
       "5210     ham\n",
       "Name: label, dtype: object"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>len_text</th>\n",
       "      <th>digits</th>\n",
       "      <th>non_alpha_char</th>\n",
       "      <th>processed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>3266</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>happy sad one thing past good morning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>579</td>\n",
       "      <td>116</td>\n",
       "      <td>0</td>\n",
       "      <td>39</td>\n",
       "      <td>let pool money together buy bunch lotto ticket...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4680</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>im inside officestill fill formsdon know leave</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4117</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>izzit still rain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3835</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>hang brother family</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      len_text  digits  non_alpha_char  \\\n",
       "3266        74       0              25   \n",
       "579        116       0              39   \n",
       "4680        70       0              15   \n",
       "4117        25       0               5   \n",
       "3835        42       0               7   \n",
       "\n",
       "                                         processed_text  \n",
       "3266              happy sad one thing past good morning  \n",
       "579   let pool money together buy bunch lotto ticket...  \n",
       "4680     im inside officestill fill formsdon know leave  \n",
       "4117                                   izzit still rain  \n",
       "3835                                hang brother family  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_val.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3266    ham\n",
       "579     ham\n",
       "4680    ham\n",
       "4117    ham\n",
       "3835    ham\n",
       "Name: label, dtype: object"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_val.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need to convert the texts into numerical vectors:\n",
    "- Tokenization: Divide the texts into words or smaller sub-texts, which will enable good generalization of relationship between the texts and the labels. This determines the “vocabulary” of the dataset (set of unique tokens present in the data).\n",
    "- Vectorization: Define a good numerical measure to characterize these texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_text = x_train['processed_text']\n",
    "\n",
    "x_val_text = x_val['processed_text']\n",
    "\n",
    "x_train_tfidf, x_val_tfidf = ngram_vectorize(train_texts = x_train_text.values\n",
    "                                             , train_labels = y_train.values\n",
    "                                             , val_texts = x_val_text.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Add new features to sparse matrix built from the text of trainset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<4457x7699 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 57153 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_new = x_train_tfidf\n",
    "\n",
    "for feature in ['len_text', 'digits', 'non_alpha_char']:\n",
    "    x_train_new = add_feature(x_train_new, x_train[feature].values.reshape(1, -1))\n",
    "\n",
    "x_train_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Add new features to sparse matrix built from text of testset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1115x7699 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 12311 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_val_new = x_val_tfidf\n",
    "\n",
    "for feature in ['len_text', 'digits', 'non_alpha_char']:\n",
    "    x_val_new = add_feature(x_val_new, x_val[feature].values.reshape(1, -1))\n",
    "\n",
    "x_val_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "#(y_true, y_pred, normalize=True, sample_weight=None)[source]\n",
    "\n",
    "clf = GaussianNB()\n",
    "# A sparse matrix was passed, but dense data is required.\n",
    "clf.fit(x_train_new.toarray(), y_train)\n",
    "\n",
    "y_preds = clf.predict(x_val_new.toarray())\n",
    "\n",
    "round(accuracy_score(y_val, y_preds), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ham' 'spam']\n"
     ]
    }
   ],
   "source": [
    "print(clf.classes_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Evaluation Metrics**\n",
    "-  The accuracy will yield misleading results if the data set is unbalanced; that is, when the numbers of observations in different classes vary greatly. Confusion Matrix and Classification Report will help us to evaluate the performance for each class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1) Confusion Matrix**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an supervised machine learning algorithm.\n",
    "\n",
    "- Each row of the matrix represents the instances in a predicted class while each column represents the instances in an actual class. There are four ways to check if the predictions are right or wrong:\n",
    "\n",
    "    - 1) True Positive (TP): The class was positive and predicted positive.\n",
    "    - 2) True Negative (TN): The class was negative and predicted negative.\n",
    "    - 3) False Negative (FN): The class was positive but predicted negative\n",
    "    - 4) False Positive (FP) : The case was negative but predicted positive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Actual Ham</th>\n",
       "      <th>Actual Spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Predicted Ham</td>\n",
       "      <td>878</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Predicted Spam</td>\n",
       "      <td>13</td>\n",
       "      <td>140</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Actual Ham  Actual Spam\n",
       "Predicted Ham          878           84\n",
       "Predicted Spam          13          140"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_confusion_matrix =  make_confusion_matrix(confusion_matrix(y_val, y_preds)\n",
    "                                              , columns = ['Actual Ham', 'Actual Spam']\n",
    "                                              , index = ['Predicted Ham', 'Predicted Spam'])\n",
    "df_confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The diagonal elements:\n",
    "    - Show the number of correct classifications for each class: 878 and 140 for the classes spam and ham, respectively. \n",
    "    - The correction of prediction = True Positive (TP) + True Negative (TN) = 878 + 140 = 918\n",
    "\n",
    "- The off-diagonal elements: \n",
    "    - Provides the misclassifications for each class: 84 and 13 for classes spam and ham, respectively. \n",
    "    - The misclassification = False Negative (FN) + False Positive (FP) = 84 + 13 = 97"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2) Classification Report**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The report shows the main classification metrics precision, recall and f1-score on a per-class basis. The metrics are calculated by using true and false positives, true and false negatives. Positive and negative in this case are generic names for the predicted classes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Precision – What percent of your predictions were correct ?**\n",
    "\n",
    "- Precision is the ability of a classifier not to label an instance positive that is actually negative. For each class it is defined as the ratio of true positives to the sum of true and false positives.\n",
    "\n",
    "- Precision = Accuracy of positive predictions.\n",
    "\n",
    "- Precision = TP/(TP + FP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Recall – What percent of the positive cases did you catch ?** \n",
    "\n",
    "- Recall is the ability of a classifier to find all positive instances. For each class it is defined as the ratio of true positives to the sum of true positives (TP) and false negatives (FN).\n",
    "\n",
    "- Recall: Fraction of positives that were correctly identified.\n",
    "- Recall = TP/(TP+FN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**F1 score – What percent of positive predictions were correct?**\n",
    "\n",
    "- The F1 score is a weighted harmonic mean of precision and recall such that the best score is 1.0 and the worst is 0.0. Generally speaking, F1 scores are lower than accuracy measures as they embed precision and recall into their computation. As a rule of thumb, the weighted average of F1 should be used to compare classifier models, not global accuracy.\n",
    "\n",
    "- F1 Score = 2*(Recall * Precision) / (Recall + Precision)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         ham       0.99      0.91      0.95       962\n",
      "        spam       0.62      0.92      0.74       153\n",
      "\n",
      "    accuracy                           0.91      1115\n",
      "   macro avg       0.81      0.91      0.85      1115\n",
      "weighted avg       0.94      0.91      0.92      1115\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "# Thus in binary classification, \n",
    "# the count of true negatives is \n",
    "# , false negatives is \n",
    "#, true positives is  \n",
    "# and false positives is .\n",
    "print (classification_report(y_val , y_preds, target_names = ['ham', 'spam']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References**\n",
    "- 1) https://stats.stackexchange.com/questions/336455/fpr-false-positive-rate-vs-fdr-false-discovery-rate/340079#340079\n",
    "- 2) https://stats.stackexchange.com/questions/459994/how-to-interpret-the-confusion-matrix\n",
    "- 3) https://en.wikipedia.org/wiki/Type_I_and_type_II_errors#False_positive_and_false_negative_rates\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}