msan621_ml1/ML-Lecture7.2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('/Users/tlee010/Desktop/github_repos/fastai/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai.nlp import *\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "# from torchtext import vocab, data, datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.0058509219, 0.99243325)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from fastai.io import *\n",
    "path = './'\n",
    "FILENAME='mnist.pkl.gz'\n",
    "\n",
    "def load_mnist(filename):\n",
    "    return pickle.load(gzip.open(filename, 'rb'), encoding='latin-1')\n",
    "\n",
    "((x, y), (x_valid, y_valid), _) = load_mnist(path+FILENAME)\n",
    "mean = x.mean()\n",
    "std = x.std()\n",
    "x=(x-mean)/std\n",
    "mean, std, x.mean(), x.std()\n",
    "x_valid = (x_valid-mean)/std\n",
    "x_valid.mean(), x_valid.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "md = ImageClassifierData.from_arrays(path, (x,y), (x_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### if you are missing the en library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!python -m spacy download en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Auto Encoder\n",
    "\n",
    "When you have unlabeled data. How can you create a NN that makes a features for you if you dont have the dependent variable. Lets have our input be the output as well. \n",
    "\n",
    "https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44629\n",
    "\n",
    "Reconstruct the details of this insurance policy. \n",
    "\n",
    "Less activations than inputs. Will compress features down into fewer features and then expand it back out. Great way for making embeddings and features even when you don't have labels\n",
    "\n",
    "#### Denoising the Auto Encoder \n",
    "\n",
    "Start with a noisy version and then come out with the clean dataset. The 15% were randomly replaced with another row. The dependent variable would stay the same. Can show a lot of different rows. Gave him features (first layer weights), then took the NN and trained it on the claims data.\n",
    "\n",
    "2 Models would have been enough to win.\n",
    "\n",
    "#### He did no feature engineering \n",
    "\n",
    "#### Embeddings\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From last week"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterators & Streaming\n",
    "\n",
    "Its a object that we call next on. We can pull different pieces, ordered or randomized that can be pulled for mini-batches.\n",
    "\n",
    "<img src='http://corochann.com/wp-content/uploads/2017/06/ParallelSequentialIterator.png' style='width:600px'/>\n",
    "\n",
    "Pytorch is designed to handle this concept too, of predicting a stream of data (or the next item). `fastai` is also designed around this concept of iterators. `DataLoader` is created with the same concept, but with multiprocessing, which is multi threaded applications ( so you can use your entire processor) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variable - keeps track of computations being made to a tensor. It's a wrapper\n",
    "\n",
    "#### pytorch - autograd\n",
    "\n",
    "http://pytorch.org/docs/master/autograd.html\n",
    "\n",
    "If we call `back` it will calculate the gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_weights(*dims): return nn.Parameter(torch.randn(*dims)/dims[0])\n",
    "\n",
    "class LogReg(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.l1_w = get_weights(28*28, 10)  # Layer 1 weights\n",
    "        self.l1_b = get_weights(10)         # Layer 1 bias\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = torch.matmul(x, self.l1_w) + self.l1_b  # Linear Layer\n",
    "        x = torch.log(torch.exp(x)/(1 + torch.exp(x).sum(dim=0)))        # Non-linear (LogSoftmax) Layer\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(x, y):\n",
    "    y_pred = to_np(net2.forward(Variable(x)))#.cuda())))\n",
    "    return np.sum(y_pred.argmax(axis=1) == to_np(y))/len(y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A re-write of `fit`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.910429936306\n"
     ]
    }
   ],
   "source": [
    "net2 = LogReg()#.cuda()\n",
    "loss=nn.NLLLoss()\n",
    "learning_rate = 1e-3\n",
    "optimizer=optim.Adam(net2.parameters(), lr=learning_rate)\n",
    "\n",
    "for epoch in range(1):\n",
    "    losses=[]\n",
    "    dl = iter(md.trn_dl)\n",
    "    for t in range(len(dl)):\n",
    "        # Forward pass: compute predicted y by passing x to the model.\n",
    "        xt, yt = next(dl)\n",
    "        y_pred = net2.forward(Variable(xt))#.cuda())\n",
    "\n",
    "        # Compute and print loss.\n",
    "        l = loss(y_pred, Variable(yt))#.cuda())\n",
    "        losses.append(l)\n",
    "\n",
    "        # Before the backward pass, use the optimizer object to zero all of the\n",
    "        # gradients for the variables it will update (which are the learnable weights\n",
    "        # of the model)\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # Backward pass: compute gradient of the loss with respect to model\n",
    "        # parameters\n",
    "        l.backward()\n",
    "        # print(loss.data)\n",
    "\n",
    "        # Calling the step function on an Optimizer makes an update to its\n",
    "        # parameters\n",
    "        optimizer.step()\n",
    "    \n",
    "    val_dl = iter(md.val_dl)\n",
    "    val_scores = [score(*next(val_dl)) for i in range(len(val_dl))]\n",
    "    print(np.mean(val_scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Within a `fit` lets rewrite some of the functions (`gradient` and `step`)\n",
    "\n",
    "```python\n",
    "optimizer.zero_grad()\n",
    "\n",
    "optimizer.step()\n",
    "```\n",
    "\n",
    "This code will be replaced with the following:\n",
    "\n",
    "```python\n",
    "        if w.grad is not None:\n",
    "            w.grad.data.zero_()\n",
    "            b.grad.data.zero_()   \n",
    "\n",
    "        # Backward pass: compute gradient of the loss with respect to model parameters\n",
    "        l.backward()\n",
    "        w.data -= w.grad.data * lr\n",
    "        b.data -= b.grad.data * lr\n",
    "```\n",
    "\n",
    "All gradients have to be added together to get that gradient for that parameter.\n",
    "\n",
    "- `w` is the variable we set before\n",
    "- `.grad` referring to the gradient part of the variable\n",
    "- `.data.zero_` will zero the gradients, note this is the initializing\n",
    "\n",
    "```python\n",
    "    w.grad.data.zero_()\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.905155254777\n"
     ]
    }
   ],
   "source": [
    "net2 = LogReg()#.cuda()\n",
    "loss_fn=nn.NLLLoss()\n",
    "lr = 1e-2\n",
    "w,b = net2.l1_w,net2.l1_b\n",
    "\n",
    "for epoch in range(1):\n",
    "    losses=[]\n",
    "    dl = iter(md.trn_dl)\n",
    "    for t in range(len(dl)):\n",
    "        # Forward pass: compute predicted y by passing x to the model.\n",
    "        xt, yt = next(dl)\n",
    "        y_pred = net2.forward(Variable(xt))#.cuda())\n",
    "\n",
    "        # Compute and print loss.\n",
    "        l = loss(y_pred, Variable(yt))#.cuda())\n",
    "        losses.append(loss)\n",
    "\n",
    "        # Before the backward pass, zero the gradients for all of the parameters\n",
    "        if w.grad is not None:\n",
    "            w.grad.data.zero_()\n",
    "            b.grad.data.zero_()   \n",
    "\n",
    "        # Backward pass: compute gradient of the loss with respect to model parameters\n",
    "        l.backward()\n",
    "        w.data -= w.grad.data * lr\n",
    "        b.data -= b.grad.data * lr\n",
    "    \n",
    "    val_dl = iter(md.val_dl)\n",
    "    val_scores = [score(*next(val_dl)) for i in range(len(val_dl))]\n",
    "    print(np.mean(val_scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going backwards\n",
    "\n",
    "**Momentum** `fastai` can speed up the iteration rate and looping. \n",
    "\n",
    "- `optimizer.step()` - updates \n",
    "- `optimizer.zero_grad()` -\n",
    "- `fit` - is the loop that cycles through\n",
    "- `nn.Softmax` - does the softmax calculation\n",
    "- `nn.Linear` - does the linear multiplication\n",
    "\n",
    "### A word on learning rate\n",
    "\n",
    "Sometimes if your learning rate is too large, it gets hard to converge near the later iterations\n",
    "\n",
    "<img src='https://qph.ec.quoracdn.net/main-qimg-f3972c89625c0451f0ea92a4c0ea0728' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### will pull out the parameters of the model\n",
    "\n",
    "How many weights vs. the size of the data. Since our dataset is small, and our weights are very large, chances are that we will dramatically overfit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-25-112692b70a57>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-25-112692b70a57>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    t = [ 0.numel() for o in net2.parameters]\u001b[0m\n\u001b[0m                ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "t = [ 0.numel() for o in net.parameters]\n",
    "t, sum(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regularization \n",
    "\n",
    "Add new terms to the loss function. If we add L1 and L2 norms of the weights to the loss functions, it incentives the coefficients to be zlose to zero. See the extra term below.\n",
    "\n",
    "$$ loss =\\frac{1}{n} \\sum{(wX-y)^2} + \\alpha \\sum{w^2} $$\n",
    "\n",
    "How to implement? \n",
    "\n",
    "1. Change the loss function\n",
    "2. Change the training loop to add the derivative adjustment <-- **weight decay** (not the same as momentum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = nn.Sequential(\n",
    "    nn.Linear(28*28, 100),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(100, 10),\n",
    "    nn.LogSoftmax()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss=nn.NLLLoss()\n",
    "metrics=[accuracy]\n",
    "opt=optim.SGD(net.parameters(), 1e-1, momentum=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0f063b825ccf42cd85b3207990329c0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=5), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.33271  0.28169  0.92725]                         \n",
      "[ 1.       0.22736  0.24474  0.94715]                         \n",
      "[ 2.       0.23459  0.26795  0.94118]                         \n",
      "[ 3.       0.21125  0.27423  0.94805]                         \n",
      "[ 4.       0.21197  0.32224  0.9384 ]                         \n",
      "\n"
     ]
    }
   ],
   "source": [
    "fit(net, md, epochs=5, crit=loss, opt=opt, metrics=metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20be2ca45d3f41cbab36afc018455f1a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=5), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       1.11321  1.13041  0.62271]                        \n",
      "[ 1.       1.08937  1.06073  0.70482]                        \n",
      "[ 2.       1.04438  1.35849  0.46795]                        \n",
      "[ 3.       1.08236  1.12079  0.63515]                        \n",
      "[ 4.       1.08296  1.20425  0.62988]                        \n",
      "\n"
     ]
    }
   ],
   "source": [
    "opt=optim.SGD(net.parameters(), 1e-1, momentum=0.9, weight_decay=0.3)\n",
    "fit(net, md, epochs=5, crit=loss, opt=opt, metrics=metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On to NLP! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2017-11-30 14:16:02--  http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
      "Resolving ai.stanford.edu... 171.64.68.10\n",
      "Connecting to ai.stanford.edu|171.64.68.10|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 84125825 (80M) [application/x-gzip]\n",
      "Saving to: ‘aclImdb_v1.tar.gz’\n",
      "\n",
      "aclImdb_v1.tar.gz   100%[===================>]  80.23M  10.6MB/s    in 10s     \n",
      "\n",
      "2017-11-30 14:16:12 (7.80 MB/s) - ‘aclImdb_v1.tar.gz’ saved [84125825/84125825]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gunzip aclImdb_v1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#!tar -xvf aclImdb_v1.tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH='aclImdb/'\n",
    "names = ['neg','pos']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn,trn_y = texts_from_folders(f'{PATH}train',names)\n",
    "val,val_y = texts_from_folders(f'{PATH}test',names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def texts_from_folders(src, names):\n",
    "    texts,labels = [],[]\n",
    "    for idx,name in enumerate(names):\n",
    "        path = os.path.join(src, name)\n",
    "        for fname in sorted(os.listdir(path)):\n",
    "            fpath = os.path.join(path, fname)\n",
    "            texts.append(open(fpath).read())\n",
    "            labels.append(idx)\n",
    "    return texts,np.array(labels)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "??texts_from_folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Story of a man who has unnatural feelings for a pig. Starts out with a opening scene that is a terrific example of absurd comedy. A formal orchestra audience is turned into an insane, violent mob by the crazy chantings of it's singers. Unfortunately it stays absurd the WHOLE time with no general narrative eventually making it just too off putting. Even those from the era should be turned off. The cryptic dialogue would make Shakespeare seem easy to a third grader. On a technical level it's better than you might think with some good cinematography by future great Vilmos Zsigmond. Future stars Sally Kirkland and Frederic Forrest can be seen briefly.\""
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First we will throw away the order of words --> Bag of words\n",
    "\n",
    "<img src='https://qph.ec.quoracdn.net/main-qimg-c47060d2f02439a44795e2fbcf2ca347-c' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's use sklearn to convert corpus to bag of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "veczr = CountVectorizer(tokenizer=tokenize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key Concepts\n",
    "\n",
    "**Vocabulary** - need to find the unique terms throughout the entire corpus\n",
    "\n",
    "**Term Document Matrix**\n",
    "\n",
    "- columns - vocab or words\n",
    "- rows -  different documents\n",
    "- cells - count of occurance\n",
    "\n",
    "#### Sentiment Approach: bays rule\n",
    "\n",
    "The probability that a document is from class 1 given that the document is from class 0. We will use a conditional probability as follows below\n",
    "\n",
    "$$ \\frac{P(C_1 | d)}{P(C_0 | d}) = \\frac {P(d|C_1)P(C_1)}{P(d)} = \\frac {P(d|C_1)P(C_1)}{P(d|C_0)P(C_0)}$$\n",
    "\n",
    "How do we calculate? \n",
    "\n",
    "1. Gather the documents of one class together\n",
    "2. Create profile made up of the words, and their probabilities\n",
    "3. Note: we add a dataset that assumes every word occurs at least one in the document\n",
    "4. We should have a probability per word, for each class\n",
    "5. For each doucment we can multiple the coefficients and that will give us a $ P(C_1 | d) $ and $ P(C_0 | d) $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### we will learn the word features from the training set and split. Then use the same framework to split the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_term_doc = veczr.fit_transform(trn)\n",
    "val_term_doc = veczr.transform(val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We now have 25000 words, with 75132 docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<25000x75132 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 3749745 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sparse Matrix \n",
    "\n",
    "Special format when most of your matrix is filled with zeros. This assumes that most of the matrix is zero, and only keep track of the elements that are non-zero."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The first feature has 93 items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x75132 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 93 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What are the features (words) ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aussie', 'aussies', 'austen', 'austeniana', 'austens']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = veczr.get_feature_names(); vocab[5000:5005]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets look at our 93 words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "w0 = set([o.lower() for o in trn[0].split(' ')]); #w0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Look up a specific word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1297"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "veczr.vocabulary_['absurd']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How many times does this word show up in document 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_term_doc[0,1297]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_term_doc[0,5000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes\n",
    "\n",
    "We define the **log-count ratio** $r$ for each word $f$:\n",
    "\n",
    "$r = \\log \\frac{\\text{ratio of feature $f$ in positive documents}}{\\text{ratio of feature $f$ in negative documents}}$\n",
    "\n",
    "where ratio of feature $f$ in positive documents is the number of times a positive document has a feature divided by the number of positive documents.\n",
    "\n",
    "`p = x[y==1].sum(0)+1` - Number of times it shows in positive corpus\n",
    "\n",
    "`p = x[y==0].sum(0)+1` - Number of times it shows in negative corpus\n",
    "\n",
    "`r = np.log((p/p.sum())/(q/q.sum()))` - summing logs is a lot easier to do, otherwise you might run into float errors\n",
    "\n",
    "`b = np.log(len(p)/len(q))` - this is our naive bays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=trn_term_doc\n",
    "y=trn_y\n",
    "\n",
    "p = x[y==1].sum(0)+1\n",
    "q = x[y==0].sum(0)+1\n",
    "r = np.log((p/p.sum())/(q/q.sum()))\n",
    "b = np.log(len(p)/len(q))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Here's our predictions from Naive Bayes ~ 80% positive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.80740000000000001"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_preds = val_term_doc @ r.T + b\n",
    "preds = pre_preds.T>0\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Binarized: instead of using the 0.66 for a word, why not just 1 if the word shows up.  83% chance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.82623999999999997"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_preds = val_term_doc.sign() @ r.T + b\n",
    "preds = pre_preds.T>0\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is how we can fit logistic regression where the features are the unigrams. So instead of using bayes, we set it up like a supervised problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.85648000000000002"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=1e8, dual=True)\n",
    "m.fit(x, y)\n",
    "preds = m.predict(val_term_doc)\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regularized version -> 0.0000001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.85528000000000004"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=1e8, dual=True)\n",
    "m.fit(trn_term_doc.sign(), y)\n",
    "preds = m.predict(val_term_doc.sign())\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regularized version -> 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.88271999999999995"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=0.1, dual=True)\n",
    "m.fit(x, y)\n",
    "preds = m.predict(val_term_doc)\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regularized version -> 0.1 with traing doc terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.88404000000000005"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=0.1, dual=True)\n",
    "m.fit(trn_term_doc.sign(), y)\n",
    "preds = m.predict(val_term_doc.sign())\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigram with NB features\n",
    "\n",
    "Instead of using words, we will use small phrases of words, or phrases. In this case we will use 3 word phrases, also referred to as trigrams\n",
    "\n",
    "<img src='https://www.researchgate.net/profile/Simon_Lin4/publication/271910222/figure/fig1/AS:295071070081024@1447361833118/Fig-1-An-example-of-unigrams-bigrams-trigrams-and-4-grams-extracted-from-the-clinical.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "veczr =  CountVectorizer(ngram_range=(1,3), tokenizer=tokenize, max_features=800000)\n",
    "trn_term_doc = veczr.fit_transform(trn)\n",
    "val_term_doc = veczr.transform(val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let regularization do the feature selection, just bound the overall space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 800000)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = veczr.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['by vast', 'by vengeance', 'by vengeance .', 'by vera', 'by vera miles']"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[200000:200005]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y=trn_y\n",
    "x=trn_term_doc.sign()\n",
    "val_x = val_term_doc.sign()\n",
    "p = x[y==1].sum(0)+1\n",
    "q = x[y==0].sum(0)+1\n",
    "r = np.log((p/p.sum())/(q/q.sum()))\n",
    "b = np.log(len(p)/len(q))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}