Sentiment_Analysis_biLSTM.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Sentiment_Analysis_biLSTM.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "qBu-ToMOtNJj"
      },
      "source": [
        "! pip install torchtext==0.8.1 "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1STWY9hT8cll"
      },
      "source": [
        "# Sentiment Analysis Using Bidirectional LSTM\n",
        "\n",
        "In this project, we'll be building an RNN-based learning model to detect sentiment (i.e. detect if a sentence is positive or negative) using PyTorch and TorchText. This will be done on movie reviews, using the [IMDb dataset](http://ai.stanford.edu/~amaas/data/sentiment/).\n",
        "\n",
        "We will use:\n",
        "- pre-trained word embeddings\n",
        "- different RNN architecture\n",
        "- bidirectional RNN\n",
        "- multi-layer RNN\n",
        "- regularization\n",
        "- optimizer\n",
        "\n",
        "But first, let's mount Google drive where you will be saving your model. Run the following code and go to the URL it displays. Copy authorization code from there, paste it here and press enter.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RLueVq9q3Iuw"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5cCrKM0z8clo"
      },
      "source": [
        "## Preparing Data\n",
        "\n",
        "One of the main concepts of TorchText is the `Field`. These define how your data should be processed. In our sentiment classification task the data consists of both the raw string of the review and the sentiment, either \"pos\" or \"neg\".\n",
        "\n",
        "We use the `TEXT` field to define how the review should be processed, and the `LABEL` field to process the sentiment.\n",
        "\n",
        "Our `TEXT` field has tokenize='spacy' as an argument. This defines that the \"tokenization\" (the act of splitting the string into discrete \"tokens\") should be done using the [spaCy](https://spacy.io/) tokenizer. If no `tokenize` argument is passed, the default is simply splitting the string on spaces.\n",
        "\n",
        "`LABEL` is defined by a `LabelField`, a special subset of the `Field` class specifically used for handling labels. We will explain the `dtype` argument later.\n",
        "\n",
        "For more on `Fields`, go [here](https://github.com/pytorch/text/blob/master/torchtext/data/field.py).\n",
        "\n",
        "We also set the random seeds for reproducibility. \n",
        "\n",
        "We'll be using *packed padded sequences*, which will make our RNN only process the non-padded elements of our sequence, and for any padded element the `output` will be a zero tensor. To use packed padded sequences, we have to tell the RNN how long the actual sequences are. We do this by setting `include_lengths = True` for our `TEXT` field. This will cause `batch.text` to now be a tuple with the first element being our sentence (a numericalized tensor that has been padded) and the second element being the actual lengths of our sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oBeVWwmS8clr"
      },
      "source": [
        "import torch\n",
        "from torchtext import data\n",
        "from torchtext import datasets\n",
        "\n",
        "SEED = 1234\n",
        "\n",
        "torch.manual_seed(SEED)\n",
        "torch.backends.cudnn.deterministic = True\n",
        "\n",
        "TEXT = data.Field(tokenize = 'spacy', include_lengths = True)\n",
        "LABEL = data.LabelField(dtype = torch.float)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ycaPUUK8clw"
      },
      "source": [
        "Another handy feature of TorchText is that it has support for common datasets used in natural language processing (NLP). \n",
        "\n",
        "The following code automatically downloads the IMDb dataset and splits it into the canonical train/test splits as `torchtext.datasets` objects. It process the data using the `Fields` we have previously defined. The IMDb dataset consists of 50,000 movie reviews, each marked as being a positive or negative review."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nNDlseJW8clx"
      },
      "source": [
        "from torchtext import datasets\n",
        "\n",
        "train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sAx4pwFiDlwn"
      },
      "source": [
        "We can see how many examples are in each split by checking their length."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pwlF3DLVDqjL"
      },
      "source": [
        "print(f'Number of training examples: {len(train_data)}')\n",
        "print(f'Number of testing examples: {len(test_data)}')\n",
        "print(vars(train_data.examples[0]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4W3HTWn_8cl1"
      },
      "source": [
        "The IMDb dataset only has train/test splits, so we need to create a validation set. We can do this with the `.split()` method. \n",
        "\n",
        "By default this splits 70/30, however by passing a `split_ratio` argument, we can change the ratio of the split, i.e. a `split_ratio` of 0.8 would mean 80% of the examples make up the training set and 20% make up the validation set. \n",
        "\n",
        "We also pass our random seed to the `random_state` argument, ensuring that we get the same train/validation split each time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dE7VucqV8cl2"
      },
      "source": [
        "import random\n",
        "\n",
        "train_data, valid_data = train_data.split(random_state = random.seed(SEED))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ZBdZeR_8cl7"
      },
      "source": [
        "Next is the use of pre-trained word embeddings. Now, instead of having our word embeddings initialized randomly, they are initialized with these pre-trained vectors. Here, we will use max vocab size of top 25000 most common words for quick training.\n",
        "\n",
        "We get these vectors simply by specifying which vectors we want and passing it as an argument to `build_vocab`. `TorchText` handles downloading the vectors and associating them with the correct words in our vocabulary.\n",
        "\n",
        "Here, we'll be using the `\"fasttext.simple.300d\" vectors\"`. `300d` indicates these vectors are 100-dimensional.\n",
        "\n",
        "You can see the other available vectors [here](https://github.com/pytorch/text/blob/master/torchtext/vocab.py#L113).\n",
        "\n",
        "The theory is that these pre-trained vectors already have words with similar semantic meaning close together in vector space, e.g. \"terrible\", \"awful\", \"dreadful\" are nearby. This gives our embedding layer a good initialization as it does not have to learn these relations from scratch.\n",
        "\n",
        "**Note**: these vectors are about 293MB, so watch out if you have a limited internet connection.\n",
        "\n",
        "By default, TorchText will initialize words in your vocabulary but not in your pre-trained embeddings to zero. We don't want this, and instead initialize them randomly by setting `unk_init` to `torch.Tensor.normal_`. This will now initialize those words via a Gaussian distribution. This initialization is for the words which appear in the examples but we have cut from the vocabulary. For example, if the sentence was \"This film is great and I love it\" but the word \"love\" was not in the vocabulary, it would become \"This film is great and I `<unk>` it\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bdBXzRPs8cl8"
      },
      "source": [
        "MAX_VOCAB_SIZE = 25_000\n",
        "\n",
        "TEXT.build_vocab(train_data, \n",
        "                 max_size = MAX_VOCAB_SIZE, \n",
        "                 vectors = \"fasttext.simple.300d\", \n",
        "                 unk_init = torch.Tensor.normal_)\n",
        "\n",
        "LABEL.build_vocab(train_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W23YhJ3DGQlv"
      },
      "source": [
        "Now, let's check unique tokens in TEXT and LABEL vocabulary.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oljXHECBGHO8"
      },
      "source": [
        "print(f\"Unique tokens in TEXT vocabulary: {len(TEXT.vocab)}\")\n",
        "print(f\"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cewOIVENGiDf"
      },
      "source": [
        "Why is the vocab size 25002 and not 25000? One of the addition tokens is the `<unk>` token and the other is a `<pad>` token.\n",
        "\n",
        "When we feed sentences into our model, we feed a _batch_ of them at a time, i.e. more than one at a time, and all sentences in the batch need to be the same size. Thus, to ensure each sentence in the batch is the same size, any shorter than the longest within the batch are padded.\n",
        "\n",
        "![](https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/assets/sentiment6.png?raw=1)\n",
        "\n",
        "We can also view the most common words in the vocabulary and their frequencies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RH7S5CgrGvAh"
      },
      "source": [
        "print(TEXT.vocab.freqs.most_common(20))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q8Nw5F2vGyIh"
      },
      "source": [
        "We can also see the vocabulary directly using either the `stoi` (**s**tring **to** **i**nt) or `itos` (**i**nt **to**  **s**tring) method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sD-GeKnbG1xX"
      },
      "source": [
        "print(TEXT.vocab.itos[:10])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oOuRwL-n8cmA"
      },
      "source": [
        "The final step of preparing the data is creating the iterators. We iterate over these in the training/evaluation loop, and they return a batch of examples (indexed and converted into tensors) at each iteration.\n",
        "\n",
        "We'll use a `BucketIterator` which is a special type of iterator that will return a batch of examples where each example is of a similar length, minimizing the amount of padding per example.\n",
        "\n",
        "We also want to place the tensors returned by the iterator on the GPU (if you're using one). PyTorch handles this using `torch.device`, we then pass this device to the iterator.\n",
        "\n",
        "Another thing for packed padded sequences all of the tensors within a batch need to be sorted by their lengths. This is handled in the iterator by setting `sort_within_batch = True`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dzC9TYIt8cmB"
      },
      "source": [
        "BATCH_SIZE = 64\n",
        "\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "print(\"We are working with \", device)\n",
        "\n",
        "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
        "    (train_data, valid_data, test_data), \n",
        "    batch_size = BATCH_SIZE,\n",
        "    sort_within_batch = True,\n",
        "    device = device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0I-SEGbB8cmG"
      },
      "source": [
        "### Different RNN Architecture\n",
        "\n",
        "As you are already aware, standard RNNs suffer from the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem). LSTMs overcome this by having an extra recurrent state called a _cell_, $c$ - which can be thought of as the \"memory\" of the LSTM - and they use use multiple _gates_ which control the flow of information into and out of the memory. For more information, go [here](https://colah.github.io/posts/2015-08-Understanding-LSTMs/). While RNN are can be written as a function of $x_t$ and $h_t$ as following:\n",
        "\n",
        "$$(h_{t+1}) = \\text{RNN}(x_t, h_t)$$\n",
        "\n",
        "We can simply think of the LSTM as a function of $x_t$, $h_t$ and $c_t$, instead of just $x_t$ and $h_t$.\n",
        "\n",
        "$$(h_{t+1}, c_{t+1}) = \\text{LSTM}(x_t, h_t, c_t)$$\n",
        "\n",
        "Thus, the model using an LSTM looks something like (with the embedding layers omitted):\n",
        "\n",
        "![](https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/assets/sentiment2.png?raw=1)\n",
        "\n",
        "The initial cell state, $c_0$, like the initial hidden state is initialized to a tensor of all zeros. The sentiment prediction is still, however, only made using the final hidden state, not the final cell state, i.e. $\\hat{y}=f(h_T)$.\n",
        "\n",
        "### Bidirectional RNN\n",
        "\n",
        "The concept behind a bidirectional RNN is simple. As well as having an RNN processing the words in the sentence from the first to the last (a forward RNN), we have a second RNN processing the words in the sentence from the **last to the first** (a backward RNN). At time step $t$, the forward RNN is processing word $x_t$, and the backward RNN is processing word $x_{T-t+1}$. \n",
        "\n",
        "In PyTorch, the hidden state (and cell state) tensors returned by the forward and backward RNNs are stacked on top of each other in a single tensor. \n",
        "\n",
        "We make our sentiment prediction using a concatenation of the last hidden state from the forward RNN (obtained from final word of the sentence), $h_T^\\rightarrow$, and the last hidden state from the backward RNN (obtained from the first word of the sentence), $h_T^\\leftarrow$, i.e. $\\hat{y}=f(h_T^\\rightarrow, h_T^\\leftarrow)$   \n",
        "\n",
        "The image below shows a bi-directional RNN, with the forward RNN in orange, the backward RNN in green and the linear layer in silver.  \n",
        "\n",
        "![](https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/assets/sentiment3.png?raw=1)\n",
        "\n",
        "### Multi-layer RNN\n",
        "\n",
        "Multi-layer RNNs (also called *deep RNNs*) are another simple concept. The idea is that we add additional RNNs on top of the initial standard RNN, where each RNN added is another *layer*. The hidden state output by the first (bottom) RNN at time-step $t$ will be the input to the RNN above it at time step $t$. The prediction is then made from the final hidden state of the final (highest) layer.\n",
        "\n",
        "The image below shows a multi-layer unidirectional RNN, where the layer number is given as a superscript. Also note that each layer needs their own initial hidden state, $h_0^L$.\n",
        "\n",
        "![](https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/assets/sentiment4.png?raw=1)\n",
        "\n",
        "### Regularization\n",
        "\n",
        "Although we've added improvements to our model, each one adds additional parameters. Without going into overfitting into too much detail, the more parameters you have in in your model, the higher the probability that your model will overfit (memorize the training data, causing  a low training error but high validation/testing error, i.e. poor generalization to new, unseen examples). To combat this, we use regularization. More specifically, we use a method of regularization called *dropout*. Dropout works by randomly *dropping out* (setting to 0) neurons in a layer during a forward pass. The probability that each neuron is dropped out is set by a hyperparameter and each neuron with dropout applied is considered indepenently. One theory about why dropout works is that a model with parameters dropped out can be seen as a \"weaker\" (less parameters) model. The predictions from all these \"weaker\" models (one for each forward pass) get averaged together withinin the parameters of the model. Thus, your one model can be thought of as an ensemble of weaker models, none of which are over-parameterized and thus should not overfit.\n",
        "\n",
        "### Implementation Details\n",
        " In this project, you will be implementing RNN and a variation of RNN called LSTM. We will implement and compare performances over following 3 different architectures:\n",
        "\n",
        "1. Unidirectional multi-layer RNN\n",
        "2. Bidirectional multi-layer RNN \n",
        "3. Bidirectional multi-layer LSTM\n",
        " \n",
        "To use RNN, we use `nn.RNN` and to use an LSTM, we use `nn.LSTM`. You can read more on the implementation of these two models [here](https://pytorch.org/docs/stable/nn.html). Also, note that the LSTM returns the `output` and a tuple of the final `hidden` state and the final `cell` state, whereas the standard RNN only returns the `output` and final `hidden` state. \n",
        "\n",
        "As the final hidden state of our LSTM and RNN has both a forward and a backward component, which will be concatenated together, the size of the input to the `nn.Linear` layer is twice that of the hidden dimension size.\n",
        "\n",
        "Implementing bidirectionality and adding additional layers are done by passing values for the `num_layers` and `bidirectional` arguments for the RNN/LSTM. \n",
        "\n",
        "Dropout is implemented by initializing an `nn.Dropout` layer (the argument is the probability of dropping out each neuron) and using it within the `forward` method after each layer we want to apply dropout to. **Note**: never use dropout on the input or output layers (`text` or `fc` in this case), you only ever want to use dropout on intermediate layers. \n",
        "\n",
        "The final hidden state, `hidden`, has a shape of _**[num layers * num directions, batch size, hid dim]**_. These are ordered: **[forward_layer_0, backward_layer_0, forward_layer_1, backward_layer 1, ..., forward_layer_n, backward_layer n]**. As we want the final (top) layer forward and backward hidden states, we get the top two hidden layers from the first dimension, `hidden[-2,:,:]` and `hidden[-1,:,:]`, and concatenate them together before passing them to the linear layer (after applying dropout). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDnfDCeUeD1W"
      },
      "source": [
        "## Project Task 3 - Bidirectional Multi-Layer LSTM\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qpwD1ips8cmH"
      },
      "source": [
        "import torch.nn as nn\n",
        "\n",
        "class LSTM(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, \n",
        "                 bidirectional, dropout, pad_idx):\n",
        "        \n",
        "        super().__init__()\n",
        "        \n",
        "        #TO-DO\n",
        "        #1. Initialize Embedding Layer\n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)\n",
        "        #2. Initialize LSTM layer\n",
        "        self.rnn = nn.LSTM(embedding_dim, \n",
        "                           hidden_dim, \n",
        "                           num_layers=n_layers, \n",
        "                           bidirectional=bidirectional, \n",
        "                           dropout=dropout)\n",
        "        #3. Initialize a fully connected layer with Linear transformation\n",
        "        self.fc = nn.Linear(hidden_dim * 2, output_dim)\n",
        "        #4. Initialize Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "        \n",
        "    def forward(self, text, text_lengths):\n",
        "        #text = [sent_len, batch_size]\n",
        "\n",
        "        #TO-DO\n",
        "        #1. Apply embedding layer that matches each word to its vector and apply dropout. Dim [sent_len, batch_size, emb_dim]\n",
        "        embedded = self.dropout(self.embedding(text))\n",
        "        #2. Run the LSTM along the sentences of length sent_len. #output = [sent len, batch size, hid dim * num directions]; #hidden = [num layers * num directions, batch size, hid dim]\n",
        "        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths.to('cpu'))\n",
        "        packed_output, (hidden, cell) = self.rnn(packed_embedded)\n",
        "        output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)\n",
        "        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))\n",
        "        #3. Concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers and apply dropout\n",
        "            \n",
        "        return self.fc(hidden)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WZyO4nBJ8cmK"
      },
      "source": [
        "We now create an instance of our LSTM class with the parameters and arguments for the number of layers, bidirectionality and dropout probability.\n",
        "\n",
        "To ensure the pre-trained vectors can be loaded into the model, the `EMBEDDING_DIM` must be equal to that of the pre-trained GloVe vectors loaded earlier.\n",
        "\n",
        "We get our pad token index from the vocabulary, getting the actual string representing the pad token from the field's `pad_token` attribute, which is `<pad>` by default."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2kb_UKfO8cmL"
      },
      "source": [
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = 300\n",
        "HIDDEN_DIM = 256\n",
        "OUTPUT_DIM = 1\n",
        "N_LAYERS = 2\n",
        "BIDIRECTIONAL = True\n",
        "DROPOUT = 0.5\n",
        "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
        "\n",
        "model = LSTM(INPUT_DIM, \n",
        "            EMBEDDING_DIM, \n",
        "            HIDDEN_DIM, \n",
        "            OUTPUT_DIM, \n",
        "            N_LAYERS, \n",
        "            BIDIRECTIONAL, \n",
        "            DROPOUT,\n",
        "            PAD_IDX)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NqudxZk28cmP"
      },
      "source": [
        "We'll print out the number of parameters in our model. \n",
        "\n",
        "Notice how we have almost twice as many parameters as before!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qOI-V77X8cmQ"
      },
      "source": [
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "print(f'The model has {count_parameters(model):,} trainable parameters')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bhh_Jswe6jy-"
      },
      "source": [
        "The final addition is copying the pre-trained word embeddings we loaded earlier into the `embedding` layer of our model.\n",
        "\n",
        "We retrieve the embeddings from the field's vocab, and check they're the correct size, _**[vocab size, embedding dim]**_ "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GbKCFy3g6jaK"
      },
      "source": [
        "pretrained_embeddings = TEXT.vocab.vectors\n",
        "\n",
        "print(pretrained_embeddings.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O20Yq9MI6pW3"
      },
      "source": [
        "We then replace the initial weights of the `embedding` layer with the pre-trained embeddings.\n",
        "\n",
        "**Note**: this should always be done on the `weight.data` and not the `weight`!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bNJ3-DOp6sos"
      },
      "source": [
        "model.embedding.weight.data.copy_(pretrained_embeddings)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GW6Qf64n6uRS"
      },
      "source": [
        "As our `<unk>` and `<pad>` token aren't in the pre-trained vocabulary they have been initialized using `unk_init` (an $\\mathcal{N}(0,1)$ distribution) when building our vocab. It is preferable to initialize them both to all zeros to explicitly tell our model that, initially, they are irrelevant for determining sentiment. \n",
        "\n",
        "We do this by manually setting their row in the embedding weights matrix to zeros. We get their row by finding the index of the tokens, which we have already done for the padding index.\n",
        "\n",
        "**Note**: like initializing the embeddings, this should be done on the `weight.data` and not the `weight`!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4HlPV4_b66xz"
      },
      "source": [
        "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
        "\n",
        "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "\n",
        "print(model.embedding.weight.data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkx31oLH683E"
      },
      "source": [
        "We can now see the first two rows of the embedding weights matrix have been set to zeros. As we passed the index of the pad token to the `padding_idx` of the embedding layer it will remain zeros throughout training, however the `<unk>` token embedding will be learned."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q5A-I_TI8cmi"
      },
      "source": [
        "## Train the Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7mPTDOJh8cmk"
      },
      "source": [
        "Now to training the model.\n",
        "\n",
        "Here we will be using `Adam` optimizer for training. This optimizer adapts the learning rate for each parameter, giving parameters that are updated more frequently lower learning rates and parameters that are updated infrequently higher learning rates. More information about `Adam` (and other optimizers) can be found [here](http://ruder.io/optimizing-gradient-descent/index.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0vSUInUw8cml"
      },
      "source": [
        "import torch.optim as optim\n",
        "\n",
        "optimizer = optim.Adam(model.parameters())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VDy35vsA8cmo"
      },
      "source": [
        "Next, we'll define our loss function. In PyTorch this is commonly called a criterion. \n",
        "\n",
        "The loss function here is _binary cross entropy with logits_. \n",
        "\n",
        "Our model currently outputs an unbound real number. As our labels are either 0 or 1, we want to restrict the predictions to a number between 0 and 1. We do this using the _sigmoid_ or _logit_ functions. \n",
        "\n",
        "We then use this this bound scalar to calculate the loss using binary cross entropy. \n",
        "\n",
        "The `BCEWithLogitsLoss` criterion carries out both the sigmoid and the binary cross entropy steps.\n",
        "\n",
        "We define the criterion and place the model and criterion on the GPU (if available) by using `.to`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fAG2gsbe8cmp"
      },
      "source": [
        "criterion = nn.BCEWithLogitsLoss()\n",
        "\n",
        "model = model.to(device)\n",
        "criterion = criterion.to(device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3CJTM7j8cmw"
      },
      "source": [
        "The criterion function calculates the loss, however we have to write our own function to calculate the accuracy. \n",
        "\n",
        "This function first feeds the predictions through a sigmoid layer, squashing the values between 0 and 1, we then round them to the nearest integer. This rounds any value greater than 0.5 to 1 (a positive sentiment) and the rest to 0 (a negative sentiment).\n",
        "\n",
        "We then calculate how many rounded predictions equal the actual labels and average it across the batch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zFihngLA8cmw"
      },
      "source": [
        "def binary_accuracy(preds, y):\n",
        "    \"\"\"\n",
        "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
        "    \"\"\"\n",
        "    #round predictions to the closest integer\n",
        "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
        "    correct = (rounded_preds == y).float() #convert into float for division \n",
        "    acc = correct.sum() / len(correct)\n",
        "    return acc"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGmt9DlO8cm1"
      },
      "source": [
        "We define a function for training our model. The `train` function iterates over all examples, one batch at a time. \n",
        "\n",
        "`model.train()` is used to put the model in \"training mode\", which turns on _dropout_ and _batch normalization_.\n",
        "\n",
        "**Note**: as we are using dropout, we must remember to use `model.train()` to ensure the dropout is \"turned on\" while training.\n",
        "\n",
        "For each batch, we first zero the gradients. Each parameter in a model has a `grad` attribute which stores the gradient calculated by the `criterion`. PyTorch does not automatically remove (or \"zero\") the gradients calculated from the last gradient calculation, so they must be manually zeroed.\n",
        "\n",
        "We then feed the batch of sentences, `batch.text`, into the model. As we have set `include_lengths = True`, our `batch.text` is now a tuple with the first element being the numericalized tensor and the second element being the actual lengths of each sequence. We separate these into their own variables, `text` and `text_lengths`, before passing them to the model. \n",
        "\n",
        "Note, you do not need to do `model.forward(text, text_lengths)`, simply calling the model works. The `squeeze` is needed as the predictions are initially size _**[text, text_lengths, 1]**_, and we need to remove the dimension of size 1 as PyTorch expects the predictions input to our criterion function to be of size _**[text, text_lengths]**_.\n",
        "\n",
        "The loss and accuracy are then calculated using our predictions and the labels, `batch.label`, with the loss being averaged over all examples in the batch.\n",
        "\n",
        "We calculate the gradient of each parameter with `loss.backward()`, and then update the parameters using the gradients and optimizer algorithm with `optimizer.step()`.\n",
        "\n",
        "The loss and accuracy is accumulated across the epoch, the `.item()` method is used to extract a scalar from a tensor which only contains a single value.\n",
        "\n",
        "Finally, we return the loss and accuracy, averaged across the epoch. The `len` of an iterator is the number of batches in the iterator.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cT-BG8jQ8cm2"
      },
      "source": [
        "def train(model, iterator, optimizer, criterion):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.train()\n",
        "    \n",
        "    for batch in iterator:\n",
        "        \n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        text, text_lengths = batch.text\n",
        "        \n",
        "        predictions = model(text, text_lengths).squeeze(1)\n",
        "        \n",
        "        loss = criterion(predictions, batch.label)\n",
        "        \n",
        "        acc = binary_accuracy(predictions, batch.label)\n",
        "        \n",
        "        loss.backward()\n",
        "        \n",
        "        optimizer.step()\n",
        "        \n",
        "        epoch_loss += loss.item()\n",
        "        epoch_acc += acc.item()\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_DpAyYjr8cm6"
      },
      "source": [
        "Then we define a function for evaluating our model, again remembering to separate `batch.text`.\n",
        "\n",
        "`evaluate` is similar to `train`, with a few modifications as you don't want to update the parameters when evaluating.\n",
        "\n",
        "`model.eval()` puts the model in \"evaluation mode\", this turns off _dropout_ and _batch normalization_.\n",
        "\n",
        "No gradients are calculated on PyTorch operations inside the `with no_grad()` block. This causes less memory to be used and speeds up computation.\n",
        "\n",
        "The rest of the function is the same as `train`, with the removal of `optimizer.zero_grad()`, `loss.backward()` and `optimizer.step()`, as we do not update the model's parameters when evaluating."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NBJkW2Z78cm7"
      },
      "source": [
        "def evaluate(model, iterator, criterion):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.eval()\n",
        "    \n",
        "    with torch.no_grad():\n",
        "    \n",
        "        for batch in iterator:\n",
        "            \n",
        "            text, text_lengths = batch.text\n",
        "            \n",
        "            predictions = model(text, text_lengths).squeeze(1)\n",
        "            \n",
        "            loss = criterion(predictions, batch.label)\n",
        "            \n",
        "            acc = binary_accuracy(predictions, batch.label)\n",
        "\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_acc += acc.item()\n",
        "\n",
        "\n",
        "\n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z425A2Cn8cm-"
      },
      "source": [
        "And also create a nice function to tell us how long our epochs are taking."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ht1SqRjM8cnA"
      },
      "source": [
        "import time\n",
        "\n",
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vxxaC3Ls8cnE"
      },
      "source": [
        "We then train the model through multiple epochs, an epoch being a complete pass through all examples in the training and validation sets.\n",
        "\n",
        "At each epoch, if the validation loss is the best we have seen so far, we'll save the parameters of the model on the mounted gdrive, and then after training has finished we'll use that model on the test set. You can see the saved model in the left panel. Click on the folder option and go to gdrive -> My Drive. \n",
        "\n",
        "**Note**: you are required to submit this \"bilstm_model.pt\" file for grading."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lv4rfx0G8cnF"
      },
      "source": [
        "N_EPOCHS = 5\n",
        "path = F\"/content/gdrive/My Drive/bilstm_model.pt\"\n",
        "best_valid_loss = float('inf')\n",
        "\n",
        "for epoch in range(N_EPOCHS):\n",
        "\n",
        "    start_time = time.time()\n",
        "    \n",
        "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
        "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
        "    \n",
        "    end_time = time.time()\n",
        "\n",
        "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "    \n",
        "    if valid_loss < best_valid_loss:\n",
        "        best_valid_loss = valid_loss\n",
        "        torch.save(model.state_dict(), path)\n",
        "    \n",
        "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
        "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "14WfUZua8cnI"
      },
      "source": [
        "The train accuracy here seems to be improving now over 5 epochs.\n",
        "\n",
        "Finally, we calculate the test loss and accuracy, which we get from our parameters that gave us the best validation loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RJVA1JMt8cnJ"
      },
      "source": [
        "model.load_state_dict(torch.load(path, map_location=device))\n",
        "\n",
        "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
        "\n",
        "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AODvyprH8cnM"
      },
      "source": [
        "## User Input\n",
        "\n",
        "We can now use our model to predict the sentiment of any sentence we give it. As it has been trained on movie reviews, the sentences provided should also be movie reviews.\n",
        "\n",
        "When using a model for inference it should always be in evaluation mode. If this tutorial is followed step-by-step then it should already be in evaluation mode (from doing `evaluate` on the test set), however we explicitly set it to avoid any risk.\n",
        "\n",
        "Our `predict_sentiment` function does a few things:\n",
        "- sets the model to evaluation mode\n",
        "- tokenizes the sentence, i.e. splits it from a raw string into a list of tokens\n",
        "- indexes the tokens by converting them into their integer representation from our vocabulary\n",
        "- gets the length of our sequence\n",
        "- converts the indexes, which are a Python list into a PyTorch tensor\n",
        "- add a batch dimension by `unsqueeze`ing \n",
        "- converts the length into a tensor\n",
        "- squashes the output prediction from a real number between 0 and 1 with the `sigmoid` function\n",
        "- converts the tensor holding a single value into an integer with the `item()` method\n",
        "\n",
        "We are expecting reviews with a negative sentiment to return a value close to 0 and positive reviews to return a value close to 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wqIHzbpC8cnO"
      },
      "source": [
        "import spacy\n",
        "nlp = spacy.load('en')\n",
        "\n",
        "def predict_sentiment(model, sentence):\n",
        "    model.eval()\n",
        "    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]\n",
        "    indexed = [TEXT.vocab.stoi[t] for t in tokenized]\n",
        "    length = [len(indexed)]\n",
        "    tensor = torch.LongTensor(indexed).to(device)\n",
        "    tensor = tensor.unsqueeze(1)\n",
        "    length_tensor = torch.LongTensor(length)\n",
        "    prediction = torch.sigmoid(model(tensor, length_tensor))\n",
        "    return prediction.item()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vTEe1Qvc8cnQ"
      },
      "source": [
        "An example negative review..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N-KsB4Yl8cnR"
      },
      "source": [
        "predict_sentiment(model, \"This film is terrible\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UO1Y9IFv8cnU"
      },
      "source": [
        "An example positive review..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mxHYvP3O8cnW"
      },
      "source": [
        "predict_sentiment(model, \"This film is great\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P_rnR7T28cnY"
      },
      "source": [
        "However, it's still not perfect for this example..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nFbQg-wSRVDu"
      },
      "source": [
        "predict_sentiment(model, \"Holy crap this film is good\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BoqRQc18GE_k"
      },
      "source": [
        "Here we can see that bi-LSTM architecture is not only giving better results for sentiment analysis task, but it has also overcome vanishing gradient problem that RNN suffers from. \n",
        "\n",
        "However, it's still not perfect and there are many more models to improve upon this task, most recent ones use attention (why not try fancy new thing in the market? ;))\n",
        "\n",
        "If you are done with all the implementations, go ahead and submit the notebooks and trained model (.pt files) using submit script.\n",
        "\n"
      ]
    }
  ]
}