4. LSTM POS Sequence Labeling.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "HW_4.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/asliakalin/ML/blob/master/LSTM_POS_Sequence_Labeling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5KTrLJKD-l7",
        "colab_type": "text"
      },
      "source": [
        "# Homework 4: Neural Sequence Labeling\n",
        "\n",
        "**Due March 4, 2020 at 11:59PM**\n",
        "\n",
        "\n",
        "In this homework, you will be implementing, training, and evaluating an LSTM for part-of-speech tagging using the PyTorch library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DFgEH9BWEqPV",
        "colab_type": "text"
      },
      "source": [
        "**Before beginning, please switch your Colab session to a GPU runtime** \n",
        "\n",
        "Go to Runtime > Change runtime type > Hardware accelerator > GPU"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xALTxzWIEkkE",
        "colab_type": "text"
      },
      "source": [
        "### Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kk8GRWIkbrU1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# import libraries\n",
        "import torch\n",
        "import numpy as np\n",
        "import torch.nn as nn\n",
        "from torch.nn.utils.rnn import pad_sequence, pad_packed_sequence, pack_padded_sequence"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t87Vex5_b5MI",
        "colab_type": "code",
        "outputId": "12d082ae-a1c7-484d-87a4-cc3f47897992",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# if this cell prints \"Running on cpu\", you must switch runtime environments\n",
        "# go to Runtime > Change runtime type > Hardware accelerator > GPU\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(\"Running on {}\".format(device))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Running on cuda\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3nDS9MwFCVf",
        "colab_type": "text"
      },
      "source": [
        "### Download & Load Pretrained Embeddings\n",
        "\n",
        "In this assignment, we will be using GloVe pretrained word embeddings. You can read more about GloVe here: https://nlp.stanford.edu/projects/glove/\n",
        "\n",
        "**Note**: this section will take *several minutes*, since the embedding files are large. Files in Colab may be cached between sessions, so you may or may not need to redownload the files each time you reconnect. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "csWld6ckFNL1",
        "colab_type": "code",
        "outputId": "24a60606-3ab3-47b8-807c-cdc8f8e00a95",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 461
        }
      },
      "source": [
        "# download pretrained word embeddings\n",
        "!wget http://nlp.stanford.edu/data/glove.6B.zip\n",
        "!unzip glove*.zip"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-03-29 00:57:45--  http://nlp.stanford.edu/data/glove.6B.zip\n",
            "Resolving nlp.stanford.edu (nlp.stanford.edu)... 171.64.67.140\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:80... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://nlp.stanford.edu/data/glove.6B.zip [following]\n",
            "--2020-03-29 00:57:45--  https://nlp.stanford.edu/data/glove.6B.zip\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip [following]\n",
            "--2020-03-29 00:57:45--  http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip\n",
            "Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22\n",
            "Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 862182613 (822M) [application/zip]\n",
            "Saving to: ‘glove.6B.zip’\n",
            "\n",
            "glove.6B.zip        100%[===================>] 822.24M  2.20MB/s    in 6m 27s  \n",
            "\n",
            "2020-03-29 01:04:12 (2.12 MB/s) - ‘glove.6B.zip’ saved [862182613/862182613]\n",
            "\n",
            "Archive:  glove.6B.zip\n",
            "  inflating: glove.6B.50d.txt        \n",
            "  inflating: glove.6B.100d.txt       \n",
            "  inflating: glove.6B.200d.txt       \n",
            "  inflating: glove.6B.300d.txt       \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iiuJ5eylL0A0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def read_embeddings(filename, vocab_size=10000):\n",
        "  \"\"\"\n",
        "  Utility function, loads in the `vocab_size` most common embeddings from `filename`\n",
        "  \n",
        "  Arguments:\n",
        "  - filename:     path to file\n",
        "                  automatically infers correct embedding dimension from filename\n",
        "  - vocab_size:   maximum number of embeddings to load\n",
        "\n",
        "  Returns \n",
        "  - embeddings:   torch.FloatTensor matrix of size (vocab_size x word_embedding_dim)\n",
        "  - vocab:        dictionary mapping word (str) to index (int) in embedding matrix\n",
        "  \"\"\"\n",
        "\n",
        "  # get the embedding size from the first embedding\n",
        "  with open(filename, encoding=\"utf-8\") as file:\n",
        "    word_embedding_dim = len(file.readline().split(\" \")) - 1\n",
        "\n",
        "  vocab = {}\n",
        "\n",
        "  embeddings = np.zeros((vocab_size, word_embedding_dim))\n",
        "\n",
        "  with open(filename, encoding=\"utf-8\") as file:\n",
        "    for idx, line in enumerate(file):\n",
        "\n",
        "      if idx + 2 >= vocab_size:\n",
        "        break\n",
        "\n",
        "      cols = line.rstrip().split(\" \")\n",
        "      val = np.array(cols[1:])\n",
        "      word = cols[0]\n",
        "      embeddings[idx + 2] = val\n",
        "      vocab[word] = idx + 2\n",
        "  \n",
        "  # a FloatTensor is a multidimensional matrix\n",
        "  # that contains 32-bit floats in every entry\n",
        "  # https://pytorch.org/docs/stable/tensors.html\n",
        "  return torch.FloatTensor(embeddings), vocab"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Ah3clY7lHNZ",
        "colab_type": "text"
      },
      "source": [
        "Running the cell below lists all the files in the current directory. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4D__oK6mQ6hs",
        "colab_type": "code",
        "outputId": "eefbbd61-c33c-4e15-87e0-f80b19d6c50b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 212
        }
      },
      "source": [
        "!ls -lh"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total 3.0G\n",
            "-rw-rw-r-- 1 root root 332M Aug  4  2014 glove.6B.100d.txt\n",
            "-rw-rw-r-- 1 root root 662M Aug  4  2014 glove.6B.200d.txt\n",
            "-rw-rw-r-- 1 root root 990M Aug 27  2014 glove.6B.300d.txt\n",
            "-rw-rw-r-- 1 root root 164M Aug  4  2014 glove.6B.50d.txt\n",
            "-rw-r--r-- 1 root root 823M Oct 25  2015 glove.6B.zip\n",
            "-rw-r--r-- 1 root root 209K Mar 29 00:56 pos.dev\n",
            "-rw-r--r-- 1 root root  319 Mar 29 00:56 pos.tagset\n",
            "-rw-r--r-- 1 root root 128K Mar 29 00:56 pos.test\n",
            "-rw-r--r-- 1 root root 1.7M Mar 29 00:56 pos.train\n",
            "drwxr-xr-x 1 root root 4.0K Mar 18 16:23 sample_data\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aoSWrCwZllg6",
        "colab_type": "text"
      },
      "source": [
        "You should see several embedding files, which are all formatted as\n",
        "\n",
        "```\n",
        "glove.6B.<emb_dim>d.txt\n",
        "```\n",
        "\n",
        "Each `txt` file contains `emb_dim` dimensional embeddings for 400,000 unique, uncased words. The script below loads the `vocab_size` most common words from the embedding file into a matrix we can give to our model. All other words will later be mapped to the `UNKNOWN` embedding."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xL8WuEZoOFbs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# this loads the 10,000 most common word 50-dimensional embeddings\n",
        "vocab_size = 10000 \n",
        "embeddings, vocab = read_embeddings('glove.6B.50d.txt', vocab_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RhXTkpTqGR1H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!wget https://raw.githubusercontent.com/dbamman/nlp20/master/HW_4/pos.train\n",
        "!wget https://raw.githubusercontent.com/dbamman/nlp20/master/HW_4/pos.dev\n",
        "!wget https://raw.githubusercontent.com/dbamman/nlp20/master/HW_4/pos.test\n",
        "!wget https://raw.githubusercontent.com/dbamman/nlp20/master/HW_4/pos.tagset"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "py9AStUOJnPB",
        "colab_type": "text"
      },
      "source": [
        "## Part 1: Batching the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a5jWl-z8w_5t",
        "colab_type": "text"
      },
      "source": [
        "Implement the `get_batches` function in the `Dataset` class below. \n",
        "\n",
        "**Please make sure that**\n",
        "\n",
        "*   Your implementation is self-contained. That is, all helper functions and variables are defined within `get_batches`.\n",
        "*   Your implementation can handle variable batch sizes. You may not assume that the value with always be 32\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-ePEcb46_zGS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def read_tagset(tag_file):\n",
        "  \"\"\"\n",
        "  Utility function, loads tag file into a dictionary from tag string to tag index\n",
        "\n",
        "  Arguments:\n",
        "  - tag_file:   file location of the tagset\n",
        "\n",
        "  Outputs:\n",
        "  - tagset:     a dictionary mapping tag strings (e.g. \"VB\") to a unique index\n",
        "  \"\"\"\n",
        "  tagset = {}\n",
        "  with open(tag_file, encoding='utf8') as f:\n",
        "    for line in f:\n",
        "      columns = line.rstrip().split('\\t')\n",
        "      tag = columns[0]\n",
        "      tag_id = int(columns[1])\n",
        "      tagset[tag] = tag_id\n",
        "  \n",
        "  return tagset"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8mMQIJ_vVuiz",
        "colab_type": "text"
      },
      "source": [
        "The cells below download the data files and construct the corresponding `Dataset` objects. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2KWyqb2HcLop",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Dataset():\n",
        "  def __init__(self, filename, is_labeled):\n",
        "    self.is_labeled = is_labeled\n",
        "    # if the file is not labeled, the Dataset has no tags (see read_data)\n",
        "    if is_labeled:\n",
        "      self.sentences, self.tags = self.read_data(filename, is_labeled)\n",
        "    else:\n",
        "      self.sentences = self.read_data(filename, is_labeled)\n",
        "      self.tags = None\n",
        "\n",
        "  def read_data(self, filename, is_labeled):\n",
        "    \"\"\"\n",
        "    Utility function, loads text file into a list of sentence and tag strings\n",
        "\n",
        "    Arguments:\n",
        "    - filename:     path to file\n",
        "    - is_labeled:   whether the file contains tags for each word or not\n",
        "        > if True, we assume each line is formatted as \"<word>\\t<tag>\\n\"\n",
        "        > if False, we assume each line is formatted as \"<word>\\n\"\n",
        "\n",
        "    Returns:\n",
        "    - sentences:    a list of sentences, where each sentence is a list \n",
        "                    words (strings)\n",
        "\n",
        "    if is_labeled=True, also returns\n",
        "    - tags:         a list of tags for each sentence, where tags[i] contains\n",
        "                    a list of tags (strings) that correspond to the words in \n",
        "                    sentences[i]\n",
        "    \"\"\"\n",
        "    sentences = []\n",
        "    tags = []\n",
        "\n",
        "    current_sentence = []\n",
        "    current_tags = []\n",
        "\n",
        "    with open(filename, encoding='utf8') as f:\n",
        "      # iterate over the lines in the file\n",
        "      for line in f:\n",
        "        if len(line) == 0:\n",
        "          continue\n",
        "        if line == '\\n':\n",
        "          if len(current_sentence) != 0:\n",
        "            sentences.append(current_sentence)\n",
        "            tags.append(current_tags)\n",
        "\n",
        "          current_sentence = []\n",
        "          current_tags = []\n",
        "        else:\n",
        "          if is_labeled:\n",
        "            columns = line.rstrip().split('\\t')\n",
        "            word = columns[0].lower()\n",
        "            tag = columns[1]\n",
        "\n",
        "            current_sentence.append(word)\n",
        "            current_tags.append(tag)\n",
        "          else:\n",
        "            column = line.rstrip().split('\\t')\n",
        "            word = column[0].lower()\n",
        "            current_sentence.append(word)\n",
        "      \n",
        "      if is_labeled:\n",
        "        return sentences, tags\n",
        "      else:\n",
        "        return sentences\n",
        "\n",
        "\n",
        "  def get_batches(self, batch_size, vocab, tagset):\n",
        "    \"\"\"\n",
        "    Batches the data into mini-batches of size `batch_size`\n",
        "    Arguments:\n",
        "    - batch_size:       the desired output batch size\n",
        "    - vocab:            a dictionary mapping word strings to indices\n",
        "    - tagset:           a dictionary mapping tag strings to indices\n",
        "\n",
        "    Outputs:\n",
        "    if is_labeled=True:\n",
        "    - batched_word_indices:     a list of matrices of dimension (batch_size x max_seq_len)\n",
        "    - batched_tag_indices:      a list of matrices of dimension (batch_size x max_seq_len)\n",
        "    - batched_lengths:          a list of arrays of length (batch_size)\n",
        "    if is_labeled=False:\n",
        "    - batched_word_indices:     a list of matrices of dimension (batch_size x max_seq_len)\n",
        "    - batched_lengths:          a list of arrays of length (batch_size)\n",
        "\n",
        "    \"\"\"\n",
        "    PAD_INDEX = 0             # reserved for padding words\n",
        "    UNKNOWN_INDEX = 1         # reserved for unknown words\n",
        "    IGNORE_TAG_INDEX = -100   # reserved for padding tags\n",
        "\n",
        "    # randomly shuffle the data\n",
        "    np.random.seed(159) # DON'T CHANGE THIS\n",
        "    shuffle = np.random.permutation(range(len(self.sentences)))\n",
        "    sentences = [self.sentences[i] for i in shuffle]\n",
        "    if self.is_labeled:\n",
        "      tags = [self.tags[i] for i in shuffle]\n",
        "    else:\n",
        "      tags = None\n",
        "\n",
        "    #############################\n",
        "    #       YOUR CODE HERE      #\n",
        "    #############################\n",
        "    \n",
        "    # Helper functions:\n",
        "    def replace(word, tag):\n",
        "      if tag:\n",
        "        if word in tagset:\n",
        "          return tagset[word]\n",
        "      else:\n",
        "        if word in vocab:\n",
        "          return vocab[word]\n",
        "        else:\n",
        "          return UNKNOWN_INDEX\n",
        "    \n",
        "    def pad(sentence, max_length, tag, rep=0):\n",
        "      #if rep == 0:\n",
        "      if len(sentence) != max_length:\n",
        "          diff = max_length - len(sentence)\n",
        "          add = [PAD_INDEX] * diff if not tag else [IGNORE_TAG_INDEX] * diff\n",
        "          sentence += add\n",
        "      #else: \n",
        "      #    while rep > 0:\n",
        "      #      add_val = [[PAD_INDEX]*len(sentence[0])] if not tag else [[IGNORE_TAG_INDEX] *len(sentence[0])]\n",
        "      #      sentence += add_val\n",
        "      #      rep -= 1\n",
        "      return sentence \n",
        "\n",
        "    leftover = len(sentences) % batch_size != 0\n",
        "    num_of_batches = len(sentences)//batch_size if not leftover else (len(sentences)//batch_size) + 1\n",
        "    batched_word_indices = np.empty(num_of_batches, dtype=object)\n",
        "    batched_tag_indices = np.empty(num_of_batches, dtype=object)\n",
        "    batched_lengths = np.empty(num_of_batches, dtype=object)\n",
        "\n",
        "    #print(batch_size, num_of_batches, len(sentences))\n",
        "    for i  in range(num_of_batches):\n",
        "      batch = sentences[i*batch_size:(i+1)*batch_size]\n",
        "      lengths = [len(item) for item in batch]\n",
        "      max_seq_len = max(lengths)\n",
        "      indiced = [pad([replace(i, False) for i in sen], max_seq_len, False) for sen in batch]\n",
        "      #sentence_pad = batch_size - len(indiced)\n",
        "      #padded_sen = pad(indiced, len(indiced), False, sentence_pad)\n",
        "\n",
        "      tag_batch = tags[i*batch_size:(i+1)*batch_size] if self.is_labeled else None\n",
        "      tagged = [pad([replace(i, True) for i in sen], max_seq_len, True) for sen in tag_batch] if self.is_labeled else None\n",
        "      #padded_tag = pad(tagged, len(tagged), True, sentence_pad)if self.is_labeled else None\n",
        "\n",
        "      batched_word_indices[i] = np.array(indiced)\n",
        "      batched_tag_indices[i] = np.array(tagged)\n",
        "      batched_lengths[i] = np.array(lengths)\n",
        "    \n",
        "    #############################\n",
        "    #       DO NOT MODIFY       #\n",
        "    #############################\n",
        "    if self.is_labeled:\n",
        "      return np.array(batched_word_indices), np.array(batched_tag_indices), np.array(batched_lengths)\n",
        "    else:\n",
        "      return np.array(batched_word_indices), np.array(batched_lengths)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "vtP5seIgBORT",
        "colab": {}
      },
      "source": [
        "# read the files\n",
        "tagset = read_tagset('pos.tagset')\n",
        "train_dataset = Dataset('pos.train', is_labeled=True)\n",
        "dev_dataset = Dataset('pos.dev', is_labeled=True)\n",
        "test_dataset = Dataset('pos.test', is_labeled=False)\n",
        "\n",
        "BATCH_SIZE = 32"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMAZmDFSYjWF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# these should run without errors if implemented correctly\n",
        "#print(\"train\")\n",
        "train_batch_idx, train_batch_tags, train_batch_lens = train_dataset.get_batches(BATCH_SIZE, vocab, tagset)\n",
        "#print(\"dev\")\n",
        "dev_batch_idx, dev_batch_tags, dev_batch_lens = dev_dataset.get_batches(BATCH_SIZE, vocab, tagset)\n",
        "#print(\"test\")\n",
        "test_batch_idx, test_batch_lens = test_dataset.get_batches(BATCH_SIZE, vocab, tagset)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L-nnX7WxqDJ3",
        "colab_type": "text"
      },
      "source": [
        "### Part 2: Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oMtkCBt_wzIS",
        "colab_type": "text"
      },
      "source": [
        "Next, we will implement utility functions that will later be used to assess our model's perfomance. \n",
        "\n",
        "**Please make sure that**\n",
        "\n",
        "*   Your implementation is self-contained. That is, keep all helper functions or variables inside of your function.\n",
        "*   Your implementation does not import any additional libraries. You will not receive credit if you do."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OQLiM0ukG-4W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# The accuracy function has been implemented for you\n",
        "\n",
        "def accuracy(true, pred):\n",
        "  \"\"\"\n",
        "  Arguments:\n",
        "  - true:       a list of true label values (integers)\n",
        "  - pred:       a list of predicted label values (integers)\n",
        "\n",
        "  Output:\n",
        "  - accuracy:   the prediction accuracy\n",
        "  \"\"\"\n",
        "  true = np.array(true)\n",
        "  pred = np.array(pred)\n",
        "\n",
        "  num_correct = sum(true == pred)\n",
        "  num_total = len(true)\n",
        "  return num_correct / num_total"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ChJjUu45qFM8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def confusion_matrix(true, pred, num_tags):\n",
        "  \"\"\"\n",
        "  Arguments:\n",
        "  - true:       a list of true label values (integers)\n",
        "  - pred:       a list of predicted label values (integers)\n",
        "  - num_tags:   the number of possible tags\n",
        "                true and pred will both contain integers between\n",
        "                0 and num_tags - 1 (inclusive)\n",
        "\n",
        "  Output: \n",
        "  - confusion_matrix:   a (num_tags x num_tags) matrix of integers\n",
        "\n",
        "  confusion_matrix[i][j] = # predictions where true label\n",
        "  was i and predicted label was j\n",
        "  \"\"\"\n",
        "\n",
        "  confusion_matrix = np.zeros((num_tags, num_tags))\n",
        "\n",
        "  #############################\n",
        "  #       YOUR CODE HERE      #\n",
        "  #############################\n",
        "\n",
        "  for i in range(len(true)):\n",
        "    true_label = true[i]\n",
        "    pred_label = pred[i]\n",
        "    confusion_matrix[true_label][pred_label] += 1\n",
        "      \n",
        "  return confusion_matrix"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Hdj6QSaBV9S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def precision(true, pred, num_tags):\n",
        "  \"\"\"\n",
        "  Arguments:\n",
        "  - true:       a list of true label values (integers)\n",
        "  - pred:       a list of predicted label values (integers)\n",
        "  - num_tags:   the number of possible tags\n",
        "                true and pred will both contain integers between\n",
        "                0 and num_tags - 1 (inclusive)\n",
        "\n",
        "  Output: \n",
        "  - precision:  an array of length num_tags, where precision[i]\n",
        "                gives the precision of class i\n",
        "\n",
        "  Hints:  the confusion matrix may be useful\n",
        "          be careful about zero division\n",
        "  \"\"\"\n",
        "\n",
        "  precision = np.zeros(num_tags)\n",
        "\n",
        "  #############################\n",
        "  #       YOUR CODE HERE      #\n",
        "  #############################\n",
        "\n",
        "  matrix = confusion_matrix(true, pred, num_tags)\n",
        "  for i in range(num_tags):\n",
        "    num = matrix[i][i]\n",
        "    denom = sum(matrix.T[i])\n",
        "    precision[i] = num/denom if denom != 0 else 0\n",
        "      \n",
        "  return precision"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YJL55TnOBVxl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def recall(true, pred, num_tags):\n",
        "  \"\"\"\n",
        "  Arguments:\n",
        "  - true:       a list of true label values (integers)\n",
        "  - pred:       a list of predicted label values (integers)\n",
        "  - num_tags:   the number of possible tags\n",
        "                true and pred will both contain integers between\n",
        "                0 and num_tags - 1 (inclusive)\n",
        "\n",
        "  Output: \n",
        "  - recall:     an array of length num_tags, where recall[i]\n",
        "                gives the recall of class i\n",
        "\n",
        "  Hints:  the confusion matrix may be useful\n",
        "          be careful about zero division\n",
        "  \"\"\"\n",
        "\n",
        "  \"\"\"\n",
        "  YOUR CODE HERE\n",
        "  \"\"\"\n",
        "  recall = np.zeros(num_tags)\n",
        "\n",
        "  #############################\n",
        "  #       YOUR CODE HERE      #\n",
        "  #############################\n",
        "\n",
        "  matrix = confusion_matrix(true, pred, num_tags)\n",
        "  for i in range(num_tags):\n",
        "    num = matrix[i][i] \n",
        "    denom = sum(matrix[i])\n",
        "    recall[i] = num/denom if denom != 0 else 0\n",
        "     \n",
        "  return recall"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7drr7z1VBVjD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def f1_score(true, pred, num_tags):\n",
        "  \"\"\"\n",
        "  Arguments:\n",
        "  - true:       a list of true label values (integers)\n",
        "  - pred:       a list of predicted label values (integers)\n",
        "  - num_tags:   the number of possible tags\n",
        "                true and pred will both contain integers between\n",
        "                0 and num_tags - 1 (inclusive)\n",
        "\n",
        "  Output: \n",
        "  - f1:         an array of length num_tags, where f1[i]\n",
        "                gives the recall of class i\n",
        "  \"\"\"\n",
        "  f1 = np.zeros(num_tags)\n",
        "\n",
        "  #############################\n",
        "  #       YOUR CODE HERE      #\n",
        "  #############################\n",
        "  pres = precision(true, pred, num_tags)\n",
        "  rec = recall(true, pred, num_tags)\n",
        "\n",
        "  for i in range(num_tags):\n",
        "      f1[i] = (2*pres[i]*rec[i])/(pres[i]+rec[i]) if (pres[i]+rec[i]) != 0 else 0\n",
        "\n",
        "  return f1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GS1p55P5UGv4",
        "colab_type": "text"
      },
      "source": [
        "### Part 3: Building the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6IPoFwqfoFOO",
        "colab_type": "text"
      },
      "source": [
        "Fill in the blanks in `LSTMTagger`'s `__init__` function. If you get stuck, you can reference PyTorch's [torch.nn documentation](https://pytorch.org/docs/stable/nn.html) or [this official tutorial](https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html) on LSTM sequence labeling."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L6J3z3T0USI8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class LSTMTagger(nn.Module):\n",
        "  \"\"\"\n",
        "  An LSTM model for sequence labeling\n",
        "\n",
        "  Initialization Arguments:\n",
        "  - embeddings:   a matrix of size (vocab_size, emb_dim)\n",
        "                  containing pretrained embedding weights\n",
        "  - hidden_dim:   the LSTM's hidden layer size\n",
        "  - tagset_size:  the number of possible output tags\n",
        "\n",
        "  \"\"\"\n",
        "  def __init__(self, embeddings, hidden_dim, tagset_size):\n",
        "    super().__init__()\n",
        "  \n",
        "    self.hidden_dim = hidden_dim\n",
        "    self.num_labels = tagset_size\n",
        "\n",
        "    #############################\n",
        "    #       YOUR CODE HERE      #\n",
        "    #############################\n",
        "\n",
        "    # Initialize a PyTorch embeddings layer using the pretrained embedding weights\n",
        "    self.embeddings = nn.Embedding(len(embeddings), len(embeddings.T), _weight=embeddings)\n",
        "\n",
        "    # Initialize an LSTM layer\n",
        "    self.lstm = nn.LSTM(len(embeddings.T), hidden_dim)\n",
        "\n",
        "    # Initialize a single feedforward layer\n",
        "    self.hidden2tag = nn.Linear(hidden_dim, tagset_size)\n",
        "  \n",
        "  def forward(self, indices, lengths):\n",
        "    \"\"\"\n",
        "    Runs a batched sequence through the model and returns output logits\n",
        "\n",
        "    Arguments:\n",
        "    - indices:  a matrix of size (batch_size x max_seq_len)\n",
        "                containing the word indices of sentences in the batch\n",
        "    - lengths:  a vector of size (batch_size) containing the\n",
        "                original lengths of the sequences before padding\n",
        "\n",
        "    Output:\n",
        "    - logits:   a matrix of size (batch_size x max_seq_len x num_tags)\n",
        "                gives a score to each possible tag for each word\n",
        "                in each sentence \n",
        "    \"\"\"\n",
        "    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "    # cast arrays as PyTorch data types and move to GPU memory\n",
        "    indices = torch.LongTensor(indices).to(device)\n",
        "    lengths = torch.LongTensor(lengths).to(device)\n",
        "    \n",
        "    # convert word indices to word embeddings\n",
        "    embeddings = self.embeddings(indices)\n",
        "\n",
        "    # pack/pad handles variable length sequence batching\n",
        "    # see here if you're curious: https://gist.github.com/HarshTrivedi/f4e7293e941b17d19058f6fb90ab0fec\n",
        "    packed_input_embs = pack_padded_sequence(embeddings, lengths, batch_first=True, enforce_sorted=False)\n",
        "    # run input through LSTM layer\n",
        "    packed_output, _ = self.lstm(packed_input_embs)\n",
        "    # unpack sequences into original format\n",
        "    padded_output, output_lengths = pad_packed_sequence(packed_output, batch_first=True)\n",
        "\n",
        "    logits = self.hidden2tag(padded_output)\n",
        "    return logits\n",
        "\n",
        "  def run_training(self, train_dataset, dev_dataset, batch_size, vocab, tagset,\n",
        "                         lr=5e-4, num_epochs=100, eval_every=5):\n",
        "    \"\"\"\n",
        "    Trains the model on the training data with a learning rate of lr\n",
        "    for num_epochs. Evaluates the model on the dev data eval_every epochs.\n",
        "\n",
        "    Arguments:\n",
        "    - train_dataset:  Dataset object containing the training data\n",
        "    - dev_dataset:    Dataset object containing the dev data\n",
        "    - batch_size:     batch size for train/dev data\n",
        "    - vocab:          a dictionary mapping word strings to indices\n",
        "    - tagset:         a dictionary mapping tag strings to indices\n",
        "    - lr:             learning rate\n",
        "    - num_epochs:     number of epochs to train for\n",
        "    - eval_every:     evaluation is run eval_every epochs\n",
        "    \"\"\"\n",
        "    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "    if str(device) == 'cpu':\n",
        "      print(\"Training only supported in GPU environment\")\n",
        "      return\n",
        "\n",
        "    # clear unreferenced data/models from GPU memory \n",
        "    torch.cuda.empty_cache()\n",
        "    # move model to GPU memory\n",
        "    self.to(device)\n",
        "\n",
        "    # set the optimizer (Adam) and loss function (CrossEnt)\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
        "    loss_function = nn.CrossEntropyLoss(ignore_index=-100)\n",
        "\n",
        "    # batch training and dev data\n",
        "    train_batch_idx, train_batch_tags, train_batch_lens = train_dataset.get_batches(BATCH_SIZE, vocab, tagset)\n",
        "    dev_batch_idx, dev_batch_tags, dev_batch_lens = dev_dataset.get_batches(BATCH_SIZE, vocab, tagset)\n",
        "\n",
        "    print(\"**** TRAINING *****\")\n",
        "    for i in range(num_epochs):\n",
        "      # sets the model in train mode\n",
        "      self.train()\n",
        "\n",
        "      total_loss = 0\n",
        "      for b in range(len(train_batch_idx)):\n",
        "        # compute the logits\n",
        "        logits = model.forward(train_batch_idx[b], train_batch_lens[b])\n",
        "        # move labels to GPU memory\n",
        "        labels = torch.LongTensor(train_batch_tags[b]).to(device)\n",
        "        # compute the loss with respect to true labels\n",
        "        loss = loss_function(logits.view(-1, len(tagset)), labels.view(-1))\n",
        "        total_loss += loss\n",
        "        # propagate gradients backward\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        # set model gradients to zero before performing next forward pass\n",
        "        self.zero_grad()\n",
        "\n",
        "      print(\"Epoch {} | Loss: {}\".format(i, total_loss))\n",
        "\n",
        "      if (i + 1) % eval_every == 0:\n",
        "        print(\"**** EVALUATION *****\")\n",
        "        # sets the model in evaluate mode (no gradients)\n",
        "        self.eval()\n",
        "        # compute dev f1 score\n",
        "        acc, true, pred = self.evaluate(dev_batch_idx, dev_batch_lens, dev_batch_tags, tagset)\n",
        "        print(\"Dev Accuracy: {}\".format(acc))\n",
        "        print(\"**********************\")\n",
        "\n",
        "  def evaluate(self, batched_sentences, batched_lengths, batched_labels, tagset):\n",
        "    \"\"\"\n",
        "    Evaluate the model's predictions on the provided dataset. \n",
        "\n",
        "    Arguments:\n",
        "    - batched_sentences:  a list of matrices, each of size (batch_size x max_seq_len),\n",
        "                          containing the word indices of sentences in the batch\n",
        "    - batched_lengths:    a list of vectors, each of size (batch_size), containing the\n",
        "                          original lengths of the sequences before padding\n",
        "    - batched_labels:     a list of matrices, each of size (batch_size x max_seq_len),\n",
        "                          containing the tag indices corresponding to sentences in the batch\n",
        "    - num_tags:           the number of possible output tags\n",
        "\n",
        "    Output:\n",
        "    - accuracy:           the model's prediction accuracy\n",
        "    - all_true_labels:    a flattened list of all true labels\n",
        "    - all_predictions:    a flattened list of all of the model's corresponding predictions\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "    all_true_labels = []\n",
        "    all_predictions = []\n",
        "\n",
        "    for b in range(len(batched_sentences)):\n",
        "      logits = self.forward(batched_sentences[b], batched_lengths[b])\n",
        "      batch_predictions = torch.argmax(logits, dim=-1).cpu().numpy()\n",
        "\n",
        "      batch_size, _ = batched_sentences[b].shape\n",
        "\n",
        "      for i in range(batch_size):\n",
        "        tags = batched_labels[b][i]\n",
        "        preds = batch_predictions[i]\n",
        "        \n",
        "        seq_len = int(batched_lengths[b][i])\n",
        "        for j in range(seq_len):\n",
        "          all_predictions.append(int(preds[j]))\n",
        "          all_true_labels.append(int(tags[j]))\n",
        "      \n",
        "    \n",
        "    acc = accuracy(all_true_labels, all_predictions)\n",
        "      \n",
        "    return acc, all_true_labels, all_predictions"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A4AOB94R9RFo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def set_seed(seed):\n",
        "  \"\"\"\n",
        "  Sets random seeds and sets model in deterministic\n",
        "  training mode. Ensures reproducible results\n",
        "  \"\"\"\n",
        "  torch.manual_seed(seed)\n",
        "  torch.backends.cudnn.deterministic = True\n",
        "  torch.backends.cudnn.benchmark = False\n",
        "  np.random.seed(seed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-AuNeDk9qAM_",
        "colab_type": "text"
      },
      "source": [
        "## Training the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3WTEvLeVuNWl",
        "colab_type": "text"
      },
      "source": [
        "Run the cells below to train your model. If all of the previous sections are implemented correctly, you should see\n",
        "\n",
        "\n",
        "*   the loss decreasing consistently for every epoch\n",
        "*   the dev accuracy increasing until convergence around ~0.88\n",
        "\n",
        "The staff solution achieves an accuracy of 0.880 after 25 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S9NqwYnfU2WB",
        "colab_type": "code",
        "outputId": "138fe855-2e74-41a0-dfbd-853c191f7af6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 745
        }
      },
      "source": [
        "# sets the random seed – DO NOT change this\n",
        "# this ensures deterministic results that are comparable with the staff values\n",
        "set_seed(159)\n",
        "\n",
        "HIDDEN_SIZE = 64\n",
        "# intialize a new LSTMTagger model\n",
        "model = LSTMTagger(embeddings, HIDDEN_SIZE, len(tagset))\n",
        "# train the model\n",
        "model.run_training(train_dataset, dev_dataset, BATCH_SIZE, vocab, tagset,   \n",
        "                   lr=5e-4, num_epochs=25, eval_every=5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "**** TRAINING *****\n",
            "Epoch 0 | Loss: 999.9422607421875\n",
            "Epoch 1 | Loss: 442.2580871582031\n",
            "Epoch 2 | Loss: 275.37127685546875\n",
            "Epoch 3 | Loss: 212.16624450683594\n",
            "Epoch 4 | Loss: 181.42295837402344\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.8586200039769338\n",
            "**********************\n",
            "Epoch 5 | Loss: 163.11807250976562\n",
            "Epoch 6 | Loss: 150.56387329101562\n",
            "Epoch 7 | Loss: 141.14601135253906\n",
            "Epoch 8 | Loss: 133.6881866455078\n",
            "Epoch 9 | Loss: 127.54783630371094\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.8754026645456353\n",
            "**********************\n",
            "Epoch 10 | Loss: 122.3573989868164\n",
            "Epoch 11 | Loss: 117.87614440917969\n",
            "Epoch 12 | Loss: 113.92591094970703\n",
            "Epoch 13 | Loss: 110.38809967041016\n",
            "Epoch 14 | Loss: 107.16902923583984\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.8791409823026447\n",
            "**********************\n",
            "Epoch 15 | Loss: 104.20204162597656\n",
            "Epoch 16 | Loss: 101.439453125\n",
            "Epoch 17 | Loss: 98.83808135986328\n",
            "Epoch 18 | Loss: 96.3730239868164\n",
            "Epoch 19 | Loss: 94.02764129638672\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.8803738317757009\n",
            "**********************\n",
            "Epoch 20 | Loss: 91.78711700439453\n",
            "Epoch 21 | Loss: 89.6347427368164\n",
            "Epoch 22 | Loss: 87.56659698486328\n",
            "Epoch 23 | Loss: 85.56128692626953\n",
            "Epoch 24 | Loss: 83.62176513671875\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.8802147544243388\n",
            "**********************\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JH5giKgvJp0c",
        "colab_type": "text"
      },
      "source": [
        "Once the model is trained, run the cells below to print the precision, recall, and $F_1$ score per class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TLqaZ_6cMMPM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def eval_per_class(model, dataset, vocab, tagset):\n",
        "  \"\"\"\n",
        "  Prints precision, recall, and F1 for each class in the tagset\n",
        "  \"\"\"\n",
        "  # batch the data\n",
        "  batched_idx, batched_tags, batched_lens = dev_dataset.get_batches(BATCH_SIZE, vocab, tagset)\n",
        "  # compute idx --> tag from tag --> idx\n",
        "  reverse_tagset = {v: k for k,v in tagset.items()}\n",
        "  # evaluate model on hold-out set\n",
        "  acc, true, pred = model.evaluate(batched_idx, batched_lens, batched_tags, tagset)\n",
        "  true = np.array(true)\n",
        "  pred = np.array(pred)\n",
        "\n",
        "  pr = precision(true, pred, len(tagset))\n",
        "  re = recall(true, pred, len(tagset))\n",
        "  f1 = f1_score(true, pred, len(tagset))\n",
        "\n",
        "  for idx, tag in reverse_tagset.items():\n",
        "    print(\"***********************\")\n",
        "    print(\"TAG: {}\".format(tag))\n",
        "    num_pred = np.sum(pred == idx)\n",
        "    num_true = np.sum(true == idx)\n",
        "    print(\"({} pred, {} true)\".format(num_pred, num_true))\n",
        "\n",
        "    print(\"PRECISION: \\t{:.3f}\".format(pr[idx]))\n",
        "    print(\"RECALL: \\t{:.3f}\".format(re[idx]))\n",
        "    print(\"F1 SCORE: \\t{:.3f}\".format(f1[idx]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-tTEpCBsuYuP",
        "colab_type": "code",
        "outputId": "96e27391-fa48-428c-aa29-4fbbb3e856ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 5350
        }
      },
      "source": [
        "eval_per_class(model, dev_dataset, vocab, tagset)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***********************\n",
            "TAG: $\n",
            "(13 pred, 14 true)\n",
            "PRECISION: \t1.000\n",
            "RECALL: \t0.929\n",
            "F1 SCORE: \t0.963\n",
            "***********************\n",
            "TAG: ''\n",
            "(85 pred, 88 true)\n",
            "PRECISION: \t0.882\n",
            "RECALL: \t0.852\n",
            "F1 SCORE: \t0.867\n",
            "***********************\n",
            "TAG: ,\n",
            "(949 pred, 936 true)\n",
            "PRECISION: \t0.954\n",
            "RECALL: \t0.967\n",
            "F1 SCORE: \t0.960\n",
            "***********************\n",
            "TAG: -LRB-\n",
            "(107 pred, 117 true)\n",
            "PRECISION: \t0.972\n",
            "RECALL: \t0.889\n",
            "F1 SCORE: \t0.929\n",
            "***********************\n",
            "TAG: -RRB-\n",
            "(117 pred, 120 true)\n",
            "PRECISION: \t0.949\n",
            "RECALL: \t0.925\n",
            "F1 SCORE: \t0.937\n",
            "***********************\n",
            "TAG: .\n",
            "(1461 pred, 1503 true)\n",
            "PRECISION: \t0.988\n",
            "RECALL: \t0.960\n",
            "F1 SCORE: \t0.974\n",
            "***********************\n",
            "TAG: :\n",
            "(103 pred, 106 true)\n",
            "PRECISION: \t0.922\n",
            "RECALL: \t0.896\n",
            "F1 SCORE: \t0.909\n",
            "***********************\n",
            "TAG: ADD\n",
            "(19 pred, 81 true)\n",
            "PRECISION: \t0.368\n",
            "RECALL: \t0.086\n",
            "F1 SCORE: \t0.140\n",
            "***********************\n",
            "TAG: AFX\n",
            "(1 pred, 4 true)\n",
            "PRECISION: \t0.000\n",
            "RECALL: \t0.000\n",
            "F1 SCORE: \t0.000\n",
            "***********************\n",
            "TAG: CC\n",
            "(780 pred, 781 true)\n",
            "PRECISION: \t0.990\n",
            "RECALL: \t0.988\n",
            "F1 SCORE: \t0.989\n",
            "***********************\n",
            "TAG: CD\n",
            "(325 pred, 378 true)\n",
            "PRECISION: \t0.858\n",
            "RECALL: \t0.738\n",
            "F1 SCORE: \t0.794\n",
            "***********************\n",
            "TAG: DT\n",
            "(1967 pred, 1943 true)\n",
            "PRECISION: \t0.970\n",
            "RECALL: \t0.982\n",
            "F1 SCORE: \t0.976\n",
            "***********************\n",
            "TAG: EX\n",
            "(51 pred, 56 true)\n",
            "PRECISION: \t0.961\n",
            "RECALL: \t0.875\n",
            "F1 SCORE: \t0.916\n",
            "***********************\n",
            "TAG: FW\n",
            "(2 pred, 30 true)\n",
            "PRECISION: \t1.000\n",
            "RECALL: \t0.067\n",
            "F1 SCORE: \t0.125\n",
            "***********************\n",
            "TAG: GW\n",
            "(16 pred, 32 true)\n",
            "PRECISION: \t0.375\n",
            "RECALL: \t0.188\n",
            "F1 SCORE: \t0.250\n",
            "***********************\n",
            "TAG: HYPH\n",
            "(89 pred, 95 true)\n",
            "PRECISION: \t0.854\n",
            "RECALL: \t0.800\n",
            "F1 SCORE: \t0.826\n",
            "***********************\n",
            "TAG: IN\n",
            "(2461 pred, 2353 true)\n",
            "PRECISION: \t0.909\n",
            "RECALL: \t0.951\n",
            "F1 SCORE: \t0.929\n",
            "***********************\n",
            "TAG: JJ\n",
            "(1730 pred, 1655 true)\n",
            "PRECISION: \t0.810\n",
            "RECALL: \t0.847\n",
            "F1 SCORE: \t0.828\n",
            "***********************\n",
            "TAG: JJR\n",
            "(40 pred, 47 true)\n",
            "PRECISION: \t0.675\n",
            "RECALL: \t0.574\n",
            "F1 SCORE: \t0.621\n",
            "***********************\n",
            "TAG: JJS\n",
            "(69 pred, 84 true)\n",
            "PRECISION: \t0.913\n",
            "RECALL: \t0.750\n",
            "F1 SCORE: \t0.824\n",
            "***********************\n",
            "TAG: LS\n",
            "(9 pred, 5 true)\n",
            "PRECISION: \t0.333\n",
            "RECALL: \t0.600\n",
            "F1 SCORE: \t0.429\n",
            "***********************\n",
            "TAG: MD\n",
            "(356 pred, 358 true)\n",
            "PRECISION: \t0.986\n",
            "RECALL: \t0.980\n",
            "F1 SCORE: \t0.983\n",
            "***********************\n",
            "TAG: NFP\n",
            "(30 pred, 60 true)\n",
            "PRECISION: \t0.767\n",
            "RECALL: \t0.383\n",
            "F1 SCORE: \t0.511\n",
            "***********************\n",
            "TAG: NN\n",
            "(3495 pred, 3336 true)\n",
            "PRECISION: \t0.813\n",
            "RECALL: \t0.851\n",
            "F1 SCORE: \t0.832\n",
            "***********************\n",
            "TAG: NNP\n",
            "(2048 pred, 1816 true)\n",
            "PRECISION: \t0.662\n",
            "RECALL: \t0.746\n",
            "F1 SCORE: \t0.701\n",
            "***********************\n",
            "TAG: NNPS\n",
            "(23 pred, 63 true)\n",
            "PRECISION: \t0.783\n",
            "RECALL: \t0.286\n",
            "F1 SCORE: \t0.419\n",
            "***********************\n",
            "TAG: NNS\n",
            "(975 pred, 929 true)\n",
            "PRECISION: \t0.790\n",
            "RECALL: \t0.829\n",
            "F1 SCORE: \t0.809\n",
            "***********************\n",
            "TAG: PDT\n",
            "(2 pred, 21 true)\n",
            "PRECISION: \t0.000\n",
            "RECALL: \t0.000\n",
            "F1 SCORE: \t0.000\n",
            "***********************\n",
            "TAG: POS\n",
            "(85 pred, 84 true)\n",
            "PRECISION: \t0.953\n",
            "RECALL: \t0.964\n",
            "F1 SCORE: \t0.959\n",
            "***********************\n",
            "TAG: PRP\n",
            "(1486 pred, 1487 true)\n",
            "PRECISION: \t0.990\n",
            "RECALL: \t0.989\n",
            "F1 SCORE: \t0.990\n",
            "***********************\n",
            "TAG: PRP$\n",
            "(313 pred, 315 true)\n",
            "PRECISION: \t0.984\n",
            "RECALL: \t0.978\n",
            "F1 SCORE: \t0.981\n",
            "***********************\n",
            "TAG: RB\n",
            "(1152 pred, 1292 true)\n",
            "PRECISION: \t0.909\n",
            "RECALL: \t0.810\n",
            "F1 SCORE: \t0.857\n",
            "***********************\n",
            "TAG: RBR\n",
            "(30 pred, 22 true)\n",
            "PRECISION: \t0.433\n",
            "RECALL: \t0.591\n",
            "F1 SCORE: \t0.500\n",
            "***********************\n",
            "TAG: RBS\n",
            "(20 pred, 20 true)\n",
            "PRECISION: \t0.650\n",
            "RECALL: \t0.650\n",
            "F1 SCORE: \t0.650\n",
            "***********************\n",
            "TAG: RP\n",
            "(64 pred, 75 true)\n",
            "PRECISION: \t0.703\n",
            "RECALL: \t0.600\n",
            "F1 SCORE: \t0.647\n",
            "***********************\n",
            "TAG: SYM\n",
            "(4 pred, 20 true)\n",
            "PRECISION: \t0.750\n",
            "RECALL: \t0.150\n",
            "F1 SCORE: \t0.250\n",
            "***********************\n",
            "TAG: TO\n",
            "(347 pred, 359 true)\n",
            "PRECISION: \t0.859\n",
            "RECALL: \t0.830\n",
            "F1 SCORE: \t0.844\n",
            "***********************\n",
            "TAG: UH\n",
            "(61 pred, 116 true)\n",
            "PRECISION: \t0.918\n",
            "RECALL: \t0.483\n",
            "F1 SCORE: \t0.633\n",
            "***********************\n",
            "TAG: VB\n",
            "(1074 pred, 1122 true)\n",
            "PRECISION: \t0.926\n",
            "RECALL: \t0.887\n",
            "F1 SCORE: \t0.906\n",
            "***********************\n",
            "TAG: VBD\n",
            "(509 pred, 520 true)\n",
            "PRECISION: \t0.888\n",
            "RECALL: \t0.869\n",
            "F1 SCORE: \t0.879\n",
            "***********************\n",
            "TAG: VBG\n",
            "(342 pred, 384 true)\n",
            "PRECISION: \t0.874\n",
            "RECALL: \t0.779\n",
            "F1 SCORE: \t0.824\n",
            "***********************\n",
            "TAG: VBN\n",
            "(480 pred, 476 true)\n",
            "PRECISION: \t0.844\n",
            "RECALL: \t0.851\n",
            "F1 SCORE: \t0.847\n",
            "***********************\n",
            "TAG: VBP\n",
            "(770 pred, 771 true)\n",
            "PRECISION: \t0.921\n",
            "RECALL: \t0.920\n",
            "F1 SCORE: \t0.920\n",
            "***********************\n",
            "TAG: VBZ\n",
            "(648 pred, 643 true)\n",
            "PRECISION: \t0.952\n",
            "RECALL: \t0.960\n",
            "F1 SCORE: \t0.956\n",
            "***********************\n",
            "TAG: WDT\n",
            "(105 pred, 106 true)\n",
            "PRECISION: \t0.790\n",
            "RECALL: \t0.783\n",
            "F1 SCORE: \t0.787\n",
            "***********************\n",
            "TAG: WP\n",
            "(120 pred, 113 true)\n",
            "PRECISION: \t0.900\n",
            "RECALL: \t0.956\n",
            "F1 SCORE: \t0.927\n",
            "***********************\n",
            "TAG: WP$\n",
            "(0 pred, 2 true)\n",
            "PRECISION: \t0.000\n",
            "RECALL: \t0.000\n",
            "F1 SCORE: \t0.000\n",
            "***********************\n",
            "TAG: WRB\n",
            "(112 pred, 113 true)\n",
            "PRECISION: \t1.000\n",
            "RECALL: \t0.991\n",
            "F1 SCORE: \t0.996\n",
            "***********************\n",
            "TAG: XX\n",
            "(0 pred, 3 true)\n",
            "PRECISION: \t0.000\n",
            "RECALL: \t0.000\n",
            "F1 SCORE: \t0.000\n",
            "***********************\n",
            "TAG: ``\n",
            "(100 pred, 91 true)\n",
            "PRECISION: \t0.840\n",
            "RECALL: \t0.923\n",
            "F1 SCORE: \t0.880\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lefzFwCD8AJU",
        "colab_type": "text"
      },
      "source": [
        "## Part 4: Model Exploration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dxZ_Wn75uw7n",
        "colab_type": "text"
      },
      "source": [
        "Congratulations, you've just trained a neural network!\n",
        "\n",
        "Now, improve the `LSTMTagger` model and implementing the `init` function in the `FancyTagger` class below. \n",
        "* Feel free to replace the `forward` function inherited from `LSTMTagger` if \n",
        "you need to, but it should not be necessary to receive full credit. Credit will be awarded based on the performance on a holdout test set. \n",
        "* Do not modify any of the cells above when completing part 4. Instead, insert cells below if you need to perform any additional computations. \n",
        "* You are allowed to use any function in `torch.nn`. You are **not** allowed to import any libraries or use implementations copied from the internet. \n",
        "\n",
        "Before submitting, please describe your modifications below:\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Ya-aaGh6l8D",
        "colab_type": "text"
      },
      "source": [
        "For the FancyTagger model I first **increased the vocab size from 10K to 400K** to increase the number of weights in the model. Then I **increased the dimension of word embeddings from 50d to 100d** for similar reasons. I also **converted the model into bidirectional LSTM** and therefore doubled the input dimension for the hidden layer. I kept the **batch size same as 32** but I **increased the hidden dimension size to 256**. I also **decreased the learning rate to 5e-5** to be able to take smaller steps. In accordance with the smaller steps I increased the **number of epochs from 25 to 60** in the FancyTagger model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hKz2PLbu5d8Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class FancyTagger(LSTMTagger):\n",
        "  \"\"\"\n",
        "  An improved neural model for sequence labeling\n",
        "\n",
        "  Starter code from LSTMTagger has already been provided, but\n",
        "  feel free to change the init and forward function internals\n",
        "  if your model design requires it (though this is not necessary\n",
        "  to receive full credit).\n",
        "\n",
        "  You may use any component in torch.nn. You may NOT\n",
        "  import any additional libraries/modules. \n",
        "\n",
        "  \"\"\"\n",
        "  def __init__(self, embeddings, hidden_dim, tagset_size):\n",
        "    # initializes the parent LSTMTagger class\n",
        "    # inherits forward, evaluate, and run_training methods\n",
        "    super().__init__(embeddings, hidden_dim, tagset_size)\n",
        "  \n",
        "    self.hidden_dim = hidden_dim\n",
        "    self.num_labels = tagset_size\n",
        "\n",
        "    #############################\n",
        "    #       YOUR CODE HERE      #\n",
        "    #############################\n",
        "    # Initialize a PyTorch embeddings layer using the pretrained embedding weights\n",
        "    self.embeddings = nn.Embedding(len(embeddings), len(embeddings.T), _weight=embeddings)\n",
        "    \n",
        "    # Initialize an LSTM layer\n",
        "    self.lstm = nn.LSTM(len(embeddings.T), hidden_dim, bidirectional=True)\n",
        "\n",
        "    # Initialize a single feedforward layer\n",
        "    self.hidden2tag = nn.Linear(hidden_dim*2, tagset_size)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mGDY4ymJvo3h",
        "colab_type": "text"
      },
      "source": [
        "Run the training script below to train the `FancyTagger` model. Again, feel free to adjust any hyperparameters if necessary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J1Ou5V0P52EF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "set_seed(159)\n",
        "vocab_size = 400000\n",
        "embeddings, vocab = read_embeddings('glove.6B.100d.txt', vocab_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8lnp-tWl9Vbo",
        "colab_type": "code",
        "outputId": "25f2faf7-58d4-40a9-a14b-2747a3de52c6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1830
        }
      },
      "source": [
        "# adjust hypermeters\n",
        "HIDDEN_SIZE = 256\n",
        "BATCH_SIZE = 32\n",
        "model = FancyTagger(embeddings, HIDDEN_SIZE, len(tagset))\n",
        "print(model)\n",
        "model.run_training(train_dataset, dev_dataset, BATCH_SIZE, vocab, tagset,   \n",
        "                   lr=5e-5, num_epochs=60, eval_every=5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "FancyTagger(\n",
            "  (embeddings): Embedding(400000, 100)\n",
            "  (lstm): LSTM(100, 256, batch_first=True, bidirectional=True)\n",
            "  (hidden2tag): Linear(in_features=512, out_features=50, bias=True)\n",
            ")\n",
            "**** TRAINING *****\n",
            "Epoch 0 | Loss: 1174.2171630859375\n",
            "Epoch 1 | Loss: 797.6525268554688\n",
            "Epoch 2 | Loss: 547.1670532226562\n",
            "Epoch 3 | Loss: 405.5942687988281\n",
            "Epoch 4 | Loss: 319.2713623046875\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.8104195665142175\n",
            "**********************\n",
            "Epoch 5 | Loss: 262.20123291015625\n",
            "Epoch 6 | Loss: 222.32269287109375\n",
            "Epoch 7 | Loss: 193.39950561523438\n",
            "Epoch 8 | Loss: 171.76878356933594\n",
            "Epoch 9 | Loss: 155.13783264160156\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.8866573871545039\n",
            "**********************\n",
            "Epoch 10 | Loss: 142.01303100585938\n",
            "Epoch 11 | Loss: 131.38795471191406\n",
            "Epoch 12 | Loss: 122.57825469970703\n",
            "Epoch 13 | Loss: 115.11605072021484\n",
            "Epoch 14 | Loss: 108.67938995361328\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9053489759395507\n",
            "**********************\n",
            "Epoch 15 | Loss: 103.04467010498047\n",
            "Epoch 16 | Loss: 98.0530014038086\n",
            "Epoch 17 | Loss: 93.58589172363281\n",
            "Epoch 18 | Loss: 89.55413818359375\n",
            "Epoch 19 | Loss: 85.88578796386719\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9143368462915092\n",
            "**********************\n",
            "Epoch 20 | Loss: 82.52377319335938\n",
            "Epoch 21 | Loss: 79.42571258544922\n",
            "Epoch 22 | Loss: 76.55421447753906\n",
            "Epoch 23 | Loss: 73.88011932373047\n",
            "Epoch 24 | Loss: 71.3786392211914\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9188307814674885\n",
            "**********************\n",
            "Epoch 25 | Loss: 69.02809143066406\n",
            "Epoch 26 | Loss: 66.81028747558594\n",
            "Epoch 27 | Loss: 64.71075439453125\n",
            "Epoch 28 | Loss: 62.717193603515625\n",
            "Epoch 29 | Loss: 60.819068908691406\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9224895605488168\n",
            "**********************\n",
            "Epoch 30 | Loss: 59.0070686340332\n",
            "Epoch 31 | Loss: 57.27428436279297\n",
            "Epoch 32 | Loss: 55.613277435302734\n",
            "Epoch 33 | Loss: 54.017940521240234\n",
            "Epoch 34 | Loss: 52.482757568359375\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9243587194273215\n",
            "**********************\n",
            "Epoch 35 | Loss: 51.00301742553711\n",
            "Epoch 36 | Loss: 49.57396697998047\n",
            "Epoch 37 | Loss: 48.191184997558594\n",
            "Epoch 38 | Loss: 46.84977722167969\n",
            "Epoch 39 | Loss: 45.54923629760742\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.925074567508451\n",
            "**********************\n",
            "Epoch 40 | Loss: 44.289520263671875\n",
            "Epoch 41 | Loss: 43.06735610961914\n",
            "Epoch 42 | Loss: 41.9190788269043\n",
            "Epoch 43 | Loss: 40.72274398803711\n",
            "Epoch 44 | Loss: 39.60173797607422\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9253927222111752\n",
            "**********************\n",
            "Epoch 45 | Loss: 38.48536682128906\n",
            "Epoch 46 | Loss: 37.414215087890625\n",
            "Epoch 47 | Loss: 36.36381912231445\n",
            "Epoch 48 | Loss: 35.337669372558594\n",
            "Epoch 49 | Loss: 34.334373474121094\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9255517995625373\n",
            "**********************\n",
            "Epoch 50 | Loss: 33.36187744140625\n",
            "Epoch 51 | Loss: 32.41389083862305\n",
            "Epoch 52 | Loss: 31.473651885986328\n",
            "Epoch 53 | Loss: 30.566116333007812\n",
            "Epoch 54 | Loss: 29.687911987304688\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9264664943328693\n",
            "**********************\n",
            "Epoch 55 | Loss: 28.793956756591797\n",
            "Epoch 56 | Loss: 27.942474365234375\n",
            "Epoch 57 | Loss: 27.093955993652344\n",
            "Epoch 58 | Loss: 26.274850845336914\n",
            "Epoch 59 | Loss: 25.477949142456055\n",
            "**** EVALUATION *****\n",
            "Dev Accuracy: 0.9264267249950289\n",
            "**********************\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgLM__WZw4wz",
        "colab_type": "text"
      },
      "source": [
        "### Save Predictions\n",
        "\n",
        "When you are satisfied with your `FancyTagger`'s performance on the dev set, run the cell below to write your predictions on the test set to a text file. \n",
        "\n",
        "You can download `predictions.txt` by going to \n",
        "**View > Table of Contents > Files**\n",
        "\n",
        "Please submit this `predictions.txt` file to Gradescope. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ddSD3-FN9Zzp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "assert isinstance(model, FancyTagger), 'Please assign your FancyTagger to a variable named model'\n",
        "\n",
        "test_batch_idx, test_batch_lens = test_dataset.get_batches(BATCH_SIZE, vocab, tagset)\n",
        "\n",
        "predictions = []\n",
        "\n",
        "for b in range(len(test_batch_idx)):\n",
        "  logits = model.forward(test_batch_idx[b], test_batch_lens[b])\n",
        "  batch_predictions = torch.argmax(logits, dim=-1).cpu().numpy()\n",
        "\n",
        "  batch_size, _ = test_batch_idx[b].shape\n",
        "\n",
        "  for i in range(batch_size):\n",
        "    preds = batch_predictions[i]\n",
        "    \n",
        "    seq_len = int(test_batch_lens[b][i])\n",
        "    for j in range(seq_len):\n",
        "      predictions.append(int(preds[j]))\n",
        "  \n",
        "\n",
        "with open('predictions.txt', 'w') as f:\n",
        "  for p in predictions:\n",
        "    f.write(str(p) + \"\\n\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nxlLtPzvaH6d",
        "colab_type": "text"
      },
      "source": [
        "## Debugging"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "fYKo8jfcPtt6",
        "colab": {}
      },
      "source": [
        "b = 3\n",
        "arr = [[\"Asli\", \"loves\"], [\"ice\", \"cream\", \"and\"], [\"she\", \"also\", \"should\", \"finish\"], [\"this\", \"hw\", \"asap\"], [\"so\", \"she\",\"is\",\"using\",\"this\",\"cell\",\"as\"],[\"debugging\",\"method\"],\\\n",
        "       [\"lets\",\"see\",\"how\",\"things\",\"are\",\"working\"], [\"ok\",\"last\",\"one\"],[\"haha\",\"i\",\"lied\",\"sooo\"],[\"its\",\"here\"]]\n",
        "n = len(arr)//b if not (len(arr)%b != 0) else (len(arr)//b) + 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AnU1IfD3aJeX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def replace(word, tag):\n",
        "      if word in vocab:\n",
        "        return vocab[word] if not tag else tagset[word]\n",
        "      else:\n",
        "        return \"UNKNOWN_INDEX\"\n",
        "    \n",
        "def pad(sentence, max_length, tag):\n",
        "      assert(len(sentence)<=max_length)\n",
        "      if len(sentence) != max_length:\n",
        "          diff = max_length - len(sentence)\n",
        "          add = [\"PAD_INDEX\"] * diff if not tag else [\"IGNORE_TAG_INDEX\"] * diff\n",
        "          sentence += add\n",
        "      return sentence \n",
        "\n",
        "for i  in range(n):\n",
        "      matrix = np.array([])\n",
        "      batch = arr[i*b:(i+1)*b]\n",
        "      lengts = [len(item) for item in batch]\n",
        "      longest = max(lengts)\n",
        "      indiced = [pad([replace(i, False) for i in sentence], longest, False) for sentence in batch]\n",
        "      padded = pad(indiced, len(indiced), False)\n",
        "      print(indiced)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jhB-ZTwWaQRT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}