baseline.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.0"
    },
    "colab": {
      "name": "Tarea_2.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z0tyIsliieNr",
        "colab_type": "text"
      },
      "source": [
        "# Tarea 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W_dxGEs3iiau",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## 1.- Introducción\n",
        "\n",
        "### Objetivo\n",
        "\n",
        "En esta tarea abarcaremos la resolución de problemas de [Sequence Labelling](https://en.wikipedia.org/wiki/Sequence_labeling) (etiquetado de sequencias).\n",
        "\n",
        "**¿Qué es sequence labelling?** \n",
        "\n",
        "En breves palabras, dada una secuencia de tokens (frase u oración) sequence labelling tiene por objetivo asignar una etiqueta a cada token de dicha secuencia.\n",
        "\n",
        "**Named Entity Recognition (NER)**\n",
        "\n",
        "Esta tarea consiste en localizar y clasificar los tokens de una oración que representen entidades nombradas. Es decir, tokens que simbolicen (1) **personas**, (2) **organizaciones**, (3) **lugares** y (4) **adjetivos, eventos y otras entidades que no entren en las categorías anteriores** deberán ser taggeados como (1) **PER**, (2) **ORG**, (3) **LOC** y (4) **MISC** respectivamente. Adicionalmente, dado que existen entidades representadas en más de un token (como La Serena), se utiliza la notación BIO como prefijo al tag: Beginning, Inside, Outside. Es decir, si encuentro una entidad, el primer token etiquetado será precedido por B, el segundo por I y los n restantes por I. Por otra parte, si el token no representa ninguna entidad nombrada, se representa por O. Un ejemplo de esto es:\n",
        "\n",
        "Por ejemplo:\n",
        "\n",
        "```\n",
        "Felipe B-PER\n",
        "Bravo I-PER\n",
        "es O\n",
        "el O\n",
        "profesor O\n",
        "de O\n",
        "PLN B-MISC\n",
        "de O\n",
        "la O\n",
        "Universidad B-ORG\n",
        "de I-ORG\n",
        "Chile I-ORG\n",
        ". O\n",
        "```\n",
        "\n",
        "Para mayor información, visitar [NER en Wikipedia](https://es.wikipedia.org/wiki/Reconocimiento_de_entidades_nombradas).\n",
        "\n",
        "**En esta tarea usaremos un dataset de noticias etiquetadas en español.**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9spX-Hkh8YJg",
        "colab_type": "text"
      },
      "source": [
        "### Métodos\n",
        "\n",
        "La idea principal de este notebook es tener un baseline para implementar la tarea 2. Para esto, construiremos una solución utilizando una [red recurrente](https://github.com/dccuchile/CC6205/blob/master/slides/NLP-RNN.pdf)  `LSTM` simple (un solo nivel y sin bi-direccionalidad). \n",
        "\n",
        "En este notebook veremos el proceso de cargar los datasets, haremos batches de texto y padding para poder entrenar el modelo utilizando los batches.\n",
        "\n",
        "Finalmente, mostraremos como construir un output para que lo puedan probar en la tarea en codelab.\n",
        "\n",
        "Se espera que para la tarea, ustedes complementen y mejoren el baseline a partir de varias mejoras que pueden ser aplicadas al modelo. Algunas sugerencias:\n",
        "\n",
        "*   Inicializar los embeddings con modelos pre-entrenados. (glove, etc...)\n",
        "*   Probar bi-direccionalidad.\n",
        "*   Probar varios niveles en las redes recurrentes.\n",
        "*   Probar teacher forcing.\n",
        "*   Incluir dropout.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5EWNqBnG_Atg",
        "colab_type": "text"
      },
      "source": [
        "## 2.- Preprocesamiento"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LMgKjfYC_Go-",
        "colab_type": "text"
      },
      "source": [
        "Para el pre-procesamiento utilizaremos la biblioteca [`torchtext`](https://github.com/pytorch/text). Como su nombre lo indica, torchtext brinda funcionalidades para hacer procesamiento de texto con pytorch."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eIBDzEwJB2NQ",
        "colab_type": "text"
      },
      "source": [
        "### Algunas referencias \n",
        "\n",
        "\n",
        "https://github.com/pytorch/text\n",
        "\n",
        "https://pytorch.org/text/index.html\n",
        "\n",
        "https://torchtext.readthedocs.io/en/latest/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kHuIwP_oMGU-",
        "colab_type": "code",
        "outputId": "2e9b02cb-4de8-4152-eb03-8f7003a92708",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 321
        }
      },
      "source": [
        "# Instalar torchtext\n",
        "!pip3 install --upgrade torchtext"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torchtext\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/43/94/929d6bd236a4fb5c435982a7eb9730b78dcd8659acf328fd2ef9de85f483/torchtext-0.4.0-py3-none-any.whl (53kB)\n",
            "\r\u001b[K     |██████▏                         | 10kB 18.9MB/s eta 0:00:01\r\u001b[K     |████████████▍                   | 20kB 2.1MB/s eta 0:00:01\r\u001b[K     |██████████████████▌             | 30kB 3.0MB/s eta 0:00:01\r\u001b[K     |████████████████████████▊       | 40kB 2.0MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▉ | 51kB 2.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 61kB 2.3MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext) (4.28.1)\n",
            "Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.6/dist-packages (from torchtext) (1.17.4)\n",
            "Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.6/dist-packages (from torchtext) (2.21.0)\n",
            "Requirement already satisfied, skipping upgrade: torch in /usr/local/lib/python3.6/dist-packages (from torchtext) (1.3.1)\n",
            "Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from torchtext) (1.12.0)\n",
            "Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext) (2.8)\n",
            "Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext) (1.24.3)\n",
            "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext) (2019.9.11)\n",
            "Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext) (3.0.4)\n",
            "Installing collected packages: torchtext\n",
            "  Found existing installation: torchtext 0.3.1\n",
            "    Uninstalling torchtext-0.3.1:\n",
            "      Successfully uninstalled torchtext-0.3.1\n",
            "Successfully installed torchtext-0.4.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zh_-imJMiiaw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "from torchtext import data, datasets"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1mG563jgBnGM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Garantizar reproducibilidad\n",
        "SEED = 1234\n",
        "torch.manual_seed(SEED)\n",
        "torch.backends.cudnn.deterministic = True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uMud7YGMBZvg",
        "colab_type": "text"
      },
      "source": [
        "### Torchtext: Data y Fields\n",
        "\n",
        " `TORCHTEXT.DATA` \n",
        "\n",
        "El primer módulo que usaremos, `DATA`, nos proporciona las siguientes funcionalidades:\n",
        "\n",
        "*   Capacidad para definir un pipeline de preprocesamiento\n",
        "*   Batching, padding, y numericalizing (incluida la creación de un vocabulario)\n",
        "*   Divisiones de conjuntos de datos (entrenamiento, validación, prueba)\n",
        "*   Cargar datasets personalizados\n",
        "\n",
        "`DATA.Field` \n",
        "\n",
        "* Define un tipo de datos junto con instrucciones para convertir el texto a Tensor.\n",
        "* La clase Field modela tipos de datos comunes para procesamiento de texto  que pueden ser representados por tensores.\n",
        "* Contiene un objeto Vocab que define el conjunto de valores posibles para elementos del Field y sus representaciones numéricas correspondientes.\n",
        "* El objeto Field también contiene otros parámetros relacionados con la forma en que se debe numericalizar un tipo de datos, como un método de tokenización y el tipo de Tensor que se debe producir.\n",
        "* Si un Field se comparte entre dos columnas en un conjunto de datos (por ejemplo, preguntas y respuestas en un dataset de Question Answering), tendrán un vocabulario compartido.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dV7UM2DmSuY9",
        "colab_type": "text"
      },
      "source": [
        "### Definir nuestros fields\n",
        "\n",
        "\n",
        "Recordemos como está definido el dataset:\n",
        "\n",
        "```\n",
        "El O\n",
        "Abogado B-PER\n",
        "General I-PER\n",
        "del I-PER\n",
        "Estado I-PER\n",
        ", O\n",
        "Daryl B-PER\n",
        "Williams I-PER\n",
        "```\n",
        "\n",
        "\n",
        "A partir de esto, podemos definir los siguientes fields:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3DcM_IjgCdzz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Primer Field: TEXT. Representan los tokens de la secuencia\n",
        "TEXT = data.Field(lower=False)\n",
        "\n",
        "# Segundo Field: NER_TAGS. Representan los Tags asociados a cada palabra.\n",
        "NER_TAGS = data.Field(unk_token=None)\n",
        "\n",
        "fields = ((\"text\", TEXT), (\"nertags\", NER_TAGS))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BehSou6rCvwg",
        "colab_type": "text"
      },
      "source": [
        "### Obtener datos\n",
        "\n",
        "Ahora, obtenemos los datos desde la fuente:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lbT0g_kC18Jb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!wget https://github.com/dccuchile/CC6205/releases/download/Data/train_NER_esp.txt  # Dataset de Entrenamiento\n",
        "!wget https://github.com/dccuchile/CC6205/releases/download/Data/val_NER_esp.txt    # Dataset de Validación (Para probar y ajustar el modelo)\n",
        "!wget https://github.com/dccuchile/CC6205/releases/download/Data/test_NER_esp.txt   # Dataset de la Competencia. Estos datos solo contienen los tokens. ¡¡SON LOS QUE DEBEN SER PREDICHOS!!"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xCKTJOdgC5eC",
        "colab_type": "text"
      },
      "source": [
        "### Cargar datasets usando SequenceTaggingDataset\n",
        "\n",
        "\n",
        "Ahora, usamos la clase `sequenceTaggingDataset` para cargar los datasets.\n",
        "\n",
        "El objetivo de cargar por medio de esta clase los datasets es poder cargar los datos de forma comprensible para pytorch. Para esto, usan el arreglo `field` para definir como se leera cada linea.   \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HsHdGml62J21",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_data = datasets.SequenceTaggingDataset('train_NER_esp.txt', fields, encoding=\"iso-8859-1\", separator=\" \")\n",
        "valid_data = datasets.SequenceTaggingDataset('val_NER_esp.txt', fields, encoding=\"iso-8859-1\", separator=\" \")\n",
        "test_data = datasets.SequenceTaggingDataset('test_NER_esp.txt', fields, encoding=\"iso-8859-1\", separator=\" \")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hu7q3HCliia5",
        "colab_type": "code",
        "outputId": "e6e5bbe9-f13d-4faf-f6e4-8ce8205057d9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "print(f\"Number of training examples: {len(train_data)}\")\n",
        "print(f\"Number of validation examples: {len(valid_data)}\")\n",
        "print(f\"Number of testing examples: {len(test_data)}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of training examples: 8323\n",
            "Number of validation examples: 1915\n",
            "Number of testing examples: 1517\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fDRnhXAdFGL-",
        "colab_type": "text"
      },
      "source": [
        "Visualizemos un ejemplo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MnD0VMvviia9",
        "colab_type": "code",
        "outputId": "4d5c0b8e-cbd9-46f4-d364-2e89ee1660cf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "random_train_item = torch.randint(0, len(train_data), [1])\n",
        "\n",
        "random_example = vars(train_data.examples[random_train_item])\n",
        "\n",
        "for word, tag in zip(random_example['text'], random_example['nertags']):\n",
        "  print(word, '\\t', tag)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "La \t O\n",
            "quinta \t O\n",
            "edición \t O\n",
            "del \t O\n",
            "Festival \t B-MISC\n",
            "de \t I-MISC\n",
            "Teatro \t I-MISC\n",
            "en \t I-MISC\n",
            "la \t I-MISC\n",
            "Calle \t I-MISC\n",
            ", \t O\n",
            "que \t O\n",
            "organiza \t O\n",
            "anualmente \t O\n",
            "el \t O\n",
            "Ayuntamiento \t B-ORG\n",
            "de \t I-ORG\n",
            "Villanueva \t I-ORG\n",
            "de \t I-ORG\n",
            "la \t I-ORG\n",
            "Serena \t I-ORG\n",
            "con \t O\n",
            "la \t O\n",
            "colaboración \t O\n",
            "de \t O\n",
            "la \t O\n",
            "Consejería \t B-ORG\n",
            "de \t I-ORG\n",
            "Cultura \t I-ORG\n",
            "de \t I-ORG\n",
            "la \t I-ORG\n",
            "Junta \t I-ORG\n",
            "de \t I-ORG\n",
            "Extremadura \t I-ORG\n",
            ", \t O\n",
            "tendrá \t O\n",
            "lugar \t O\n",
            "del \t O\n",
            "12 \t O\n",
            "al \t O\n",
            "15 \t O\n",
            "de \t O\n",
            "julio \t O\n",
            ", \t O\n",
            "anunció \t O\n",
            "hoy \t O\n",
            "el \t O\n",
            "concejal \t O\n",
            "de \t O\n",
            "Cultura \t B-MISC\n",
            "villanovense \t O\n",
            ", \t O\n",
            "Gregorio \t B-PER\n",
            "Gil \t I-PER\n",
            "Ruedas \t I-PER\n",
            ". \t O\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2l05KYy5FSUy",
        "colab_type": "text"
      },
      "source": [
        "### Construir los vocabularios para el texto y las etiquetas\n",
        "\n",
        "El siguiente paso consiste en construir los vocabularios: Objetos que contienen todos los tokens (de entrenamiento) posibles para ambos fields. Para esto, hacemos uso del método `Field.build_vocab` sobre cada uno de los fields. \n",
        "\n",
        "Si se proporciona un objeto Dataset, se utilizan todas las columnas correspondientes a este field. También se pueden proporcionar columnas individuales directamente."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PBhp7WICiibL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "TEXT.build_vocab(train_data) #,\n",
        "                 # unk_init = torch.Tensor.normal_,\n",
        "                 #vectors = \"glove.6B.100d\") #Esto es una pista de como usar vectores pre-entrenados.\n",
        "\n",
        "NER_TAGS.build_vocab(train_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M4OgUKM_iibO",
        "colab_type": "code",
        "outputId": "04996ee3-cbcd-4cc5-c897-a55ba51aa7af",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "print(f\"Tokens únicos en TEXT: {len(TEXT.vocab)}\")\n",
        "print(f\"Tokens únicos en NER_TAGS: {len(NER_TAGS.vocab)}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tokens únicos en TEXT: 26101\n",
            "Tokens únicos en NER_TAGS: 10\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5PQhbctGjgU",
        "colab_type": "text"
      },
      "source": [
        "Veamos las posibles etiquetas que se le pueden asignar a una palabra:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d4FeyL9nFnId",
        "colab_type": "code",
        "outputId": "b494c320-587a-4d71-f2cc-24321e9ec1e9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(NER_TAGS.vocab.itos)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['<pad>', 'O', 'B-ORG', 'I-ORG', 'B-LOC', 'B-PER', 'I-PER', 'I-MISC', 'B-MISC', 'I-LOC']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HYQDoUqSHFKj",
        "colab_type": "text"
      },
      "source": [
        "Las cuales coinciden con lo estipulado al comienzo. La única diferencia es la presencia de \\<pad\\>, el cual es generado por el dataloader.\n",
        "\n",
        "Veamos ahora los tokens mas frecuentes y especiales:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m5eSLm4diibR",
        "colab_type": "code",
        "outputId": "c21ff190-00f1-41fb-df40-17d9bfd25c24",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# Tokens mas frecuentes\n",
        "print(TEXT.vocab.freqs.most_common(20))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('de', 17657), (',', 14716), ('la', 9571), ('que', 7516), ('.', 7263), ('el', 6905), ('en', 6484), ('\"', 5691), ('y', 5336), ('a', 4304), ('del', 3742), ('los', 3705), ('se', 2495), ('por', 2473), ('las', 2368), (')', 2099), ('(', 2094), ('con', 2001), ('un', 1985), ('para', 1692)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dk-RKVnFiibV",
        "colab_type": "code",
        "outputId": "0917d680-8f83-4c6a-b955-036a23d8e9ae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Tokens con los primeros 10 indices del vocabulario.\n",
        "print(TEXT.vocab.itos[:10])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['<unk>', '<pad>', 'de', ',', 'la', 'que', '.', 'el', 'en', '\"']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wcz7BU_6ETBs",
        "colab_type": "code",
        "outputId": "ef6f9d3e-5922-4f95-e885-ed370163baf7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "#Tokens especiales en el texto.\n",
        "for tok in TEXT.vocab.itos:\n",
        "    if tok[0] == \"<\":\n",
        "        print(tok)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<unk>\n",
            "<pad>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qrYvF3X0sjWL",
        "colab_type": "text"
      },
      "source": [
        "### Frecuencia de los Tags\n",
        "\n",
        "Visualizemos rápidamente las cantidades y frecuencias de cada tag:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuXOsbJUiibh",
        "colab_type": "code",
        "outputId": "9fcdc233-3bee-4474-9aec-27501f4c0009",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        }
      },
      "source": [
        "def tag_percentage(tag_counts):\n",
        "    \n",
        "    total_count = sum([count for tag, count in tag_counts])\n",
        "    tag_counts_percentages = [(tag, count, count/total_count) for tag, count in tag_counts]\n",
        "  \n",
        "    return tag_counts_percentages\n",
        "\n",
        "print(\"Tag Ocurrencia Porcentaje\\n\")\n",
        "\n",
        "for tag, count, percent in tag_percentage(NER_TAGS.vocab.freqs.most_common()):\n",
        "    print(f\"{tag}\\t{count}\\t{percent*100:4.1f}%\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tag Ocurrencia Porcentaje\n",
            "\n",
            "O\t231920\t87.6%\n",
            "B-ORG\t7390\t 2.8%\n",
            "I-ORG\t4992\t 1.9%\n",
            "B-LOC\t4913\t 1.9%\n",
            "B-PER\t4321\t 1.6%\n",
            "I-PER\t3903\t 1.5%\n",
            "I-MISC\t3212\t 1.2%\n",
            "B-MISC\t2173\t 0.8%\n",
            "I-LOC\t1891\t 0.7%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZizJJAeIoZ9",
        "colab_type": "text"
      },
      "source": [
        "## 3.- El modelo\n",
        "\n",
        "### Definición de la red\n",
        "\n",
        "Teniendo ya cargado los datos, toca definir nuestro modelo."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IqdauQg9iibn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BATCH_SIZES = (64,2,2)\n",
        "\n",
        "# Usar cuda si es que está disponible.\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "# Dividir datos entre entrenamiento y test\n",
        "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
        "    (train_data, valid_data, test_data), \n",
        "    batch_sizes = BATCH_SIZES,\n",
        "    device = device,\n",
        "    sort=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yezpk8CYvFY4",
        "colab_type": "text"
      },
      "source": [
        "La red base tendrá una capa de embedding, unas cuantas LSTM y una capa de salida. Además, contará con dropout para el entrenamiento.\n",
        "\n",
        "Esta será definida en la siguiente clase:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5NlXq7lNiibq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "# Definir la red\n",
        "class RNNNer(nn.Module):\n",
        "    def __init__(self, \n",
        "                 vocab_size, \n",
        "                 embedding_dim, \n",
        "                 hidden_dim, \n",
        "                 output_dim, \n",
        "                 n_layers, \n",
        "                 bidirectional, \n",
        "                 dropout, \n",
        "                 pad_idx):\n",
        "        \n",
        "        super().__init__()\n",
        "        \n",
        "        # Capa de embedding\n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)\n",
        "        \n",
        "        # Capa LSTM\n",
        "        self.rnn = nn.LSTM(embedding_dim, \n",
        "                           hidden_dim, \n",
        "                           num_layers = n_layers, \n",
        "                           bidirectional = bidirectional)\n",
        "        \n",
        "        # Capa de salida\n",
        "        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)\n",
        "        \n",
        "        # Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        \n",
        "    def forward(self, text):\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "        \n",
        "        # Convertir lo enviado a embedding\n",
        "        embedded = F.relu(self.dropout(self.embedding(text)))\n",
        "        \n",
        "        #embedded = [sent len, batch size, emb dim]\n",
        "        \n",
        "        # Pasar los embeddings por la rnn (LSTM)\n",
        "        outputs, (hidden, cell) = self.rnn(embedded)\n",
        "        \n",
        "        #output = [sent len, batch size, hid dim * n directions]\n",
        "        #hidden/cell = [n layers * n directions, batch size, hid dim]\n",
        "        \n",
        "        # Predecir usando la capa de salida.\n",
        "        predictions = self.fc(self.dropout(outputs))\n",
        "        \n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "        \n",
        "        return predictions"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u2_sze6WvIGi",
        "colab_type": "text"
      },
      "source": [
        "## Inicialización de la red\n",
        "\n",
        "Ahora, definimos los hiperparámetros de la red:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E6KMhMfViibv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Hiperparámetros de la red\n",
        "\n",
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = 128\n",
        "HIDDEN_DIM = 512\n",
        "OUTPUT_DIM = len(NER_TAGS.vocab)\n",
        "N_LAYERS = 3\n",
        "BIDIRECTIONAL = True\n",
        "DROPOUT = 0.25 \n",
        "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
        "\n",
        "# Creamos nuestro modelo.\n",
        "model = RNNNer(INPUT_DIM, \n",
        "                     EMBEDDING_DIM, \n",
        "                     HIDDEN_DIM, \n",
        "                     OUTPUT_DIM, \n",
        "                     N_LAYERS, \n",
        "                     BIDIRECTIONAL, \n",
        "                     DROPOUT, \n",
        "                     PAD_IDX)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BuJ0Ow2SvQoY",
        "colab_type": "text"
      },
      "source": [
        "E iniciamos los pesos de la red de forma aleatoria (Usando una distribución normal)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TNhyy0egiiby",
        "colab_type": "code",
        "outputId": "12a18a0c-6b64-4282-8a73-0f80190b0db6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "def init_weights(m):\n",
        "    for name, param in m.named_parameters():\n",
        "        nn.init.normal_(param.data, mean=0, std=0.1)\n",
        "        \n",
        "model.apply(init_weights)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RNNNer(\n",
              "  (embedding): Embedding(26101, 128, padding_idx=1)\n",
              "  (rnn): LSTM(128, 512, num_layers=3, bidirectional=True)\n",
              "  (fc): Linear(in_features=1024, out_features=10, bias=True)\n",
              "  (dropout): Dropout(p=0.25, inplace=False)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P_n1nBlPiib3",
        "colab_type": "code",
        "outputId": "aa9bd757-e135-48f7-a0b8-f8dda7854a28",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "print(f'El modelo tiene {count_parameters(model):,} parámetros entrenables.')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "El modelo tiene 18,580,106 parámetros entrenables.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g-QMo2xCiib9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#pretrained_embeddings = TEXT.vocab.vectors\n",
        "#print(pretrained_embeddings.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qmVWSRiCiicD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#model.embedding.weight.data.copy_(pretrained_embeddings)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-HfgnB_Vv0uX",
        "colab_type": "text"
      },
      "source": [
        "Por último, definimos los embeddings que representan a  \\<UNK\\> y \\<PAD\\> como [0, 0, ..., 0]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DTITlfbZiicH",
        "colab_type": "code",
        "outputId": "0f7637ca-c5aa-4fe7-d524-46265ea4c30a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "source": [
        "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
        "\n",
        "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "\n",
        "# Los primeros 2 embeddings representan UNK y PAD\n",
        "print(model.embedding.weight.data)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
            "        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
            "        [ 0.0931, -0.0657,  0.0369,  ...,  0.0365, -0.0111, -0.0174],\n",
            "        ...,\n",
            "        [-0.2111,  0.0855, -0.0953,  ..., -0.1742,  0.1881, -0.0942],\n",
            "        [-0.0228, -0.1057,  0.1015,  ..., -0.0419, -0.0931,  0.1250],\n",
            "        [-0.0398,  0.0244, -0.0247,  ..., -0.0619, -0.1174,  0.1016]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fz39wa78wGYR",
        "colab_type": "text"
      },
      "source": [
        "## Entrenamiento y evaluación de la red\n",
        "\n",
        "Teniendo ya definida la red, comenzamos con el proceso de entrenamiento.\n",
        "Lo primero es definir el optimizador (`ADAM`) y la loss que usaremos (`CrossEntropy`).\n",
        "\n",
        "Referencias: https://github.com/dccuchile/CC6205/blob/master/slides/NLP-linear.pdf"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gsHhWRWWiicK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch.optim as optim\n",
        "\n",
        "# Optimizador\n",
        "optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "TAG_PAD_IDX = NER_TAGS.vocab.stoi[NER_TAGS.pad_token]\n",
        "\n",
        "# Loss: Cross Entropy\n",
        "criterion = nn.CrossEntropyLoss(ignore_index = TAG_PAD_IDX)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dqr0AJ6_iicR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "model = model.to(device)\n",
        "criterion = criterion.to(device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B21E1eAFId16",
        "colab_type": "text"
      },
      "source": [
        "Por otra parte, definiremos las métricas que serán usadas en la competencia: `precision`, `recall` y `f1`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9mUOOLEWiicU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Definimos las métricas\n",
        "\n",
        "from sklearn.metrics import f1_score, precision_score, recall_score\n",
        "import warnings\n",
        "import sklearn.exceptions\n",
        "warnings.filterwarnings(\"ignore\", category=sklearn.exceptions.UndefinedMetricWarning)\n",
        "\n",
        "\n",
        "def calculate_metrics(preds, y):\n",
        "    \"\"\"\n",
        "    Calcula precision, recall y f1 de cada batch.\n",
        "    \"\"\"\n",
        "\n",
        "    # Obtener el indice de la clase con probabilidad mayor. (clases)\n",
        "    y_pred = preds.argmax(dim = 1, keepdim = True) \n",
        "    # Obtenemos los indices distintos de 0.\n",
        "\n",
        "    y_pred = y_pred.view(-1).to('cpu')\n",
        "    y_true = y.to('cpu')\n",
        "\n",
        "    f1 = f1_score(y_true, y_pred, average='macro')\n",
        "    precision = precision_score(y_true, y_pred, average='macro')\n",
        "    recall = recall_score(y_true, y_pred, average='macro')\n",
        "\n",
        "    return precision, recall, f1\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQUVEEerIm-7",
        "colab_type": "text"
      },
      "source": [
        "### Definimos el entrenamiento\n",
        "\n",
        "Algunos conceptos previos: \n",
        "\n",
        "- `epoca` : una pasada completa de una dataset.\n",
        "- `batch`: una fracción de la época. Se utilizan para entrenar mas rápidamente la red. (mas eficiente pasar n datos que uno en cada ejecución del backpropagation)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8xlq48WjiW6U",
        "colab_type": "text"
      },
      "source": [
        "#### `train`\n",
        "\n",
        "Esta función está encargada de entrenar la red en una época. Para esto, por cada batch de la época actual, predice los tags del texto, calcula su loss y luego hace backpropagation para actualizar los pesos de la red.\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DV6YLt0oiicW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train(model, iterator, optimizer, criterion):\n",
        "\n",
        "    epoch_loss = 0\n",
        "    epoch_precision = 0\n",
        "    epoch_recall = 0\n",
        "    epoch_f1 = 0\n",
        "    \n",
        "    model.train()\n",
        "    \n",
        "    # Por cada batch del iterador de la época:\n",
        "    for batch in iterator:\n",
        "\n",
        "        # Extraemos el texto y los tags del batch que estamos procesado    \n",
        "        text = batch.text\n",
        "        tags = batch.nertags\n",
        "        \n",
        "        # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        #text = [sent len, batch size]\n",
        "        \n",
        "        # Predecimos los tags del texto del batch.\n",
        "        predictions = model(text)\n",
        "        \n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "        #tags = [sent len, batch size]\n",
        "        \n",
        "        # Reordenamos los datos para calcular la loss\n",
        "        predictions = predictions.view(-1, predictions.shape[-1])\n",
        "        tags = tags.view(-1)\n",
        "        \n",
        "        #predictions = [sent len * batch size, output dim]\n",
        "        #tags = [sent len * batch size]\n",
        "        \n",
        "        # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "        loss = criterion(predictions, tags)\n",
        "                \n",
        "        # Calculamos el accuracy\n",
        "        precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "        \n",
        "        # Calculamos los gradientes\n",
        "        loss.backward()\n",
        "        \n",
        "        # Actualizamos los parámetros de la red\n",
        "        optimizer.step()\n",
        "        \n",
        "        # Actualizamos el loss y las métricas\n",
        "        epoch_loss += loss.item()\n",
        "        epoch_precision += precision\n",
        "        epoch_recall += recall\n",
        "        epoch_f1 += f1\n",
        "\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_precision / len(iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PYNcwKnAz5Hf",
        "colab_type": "text"
      },
      "source": [
        "#### `evaluate`\n",
        "\n",
        "Evalua el rendimiento actual de la red usando los datos de validación. \n",
        "\n",
        "Por cada batch de estos datos, calcula y reporta el loss y las métricas asociadas al conjunto de validación. \n",
        "\n",
        "Ya que las métricas son calculadas por cada batch, estas son retornadas promediadas por el número de batches entregados. (ver linea del return)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WsRuiUuHiicY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Evaluamos el modelo \n",
        "\n",
        "def evaluate(model, iterator, criterion):\n",
        "\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_precision = 0\n",
        "    epoch_recall = 0\n",
        "    epoch_f1 = 0\n",
        "    \n",
        "    model.eval()\n",
        "    \n",
        "    # Indicamos que ahora no guardaremos los gradientes\n",
        "    with torch.no_grad():\n",
        "        # Por cada batch\n",
        "        for batch in iterator:\n",
        "\n",
        "            text = batch.text\n",
        "            tags = batch.nertags\n",
        "\n",
        "            # Predecimos \n",
        "            predictions = model(text)\n",
        "            \n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "            \n",
        "            # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "            loss = criterion(predictions, tags)\n",
        "                    \n",
        "            # Calculamos las métricas\n",
        "            precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "            \n",
        "            # Actualizamos el loss y las métricas\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_precision += precision\n",
        "            epoch_recall += recall\n",
        "            epoch_f1 += f1\n",
        "\n",
        "    return epoch_loss / len(iterator), epoch_precision / len(iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xs-n9Y5yiica",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import time\n",
        "\n",
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hy3MVf5H0A94",
        "colab_type": "text"
      },
      "source": [
        "### Entrenamiento de la red\n",
        "\n",
        "En este cuadro de código ejecutaremos el entrenamiento de la red.\n",
        "Para esto, primero definiremos el número de épocas y luego por cada época, ejecutaremos `train` y `evaluate`.\n",
        "\n",
        "**Importante: Reiniciar el modelo**\n",
        "\n",
        "Si ejecutas nuevamente esta celda, se seguira entrenando el mismo modelo una y otra vez. \n",
        "Para reiniciar el modelo se debe ejecutar nuevamente la celda que contiene la función `init_weights`\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iK5lQqpviicf",
        "colab_type": "code",
        "outputId": "fc97b1b8-d418-4bba-c30c-4bb1f1d182ca",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 799
        }
      },
      "source": [
        "N_EPOCHS = 15\n",
        "\n",
        "best_valid_loss = float('inf')\n",
        "\n",
        "for epoch in range(N_EPOCHS):\n",
        "\n",
        "    start_time = time.time()\n",
        "    \n",
        "    # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "    # Entrenar\n",
        "    train_loss, train_precision, train_recall, train_f1 = train(model, train_iterator, optimizer, criterion)\n",
        "\n",
        "    # Evaluar\n",
        "    valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(model, valid_iterator, criterion)\n",
        "    \n",
        "    end_time = time.time()\n",
        "\n",
        "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "    # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "    # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "    if valid_loss < best_valid_loss:\n",
        "        best_valid_loss = valid_loss\n",
        "        torch.save(model.state_dict(), 'modelo_tarea_2.pt')\n",
        "    \n",
        "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "    print(f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}')\n",
        "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.695 | Train f1: 0.05 | Train precision: 0.04 | Train recall: 0.10\n",
            "\t Val. Loss: 0.804 |  Val. f1: 0.17 | Val. precision: 0.14 | Val. recall: 0.22\n",
            "Epoch: 02 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.621 | Train f1: 0.05 | Train precision: 0.04 | Train recall: 0.10\n",
            "\t Val. Loss: 0.766 |  Val. f1: 0.17 | Val. precision: 0.14 | Val. recall: 0.22\n",
            "Epoch: 03 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.507 | Train f1: 0.10 | Train precision: 0.14 | Train recall: 0.15\n",
            "\t Val. Loss: 0.444 |  Val. f1: 0.31 | Val. precision: 0.31 | Val. recall: 0.35\n",
            "Epoch: 04 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.218 | Train f1: 0.34 | Train precision: 0.37 | Train recall: 0.39\n",
            "\t Val. Loss: 0.400 |  Val. f1: 0.37 | Val. precision: 0.38 | Val. recall: 0.41\n",
            "Epoch: 05 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.156 | Train f1: 0.48 | Train precision: 0.51 | Train recall: 0.53\n",
            "\t Val. Loss: 0.328 |  Val. f1: 0.37 | Val. precision: 0.37 | Val. recall: 0.43\n",
            "Epoch: 06 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.128 | Train f1: 0.55 | Train precision: 0.58 | Train recall: 0.60\n",
            "\t Val. Loss: 0.331 |  Val. f1: 0.38 | Val. precision: 0.38 | Val. recall: 0.43\n",
            "Epoch: 07 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.105 | Train f1: 0.60 | Train precision: 0.62 | Train recall: 0.65\n",
            "\t Val. Loss: 0.404 |  Val. f1: 0.45 | Val. precision: 0.46 | Val. recall: 0.49\n",
            "Epoch: 08 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.084 | Train f1: 0.66 | Train precision: 0.67 | Train recall: 0.71\n",
            "\t Val. Loss: 0.292 |  Val. f1: 0.49 | Val. precision: 0.49 | Val. recall: 0.52\n",
            "Epoch: 09 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.067 | Train f1: 0.70 | Train precision: 0.70 | Train recall: 0.75\n",
            "\t Val. Loss: 0.295 |  Val. f1: 0.50 | Val. precision: 0.50 | Val. recall: 0.53\n",
            "Epoch: 10 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.054 | Train f1: 0.73 | Train precision: 0.72 | Train recall: 0.79\n",
            "\t Val. Loss: 0.311 |  Val. f1: 0.45 | Val. precision: 0.46 | Val. recall: 0.47\n",
            "Epoch: 11 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.050 | Train f1: 0.74 | Train precision: 0.74 | Train recall: 0.79\n",
            "\t Val. Loss: 0.293 |  Val. f1: 0.50 | Val. precision: 0.50 | Val. recall: 0.53\n",
            "Epoch: 12 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.043 | Train f1: 0.77 | Train precision: 0.76 | Train recall: 0.82\n",
            "\t Val. Loss: 0.409 |  Val. f1: 0.45 | Val. precision: 0.45 | Val. recall: 0.49\n",
            "Epoch: 13 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.043 | Train f1: 0.75 | Train precision: 0.75 | Train recall: 0.81\n",
            "\t Val. Loss: 0.348 |  Val. f1: 0.51 | Val. precision: 0.51 | Val. recall: 0.54\n",
            "Epoch: 14 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.029 | Train f1: 0.79 | Train precision: 0.77 | Train recall: 0.84\n",
            "\t Val. Loss: 0.363 |  Val. f1: 0.52 | Val. precision: 0.53 | Val. recall: 0.55\n",
            "Epoch: 15 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.025 | Train f1: 0.80 | Train precision: 0.79 | Train recall: 0.85\n",
            "\t Val. Loss: 0.341 |  Val. f1: 0.52 | Val. precision: 0.53 | Val. recall: 0.56\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MkSV5T7cGU_y",
        "colab_type": "text"
      },
      "source": [
        "### Cargar modelo preentrenado\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y27CNYfrjtQ-",
        "colab_type": "code",
        "outputId": "5c43a07f-8277-4293-fcac-2a5bb0f0656e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "model.load_state_dict(torch.load('modelo_tarea_2.pt'))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<All keys matched successfully>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "feFwZf3xhH-L",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBctQHTh0lxD",
        "colab_type": "text"
      },
      "source": [
        "### Resultados finales del set de validación\n",
        "\n",
        "Estos son los resultados de predecir el dataset de evaluación con el modelo actualmente entrenado:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s0gVbP8yiicj",
        "colab_type": "code",
        "outputId": "556c26dc-cf29-4a7e-e13a-294358442cf3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(model, valid_iterator, criterion)\n",
        "\n",
        "print(f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Val. Loss: 0.292 |  Val. f1: 0.49 | Val. precision: 0.49 | Val. recall: 0.52\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uF1ysw_Kw6zz",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## 4.- Predecir datos de la competencia\n",
        "\n",
        "Ahora, a partir de los datos de test y nuestro modelo entrenado, predeciremos las etiquetas que serán evaluadas en la competencia."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1RBs3UU4wLk3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def predict_labels(model, iterator, criterion):\n",
        "\n",
        "    model.eval()\n",
        "    list_predictions_batch = []\n",
        "    texts = []\n",
        "\n",
        "    with torch.no_grad():\n",
        "\n",
        "        for batch in iterator:\n",
        "            text = batch.text\n",
        "            texts.append(torch.transpose(text, 0, 1).tolist())\n",
        "\n",
        "            # Predecir los tags del batch\n",
        "            predictions_batch = model(text)\n",
        "            \n",
        "            # Hacer las oraciones y no las palabras el primer indice\n",
        "            predictions_batch = torch.transpose(predictions_batch, 0, 1)\n",
        "            \n",
        "            predicted_tags_batch = []\n",
        "            for predictions_sent in predictions_batch:\n",
        "                sent_tags = []\n",
        "                # extraer la clase (el indice de la probabilidad predicha mas alta)\n",
        "                for prediction_tag in predictions_sent:\n",
        "                    argmax_index = prediction_tag.topk(1)[1]\n",
        "                    sent_tags.append(argmax_index)\n",
        "\n",
        "                predicted_tags_batch.append(sent_tags)\n",
        "            \n",
        "            list_predictions_batch.append(predicted_tags_batch)\n",
        "\n",
        "    return texts, list_predictions_batch\n",
        "\n",
        "test_texts, test_predictions = predict_labels(model, test_iterator , criterion)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rgCbZpeYN1eU",
        "colab_type": "text"
      },
      "source": [
        "Y transformaremos los vectores de los tags y el texto a sus respectivos tokens partir del vocabulario de los `fields` TEXT y NER_TAGS"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "D4eSkY_MgsUn",
        "colab": {}
      },
      "source": [
        "def get_tokens_from_vocab(test_texts, field):\n",
        "  tokens = []\n",
        "  for batch in test_texts:\n",
        "      for sent in batch:\n",
        "          token_batch = []\n",
        "          for token in sent:\n",
        "              token_batch.append(field.vocab.itos[token])\n",
        "          tokens.append(token_batch)\n",
        "  return tokens"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vd8zcAygNG9n",
        "colab_type": "code",
        "outputId": "8df47c99-5230-4f9b-cd03-336ca7a09421",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "tags = get_tokens_from_vocab(test_predictions, NER_TAGS)\n",
        "sentences = get_tokens_from_vocab(test_texts, TEXT)\n",
        "\n",
        "# Ejemplo:\n",
        "print(tags[0])\n",
        "print(sentences[0])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['O', 'O', 'O', 'O', 'O', 'O', 'B-ORG', 'O', 'O']\n",
            "['La', '<unk>', ',', '23', 'may', '(', 'EFECOM', ')', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XU7dxY4jOCFY",
        "colab_type": "text"
      },
      "source": [
        "Filtramos \\<PAD\\>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e7s_L5_hOAXn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def filter_pads(sentences,tags):\n",
        "\n",
        "  filter_sentences = []\n",
        "  filter_labels = []\n",
        "  for sent,labels in zip(sentences,tags):\n",
        "      filter_sentence = []\n",
        "      filter_label = []\n",
        "      \n",
        "      for word,label in zip(sent,labels):\n",
        "          if word != '<pad>':\n",
        "              filter_sentence.append(word)\n",
        "              filter_label.append(label)\n",
        "\n",
        "      filter_sentences.append(filter_sentence)\n",
        "      filter_labels.append(filter_label)\n",
        "\n",
        "  return filter_sentences, filter_labels\n",
        "    \n",
        "filter_sentences, filter_labels = filter_pads(sentences,tags)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n8-s8YZtjMTv",
        "colab_type": "code",
        "outputId": "5041cc19-d5c5-47ec-e963-fe2c2e9b3748",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(\"Número de palabras: {}, Número de tags predichos: {}\".format(sum([len(sent) for sent in filter_sentences]),sum([len(sent) for sent in filter_labels])))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Número de palabras: 51533, Número de tags predichos: 51533\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YwQp1Ru8Oht8",
        "colab_type": "text"
      },
      "source": [
        "### Generar el archivo para la submission"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RPfZkjJGkWyq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os, shutil\n",
        "\n",
        "if (not os.path.isdir('./predictions')):\n",
        "    os.mkdir('./predictions')\n",
        "\n",
        "else:\n",
        "    # Eliminar predicciones anteriores:\n",
        "    shutil.rmtree('./predictions')\n",
        "    os.mkdir('./predictions')\n",
        "\n",
        "f = open('predictions/predictions.txt','w')\n",
        "for sent,labels in zip(sentences,filter_labels):\n",
        "    for word,label in zip(sent,labels):\n",
        "        f.write(word+' '+label+'\\n')\n",
        "    f.write('\\n')\n",
        "f.close()\n",
        "\n",
        "a = shutil.make_archive('predictions', 'zip', './predictions')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k2PqvJAmTFWR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# A veces no funciona a la primera. Ejecutar mas de una vez para obtener el archivo...\n",
        "from google.colab import files\n",
        "files.download('predictions.zip')  "
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}