assignments/transformer/nlp_2_transformer_offensive_language_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "name": "nlp-2-transformer-offensive-language-classification.ipynb",
      "private_outputs": true,
      "provenance": [],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/oliverguhr/htw-nlp-lecture/blob/master/assignments/transformer/nlp_2_transformer_offensive_language_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1x7ywAcjTMyk"
      },
      "source": [
        "# Offensive Language Classification\n",
        "\n",
        "\n",
        "## First Steps\n",
        "\n",
        "We need to download the required packages and our the training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T6cFhLiDTMyk"
      },
      "source": [
        "!pip install datasets transformers accelerate"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lqhVeVEnTMyl"
      },
      "source": [
        "!wget -c https://www.htw-dresden.de/~guhr/dist/sample/germeval2018.training.txt\n",
        "!wget -c https://www.htw-dresden.de/~guhr/dist/sample/germeval2018.test.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ONd9nMwMTMyl"
      },
      "source": [
        "import time\n",
        "import pandas as pd\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Ca0b7_IpH1M"
      },
      "source": [
        "# check if we have a GPU\n",
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5wCOi_UiTMyl"
      },
      "source": [
        "## Prepairing the data\n",
        "\n",
        "In the next step we have to load the data and adjust it a bit. The data is available in tab delimited csv. Pandas is a good choice for simple processing, but it could also be done with Python board tools."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fpvStxVhTMyl"
      },
      "source": [
        "test_df = pd.read_csv(\"germeval2018.test.txt\", sep='\\t', header=0,encoding=\"utf-8\")\n",
        "train_df = pd.read_csv(\"germeval2018.training.txt\", sep='\\t', header=0,encoding=\"utf-8\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H9CkOlvaTMyl"
      },
      "source": [
        "train_df.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W4BsXBCsTMyn"
      },
      "source": [
        "# Since we do not need the label 2 columns, we can delete them.\n",
        "test_df.drop(columns=['label2'], inplace=True)\n",
        "train_df.drop(columns=['label2'], inplace=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q4P20Gf_TMyn"
      },
      "source": [
        "def clean_text (text):\n",
        "    #text = text.str.lower() # lowercase\n",
        "    #text = text.str.replace(r\"\\#\",\"\") # replaces hashtags\n",
        "    #text = text.str.replace(r\"http\\S+\",\"URL\")  # remove URL addresses\n",
        "    #text = text.str.replace(r\"@\",\"\")\n",
        "    #text = text.str.replace(r\"[^A-Za-z0-9öäüÖÄÜß()!?]\", \" \")\n",
        "    #text = text.str.replace(\"\\s{2,}\", \" \")\n",
        "    return text\n",
        "\n",
        "def convert_label(label):\n",
        "    return 1 if label == \"OFFENSE\" else 0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p690qluXTMyn"
      },
      "source": [
        "train_df[\"text\"]=clean_text(train_df[\"text\"])\n",
        "test_df[\"text\"]=clean_text(test_df[\"text\"])\n",
        "train_df[\"label\"]=train_df[\"label\"].map(convert_label)\n",
        "test_df[\"label\"]=test_df[\"label\"].map(convert_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9BIixoz-TMyn"
      },
      "source": [
        "# this is  how our data set looks now\n",
        "train_df.head() "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HeE1qHXhTMyo"
      },
      "source": [
        "len(train_df.loc[train_df[\"label\"]==1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.utils import shuffle\n",
        "train_df = shuffle(train_df)"
      ],
      "metadata": {
        "id": "XYIke-q7Oqfz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T1_tIVzLTMyo"
      },
      "source": [
        "How many datasets do we have in our Train/Valid/Test sets?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6rCCzWaJTMyo"
      },
      "source": [
        "print(f\"Test exampels \\t {len(test_df) }\")\n",
        "print(f\"Train exampels \\t {len(train_df[500:])}\")\n",
        "print(f\"Valid exampels \\t {len(train_df[:500])}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ObJ7KjhYDxCX"
      },
      "source": [
        "In the next step we convert the data in a format that our ml lib can use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5zRn0t3oTMyp"
      },
      "source": [
        "from datasets import Dataset\n",
        "\n",
        "train_dataset = Dataset.from_pandas(train_df[500:])\n",
        "valid_dataset = Dataset.from_pandas(train_df[:500])\n",
        "test_dataset = Dataset.from_pandas(test_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2qt9p2yeTMyp"
      },
      "source": [
        "# What is the shape of our dataset?\n",
        "train_dataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fUhmAob8TMyp"
      },
      "source": [
        "## Encoding of the data \n",
        "\n",
        "We convert our texts into token that our model can process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jxDv4WeXTMyp"
      },
      "source": [
        "from transformers import AutoTokenizer\n",
        "from datasets import load_dataset, load_metric, list_metrics\n",
        "\n",
        "\n",
        "# try out different models :) \n",
        "\n",
        "model_checkpoint =\"distilbert-base-multilingual-cased\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!rm -rf ./test-offsive-language/checkpoint*"
      ],
      "metadata": {
        "id": "ZFuavLu5UlwZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZkNiW-kITMyp"
      },
      "source": [
        "demo_tokens = tokenizer([\"Mehr Daten führen oftmals zu besseren Ergebnissen.\", \"And this is a second sentence\"],add_special_tokens=True, truncation=True)\n",
        "demo_tokens"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tjSb3j1RTMyp"
      },
      "source": [
        "tokenizer.convert_ids_to_tokens(demo_tokens['input_ids'][0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2CBg2qhVTMyp"
      },
      "source": [
        "def example_tokenizer(examples):\n",
        "    return tokenizer(examples[\"text\"], truncation=True,padding=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tWghamclTMyp"
      },
      "source": [
        "encoded_train_dataset = train_dataset.map(example_tokenizer, batched=True)\n",
        "encoded_valid_dataset = valid_dataset.map(example_tokenizer, batched=True)\n",
        "encoded_test_dataset = test_dataset.map(example_tokenizer, batched=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xhkxSIVLTMyq"
      },
      "source": [
        "## The training \\o/\n",
        "\n",
        "Now we can train our model. To do this, we need to define a number of settings (hyperparameters):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X4DtlapiTMyq"
      },
      "source": [
        "from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer\n",
        "\n",
        "model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint, num_labels=2)\n",
        "\n",
        "batch_size = 8\n",
        "\n",
        "args = TrainingArguments(\n",
        "    \"test-offsive-language\",\n",
        "    evaluation_strategy = \"steps\",\n",
        "    save_strategy= \"steps\",\n",
        "    learning_rate=3e-5,\n",
        "    per_device_train_batch_size=batch_size,\n",
        "    per_device_eval_batch_size=batch_size,\n",
        "    gradient_accumulation_steps=4,\n",
        "    num_train_epochs=2,\n",
        "    eval_steps=50,\n",
        "    save_steps=50,\n",
        "    warmup_steps=50,\n",
        "    logging_steps=10,\n",
        "    weight_decay=0.001,\n",
        "    load_best_model_at_end=True,\n",
        "    overwrite_output_dir=True,\n",
        "    metric_for_best_model=\"f1\",\n",
        "    save_total_limit=2,    \n",
        "    fp16=True    \n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import accuracy_score, f1_score\n",
        "\n",
        "def compute_metrics(pred):\n",
        "  labels = pred.label_ids\n",
        "  preds = pred.predictions.argmax(-1)\n",
        "  f1 = f1_score(labels, preds, average=\"macro\")\n",
        "  acc = accuracy_score(labels, preds)\n",
        "  return {\"accuracy\": acc, \"f1\": f1}"
      ],
      "metadata": {
        "id": "Rh_wjh5TKhY8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g0v5GJXmTMyq"
      },
      "source": [
        "trainer = Trainer(\n",
        "    model,\n",
        "    args,\n",
        "    train_dataset=encoded_train_dataset,\n",
        "    eval_dataset=encoded_valid_dataset,        \n",
        "    tokenizer=tokenizer,\n",
        "    compute_metrics=compute_metrics\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y_lUSvpDTMyq"
      },
      "source": [
        "trainer.train()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i98JNibNTMyr"
      },
      "source": [
        "#trainer.model.to(\"cuda\")\n",
        "trainer.evaluate()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rfEjiV_p7BRh"
      },
      "source": [
        "# How much GPU memory did we use?\n",
        "!nvidia-smi\n",
        "#import torch\n",
        "#torch.cuda.empty_cache()\n",
        "#!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-w-nuYI8TMyr"
      },
      "source": [
        "#tensorboard --logdir runs\n",
        "%load_ext tensorboard\n",
        "#%reload_ext tensorboard\n",
        "%tensorboard --logdir /content/test-offsive-language/runs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tv-PghAYTMyr"
      },
      "source": [
        "## Testing the model\n",
        "\n",
        "The next step is to test the model with the provided test data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jYI9LEbvTMyr"
      },
      "source": [
        "result = trainer.predict(encoded_test_dataset)\n",
        "result.metrics[\"test_f1\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YiYw4kS3TMyr"
      },
      "source": [
        "import torch\n",
        "\n",
        "#trainer.prediction_step(trainer.model,tokenizer(\"das ist ein test\"),False)\n",
        "trainer.model.cpu()\n",
        "#trainer.model.num_parameters()\n",
        "encoded_texts = tokenizer([\"du bist so dumm\", \"du bist toll\"],padding=True, return_tensors=\"pt\")\n",
        "print(encoded_texts)\n",
        "logits = trainer.model(**encoded_texts)\n",
        "probabilities = torch.softmax(logits[0],dim=1)\n",
        "print(probabilities)\n",
        "class_label = torch.argmax(probabilities,dim=1)\n",
        "print(class_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylTzH9P8uu-8"
      },
      "source": [
        "How can we predict a sigle test example and how long does it take on a cpu?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_eyM790HTMyr"
      },
      "source": [
        "def predict(text):\n",
        "    trainer.model.cpu()\n",
        "    #trainer.model.num_parameters()\n",
        "    encoded_texts = tokenizer(text, return_tensors=\"pt\")\n",
        "    #print(encoded_texts)\n",
        "    logits = trainer.model(**encoded_texts)\n",
        "    probabilities = torch.softmax(logits[0],dim=1)\n",
        "    #print(probabilities)\n",
        "    class_label = torch.argmax(probabilities)\n",
        "    return class_label\n",
        "    #print(class_label)\n",
        "\n",
        "%timeit predict(\"du bist so toll\")\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lsL5FUdTTMys"
      },
      "source": [
        "# Tutorial:\n",
        "\n",
        "Our results are already quite good - but we can still improve the results.  First get familiar with the notebook - change a few parameters like learning rate and number of epochs and see how they change the results. \n",
        "\n",
        "**Your task is to improve the classification score.**\n",
        "\n",
        "Here are some ideas how you can improve the score.\n",
        "\n",
        "* Test different models. The [Model Hub](https://huggingface.co/models) lists a number of German models with which you can improve the results. \n",
        "\n",
        "* About 5000 sampels in the data set are comparatively few for this problem. You may find more data sets that you can add to the current training data set.\n",
        "\n",
        "* A number of multilingual models are available in the [Model Hub](https://huggingface.co/models). These models have been trained with different languages. You could also try adding English to the German dataset to train a multilingual model. This may also be better on the German data. \n",
        "\n",
        "Data augmentation is a procedure to create new data sets by modifying existing data sets. It is important that the statement does not change (the class remains the same).\n",
        "\n",
        "* You can replace synonyms words and thus generate new data sets. An example:\n",
        "\n",
        "> \"Can you still believe all this crap?\" -> \"Can you still believe all this crap?\"\n",
        "\n",
        "* Everything is allowed here. Try translating texts from German to English and back to German. If the meaning is preserved, the result can also be used for training. A small example with Google Translate:\n",
        "\n",
        "> Deutsch: \"Kann man diesen ganzen Scheiß noch glauben?\" \n",
        "\n",
        "> Englisch: \"Can you still believe all this shit?\"\n",
        "\n",
        "> Deutsch: \"Kannst du all diese Scheiße noch glauben?\"\n",
        "\n",
        "\n"
      ]
    }
  ]
}