Copy_of_01_how_to_train.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of 01_how-to-train.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/QazQazaq/transformers/blob/master/Copy_of_01_how_to_train.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e67Ut53QYEdU",
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "508869a4-6437-4d51-d9cb-d3f8ea8127de"
      },
      "source": [
        "#@title\n",
        "%%html\n",
        "<div style=\"background-color: pink;\">\n",
        "  Notebook written in collaboration with <a href=\"https://github.com/aditya-malte\">Aditya Malte</a>.\n",
        "  <br>\n",
        "  The Notebook is on GitHub, so contributions are more than welcome.\n",
        "</div>\n",
        "<br>\n",
        "<div style=\"background-color: yellow;\">\n",
        "  Aditya wrote another notebook with a slightly different use case and methodology, please check it out.\n",
        "  <br>\n",
        "  <a target=\"_blank\" href=\"https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b\">\n",
        "    https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b\n",
        "  </a>\n",
        "</div>\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div style=\"background-color: pink;\">\n",
              "  Notebook written in collaboration with <a href=\"https://github.com/aditya-malte\">Aditya Malte</a>.\n",
              "  <br>\n",
              "  The Notebook is on GitHub, so contributions are more than welcome.\n",
              "</div>\n",
              "<br>\n",
              "<div style=\"background-color: yellow;\">\n",
              "  Aditya wrote another notebook with a slightly different use case and methodology, please check it out.\n",
              "  <br>\n",
              "  <a target=\"_blank\" href=\"https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b\">\n",
              "    https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b\n",
              "  </a>\n",
              "</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M1oqh0F6W3ad"
      },
      "source": [
        "# How to train a new language model from scratch using Transformers and Tokenizers\n",
        "\n",
        "### Notebook edition (link to blogpost [link](https://huggingface.co/blog/how-to-train)). Last update May 15, 2020\n",
        "\n",
        "\n",
        "Over the past few months, we made several improvements to our [`transformers`](https://github.com/huggingface/transformers) and [`tokenizers`](https://github.com/huggingface/tokenizers) libraries, with the goal of making it easier than ever to **train a new language model from scratch**.\n",
        "\n",
        "In this post we’ll demo how to train a “small” model (84 M parameters = 6 layers, 768 hidden size, 12 attention heads) – that’s the same number of layers & heads as DistilBERT – on **Esperanto**. We’ll then fine-tune the model on a downstream task of part-of-speech tagging.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oK7PPVm2XBgr"
      },
      "source": [
        "## 1. Find a dataset\n",
        "\n",
        "First, let us find a corpus of text in Esperanto. Here we’ll use the Esperanto portion of the [OSCAR corpus](https://traces1.inria.fr/oscar/) from INRIA.\n",
        "OSCAR is a huge multilingual corpus obtained by language classification and filtering of [Common Crawl](https://commoncrawl.org/) dumps of the Web.\n",
        "\n",
        "<img src=\"https://huggingface.co/blog/assets/01_how-to-train/oscar.png\" style=\"margin: auto; display: block; width: 260px;\">\n",
        "\n",
        "The Esperanto portion of the dataset is only 299M, so we’ll concatenate with the Esperanto sub-corpus of the [Leipzig Corpora Collection](https://wortschatz.uni-leipzig.de/en/download), which is comprised of text from diverse sources like news, literature, and wikipedia.\n",
        "\n",
        "The final training corpus has a size of 3 GB, which is still small – for your model, you will get better results the more data you can get to pretrain on. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HOk4iZ9YZvec",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "51da9a50-0c20-4e9b-ac55-00d485bf93e8"
      },
      "source": [
        "# in this notebook we'll only get one of the files (the Oscar one) for the sake of simplicity and performance\n",
        "!wget -c https://cdn-datasets.huggingface.co/oscar/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-03-11 14:32:44--  https://cdn-datasets.huggingface.co/oscar/\n",
            "Resolving cdn-datasets.huggingface.co (cdn-datasets.huggingface.co)... 13.226.50.79, 13.226.50.50, 13.226.50.129, ...\n",
            "Connecting to cdn-datasets.huggingface.co (cdn-datasets.huggingface.co)|13.226.50.79|:443... connected.\n",
            "HTTP request sent, awaiting response... 404 Not Found\n",
            "2021-03-11 14:32:44 ERROR 404: Not Found.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0TKaTdOEjU8V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "66764431-dfb1-4694-92ce-88d1d645bf49"
      },
      "source": [
        "!wget https://object.pouta.csc.fi/Tatoeba-Challenge/abk-eng.tar\r\n",
        "import tarfile\r\n",
        "my_tar = tarfile.open('abk-eng.tar')\r\n",
        "my_tar.extractall()\r\n",
        "import gzip\r\n",
        "\r\n",
        "input = gzip.GzipFile(\"/content/data/abk-eng/train.src.gz\", 'rb')\r\n",
        "s = input.read()\r\n",
        "input.close()\r\n",
        "output = open(\"/content/data/abk-eng/train.src\", 'wb')\r\n",
        "output.write(s)\r\n",
        "output.close()\r\n",
        "\r\n",
        "print(\"done\")\r\n",
        "import gzip\r\n",
        "\r\n",
        "input = gzip.GzipFile(\"/content/data/abk-eng/train.trg.gz\", 'rb')\r\n",
        "s = input.read()\r\n",
        "input.close()\r\n",
        "output = open(\"/content/data/abk-eng/train.trg\", 'wb')\r\n",
        "output.write(s)\r\n",
        "output.close()\r\n",
        "\r\n",
        "print(\"done\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-03-11 16:10:14--  https://object.pouta.csc.fi/Tatoeba-Challenge/abk-eng.tar\n",
            "Resolving object.pouta.csc.fi (object.pouta.csc.fi)... 86.50.254.18, 86.50.254.19\n",
            "Connecting to object.pouta.csc.fi (object.pouta.csc.fi)|86.50.254.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1792000 (1.7M) [application/x-tar]\n",
            "Saving to: ‘abk-eng.tar’\n",
            "\n",
            "abk-eng.tar         100%[===================>]   1.71M  1.72MB/s    in 1.0s    \n",
            "\n",
            "2021-03-11 16:10:18 (1.72 MB/s) - ‘abk-eng.tar’ saved [1792000/1792000]\n",
            "\n",
            "done\n",
            "done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G-kkz81OY6xH"
      },
      "source": [
        "## 2. Train a tokenizer\n",
        "\n",
        "We choose to train a byte-level Byte-pair encoding tokenizer (the same as GPT-2), with the same special tokens as RoBERTa. Let’s arbitrarily pick its size to be 52,000.\n",
        "\n",
        "We recommend training a byte-level BPE (rather than let’s say, a WordPiece tokenizer like BERT) because it will start building its vocabulary from an alphabet of single bytes, so all words will be decomposable into tokens (no more `<unk>` tokens!).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5duRggBRZKvP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 129
        },
        "outputId": "db38f186-ec9c-404b-8666-6e73ba06624f"
      },
      "source": [
        "# We won't need TensorFlow here\n",
        "!pip uninstall -y tensorflow\n",
        "# Install `transformers` from master\n",
        "!pip install git+https://github.com/huggingface/transformers\n",
        "!piamp list | grep -E 'transformers|tokenizers'\n",
        "# transformers version at notebook update --- 2.11.0\n",
        "# tokenizers version at notebook update --- 0.8.0rc1\n",
        "\n",
        "transformers-cli login\n",
        "\n"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "error",
          "ename": "SyntaxError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-37-581f14eedc2f>\"\u001b[0;36m, line \u001b[0;32m9\u001b[0m\n\u001b[0;31m    transformers-cli login\u001b[0m\n\u001b[0m                         ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IMnymRDLe0hi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8ed11820-1103-443b-c3cc-ea1a8e6073cf"
      },
      "source": [
        "%%time \n",
        "#from pathlib import Path\n",
        "\n",
        "from tokenizers import ByteLevelBPETokenizer\n",
        "\n",
        "#paths = [str(x) for x in Path(\".\").glob(\"**/*.txt\")]\n",
        "\n",
        "# Initialize a tokenizer\n",
        "tokenizer = ByteLevelBPETokenizer()\n",
        "\n",
        "# Customize training\n",
        "tokenizer.train(files='/content/data/abk-eng/train.src', vocab_size=52_000, min_frequency=2, special_tokens=[\n",
        "    \"<s>\",\n",
        "    \"<pad>\",\n",
        "    \"</s>\",\n",
        "    \"<unk>\",\n",
        "    \"<mask>\",\n",
        "])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 6.08 s, sys: 1.02 s, total: 7.1 s\n",
            "Wall time: 2.31 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Ei7bqpRf1LH"
      },
      "source": [
        "Now let's save files to disk"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EIS-irI0f32P",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a50a2e18-4e0a-4384-f249-4efe05f3cff3"
      },
      "source": [
        "!mkdir EsperBERTo\n",
        "tokenizer.save_model(\"EsperBERTo\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['EsperBERTo/vocab.json', 'EsperBERTo/merges.txt']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lOOfYSuQhSqT"
      },
      "source": [
        "🔥🔥 Wow, that was fast! ⚡️🔥\n",
        "\n",
        "We now have both a `vocab.json`, which is a list of the most frequent tokens ranked by frequency, and a `merges.txt` list of merges.\n",
        "\n",
        "```json\n",
        "{\n",
        "\t\"<s>\": 0,\n",
        "\t\"<pad>\": 1,\n",
        "\t\"</s>\": 2,\n",
        "\t\"<unk>\": 3,\n",
        "\t\"<mask>\": 4,\n",
        "\t\"!\": 5,\n",
        "\t\"\\\"\": 6,\n",
        "\t\"#\": 7,\n",
        "\t\"$\": 8,\n",
        "\t\"%\": 9,\n",
        "\t\"&\": 10,\n",
        "\t\"'\": 11,\n",
        "\t\"(\": 12,\n",
        "\t\")\": 13,\n",
        "\t# ...\n",
        "}\n",
        "\n",
        "# merges.txt\n",
        "l a\n",
        "Ġ k\n",
        "o n\n",
        "Ġ la\n",
        "t a\n",
        "Ġ e\n",
        "Ġ d\n",
        "Ġ p\n",
        "# ...\n",
        "```\n",
        "\n",
        "What is great is that our tokenizer is optimized for Esperanto. Compared to a generic tokenizer trained for English, more native words are represented by a single, unsplit token. Diacritics, i.e. accented characters used in Esperanto – `ĉ`, `ĝ`, `ĥ`, `ĵ`, `ŝ`, and `ŭ` – are encoded natively. We also represent sequences in a more efficient manner. Here on this corpus, the average length of encoded sequences is ~30% smaller as when using the pretrained GPT-2 tokenizer.\n",
        "\n",
        "Here’s  how you can use it in `tokenizers`, including handling the RoBERTa special tokens – of course, you’ll also be able to use it directly from `transformers`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tKVWB8WShT-z"
      },
      "source": [
        "from tokenizers.implementations import ByteLevelBPETokenizer\n",
        "from tokenizers.processors import BertProcessing\n",
        "\n",
        "\n",
        "tokenizer = ByteLevelBPETokenizer(\n",
        "    \"./EsperBERTo/vocab.json\",\n",
        "    \"./EsperBERTo/merges.txt\",\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hO5M3vrAhcuj"
      },
      "source": [
        "tokenizer._tokenizer.post_processor = BertProcessing(\n",
        "    (\"</s>\", tokenizer.token_to_id(\"</s>\")),\n",
        "    (\"<s>\", tokenizer.token_to_id(\"<s>\")),\n",
        ")\n",
        "tokenizer.enable_truncation(max_length=512)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E3Ye27nchfzq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9350409e-20ff-43e4-cd69-09c113e73343"
      },
      "source": [
        "tokenizer.encode(\"Mi estas Julien.\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Encoding(num_tokens=13, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X8ya5_7rhjKS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "428d00c0-ece4-4452-e33d-4cd28af78f43"
      },
      "source": [
        "tokenizer.encode(\"Mi estas Julien.\").tokens"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<s>', 'M', 'i', 'Ġ', 'est', 'as', 'Ġ', 'J', 'ul', 'i', 'en', '.', '</s>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WQpUC_CDhnWW"
      },
      "source": [
        "## 3. Train a language model from scratch\n",
        "\n",
        "**Update:** This section follows along the [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/legacy/run_language_modeling.py) script, using our new [`Trainer`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py) directly. Feel free to pick the approach you like best.\n",
        "\n",
        "> We’ll train a RoBERTa-like model, which is a BERT-like with a couple of changes (check the [documentation](https://huggingface.co/transformers/model_doc/roberta.html) for more details).\n",
        "\n",
        "As the model is BERT-like, we’ll train it on a task of *Masked language modeling*, i.e. the predict how to fill arbitrary tokens that we randomly mask in the dataset. This is taken care of by the example script.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kD140sFjh0LQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "857f02d6-aaef-4ca6-9373-a48f54863f92"
      },
      "source": [
        "# Check that we have a GPU\n",
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Thu Mar 11 16:11:05 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.56       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   36C    P0    27W / 250W |      0MiB / 16280MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VNZZs-r6iKAV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3b2639da-c523-433a-ac11-cec9d09e2897"
      },
      "source": [
        "# Check that PyTorch sees it\n",
        "import torch\n",
        "torch.cuda.is_available()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u0qQzgrBi1OX"
      },
      "source": [
        "### We'll define the following config for the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LTXXutqeDzPi"
      },
      "source": [
        "from transformers import RobertaConfig\n",
        "\n",
        "config = RobertaConfig(\n",
        "    vocab_size=52_000,\n",
        "    max_position_embeddings=514,\n",
        "    num_attention_heads=12,\n",
        "    num_hidden_layers=6,\n",
        "    type_vocab_size=1,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yAwQ82JiE5pi"
      },
      "source": [
        "Now let's re-create our tokenizer in transformers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4keFBUjQFOD1"
      },
      "source": [
        "from transformers import RobertaTokenizerFast\n",
        "\n",
        "tokenizer = RobertaTokenizerFast.from_pretrained(\"./EsperBERTo\", max_len=512)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6yNCw-3hFv9h"
      },
      "source": [
        "Finally let's initialize our model.\n",
        "\n",
        "**Important:**\n",
        "\n",
        "As we are training from scratch, we only initialize from a config, not from an existing pretrained model or checkpoint."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BzMqR-dzF4Ro"
      },
      "source": [
        "from transformers import RobertaForMaskedLM\n",
        "\n",
        "model = RobertaForMaskedLM(config=config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jU6JhBSTKiaM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "810d0dee-e50e-47ed-e142-1750e2251701"
      },
      "source": [
        "model.num_parameters()\n",
        "# => 84 million parameters"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "83504416"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jBtUHRMliOLM"
      },
      "source": [
        "### Now let's build our training Dataset\n",
        "\n",
        "We'll build our dataset by applying our tokenizer to our text file.\n",
        "\n",
        "Here, as we only have one text file, we don't even need to customize our `Dataset`. We'll just use the `LineByLineDataset` out-of-the-box."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GlvP_A-THEEl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f4924cd1-0336-4c42-a313-557283f47292"
      },
      "source": [
        "%%time\n",
        "from transformers import LineByLineTextDataset\n",
        "\n",
        "dataset = LineByLineTextDataset(\n",
        "    tokenizer=tokenizer,\n",
        "    file_path=\"/content/data/abk-eng/train.src\",\n",
        "    block_size=128,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/data/datasets/language_modeling.py:128: FutureWarning: This dataset will be removed from the library soon, preprocessing should be handled with the 🤗 Datasets library. You can have a look at this example script for pointers: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py\n",
            "  FutureWarning,\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 2.99 s, sys: 532 ms, total: 3.52 s\n",
            "Wall time: 1.79 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hDLs73HcIHk5"
      },
      "source": [
        "Like in the [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) script, we need to define a data_collator.\n",
        "\n",
        "This is just a small helper that will help us batch different samples of the dataset together into an object that PyTorch knows how to perform backprop on."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zTgWPa9Dipk2"
      },
      "source": [
        "from transformers import DataCollatorForLanguageModeling\n",
        "\n",
        "data_collator = DataCollatorForLanguageModeling(\n",
        "    tokenizer=tokenizer, mlm=True, mlm_probability=0.15\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ri2BIQKqjfHm"
      },
      "source": [
        "### Finally, we are all set to initialize our Trainer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YpvnFFmZJD-N"
      },
      "source": [
        "from transformers import Trainer, TrainingArguments\n",
        "\n",
        "training_args = TrainingArguments(\n",
        "    output_dir=\"./EsperBERTo\",\n",
        "    overwrite_output_dir=True,\n",
        "    num_train_epochs=1,\n",
        "    per_gpu_train_batch_size=64,\n",
        "    save_steps=10_000,\n",
        "    save_total_limit=2,\n",
        ")\n",
        "\n",
        "trainer = Trainer(\n",
        "    model=model,\n",
        "    args=training_args,\n",
        "    data_collator=data_collator,\n",
        "    train_dataset=dataset,\n",
        "    #prediction_loss_only=True\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o6sASa36Nf-N"
      },
      "source": [
        "### Start training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VmaHZXzmkNtJ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 199
        },
        "outputId": "d63cd58b-60cb-42cf-f573-aef1fcd43428"
      },
      "source": [
        "%%time\n",
        "trainer.train()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.\n",
            "Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.\n",
            "Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "        </style>\n",
              "      \n",
              "      <progress value='393' max='393' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [393/393 02:49, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 1min 6s, sys: 1min 43s, total: 2min 50s\n",
            "Wall time: 2min 50s\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TrainOutput(global_step=393, training_loss=8.391918386211833, metrics={'train_runtime': 170.191, 'train_samples_per_second': 2.309, 'total_flos': 602296977530496.0, 'epoch': 1.0, 'init_mem_cpu_alloc_delta': 2573358, 'init_mem_gpu_alloc_delta': 334180352, 'init_mem_cpu_peaked_delta': 18306, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 586754, 'train_mem_gpu_alloc_delta': 1010782208, 'train_mem_cpu_peaked_delta': 1281085, 'train_mem_gpu_peaked_delta': 10426161152})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ZkooHz1-_2h"
      },
      "source": [
        "#### 🎉 Save final model (+ tokenizer + config) to disk"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QDNgPls7_l13"
      },
      "source": [
        "trainer.save_model(\"./EsperBERTo\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pA9V3G_aCRrC"
      },
      "source": [
        "from torch.utils.data import Dataset\r\n",
        "\r\n",
        "class EsperantoDataset(Dataset):\r\n",
        "    def __init__(self, evaluate: bool = False):\r\n",
        "        tokenizer = ByteLevelBPETokenizer(\r\n",
        "            \"./models/EsperBERTo-small/vocab.json\",\r\n",
        "            \"./models/EsperBERTo-small/merges.txt\",\r\n",
        "        )\r\n",
        "        tokenizer._tokenizer.post_processor = BertProcessing(\r\n",
        "            (\"</s>\", tokenizer.token_to_id(\"</s>\")),\r\n",
        "            (\"<s>\", tokenizer.token_to_id(\"<s>\")),\r\n",
        "        )\r\n",
        "        tokenizer.enable_truncation(max_length=512)\r\n",
        "        # or use the RobertaTokenizer from `transformers` directly.\r\n",
        "\r\n",
        "        self.examples = []\r\n",
        "\r\n",
        "        src_files = Path(\"./data/\").glob(\"*-eval.txt\") if evaluate else Path(\"./data/\").glob(\"*-train.txt\")\r\n",
        "        for src_file in src_files:\r\n",
        "            print(\"🔥\", src_file)\r\n",
        "            lines = src_file.read_text(encoding=\"utf-8\").splitlines()\r\n",
        "            self.examples += [x.ids for x in tokenizer.encode_batch(lines)]\r\n",
        "\r\n",
        "    def __len__(self):\r\n",
        "        return len(self.examples)\r\n",
        "\r\n",
        "    def __getitem__(self, i):\r\n",
        "        # We’ll pad at the batch level.\r\n",
        "        return torch.tensor(self.examples[i])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wod9q_pHCRtY"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d0caceCy_p1-"
      },
      "source": [
        "## 4. Check that the LM actually trained"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iIQJ8ND_AEhl"
      },
      "source": [
        "Aside from looking at the training and eval losses going down, the easiest way to check whether our language model is learning anything interesting is via the `FillMaskPipeline`.\n",
        "\n",
        "Pipelines are simple wrappers around tokenizers and models, and the 'fill-mask' one will let you input a sequence containing a masked token (here, `<mask>`) and return a list of the most probable filled sequences, with their probabilities.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ltXgXyCbAJLY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c64f6bd2-5001-4dc1-9746-effcd3d2d25d"
      },
      "source": [
        "from transformers import pipeline\n",
        "\n",
        "fill_mask = pipeline(\n",
        "    \"fill-mask\",\n",
        "    model=\"./EsperBERTo\",\n",
        "    tokenizer=\"./EsperBERTo\"\n",
        ")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of RobertaModel were not initialized from the model checkpoint at ./EsperBERTo and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UIvgZ3S6AO0z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cf809780-74d3-4c86-bcbc-f33f2b774efb"
      },
      "source": [
        "# The sun <mask>.\n",
        "# =>\n",
        "\n",
        "fill_mask(\"Иҟалап Давид игәҭакы ахьизынамыгӡаз <mask>.\")"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'score': 0.04219769313931465,\n",
              "  'sequence': 'Иҟалап Давид игәҭакы ахьизынамыгӡаз,.',\n",
              "  'token': 16,\n",
              "  'token_str': ','},\n",
              " {'score': 0.01041566114872694,\n",
              "  'sequence': 'Иҟалап Давид игәҭакы ахьизынамыгӡаз уи.',\n",
              "  'token': 375,\n",
              "  'token_str': ' уи'},\n",
              " {'score': 0.009070862084627151,\n",
              "  'sequence': 'Иҟалап Давид игәҭакы ахьизынамыгӡаз Иегова.',\n",
              "  'token': 370,\n",
              "  'token_str': ' Иегова'},\n",
              " {'score': 0.008338535204529762,\n",
              "  'sequence': 'Иҟалап Давид игәҭакы ахьизынамыгӡаз насгьы.',\n",
              "  'token': 391,\n",
              "  'token_str': ' насгьы'},\n",
              " {'score': 0.005623785313218832,\n",
              "  'sequence': 'Иҟалап Давид игәҭакы ахьизынамыгӡаз ауаа.',\n",
              "  'token': 459,\n",
              "  'token_str': ' ауаа'}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i0qCyyhNAWZi"
      },
      "source": [
        "Ok, simple syntax/grammar works. Let’s try a slightly more interesting prompt:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YZ9HSQxAAbme",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0ebed3dc-63c6-49c7-8cc4-656add0dea49"
      },
      "source": [
        "fill_mask(\"Иҟалап Давид  <mask>.\")\n",
        "\n",
        "# This is the beginning of a beautiful <mask>.\n",
        "# =>"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'score': 0.036667462438344955,\n",
              "  'sequence': 'Иҟалап Давид,.',\n",
              "  'token': 16,\n",
              "  'token_str': ','},\n",
              " {'score': 0.020548192784190178,\n",
              "  'sequence': 'Иҟалап Давид  Иегова.',\n",
              "  'token': 370,\n",
              "  'token_str': ' Иегова'},\n",
              " {'score': 0.013288303278386593,\n",
              "  'sequence': 'Иҟалап Давид  уи.',\n",
              "  'token': 375,\n",
              "  'token_str': ' уи'},\n",
              " {'score': 0.009317377582192421,\n",
              "  'sequence': 'Иҟалап Давид  Анцәа.',\n",
              "  'token': 397,\n",
              "  'token_str': ' Анцәа'},\n",
              " {'score': 0.008055318146944046,\n",
              "  'sequence': 'Иҟалап Давид :.',\n",
              "  'token': 30,\n",
              "  'token_str': ':'}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L6E6iZfuTWhL",
        "outputId": "7f8d44a5-a1d2-46c6-e88d-5a530aa9d535"
      },
      "source": [
        "pip install git-lfs"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting git-lfs\n",
            "  Downloading https://files.pythonhosted.org/packages/eb/40/cd243be7ba9bd9d83fd8515b1aed7e0b822c76cab4bf60398a1b7e024f00/git_lfs-1.6-py2.py3-none-any.whl\n",
            "Installing collected packages: git-lfs\n",
            "Successfully installed git-lfs-1.6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v599G-TLRwSk",
        "outputId": "cca77751-dc24-4632-c403-edb1f049f3ed"
      },
      "source": [
        "\r\n",
        "\r\n",
        "\r\n",
        "!git clone https://huggingface.co/QA/Ab\r\n",
        "\r\n"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "git: 'lfs' is not a git command. See 'git --help'.\n",
            "\n",
            "The most similar command is\n",
            "\tlog\n",
            "fatal: destination path 'Ab' already exists and is not an empty directory.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pA80eXv0UEZW",
        "outputId": "f2958b1e-af5f-4499-ac7b-0325b4c578f5"
      },
      "source": [
        "trainer.save_model(\"https://huggingface.co/QA/Ab/blob/main\")\r\n",
        "tokenizer.save_pretrained(\"https://huggingface.co/QA/Ab/blob/main\")"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('https://huggingface.co/QA/Ab/blob/main/tokenizer_config.json',\n",
              " 'https://huggingface.co/QA/Ab/blob/main/special_tokens_map.json',\n",
              " 'https://huggingface.co/QA/Ab/blob/main/vocab.json',\n",
              " 'https://huggingface.co/QA/Ab/blob/main/merges.txt',\n",
              " 'https://huggingface.co/QA/Ab/blob/main/added_tokens.json')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 129
        },
        "id": "ztlHMGbBUEgw",
        "outputId": "4b5f1935-9382-4758-82a3-0afa6e9735b8"
      },
      "source": [
        "sudo apt-get install git-lfs"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "error",
          "ename": "SyntaxError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-48-becfd580ee34>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    sudo apt-get install git-lfs\u001b[0m\n\u001b[0m           ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9Zt9SUBdVZCy",
        "outputId": "1c5ea7d6-a65b-4add-8f13-46f2e79ae97a"
      },
      "source": [
        "!git clone https://QA:Bug718is@huggingface.co/QA/Ab"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fatal: destination path 'Ab' already exists and is not an empty directory.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W7Gy1KilV1ya",
        "outputId": "90122531-899c-49dc-e787-7450e81d93ee"
      },
      "source": [
        "cd Ab"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/Ab\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ye1cJFh1WEGm",
        "outputId": "57a9cbfd-ce65-4337-c88e-6ffab33c873e"
      },
      "source": [
        "!sudo apt-get install git-lfs"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following NEW packages will be installed:\n",
            "  git-lfs\n",
            "0 upgraded, 1 newly installed, 0 to remove and 29 not upgraded.\n",
            "Need to get 2,129 kB of archives.\n",
            "After this operation, 7,662 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 git-lfs amd64 2.3.4-1 [2,129 kB]\n",
            "Fetched 2,129 kB in 1s (1,572 kB/s)\n",
            "debconf: unable to initialize frontend: Dialog\n",
            "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)\n",
            "debconf: falling back to frontend: Readline\n",
            "debconf: unable to initialize frontend: Readline\n",
            "debconf: (This frontend requires a controlling tty.)\n",
            "debconf: falling back to frontend: Teletype\n",
            "dpkg-preconfigure: unable to re-open stdin: \n",
            "Selecting previously unselected package git-lfs.\n",
            "(Reading database ... 160975 files and directories currently installed.)\n",
            "Preparing to unpack .../git-lfs_2.3.4-1_amd64.deb ...\n",
            "Unpacking git-lfs (2.3.4-1) ...\n",
            "Setting up git-lfs (2.3.4-1) ...\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rsmd0ImaiKNf"
      },
      "source": [
        "!transformers-cli login\r\n",
        "!transformers-cli repo create Abkho"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tEamra4UXnbp"
      },
      "source": [
        "!git clone https://QA:Bug718is@huggingface.co/QA/Abkho\r\n",
        "!cd Abkho\r\n",
        "!git config --global user.email \"QA:Bug718is@huggingface.co\"\r\n",
        "# Tip: using the same email than for your huggingface.co account will link your commits to your profile\r\n",
        "!git config --global user.name \"QA\"\r\n",
        "!git add .\r\n",
        "!git commit -m \"Initial commit\"\r\n",
        "!git push"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RZziOY_uXneI",
        "outputId": "5f066881-9176-483a-b214-af422672d098"
      },
      "source": [
        "!git add .\r\n",
        "!git commit -m \"Initial commit\"\r\n",
        "!git push\r\n"
      ],
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "On branch main\n",
            "Your branch is ahead of 'origin/main' by 4 commits.\n",
            "  (use \"git push\" to publish your local commits)\n",
            "\n",
            "nothing to commit, working tree clean\n",
            "fatal: could not read Username for 'https://huggingface.co': No such device or address\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j69vJ_FieKQH",
        "outputId": "dad437d5-bbba-4473-c542-372ba55c3ec6"
      },
      "source": [
        "model.save_pretrained(\"https://huggingface.co/QA/Abkho\")\r\n",
        "tokenizer.save_pretrained(\"https://huggingface.co/QA/Abkho\")"
      ],
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('https://huggingface.co/QA/Abkha/tokenizer_config.json',\n",
              " 'https://huggingface.co/QA/Abkha/special_tokens_map.json',\n",
              " 'https://huggingface.co/QA/Abkha/vocab.json',\n",
              " 'https://huggingface.co/QA/Abkha/merges.txt',\n",
              " 'https://huggingface.co/QA/Abkha/added_tokens.json')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "15EURywxgEWI",
        "outputId": "726ab983-9b85-4c8a-e9c7-93ec12d36b51"
      },
      "source": [
        "!git add --all\r\n",
        "!git status\r\n"
      ],
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "On branch main\n",
            "Your branch is ahead of 'origin/main' by 4 commits.\n",
            "  (use \"git push\" to publish your local commits)\n",
            "\n",
            "Changes to be committed:\n",
            "  (use \"git reset HEAD <file>...\" to unstage)\n",
            "\n",
            "\t\u001b[32mnew file:   https:/huggingface.co/QA/Abkha/config.json\u001b[m\n",
            "\t\u001b[32mnew file:   https:/huggingface.co/QA/Abkha/merges.txt\u001b[m\n",
            "\t\u001b[32mnew file:   https:/huggingface.co/QA/Abkha/pytorch_model.bin\u001b[m\n",
            "\t\u001b[32mnew file:   https:/huggingface.co/QA/Abkha/special_tokens_map.json\u001b[m\n",
            "\t\u001b[32mnew file:   https:/huggingface.co/QA/Abkha/tokenizer_config.json\u001b[m\n",
            "\t\u001b[32mnew file:   https:/huggingface.co/QA/Abkha/vocab.json\u001b[m\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "psJUstthdKeU",
        "outputId": "a68c0030-85ed-4467-d5c4-e81df68ec15b"
      },
      "source": [
        "!transformers-cli upload"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[1m\u001b[31mDeprecated: used to be the way to upload a model to S3. We now use a git-based system for storing models and other artifacts. Use the `repo create` command instead.\u001b[0m\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6RsGaD1qAfLP"
      },
      "source": [
        "## 5. Share your model 🎉"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5oESe8djApQw"
      },
      "source": [
        "Finally, when you have a nice model, please think about sharing it with the community:\n",
        "\n",
        "- upload your model using the CLI: `transformers-cli upload`\n",
        "- write a README.md model card and add it to the repository under `model_cards/`. Your model card should ideally include:\n",
        "    - a model description,\n",
        "    - training params (dataset, preprocessing, hyperparameters), \n",
        "    - evaluation results,\n",
        "    - intended uses & limitations\n",
        "    - whatever else is helpful! 🤓\n",
        "\n",
        "### **TADA!**\n",
        "\n",
        "➡️ Your model has a page on http://huggingface.co/models and everyone can load it using `AutoModel.from_pretrained(\"username/model_name\")`.\n",
        "\n",
        "[![tb](https://huggingface.co/blog/assets/01_how-to-train/model_page.png)](https://huggingface.co/julien-c/EsperBERTo-small)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aw9ifsgqBI2o"
      },
      "source": [
        "If you want to take a look at models in different languages, check https://huggingface.co/models\n",
        "\n",
        "[![all models](https://huggingface.co/front/thumbnails/models.png)](https://huggingface.co/models)\n"
      ]
    }
  ]
}