FineTuningLLMs101.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xeKthJYgEQg_"
   },
   "source": [
    "![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/title.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mRrjIkhQEQhA"
   },
   "source": [
    "## Who Am I?\n",
    "\n",
    "Data scientist, teacher, author of the ***Deep Learning with PyTorch Step-by-Step*** series.\n",
    "\n",
    "[![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/new_books.png)](https://pytorchstepbystep.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mJcITc7sEQhB"
   },
   "source": [
    "### Upcoming Book: The Ultimate Guide to Fine-Tuning LLMs with PyTorch and Hugging Face"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qb09p005EQhC"
   },
   "source": [
    "<center><a href=\"https://leanpub.com/finetuning\"><img src=\"https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/cover.png\" width=\"200\"/></a></center>\n",
    "<br>\n",
    "<h3><center>Visit <a href=\"https://leanpub.com/finetuning\">leanpub.com/finetuning</a> and get a discount coupon when it's released!</center></h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E2oovLQ2EQhD"
   },
   "source": [
    "## Agenda\n",
    "\n",
    "- Why Fine-Tune?\n",
    "- Types of Fine-Tuning\n",
    "- The Main Challenge\n",
    "- Loading a Quantized Base Model\n",
    "- Formatting Your Dataset\n",
    "- Setting Up Low-Rank Adapters (LoRA)\n",
    "- Fine-Tuning with SFTTrainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XGGHYHAuEQhD"
   },
   "source": [
    "## Why Fine-Tune?\n",
    "\n",
    "- Pretrained (base) models are **next word predictors**: they do not understand **questions** or **instructions** naturally.\n",
    "\n",
    "\n",
    "- If prompted with a question, a base model is likely to reply with even **more questions** insteaf of answers.\n",
    "\n",
    "\n",
    "- Fine-tuning can be used to:\n",
    "  - **change the behavior** of the model\n",
    "  - **change the style** of its answers\n",
    "  - **increase its knowledge**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "umVXbIp5EQhE"
   },
   "source": [
    "## Types of Fine-Tuning\n",
    "\n",
    "- **Self-supervised fine-tuning**: it works by training the model to predict the next word - just like in (base) model pretraining.\n",
    "  - it may be used to change **the style** of a model's answers to the user prompt\n",
    "  - in self-supervised fine-tuning, **the labels are the same as the inputs**, except that they are **shifted** by one position\n",
    "\n",
    "\n",
    "- **Supervised fine-tuning**: it works by training the model on **pairs of inputs/outputs** such as **question/answer** or **prompt/response**.\n",
    "  - it may be used to teach the model how to **respond to user prompts** related to a given **domain**\n",
    "\n",
    "\n",
    "- **Reinforcement-based fine-tuning**: it works by training the model on **labeled outputs** (good/bad or appropriate/inappropriate)\n",
    "  - it may be used to **align the model** or **improve its answers**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BZdYvjXsEQhE"
   },
   "source": [
    "## The Main Challenge\n",
    "\n",
    "Fine-tuning LLMs feels like trying to fit this:\n",
    "\n",
    "![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/elephant.png)\n",
    "\n",
    "into this:\n",
    "\n",
    "![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/beetle.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YzcOwMvcEQhF"
   },
   "source": [
    "### Attention Is All You Need\n",
    "\n",
    "- LLMs are Transformer Decoders\n",
    "- The majority of the model's parameters are in the attention heads\n",
    "\n",
    "![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/stacked_layers.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G48picD2EQhG"
   },
   "source": [
    "### We have MANY problems!\n",
    "\n",
    "1. Base model takes a lot of space (p)\n",
    "2. Optimizer's states (Adam) takes even more space (2p)\n",
    "3. Gradients take a lot of space (p)\n",
    "4. Activations take a lot of space, especially if the sequences are long\n",
    "5. Attention is quadratic on the sequence length (10x longer => 100x more expensive to handle)\n",
    "\n",
    "p = # of parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ekIGgo1PEQhG"
   },
   "source": [
    "### Combine many solutions, we must!\n",
    "\n",
    "1. ~~Base model takes a lot of space (p)~~\n",
    "\n",
    "- **Solution**: **Quantization** reduces the model's memory footprint.\n",
    "\n",
    "2. ~~Optimizer's states (Adam) takes even more space (2p)~~\n",
    "\n",
    "- **Solutions**: While **8-bit Adam** may reduce the optimizer's memory footprint (especially if you're updating all parameters), **LoRA** drastically reduces the number of trainable parameters, thus making the optimizer a non-issue,\n",
    "\n",
    "3. ~~Gradients take a lot of space (p)~~\n",
    "\n",
    "- **Solutions**: **Checkpointing** trades compute for memory (discarding and recomputing values as they are needed, instead of caching them) and **LoRA** reduces the number of trainable parameters, so fewer gradients are computed overall.\n",
    "\n",
    "4. ~~Activations take a lot of space, especially if the sequences are long~~\n",
    "\n",
    "- **Solution**: **Checkpointing** once again trades compute for memory.\n",
    "\n",
    "5. ~~Attention is quadratic on the sequence length (10x longer => 100x more expensive to handle)~~\n",
    "\n",
    "- **Solution**: **Flash Attention** removes several inefficiencies of the original implementation, thus making memory requirements **linearly** proportional to sequence length! However, it only works on higher-end GPUs (it won't work on Colab)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QeaxyMxIEQhG"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kODUm5BmEQhI"
   },
   "outputs": [],
   "source": [
    "!pip install -U transformers datasets trl bitsandbytes peft matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "LE3FafblEQhI"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
    "from datasets import load_dataset\n",
    "from trl import SFTConfig, SFTTrainer, setup_chat_format\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BGxWM_J9EQhJ"
   },
   "source": [
    "## Loading a Quantized Based Model\n",
    "\n",
    "### Model's Parameters (FP32)\n",
    "\n",
    "![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/quant1.png)\n",
    "\n",
    "Source:[Quantization Fundamentals with Hugging Face](https://www.deeplearning.ai/short-courses/quantization-fundamentals-with-hugging-face/)\n",
    "\n",
    "### Quantizing Parameters (FP32 -> INT8)\n",
    "![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/quant2.png)\n",
    "\n",
    "Source:[Quantization Fundamentals with Hugging Face](https://www.deeplearning.ai/short-courses/quantization-fundamentals-with-hugging-face/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xbPZQ4aNEQhJ",
    "outputId": "ecd1995a-795c-43a2-a5e4-53c5519d1786"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`low_cpu_mem_usage` was None, now default to True since model is quantized.\n"
     ]
    }
   ],
   "source": [
    "bnb_config = BitsAndBytesConfig(load_in_8bit=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(\"facebook/opt-350m\",\n",
    "                                             quantization_config=bnb_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8AOI2ZnmFdzH"
   },
   "source": [
    "The quantized model takes up roughly 360 Mb or RAM, which is a little bit over 1/4 of its original size (1324 Mb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5WXLJ9QNEQhJ",
    "outputId": "4f1cabc5-1509-4dc3-c49f-6c38f9442e7a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "359.354368\n"
     ]
    }
   ],
   "source": [
    "print(model.get_memory_footprint()/1e6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3gAVQKRsFHpt"
   },
   "source": [
    "Notice that all `Linear` layers were replaced by their `Linear8bitLt` quantized versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Oe2hIoBbEQhJ",
    "outputId": "df045aaa-e493-41d0-8213-b5f25c37923d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OPTForCausalLM(\n",
       "  (model): OPTModel(\n",
       "    (decoder): OPTDecoder(\n",
       "      (embed_tokens): Embedding(50272, 512, padding_idx=1)\n",
       "      (embed_positions): OPTLearnedPositionalEmbedding(2050, 1024)\n",
       "      (project_out): Linear8bitLt(in_features=1024, out_features=512, bias=False)\n",
       "      (project_in): Linear8bitLt(in_features=512, out_features=1024, bias=False)\n",
       "      (layers): ModuleList(\n",
       "        (0-23): 24 x OPTDecoderLayer(\n",
       "          (self_attn): OPTSdpaAttention(\n",
       "            (k_proj): Linear8bitLt(in_features=1024, out_features=1024, bias=True)\n",
       "            (v_proj): Linear8bitLt(in_features=1024, out_features=1024, bias=True)\n",
       "            (q_proj): Linear8bitLt(in_features=1024, out_features=1024, bias=True)\n",
       "            (out_proj): Linear8bitLt(in_features=1024, out_features=1024, bias=True)\n",
       "          )\n",
       "          (activation_fn): ReLU()\n",
       "          (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
       "          (fc1): Linear8bitLt(in_features=1024, out_features=4096, bias=True)\n",
       "          (fc2): Linear8bitLt(in_features=4096, out_features=1024, bias=True)\n",
       "          (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (lm_head): Linear(in_features=512, out_features=50272, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sG1A7Y6sEQhJ"
   },
   "source": [
    "## Formatting Your Dataset\n",
    "\n",
    "<center><a href=\"https://imgflip.com/i/90l40r\"><img src=\"https://i.imgflip.com/90l40r.jpg\" title=\"made at imgflip.com\"/></a><div></div></center>\n",
    "\n",
    "The `SFTTrainer` we'll use later can handle a couple of default formats, namely:\n",
    "- conversational format\n",
    "\n",
    "```{\"messages\": [\n",
    "    {\"role\": \"system\", \"content\": \"You are helpful\"},\n",
    "    {\"role\": \"user\", \"content\": \"What's the capital of France?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"...\"}\n",
    "]}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lPKb-T0vEQhL"
   },
   "source": [
    "- instruction format\n",
    "\n",
    "```\n",
    "{\"prompt\": \"<prompt text>\", \"completion\": \"<ideal generated text>\"}\n",
    "{\"prompt\": \"<prompt text>\", \"completion\": \"<ideal generated text>\"}\n",
    "```\n",
    "\n",
    "So, to make it easier to train, we're renaming the columns accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 113,
     "referenced_widgets": [
      "fb4bd49a8e9242f2a8e8a203fbf819d9",
      "ce54c54384ad4a06b0d1a8ad05103921",
      "b18b1cfeaf254210b3f1385a9c20ba79",
      "a0792e7ab9714a6aba3f5a675585da4e",
      "7da4af82216d4d56b72d69642b51428e",
      "9e3cefeade124117a19786c331952b2e",
      "323818bc24df45e6b272d0a8ec6349a6",
      "aaed9f2d60654d08a53ca10b4f3c22cf",
      "76d11f29b0154ef0b09e29c1c79edcd2",
      "9e7640c1b9dd4ef39e4a661b2c920956",
      "ed0e3855c7254c1e9971123c8c1be76b",
      "17b7290809b34b67a4da25509a8d72e8",
      "b45ddfd9d2d1476da27afd3d889075f4",
      "aa5747f60feb40e090b86fea0e0e7a48",
      "6f3ab98e43864ba29b00cc7d58f67193",
      "a2653cc73f6c433c824de18b6838839f",
      "741638a62d12443fb2036ef6a82e76b2",
      "bb97f148c5784b7690bc61cef6c94469",
      "e05986b01b9140ce980a15d6c74c77a8",
      "40110b82da934021972fac80dd1f1600",
      "7c6c41ca531c4411a2852e795f828dc1",
      "c4a642a94f3f4b45a62a1ead999db4db",
      "c51b38a0917d4a5f95384ad6640eec9e",
      "a10b58bcb72f44ea916c82310786b9d6",
      "b1d77b680f094e49a7db10fecbb0780f",
      "792d52d64ca045b29e3df8e24a479520",
      "0a165baed61b4894a58659acaf35eee3",
      "2f1a805e7fc845c893939a9e45589361",
      "50e6594c0a8f42469d55e1275f8d280d",
      "ffe3fa710405484092dd35c2902853da",
      "28ad6d28984e48a28bd447f0005dbe65",
      "80f7f767c20f495d9d616889f9ae6727",
      "4900ab1d1ff840e1a57a496059052854"
     ]
    },
    "id": "qc6YWqEiEQhL",
    "outputId": "99e3ce22-7c83-484a-e642-583aa67da9bb"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb4bd49a8e9242f2a8e8a203fbf819d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/531 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17b7290809b34b67a4da25509a8d72e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "sentences.csv:   0%|          | 0.00/98.4k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c51b38a0917d4a5f95384ad6640eec9e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split:   0%|          | 0/720 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset = load_dataset(\"dvgodoy/yoda_sentences\", split=\"train\")\n",
    "dataset = dataset.rename_column(\"sentence\", \"prompt\")\n",
    "dataset = dataset.rename_column(\"translation_extra\", \"completion\")\n",
    "dataset = dataset.remove_columns([\"translation\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ApOxKdy1EQhL",
    "outputId": "b440fc56-866d-4082-90a2-8624b87f2536"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'prompt': 'The birch canoe slid on the smooth planks.',\n",
       " 'completion': 'On the smooth planks, the birch canoe slid. Yes, hrrrm.'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eB6dPZyrEQhL"
   },
   "source": [
    "### Tokenizer\n",
    "\n",
    "The tokenizer is an important piece of the process.\n",
    "\n",
    "If we don't get the special tokens right, the model will either generate gibberish or be an annoying chatterbox.\n",
    "\n",
    "A few important things to remember:\n",
    "\n",
    "1. If you're padding, you should use **left** padding.\n",
    "\n",
    "  - left-padding: `<|pad|><|pad|><|pad|>This is a short sentence.</s>`\n",
    "\n",
    "  - right-padding: `This is a short sentence.<|pad|><|pad|><|pad|></s>`\n",
    "  \n",
    "Remember, LLMs are **decoders**, they predict the **next word** at the **end of the sentence**.\n",
    "\n",
    "Right-padding makes it too easy for the LLM to generate endless sequences of `<|pad|>` tokens.\n",
    "\n",
    "You do not want that.\n",
    "\n",
    "2. **DO NOT** set the **padding token** to be the same as the **EOS (ending) token**.\n",
    "\n",
    "  - proper padding token: `<|pad|><|pad|><|pad|>This is a short sentence.</s>`\n",
    "\n",
    "  - EOS as padding token: `</s></s></s>This is a short sentence.</s>`\n",
    "\n",
    "Using the EOS token as padding token **will** confuse the model, as it will learn that there may be more words after the EOS token (because **there are** words after the left-padded tokens.\n",
    "\n",
    "You do no want that either."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 145,
     "referenced_widgets": [
      "9afdf8122a444753841024792d9d8c8e",
      "2414528cfdf1453fb706e91b15ecb54f",
      "9cb54126f17d448dafad51fdff2b1493",
      "a22c2bf469664473ae178a19796d743f",
      "5f6c9085c62c4fc6b3f125b29ffc3f6c",
      "a1b4fd3755f14b11973be5d034e395b9",
      "fded883d80d74bd78e47ef94d40499c7",
      "018a3aa70bce4f6f85c16ca81c0ff40d",
      "226d1be905be4d34810b0023226959f6",
      "11092c1630b9408ca152e9cffec36e97",
      "02e06e7e119c4a2b9baabd4a8e0e065e",
      "4b8ccf5250b3429c9cc71272274f2b0c",
      "65eb06ea48ee4bcaa52929d1de28aaf8",
      "4b0fd8ab0cbd40d48b9a0078552da875",
      "ce14261a6ca4435182f36cc78d476a1c",
      "57320a7047774ad3bf3ab6881993ac6a",
      "6c844772e5084d2db32a8b6ccf2e18a5",
      "1f3b901a2b8d48b68f39e7e67fc57b3a",
      "11dda81baf17409296548b50f9f8977b",
      "0c6f74854d184962830b221b5a462323",
      "2fd08216ea134591b705374c2b6832cf",
      "2b5c132ebbd243279286c9f703ddfb37",
      "492feff6ea7d4bb68b803b5f12dacd0b",
      "28071d719572429b927f36da16807fa9",
      "5d5ce6b80fd44c02b023edb13dff57f7",
      "6422020f711948b985f1774f3c221030",
      "fe0031cc8afa4bee90f54b4739613d8a",
      "0b55cf228b32451ea28058ab86348d9e",
      "745663eee74e4ac0a4660e0437d67cd8",
      "6a231d8921994c1abcbefc81f253177e",
      "5da9d9d834904f419a99a92ee95151e5",
      "d980ea5331ef4ab58be0d6a17fcc55b0",
      "d95a894605fa46d49e8306914683248b",
      "c46602ad45684968aec7564d60331756",
      "11fcac8bc6584273becc268854e746c9",
      "14bfebbb42cd412cb0245f6d6b8b2072",
      "7f73504ba50f47d3af0cdc8b474eb671",
      "21f53b133fd6458ca8b903a987323271",
      "e7e1212abeea4cf2a14170071393611a",
      "73d608ffe83142feb4a09e703b4742c3",
      "bf0b8c18e0014d60ae77c6bcf3a6edea",
      "8bb5bfcc92914cfeacd0b1642592b7d7",
      "3fa917fb47794f2583c0cd93bc9e87b3",
      "46c2a2f0ce214f949736e6653ccec21f"
     ]
    },
    "id": "ZFvf8h3PEQhM",
    "outputId": "3fcf29c2-d358-4c92-8b4e-05fa5e0e956e"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9afdf8122a444753841024792d9d8c8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/685 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b8ccf5250b3429c9cc71272274f2b0c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/899k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "492feff6ea7d4bb68b803b5f12dacd0b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c46602ad45684968aec7564d60331756",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/441 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model.config._name_or_path,\n",
    "    padding_side='left',\n",
    "    pad_token='<|pad|>',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rLQwbN-tEQhM",
    "outputId": "f4616cf2-a1f4-46af-ebf3-00276bdc8b00"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('<|pad|>', '</s>', '</s>')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.pad_token, tokenizer.eos_token, tokenizer.bos_token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oUon5tuFEQhN"
   },
   "source": [
    "Now that we have both model and tokenizer, we can take them for a spin and use them to generate a few words.\n",
    "\n",
    "Notice that it still is the base pretrained model. It does not know instructions, and it does not know Yoda."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "8KXn65I1EQhN"
   },
   "outputs": [],
   "source": [
    "def generate(model, tokenizer, prompt, max_new_tokens=100):\n",
    "    tokenized_input = tokenizer(prompt, add_special_tokens=False, return_tensors=\"pt\")\n",
    "    input_ids = tokenized_input[\"input_ids\"].cuda()\n",
    "\n",
    "    model.eval()\n",
    "    generation_output = model.generate(\n",
    "        input_ids=input_ids,\n",
    "        max_new_tokens=max_new_tokens,\n",
    "        eos_token_id=tokenizer.eos_token_id,\n",
    "        pad_token_id=tokenizer.pad_token_id\n",
    "    )\n",
    "    output = tokenizer.batch_decode(generation_output, skip_special_tokens=False)[0]\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZK5iBB_YEQhN",
    "outputId": "436471af-5e31-48d5-f675-6528be8bcc77"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Force is strong in you!\n",
      "I'm not sure if I'm ready to be a Jedi.\n",
      "I'm not sure if I'm ready to be a Jedi.\n",
      "I'm not sure if I'm ready to be a Jedi.\n",
      "I'm not sure if I'm ready to be a Jedi.\n",
      "I'm not sure if I'm ready to be a Jedi.\n",
      "I'm not sure if I'm ready to be a Jedi.\n",
      "I'm not sure if I'm ready to be a Jedi.\n",
      "I\n"
     ]
    }
   ],
   "source": [
    "print(generate(model, tokenizer, 'The Force is strong in you!'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oU1xh_p2EQhN"
   },
   "source": [
    "#### Chat Template\n",
    "\n",
    "We're trusting that `SFTTrainer` will handle our dataset appropriately, since we renamed the columns accordingly.\n",
    "\n",
    "It **will**, but **only** if the provided tokenizer has a **chat template**, that is, if it specifies the following:\n",
    "\n",
    "- which special tokens should be used and where they should be placed\n",
    "- where the specified columns (`prompt` and `completion`) should be placed\n",
    "- what is the **generation prompt** (more on that in the last section)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "yBz3gESEEQhO"
   },
   "outputs": [],
   "source": [
    "tokenizer.chat_template"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BsdyWmvhEQhO"
   },
   "source": [
    "Unfortunately, our model/tokenizer does not come with a chat template.\n",
    "\n",
    "It makes sense, since the base model was not fine-tuned for instructions.\n",
    "\n",
    "However, we can define a template or, better yet, use [ChatML](https://github.com/openai/openai-python/blob/release-v0.28.0/chatml.md) as default template calling the `setup_chat_format()` function.\n",
    "\n",
    "**IMPORTANT #1**: the `setup_chat_format()` function forcibly resize the embedding layers to match the tokenizer's vocabulary length.\n",
    "\n",
    "However, most models have **spare space** in their embedding layers, meaning it is possible to **add extra tokens without resizing the embedding layers**.\n",
    "\n",
    "For performance reasons, it is commonplace to keep the embedding layer's length as a multiple of a power of 2 (that's what we're doing in the `resize_to_multiple_of` argument since our model's embedding layer has 50,272 elements while the tokenizer has only 50,266).\n",
    "\n",
    "**IMPORTANT #2**: if your model's embedding layer gets resized (whether it's shorter or longer than it was), you'll **have to save the embeddings** after fine-tuning, thus making your adapter larger, and requiring extra steps for loading it afterward.\n",
    "\n",
    "**IMPORTANT #3**: the `setup_chat_format()` function **sets the EOS token as the pad token**, effectively undoing what we did above, so we need to fix it once again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "zganniuMEQhO"
   },
   "outputs": [],
   "source": [
    "if tokenizer.chat_template is None:\n",
    "    model, tokenizer = setup_chat_format(model, tokenizer, resize_to_multiple_of=32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eaFzgM-BEQhP",
    "outputId": "818bc7cf-6a97-453e-e09e-9b3ab200f14d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('<|im_end|>', '<|im_end|>', '<|im_start|>')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.pad_token, tokenizer.eos_token, tokenizer.bos_token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mWwVt5XyEQhP"
   },
   "source": [
    "Uh-oh! Unfortunately, the function does exactly what I advised you **not** to do: the padding token was set to be the same as the EOS token.\n",
    "\n",
    "Let's fix it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CHKAnuFVEQhP",
    "outputId": "aa275a55-b6ce-4a42-a29a-94b92a27dba5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('<|pad|>', '<|im_end|>', '<|im_start|>')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.pad_token = '<|pad|>'\n",
    "tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
    "\n",
    "tokenizer.pad_token, tokenizer.eos_token, tokenizer.bos_token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZJI9MEzKEQhP"
   },
   "source": [
    "Now, let's see how it will format an example from our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6-mbEiKCEQhQ",
    "outputId": "e9eca279-d3b0-4e11-dba7-edda87bb083b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "The birch canoe slid on the smooth planks.<|im_end|>\n",
      "<|im_start|>assistant\n",
      "On the smooth planks, the birch canoe slid. Yes, hrrrm.<|im_end|>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": dataset[0]['prompt']},\n",
    "    {\"role\": \"assistant\", \"content\": dataset[0]['completion']}\n",
    "]\n",
    "print(tokenizer.apply_chat_template(messages, tokenize=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kBBVuIJMGZck"
   },
   "source": [
    "Notice that each interaction is wrapped in `<|im_start|>` and `<im_end|>` tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y8d-t3p-EQhQ"
   },
   "source": [
    "## Setting Up Low-Rank Adapters (LoRA)\n",
    "\n",
    "### What is LoRA?\n",
    "\n",
    "LoRA is a **parameter-efficient fine-tuning (PEFT)** technique that attaches **adapters** to linear layers (mostly in attention heads) to allow models to be fine-tuned without updating the base model's parameters.\n",
    "\n",
    "![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/lora.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "W9nDboWaEQhQ"
   },
   "outputs": [],
   "source": [
    "# Without LoRA\n",
    "def regular_forward_matmul(x, W):\n",
    "    h = x @ W\n",
    "    return h\n",
    "\n",
    "# With LoRA\n",
    "def lora_forward_matmul(x, W, W_A, W_B, alpha):\n",
    "    h = x @ W\n",
    "    h += x @ (W_A @ W_B)*alpha"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DHWoE-T6EQhQ"
   },
   "source": [
    "$$\n",
    "\\Large\n",
    "W_{\\text{LoRA}} = W_{\\text{orig}}+\\frac{\\alpha}{r}\\Delta W = W_{\\text{orig}}+\\frac{\\alpha}{r}BA\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MFtd62eHEQhQ"
   },
   "source": [
    "![](https://raw.githubusercontent.com/dvgodoy/FineTuningLLMs101_ODSC_Europe2024/main/images/lora_diagram.png)\n",
    "Source: [PEFT documentation](https://huggingface.co/docs/peft/main/en/conceptual_guides/lora#merge-lora-weights-into-the-base-model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KZkjaHKkEQhR"
   },
   "source": [
    "### PEFT\n",
    "\n",
    "It is a library for efficiently adapting large pretrained models. It is integrated with the Transformers, Diffusers, and Accelerate libraries.\n",
    "\n",
    "#### Steps:\n",
    "\n",
    "1. Instantiate a base model (done!)\n",
    "2. Prepare the model using the `prepare_model_for_kbit_training()` method (cell below)\n",
    "3. Create an instance of the configuration ([`LoraConfig`](https://huggingface.co/docs/peft/main/en/conceptual_guides/lora#common-lora-parameters-in-peft)) object (cell below)\n",
    "4. Apply the LoRA configuration to the base model using the `get_peft_model()` method (cell below)\n",
    "5. Train the PEFT model as you would normally train any other model (next section)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ilxMPNXDEQhR",
    "outputId": "f902c3db-551d-4b85-aedb-6175cf7d62d9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PeftModelForCausalLM(\n",
       "  (base_model): LoraModel(\n",
       "    (model): OPTForCausalLM(\n",
       "      (model): OPTModel(\n",
       "        (decoder): OPTDecoder(\n",
       "          (embed_tokens): Embedding(50272, 512, padding_idx=1)\n",
       "          (embed_positions): OPTLearnedPositionalEmbedding(2050, 1024)\n",
       "          (project_out): Linear8bitLt(in_features=1024, out_features=512, bias=False)\n",
       "          (project_in): Linear8bitLt(in_features=512, out_features=1024, bias=False)\n",
       "          (layers): ModuleList(\n",
       "            (0-23): 24 x OPTDecoderLayer(\n",
       "              (self_attn): OPTSdpaAttention(\n",
       "                (k_proj): Linear8bitLt(in_features=1024, out_features=1024, bias=True)\n",
       "                (v_proj): lora.Linear8bitLt(\n",
       "                  (base_layer): Linear8bitLt(in_features=1024, out_features=1024, bias=True)\n",
       "                  (lora_dropout): ModuleDict(\n",
       "                    (default): Dropout(p=0.05, inplace=False)\n",
       "                  )\n",
       "                  (lora_A): ModuleDict(\n",
       "                    (default): Linear(in_features=1024, out_features=8, bias=False)\n",
       "                  )\n",
       "                  (lora_B): ModuleDict(\n",
       "                    (default): Linear(in_features=8, out_features=1024, bias=False)\n",
       "                  )\n",
       "                  (lora_embedding_A): ParameterDict()\n",
       "                  (lora_embedding_B): ParameterDict()\n",
       "                  (lora_magnitude_vector): ModuleDict()\n",
       "                )\n",
       "                (q_proj): lora.Linear8bitLt(\n",
       "                  (base_layer): Linear8bitLt(in_features=1024, out_features=1024, bias=True)\n",
       "                  (lora_dropout): ModuleDict(\n",
       "                    (default): Dropout(p=0.05, inplace=False)\n",
       "                  )\n",
       "                  (lora_A): ModuleDict(\n",
       "                    (default): Linear(in_features=1024, out_features=8, bias=False)\n",
       "                  )\n",
       "                  (lora_B): ModuleDict(\n",
       "                    (default): Linear(in_features=8, out_features=1024, bias=False)\n",
       "                  )\n",
       "                  (lora_embedding_A): ParameterDict()\n",
       "                  (lora_embedding_B): ParameterDict()\n",
       "                  (lora_magnitude_vector): ModuleDict()\n",
       "                )\n",
       "                (out_proj): Linear8bitLt(in_features=1024, out_features=1024, bias=True)\n",
       "              )\n",
       "              (activation_fn): ReLU()\n",
       "              (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
       "              (fc1): Linear8bitLt(in_features=1024, out_features=4096, bias=True)\n",
       "              (fc2): Linear8bitLt(in_features=4096, out_features=1024, bias=True)\n",
       "              (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (lm_head): Linear(in_features=512, out_features=50272, bias=False)\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training\n",
    "\n",
    "model = prepare_model_for_kbit_training(model)\n",
    "\n",
    "config = LoraConfig(\n",
    "    r=8,                   # the rank of the adapter, the lower the fewer parameters you'll need to train\n",
    "    lora_alpha=16,         # multiplier, usually 2*r\n",
    "    bias=\"none\",           # BEWARE: training biases *modifies* base model's behavior\n",
    "    lora_dropout=0.05,\n",
    "    task_type=\"CAUSAL_LM\",\n",
    ")\n",
    "\n",
    "model = get_peft_model(model, config)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lX5o_I0kGsxn"
   },
   "source": [
    "Notice that the attention layers (`v_proj`, and `q_proj`) which already were quantized (`Linear8bitLt`), now have a set of several much smaller LoRA-related `Linear` layers attached to them.\n",
    "\n",
    "These extra layers will make the model slightly larger:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "G6EkXP18EQhR",
    "outputId": "a866dbc7-d428-4c0d-938e-f7b615f8ce22"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "418.816\n"
     ]
    }
   ],
   "source": [
    "print(model.get_memory_footprint()/1e6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Uz2IWEAEHLAP"
   },
   "source": [
    "But, overall, only a tiny fraction of the total number of parameters is trainable now, thanks to LoRA!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vNLbOq6GEQhR",
    "outputId": "006d8750-a53d-4edb-c519-c4572055a756"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "786.432\n",
      "0.2368893467652883\n"
     ]
    }
   ],
   "source": [
    "trainable_parms, tot_parms = model.get_nb_trainable_parameters()\n",
    "print(trainable_parms/1e3)\n",
    "print(100*trainable_parms/tot_parms)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o7OblnYAEQhR"
   },
   "source": [
    "## Fine-Tuning with SFTTrainer\n",
    "\n",
    "The `SFTTrainer` (Supervised Fine-Tuning Trainer) object takes:\n",
    "- a model\n",
    "- a tokenizer\n",
    "- a dataset\n",
    "- a configuration object\n",
    "\n",
    "We already have the first three elements, let's work on the last one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eHnCCh1hEQhS"
   },
   "source": [
    "### `SFTConfig`\n",
    "\n",
    "#### Gradient Checkpointing\n",
    "\n",
    "This is an overlooked hero in most tutorials.\n",
    "\n",
    "The memory gains, if the sequences are long (e.g. 1,000 tokens), are HUGE.\n",
    "\n",
    "#### Gradient Accumulation\n",
    "\n",
    "Next to the huge elephant (the LLM) we must be able to fit **at least one data point** in the GPU RAM.\n",
    "\n",
    "A mini-batch of one is no fun, though, so we turn to **gradient accumulation**.\n",
    "\n",
    "The mini-batch size (`per_device_train_batch_size`) will actually stand for a **micro-batch** which works as an intermediate step. After a few micro-batches (`gradient_accumulation_steps`), the model update will be triggered.\n",
    "\n",
    "The effective mini-batch size is: `gradient_accumulation_steps` x `per_device_train_batch_size`\n",
    "\n",
    "#### Auto-find Batch Size\n",
    "\n",
    "Is there anything more annoying than OOM errors? Do you like trying out mini/micro batch sizes on your own? Me neither!\n",
    "\n",
    "If set to `True`, it will start with `per_device_train_batch_size` and, should it result in OOM error, it halves the size and try again until it works.\n",
    "\n",
    "#### Optim\n",
    "\n",
    "The optimizer it should use. We can choose a paged 8-bit optimizer (`\"paged_adamw_8bit\"`) to squeeze the most space out of it.\n",
    "\n",
    "In our case, though, it won't make much of a difference since LoRA reduced the number of trainable parameters to a handful.\n",
    "\n",
    "#### Max Sequence Length\n",
    "\n",
    "The sequence length is one of the main drivers of memory usage, especially since we cannot use Flash Attention.\n",
    "\n",
    "So, we should choose a value that's not much more than what we actually need.\n",
    "\n",
    "How long are our sequences (considering **both** prompt and completion, that is):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 518,
     "referenced_widgets": [
      "e3567fd081844a4885c67b52cfceb93d",
      "8c2428425d444b0eb627d3a730abb05a",
      "df9f906da04d4ac8a77cce8f35818847",
      "1aa93986329f4241a4e5f93ca40b2e46",
      "c4c3ac015336423ca5cd47b94f324848",
      "00e395b7efa1445fbda1840d94bbda45",
      "40c2599b769c48898d69c1bfa7c1930e",
      "acf32e9bab24410ca4cbd73951a7b9f0",
      "84082bcedde2416587f2b3f3c662518f",
      "2abfa7db9e1646fc8947b083bba849e8",
      "34e5ab638f5f4664977231d4629d2519"
     ]
    },
    "id": "YtHengPcEQhS",
    "outputId": "9cafa577-525e-41d5-a87b-06a392092f49"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e3567fd081844a4885c67b52cfceb93d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/720 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(array([  5.,  18.,  91.,  55., 224., 136., 128.,  23.,  31.,   9.]),\n",
       " array([10. , 11.5, 13. , 14.5, 16. , 17.5, 19. , 20.5, 22. , 23.5, 25. ]),\n",
       " <BarContainer object of 10 artists>)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGdCAYAAAA44ojeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAdvklEQVR4nO3df6zV9X3H8dfl1xWVe9lV4XorIF2riFXr1OFtnbWTCMj8UWmmzjltiGbdxUbZrNJVrV0TnHPqNFSypNM1rd1qMq1iRsdAoU0v2GKN01iqBkWHF6yEewULXr1nfzSc7CqKwL2cz708HslJvN/v9577/uTquU+/53zPqatUKpUAABRkSK0HAAB4L4ECABRHoAAAxREoAEBxBAoAUByBAgAUR6AAAMURKABAcYbVeoA90dPTk/Xr12fUqFGpq6ur9TgAwEdQqVTy5ptvpqWlJUOGfPg5kgEZKOvXr8+4ceNqPQYAsAdeeeWVHHHEER96zIAMlFGjRiX53QIbGhpqPA0A8FF0dXVl3Lhx1b/jH2ZABsqOp3UaGhoECgAMMB/l5RleJAsAFEegAADFESgAQHEECgBQHIECABRHoAAAxREoAEBxBAoAUByBAgAUR6AAAMURKABAcQQKAFAcgQIAFEegAADFGVbrAYD+deT1j9Z6hN320i0zaz0CUGPOoAAAxREoAEBxBAoAUByBAgAUR6AAAMURKABAcQQKAFAcgQIAFEegAADFESgAQHEECgBQHIECABRHoAAAxREoAEBxBAoAUByBAgAUR6AAAMURKABAcQQKAFAcgQIAFEegAADFESgAQHEECgBQHIECABRHoAAAxREoAEBxBAoAUByBAgAUR6AAAMURKABAcQQKAFAcgQIAFEegAADF2a1AmT9/fk455ZSMGjUqY8aMyfnnn581a9b0Ombbtm1pa2vLIYcckoMPPjizZs3Khg0beh2zbt26zJw5MwceeGDGjBmTa6+9Nu+8887erwYAGBR2K1CWL1+etra2rFy5MkuWLEl3d3fOOuusbN26tXrMNddck0ceeSQPPPBAli9fnvXr1+eCCy6o7n/33Xczc+bMvP322/nZz36Wf/3Xf819992XG2+8se9WBQAMaHWVSqWyp9/8+uuvZ8yYMVm+fHlOP/30dHZ25rDDDsv999+fL37xi0mSX/3qVznmmGPS3t6eU089Nf/5n/+ZP/mTP8n69eszduzYJMnChQtz3XXX5fXXX8+IESN2+XO7urrS2NiYzs7ONDQ07On4sF848vpHaz3Cbnvplpm1HgHoB7vz93uvXoPS2dmZJGlqakqSrF69Ot3d3Zk6dWr1mEmTJmX8+PFpb29PkrS3t+e4446rxkmSTJs2LV1dXXn22Wd3+nO2b9+erq6uXjcAYPDa40Dp6enJ1Vdfnc9+9rP51Kc+lSTp6OjIiBEjMnr06F7Hjh07Nh0dHdVj/n+c7Ni/Y9/OzJ8/P42NjdXbuHHj9nRsAGAA2ONAaWtryzPPPJN/+7d/68t5dmrevHnp7Oys3l555ZV+/5kAQO0M25NvmjNnThYtWpQVK1bkiCOOqG5vbm7O22+/nc2bN/c6i7Jhw4Y0NzdXj3niiSd63d+Oq3x2HPNe9fX1qa+v35NRAYABaLfOoFQqlcyZMycPPvhgli1blokTJ/baf9JJJ2X48OFZunRpdduaNWuybt26tLa2JklaW1vzP//zP9m4cWP1mCVLlqShoSGTJ0/em7UAAIPEbp1BaWtry/33358f/ehHGTVqVPU1I42NjRk5cmQaGxsze/bszJ07N01NTWloaMhVV12V1tbWnHrqqUmSs846K5MnT86ll16aW2+9NR0dHfn617+etrY2Z0kAgCS7GSj33HNPkuSMM87otf3ee+/N5ZdfniS54447MmTIkMyaNSvbt2/PtGnT8u1vf7t67NChQ7No0aJ8+ctfTmtraw466KBcdtll+eY3v7l3KwEABo29eh+UWvE+KPDReR8UoBT77H1QAAD6g0ABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOLsdqCsWLEi55xzTlpaWlJXV5eHHnqo1/7LL788dXV1vW7Tp0/vdcymTZtyySWXpKGhIaNHj87s2bOzZcuWvVoIADB47HagbN26NSeccEIWLFjwgcdMnz49r732WvX2gx/8oNf+Sy65JM8++2yWLFmSRYsWZcWKFbnyyit3f3oAYFAatrvfMGPGjMyYMeNDj6mvr09zc/NO9z333HNZvHhxfv7zn+fkk09Oktx99905++yzc9ttt6WlpWV3RwIABpl+eQ3K448/njFjxuToo4/Ol7/85bzxxhvVfe3t7Rk9enQ1TpJk6tSpGTJkSFatWtUf4wAAA8xun0HZlenTp+eCCy7IxIkT8+KLL+ZrX/taZsyYkfb29gwdOjQdHR0ZM2ZM7yGGDUtTU1M6Ojp2ep/bt2/P9u3bq193dXX19dgAQEH6PFAuuuii6j8fd9xxOf744/P7v//7efzxx3PmmWfu0X3Onz8/N998c1+NCAAUrt8vM/74xz+eQw89NC+88EKSpLm5ORs3bux1zDvvvJNNmzZ94OtW5s2bl87OzurtlVde6e+xAYAa6vdAefXVV/PGG2/k8MMPT5K0trZm8+bNWb16dfWYZcuWpaenJ1OmTNnpfdTX16ehoaHXDQAYvHb7KZ4tW7ZUz4Ykydq1a/PUU0+lqakpTU1NufnmmzNr1qw0NzfnxRdfzFe/+tV84hOfyLRp05IkxxxzTKZPn54rrrgiCxcuTHd3d+bMmZOLLrrIFTwAQJI9OIPyi1/8IieeeGJOPPHEJMncuXNz4okn5sYbb8zQoUPz9NNP59xzz81RRx2V2bNn56STTspPfvKT1NfXV+/j+9//fiZNmpQzzzwzZ599dk477bT88z//c9+tCgAY0Hb7DMoZZ5yRSqXygft//OMf7/I+mpqacv/99+/ujwYA9hM+iwcAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOMNqPQDAex15/aO1HmG3vXTLzFqPAIOKMygAQHEECgBQHIECABRHoAAAxREoAEBxBAoAUByBAgAUR6AAAMURKABAcQQKAFAcgQIAFMdn8QD0AZ8fBH3LGRQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIqz24GyYsWKnHPOOWlpaUldXV0eeuihXvsrlUpuvPHGHH744Rk5cmSmTp2a559/vtcxmzZtyiWXXJKGhoaMHj06s2fPzpYtW/ZqIQDA4LHbgbJ169accMIJWbBgwU7333rrrbnrrruycOHCrFq1KgcddFCmTZuWbdu2VY+55JJL8uyzz2bJkiVZtGhRVqxYkSuvvHLPVwEADCrDdvcbZsyYkRkzZux0X6VSyZ133pmvf/3rOe+885Ik3/3udzN27Ng89NBDueiii/Lcc89l8eLF+fnPf56TTz45SXL33Xfn7LPPzm233ZaWlpa9WA4AMBj06WtQ1q5dm46OjkydOrW6rbGxMVOmTEl7e3uSpL29PaNHj67GSZJMnTo1Q4YMyapVq3Z6v9u3b09XV1evGwAwePVpoHR0dCRJxo4d22v72LFjq/s6OjoyZsyYXvuHDRuWpqam6jHvNX/+/DQ2NlZv48aN68uxAYDCDIireObNm5fOzs7q7ZVXXqn1SABAP+rTQGlubk6SbNiwodf2DRs2VPc1Nzdn48aNvfa/88472bRpU/WY96qvr09DQ0OvGwAwePVpoEycODHNzc1ZunRpdVtXV1dWrVqV1tbWJElra2s2b96c1atXV49ZtmxZenp6MmXKlL4cBwAYoHb7Kp4tW7bkhRdeqH69du3aPPXUU2lqasr48eNz9dVX51vf+lY++clPZuLEibnhhhvS0tKS888/P0lyzDHHZPr06bniiiuycOHCdHd3Z86cObnoootcwQMAJNmDQPnFL36Rz3/+89Wv586dmyS57LLLct999+WrX/1qtm7dmiuvvDKbN2/OaaedlsWLF+eAAw6ofs/3v//9zJkzJ2eeeWaGDBmSWbNm5a677uqD5QAAg0FdpVKp1HqI3dXV1ZXGxsZ0dnZ6PQrswpHXP1rrESjUS7fMrPUI7Gd25+/3gLiKBwDYvwgUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACjObr9RG/SVgfj+HN43AmDfcAYFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4w2o9AAwkR17/aK1HANgvOIMCABRHoAAAxREoAEBx+jxQvvGNb6Surq7XbdKkSdX927ZtS1tbWw455JAcfPDBmTVrVjZs2NDXYwAAA1i/nEE59thj89prr1VvP/3pT6v7rrnmmjzyyCN54IEHsnz58qxfvz4XXHBBf4wBAAxQ/XIVz7Bhw9Lc3Py+7Z2dnfnOd76T+++/P3/8x3+cJLn33ntzzDHHZOXKlTn11FP7YxwAYIDplzMozz//fFpaWvLxj388l1xySdatW5ckWb16dbq7uzN16tTqsZMmTcr48ePT3t7+gfe3ffv2dHV19boBAINXnwfKlClTct9992Xx4sW55557snbt2vzRH/1R3nzzzXR0dGTEiBEZPXp0r+8ZO3ZsOjo6PvA+58+fn8bGxupt3LhxfT02AFCQPn+KZ8aMGdV/Pv744zNlypRMmDAhP/zhDzNy5Mg9us958+Zl7ty51a+7urpECgAMYv1+mfHo0aNz1FFH5YUXXkhzc3PefvvtbN68udcxGzZs2OlrVnaor69PQ0NDrxsAMHj1e6Bs2bIlL774Yg4//PCcdNJJGT58eJYuXVrdv2bNmqxbty6tra39PQoAMED0+VM8f/M3f5NzzjknEyZMyPr163PTTTdl6NChufjii9PY2JjZs2dn7ty5aWpqSkNDQ6666qq0tra6ggcAqOrzQHn11Vdz8cUX54033shhhx2W0047LStXrsxhhx2WJLnjjjsyZMiQzJo1K9u3b8+0adPy7W9/u6/HAAAGsLpKpVKp9RC7q6urK42Njens7PR6lAHMJwNDbb10y8xaj8B+Znf+fvssHgCgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKM6zWAwDAR3Xk9Y/WeoTd9tItM2s9woDkDAoAUByBAgAUR6AAAMURKABAcQQKAFAcgQIAFEegAADFESgAQHEECgBQHIECABRHoAAAxfFZPAD7qYH4uTbsP5xBAQCKI1AAgOIIFACgOAIFACiOQAEAiiNQAIDiCBQAoDjeB2WQ8H4GAAwmzqAAAMURKABAcQQKAFAcgQIAFEegAADFESgAQHEECgBQHIECABRHoAAAxREoAEBxBAoAUByBAgAUR6AAAMXxacYA0I8G6qfNv3TLzJr+fGdQAIDiCBQAoDgCBQAojkABAIojUACA4riKZycG6iuuAWCwcAYFACiOQAEAiiNQAIDi1DRQFixYkCOPPDIHHHBApkyZkieeeKKW4wAAhahZoPz7v/975s6dm5tuuilPPvlkTjjhhEybNi0bN26s1UgAQCFqFii33357rrjiinzpS1/K5MmTs3Dhwhx44IH5l3/5l1qNBAAUoiaXGb/99ttZvXp15s2bV902ZMiQTJ06Ne3t7e87fvv27dm+fXv1687OziRJV1dXv8zXs/2tfrlfABgo+uNv7I77rFQquzy2JoHym9/8Ju+++27Gjh3ba/vYsWPzq1/96n3Hz58/PzfffPP7to8bN67fZgSA/Vnjnf1332+++WYaGxs/9JgB8UZt8+bNy9y5c6tf9/T0ZNOmTTnkkENSV1fXpz+rq6sr48aNyyuvvJKGhoY+ve8SWe/gZr2Dm/UOboNxvZVKJW+++WZaWlp2eWxNAuXQQw/N0KFDs2HDhl7bN2zYkObm5vcdX19fn/r6+l7bRo8e3Z8jpqGhYdD8C/FRWO/gZr2Dm/UOboNtvbs6c7JDTV4kO2LEiJx00klZunRpdVtPT0+WLl2a1tbWWowEABSkZk/xzJ07N5dddllOPvnk/OEf/mHuvPPObN26NV/60pdqNRIAUIiaBcqFF16Y119/PTfeeGM6Ojry6U9/OosXL37fC2f3tfr6+tx0003ve0ppsLLewc16BzfrHdz2t/W+V13lo1zrAwCwD/ksHgCgOAIFACiOQAEAiiNQAIDi7JeBsmLFipxzzjlpaWlJXV1dHnrooV77K5VKbrzxxhx++OEZOXJkpk6dmueff742w/aRD1tzd3d3rrvuuhx33HE56KCD0tLSkr/4i7/I+vXrazfwXtrV7/j/+8u//MvU1dXlzjvv3Gfz9bWPst7nnnsu5557bhobG3PQQQfllFNOybp16/b9sH1gV+vdsmVL5syZkyOOOCIjR46sfiDpQDR//vyccsopGTVqVMaMGZPzzz8/a9as6XXMtm3b0tbWlkMOOSQHH3xwZs2a9b43whxIdrXmTZs25aqrrsrRRx+dkSNHZvz48fnKV75S/Zy2geaj/I53qFQqmTFjxi4f1waD/TJQtm7dmhNOOCELFizY6f5bb701d911VxYuXJhVq1bloIMOyrRp07Jt27Z9PGnf+bA1v/XWW3nyySdzww035Mknn8x//Md/ZM2aNTn33HNrMGnf2NXveIcHH3wwK1eu/Ehvu1yyXa33xRdfzGmnnZZJkybl8ccfz9NPP50bbrghBxxwwD6etG/sar1z587N4sWL873vfS/PPfdcrr766syZMycPP/zwPp507y1fvjxtbW1ZuXJllixZku7u7px11lnZunVr9ZhrrrkmjzzySB544IEsX74869evzwUXXFDDqffOrta8fv36rF+/PrfddlueeeaZ3HfffVm8eHFmz55d48n3zEf5He9w55139vlHvBSrsp9LUnnwwQerX/f09FSam5sr//AP/1Ddtnnz5kp9fX3lBz/4QQ0m7HvvXfPOPPHEE5UklZdffnnfDNWPPmi9r776auVjH/tY5ZlnnqlMmDChcscdd+zz2frDztZ74YUXVv78z/+8NgP1s52t99hjj61885vf7LXtD/7gDyp/+7d/uw8n6x8bN26sJKksX768Uqn87vFp+PDhlQceeKB6zHPPPVdJUmlvb6/VmH3qvWvemR/+8IeVESNGVLq7u/fhZP3jg9b7y1/+svKxj32s8tprr32kx/GBbr88g/Jh1q5dm46OjkydOrW6rbGxMVOmTEl7e3sNJ9u3Ojs7U1dX1++feVQrPT09ufTSS3Pttdfm2GOPrfU4/aqnpyePPvpojjrqqEybNi1jxozJlClTBvXp4c985jN5+OGH87//+7+pVCp57LHH8utf/zpnnXVWrUfbazuexmhqakqSrF69Ot3d3b0esyZNmpTx48cPmses9675g45paGjIsGED4jNwP9TO1vvWW2/lz/7sz7JgwYKdfmbdYCRQ3qOjoyNJ3veOtmPHjq3uG+y2bduW6667LhdffPGg+oCq/+/v//7vM2zYsHzlK1+p9Sj9buPGjdmyZUtuueWWTJ8+Pf/1X/+VL3zhC7nggguyfPnyWo/XL+6+++5Mnjw5RxxxREaMGJHp06dnwYIFOf3002s92l7p6enJ1Vdfnc9+9rP51Kc+leR3j1kjRox43/9MDJbHrJ2t+b1+85vf5O/+7u9y5ZVX7uPp+t4Hrfeaa67JZz7zmZx33nk1nG7fGvipSZ/q7u7On/7pn6ZSqeSee+6p9Tj9YvXq1fmnf/qnPPnkk/vFc7k9PT1JkvPOOy/XXHNNkuTTn/50fvazn2XhwoX53Oc+V8vx+sXdd9+dlStX5uGHH86ECROyYsWKtLW1paWlpdeZhoGmra0tzzzzTH7605/WepR9Zldr7urqysyZMzN58uR84xvf2LfD9YOdrffhhx/OsmXL8stf/rKGk+17zqC8x45TZ+99BfyGDRsG/Wm1HXHy8ssvZ8mSJYP27MlPfvKTbNy4MePHj8+wYcMybNiwvPzyy/nrv/7rHHnkkbUer88deuihGTZsWCZPntxr+zHHHDNgr+L5ML/97W/zta99LbfffnvOOeecHH/88ZkzZ04uvPDC3HbbbbUeb4/NmTMnixYtymOPPZYjjjiiur25uTlvv/12Nm/e3Ov4wfCY9UFr3uHNN9/M9OnTM2rUqDz44IMZPnx4DabsOx+03mXLluXFF1/M6NGjq49ZSTJr1qycccYZNZq2/wmU95g4cWKam5uzdOnS6raurq6sWrUqra2tNZysf+2Ik+effz7//d//nUMOOaTWI/WbSy+9NE8//XSeeuqp6q2lpSXXXnttfvzjH9d6vD43YsSInHLKKe+7bPHXv/51JkyYUKOp+k93d3e6u7szZEjvh7ehQ4dWzyYNJJVKJXPmzMmDDz6YZcuWZeLEib32n3TSSRk+fHivx6w1a9Zk3bp1A/Yxa1drTn73uHzWWWdlxIgRefjhhwfsFWnJrtd7/fXXv+8xK0nuuOOO3HvvvTWYeN/YL5/i2bJlS1544YXq12vXrs1TTz2VpqamjB8/PldffXW+9a1v5ZOf/GQmTpyYG264IS0tLTn//PNrN/Re+rA1H3744fniF7+YJ598MosWLcq7775bfe66qakpI0aMqNXYe2xXv+P3Btjw4cPT3Nyco48+el+P2id2td5rr702F154YU4//fR8/vOfz+LFi/PII4/k8ccfr93Qe2FX6/3c5z6Xa6+9NiNHjsyECROyfPnyfPe7383tt99ew6n3TFtbW+6///786Ec/yqhRo6r/bTY2NmbkyJFpbGzM7NmzM3fu3DQ1NaWhoSFXXXVVWltbc+qpp9Z4+j2zqzXviJO33nor3/ve99LV1ZWurq4kyWGHHZahQ4fWcvzdtqv1Njc37/Rs2Pjx43cab4NGTa8hqpHHHnuskuR9t8suu6xSqfzuUuMbbrihMnbs2Ep9fX3lzDPPrKxZs6a2Q++lD1vz2rVrd7ovSeWxxx6r9eh7ZFe/4/ca6JcZf5T1fuc736l84hOfqBxwwAGVE044ofLQQw/VbuC9tKv1vvbaa5XLL7+80tLSUjnggAMqRx99dOUf//EfKz09PbUdfA980H+b9957b/WY3/72t5W/+qu/qvze7/1e5cADD6x84QtfqLz22mu1G3ov7WrNH/T7T1JZu3ZtTWffEx/ld7yz7xnslxnXVSqVSt/lDgDA3vMaFACgOAIFACiOQAEAiiNQAIDiCBQAoDgCBQAojkABAIojUACA4ggUAKA4AgUAKI5AAQCKI1AAgOL8H+chm8YrR1xeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(dataset.map(lambda row: {'len': len(row['prompt'].split()) + len(row['completion'].split())})['len'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-XbWZLPXEQhS"
   },
   "source": [
    "The longest sequences have 25 words.\n",
    "\n",
    "That's roughly 30 tokens (as a rule of thumb, **1 token = 0.8 word**).\n",
    "\n",
    "We'll also add some special tokens to it, so it's surely shorter than 40 tokens in total.\n",
    "\n",
    "We may give the user some wiggle room to come up with slightly longer sentences, so let's make it 64 tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 106,
     "referenced_widgets": [
      "4505dd5cf9324bd6b511d3fb1d0881aa",
      "ac6e3fa9a62d4a178489f9d8c6e0239d",
      "3389d70a664347be882edddc945e0ab0",
      "2b7be946f2aa400c8e886fb19d58825b",
      "7dcd7021d9a54613872f902a24788191",
      "411b31cf20c14127bb22777035090dd7",
      "bfecaf88ff994b669a8ff72681bbe8af",
      "ae35335f77cb4291ae0d434753f2ec5f",
      "849601cbfbad4ac78cbe647ae57fd5fd",
      "f829c70029444a76b50c6ee075717401",
      "a39834092155453885d7ca03534dec09"
     ]
    },
    "id": "wim0ViZ4EQhT",
    "outputId": "ca83a91d-3ff6-401a-c918-860d569d4125"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4505dd5cf9324bd6b511d3fb1d0881aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/720 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:403: UserWarning: You passed a processing_class with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to overflow issues when training a model in half-precision. You might consider adding `processing_class.padding_side = 'right'` to your code.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "sft_config = SFTConfig(\n",
    "    # Checkpointing\n",
    "    gradient_checkpointing=True,    # this saves a LOT of memory\n",
    "    gradient_checkpointing_kwargs={'use_reentrant': False}, # set this to avoid exceptions in newer versions of PyTorch\n",
    "    # Gradient Accumulation / Batch size\n",
    "    gradient_accumulation_steps=4,  # actual batch (for updating) is 4x micro-batch size\n",
    "    per_device_train_batch_size=8,  # the initial (micro) batch size to start off with\n",
    "    auto_find_batch_size=True,      # if (micro) batch size would cause OOM, halves its size until it works\n",
    "    # Optimizer\n",
    "    optim='paged_adamw_8bit',       # 8-bit Adam optimizer - doesn't help much if you're using LoRA!\n",
    "    # Seq Length\n",
    "    max_seq_length=64,\n",
    "    ## These are typical training parameters\n",
    "    num_train_epochs=10,            # 10 epochs take about 10 minutes on Colab's T4 GPU\n",
    "    learning_rate=3e-4,\n",
    "    logging_steps=10,\n",
    "    logging_dir=\"./logs\",\n",
    "    output_dir='./tmp',\n",
    "    report_to='none'\n",
    ")\n",
    "\n",
    "trainer = SFTTrainer(\n",
    "    model=model,\n",
    "    tokenizer=tokenizer,\n",
    "    args=sft_config,\n",
    "    train_dataset=dataset,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eHBCU9oNEQhT"
   },
   "source": [
    "The `SFTTrainer` had already preprocessed our dataset, so we can peek inside and see how each mini-batch was assembled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "IqZZaPE5EQhT"
   },
   "outputs": [],
   "source": [
    "dl = trainer.get_train_dataloader()\n",
    "batch = next(iter(dl))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZpN8Q9zPHwMt"
   },
   "source": [
    "The mini-batch has eight elements as specified in the `per_device_train_batch_size` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MiSbckeIHm47",
    "outputId": "20828e15-e105-4ae9-fb3a-706041a8254b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(batch['input_ids'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "85A2h7ntH5Lf"
   },
   "source": [
    "Let's check the labels, after all, we didn't provide any, right?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZyttnYrEH5dx",
    "outputId": "0d7ebdb6-90aa-45cd-c1dc-bd17b2674c7c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([50265, 50265, 50265, 50265, 50265, 50265, 50265, 50265, 50266, 12105,\n",
       "         50118,  1213,   362,    49,  1159,    31,     5,   285,   334,     4,\n",
       "         50267, 50118, 50266,  2401, 33388, 50118,  7605,     5,   285,   334,\n",
       "             6,    49,  1159,    51,   362,     4,   289,   338, 41311,     4,\n",
       "         50267, 50118], device='cuda:0'),\n",
       " tensor([ -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100, 50266, 12105,\n",
       "         50118,  1213,   362,    49,  1159,    31,     5,   285,   334,     4,\n",
       "         50267, 50118, 50266,  2401, 33388, 50118,  7605,     5,   285,   334,\n",
       "             6,    49,  1159,    51,   362,     4,   289,   338, 41311,     4,\n",
       "         50267, 50118], device='cuda:0'))"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch['input_ids'][0], batch['labels'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KJ1jHu9mIyfl"
   },
   "source": [
    "The **labels were added automatically, and they are exactly the same as the inputs** (thus making this a case of self-supervised fine-tuning).\n",
    "\n",
    "The shifting of the labels is going to be handled automatically as well, there's no need to be concerned about it.\n",
    "\n",
    "Did you notice there's a difference in the beginning of both sequences? Can you guess which token that is?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 36
    },
    "id": "GKtwKecHItQb",
    "outputId": "cbe13fa9-9bf9-4a30-ac11-2326e8cbb7b1"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'<|pad|>'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.convert_ids_to_tokens(50265)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ue4wpMFyEQhU"
   },
   "source": [
    "### Training\n",
    "\n",
    "Now, we call the `train()` method and wait. It takes around 10 minutes to train for 10 epochs using Colab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 879
    },
    "id": "0cglB4p8EQhU",
    "outputId": "3b6dc5c7-592f-4b33-ec1e-dfd29d8e4799"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:316: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
      "  warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
      "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='220' max='220' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [220/220 08:29, Epoch 9/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.854800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>2.581800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>2.032500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40</td>\n",
       "      <td>1.815400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>1.653700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>1.564600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>70</td>\n",
       "      <td>1.530200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>80</td>\n",
       "      <td>1.459700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>90</td>\n",
       "      <td>1.459300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>1.421500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>110</td>\n",
       "      <td>1.424400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>120</td>\n",
       "      <td>1.413700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>130</td>\n",
       "      <td>1.366600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>140</td>\n",
       "      <td>1.361700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>1.362100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>160</td>\n",
       "      <td>1.345900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>170</td>\n",
       "      <td>1.330800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>180</td>\n",
       "      <td>1.348700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>190</td>\n",
       "      <td>1.336400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>1.323700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>210</td>\n",
       "      <td>1.323200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>220</td>\n",
       "      <td>1.332500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=220, training_loss=1.6201479825106535, metrics={'train_runtime': 514.22, 'train_samples_per_second': 14.002, 'train_steps_per_second': 0.428, 'total_flos': 513663367249920.0, 'train_loss': 1.6201479825106535, 'epoch': 9.777777777777779})"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "071c2973-6f46-4a64-ae00-a2ee5b6ba72b"
   },
   "source": [
    "After 10 epochs, the loss should be around 1.4. After 30 epochs. around 1.2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d4f19b3a"
   },
   "source": [
    "### Loading the Adapter from the Hub\n",
    "\n",
    "I have already trained the model for 30 epochs and pushed it to the hub, so we can try it right away without having to wait for the training process to finish. Moreover, since we modified the tokenizer, I have pushed it to the hub as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 292,
     "referenced_widgets": [
      "fa8f45f9408d48478ae2affe5794b31e",
      "e430d0763a6f4055b3aa22754e39caa3",
      "bd6dd9ec7c2f46d98ca2bd8bbb5b35a1",
      "bfeabd04140f4f43b7094266959e4304",
      "c7b269ebb72242b18803ebf29044d6f5",
      "5e44a9c09f9941108af3eb3aace57158",
      "3b2481073ed34f63a51d887e1aba1d9a",
      "34315f63fbfc41a895f9cd4aa75220bb",
      "8b1a5dd19d324c4680d287b721d938f5",
      "5931b6b9c6ad45bcafe1cf44f270af8a",
      "8be7c43233764b1ea94b115cde0c90e2",
      "3f589071daac409db157c589377858fe",
      "4b4ab4793c394a99b55e3390fdea131e",
      "3d22cdf3089b4c9ea39cf743376b7a2e",
      "3aefb94e617d43c6bd3c35c4723ad9ac",
      "66e4410a75e043899cbbe27dbf620ef1",
      "8d630ebbfdbe4ee9a2cffe636971a5f1",
      "d96f3e126503480291bf7aa4627d4926",
      "c8f1805c2a0243b3a7b5ae9bf1733cd8",
      "3b22b82d87b74fbda60e2af02368a4d6",
      "5a0d7b4272fe40079c235f7623d2693d",
      "e86b545b629b4a47b26505a00cf23bd8",
      "47c715eb4bf64d479c2a807ac5800920",
      "831cdb6847514df3b5e4e5c9c9ece725",
      "12aaa39c3bfe4a79a7024d37970f53d8",
      "f972e99b9db744f79d18fdb5e2c417b6",
      "995923f196f247d7b3bf79453a4678a7",
      "efcca0c56aa94c7a9d6cc9233324fd7c",
      "e6d6249fd1264b8b9671961b46d1fb8d",
      "2122d86af7d142c7b344ba9675821c4b",
      "a44cd12ce1ee41f481bd7e035e001bc1",
      "9447e14f052f4aa3986612d00ccfa863",
      "d7fb33bb215d4ebcb5b3adc84927e3b6",
      "66f35f63a0444554a885343883b437cc",
      "d79c3d9b7340430391b8136dcfe2701a",
      "68ee71cfc4e142e99023b93a055c2af0",
      "6cb47fae8ad4465baa67b5720c2945c8",
      "cd3e94a3d85c42fba2f3269ee02b3405",
      "3c8915c02d1a42a4963c0d10394f8253",
      "f09a91a8e7204837ac970d0f6b44b257",
      "2ca1f0cb802041c7a601b1230095b36c",
      "58bcf3d8d66a478083628a46d6c9b9c9",
      "d8c32e2888514a0095b8efa8d8572aa2",
      "0214c67fa4d9477fb04a3b7e30204bca",
      "6635e0d0613a4b0f83b2b3be46e12509",
      "7ba314fb6c0744cc8d3a0e5004a7050b",
      "68a8da0b470f4579b8c9fbdac9f059c0",
      "40d04535ce86490e9d3747f9afeb7ad5",
      "aa04d96248734b56adf78f32c23803c7",
      "527f8a253bfb4e8192d270b40c0a944e",
      "117c7b7719764ef180af250234a8b7c4",
      "60ac1e4dda1544cd8e5b37e21ccdae5e",
      "bbc17319d7fb4b9596efb8cfe99a6557",
      "9b49f0e967df49e8aaf53f0eb152fedc",
      "0e2b7002cdfc4551adce422b7118fd78",
      "5cac063a4e574114955254aafa01c91c",
      "5a4c50f2acf64d8d8e390c75bd2974ed",
      "2acabcccda8d4d00be0025368a41a83e",
      "bd6775a8558e4707973f7ac3c80ad53a",
      "b2339fe8cb6d427a99eb3cc31ced2e17",
      "3b058577973b42f18b4b5e7d3600b5a5",
      "ec86a9631ebf4c12bb5e8df7bae26059",
      "5c665fe8a67e447e82a8068fb28cd89b",
      "48a253a886c548e9a3350b72352a7835",
      "46f823a933334476a8c04bef8b9674f4",
      "93b9e53b9b2b44b78113fc53e0880ea2",
      "d2dcfe0b2f73433db444cc9a1abc8a17",
      "c2933bd065b64ab98ae353fcf8029253",
      "d06082f8c18b4e9b83e4c31a35b163f3",
      "b614776866e6430a8e54970ec6831fb1",
      "64b36cc577e6421fa9cbc13a3251b750",
      "d1d7ab83c20f4f53971bbde959a8d543",
      "a9871e96c328439589a648f981f64ac1",
      "fe03c488f16d43fe8c62cb3aff21b163",
      "6282bff79a304ee1bcc4d1bb5d0ad275",
      "8827256dc2aa4a9d8f580111f676a48a",
      "a845418aeefd4c3aa05a7296173a307a",
      "e1659081e2424f7598d2e8bd9b210e23",
      "81121be399ea4302bab286de4fa717fa",
      "99e6ed6d7d86495f93814b040677e362",
      "94d389650cf54d90945c940e3d238343",
      "6f3095e20e4c4191aa8c050851f3a76e",
      "c70ce598e1b14886b21c496c236358a6",
      "3d7d381ca4424528b893d1e3c1443e12",
      "bfa45efdf46d4e6aacfd932c45c45f6a",
      "cb2f2dc5c6134be68fff89ae91b37fbd",
      "73c96de5ac1f4a4599f9f681daa085d0",
      "ecaabbdc8f4e4c57b8b0317098c3f38a"
     ]
    },
    "id": "d297b8e5",
    "outputId": "542552f9-800b-4885-c057-f32761c39874"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa8f45f9408d48478ae2affe5794b31e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/1.37k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f589071daac409db157c589377858fe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/798k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47c715eb4bf64d479c2a807ac5800920",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66f35f63a0444554a885343883b437cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/2.11M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6635e0d0613a4b0f83b2b3be46e12509",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "added_tokens.json:   0%|          | 0.00/71.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5cac063a4e574114955254aafa01c91c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/552 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d2dcfe0b2f73433db444cc9a1abc8a17",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "adapter_config.json:   0%|          | 0.00/642 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1659081e2424f7598d2e8bd9b210e23",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "adapter_model.safetensors:   0%|          | 0.00/3.16M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead.\n"
     ]
    }
   ],
   "source": [
    "# After training, I pushed it to the hub like this\n",
    "# from huggingface_hub import login\n",
    "# login()\n",
    "# trainer.model.push_to_hub('dvgodoy/opt-350m-lora-yoda')\n",
    "# trainer.tokenizer.push_to_hub('dvgodoy/opt-350m-lora-yoda')\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"dvgodoy/opt-350m-lora-yoda\")\n",
    "model = AutoModelForCausalLM.from_pretrained(\"dvgodoy/opt-350m-lora-yoda\", device_map='auto')\n",
    "\n",
    "# overwriting the model in the trainer as well, so we can save it as if it were fully trained\n",
    "trainer.model = model\n",
    "trainer.tokenizer = tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RZvxYdw0JSWw"
   },
   "source": [
    "## Trying Our Model\n",
    "\n",
    "Now, our model *should* be able to understand the chat template, and produce a Yoda-like sentence as a response to any (short) sentence we provide it.\n",
    "\n",
    "So, the model requires its inputs to be formatted as such. We need to build a list of \"messages\" (only ours, the `user`, in this case) and - literally - **prompt** the model to answer by indicating it is **its turn to write**.\n",
    "\n",
    "This is the purpose of the `add_generation_prompt` argument: it adds `<|im_start|>assistant` to the end of the conversation, so the model can predict the next word (and again and again, until it predicts an `<|endoftext|>` token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "id": "WZpxxClZEQhV"
   },
   "outputs": [],
   "source": [
    "def gen_prompt(tokenizer, sentence, add_generation_prompt=True):\n",
    "    converted_sample = [\n",
    "        {\"role\": \"user\", \"content\": sentence},\n",
    "    ]\n",
    "    prompt = tokenizer.apply_chat_template(converted_sample, tokenize=False, add_generation_prompt=True)\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1TbOURqmEQhV",
    "outputId": "d987f0e0-303b-468f-dd2a-2f947778491b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "Do not give up so soon!<|im_end|>\n",
      "<|im_start|>assistant\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sentence = 'Do not give up so soon!'\n",
    "prompt = gen_prompt(tokenizer, sentence)\n",
    "print(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u845uOtEKSlp"
   },
   "source": [
    "The prompt seems about right, let's use it to generate a completion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sHXe_qB8EQhV",
    "outputId": "08c423fb-9eaa-4590-d153-f99899333df6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "Do not give up so soon!<|im_end|>\n",
      "<|im_start|>assistant\n",
      "So soon, do not give up, you must.<|im_end|>\n"
     ]
    }
   ],
   "source": [
    "print(generate(model, tokenizer, prompt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CcG_TNuWKWpA"
   },
   "source": [
    "Awesome! It works!\n",
    "\n",
    "If you trained it for 10 epochs only, it works well most of the time. If you train it for longer, say, 30 epochs, it will be even better.\n",
    "\n",
    "Let's try a few more sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pwmpp1c8EQhW",
    "outputId": "50ccbaa6-7dc7-437c-a6c4-488057b791eb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "Jump over the lazy fox.<|im_end|>\n",
      "<|im_start|>assistant\n",
      "Over the lazy fox, jump, you must.<|im_end|>\n"
     ]
    }
   ],
   "source": [
    "sentence = 'Jump over the lazy fox.'\n",
    "print(generate(model, tokenizer, gen_prompt(tokenizer, sentence)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OgzPlvweEQhW",
    "outputId": "7b324bde-6e20-4c9b-d061-746bf091c75a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "The Force is strong in this one!<|im_end|>\n",
      "<|im_start|>assistant\n",
      "Strong in this one, the Force is.<|im_end|>\n"
     ]
    }
   ],
   "source": [
    "sentence = 'The Force is strong in this one!'\n",
    "print(generate(model, tokenizer, gen_prompt(tokenizer, sentence)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3faO0uMNEQhW",
    "outputId": "af055221-15e0-4504-f096-98fd1cf73c49"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "Fine-tuning LLMs is easy.<|im_end|>\n",
      "<|im_start|>assistant\n",
      "Easy, fine-tuning LLMs is.<|im_end|>\n"
     ]
    }
   ],
   "source": [
    "sentence = 'Fine-tuning LLMs is easy.'\n",
    "print(generate(model, tokenizer, gen_prompt(tokenizer, sentence)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4ZTz7jIiKae2"
   },
   "source": [
    "We can also try a sentence from the training set itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PXP5GfMiEQhW",
    "outputId": "eb819b12-5273-43c2-cd69-804dbcda5b5e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "Whitings are small fish caught in nets.<|im_end|>\n",
      "<|im_start|>assistant\n",
      "In nets, small fish caught, whitings are.<|im_end|>\n",
      "\n",
      "Completion:\n",
      "Caught in nets, small fish, whitings are. Yes, hrrmmm.\n"
     ]
    }
   ],
   "source": [
    "i = 235\n",
    "print(generate(model, tokenizer, gen_prompt(tokenizer, dataset[i]['prompt'])))\n",
    "print(f\"\\nCompletion:\\n{dataset[i]['completion']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mtEP-sn6MO0A",
    "outputId": "b00c6f2b-f4b7-478d-fd55-026cd649ee3e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "The model speaks like Yoda.<|im_end|>\n",
      "<|im_start|>assistant\n",
      "Like Yoda, the model speak.<|im_end|>\n"
     ]
    }
   ],
   "source": [
    "sentence = 'The model speaks like Yoda.'\n",
    "print(generate(model, tokenizer, gen_prompt(tokenizer, sentence)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d6yQkOzjKeYV"
   },
   "source": [
    "**Like Yoda, the model speaks. Yes, hrmm.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8b4ffc47"
   },
   "source": [
    "## Saving The Model\n",
    "\n",
    "You may also save both adapter and tokenizer to disk, instead of pushing them to the hub:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bqqhHTJrMc-U",
    "outputId": "39cf6b27-d0fa-446f-8c78-810660362ae2"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('yoda_adapter/tokenizer_config.json',\n",
       " 'yoda_adapter/special_tokens_map.json',\n",
       " 'yoda_adapter/vocab.json',\n",
       " 'yoda_adapter/merges.txt',\n",
       " 'yoda_adapter/added_tokens.json',\n",
       " 'yoda_adapter/tokenizer.json')"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# trainer.save_model('yoda_adapter')\n",
    "\n",
    "# it is equivalent to saving both model and tokenizer\n",
    "trainer.model.save_pretrained('yoda_adapter')\n",
    "# if you modified the tokenizer, you need to save it as well\n",
    "trainer.tokenizer.save_pretrained('yoda_adapter')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2afp9NqfBdgI"
   },
   "source": [
    "## BONUS\n",
    "\n",
    "### Loading with Resized Embeddings\n",
    "\n",
    "If your embeddings were resized, you'd need to load the base model first, resize its embeddings accordingly, and **only then** load your adapter using `PeftModel.from_pretrained()` instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "056c2b8a-d67f-4a89-a53a-b5e17e4fa6a9",
    "outputId": "7f555163-d45e-49a8-c690-9169c05bab32"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PeftModelForCausalLM(\n",
       "  (base_model): LoraModel(\n",
       "    (model): OPTForCausalLM(\n",
       "      (model): OPTModel(\n",
       "        (decoder): OPTDecoder(\n",
       "          (embed_tokens): Embedding(50272, 512, padding_idx=1)\n",
       "          (embed_positions): OPTLearnedPositionalEmbedding(2050, 1024)\n",
       "          (project_out): Linear(in_features=1024, out_features=512, bias=False)\n",
       "          (project_in): Linear(in_features=512, out_features=1024, bias=False)\n",
       "          (layers): ModuleList(\n",
       "            (0-23): 24 x OPTDecoderLayer(\n",
       "              (self_attn): OPTAttention(\n",
       "                (k_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (v_proj): lora.Linear(\n",
       "                  (base_layer): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                  (lora_dropout): ModuleDict(\n",
       "                    (default): Dropout(p=0.05, inplace=False)\n",
       "                  )\n",
       "                  (lora_A): ModuleDict(\n",
       "                    (default): Linear(in_features=1024, out_features=8, bias=False)\n",
       "                  )\n",
       "                  (lora_B): ModuleDict(\n",
       "                    (default): Linear(in_features=8, out_features=1024, bias=False)\n",
       "                  )\n",
       "                  (lora_embedding_A): ParameterDict()\n",
       "                  (lora_embedding_B): ParameterDict()\n",
       "                  (lora_magnitude_vector): ModuleDict()\n",
       "                )\n",
       "                (q_proj): lora.Linear(\n",
       "                  (base_layer): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                  (lora_dropout): ModuleDict(\n",
       "                    (default): Dropout(p=0.05, inplace=False)\n",
       "                  )\n",
       "                  (lora_A): ModuleDict(\n",
       "                    (default): Linear(in_features=1024, out_features=8, bias=False)\n",
       "                  )\n",
       "                  (lora_B): ModuleDict(\n",
       "                    (default): Linear(in_features=8, out_features=1024, bias=False)\n",
       "                  )\n",
       "                  (lora_embedding_A): ParameterDict()\n",
       "                  (lora_embedding_B): ParameterDict()\n",
       "                  (lora_magnitude_vector): ModuleDict()\n",
       "                )\n",
       "                (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              )\n",
       "              (activation_fn): ReLU()\n",
       "              (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
       "              (fc1): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "              (fc2): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "              (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (lm_head): Linear(in_features=512, out_features=50272, bias=False)\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from peft import PeftModel\n",
    "\n",
    "loaded_tokenizer = AutoTokenizer.from_pretrained('yoda_adapter')\n",
    "base_model = AutoModelForCausalLM.from_pretrained(\"facebook/opt-350m\", device_map='auto')\n",
    "num_embeddings = base_model.get_input_embeddings().num_embeddings\n",
    "if num_embeddings < len(loaded_tokenizer):\n",
    "  base_model.resize_token_embeddings(len(loaded_tokenizer))\n",
    "model_to_merge = PeftModel.from_pretrained(base_model, 'yoda_adapter')\n",
    "model_to_merge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uVi-hDAQEpap"
   },
   "source": [
    "### Merging Adapters\n",
    "\n",
    "It is also possible to merge the adapters into the base model, resulting in what would have been the result of directly fine-tuning the linear layers that were adapted. It is as simgple as calling the (adapted) model's `merge_and_unload()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Xj4L7MrLBpcV",
    "outputId": "ecb2a2ca-983f-4181-8893-4b0a53a18522"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OPTForCausalLM(\n",
       "  (model): OPTModel(\n",
       "    (decoder): OPTDecoder(\n",
       "      (embed_tokens): Embedding(50272, 512, padding_idx=1)\n",
       "      (embed_positions): OPTLearnedPositionalEmbedding(2050, 1024)\n",
       "      (project_out): Linear(in_features=1024, out_features=512, bias=False)\n",
       "      (project_in): Linear(in_features=512, out_features=1024, bias=False)\n",
       "      (layers): ModuleList(\n",
       "        (0-23): 24 x OPTDecoderLayer(\n",
       "          (self_attn): OPTAttention(\n",
       "            (k_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "            (v_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "            (q_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "            (out_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "          )\n",
       "          (activation_fn): ReLU()\n",
       "          (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
       "          (fc1): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          (fc2): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "          (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (lm_head): Linear(in_features=512, out_features=50272, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_model = model_to_merge.merge_and_unload()\n",
    "merged_model.save_pretrained('yoda_model')\n",
    "merged_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UWUA5sj6FKg7"
   },
   "source": [
    "Let's take the merged model for a quick spin to make sure it is working fine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1a2e7a66-acd7-4836-800d-6c8fc65fd2b1",
    "outputId": "f1f5ccf1-86d0-4278-e15a-352ea30da4a2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "Jump over the lazy fox.<|im_end|>\n",
      "<|im_start|>assistant\n",
      "Over the lazy fox, jump, you must.<|im_end|>\n"
     ]
    }
   ],
   "source": [
    "sentence = 'Jump over the lazy fox.'\n",
    "print(generate(merged_model, tokenizer, gen_prompt(tokenizer, sentence)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sXfb0OTGKvk5"
   },
   "source": [
    "And that's a wrap!\n",
    "\n",
    "Now, it's your time to try and fine-tune your own models!\n",
    "\n",
    "<center><h1>THANK YOU!</h1></center>"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "00e395b7efa1445fbda1840d94bbda45": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "018a3aa70bce4f6f85c16ca81c0ff40d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0214c67fa4d9477fb04a3b7e30204bca": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "02e06e7e119c4a2b9baabd4a8e0e065e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0a165baed61b4894a58659acaf35eee3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0b55cf228b32451ea28058ab86348d9e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0c6f74854d184962830b221b5a462323": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "0e2b7002cdfc4551adce422b7118fd78": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "11092c1630b9408ca152e9cffec36e97": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "117c7b7719764ef180af250234a8b7c4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "11dda81baf17409296548b50f9f8977b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "11fcac8bc6584273becc268854e746c9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e7e1212abeea4cf2a14170071393611a",
      "placeholder": "​",
      "style": "IPY_MODEL_73d608ffe83142feb4a09e703b4742c3",
      "value": "special_tokens_map.json: 100%"
     }
    },
    "12aaa39c3bfe4a79a7024d37970f53d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2122d86af7d142c7b344ba9675821c4b",
      "max": 456318,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a44cd12ce1ee41f481bd7e035e001bc1",
      "value": 456318
     }
    },
    "14bfebbb42cd412cb0245f6d6b8b2072": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bf0b8c18e0014d60ae77c6bcf3a6edea",
      "max": 441,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_8bb5bfcc92914cfeacd0b1642592b7d7",
      "value": 441
     }
    },
    "17b7290809b34b67a4da25509a8d72e8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b45ddfd9d2d1476da27afd3d889075f4",
       "IPY_MODEL_aa5747f60feb40e090b86fea0e0e7a48",
       "IPY_MODEL_6f3ab98e43864ba29b00cc7d58f67193"
      ],
      "layout": "IPY_MODEL_a2653cc73f6c433c824de18b6838839f"
     }
    },
    "1aa93986329f4241a4e5f93ca40b2e46": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2abfa7db9e1646fc8947b083bba849e8",
      "placeholder": "​",
      "style": "IPY_MODEL_34e5ab638f5f4664977231d4629d2519",
      "value": " 720/720 [00:00&lt;00:00, 10870.65 examples/s]"
     }
    },
    "1f3b901a2b8d48b68f39e7e67fc57b3a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2122d86af7d142c7b344ba9675821c4b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "21f53b133fd6458ca8b903a987323271": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "226d1be905be4d34810b0023226959f6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2414528cfdf1453fb706e91b15ecb54f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a1b4fd3755f14b11973be5d034e395b9",
      "placeholder": "​",
      "style": "IPY_MODEL_fded883d80d74bd78e47ef94d40499c7",
      "value": "tokenizer_config.json: 100%"
     }
    },
    "28071d719572429b927f36da16807fa9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0b55cf228b32451ea28058ab86348d9e",
      "placeholder": "​",
      "style": "IPY_MODEL_745663eee74e4ac0a4660e0437d67cd8",
      "value": "merges.txt: 100%"
     }
    },
    "28ad6d28984e48a28bd447f0005dbe65": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2abfa7db9e1646fc8947b083bba849e8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2acabcccda8d4d00be0025368a41a83e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5c665fe8a67e447e82a8068fb28cd89b",
      "max": 552,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_48a253a886c548e9a3350b72352a7835",
      "value": 552
     }
    },
    "2b5c132ebbd243279286c9f703ddfb37": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2b7be946f2aa400c8e886fb19d58825b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f829c70029444a76b50c6ee075717401",
      "placeholder": "​",
      "style": "IPY_MODEL_a39834092155453885d7ca03534dec09",
      "value": " 720/720 [00:00&lt;00:00, 2122.28 examples/s]"
     }
    },
    "2ca1f0cb802041c7a601b1230095b36c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2f1a805e7fc845c893939a9e45589361": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2fd08216ea134591b705374c2b6832cf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "323818bc24df45e6b272d0a8ec6349a6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3389d70a664347be882edddc945e0ab0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ae35335f77cb4291ae0d434753f2ec5f",
      "max": 720,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_849601cbfbad4ac78cbe647ae57fd5fd",
      "value": 720
     }
    },
    "34315f63fbfc41a895f9cd4aa75220bb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "34e5ab638f5f4664977231d4629d2519": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3aefb94e617d43c6bd3c35c4723ad9ac": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5a0d7b4272fe40079c235f7623d2693d",
      "placeholder": "​",
      "style": "IPY_MODEL_e86b545b629b4a47b26505a00cf23bd8",
      "value": " 798k/798k [00:01&lt;00:00, 496kB/s]"
     }
    },
    "3b058577973b42f18b4b5e7d3600b5a5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3b22b82d87b74fbda60e2af02368a4d6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "3b2481073ed34f63a51d887e1aba1d9a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3c8915c02d1a42a4963c0d10394f8253": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3d22cdf3089b4c9ea39cf743376b7a2e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c8f1805c2a0243b3a7b5ae9bf1733cd8",
      "max": 798293,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_3b22b82d87b74fbda60e2af02368a4d6",
      "value": 798293
     }
    },
    "3d7d381ca4424528b893d1e3c1443e12": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3f589071daac409db157c589377858fe": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4b4ab4793c394a99b55e3390fdea131e",
       "IPY_MODEL_3d22cdf3089b4c9ea39cf743376b7a2e",
       "IPY_MODEL_3aefb94e617d43c6bd3c35c4723ad9ac"
      ],
      "layout": "IPY_MODEL_66e4410a75e043899cbbe27dbf620ef1"
     }
    },
    "3fa917fb47794f2583c0cd93bc9e87b3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "40110b82da934021972fac80dd1f1600": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "40c2599b769c48898d69c1bfa7c1930e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "40d04535ce86490e9d3747f9afeb7ad5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9b49f0e967df49e8aaf53f0eb152fedc",
      "placeholder": "​",
      "style": "IPY_MODEL_0e2b7002cdfc4551adce422b7118fd78",
      "value": " 71.0/71.0 [00:00&lt;00:00, 5.56kB/s]"
     }
    },
    "411b31cf20c14127bb22777035090dd7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4505dd5cf9324bd6b511d3fb1d0881aa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ac6e3fa9a62d4a178489f9d8c6e0239d",
       "IPY_MODEL_3389d70a664347be882edddc945e0ab0",
       "IPY_MODEL_2b7be946f2aa400c8e886fb19d58825b"
      ],
      "layout": "IPY_MODEL_7dcd7021d9a54613872f902a24788191"
     }
    },
    "46c2a2f0ce214f949736e6653ccec21f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "46f823a933334476a8c04bef8b9674f4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "47c715eb4bf64d479c2a807ac5800920": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_831cdb6847514df3b5e4e5c9c9ece725",
       "IPY_MODEL_12aaa39c3bfe4a79a7024d37970f53d8",
       "IPY_MODEL_f972e99b9db744f79d18fdb5e2c417b6"
      ],
      "layout": "IPY_MODEL_995923f196f247d7b3bf79453a4678a7"
     }
    },
    "48a253a886c548e9a3350b72352a7835": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4900ab1d1ff840e1a57a496059052854": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "492feff6ea7d4bb68b803b5f12dacd0b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_28071d719572429b927f36da16807fa9",
       "IPY_MODEL_5d5ce6b80fd44c02b023edb13dff57f7",
       "IPY_MODEL_6422020f711948b985f1774f3c221030"
      ],
      "layout": "IPY_MODEL_fe0031cc8afa4bee90f54b4739613d8a"
     }
    },
    "4b0fd8ab0cbd40d48b9a0078552da875": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_11dda81baf17409296548b50f9f8977b",
      "max": 898822,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_0c6f74854d184962830b221b5a462323",
      "value": 898822
     }
    },
    "4b4ab4793c394a99b55e3390fdea131e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8d630ebbfdbe4ee9a2cffe636971a5f1",
      "placeholder": "​",
      "style": "IPY_MODEL_d96f3e126503480291bf7aa4627d4926",
      "value": "vocab.json: 100%"
     }
    },
    "4b8ccf5250b3429c9cc71272274f2b0c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_65eb06ea48ee4bcaa52929d1de28aaf8",
       "IPY_MODEL_4b0fd8ab0cbd40d48b9a0078552da875",
       "IPY_MODEL_ce14261a6ca4435182f36cc78d476a1c"
      ],
      "layout": "IPY_MODEL_57320a7047774ad3bf3ab6881993ac6a"
     }
    },
    "50e6594c0a8f42469d55e1275f8d280d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "527f8a253bfb4e8192d270b40c0a944e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "57320a7047774ad3bf3ab6881993ac6a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "58bcf3d8d66a478083628a46d6c9b9c9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "5931b6b9c6ad45bcafe1cf44f270af8a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5a0d7b4272fe40079c235f7623d2693d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5a4c50f2acf64d8d8e390c75bd2974ed": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3b058577973b42f18b4b5e7d3600b5a5",
      "placeholder": "​",
      "style": "IPY_MODEL_ec86a9631ebf4c12bb5e8df7bae26059",
      "value": "special_tokens_map.json: 100%"
     }
    },
    "5c665fe8a67e447e82a8068fb28cd89b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5cac063a4e574114955254aafa01c91c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5a4c50f2acf64d8d8e390c75bd2974ed",
       "IPY_MODEL_2acabcccda8d4d00be0025368a41a83e",
       "IPY_MODEL_bd6775a8558e4707973f7ac3c80ad53a"
      ],
      "layout": "IPY_MODEL_b2339fe8cb6d427a99eb3cc31ced2e17"
     }
    },
    "5d5ce6b80fd44c02b023edb13dff57f7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6a231d8921994c1abcbefc81f253177e",
      "max": 456318,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5da9d9d834904f419a99a92ee95151e5",
      "value": 456318
     }
    },
    "5da9d9d834904f419a99a92ee95151e5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "5e44a9c09f9941108af3eb3aace57158": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5f6c9085c62c4fc6b3f125b29ffc3f6c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "60ac1e4dda1544cd8e5b37e21ccdae5e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6282bff79a304ee1bcc4d1bb5d0ad275": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "6422020f711948b985f1774f3c221030": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d980ea5331ef4ab58be0d6a17fcc55b0",
      "placeholder": "​",
      "style": "IPY_MODEL_d95a894605fa46d49e8306914683248b",
      "value": " 456k/456k [00:00&lt;00:00, 641kB/s]"
     }
    },
    "64b36cc577e6421fa9cbc13a3251b750": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "65eb06ea48ee4bcaa52929d1de28aaf8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6c844772e5084d2db32a8b6ccf2e18a5",
      "placeholder": "​",
      "style": "IPY_MODEL_1f3b901a2b8d48b68f39e7e67fc57b3a",
      "value": "vocab.json: 100%"
     }
    },
    "6635e0d0613a4b0f83b2b3be46e12509": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7ba314fb6c0744cc8d3a0e5004a7050b",
       "IPY_MODEL_68a8da0b470f4579b8c9fbdac9f059c0",
       "IPY_MODEL_40d04535ce86490e9d3747f9afeb7ad5"
      ],
      "layout": "IPY_MODEL_aa04d96248734b56adf78f32c23803c7"
     }
    },
    "66e4410a75e043899cbbe27dbf620ef1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "66f35f63a0444554a885343883b437cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d79c3d9b7340430391b8136dcfe2701a",
       "IPY_MODEL_68ee71cfc4e142e99023b93a055c2af0",
       "IPY_MODEL_6cb47fae8ad4465baa67b5720c2945c8"
      ],
      "layout": "IPY_MODEL_cd3e94a3d85c42fba2f3269ee02b3405"
     }
    },
    "68a8da0b470f4579b8c9fbdac9f059c0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_60ac1e4dda1544cd8e5b37e21ccdae5e",
      "max": 71,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_bbc17319d7fb4b9596efb8cfe99a6557",
      "value": 71
     }
    },
    "68ee71cfc4e142e99023b93a055c2af0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2ca1f0cb802041c7a601b1230095b36c",
      "max": 2109135,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_58bcf3d8d66a478083628a46d6c9b9c9",
      "value": 2109135
     }
    },
    "6a231d8921994c1abcbefc81f253177e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c844772e5084d2db32a8b6ccf2e18a5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6cb47fae8ad4465baa67b5720c2945c8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d8c32e2888514a0095b8efa8d8572aa2",
      "placeholder": "​",
      "style": "IPY_MODEL_0214c67fa4d9477fb04a3b7e30204bca",
      "value": " 2.11M/2.11M [00:02&lt;00:00, 970kB/s]"
     }
    },
    "6f3095e20e4c4191aa8c050851f3a76e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6f3ab98e43864ba29b00cc7d58f67193": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7c6c41ca531c4411a2852e795f828dc1",
      "placeholder": "​",
      "style": "IPY_MODEL_c4a642a94f3f4b45a62a1ead999db4db",
      "value": " 98.4k/98.4k [00:00&lt;00:00, 450kB/s]"
     }
    },
    "73c96de5ac1f4a4599f9f681daa085d0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "73d608ffe83142feb4a09e703b4742c3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "741638a62d12443fb2036ef6a82e76b2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "745663eee74e4ac0a4660e0437d67cd8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "76d11f29b0154ef0b09e29c1c79edcd2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "792d52d64ca045b29e3df8e24a479520": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_80f7f767c20f495d9d616889f9ae6727",
      "placeholder": "​",
      "style": "IPY_MODEL_4900ab1d1ff840e1a57a496059052854",
      "value": " 720/720 [00:00&lt;00:00, 4208.89 examples/s]"
     }
    },
    "7ba314fb6c0744cc8d3a0e5004a7050b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_527f8a253bfb4e8192d270b40c0a944e",
      "placeholder": "​",
      "style": "IPY_MODEL_117c7b7719764ef180af250234a8b7c4",
      "value": "added_tokens.json: 100%"
     }
    },
    "7c6c41ca531c4411a2852e795f828dc1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7da4af82216d4d56b72d69642b51428e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7dcd7021d9a54613872f902a24788191": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7f73504ba50f47d3af0cdc8b474eb671": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3fa917fb47794f2583c0cd93bc9e87b3",
      "placeholder": "​",
      "style": "IPY_MODEL_46c2a2f0ce214f949736e6653ccec21f",
      "value": " 441/441 [00:00&lt;00:00, 33.7kB/s]"
     }
    },
    "80f7f767c20f495d9d616889f9ae6727": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "81121be399ea4302bab286de4fa717fa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c70ce598e1b14886b21c496c236358a6",
      "placeholder": "​",
      "style": "IPY_MODEL_3d7d381ca4424528b893d1e3c1443e12",
      "value": "adapter_model.safetensors: 100%"
     }
    },
    "831cdb6847514df3b5e4e5c9c9ece725": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_efcca0c56aa94c7a9d6cc9233324fd7c",
      "placeholder": "​",
      "style": "IPY_MODEL_e6d6249fd1264b8b9671961b46d1fb8d",
      "value": "merges.txt: 100%"
     }
    },
    "84082bcedde2416587f2b3f3c662518f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "849601cbfbad4ac78cbe647ae57fd5fd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8827256dc2aa4a9d8f580111f676a48a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8b1a5dd19d324c4680d287b721d938f5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8bb5bfcc92914cfeacd0b1642592b7d7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8be7c43233764b1ea94b115cde0c90e2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8c2428425d444b0eb627d3a730abb05a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_00e395b7efa1445fbda1840d94bbda45",
      "placeholder": "​",
      "style": "IPY_MODEL_40c2599b769c48898d69c1bfa7c1930e",
      "value": "Map: 100%"
     }
    },
    "8d630ebbfdbe4ee9a2cffe636971a5f1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "93b9e53b9b2b44b78113fc53e0880ea2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9447e14f052f4aa3986612d00ccfa863": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "94d389650cf54d90945c940e3d238343": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_73c96de5ac1f4a4599f9f681daa085d0",
      "placeholder": "​",
      "style": "IPY_MODEL_ecaabbdc8f4e4c57b8b0317098c3f38a",
      "value": " 3.16M/3.16M [00:00&lt;00:00, 11.9MB/s]"
     }
    },
    "995923f196f247d7b3bf79453a4678a7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "99e6ed6d7d86495f93814b040677e362": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bfa45efdf46d4e6aacfd932c45c45f6a",
      "max": 3159096,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cb2f2dc5c6134be68fff89ae91b37fbd",
      "value": 3159096
     }
    },
    "9afdf8122a444753841024792d9d8c8e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2414528cfdf1453fb706e91b15ecb54f",
       "IPY_MODEL_9cb54126f17d448dafad51fdff2b1493",
       "IPY_MODEL_a22c2bf469664473ae178a19796d743f"
      ],
      "layout": "IPY_MODEL_5f6c9085c62c4fc6b3f125b29ffc3f6c"
     }
    },
    "9b49f0e967df49e8aaf53f0eb152fedc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9cb54126f17d448dafad51fdff2b1493": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_018a3aa70bce4f6f85c16ca81c0ff40d",
      "max": 685,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_226d1be905be4d34810b0023226959f6",
      "value": 685
     }
    },
    "9e3cefeade124117a19786c331952b2e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9e7640c1b9dd4ef39e4a661b2c920956": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a0792e7ab9714a6aba3f5a675585da4e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9e7640c1b9dd4ef39e4a661b2c920956",
      "placeholder": "​",
      "style": "IPY_MODEL_ed0e3855c7254c1e9971123c8c1be76b",
      "value": " 531/531 [00:00&lt;00:00, 25.1kB/s]"
     }
    },
    "a10b58bcb72f44ea916c82310786b9d6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2f1a805e7fc845c893939a9e45589361",
      "placeholder": "​",
      "style": "IPY_MODEL_50e6594c0a8f42469d55e1275f8d280d",
      "value": "Generating train split: 100%"
     }
    },
    "a1b4fd3755f14b11973be5d034e395b9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a22c2bf469664473ae178a19796d743f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_11092c1630b9408ca152e9cffec36e97",
      "placeholder": "​",
      "style": "IPY_MODEL_02e06e7e119c4a2b9baabd4a8e0e065e",
      "value": " 685/685 [00:00&lt;00:00, 31.7kB/s]"
     }
    },
    "a2653cc73f6c433c824de18b6838839f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a39834092155453885d7ca03534dec09": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a44cd12ce1ee41f481bd7e035e001bc1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "a845418aeefd4c3aa05a7296173a307a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a9871e96c328439589a648f981f64ac1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "aa04d96248734b56adf78f32c23803c7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "aa5747f60feb40e090b86fea0e0e7a48": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e05986b01b9140ce980a15d6c74c77a8",
      "max": 98437,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_40110b82da934021972fac80dd1f1600",
      "value": 98437
     }
    },
    "aaed9f2d60654d08a53ca10b4f3c22cf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ac6e3fa9a62d4a178489f9d8c6e0239d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_411b31cf20c14127bb22777035090dd7",
      "placeholder": "​",
      "style": "IPY_MODEL_bfecaf88ff994b669a8ff72681bbe8af",
      "value": "Map: 100%"
     }
    },
    "acf32e9bab24410ca4cbd73951a7b9f0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ae35335f77cb4291ae0d434753f2ec5f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b18b1cfeaf254210b3f1385a9c20ba79": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_aaed9f2d60654d08a53ca10b4f3c22cf",
      "max": 531,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_76d11f29b0154ef0b09e29c1c79edcd2",
      "value": 531
     }
    },
    "b1d77b680f094e49a7db10fecbb0780f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ffe3fa710405484092dd35c2902853da",
      "max": 720,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_28ad6d28984e48a28bd447f0005dbe65",
      "value": 720
     }
    },
    "b2339fe8cb6d427a99eb3cc31ced2e17": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b45ddfd9d2d1476da27afd3d889075f4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_741638a62d12443fb2036ef6a82e76b2",
      "placeholder": "​",
      "style": "IPY_MODEL_bb97f148c5784b7690bc61cef6c94469",
      "value": "sentences.csv: 100%"
     }
    },
    "b614776866e6430a8e54970ec6831fb1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8827256dc2aa4a9d8f580111f676a48a",
      "placeholder": "​",
      "style": "IPY_MODEL_a845418aeefd4c3aa05a7296173a307a",
      "value": " 642/642 [00:00&lt;00:00, 42.7kB/s]"
     }
    },
    "bb97f148c5784b7690bc61cef6c94469": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "bbc17319d7fb4b9596efb8cfe99a6557": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "bd6775a8558e4707973f7ac3c80ad53a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_46f823a933334476a8c04bef8b9674f4",
      "placeholder": "​",
      "style": "IPY_MODEL_93b9e53b9b2b44b78113fc53e0880ea2",
      "value": " 552/552 [00:00&lt;00:00, 26.6kB/s]"
     }
    },
    "bd6dd9ec7c2f46d98ca2bd8bbb5b35a1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_34315f63fbfc41a895f9cd4aa75220bb",
      "max": 1368,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_8b1a5dd19d324c4680d287b721d938f5",
      "value": 1368
     }
    },
    "bf0b8c18e0014d60ae77c6bcf3a6edea": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bfa45efdf46d4e6aacfd932c45c45f6a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bfeabd04140f4f43b7094266959e4304": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5931b6b9c6ad45bcafe1cf44f270af8a",
      "placeholder": "​",
      "style": "IPY_MODEL_8be7c43233764b1ea94b115cde0c90e2",
      "value": " 1.37k/1.37k [00:00&lt;00:00, 108kB/s]"
     }
    },
    "bfecaf88ff994b669a8ff72681bbe8af": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c2933bd065b64ab98ae353fcf8029253": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d1d7ab83c20f4f53971bbde959a8d543",
      "placeholder": "​",
      "style": "IPY_MODEL_a9871e96c328439589a648f981f64ac1",
      "value": "adapter_config.json: 100%"
     }
    },
    "c46602ad45684968aec7564d60331756": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_11fcac8bc6584273becc268854e746c9",
       "IPY_MODEL_14bfebbb42cd412cb0245f6d6b8b2072",
       "IPY_MODEL_7f73504ba50f47d3af0cdc8b474eb671"
      ],
      "layout": "IPY_MODEL_21f53b133fd6458ca8b903a987323271"
     }
    },
    "c4a642a94f3f4b45a62a1ead999db4db": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c4c3ac015336423ca5cd47b94f324848": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c51b38a0917d4a5f95384ad6640eec9e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a10b58bcb72f44ea916c82310786b9d6",
       "IPY_MODEL_b1d77b680f094e49a7db10fecbb0780f",
       "IPY_MODEL_792d52d64ca045b29e3df8e24a479520"
      ],
      "layout": "IPY_MODEL_0a165baed61b4894a58659acaf35eee3"
     }
    },
    "c70ce598e1b14886b21c496c236358a6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c7b269ebb72242b18803ebf29044d6f5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c8f1805c2a0243b3a7b5ae9bf1733cd8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cb2f2dc5c6134be68fff89ae91b37fbd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "cd3e94a3d85c42fba2f3269ee02b3405": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ce14261a6ca4435182f36cc78d476a1c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2fd08216ea134591b705374c2b6832cf",
      "placeholder": "​",
      "style": "IPY_MODEL_2b5c132ebbd243279286c9f703ddfb37",
      "value": " 899k/899k [00:02&lt;00:00, 420kB/s]"
     }
    },
    "ce54c54384ad4a06b0d1a8ad05103921": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9e3cefeade124117a19786c331952b2e",
      "placeholder": "​",
      "style": "IPY_MODEL_323818bc24df45e6b272d0a8ec6349a6",
      "value": "README.md: 100%"
     }
    },
    "d06082f8c18b4e9b83e4c31a35b163f3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fe03c488f16d43fe8c62cb3aff21b163",
      "max": 642,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6282bff79a304ee1bcc4d1bb5d0ad275",
      "value": 642
     }
    },
    "d1d7ab83c20f4f53971bbde959a8d543": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d2dcfe0b2f73433db444cc9a1abc8a17": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c2933bd065b64ab98ae353fcf8029253",
       "IPY_MODEL_d06082f8c18b4e9b83e4c31a35b163f3",
       "IPY_MODEL_b614776866e6430a8e54970ec6831fb1"
      ],
      "layout": "IPY_MODEL_64b36cc577e6421fa9cbc13a3251b750"
     }
    },
    "d79c3d9b7340430391b8136dcfe2701a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3c8915c02d1a42a4963c0d10394f8253",
      "placeholder": "​",
      "style": "IPY_MODEL_f09a91a8e7204837ac970d0f6b44b257",
      "value": "tokenizer.json: 100%"
     }
    },
    "d7fb33bb215d4ebcb5b3adc84927e3b6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d8c32e2888514a0095b8efa8d8572aa2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d95a894605fa46d49e8306914683248b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d96f3e126503480291bf7aa4627d4926": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d980ea5331ef4ab58be0d6a17fcc55b0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "df9f906da04d4ac8a77cce8f35818847": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_acf32e9bab24410ca4cbd73951a7b9f0",
      "max": 720,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_84082bcedde2416587f2b3f3c662518f",
      "value": 720
     }
    },
    "e05986b01b9140ce980a15d6c74c77a8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e1659081e2424f7598d2e8bd9b210e23": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_81121be399ea4302bab286de4fa717fa",
       "IPY_MODEL_99e6ed6d7d86495f93814b040677e362",
       "IPY_MODEL_94d389650cf54d90945c940e3d238343"
      ],
      "layout": "IPY_MODEL_6f3095e20e4c4191aa8c050851f3a76e"
     }
    },
    "e3567fd081844a4885c67b52cfceb93d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8c2428425d444b0eb627d3a730abb05a",
       "IPY_MODEL_df9f906da04d4ac8a77cce8f35818847",
       "IPY_MODEL_1aa93986329f4241a4e5f93ca40b2e46"
      ],
      "layout": "IPY_MODEL_c4c3ac015336423ca5cd47b94f324848"
     }
    },
    "e430d0763a6f4055b3aa22754e39caa3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5e44a9c09f9941108af3eb3aace57158",
      "placeholder": "​",
      "style": "IPY_MODEL_3b2481073ed34f63a51d887e1aba1d9a",
      "value": "tokenizer_config.json: 100%"
     }
    },
    "e6d6249fd1264b8b9671961b46d1fb8d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e7e1212abeea4cf2a14170071393611a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e86b545b629b4a47b26505a00cf23bd8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ec86a9631ebf4c12bb5e8df7bae26059": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ecaabbdc8f4e4c57b8b0317098c3f38a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ed0e3855c7254c1e9971123c8c1be76b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "efcca0c56aa94c7a9d6cc9233324fd7c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f09a91a8e7204837ac970d0f6b44b257": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f829c70029444a76b50c6ee075717401": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f972e99b9db744f79d18fdb5e2c417b6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9447e14f052f4aa3986612d00ccfa863",
      "placeholder": "​",
      "style": "IPY_MODEL_d7fb33bb215d4ebcb5b3adc84927e3b6",
      "value": " 456k/456k [00:00&lt;00:00, 612kB/s]"
     }
    },
    "fa8f45f9408d48478ae2affe5794b31e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e430d0763a6f4055b3aa22754e39caa3",
       "IPY_MODEL_bd6dd9ec7c2f46d98ca2bd8bbb5b35a1",
       "IPY_MODEL_bfeabd04140f4f43b7094266959e4304"
      ],
      "layout": "IPY_MODEL_c7b269ebb72242b18803ebf29044d6f5"
     }
    },
    "fb4bd49a8e9242f2a8e8a203fbf819d9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ce54c54384ad4a06b0d1a8ad05103921",
       "IPY_MODEL_b18b1cfeaf254210b3f1385a9c20ba79",
       "IPY_MODEL_a0792e7ab9714a6aba3f5a675585da4e"
      ],
      "layout": "IPY_MODEL_7da4af82216d4d56b72d69642b51428e"
     }
    },
    "fded883d80d74bd78e47ef94d40499c7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fe0031cc8afa4bee90f54b4739613d8a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fe03c488f16d43fe8c62cb3aff21b163": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ffe3fa710405484092dd35c2902853da": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}