2 Summarization.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"accelerator":"GPU","anaconda-cloud":{},"colab":{"name":"2 Summarization.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.5"}},"cells":[{"cell_type":"markdown","metadata":{"id":"Lvk67QawFLp4"},"source":["# Introduction"]},{"cell_type":"markdown","metadata":{"id":"w9k8OvG5FLqU"},"source":["<center><h3>**Welcome to the Summarization Notebook.**</h3></center>\n","\n","In this assignment, you are going to train a neural network to summarize news articles.\n","Your neural network is going to learn from example, as we provide you with (article, summary) pairs.\n","We provide you with a **toy dataset** made of only articles about police related news.\n","Usual datasets can be 20x larger in size, but we have reduced it for computational purposes.\n","\n","You will do this using a Transformer network, from the __[Attention is all you need](http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf)__ paper.\n","In this assignment you will:\n","- Learn to process text into sub-word tokens, to avoid fixed vocabulary sizes, and UNK tokens.\n","- Implement the key conceptual blocks of a Transformer.\n","- Use a Transformer to read a news article, and produce a summary.\n","- Perform operations on learned word-vectors to examine what the model has learned.\n","\n","    \n","** Before you start **\n","\n","You should read the Attention is all you need paper.\n","We are providing you with skeleton code for the Transformer, but there will have to implement 5 conceptual blocks of the transformer yourself:\n","-  AttentionQKV: the Query, Key, Value attention mechanism at the center of the Transformer\n","- MultiHeadAttention: the multiple heads that enable each input to attend at many places at once.\n","- PositionEmbedding: the sinusoid-based position embedding of the Transformer.\n","- Encoder & Decoder: The encoder (that reads inputs, such as news articles), the decoder (that produces the output summary, one token at a time)\n","- Full Transformer: piecing it all together."]},{"cell_type":"markdown","metadata":{"id":"fJVenhu9FLqX"},"source":["All dataset files should be placed in the `dataset/` folder of this assignment.\n","\n","If you are using Google Colab, follow the instructions to mount your Google Drive onto the remote machine."]},{"cell_type":"markdown","metadata":{"id":"LLzv6ksIFLqZ"},"source":["# Library imports"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IdFfsm8rI5ZG","executionInfo":{"status":"ok","timestamp":1617255855305,"user_tz":420,"elapsed":7367,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"cdf51b34-a81c-4dd5-fd95-d648b382dc58"},"source":["!pip install segtok\n","!pip install sentencepiece"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Collecting segtok\n","  Downloading https://files.pythonhosted.org/packages/41/08/582dab5f4b1d5ca23bc6927b4bb977c8ff7f3a87a3b98844ef833e2f5623/segtok-1.5.10.tar.gz\n","Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from segtok) (2019.12.20)\n","Building wheels for collected packages: segtok\n","  Building wheel for segtok (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for segtok: filename=segtok-1.5.10-cp37-none-any.whl size=25019 sha256=21322f707d0bc481f120b05a64ef3b5f19b500374b6a81b1d92383d9a0611684\n","  Stored in directory: /root/.cache/pip/wheels/b4/39/f6/9ca1c5cabde964d728023b5751c3a206a5c8cc40252321fb6b\n","Successfully built segtok\n","Installing collected packages: segtok\n","Successfully installed segtok-1.5.10\n","Collecting sentencepiece\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f5/99/e0808cb947ba10f575839c43e8fafc9cc44e4a7a2c8f79c60db48220a577/sentencepiece-0.1.95-cp37-cp37m-manylinux2014_x86_64.whl (1.2MB)\n","\u001b[K     |████████████████████████████████| 1.2MB 19.4MB/s \n","\u001b[?25hInstalling collected packages: sentencepiece\n","Successfully installed sentencepiece-0.1.95\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"fuuqU4Bvhx7-"},"source":["Run the first of the following two cells if you are running the homework locally, and run the second cell if you are running the homework in Colab"]},{"cell_type":"code","metadata":{"id":"aGyL5lIXI7qd"},"source":["DRIVE=False\n","root_folder = \"\"\n","dataset_folder = \"dataset/\""],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"kQS9KpDJhx7_","executionInfo":{"status":"ok","timestamp":1617255894694,"user_tz":420,"elapsed":31185,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"5678a2e0-8a72-4580-df9b-e8828ea8a1ad"},"source":["from google.colab import drive\n","drive.mount('/content/drive')\n","root_folder = \"/content/drive/My Drive/cs182_hw3/\"\n","dataset_folder = \"/content/drive/My Drive/cs182_hw3_public/dataset/\""],"execution_count":2,"outputs":[{"output_type":"stream","text":["Mounted at /content/drive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"qdIkEfLxFLqb","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617246271752,"user_tz":420,"elapsed":609,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"0a2f2309-b45b-4388-ad3f-4d3e05da9977"},"source":["# This cell autoreloads the notebook when you change you python file code.\n","# If you think the notebook did not reload, rerun this cell.\n","%load_ext autoreload\n","%autoreload 2"],"execution_count":null,"outputs":[{"output_type":"stream","text":["The autoreload extension is already loaded. To reload it, use:\n","  %reload_ext autoreload\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"hVY5twBBFLqe","executionInfo":{"status":"ok","timestamp":1617255930527,"user_tz":420,"elapsed":292,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}}},"source":["import os\n","import sys\n","sys.path.append(root_folder)\n","from transformer import Transformer\n","import sentencepiece as spm\n","import torch as th\n","from torch import nn\n","from torch.nn import functional as F\n","from torch import optim\n","import numpy as np\n","import json\n","import capita\n","import os\n","from transformer_utils import set_device\n","import gc\n","from utils import validate_to_array, model_out_to_list\n","\n","list_to_device = lambda th_obj: [tensor.to(device) for tensor in th_obj]"],"execution_count":4,"outputs":[]},{"cell_type":"code","metadata":{"id":"1IDVCg83FLqi","executionInfo":{"status":"ok","timestamp":1617255933555,"user_tz":420,"elapsed":1176,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}}},"source":["# Load the word piece model that will be used to tokenize the texts into\n","# word pieces with a vocabulary size of 10000\n","sp = spm.SentencePieceProcessor()\n","sp.Load(root_folder+\"dataset/wp_vocab10000.model\")\n","\n","vocab = [line.split('\\t')[0] for line in open(root_folder+\"dataset/wp_vocab10000.vocab\", \"r\")]\n","pad_index = vocab.index('#')\n","\n","def pad_sequence(numerized, pad_index, to_length):\n","    pad = numerized[:to_length]\n","    padded = pad + [pad_index] * (to_length - len(pad))\n","    mask = [w != pad_index for w in padded]\n","    return padded, mask"],"execution_count":5,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"591lVK1BFLqk"},"source":["# Building blocks of a Transformer\n"]},{"cell_type":"markdown","metadata":{"id":"Ekya1G4qFLqm"},"source":["**TODO**:\n","\n","Implement the 5 blocks of the Transformer. In order to finish this section, you should get very small error <1e-7 on each of the 5 checks in this section.\n","\n","\n","The Transformer is split into 3 files: transformer_attention.py, transformer_utils.py and transformer.py\n","\n","Each section below gives you directions and a way to verify your code works properly.\n","\n","You do not need to modify the rest of the code provided, but should read it to understand overall architecture.\n","\n","Our Transformer is built as a Pytorch model, a standard that is good for you to get accustomed to.\n","\n"]},{"cell_type":"markdown","metadata":{"id":"bG8RDjwKFLqo"},"source":["## (1) Implementing the Query-Key-Value Attention (AttentionQKV)"]},{"cell_type":"markdown","metadata":{"id":"WurZuO8eFLqp"},"source":["This part is located in AttentionQKV in transformer_attention.py. You must implement the call function of the class.\n","You will need to implement the mathematical procedure of AttentionQKV that is described in the [Attention is all you need paper](https://arxiv.org/pdf/1706.03762.pdf)."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"kjrFRxw3FLqr","executionInfo":{"status":"ok","timestamp":1617255937047,"user_tz":420,"elapsed":804,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"6821b8b4-66e3-422c-d4df-7222007313ea"},"source":["from transformer_attention import AttentionQKV\n","\n","batch_size = 2;\n","n_queries = 3;\n","n_keyval = 5;\n","depth_k = 2;\n","depth_v = 2\n","\n","with open(root_folder+\"transformer_checks/attention_qkv_io.json\", \"r\") as f:\n","    io = json.load(f)\n","    queries = th.tensor(io['queries'])\n","    keys = th.tensor(io['keys'])\n","    values = th.tensor(io['values'])\n","    expected_output  = th.tensor(io['output'])\n","    expected_weights = th.tensor(io['weights'])\n","\n","attn_qkv = AttentionQKV()\n","output, weights = attn_qkv(queries, keys, values)\n","validate_to_array(model_out_to_list,((queries,keys,values),attn_qkv),'attentionqkv', root_folder)\n","print(\"Total error on the output:\",th.sum(th.abs(expected_output-output)).item(), \"(should be 0.0 or close to 0.0)\")\n","print(\"Total error on the weights:\",th.sum(th.abs(expected_weights-weights)).item(), \"(should be 0.0 or close to 0.0)\")"],"execution_count":6,"outputs":[{"output_type":"stream","text":["Total error on the output: 3.129243850708008e-07 (should be 0.0 or close to 0.0)\n","Total error on the weights: 1.6205012798309326e-07 (should be 0.0 or close to 0.0)\n"],"name":"stdout"},{"output_type":"stream","text":["/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order, subok=True)\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"id":"WvgZcum5FLqu"},"source":["## (2) Implementing Multi-head attention"]},{"cell_type":"markdown","metadata":{"id":"puN9fYvyFLqu"},"source":["This part is located in the class MultiHeadProjection in transformer_attention.py.\n","You must implement the call, \\_split_heads, and \\_combine_heads functions.\n","\n","**Procedure**\n","\n","The objective is to leverage the AttentionQKV class you already wrote.\n","\n","Your input are the queries, keys, values as 3-d tensors (batch_size, sequence_length, feature_size).\n","\n","Split them into 4-d tensors (batch_size, n_heads, sequence_length, new_feature_size). Where:\n","$$feature\\_size = n\\_heads * new_{feature\\_size}.$$\n","\n","You can then feed the split qkv to your implemented AttentionQKV, which will treat each head as an independent attention function.\n","\n","Then the output must be combined back into a 3-d tensor.\n","You can test the validity of your implementation in the cell below."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"nZPz8ux1FLqw","executionInfo":{"status":"ok","timestamp":1617255941538,"user_tz":420,"elapsed":552,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"e8923d31-5729-4a2e-9b19-1514e3ee9185"},"source":["from transformer_attention import MultiHeadProjection\n","\n","batch_size = 2;\n","n_queries = 3;\n","n_heads = 4\n","n_keyval = 5;\n","depth_k = 8;\n","depth_v = 8;\n","\n","with open(root_folder+\"transformer_checks/multihead_io.json\", \"r\") as f:\n","    io = json.load(f)\n","    queries = th.tensor(io['queries'])\n","    keys = th.tensor(io['keys'])\n","    values = th.tensor(io['values'])\n","    expected_output  = th.tensor(io['output'])\n","\n","mhp = MultiHeadProjection(n_heads, (depth_k,depth_v))\n","multihead_output = mhp((queries, keys, values))\n","validate_to_array(model_out_to_list,(((queries,keys,values),),mhp),'multihead', root_folder)\n","print(\"Total error on the output:\",th.sum(th.abs(expected_output-multihead_output)).item(), \"(should be 0.0 or close to 0.0)\")"],"execution_count":7,"outputs":[{"output_type":"stream","text":["Total error on the output: 1.369975507259369e-06 (should be 0.0 or close to 0.0)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ZGU6SQ2RFLqz"},"source":["## (3) Position Embedding "]},{"cell_type":"markdown","metadata":{"id":"W_MEJtfTFLq0"},"source":["You must implement the FeedForward and PositionEmbedding classes in transformer.py.\n","\n","\n","The cell below helps you verify the validity of your implementation\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"AyKCvdAnFLq1","executionInfo":{"status":"ok","timestamp":1617255973034,"user_tz":420,"elapsed":10970,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"20bbc88f-f888-41d4-f19a-f185d3c7cad9"},"source":["from transformer import PositionEmbedding\n","\n","batch_size = 2;\n","sequence_length = 3;\n","dim = 4;\n","\n","with open(root_folder+\"transformer_checks/position_embedding_io.json\", \"r\") as f:\n","    io = json.load(f)\n","    inputs = th.tensor(io['inputs'])\n","    expected_output  = th.tensor(io['output'])\n","\n","pos_emb = PositionEmbedding(dim)\n","(inputs,expected_output,pos_emb) = list_to_device((inputs,expected_output,pos_emb))\n","output_t = pos_emb(inputs)\n","validate_to_array(model_out_to_list,((inputs,),pos_emb),'position_embedding', root_folder)\n","print(\"Total error on the output:\",th.sum(th.abs(expected_output-output_t)).item(), \"(should be 0.0 or close to 0.0)\")"],"execution_count":11,"outputs":[{"output_type":"stream","text":["Total error on the output: 2.980232238769531e-07 (should be 0.0 or close to 0.0)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ZpmSj3lKFLq3"},"source":["## (4) Transformer Encoder / Transformer Decoder"]},{"cell_type":"markdown","metadata":{"id":"k_OPPPkQFLq4"},"source":["You now have all the blocks needed to implement the Transformer.\n","For this part, you have to fill in 2 classes in the transformer.py file: TransformerEncoderBlock, TransformerDecoderBlock.\n","\n","The code below will verify the accuracy of each block"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-eTL20vGFLq5","executionInfo":{"status":"ok","timestamp":1617255982205,"user_tz":420,"elapsed":882,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"4261fba4-a0f3-4ddf-a79c-1bf2b764e5ef"},"source":["from transformer import TransformerEncoderBlock\n","\n","batch_size = 2\n","sequence_length = 5\n","hidden_size = 6\n","filter_size = 12\n","n_heads = 2\n","\n","with open(root_folder+\"transformer_checks/transformer_encoder_block_io_new.json\", \"r\") as f:\n","    io = json.load(f)\n","    inputs = th.tensor(io['inputs'])\n","    expected_output = th.tensor(io['output'])\n","enc_block = TransformerEncoderBlock(input_size=6, n_heads=n_heads, filter_size=filter_size, hidden_size=hidden_size)\n","# th.save(enc_block.state_dict(),root_folder+\"transformer_checks/transformer_encoder_block\")\n","enc_block.load_state_dict(th.load(root_folder+\"transformer_checks/transformer_encoder_block\"))\n","(inputs,expected_output,enc_block) = list_to_device((inputs,expected_output,enc_block))\n","output_t = enc_block(inputs)\n","validate_to_array(model_out_to_list,((inputs,),enc_block),'encoder_block', root_folder)\n","print(\"Total error on the output:\",th.sum(th.abs(expected_output-output_t)).item(), \"(should be 0.0 or close to 0.0)\")"],"execution_count":12,"outputs":[{"output_type":"stream","text":["Total error on the output: 4.217028617858887e-06 (should be 0.0 or close to 0.0)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"tdyhiZtsFLq7","executionInfo":{"status":"ok","timestamp":1617255985117,"user_tz":420,"elapsed":832,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"a5b33754-84f4-4d9a-ea88-88d71026e4cf"},"source":["from transformer import TransformerDecoderBlock\n","batch_size = 2\n","encoder_length = 5\n","decoder_length = 3\n","hidden_size = 6\n","filter_size = 12\n","n_heads = 2\n","\n","with open(root_folder+\"transformer_checks/transformer_decoder_block_io_new.json\", \"r\") as f:\n","    io = json.load(f)\n","    decoder_inputs = th.tensor(io['decoder_inputs'])\n","    encoder_output = th.tensor(io['encoder_output'])\n","    expected_output = th.tensor(io['output'])\n","\n","dec_block = TransformerDecoderBlock(input_size=6, n_heads=n_heads, filter_size=filter_size, hidden_size=hidden_size)\n","dec_block.load_state_dict(th.load(root_folder+\"transformer_checks/transformer_decoder_block\"))\n","(decoder_inputs,encoder_output,expected_output,dec_block) = list_to_device((decoder_inputs,encoder_output,expected_output,dec_block))\n","output_t = dec_block(decoder_inputs, encoder_output)\n","validate_to_array(model_out_to_list,((decoder_inputs, encoder_output),dec_block),'decoder_block', root_folder)\n","print(\"Total error on the output:\",th.sum(th.abs(expected_output-output_t)).item(), \"(should be 0.0 or close to 0.0)\")\n"],"execution_count":13,"outputs":[{"output_type":"stream","text":["Total error on the output: 4.231929779052734e-06 (should be 0.0 or close to 0.0)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"FzuLYetAFLq9"},"source":["## (5) Transformer"]},{"cell_type":"markdown","metadata":{"id":"j4yTgzFzFLq_"},"source":["This is the final high-level function that pieces it all together.\n","\n","You have to implement the call function of the Transformer class in the `transformer.py` file.\n","\n","The block below verifies your implementation is correct."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"5QHb6nyeFLrB","executionInfo":{"status":"ok","timestamp":1617255991922,"user_tz":420,"elapsed":910,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"684ef348-0a42-4931-954a-78fc8e3f19b9"},"source":["from transformer import Transformer\n","\n","batch_size = 2\n","vocab_size = 11\n","n_layers = 3\n","n_heads = 4\n","d_model = 8\n","d_filter = 16\n","input_length = 5\n","output_length = 3\n","\n","with open(root_folder+\"transformer_checks/transformer_io_new.json\", \"r\") as f:\n","    io = json.load(f)\n","    enc_input = th.tensor(io['enc_input'])\n","    dec_input = th.tensor(io['dec_input'])\n","    enc_mask = th.tensor(io['enc_mask'])\n","    dec_mask = th.tensor(io['dec_mask'])\n","    expected_output = th.tensor(io['output'])\n","transformer = Transformer(vocab_size=vocab_size, n_layers=n_layers, n_heads=n_heads, d_model=d_model, d_filter=d_filter)\n","transformer.load_state_dict(th.load(root_folder+\"transformer_checks/transformer\"))\n","(enc_input,dec_input,enc_mask,dec_mask,expected_output,transformer) \\\n","    = list_to_device((enc_input,dec_input,enc_mask,dec_mask,expected_output,transformer))\n","output_t = transformer(enc_input, target_sequence=dec_input, encoder_mask=enc_mask, decoder_mask=dec_mask)\n","validate_to_array(model_out_to_list, ((enc_input, dec_input, enc_mask, dec_mask),transformer),'transformer', root_folder)\n","print(\"Total error on the output:\",th.sum(th.abs(expected_output-output_t)).item(), \"(should be 0.0 or close to 0.0)\")"],"execution_count":14,"outputs":[{"output_type":"stream","text":["Total error on the output: 5.14984130859375e-05 (should be 0.0 or close to 0.0)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6usohlnXFLrD"},"source":["# Training the model"]},{"cell_type":"markdown","metadata":{"id":"6pukC3qFFLrF"},"source":["Your objective is to train the Language on the dataset you are provided to reach a **validation loss <= 6.50**\n","\n","Careful: we will be testing this loss on an unreleased test set, so make sure to evaluate properly on a validation set and not overfit.\n","\n","You must save the model you want us to test under: models/final_transformer_summarization (the .index, .meta and .data files)\n","\n","**Advice**:\n","- It should be possible to attain validation loss <= 6.50 with the model dimensions we've specified (n_layers=6, d_model=104, d_filter=416), but you can tune these hyperparameters. Increasing d_model will yield better model, at the cost of longer training time.\n","- You should try tuning the learning rate, as well as what optimizer you use.\n","- You might need to train for a few (up to 2 hours) to obtain our expected loss. Remember to tune your hyperparameters first, once you find ones that work well, let it train for longer.\n","\n","**Dataset**: as in the previous notebook, make sure the dataset files are in the `dataset` folder. These can be found on the Google Drive.\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"60AayLbxFLrH","executionInfo":{"status":"ok","timestamp":1617256009753,"user_tz":420,"elapsed":13101,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"2cb9a956-05dd-48ff-a7ee-6955050f2a7e"},"source":["with open(root_folder+\"dataset/summarization_dataset_preprocessed.json\", \"r\") as f:\n","    dataset = json.load(f)\n","\n","# We load the dataset, and split it into 2 sub-datasets based on if they are training or validation.\n","# Feel free to split this dataset another way, but remember, a validation set is important, to have an idea of \n","# the amount of overfitting that has occurred!\n","\n","d_train = [d for d in dataset if d['cut'] == 'training']\n","d_valid = [d for d in dataset if d['cut'] == 'evaluation']\n","\n","len(d_train), len(d_valid)"],"execution_count":15,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(61055, 1558)"]},"metadata":{"tags":[]},"execution_count":15}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"DSNmSXaxFLrI","executionInfo":{"status":"ok","timestamp":1617256017384,"user_tz":420,"elapsed":318,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"f0cb19a9-5842-43fd-9ac3-eb7315d30188"},"source":["# An example (article, summary) pair in the training data:\n","\n","print(d_train[145]['story'])\n","print(\"=======================\\n=======================\")\n","print(d_train[145]['summary'])"],"execution_count":16,"outputs":[{"output_type":"stream","text":["Tbilisi, Georgia (CNN)Police have shot and killed a white tiger that killed a man Wednesday in Tbilisi, Georgia, a Ministry of Internal Affairs representative said, after severe flooding allowed hundreds of wild animals to escape the city zoo. \n","The tiger attack happened at a warehouse in the city center. The animal had been unaccounted for since the weekend floods destroyed the zoo premises.\n","The man killed, who was 43, worked in a company based in the warehouse, the Ministry of Internal Affairs said. Doctors said he was attacked in the throat and died before reaching the hospital. \n","Experts are still searching the warehouse, the ministry said, adding that earlier reports that the tiger had injured a second man were unfounded. \n","The zoo administration said Wednesday that another tiger was still missing. It was unable to confirm if the creature was dead or had escaped alive.\n","Georgian Prime Minister Irakli Garibashvili apologized to the public, saying he had been misinformed by the zoo's management when he'd previously said there were no more dangerous animals on the run.\n","City residents were urged to stay indoors for their own safety in the immediate aftermath of the floods. Volunteers have since been helping city workers with the cleanup operation.\n","At least 19 people died in the flooding, according to Civil Georgia, a news website run by the nongovernmental organization United Nations Association of Georgia. Six more remained missing, it said Tuesday, citing the State Security and Crisis Management Council.\n","Meanwhile, the zoo lost about half of its 600 animals, including lions, tigers, bears and wolves, in the natural disaster. \n","Some animals have since been recaptured, Civil Georgia reported. Others died in the floods or have been killed by police as they scour the streets for escapees.\n","Russian state news outlet RT.com  that an African penguin had made it 60 kilometers (37 miles) downriver from Tbilisi before being caught alive in a dragnet on the border with Azerbaijan. \n","Video from the city showed a large crocodile being restrained by rescuers, as well as a hippopotamus standing in floodwaters, looking confused.\n","The latter was eventually cornered in a city square before being tranquilized and recaptured.\n","One terrified bear escaped the flood by perching on a window ledge.\n","Video footage also showed devastation across swaths of the Georgian capital, where flash floods swept away roads, at least one house and many trees. The corpses of dead animals could be seen amid the wreckage.\n","The problems began before midnight Saturday when heavy rainfall turned the Vere River, usually little more than a stream through the center of Tbilisi, into a raging torrent, according to Civil Georgia.\n","Images on Tbilisi City Hall's Facebook page showed roads washed out, hillsides collapsed and vehicles tossed about like toys. Rescue workers carried people on their shoulders through waist-high water.\n","Garibashvili extended his condolences Tuesday to the families of those killed in the flooding.\n","He also proposed the creation of a park in the zoo premises to honor those lost. \"It will be a park of solidarity, a symbol of our unity, selflessness, and mutual support,\" he said in a statement on his website.\n","President Georgi Margvelashvili earlier said the capital's mayoral office would help those who had lost out financially as a result of the floods.\n","\"The situation is difficult, but it can be handled except for the fact that we cannot bring back those who died,\" he said.\n","According to the World Wildlife Fund, as few as 3,200 tigers exist in the wild today.\n","\n","Journalist Eka Kadagishvili reported from Tbilisi, and Laura Smith-Spark wrote from London. CNN's Kimberly Hutcherson contributed to this report.\n","=======================\n","=======================\n","Police have shot dead a tiger that killed a man in Tbilisi, Georgia, a government official says, after zoo animals escaped in weekend flooding.\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"CRRN1hJfFLrJ"},"source":["Similarly to the previous assignment, we create a function to get a random batch to train on, given a dataset."]},{"cell_type":"code","metadata":{"id":"3OLRb5bWFLrK","executionInfo":{"status":"ok","timestamp":1617256019854,"user_tz":420,"elapsed":288,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}}},"source":["def build_batch(dataset, batch_size):\n","    indices = list(np.random.randint(0, len(dataset), size=batch_size))\n","    \n","    batch = [dataset[i] for i in indices]\n","    batch_input = np.array([a['input'] for a in batch])\n","    batch_input_mask = np.array([a['input_mask'] for a in batch])\n","    batch_output = np.array([a['output'] for a in batch])\n","    batch_output_mask = np.array([a['output_mask'] for a in batch])\n","    \n","    return batch_input, batch_input_mask, batch_output, batch_output_mask"],"execution_count":17,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"T4aVeGB-FLrL"},"source":["We now instantiate the Transformer with our sets of hyperparameters specific to the task of summarization.\n","In summarization, we are going to go from documents with up to 400 words, to documents with up to 100 words.\n","The vocabulary size is set for you, and is of 10,000 words (we are using WordPieces, [here is a paper about subword encoding](http://aclweb.org/anthology/P18-1007), if you are interested)."]},{"cell_type":"code","metadata":{"id":"61JXS8xNFLrL","executionInfo":{"status":"ok","timestamp":1617256024654,"user_tz":420,"elapsed":321,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}}},"source":["# Use this trainer to train a Transformer model\n","\n","class TransformerTrainer(nn.Module):\n","    def __init__(self, vocab_size, d_model, input_length, output_length, n_layers, d_filter, dropout=0, learning_rate=1e-3):\n","        super().__init__()\n","        self.model = Transformer(vocab_size=vocab_size, d_model=d_model, n_layers=n_layers, d_filter=d_filter)\n","\n","        # Summarization loss\n","        criterion = nn.CrossEntropyLoss(reduce='none')\n","        self.loss_fn = lambda pred,target,mask: (criterion(pred.permute(0,2,1),target)*mask).sum()/mask.sum()\n","        self.learning_rate = learning_rate\n","        self.optimizer = optim.Adam(self.model.parameters(), lr=learning_rate)\n","    def forward(self,batch,optimize=True):\n","        pred_logits = self.model(**batch)\n","        target,mask = batch['target_sequence'],batch['decoder_mask']\n","        loss = self.loss_fn(pred_logits,target,mask)\n","        accuracy = (th.eq(pred_logits.argmax(dim=2,keepdim=False),target).float()*mask).sum()/mask.sum()\n","        \n","        if optimize:\n","            self.optimizer.zero_grad()\n","            loss.backward()\n","            self.optimizer.step()\n","                \n","        return loss, accuracy"],"execution_count":18,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"xtAVhYXXFLrN","executionInfo":{"status":"ok","timestamp":1617256826391,"user_tz":420,"elapsed":355,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"6b288005-396f-4389-a914-93ef0a340386"},"source":["# Dataset related parameters\n","vocab_size = len(vocab)\n","ilength = 400 # Length of the article\n","olength  = 100 # Length of the summaries\n","\n","# Model related parameters, feel free to modify these.\n","n_layers = 6\n","d_model  = 160\n","d_filter = 4*d_model\n","batch_size = 16\n","\n","dropout = 0\n","learning_rate = 1e-3\n","trainer = TransformerTrainer(vocab_size, d_model, ilength, olength, n_layers, d_filter, dropout)\n","model_id = 'test1'\n","os.makedirs(root_folder+'models/part2/',exist_ok=True)\n","\n","device = th.device(\"cuda\" if th.cuda.is_available() else \"cpu\")\n","print(device)\n","set_device(device)"],"execution_count":36,"outputs":[{"output_type":"stream","text":["cuda\n"],"name":"stdout"},{"output_type":"stream","text":["/usr/local/lib/python3.7/dist-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.\n","  warnings.warn(warning.format(ret))\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2jEzZqG0FLrO","executionInfo":{"status":"ok","timestamp":1617257020148,"user_tz":420,"elapsed":191472,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"5d47251d-59f3-4b79-c2ed-a00343d22e92"},"source":["# Skeleton code, as in the previous notebook.\n","# Write code training code and save your best performing model on the\n","# validation set. We will be testing the loss on a held-out test dataset.\n","from tqdm import tqdm\n","gc.collect()\n","trainer.model.to(device)\n","trainer.model.train()\n","losses,accuracies = [],[]\n","t = tqdm(range(int(1e3)+1))\n","for i in t:\n","    # Create a random mini-batch from the training dataset\n","    batch = build_batch(d_train, batch_size)\n","    # Build the feed-dict connecting placeholders and mini-batch\n","    batch_input, batch_input_mask, batch_output, batch_output_mask = [th.tensor(tensor) for tensor in batch]\n","    batch_input, batch_input_mask, batch_output, batch_output_mask \\\n","                = list_to_device([batch_input, batch_input_mask, batch_output, batch_output_mask])\n","    batch = {'source_sequence': batch_input, 'target_sequence': batch_output,\n","            'encoder_mask': batch_input_mask, 'decoder_mask': batch_output_mask}\n","\n","    # Obtain the loss. Be careful when you use the train_op and not, as previously.\n","    train_loss, accuracy = trainer(batch)\n","    losses.append(train_loss.item()),accuracies.append(accuracy.item())\n","    if i % 10 == 0:\n","        t.set_description(f\"Iteration: {i} Loss: {np.mean(losses[-10:])} Accuracy: {np.mean(accuracies[-10:])}\")\n","    if i % 100 == 0:\n","        save_dict = dict(\n","            kwargs = dict(\n","                vocab_size=vocab_size,\n","                d_model=d_model,\n","                n_layers=n_layers, \n","                d_filter=d_filter\n","            ),\n","            model_state_dict = trainer.model.state_dict(),\n","            notes = \"\"\n","        )\n","        th.save(save_dict, root_folder+f'models/part2/model_{model_id}.pt')"],"execution_count":37,"outputs":[{"output_type":"stream","text":["\n","  0%|          | 0/1001 [00:00<?, ?it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   0%|          | 0/1001 [00:00<?, ?it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   0%|          | 1/1001 [00:00<06:13,  2.68it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   0%|          | 2/1001 [00:00<05:19,  3.13it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   0%|          | 3/1001 [00:00<04:39,  3.57it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   0%|          | 4/1001 [00:00<04:12,  3.96it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   0%|          | 5/1001 [00:01<03:51,  4.29it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   1%|          | 6/1001 [00:01<03:37,  4.57it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   1%|          | 7/1001 [00:01<03:26,  4.82it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   1%|          | 8/1001 [00:01<03:19,  4.97it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   1%|          | 9/1001 [00:01<03:15,  5.07it/s]\u001b[A\n","Iteration: 0 Loss: 89.88429260253906 Accuracy: 0.0010683761211112142:   1%|          | 10/1001 [00:02<03:12,  5.16it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   1%|          | 10/1001 [00:02<03:12,  5.16it/s] \u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   1%|          | 11/1001 [00:02<03:11,  5.17it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   1%|          | 12/1001 [00:02<03:10,  5.19it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   1%|▏         | 13/1001 [00:02<03:08,  5.24it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   1%|▏         | 14/1001 [00:02<03:08,  5.24it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   1%|▏         | 15/1001 [00:03<03:06,  5.27it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   2%|▏         | 16/1001 [00:03<03:05,  5.30it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   2%|▏         | 17/1001 [00:03<03:05,  5.31it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   2%|▏         | 18/1001 [00:03<03:05,  5.30it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   2%|▏         | 19/1001 [00:03<03:04,  5.32it/s]\u001b[A\n","Iteration: 10 Loss: 53.08656673431396 Accuracy: 0.07542454339563846:   2%|▏         | 20/1001 [00:03<03:04,  5.33it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   2%|▏         | 20/1001 [00:04<03:04,  5.33it/s] \u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   2%|▏         | 21/1001 [00:04<03:06,  5.24it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   2%|▏         | 22/1001 [00:04<03:06,  5.26it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   2%|▏         | 23/1001 [00:04<03:06,  5.25it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   2%|▏         | 24/1001 [00:04<03:05,  5.28it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   2%|▏         | 25/1001 [00:04<03:04,  5.29it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   3%|▎         | 26/1001 [00:05<03:04,  5.29it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   3%|▎         | 27/1001 [00:05<03:04,  5.27it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   3%|▎         | 28/1001 [00:05<03:04,  5.27it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   3%|▎         | 29/1001 [00:05<03:03,  5.29it/s]\u001b[A\n","Iteration: 20 Loss: 29.85770740509033 Accuracy: 0.1167919620871544:   3%|▎         | 30/1001 [00:05<03:04,  5.26it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   3%|▎         | 30/1001 [00:06<03:04,  5.26it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   3%|▎         | 31/1001 [00:06<03:05,  5.22it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   3%|▎         | 32/1001 [00:06<03:03,  5.27it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   3%|▎         | 33/1001 [00:06<03:04,  5.26it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   3%|▎         | 34/1001 [00:06<03:04,  5.25it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   3%|▎         | 35/1001 [00:06<03:02,  5.29it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   4%|▎         | 36/1001 [00:06<03:01,  5.30it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   4%|▎         | 37/1001 [00:07<03:01,  5.30it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   4%|▍         | 38/1001 [00:07<03:01,  5.30it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   4%|▍         | 39/1001 [00:07<03:02,  5.27it/s]\u001b[A\n","Iteration: 30 Loss: 20.416405773162843 Accuracy: 0.1229987595230341:   4%|▍         | 40/1001 [00:07<03:03,  5.24it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   4%|▍         | 40/1001 [00:07<03:03,  5.24it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   4%|▍         | 41/1001 [00:07<03:05,  5.17it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   4%|▍         | 42/1001 [00:08<03:03,  5.24it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   4%|▍         | 43/1001 [00:08<03:01,  5.27it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   4%|▍         | 44/1001 [00:08<03:02,  5.24it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   4%|▍         | 45/1001 [00:08<03:02,  5.25it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   5%|▍         | 46/1001 [00:08<03:01,  5.26it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   5%|▍         | 47/1001 [00:09<03:00,  5.28it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   5%|▍         | 48/1001 [00:09<03:00,  5.27it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   5%|▍         | 49/1001 [00:09<03:00,  5.28it/s]\u001b[A\n","Iteration: 40 Loss: 13.987319660186767 Accuracy: 0.12056959792971611:   5%|▍         | 50/1001 [00:09<02:59,  5.29it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   5%|▍         | 50/1001 [00:09<02:59,  5.29it/s] \u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   5%|▌         | 51/1001 [00:09<03:01,  5.24it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   5%|▌         | 52/1001 [00:10<03:01,  5.22it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   5%|▌         | 53/1001 [00:10<03:00,  5.24it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   5%|▌         | 54/1001 [00:10<03:05,  5.11it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   5%|▌         | 55/1001 [00:10<03:04,  5.14it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   6%|▌         | 56/1001 [00:10<03:02,  5.19it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   6%|▌         | 57/1001 [00:11<03:02,  5.19it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   6%|▌         | 58/1001 [00:11<03:00,  5.22it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   6%|▌         | 59/1001 [00:11<03:00,  5.23it/s]\u001b[A\n","Iteration: 50 Loss: 10.769127559661865 Accuracy: 0.1330823227763176:   6%|▌         | 60/1001 [00:11<03:00,  5.21it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   6%|▌         | 60/1001 [00:11<03:00,  5.21it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   6%|▌         | 61/1001 [00:11<03:03,  5.13it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   6%|▌         | 62/1001 [00:11<03:02,  5.16it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   6%|▋         | 63/1001 [00:12<03:01,  5.17it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   6%|▋         | 64/1001 [00:12<03:00,  5.19it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   6%|▋         | 65/1001 [00:12<03:00,  5.20it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   7%|▋         | 66/1001 [00:12<02:58,  5.23it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   7%|▋         | 67/1001 [00:12<03:00,  5.19it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   7%|▋         | 68/1001 [00:13<02:59,  5.20it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   7%|▋         | 69/1001 [00:13<02:58,  5.21it/s]\u001b[A\n","Iteration: 60 Loss: 9.631049585342407 Accuracy: 0.13810183480381966:   7%|▋         | 70/1001 [00:13<02:58,  5.23it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   7%|▋         | 70/1001 [00:13<02:58,  5.23it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   7%|▋         | 71/1001 [00:13<02:59,  5.19it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   7%|▋         | 72/1001 [00:13<02:59,  5.19it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   7%|▋         | 73/1001 [00:14<02:58,  5.21it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   7%|▋         | 74/1001 [00:14<02:57,  5.22it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   7%|▋         | 75/1001 [00:14<02:57,  5.21it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   8%|▊         | 76/1001 [00:14<02:56,  5.24it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   8%|▊         | 77/1001 [00:14<02:55,  5.25it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   8%|▊         | 78/1001 [00:15<02:56,  5.22it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   8%|▊         | 79/1001 [00:15<02:56,  5.21it/s]\u001b[A\n","Iteration: 70 Loss: 9.014417839050292 Accuracy: 0.14031582809984683:   8%|▊         | 80/1001 [00:15<02:56,  5.21it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   8%|▊         | 80/1001 [00:15<02:56,  5.21it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   8%|▊         | 81/1001 [00:15<02:58,  5.15it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   8%|▊         | 82/1001 [00:15<02:57,  5.18it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   8%|▊         | 83/1001 [00:16<02:57,  5.17it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   8%|▊         | 84/1001 [00:16<02:56,  5.19it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   8%|▊         | 85/1001 [00:16<02:56,  5.20it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   9%|▊         | 86/1001 [00:16<02:55,  5.20it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   9%|▊         | 87/1001 [00:16<02:55,  5.21it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   9%|▉         | 88/1001 [00:16<02:56,  5.18it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   9%|▉         | 89/1001 [00:17<02:56,  5.18it/s]\u001b[A\n","Iteration: 80 Loss: 7.733102607727051 Accuracy: 0.14383289143443106:   9%|▉         | 90/1001 [00:17<02:56,  5.15it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:   9%|▉         | 90/1001 [00:17<02:56,  5.15it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:   9%|▉         | 91/1001 [00:17<02:57,  5.12it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:   9%|▉         | 92/1001 [00:17<02:57,  5.13it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:   9%|▉         | 93/1001 [00:17<02:55,  5.17it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:   9%|▉         | 94/1001 [00:18<02:54,  5.20it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:   9%|▉         | 95/1001 [00:18<02:53,  5.23it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:  10%|▉         | 96/1001 [00:18<02:52,  5.24it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:  10%|▉         | 97/1001 [00:18<02:53,  5.22it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:  10%|▉         | 98/1001 [00:18<02:52,  5.23it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:  10%|▉         | 99/1001 [00:19<02:52,  5.22it/s]\u001b[A\n","Iteration: 90 Loss: 7.104217290878296 Accuracy: 0.15025152564048766:  10%|▉         | 100/1001 [00:19<02:51,  5.25it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  10%|▉         | 100/1001 [00:19<02:51,  5.25it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  10%|█         | 101/1001 [00:19<03:24,  4.39it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  10%|█         | 102/1001 [00:19<03:16,  4.57it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  10%|█         | 103/1001 [00:19<03:09,  4.73it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  10%|█         | 104/1001 [00:20<03:05,  4.84it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  10%|█         | 105/1001 [00:20<03:01,  4.93it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  11%|█         | 106/1001 [00:20<02:59,  4.99it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  11%|█         | 107/1001 [00:20<02:56,  5.05it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  11%|█         | 108/1001 [00:20<02:56,  5.06it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  11%|█         | 109/1001 [00:21<02:54,  5.10it/s]\u001b[A\n","Iteration: 100 Loss: 6.169395637512207 Accuracy: 0.16071430519223212:  11%|█         | 110/1001 [00:21<02:54,  5.09it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  11%|█         | 110/1001 [00:21<02:54,  5.09it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  11%|█         | 111/1001 [00:21<02:54,  5.09it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  11%|█         | 112/1001 [00:21<02:52,  5.16it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  11%|█▏        | 113/1001 [00:21<02:50,  5.20it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  11%|█▏        | 114/1001 [00:22<02:50,  5.21it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  11%|█▏        | 115/1001 [00:22<02:49,  5.24it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  12%|█▏        | 116/1001 [00:22<02:48,  5.25it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  12%|█▏        | 117/1001 [00:22<02:48,  5.25it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  12%|█▏        | 118/1001 [00:22<02:47,  5.27it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  12%|█▏        | 119/1001 [00:23<02:47,  5.27it/s]\u001b[A\n","Iteration: 110 Loss: 5.649755144119263 Accuracy: 0.16927123665809632:  12%|█▏        | 120/1001 [00:23<02:48,  5.23it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  12%|█▏        | 120/1001 [00:23<02:48,  5.23it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  12%|█▏        | 121/1001 [00:23<02:50,  5.17it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  12%|█▏        | 122/1001 [00:23<02:49,  5.20it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  12%|█▏        | 123/1001 [00:23<02:48,  5.21it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  12%|█▏        | 124/1001 [00:24<02:48,  5.21it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  12%|█▏        | 125/1001 [00:24<02:46,  5.25it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  13%|█▎        | 126/1001 [00:24<02:45,  5.28it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  13%|█▎        | 127/1001 [00:24<02:44,  5.30it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  13%|█▎        | 128/1001 [00:24<02:44,  5.30it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  13%|█▎        | 129/1001 [00:24<02:44,  5.30it/s]\u001b[A\n","Iteration: 120 Loss: 5.383494234085083 Accuracy: 0.16706039980053902:  13%|█▎        | 130/1001 [00:25<02:45,  5.25it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  13%|█▎        | 130/1001 [00:25<02:45,  5.25it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  13%|█▎        | 131/1001 [00:25<02:47,  5.18it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  13%|█▎        | 132/1001 [00:25<02:46,  5.21it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  13%|█▎        | 133/1001 [00:25<02:45,  5.26it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  13%|█▎        | 134/1001 [00:25<02:44,  5.26it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  13%|█▎        | 135/1001 [00:26<02:44,  5.27it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  14%|█▎        | 136/1001 [00:26<02:44,  5.27it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  14%|█▎        | 137/1001 [00:26<02:42,  5.30it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  14%|█▍        | 138/1001 [00:26<02:44,  5.26it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  14%|█▍        | 139/1001 [00:26<02:43,  5.26it/s]\u001b[A\n","Iteration: 130 Loss: 5.219062614440918 Accuracy: 0.17951793670654298:  14%|█▍        | 140/1001 [00:27<02:43,  5.27it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  14%|█▍        | 140/1001 [00:27<02:43,  5.27it/s] \u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  14%|█▍        | 141/1001 [00:27<02:45,  5.21it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  14%|█▍        | 142/1001 [00:27<02:43,  5.25it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  14%|█▍        | 143/1001 [00:27<02:43,  5.25it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  14%|█▍        | 144/1001 [00:27<02:42,  5.29it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  14%|█▍        | 145/1001 [00:27<02:41,  5.30it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  15%|█▍        | 146/1001 [00:28<02:40,  5.34it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  15%|█▍        | 147/1001 [00:28<02:40,  5.32it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  15%|█▍        | 148/1001 [00:28<02:41,  5.28it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  15%|█▍        | 149/1001 [00:28<02:40,  5.30it/s]\u001b[A\n","Iteration: 140 Loss: 4.853471469879151 Accuracy: 0.1836117058992386:  15%|█▍        | 150/1001 [00:28<02:40,  5.31it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  15%|█▍        | 150/1001 [00:29<02:40,  5.31it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  15%|█▌        | 151/1001 [00:29<02:42,  5.24it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  15%|█▌        | 152/1001 [00:29<02:40,  5.28it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  15%|█▌        | 153/1001 [00:29<02:45,  5.14it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  15%|█▌        | 154/1001 [00:29<02:46,  5.10it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  15%|█▌        | 155/1001 [00:29<02:43,  5.17it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  16%|█▌        | 156/1001 [00:30<02:45,  5.11it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  16%|█▌        | 157/1001 [00:30<02:44,  5.13it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  16%|█▌        | 158/1001 [00:30<02:43,  5.17it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  16%|█▌        | 159/1001 [00:30<02:41,  5.21it/s]\u001b[A\n","Iteration: 150 Loss: 5.1293727397918705 Accuracy: 0.1979799225926399:  16%|█▌        | 160/1001 [00:30<02:39,  5.26it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  16%|█▌        | 160/1001 [00:31<02:39,  5.26it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  16%|█▌        | 161/1001 [00:31<02:41,  5.22it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  16%|█▌        | 162/1001 [00:31<02:39,  5.25it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  16%|█▋        | 163/1001 [00:31<02:39,  5.25it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  16%|█▋        | 164/1001 [00:31<02:38,  5.29it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  16%|█▋        | 165/1001 [00:31<02:36,  5.33it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  17%|█▋        | 166/1001 [00:32<02:37,  5.31it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  17%|█▋        | 167/1001 [00:32<02:36,  5.31it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  17%|█▋        | 168/1001 [00:32<02:36,  5.33it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  17%|█▋        | 169/1001 [00:32<02:35,  5.35it/s]\u001b[A\n","Iteration: 160 Loss: 4.850295543670654 Accuracy: 0.19007109403610228:  17%|█▋        | 170/1001 [00:32<02:35,  5.35it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  17%|█▋        | 170/1001 [00:32<02:35,  5.35it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  17%|█▋        | 171/1001 [00:32<02:36,  5.31it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  17%|█▋        | 172/1001 [00:33<02:35,  5.33it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  17%|█▋        | 173/1001 [00:33<02:37,  5.26it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  17%|█▋        | 174/1001 [00:33<02:37,  5.26it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  17%|█▋        | 175/1001 [00:33<02:35,  5.31it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  18%|█▊        | 176/1001 [00:33<02:34,  5.35it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  18%|█▊        | 177/1001 [00:34<02:35,  5.31it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  18%|█▊        | 178/1001 [00:34<02:35,  5.30it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  18%|█▊        | 179/1001 [00:34<02:35,  5.28it/s]\u001b[A\n","Iteration: 170 Loss: 4.897156071662903 Accuracy: 0.19046529084444047:  18%|█▊        | 180/1001 [00:34<02:33,  5.33it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  18%|█▊        | 180/1001 [00:34<02:33,  5.33it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  18%|█▊        | 181/1001 [00:34<02:34,  5.31it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  18%|█▊        | 182/1001 [00:35<02:33,  5.33it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  18%|█▊        | 183/1001 [00:35<02:32,  5.35it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  18%|█▊        | 184/1001 [00:35<02:32,  5.37it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  18%|█▊        | 185/1001 [00:35<02:31,  5.39it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  19%|█▊        | 186/1001 [00:35<02:31,  5.39it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  19%|█▊        | 187/1001 [00:35<02:30,  5.39it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  19%|█▉        | 188/1001 [00:36<02:30,  5.41it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  19%|█▉        | 189/1001 [00:36<02:31,  5.36it/s]\u001b[A\n","Iteration: 180 Loss: 4.727618861198425 Accuracy: 0.20145384967327118:  19%|█▉        | 190/1001 [00:36<02:31,  5.35it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  19%|█▉        | 190/1001 [00:36<02:31,  5.35it/s] \u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  19%|█▉        | 191/1001 [00:36<02:33,  5.28it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  19%|█▉        | 192/1001 [00:36<02:32,  5.32it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  19%|█▉        | 193/1001 [00:37<02:32,  5.30it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  19%|█▉        | 194/1001 [00:37<02:32,  5.28it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  19%|█▉        | 195/1001 [00:37<02:32,  5.30it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  20%|█▉        | 196/1001 [00:37<02:31,  5.31it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  20%|█▉        | 197/1001 [00:37<02:31,  5.29it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  20%|█▉        | 198/1001 [00:38<02:30,  5.34it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  20%|█▉        | 199/1001 [00:38<02:31,  5.28it/s]\u001b[A\n","Iteration: 190 Loss: 4.4496019124984745 Accuracy: 0.206494502723217:  20%|█▉        | 200/1001 [00:38<02:31,  5.29it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  20%|█▉        | 200/1001 [00:38<02:31,  5.29it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  20%|██        | 201/1001 [00:38<03:01,  4.41it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  20%|██        | 202/1001 [00:38<02:52,  4.63it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  20%|██        | 203/1001 [00:39<02:44,  4.85it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  20%|██        | 204/1001 [00:39<02:39,  5.00it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  20%|██        | 205/1001 [00:39<02:35,  5.12it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  21%|██        | 206/1001 [00:39<02:34,  5.15it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  21%|██        | 207/1001 [00:39<02:31,  5.23it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  21%|██        | 208/1001 [00:40<02:31,  5.24it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  21%|██        | 209/1001 [00:40<02:31,  5.23it/s]\u001b[A\n","Iteration: 200 Loss: 4.67873067855835 Accuracy: 0.21171337813138963:  21%|██        | 210/1001 [00:40<02:30,  5.25it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  21%|██        | 210/1001 [00:40<02:30,  5.25it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  21%|██        | 211/1001 [00:40<02:32,  5.19it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  21%|██        | 212/1001 [00:40<02:32,  5.19it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  21%|██▏       | 213/1001 [00:40<02:30,  5.24it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  21%|██▏       | 214/1001 [00:41<02:29,  5.27it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  21%|██▏       | 215/1001 [00:41<02:28,  5.31it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  22%|██▏       | 216/1001 [00:41<02:28,  5.30it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  22%|██▏       | 217/1001 [00:41<02:26,  5.34it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  22%|██▏       | 218/1001 [00:41<02:26,  5.34it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  22%|██▏       | 219/1001 [00:42<02:25,  5.37it/s]\u001b[A\n","Iteration: 210 Loss: 4.212442231178284 Accuracy: 0.2192375987768173:  22%|██▏       | 220/1001 [00:42<02:26,  5.33it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  22%|██▏       | 220/1001 [00:42<02:26,  5.33it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  22%|██▏       | 221/1001 [00:42<02:27,  5.28it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  22%|██▏       | 222/1001 [00:42<02:26,  5.32it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  22%|██▏       | 223/1001 [00:42<02:25,  5.33it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  22%|██▏       | 224/1001 [00:43<02:26,  5.32it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  22%|██▏       | 225/1001 [00:43<02:26,  5.29it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  23%|██▎       | 226/1001 [00:43<02:25,  5.32it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  23%|██▎       | 227/1001 [00:43<02:25,  5.33it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  23%|██▎       | 228/1001 [00:43<02:27,  5.25it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  23%|██▎       | 229/1001 [00:43<02:25,  5.29it/s]\u001b[A\n","Iteration: 220 Loss: 4.620905566215515 Accuracy: 0.2074577882885933:  23%|██▎       | 230/1001 [00:44<02:25,  5.30it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  23%|██▎       | 230/1001 [00:44<02:25,  5.30it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  23%|██▎       | 231/1001 [00:44<02:29,  5.17it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  23%|██▎       | 232/1001 [00:44<02:27,  5.22it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  23%|██▎       | 233/1001 [00:44<02:26,  5.24it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  23%|██▎       | 234/1001 [00:44<02:25,  5.27it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  23%|██▎       | 235/1001 [00:45<02:24,  5.31it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  24%|██▎       | 236/1001 [00:45<02:24,  5.31it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  24%|██▎       | 237/1001 [00:45<02:24,  5.31it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  24%|██▍       | 238/1001 [00:45<02:23,  5.32it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  24%|██▍       | 239/1001 [00:45<02:22,  5.35it/s]\u001b[A\n","Iteration: 230 Loss: 4.374816298484802 Accuracy: 0.21919821053743363:  24%|██▍       | 240/1001 [00:46<02:22,  5.35it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  24%|██▍       | 240/1001 [00:46<02:22,  5.35it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  24%|██▍       | 241/1001 [00:46<02:23,  5.31it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  24%|██▍       | 242/1001 [00:46<02:21,  5.38it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  24%|██▍       | 243/1001 [00:46<02:19,  5.42it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  24%|██▍       | 244/1001 [00:46<02:19,  5.42it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  24%|██▍       | 245/1001 [00:46<02:19,  5.40it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  25%|██▍       | 246/1001 [00:47<02:19,  5.39it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  25%|██▍       | 247/1001 [00:47<02:20,  5.38it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  25%|██▍       | 248/1001 [00:47<02:20,  5.38it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  25%|██▍       | 249/1001 [00:47<02:19,  5.40it/s]\u001b[A\n","Iteration: 240 Loss: 4.258655834197998 Accuracy: 0.21556324362754822:  25%|██▍       | 250/1001 [00:47<02:19,  5.38it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  25%|██▍       | 250/1001 [00:48<02:19,  5.38it/s]  \u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  25%|██▌       | 251/1001 [00:48<02:20,  5.33it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  25%|██▌       | 252/1001 [00:48<02:21,  5.30it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  25%|██▌       | 253/1001 [00:48<02:19,  5.36it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  25%|██▌       | 254/1001 [00:48<02:21,  5.29it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  25%|██▌       | 255/1001 [00:48<02:22,  5.23it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  26%|██▌       | 256/1001 [00:49<02:20,  5.30it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  26%|██▌       | 257/1001 [00:49<02:21,  5.26it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  26%|██▌       | 258/1001 [00:49<02:21,  5.26it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  26%|██▌       | 259/1001 [00:49<02:19,  5.31it/s]\u001b[A\n","Iteration: 250 Loss: 4.473654818534851 Accuracy: 0.226158307492733:  26%|██▌       | 260/1001 [00:49<02:19,  5.30it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  26%|██▌       | 260/1001 [00:49<02:19,  5.30it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  26%|██▌       | 261/1001 [00:49<02:18,  5.33it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  26%|██▌       | 262/1001 [00:50<02:17,  5.38it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  26%|██▋       | 263/1001 [00:50<02:18,  5.34it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  26%|██▋       | 264/1001 [00:50<02:17,  5.35it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  26%|██▋       | 265/1001 [00:50<02:16,  5.40it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  27%|██▋       | 266/1001 [00:50<02:15,  5.43it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  27%|██▋       | 267/1001 [00:51<02:14,  5.45it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  27%|██▋       | 268/1001 [00:51<02:15,  5.41it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  27%|██▋       | 269/1001 [00:51<02:15,  5.42it/s]\u001b[A\n","Iteration: 260 Loss: 4.302098321914673 Accuracy: 0.21817392706871033:  27%|██▋       | 270/1001 [00:51<02:14,  5.42it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  27%|██▋       | 270/1001 [00:51<02:14,  5.42it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  27%|██▋       | 271/1001 [00:51<02:16,  5.34it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  27%|██▋       | 272/1001 [00:52<02:15,  5.38it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  27%|██▋       | 273/1001 [00:52<02:15,  5.38it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  27%|██▋       | 274/1001 [00:52<02:14,  5.40it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  27%|██▋       | 275/1001 [00:52<02:14,  5.41it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  28%|██▊       | 276/1001 [00:52<02:13,  5.44it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  28%|██▊       | 277/1001 [00:52<02:14,  5.40it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  28%|██▊       | 278/1001 [00:53<02:12,  5.45it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  28%|██▊       | 279/1001 [00:53<02:12,  5.45it/s]\u001b[A\n","Iteration: 270 Loss: 4.220793986320496 Accuracy: 0.22940492928028106:  28%|██▊       | 280/1001 [00:53<02:11,  5.48it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  28%|██▊       | 280/1001 [00:53<02:11,  5.48it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  28%|██▊       | 281/1001 [00:53<02:12,  5.41it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  28%|██▊       | 282/1001 [00:53<02:11,  5.45it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  28%|██▊       | 283/1001 [00:54<02:11,  5.47it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  28%|██▊       | 284/1001 [00:54<02:10,  5.47it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  28%|██▊       | 285/1001 [00:54<02:10,  5.48it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  29%|██▊       | 286/1001 [00:54<02:11,  5.42it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  29%|██▊       | 287/1001 [00:54<02:11,  5.42it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  29%|██▉       | 288/1001 [00:54<02:12,  5.37it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  29%|██▉       | 289/1001 [00:55<02:11,  5.41it/s]\u001b[A\n","Iteration: 280 Loss: 4.358880090713501 Accuracy: 0.22009641379117967:  29%|██▉       | 290/1001 [00:55<02:12,  5.35it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  29%|██▉       | 290/1001 [00:55<02:12,  5.35it/s] \u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  29%|██▉       | 291/1001 [00:55<02:13,  5.31it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  29%|██▉       | 292/1001 [00:55<02:12,  5.35it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  29%|██▉       | 293/1001 [00:55<02:12,  5.35it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  29%|██▉       | 294/1001 [00:56<02:11,  5.37it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  29%|██▉       | 295/1001 [00:56<02:12,  5.35it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  30%|██▉       | 296/1001 [00:56<02:10,  5.40it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  30%|██▉       | 297/1001 [00:56<02:09,  5.44it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  30%|██▉       | 298/1001 [00:56<02:08,  5.46it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  30%|██▉       | 299/1001 [00:57<02:08,  5.47it/s]\u001b[A\n","Iteration: 290 Loss: 4.093936777114868 Accuracy: 0.2098969727754593:  30%|██▉       | 300/1001 [00:57<02:08,  5.45it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  30%|██▉       | 300/1001 [00:57<02:08,  5.45it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  30%|███       | 301/1001 [00:57<02:39,  4.40it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  30%|███       | 302/1001 [00:57<02:30,  4.65it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  30%|███       | 303/1001 [00:57<02:22,  4.89it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  30%|███       | 304/1001 [00:58<02:17,  5.06it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  30%|███       | 305/1001 [00:58<02:14,  5.19it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  31%|███       | 306/1001 [00:58<02:14,  5.16it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  31%|███       | 307/1001 [00:58<02:13,  5.21it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  31%|███       | 308/1001 [00:58<02:10,  5.30it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  31%|███       | 309/1001 [00:59<02:10,  5.31it/s]\u001b[A\n","Iteration: 300 Loss: 4.221121549606323 Accuracy: 0.21136982142925262:  31%|███       | 310/1001 [00:59<02:10,  5.29it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  31%|███       | 310/1001 [00:59<02:10,  5.29it/s] \u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  31%|███       | 311/1001 [00:59<02:10,  5.28it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  31%|███       | 312/1001 [00:59<02:08,  5.35it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  31%|███▏      | 313/1001 [00:59<02:08,  5.37it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  31%|███▏      | 314/1001 [00:59<02:06,  5.42it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  31%|███▏      | 315/1001 [01:00<02:06,  5.40it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  32%|███▏      | 316/1001 [01:00<02:08,  5.35it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  32%|███▏      | 317/1001 [01:00<02:06,  5.40it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  32%|███▏      | 318/1001 [01:00<02:05,  5.44it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  32%|███▏      | 319/1001 [01:00<02:04,  5.47it/s]\u001b[A\n","Iteration: 310 Loss: 4.494412922859192 Accuracy: 0.2274657592177391:  32%|███▏      | 320/1001 [01:01<02:04,  5.48it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  32%|███▏      | 320/1001 [01:01<02:04,  5.48it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  32%|███▏      | 321/1001 [01:01<02:05,  5.42it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  32%|███▏      | 322/1001 [01:01<02:05,  5.43it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  32%|███▏      | 323/1001 [01:01<02:04,  5.45it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  32%|███▏      | 324/1001 [01:01<02:04,  5.42it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  32%|███▏      | 325/1001 [01:01<02:05,  5.40it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  33%|███▎      | 326/1001 [01:02<02:06,  5.34it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  33%|███▎      | 327/1001 [01:02<02:06,  5.34it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  33%|███▎      | 328/1001 [01:02<02:06,  5.33it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  33%|███▎      | 329/1001 [01:02<02:04,  5.38it/s]\u001b[A\n","Iteration: 320 Loss: 4.512817215919495 Accuracy: 0.2166271135210991:  33%|███▎      | 330/1001 [01:02<02:04,  5.38it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  33%|███▎      | 330/1001 [01:03<02:04,  5.38it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  33%|███▎      | 331/1001 [01:03<02:04,  5.38it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  33%|███▎      | 332/1001 [01:03<02:03,  5.43it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  33%|███▎      | 333/1001 [01:03<02:02,  5.45it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  33%|███▎      | 334/1001 [01:03<02:04,  5.37it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  33%|███▎      | 335/1001 [01:03<02:03,  5.39it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  34%|███▎      | 336/1001 [01:04<02:03,  5.37it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  34%|███▎      | 337/1001 [01:04<02:03,  5.38it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  34%|███▍      | 338/1001 [01:04<02:03,  5.36it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  34%|███▍      | 339/1001 [01:04<02:04,  5.33it/s]\u001b[A\n","Iteration: 330 Loss: 4.650902199745178 Accuracy: 0.2257538393139839:  34%|███▍      | 340/1001 [01:04<02:03,  5.36it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  34%|███▍      | 340/1001 [01:04<02:03,  5.36it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  34%|███▍      | 341/1001 [01:04<02:05,  5.25it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  34%|███▍      | 342/1001 [01:05<02:03,  5.32it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  34%|███▍      | 343/1001 [01:05<02:02,  5.38it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  34%|███▍      | 344/1001 [01:05<02:01,  5.41it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  34%|███▍      | 345/1001 [01:05<02:00,  5.44it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  35%|███▍      | 346/1001 [01:05<02:01,  5.39it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  35%|███▍      | 347/1001 [01:06<02:02,  5.35it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  35%|███▍      | 348/1001 [01:06<02:03,  5.28it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  35%|███▍      | 349/1001 [01:06<02:03,  5.28it/s]\u001b[A\n","Iteration: 340 Loss: 4.197521281242371 Accuracy: 0.23051896095275878:  35%|███▍      | 350/1001 [01:06<02:02,  5.31it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  35%|███▍      | 350/1001 [01:06<02:02,  5.31it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  35%|███▌      | 351/1001 [01:06<02:01,  5.33it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  35%|███▌      | 352/1001 [01:07<02:00,  5.40it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  35%|███▌      | 353/1001 [01:07<02:00,  5.37it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  35%|███▌      | 354/1001 [01:07<02:00,  5.36it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  35%|███▌      | 355/1001 [01:07<02:01,  5.31it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  36%|███▌      | 356/1001 [01:07<02:00,  5.35it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  36%|███▌      | 357/1001 [01:07<01:59,  5.41it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  36%|███▌      | 358/1001 [01:08<01:58,  5.44it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  36%|███▌      | 359/1001 [01:08<01:57,  5.44it/s]\u001b[A\n","Iteration: 350 Loss: 3.853748106956482 Accuracy: 0.23873812854290008:  36%|███▌      | 360/1001 [01:08<01:57,  5.45it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  36%|███▌      | 360/1001 [01:08<01:57,  5.45it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  36%|███▌      | 361/1001 [01:08<01:58,  5.38it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  36%|███▌      | 362/1001 [01:08<01:59,  5.35it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  36%|███▋      | 363/1001 [01:09<01:59,  5.33it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  36%|███▋      | 364/1001 [01:09<01:58,  5.38it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  36%|███▋      | 365/1001 [01:09<01:57,  5.42it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  37%|███▋      | 366/1001 [01:09<01:57,  5.43it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  37%|███▋      | 367/1001 [01:09<01:58,  5.35it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  37%|███▋      | 368/1001 [01:09<01:58,  5.33it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  37%|███▋      | 369/1001 [01:10<01:58,  5.33it/s]\u001b[A\n","Iteration: 360 Loss: 4.022741031646729 Accuracy: 0.22725036293268203:  37%|███▋      | 370/1001 [01:10<01:57,  5.37it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  37%|███▋      | 370/1001 [01:10<01:57,  5.37it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  37%|███▋      | 371/1001 [01:10<01:57,  5.35it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  37%|███▋      | 372/1001 [01:10<01:57,  5.35it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  37%|███▋      | 373/1001 [01:10<01:57,  5.37it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  37%|███▋      | 374/1001 [01:11<01:56,  5.36it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  37%|███▋      | 375/1001 [01:11<01:55,  5.41it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  38%|███▊      | 376/1001 [01:11<01:56,  5.35it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  38%|███▊      | 377/1001 [01:11<01:55,  5.38it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  38%|███▊      | 378/1001 [01:11<01:56,  5.36it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  38%|███▊      | 379/1001 [01:12<01:55,  5.39it/s]\u001b[A\n","Iteration: 370 Loss: 4.237647771835327 Accuracy: 0.22706314027309418:  38%|███▊      | 380/1001 [01:12<01:54,  5.41it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  38%|███▊      | 380/1001 [01:12<01:54,  5.41it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  38%|███▊      | 381/1001 [01:12<01:55,  5.39it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  38%|███▊      | 382/1001 [01:12<01:53,  5.44it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  38%|███▊      | 383/1001 [01:12<01:55,  5.34it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  38%|███▊      | 384/1001 [01:12<01:54,  5.38it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  38%|███▊      | 385/1001 [01:13<01:55,  5.33it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  39%|███▊      | 386/1001 [01:13<01:54,  5.38it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  39%|███▊      | 387/1001 [01:13<01:54,  5.36it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  39%|███▉      | 388/1001 [01:13<01:54,  5.34it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  39%|███▉      | 389/1001 [01:13<01:53,  5.40it/s]\u001b[A\n","Iteration: 380 Loss: 3.9053407669067384 Accuracy: 0.22894731909036636:  39%|███▉      | 390/1001 [01:14<01:53,  5.41it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  39%|███▉      | 390/1001 [01:14<01:53,  5.41it/s] \u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  39%|███▉      | 391/1001 [01:14<01:53,  5.39it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  39%|███▉      | 392/1001 [01:14<01:52,  5.41it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  39%|███▉      | 393/1001 [01:14<01:52,  5.43it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  39%|███▉      | 394/1001 [01:14<01:53,  5.36it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  39%|███▉      | 395/1001 [01:15<01:52,  5.38it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  40%|███▉      | 396/1001 [01:15<01:52,  5.38it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  40%|███▉      | 397/1001 [01:15<01:52,  5.38it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  40%|███▉      | 398/1001 [01:15<01:52,  5.37it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  40%|███▉      | 399/1001 [01:15<01:52,  5.36it/s]\u001b[A\n","Iteration: 390 Loss: 4.012944626808166 Accuracy: 0.22929058969020844:  40%|███▉      | 400/1001 [01:15<01:52,  5.35it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  40%|███▉      | 400/1001 [01:16<01:52,  5.35it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  40%|████      | 401/1001 [01:16<02:14,  4.45it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  40%|████      | 402/1001 [01:16<02:07,  4.69it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  40%|████      | 403/1001 [01:16<02:01,  4.90it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  40%|████      | 404/1001 [01:16<01:58,  5.04it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  40%|████      | 405/1001 [01:16<01:56,  5.13it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  41%|████      | 406/1001 [01:17<01:53,  5.23it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  41%|████      | 407/1001 [01:17<01:51,  5.31it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  41%|████      | 408/1001 [01:17<01:50,  5.35it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  41%|████      | 409/1001 [01:17<01:51,  5.29it/s]\u001b[A\n","Iteration: 400 Loss: 4.2969221591949465 Accuracy: 0.23891163468360901:  41%|████      | 410/1001 [01:17<01:51,  5.28it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  41%|████      | 410/1001 [01:18<01:51,  5.28it/s]  \u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  41%|████      | 411/1001 [01:18<01:53,  5.21it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  41%|████      | 412/1001 [01:18<01:52,  5.26it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  41%|████▏     | 413/1001 [01:18<01:51,  5.28it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  41%|████▏     | 414/1001 [01:18<01:50,  5.33it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  41%|████▏     | 415/1001 [01:18<01:52,  5.21it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  42%|████▏     | 416/1001 [01:19<01:50,  5.29it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  42%|████▏     | 417/1001 [01:19<01:49,  5.34it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  42%|████▏     | 418/1001 [01:19<01:48,  5.39it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  42%|████▏     | 419/1001 [01:19<01:47,  5.42it/s]\u001b[A\n","Iteration: 410 Loss: 4.279483485221863 Accuracy: 0.2371191531419754:  42%|████▏     | 420/1001 [01:19<01:46,  5.45it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  42%|████▏     | 420/1001 [01:19<01:46,  5.45it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  42%|████▏     | 421/1001 [01:19<01:46,  5.42it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  42%|████▏     | 422/1001 [01:20<01:46,  5.43it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  42%|████▏     | 423/1001 [01:20<01:46,  5.45it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  42%|████▏     | 424/1001 [01:20<01:46,  5.43it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  42%|████▏     | 425/1001 [01:20<01:47,  5.34it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  43%|████▎     | 426/1001 [01:20<01:47,  5.36it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  43%|████▎     | 427/1001 [01:21<01:46,  5.41it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  43%|████▎     | 428/1001 [01:21<01:45,  5.44it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  43%|████▎     | 429/1001 [01:21<01:45,  5.40it/s]\u001b[A\n","Iteration: 420 Loss: 3.7775474786758423 Accuracy: 0.2297242298722267:  43%|████▎     | 430/1001 [01:21<01:45,  5.41it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  43%|████▎     | 430/1001 [01:21<01:45,  5.41it/s]  \u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  43%|████▎     | 431/1001 [01:21<01:47,  5.32it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  43%|████▎     | 432/1001 [01:22<01:45,  5.38it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  43%|████▎     | 433/1001 [01:22<01:45,  5.39it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  43%|████▎     | 434/1001 [01:22<01:44,  5.42it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  43%|████▎     | 435/1001 [01:22<01:44,  5.42it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  44%|████▎     | 436/1001 [01:22<01:44,  5.40it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  44%|████▎     | 437/1001 [01:22<01:44,  5.41it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  44%|████▍     | 438/1001 [01:23<01:44,  5.41it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  44%|████▍     | 439/1001 [01:23<01:45,  5.34it/s]\u001b[A\n","Iteration: 430 Loss: 4.38478033542633 Accuracy: 0.2396463081240654:  44%|████▍     | 440/1001 [01:23<01:44,  5.35it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  44%|████▍     | 440/1001 [01:23<01:44,  5.35it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  44%|████▍     | 441/1001 [01:23<01:44,  5.35it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  44%|████▍     | 442/1001 [01:23<01:43,  5.40it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  44%|████▍     | 443/1001 [01:24<01:42,  5.43it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  44%|████▍     | 444/1001 [01:24<01:42,  5.45it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  44%|████▍     | 445/1001 [01:24<01:44,  5.31it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  45%|████▍     | 446/1001 [01:24<01:44,  5.32it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  45%|████▍     | 447/1001 [01:24<01:44,  5.28it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  45%|████▍     | 448/1001 [01:24<01:43,  5.34it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  45%|████▍     | 449/1001 [01:25<01:44,  5.30it/s]\u001b[A\n","Iteration: 440 Loss: 4.2835200548172 Accuracy: 0.22122779935598375:  45%|████▍     | 450/1001 [01:25<01:44,  5.25it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  45%|████▍     | 450/1001 [01:25<01:44,  5.25it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  45%|████▌     | 451/1001 [01:25<01:46,  5.16it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  45%|████▌     | 452/1001 [01:25<01:46,  5.14it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  45%|████▌     | 453/1001 [01:25<01:45,  5.21it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  45%|████▌     | 454/1001 [01:26<01:46,  5.15it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  45%|████▌     | 455/1001 [01:26<01:44,  5.20it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  46%|████▌     | 456/1001 [01:26<01:43,  5.26it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  46%|████▌     | 457/1001 [01:26<01:42,  5.31it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  46%|████▌     | 458/1001 [01:26<01:42,  5.32it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  46%|████▌     | 459/1001 [01:27<01:41,  5.34it/s]\u001b[A\n","Iteration: 450 Loss: 4.166892886161804 Accuracy: 0.23575427681207656:  46%|████▌     | 460/1001 [01:27<01:40,  5.37it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  46%|████▌     | 460/1001 [01:27<01:40,  5.37it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  46%|████▌     | 461/1001 [01:27<01:40,  5.37it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  46%|████▌     | 462/1001 [01:27<01:39,  5.41it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  46%|████▋     | 463/1001 [01:27<01:39,  5.41it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  46%|████▋     | 464/1001 [01:28<01:38,  5.42it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  46%|████▋     | 465/1001 [01:28<01:38,  5.43it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  47%|████▋     | 466/1001 [01:28<01:38,  5.42it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  47%|████▋     | 467/1001 [01:28<01:38,  5.41it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  47%|████▋     | 468/1001 [01:28<01:39,  5.34it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  47%|████▋     | 469/1001 [01:28<01:40,  5.28it/s]\u001b[A\n","Iteration: 460 Loss: 4.066762399673462 Accuracy: 0.23171625435352325:  47%|████▋     | 470/1001 [01:29<01:39,  5.34it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  47%|████▋     | 470/1001 [01:29<01:39,  5.34it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  47%|████▋     | 471/1001 [01:29<01:39,  5.32it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  47%|████▋     | 472/1001 [01:29<01:38,  5.37it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  47%|████▋     | 473/1001 [01:29<01:37,  5.41it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  47%|████▋     | 474/1001 [01:29<01:37,  5.43it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  47%|████▋     | 475/1001 [01:30<01:38,  5.33it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  48%|████▊     | 476/1001 [01:30<01:38,  5.30it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  48%|████▊     | 477/1001 [01:30<01:37,  5.35it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  48%|████▊     | 478/1001 [01:30<01:36,  5.40it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  48%|████▊     | 479/1001 [01:30<01:36,  5.40it/s]\u001b[A\n","Iteration: 470 Loss: 4.2471434593200685 Accuracy: 0.2341938301920891:  48%|████▊     | 480/1001 [01:30<01:36,  5.42it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  48%|████▊     | 480/1001 [01:31<01:36,  5.42it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  48%|████▊     | 481/1001 [01:31<01:37,  5.36it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  48%|████▊     | 482/1001 [01:31<01:36,  5.35it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  48%|████▊     | 483/1001 [01:31<01:35,  5.40it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  48%|████▊     | 484/1001 [01:31<01:36,  5.35it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  48%|████▊     | 485/1001 [01:31<01:36,  5.33it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  49%|████▊     | 486/1001 [01:32<01:36,  5.31it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  49%|████▊     | 487/1001 [01:32<01:35,  5.36it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  49%|████▉     | 488/1001 [01:32<01:35,  5.37it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  49%|████▉     | 489/1001 [01:32<01:35,  5.39it/s]\u001b[A\n","Iteration: 480 Loss: 4.273408222198486 Accuracy: 0.24517337828874589:  49%|████▉     | 490/1001 [01:32<01:36,  5.31it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  49%|████▉     | 490/1001 [01:33<01:36,  5.31it/s] \u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  49%|████▉     | 491/1001 [01:33<01:36,  5.27it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  49%|████▉     | 492/1001 [01:33<01:35,  5.33it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  49%|████▉     | 493/1001 [01:33<01:34,  5.35it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  49%|████▉     | 494/1001 [01:33<01:34,  5.35it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  49%|████▉     | 495/1001 [01:33<01:34,  5.37it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  50%|████▉     | 496/1001 [01:33<01:33,  5.38it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  50%|████▉     | 497/1001 [01:34<01:34,  5.33it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  50%|████▉     | 498/1001 [01:34<01:34,  5.34it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  50%|████▉     | 499/1001 [01:34<01:33,  5.35it/s]\u001b[A\n","Iteration: 490 Loss: 4.066688418388367 Accuracy: 0.2490998923778534:  50%|████▉     | 500/1001 [01:34<01:33,  5.34it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  50%|████▉     | 500/1001 [01:34<01:33,  5.34it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  50%|█████     | 501/1001 [01:35<01:53,  4.40it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  50%|█████     | 502/1001 [01:35<01:47,  4.65it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  50%|█████     | 503/1001 [01:35<01:43,  4.83it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  50%|█████     | 504/1001 [01:35<01:39,  4.99it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  50%|█████     | 505/1001 [01:35<01:36,  5.11it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  51%|█████     | 506/1001 [01:35<01:35,  5.18it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  51%|█████     | 507/1001 [01:36<01:34,  5.23it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  51%|█████     | 508/1001 [01:36<01:33,  5.26it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  51%|█████     | 509/1001 [01:36<01:33,  5.27it/s]\u001b[A\n","Iteration: 500 Loss: 3.8679433822631837 Accuracy: 0.24601156413555145:  51%|█████     | 510/1001 [01:36<01:32,  5.31it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  51%|█████     | 510/1001 [01:36<01:32,  5.31it/s] \u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  51%|█████     | 511/1001 [01:36<01:33,  5.26it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  51%|█████     | 512/1001 [01:37<01:31,  5.32it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  51%|█████     | 513/1001 [01:37<01:31,  5.36it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  51%|█████▏    | 514/1001 [01:37<01:30,  5.37it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  51%|█████▏    | 515/1001 [01:37<01:31,  5.30it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  52%|█████▏    | 516/1001 [01:37<01:31,  5.30it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  52%|█████▏    | 517/1001 [01:38<01:30,  5.33it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  52%|█████▏    | 518/1001 [01:38<01:30,  5.34it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  52%|█████▏    | 519/1001 [01:38<01:30,  5.32it/s]\u001b[A\n","Iteration: 510 Loss: 4.027422165870666 Accuracy: 0.24939169287681578:  52%|█████▏    | 520/1001 [01:38<01:30,  5.34it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  52%|█████▏    | 520/1001 [01:38<01:30,  5.34it/s] \u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  52%|█████▏    | 521/1001 [01:38<01:31,  5.26it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  52%|█████▏    | 522/1001 [01:39<01:30,  5.30it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  52%|█████▏    | 523/1001 [01:39<01:29,  5.34it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  52%|█████▏    | 524/1001 [01:39<01:29,  5.35it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  52%|█████▏    | 525/1001 [01:39<01:29,  5.31it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  53%|█████▎    | 526/1001 [01:39<01:29,  5.33it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  53%|█████▎    | 527/1001 [01:39<01:30,  5.24it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  53%|█████▎    | 528/1001 [01:40<01:30,  5.23it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  53%|█████▎    | 529/1001 [01:40<01:30,  5.21it/s]\u001b[A\n","Iteration: 520 Loss: 4.252667546272278 Accuracy: 0.2516160011291504:  53%|█████▎    | 530/1001 [01:40<01:30,  5.18it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  53%|█████▎    | 530/1001 [01:40<01:30,  5.18it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  53%|█████▎    | 531/1001 [01:40<01:30,  5.19it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  53%|█████▎    | 532/1001 [01:40<01:29,  5.25it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  53%|█████▎    | 533/1001 [01:41<01:29,  5.25it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  53%|█████▎    | 534/1001 [01:41<01:28,  5.30it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  53%|█████▎    | 535/1001 [01:41<01:27,  5.34it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  54%|█████▎    | 536/1001 [01:41<01:26,  5.37it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  54%|█████▎    | 537/1001 [01:41<01:26,  5.37it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  54%|█████▎    | 538/1001 [01:42<01:27,  5.30it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  54%|█████▍    | 539/1001 [01:42<01:27,  5.27it/s]\u001b[A\n","Iteration: 530 Loss: 4.152505898475647 Accuracy: 0.24420367628335954:  54%|█████▍    | 540/1001 [01:42<01:26,  5.30it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  54%|█████▍    | 540/1001 [01:42<01:26,  5.30it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  54%|█████▍    | 541/1001 [01:42<01:26,  5.29it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  54%|█████▍    | 542/1001 [01:42<01:27,  5.27it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  54%|█████▍    | 543/1001 [01:42<01:27,  5.26it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  54%|█████▍    | 544/1001 [01:43<01:25,  5.32it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  54%|█████▍    | 545/1001 [01:43<01:26,  5.26it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  55%|█████▍    | 546/1001 [01:43<01:25,  5.30it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  55%|█████▍    | 547/1001 [01:43<01:25,  5.34it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  55%|█████▍    | 548/1001 [01:43<01:25,  5.31it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  55%|█████▍    | 549/1001 [01:44<01:24,  5.33it/s]\u001b[A\n","Iteration: 540 Loss: 4.1840317964553835 Accuracy: 0.25216037929058077:  55%|█████▍    | 550/1001 [01:44<01:24,  5.33it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  55%|█████▍    | 550/1001 [01:44<01:24,  5.33it/s]   \u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  55%|█████▌    | 551/1001 [01:44<01:25,  5.27it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  55%|█████▌    | 552/1001 [01:44<01:24,  5.30it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  55%|█████▌    | 553/1001 [01:44<01:23,  5.35it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  55%|█████▌    | 554/1001 [01:45<01:25,  5.22it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  55%|█████▌    | 555/1001 [01:45<01:27,  5.13it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  56%|█████▌    | 556/1001 [01:45<01:26,  5.17it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  56%|█████▌    | 557/1001 [01:45<01:25,  5.21it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  56%|█████▌    | 558/1001 [01:45<01:24,  5.26it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  56%|█████▌    | 559/1001 [01:46<01:23,  5.29it/s]\u001b[A\n","Iteration: 550 Loss: 4.256097412109375 Accuracy: 0.257609598338604:  56%|█████▌    | 560/1001 [01:46<01:23,  5.27it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  56%|█████▌    | 560/1001 [01:46<01:23,  5.27it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  56%|█████▌    | 561/1001 [01:46<01:24,  5.18it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  56%|█████▌    | 562/1001 [01:46<01:23,  5.23it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  56%|█████▌    | 563/1001 [01:46<01:23,  5.26it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  56%|█████▋    | 564/1001 [01:46<01:22,  5.28it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  56%|█████▋    | 565/1001 [01:47<01:22,  5.30it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  57%|█████▋    | 566/1001 [01:47<01:21,  5.33it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  57%|█████▋    | 567/1001 [01:47<01:20,  5.36it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  57%|█████▋    | 568/1001 [01:47<01:20,  5.37it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  57%|█████▋    | 569/1001 [01:47<01:20,  5.37it/s]\u001b[A\n","Iteration: 560 Loss: 3.9485406160354612 Accuracy: 0.2548229157924652:  57%|█████▋    | 570/1001 [01:48<01:19,  5.39it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  57%|█████▋    | 570/1001 [01:48<01:19,  5.39it/s] \u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  57%|█████▋    | 571/1001 [01:48<01:20,  5.36it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  57%|█████▋    | 572/1001 [01:48<01:19,  5.38it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  57%|█████▋    | 573/1001 [01:48<01:19,  5.38it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  57%|█████▋    | 574/1001 [01:48<01:19,  5.35it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  57%|█████▋    | 575/1001 [01:49<01:20,  5.29it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  58%|█████▊    | 576/1001 [01:49<01:20,  5.29it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  58%|█████▊    | 577/1001 [01:49<01:19,  5.31it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  58%|█████▊    | 578/1001 [01:49<01:19,  5.35it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  58%|█████▊    | 579/1001 [01:49<01:18,  5.38it/s]\u001b[A\n","Iteration: 570 Loss: 3.846024775505066 Accuracy: 0.2610983595252037:  58%|█████▊    | 580/1001 [01:49<01:18,  5.38it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  58%|█████▊    | 580/1001 [01:50<01:18,  5.38it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  58%|█████▊    | 581/1001 [01:50<01:19,  5.30it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  58%|█████▊    | 582/1001 [01:50<01:18,  5.34it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  58%|█████▊    | 583/1001 [01:50<01:18,  5.32it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  58%|█████▊    | 584/1001 [01:50<01:19,  5.27it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  58%|█████▊    | 585/1001 [01:50<01:18,  5.31it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  59%|█████▊    | 586/1001 [01:51<01:19,  5.22it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  59%|█████▊    | 587/1001 [01:51<01:19,  5.22it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  59%|█████▊    | 588/1001 [01:51<01:18,  5.24it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  59%|█████▉    | 589/1001 [01:51<01:18,  5.25it/s]\u001b[A\n","Iteration: 580 Loss: 3.7768564462661742 Accuracy: 0.2653138965368271:  59%|█████▉    | 590/1001 [01:51<01:17,  5.28it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  59%|█████▉    | 590/1001 [01:52<01:17,  5.28it/s] \u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  59%|█████▉    | 591/1001 [01:52<01:18,  5.19it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  59%|█████▉    | 592/1001 [01:52<01:17,  5.26it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  59%|█████▉    | 593/1001 [01:52<01:17,  5.25it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  59%|█████▉    | 594/1001 [01:52<01:16,  5.30it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  59%|█████▉    | 595/1001 [01:52<01:16,  5.34it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  60%|█████▉    | 596/1001 [01:52<01:15,  5.36it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  60%|█████▉    | 597/1001 [01:53<01:15,  5.35it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  60%|█████▉    | 598/1001 [01:53<01:14,  5.38it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  60%|█████▉    | 599/1001 [01:53<01:15,  5.29it/s]\u001b[A\n","Iteration: 590 Loss: 3.838985562324524 Accuracy: 0.2551154404878616:  60%|█████▉    | 600/1001 [01:53<01:15,  5.33it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  60%|█████▉    | 600/1001 [01:53<01:15,  5.33it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  60%|██████    | 601/1001 [01:54<01:30,  4.44it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  60%|██████    | 602/1001 [01:54<01:27,  4.57it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  60%|██████    | 603/1001 [01:54<01:23,  4.79it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  60%|██████    | 604/1001 [01:54<01:19,  4.96it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  60%|██████    | 605/1001 [01:54<01:17,  5.10it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  61%|██████    | 606/1001 [01:54<01:17,  5.11it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  61%|██████    | 607/1001 [01:55<01:16,  5.18it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  61%|██████    | 608/1001 [01:55<01:15,  5.22it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  61%|██████    | 609/1001 [01:55<01:14,  5.23it/s]\u001b[A\n","Iteration: 600 Loss: 3.824848985671997 Accuracy: 0.24584717601537703:  61%|██████    | 610/1001 [01:55<01:15,  5.20it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  61%|██████    | 610/1001 [01:55<01:15,  5.20it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  61%|██████    | 611/1001 [01:55<01:16,  5.09it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  61%|██████    | 612/1001 [01:56<01:16,  5.09it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  61%|██████    | 613/1001 [01:56<01:15,  5.13it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  61%|██████▏   | 614/1001 [01:56<01:15,  5.10it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  61%|██████▏   | 615/1001 [01:56<01:14,  5.15it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  62%|██████▏   | 616/1001 [01:56<01:13,  5.23it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  62%|██████▏   | 617/1001 [01:57<01:13,  5.23it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  62%|██████▏   | 618/1001 [01:57<01:13,  5.24it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  62%|██████▏   | 619/1001 [01:57<01:13,  5.19it/s]\u001b[A\n","Iteration: 610 Loss: 4.2300739765167235 Accuracy: 0.2517170414328575:  62%|██████▏   | 620/1001 [01:57<01:12,  5.24it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  62%|██████▏   | 620/1001 [01:57<01:12,  5.24it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  62%|██████▏   | 621/1001 [01:57<01:12,  5.22it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  62%|██████▏   | 622/1001 [01:58<01:12,  5.26it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  62%|██████▏   | 623/1001 [01:58<01:11,  5.31it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  62%|██████▏   | 624/1001 [01:58<01:10,  5.34it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  62%|██████▏   | 625/1001 [01:58<01:10,  5.36it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  63%|██████▎   | 626/1001 [01:58<01:09,  5.39it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  63%|██████▎   | 627/1001 [01:58<01:09,  5.39it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  63%|██████▎   | 628/1001 [01:59<01:09,  5.33it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  63%|██████▎   | 629/1001 [01:59<01:10,  5.25it/s]\u001b[A\n","Iteration: 620 Loss: 3.7438381910324097 Accuracy: 0.24633481949567795:  63%|██████▎   | 630/1001 [01:59<01:10,  5.27it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  63%|██████▎   | 630/1001 [01:59<01:10,  5.27it/s] \u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  63%|██████▎   | 631/1001 [01:59<01:10,  5.25it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  63%|██████▎   | 632/1001 [01:59<01:09,  5.31it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  63%|██████▎   | 633/1001 [02:00<01:08,  5.35it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  63%|██████▎   | 634/1001 [02:00<01:09,  5.26it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  63%|██████▎   | 635/1001 [02:00<01:09,  5.29it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  64%|██████▎   | 636/1001 [02:00<01:09,  5.25it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  64%|██████▎   | 637/1001 [02:00<01:08,  5.29it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  64%|██████▎   | 638/1001 [02:01<01:08,  5.26it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  64%|██████▍   | 639/1001 [02:01<01:08,  5.30it/s]\u001b[A\n","Iteration: 630 Loss: 4.207036852836609 Accuracy: 0.25286542177200316:  64%|██████▍   | 640/1001 [02:01<01:07,  5.34it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  64%|██████▍   | 640/1001 [02:01<01:07,  5.34it/s] \u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  64%|██████▍   | 641/1001 [02:01<01:07,  5.33it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  64%|██████▍   | 642/1001 [02:01<01:07,  5.35it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  64%|██████▍   | 643/1001 [02:02<01:07,  5.33it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  64%|██████▍   | 644/1001 [02:02<01:08,  5.23it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  64%|██████▍   | 645/1001 [02:02<01:08,  5.20it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  65%|██████▍   | 646/1001 [02:02<01:08,  5.16it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  65%|██████▍   | 647/1001 [02:02<01:08,  5.16it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  65%|██████▍   | 648/1001 [02:02<01:07,  5.26it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  65%|██████▍   | 649/1001 [02:03<01:06,  5.32it/s]\u001b[A\n","Iteration: 640 Loss: 4.003090476989746 Accuracy: 0.2622426927089691:  65%|██████▍   | 650/1001 [02:03<01:05,  5.35it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  65%|██████▍   | 650/1001 [02:03<01:05,  5.35it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  65%|██████▌   | 651/1001 [02:03<01:08,  5.11it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  65%|██████▌   | 652/1001 [02:03<01:07,  5.21it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  65%|██████▌   | 653/1001 [02:03<01:06,  5.22it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  65%|██████▌   | 654/1001 [02:04<01:06,  5.21it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  65%|██████▌   | 655/1001 [02:04<01:05,  5.28it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  66%|██████▌   | 656/1001 [02:04<01:04,  5.32it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  66%|██████▌   | 657/1001 [02:04<01:04,  5.33it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  66%|██████▌   | 658/1001 [02:04<01:06,  5.19it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  66%|██████▌   | 659/1001 [02:05<01:04,  5.27it/s]\u001b[A\n","Iteration: 650 Loss: 3.5477877140045164 Accuracy: 0.2555445834994316:  66%|██████▌   | 660/1001 [02:05<01:04,  5.31it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  66%|██████▌   | 660/1001 [02:05<01:04,  5.31it/s] \u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  66%|██████▌   | 661/1001 [02:05<01:04,  5.26it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  66%|██████▌   | 662/1001 [02:05<01:03,  5.32it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  66%|██████▌   | 663/1001 [02:05<01:03,  5.30it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  66%|██████▋   | 664/1001 [02:06<01:03,  5.31it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  66%|██████▋   | 665/1001 [02:06<01:03,  5.31it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  67%|██████▋   | 666/1001 [02:06<01:03,  5.26it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  67%|██████▋   | 667/1001 [02:06<01:02,  5.32it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  67%|██████▋   | 668/1001 [02:06<01:02,  5.33it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  67%|██████▋   | 669/1001 [02:06<01:02,  5.29it/s]\u001b[A\n","Iteration: 660 Loss: 3.998008465766907 Accuracy: 0.2532303795218468:  67%|██████▋   | 670/1001 [02:07<01:02,  5.32it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  67%|██████▋   | 670/1001 [02:07<01:02,  5.32it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  67%|██████▋   | 671/1001 [02:07<01:03,  5.23it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  67%|██████▋   | 672/1001 [02:07<01:02,  5.23it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  67%|██████▋   | 673/1001 [02:07<01:02,  5.25it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  67%|██████▋   | 674/1001 [02:07<01:01,  5.31it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  67%|██████▋   | 675/1001 [02:08<01:00,  5.35it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  68%|██████▊   | 676/1001 [02:08<01:00,  5.36it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  68%|██████▊   | 677/1001 [02:08<01:00,  5.35it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  68%|██████▊   | 678/1001 [02:08<01:00,  5.35it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  68%|██████▊   | 679/1001 [02:08<00:59,  5.40it/s]\u001b[A\n","Iteration: 670 Loss: 3.960308647155762 Accuracy: 0.2472851425409317:  68%|██████▊   | 680/1001 [02:09<01:00,  5.35it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  68%|██████▊   | 680/1001 [02:09<01:00,  5.35it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  68%|██████▊   | 681/1001 [02:09<00:59,  5.33it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  68%|██████▊   | 682/1001 [02:09<00:59,  5.35it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  68%|██████▊   | 683/1001 [02:09<00:59,  5.35it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  68%|██████▊   | 684/1001 [02:09<00:59,  5.30it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  68%|██████▊   | 685/1001 [02:09<01:00,  5.26it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  69%|██████▊   | 686/1001 [02:10<00:59,  5.25it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  69%|██████▊   | 687/1001 [02:10<00:59,  5.26it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  69%|██████▊   | 688/1001 [02:10<00:59,  5.25it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  69%|██████▉   | 689/1001 [02:10<00:59,  5.28it/s]\u001b[A\n","Iteration: 680 Loss: 4.183181023597717 Accuracy: 0.2556892231106758:  69%|██████▉   | 690/1001 [02:10<00:59,  5.27it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  69%|██████▉   | 690/1001 [02:11<00:59,  5.27it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  69%|██████▉   | 691/1001 [02:11<00:59,  5.23it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  69%|██████▉   | 692/1001 [02:11<00:58,  5.31it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  69%|██████▉   | 693/1001 [02:11<00:57,  5.33it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  69%|██████▉   | 694/1001 [02:11<00:57,  5.37it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  69%|██████▉   | 695/1001 [02:11<00:56,  5.38it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  70%|██████▉   | 696/1001 [02:12<00:56,  5.37it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  70%|██████▉   | 697/1001 [02:12<00:56,  5.38it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  70%|██████▉   | 698/1001 [02:12<00:56,  5.32it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  70%|██████▉   | 699/1001 [02:12<00:56,  5.31it/s]\u001b[A\n","Iteration: 690 Loss: 3.9089762210845946 Accuracy: 0.26859765499830246:  70%|██████▉   | 700/1001 [02:12<00:56,  5.35it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  70%|██████▉   | 700/1001 [02:12<00:56,  5.35it/s]  \u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  70%|███████   | 701/1001 [02:13<01:07,  4.45it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  70%|███████   | 702/1001 [02:13<01:04,  4.65it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  70%|███████   | 703/1001 [02:13<01:01,  4.84it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  70%|███████   | 704/1001 [02:13<00:59,  5.01it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  70%|███████   | 705/1001 [02:13<00:58,  5.10it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  71%|███████   | 706/1001 [02:14<00:58,  5.07it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  71%|███████   | 707/1001 [02:14<00:56,  5.18it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  71%|███████   | 708/1001 [02:14<00:56,  5.18it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  71%|███████   | 709/1001 [02:14<00:56,  5.19it/s]\u001b[A\n","Iteration: 700 Loss: 3.946145701408386 Accuracy: 0.2725434049963951:  71%|███████   | 710/1001 [02:14<00:55,  5.25it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  71%|███████   | 710/1001 [02:14<00:55,  5.25it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  71%|███████   | 711/1001 [02:15<00:55,  5.22it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  71%|███████   | 712/1001 [02:15<00:55,  5.23it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  71%|███████   | 713/1001 [02:15<00:54,  5.24it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  71%|███████▏  | 714/1001 [02:15<00:55,  5.13it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  71%|███████▏  | 715/1001 [02:15<00:55,  5.20it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  72%|███████▏  | 716/1001 [02:15<00:54,  5.24it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  72%|███████▏  | 717/1001 [02:16<00:54,  5.18it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  72%|███████▏  | 718/1001 [02:16<00:54,  5.23it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  72%|███████▏  | 719/1001 [02:16<00:53,  5.27it/s]\u001b[A\n","Iteration: 710 Loss: 3.9357027053833007 Accuracy: 0.2593500390648842:  72%|███████▏  | 720/1001 [02:16<00:52,  5.31it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  72%|███████▏  | 720/1001 [02:16<00:52,  5.31it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  72%|███████▏  | 721/1001 [02:16<00:53,  5.24it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  72%|███████▏  | 722/1001 [02:17<00:54,  5.15it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  72%|███████▏  | 723/1001 [02:17<00:53,  5.15it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  72%|███████▏  | 724/1001 [02:17<00:53,  5.20it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  72%|███████▏  | 725/1001 [02:17<00:52,  5.23it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  73%|███████▎  | 726/1001 [02:17<00:52,  5.20it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  73%|███████▎  | 727/1001 [02:18<00:52,  5.25it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  73%|███████▎  | 728/1001 [02:18<00:51,  5.30it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  73%|███████▎  | 729/1001 [02:18<00:51,  5.31it/s]\u001b[A\n","Iteration: 720 Loss: 3.8582273006439207 Accuracy: 0.2600216045975685:  73%|███████▎  | 730/1001 [02:18<00:50,  5.34it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  73%|███████▎  | 730/1001 [02:18<00:50,  5.34it/s]   \u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  73%|███████▎  | 731/1001 [02:18<00:51,  5.21it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  73%|███████▎  | 732/1001 [02:19<00:51,  5.22it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  73%|███████▎  | 733/1001 [02:19<00:50,  5.26it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  73%|███████▎  | 734/1001 [02:19<00:50,  5.32it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  73%|███████▎  | 735/1001 [02:19<00:49,  5.36it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  74%|███████▎  | 736/1001 [02:19<00:49,  5.39it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  74%|███████▎  | 737/1001 [02:19<00:49,  5.36it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  74%|███████▎  | 738/1001 [02:20<00:49,  5.30it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  74%|███████▍  | 739/1001 [02:20<00:49,  5.28it/s]\u001b[A\n","Iteration: 730 Loss: 4.076112270355225 Accuracy: 0.25377177298069:  74%|███████▍  | 740/1001 [02:20<00:49,  5.31it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  74%|███████▍  | 740/1001 [02:20<00:49,  5.31it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  74%|███████▍  | 741/1001 [02:20<00:49,  5.26it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  74%|███████▍  | 742/1001 [02:20<00:49,  5.28it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  74%|███████▍  | 743/1001 [02:21<00:49,  5.25it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  74%|███████▍  | 744/1001 [02:21<00:48,  5.31it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  74%|███████▍  | 745/1001 [02:21<00:48,  5.33it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  75%|███████▍  | 746/1001 [02:21<00:47,  5.38it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  75%|███████▍  | 747/1001 [02:21<00:47,  5.37it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  75%|███████▍  | 748/1001 [02:22<00:47,  5.32it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  75%|███████▍  | 749/1001 [02:22<00:47,  5.35it/s]\u001b[A\n","Iteration: 740 Loss: 3.7815128564834595 Accuracy: 0.25939765125513076:  75%|███████▍  | 750/1001 [02:22<00:46,  5.38it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  75%|███████▍  | 750/1001 [02:22<00:46,  5.38it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  75%|███████▌  | 751/1001 [02:22<00:47,  5.28it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  75%|███████▌  | 752/1001 [02:22<00:47,  5.27it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  75%|███████▌  | 753/1001 [02:22<00:47,  5.27it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  75%|███████▌  | 754/1001 [02:23<00:47,  5.21it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  75%|███████▌  | 755/1001 [02:23<00:47,  5.18it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  76%|███████▌  | 756/1001 [02:23<00:47,  5.16it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  76%|███████▌  | 757/1001 [02:23<00:47,  5.17it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  76%|███████▌  | 758/1001 [02:23<00:46,  5.20it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  76%|███████▌  | 759/1001 [02:24<00:46,  5.20it/s]\u001b[A\n","Iteration: 750 Loss: 3.8081705570220947 Accuracy: 0.25507528632879256:  76%|███████▌  | 760/1001 [02:24<00:46,  5.15it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  76%|███████▌  | 760/1001 [02:24<00:46,  5.15it/s] \u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  76%|███████▌  | 761/1001 [02:24<00:47,  5.02it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  76%|███████▌  | 762/1001 [02:24<00:46,  5.10it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  76%|███████▌  | 763/1001 [02:24<00:45,  5.20it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  76%|███████▋  | 764/1001 [02:25<00:45,  5.24it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  76%|███████▋  | 765/1001 [02:25<00:45,  5.24it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  77%|███████▋  | 766/1001 [02:25<00:44,  5.25it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  77%|███████▋  | 767/1001 [02:25<00:44,  5.26it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  77%|███████▋  | 768/1001 [02:25<00:44,  5.20it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  77%|███████▋  | 769/1001 [02:26<00:44,  5.22it/s]\u001b[A\n","Iteration: 760 Loss: 4.199476742744446 Accuracy: 0.24873246103525162:  77%|███████▋  | 770/1001 [02:26<00:43,  5.27it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  77%|███████▋  | 770/1001 [02:26<00:43,  5.27it/s] \u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  77%|███████▋  | 771/1001 [02:26<00:44,  5.22it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  77%|███████▋  | 772/1001 [02:26<00:43,  5.27it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  77%|███████▋  | 773/1001 [02:26<00:42,  5.33it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  77%|███████▋  | 774/1001 [02:26<00:42,  5.37it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  77%|███████▋  | 775/1001 [02:27<00:42,  5.37it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  78%|███████▊  | 776/1001 [02:27<00:41,  5.39it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  78%|███████▊  | 777/1001 [02:27<00:41,  5.35it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  78%|███████▊  | 778/1001 [02:27<00:41,  5.33it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  78%|███████▊  | 779/1001 [02:27<00:41,  5.29it/s]\u001b[A\n","Iteration: 770 Loss: 3.741405653953552 Accuracy: 0.2465490773320198:  78%|███████▊  | 780/1001 [02:28<00:42,  5.26it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  78%|███████▊  | 780/1001 [02:28<00:42,  5.26it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  78%|███████▊  | 781/1001 [02:28<00:42,  5.18it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  78%|███████▊  | 782/1001 [02:28<00:41,  5.23it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  78%|███████▊  | 783/1001 [02:28<00:42,  5.16it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  78%|███████▊  | 784/1001 [02:28<00:41,  5.23it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  78%|███████▊  | 785/1001 [02:29<00:41,  5.21it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  79%|███████▊  | 786/1001 [02:29<00:41,  5.20it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  79%|███████▊  | 787/1001 [02:29<00:40,  5.22it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  79%|███████▊  | 788/1001 [02:29<00:40,  5.29it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  79%|███████▉  | 789/1001 [02:29<00:39,  5.34it/s]\u001b[A\n","Iteration: 780 Loss: 3.9960076808929443 Accuracy: 0.26960526704788207:  79%|███████▉  | 790/1001 [02:30<00:39,  5.39it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  79%|███████▉  | 790/1001 [02:30<00:39,  5.39it/s] \u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  79%|███████▉  | 791/1001 [02:30<00:40,  5.18it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  79%|███████▉  | 792/1001 [02:30<00:40,  5.18it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  79%|███████▉  | 793/1001 [02:30<00:39,  5.23it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  79%|███████▉  | 794/1001 [02:30<00:39,  5.25it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  79%|███████▉  | 795/1001 [02:30<00:39,  5.28it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  80%|███████▉  | 796/1001 [02:31<00:38,  5.27it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  80%|███████▉  | 797/1001 [02:31<00:39,  5.23it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  80%|███████▉  | 798/1001 [02:31<00:38,  5.23it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  80%|███████▉  | 799/1001 [02:31<00:38,  5.20it/s]\u001b[A\n","Iteration: 790 Loss: 3.8948888778686523 Accuracy: 0.2629918798804283:  80%|███████▉  | 800/1001 [02:31<00:38,  5.20it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  80%|███████▉  | 800/1001 [02:32<00:38,  5.20it/s] \u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  80%|████████  | 801/1001 [02:32<00:45,  4.35it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  80%|████████  | 802/1001 [02:32<00:43,  4.58it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  80%|████████  | 803/1001 [02:32<00:41,  4.78it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  80%|████████  | 804/1001 [02:32<00:39,  4.94it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  80%|████████  | 805/1001 [02:33<00:38,  5.06it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  81%|████████  | 806/1001 [02:33<00:37,  5.15it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  81%|████████  | 807/1001 [02:33<00:37,  5.18it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  81%|████████  | 808/1001 [02:33<00:36,  5.27it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  81%|████████  | 809/1001 [02:33<00:36,  5.29it/s]\u001b[A\n","Iteration: 800 Loss: 3.589165472984314 Accuracy: 0.2620099544525146:  81%|████████  | 810/1001 [02:33<00:36,  5.29it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  81%|████████  | 810/1001 [02:34<00:36,  5.29it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  81%|████████  | 811/1001 [02:34<00:35,  5.31it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  81%|████████  | 812/1001 [02:34<00:35,  5.31it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  81%|████████  | 813/1001 [02:34<00:35,  5.30it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  81%|████████▏ | 814/1001 [02:34<00:35,  5.32it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  81%|████████▏ | 815/1001 [02:34<00:34,  5.36it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  82%|████████▏ | 816/1001 [02:35<00:34,  5.39it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  82%|████████▏ | 817/1001 [02:35<00:33,  5.41it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  82%|████████▏ | 818/1001 [02:35<00:33,  5.41it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  82%|████████▏ | 819/1001 [02:35<00:33,  5.39it/s]\u001b[A\n","Iteration: 810 Loss: 3.819892072677612 Accuracy: 0.2601872876286507:  82%|████████▏ | 820/1001 [02:35<00:34,  5.31it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  82%|████████▏ | 820/1001 [02:36<00:34,  5.31it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  82%|████████▏ | 821/1001 [02:36<00:33,  5.31it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  82%|████████▏ | 822/1001 [02:36<00:33,  5.36it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  82%|████████▏ | 823/1001 [02:36<00:33,  5.35it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  82%|████████▏ | 824/1001 [02:36<00:33,  5.36it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  82%|████████▏ | 825/1001 [02:36<00:32,  5.36it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  83%|████████▎ | 826/1001 [02:36<00:32,  5.31it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  83%|████████▎ | 827/1001 [02:37<00:32,  5.34it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  83%|████████▎ | 828/1001 [02:37<00:32,  5.35it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  83%|████████▎ | 829/1001 [02:37<00:31,  5.38it/s]\u001b[A\n","Iteration: 820 Loss: 4.041475439071656 Accuracy: 0.27059553414583204:  83%|████████▎ | 830/1001 [02:37<00:31,  5.36it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  83%|████████▎ | 830/1001 [02:37<00:31,  5.36it/s] \u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  83%|████████▎ | 831/1001 [02:37<00:32,  5.25it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  83%|████████▎ | 832/1001 [02:38<00:31,  5.32it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  83%|████████▎ | 833/1001 [02:38<00:31,  5.35it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  83%|████████▎ | 834/1001 [02:38<00:31,  5.31it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  83%|████████▎ | 835/1001 [02:38<00:31,  5.35it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  84%|████████▎ | 836/1001 [02:38<00:30,  5.36it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  84%|████████▎ | 837/1001 [02:38<00:30,  5.38it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  84%|████████▎ | 838/1001 [02:39<00:30,  5.28it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  84%|████████▍ | 839/1001 [02:39<00:30,  5.28it/s]\u001b[A\n","Iteration: 830 Loss: 3.951684665679932 Accuracy: 0.2720264717936516:  84%|████████▍ | 840/1001 [02:39<00:30,  5.31it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  84%|████████▍ | 840/1001 [02:39<00:30,  5.31it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  84%|████████▍ | 841/1001 [02:39<00:30,  5.31it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  84%|████████▍ | 842/1001 [02:39<00:29,  5.37it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  84%|████████▍ | 843/1001 [02:40<00:29,  5.36it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  84%|████████▍ | 844/1001 [02:40<00:29,  5.32it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  84%|████████▍ | 845/1001 [02:40<00:29,  5.29it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  85%|████████▍ | 846/1001 [02:40<00:29,  5.31it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  85%|████████▍ | 847/1001 [02:40<00:29,  5.25it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  85%|████████▍ | 848/1001 [02:41<00:28,  5.31it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  85%|████████▍ | 849/1001 [02:41<00:28,  5.36it/s]\u001b[A\n","Iteration: 840 Loss: 3.6728339433670043 Accuracy: 0.256024432182312:  85%|████████▍ | 850/1001 [02:41<00:28,  5.29it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  85%|████████▍ | 850/1001 [02:41<00:28,  5.29it/s] \u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  85%|████████▌ | 851/1001 [02:41<00:28,  5.27it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  85%|████████▌ | 852/1001 [02:41<00:27,  5.35it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  85%|████████▌ | 853/1001 [02:42<00:27,  5.32it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  85%|████████▌ | 854/1001 [02:42<00:28,  5.18it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  85%|████████▌ | 855/1001 [02:42<00:28,  5.10it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  86%|████████▌ | 856/1001 [02:42<00:27,  5.18it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  86%|████████▌ | 857/1001 [02:42<00:27,  5.24it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  86%|████████▌ | 858/1001 [02:42<00:27,  5.26it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  86%|████████▌ | 859/1001 [02:43<00:26,  5.28it/s]\u001b[A\n","Iteration: 850 Loss: 4.03725528717041 Accuracy: 0.2601996064186096:  86%|████████▌ | 860/1001 [02:43<00:26,  5.32it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  86%|████████▌ | 860/1001 [02:43<00:26,  5.32it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  86%|████████▌ | 861/1001 [02:43<00:26,  5.23it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  86%|████████▌ | 862/1001 [02:43<00:26,  5.15it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  86%|████████▌ | 863/1001 [02:43<00:26,  5.23it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  86%|████████▋ | 864/1001 [02:44<00:25,  5.27it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  86%|████████▋ | 865/1001 [02:44<00:25,  5.33it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  87%|████████▋ | 866/1001 [02:44<00:25,  5.37it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  87%|████████▋ | 867/1001 [02:44<00:24,  5.40it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  87%|████████▋ | 868/1001 [02:44<00:24,  5.37it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  87%|████████▋ | 869/1001 [02:45<00:24,  5.39it/s]\u001b[A\n","Iteration: 860 Loss: 3.8851427316665648 Accuracy: 0.2623452082276344:  87%|████████▋ | 870/1001 [02:45<00:24,  5.39it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  87%|████████▋ | 870/1001 [02:45<00:24,  5.39it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  87%|████████▋ | 871/1001 [02:45<00:24,  5.37it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  87%|████████▋ | 872/1001 [02:45<00:23,  5.40it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  87%|████████▋ | 873/1001 [02:45<00:23,  5.34it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  87%|████████▋ | 874/1001 [02:45<00:23,  5.33it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  87%|████████▋ | 875/1001 [02:46<00:23,  5.26it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  88%|████████▊ | 876/1001 [02:46<00:23,  5.32it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  88%|████████▊ | 877/1001 [02:46<00:23,  5.25it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  88%|████████▊ | 878/1001 [02:46<00:23,  5.29it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  88%|████████▊ | 879/1001 [02:46<00:22,  5.32it/s]\u001b[A\n","Iteration: 870 Loss: 3.575457739830017 Accuracy: 0.26485586166381836:  88%|████████▊ | 880/1001 [02:47<00:22,  5.36it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  88%|████████▊ | 880/1001 [02:47<00:22,  5.36it/s] \u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  88%|████████▊ | 881/1001 [02:47<00:22,  5.30it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  88%|████████▊ | 882/1001 [02:47<00:22,  5.31it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  88%|████████▊ | 883/1001 [02:47<00:22,  5.35it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  88%|████████▊ | 884/1001 [02:47<00:21,  5.32it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  88%|████████▊ | 885/1001 [02:48<00:21,  5.36it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  89%|████████▊ | 886/1001 [02:48<00:21,  5.36it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  89%|████████▊ | 887/1001 [02:48<00:21,  5.30it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  89%|████████▊ | 888/1001 [02:48<00:21,  5.34it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  89%|████████▉ | 889/1001 [02:48<00:20,  5.33it/s]\u001b[A\n","Iteration: 880 Loss: 3.867086577415466 Accuracy: 0.2655946329236031:  89%|████████▉ | 890/1001 [02:48<00:20,  5.34it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  89%|████████▉ | 890/1001 [02:49<00:20,  5.34it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  89%|████████▉ | 891/1001 [02:49<00:20,  5.27it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  89%|████████▉ | 892/1001 [02:49<00:20,  5.32it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  89%|████████▉ | 893/1001 [02:49<00:20,  5.37it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  89%|████████▉ | 894/1001 [02:49<00:19,  5.41it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  89%|████████▉ | 895/1001 [02:49<00:19,  5.42it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  90%|████████▉ | 896/1001 [02:50<00:19,  5.41it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  90%|████████▉ | 897/1001 [02:50<00:19,  5.38it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  90%|████████▉ | 898/1001 [02:50<00:19,  5.36it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  90%|████████▉ | 899/1001 [02:50<00:18,  5.38it/s]\u001b[A\n","Iteration: 890 Loss: 3.784092879295349 Accuracy: 0.2654834628105164:  90%|████████▉ | 900/1001 [02:50<00:19,  5.31it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  90%|████████▉ | 900/1001 [02:51<00:19,  5.31it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  90%|█████████ | 901/1001 [02:51<00:22,  4.39it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  90%|█████████ | 902/1001 [02:51<00:21,  4.57it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  90%|█████████ | 903/1001 [02:51<00:20,  4.76it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  90%|█████████ | 904/1001 [02:51<00:19,  4.90it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  90%|█████████ | 905/1001 [02:51<00:18,  5.05it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  91%|█████████ | 906/1001 [02:52<00:18,  5.04it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  91%|█████████ | 907/1001 [02:52<00:18,  5.05it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  91%|█████████ | 908/1001 [02:52<00:18,  5.08it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  91%|█████████ | 909/1001 [02:52<00:18,  5.11it/s]\u001b[A\n","Iteration: 900 Loss: 3.572895908355713 Accuracy: 0.2654738128185272:  91%|█████████ | 910/1001 [02:52<00:17,  5.17it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  91%|█████████ | 910/1001 [02:53<00:17,  5.17it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  91%|█████████ | 911/1001 [02:53<00:17,  5.12it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  91%|█████████ | 912/1001 [02:53<00:17,  5.14it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  91%|█████████ | 913/1001 [02:53<00:16,  5.22it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  91%|█████████▏| 914/1001 [02:53<00:16,  5.25it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  91%|█████████▏| 915/1001 [02:53<00:16,  5.30it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  92%|█████████▏| 916/1001 [02:54<00:15,  5.33it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  92%|█████████▏| 917/1001 [02:54<00:15,  5.33it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  92%|█████████▏| 918/1001 [02:54<00:15,  5.29it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  92%|█████████▏| 919/1001 [02:54<00:15,  5.22it/s]\u001b[A\n","Iteration: 910 Loss: 3.7343794107437134 Accuracy: 0.2872088313102722:  92%|█████████▏| 920/1001 [02:54<00:15,  5.21it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  92%|█████████▏| 920/1001 [02:55<00:15,  5.21it/s]  \u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  92%|█████████▏| 921/1001 [02:55<00:15,  5.18it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  92%|█████████▏| 922/1001 [02:55<00:15,  5.25it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  92%|█████████▏| 923/1001 [02:55<00:14,  5.25it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  92%|█████████▏| 924/1001 [02:55<00:14,  5.26it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  92%|█████████▏| 925/1001 [02:55<00:14,  5.27it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  93%|█████████▎| 926/1001 [02:55<00:14,  5.25it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  93%|█████████▎| 927/1001 [02:56<00:14,  5.26it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  93%|█████████▎| 928/1001 [02:56<00:13,  5.30it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  93%|█████████▎| 929/1001 [02:56<00:13,  5.34it/s]\u001b[A\n","Iteration: 920 Loss: 4.16554479598999 Accuracy: 0.2716250538825989:  93%|█████████▎| 930/1001 [02:56<00:13,  5.30it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  93%|█████████▎| 930/1001 [02:56<00:13,  5.30it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  93%|█████████▎| 931/1001 [02:56<00:13,  5.27it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  93%|█████████▎| 932/1001 [02:57<00:13,  5.29it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  93%|█████████▎| 933/1001 [02:57<00:12,  5.25it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  93%|█████████▎| 934/1001 [02:57<00:12,  5.29it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  93%|█████████▎| 935/1001 [02:57<00:12,  5.34it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  94%|█████████▎| 936/1001 [02:57<00:12,  5.38it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  94%|█████████▎| 937/1001 [02:58<00:11,  5.36it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  94%|█████████▎| 938/1001 [02:58<00:11,  5.33it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  94%|█████████▍| 939/1001 [02:58<00:11,  5.33it/s]\u001b[A\n","Iteration: 930 Loss: 3.776733922958374 Accuracy: 0.2705623745918274:  94%|█████████▍| 940/1001 [02:58<00:11,  5.27it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  94%|█████████▍| 940/1001 [02:58<00:11,  5.27it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  94%|█████████▍| 941/1001 [02:58<00:11,  5.21it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  94%|█████████▍| 942/1001 [02:58<00:11,  5.22it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  94%|█████████▍| 943/1001 [02:59<00:10,  5.28it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  94%|█████████▍| 944/1001 [02:59<00:10,  5.29it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  94%|█████████▍| 945/1001 [02:59<00:10,  5.33it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  95%|█████████▍| 946/1001 [02:59<00:10,  5.33it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  95%|█████████▍| 947/1001 [02:59<00:10,  5.37it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  95%|█████████▍| 948/1001 [03:00<00:09,  5.33it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  95%|█████████▍| 949/1001 [03:00<00:09,  5.33it/s]\u001b[A\n","Iteration: 940 Loss: 3.5713250637054443 Accuracy: 0.2713920921087265:  95%|█████████▍| 950/1001 [03:00<00:09,  5.30it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  95%|█████████▍| 950/1001 [03:00<00:09,  5.30it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  95%|█████████▌| 951/1001 [03:00<00:09,  5.22it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  95%|█████████▌| 952/1001 [03:00<00:09,  5.25it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  95%|█████████▌| 953/1001 [03:01<00:09,  5.26it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  95%|█████████▌| 954/1001 [03:01<00:09,  5.19it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  95%|█████████▌| 955/1001 [03:01<00:08,  5.21it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  96%|█████████▌| 956/1001 [03:01<00:08,  5.20it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  96%|█████████▌| 957/1001 [03:01<00:08,  5.15it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  96%|█████████▌| 958/1001 [03:02<00:08,  5.14it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  96%|█████████▌| 959/1001 [03:02<00:08,  5.19it/s]\u001b[A\n","Iteration: 950 Loss: 3.6553954124450683 Accuracy: 0.27026888728141785:  96%|█████████▌| 960/1001 [03:02<00:07,  5.25it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  96%|█████████▌| 960/1001 [03:02<00:07,  5.25it/s]   \u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  96%|█████████▌| 961/1001 [03:02<00:07,  5.21it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  96%|█████████▌| 962/1001 [03:02<00:07,  5.25it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  96%|█████████▌| 963/1001 [03:02<00:07,  5.28it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  96%|█████████▋| 964/1001 [03:03<00:06,  5.32it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  96%|█████████▋| 965/1001 [03:03<00:06,  5.35it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  97%|█████████▋| 966/1001 [03:03<00:06,  5.38it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  97%|█████████▋| 967/1001 [03:03<00:06,  5.38it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  97%|█████████▋| 968/1001 [03:03<00:06,  5.40it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  97%|█████████▋| 969/1001 [03:04<00:05,  5.42it/s]\u001b[A\n","Iteration: 960 Loss: 3.912655305862427 Accuracy: 0.260349802672863:  97%|█████████▋| 970/1001 [03:04<00:05,  5.42it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  97%|█████████▋| 970/1001 [03:04<00:05,  5.42it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  97%|█████████▋| 971/1001 [03:04<00:05,  5.35it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  97%|█████████▋| 972/1001 [03:04<00:05,  5.39it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  97%|█████████▋| 973/1001 [03:04<00:05,  5.35it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  97%|█████████▋| 974/1001 [03:05<00:05,  5.32it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  97%|█████████▋| 975/1001 [03:05<00:04,  5.29it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  98%|█████████▊| 976/1001 [03:05<00:04,  5.28it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  98%|█████████▊| 977/1001 [03:05<00:04,  5.33it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  98%|█████████▊| 978/1001 [03:05<00:04,  5.36it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  98%|█████████▊| 979/1001 [03:05<00:04,  5.38it/s]\u001b[A\n","Iteration: 970 Loss: 3.7743930578231812 Accuracy: 0.2708634316921234:  98%|█████████▊| 980/1001 [03:06<00:03,  5.34it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  98%|█████████▊| 980/1001 [03:06<00:03,  5.34it/s] \u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  98%|█████████▊| 981/1001 [03:06<00:03,  5.16it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  98%|█████████▊| 982/1001 [03:06<00:03,  5.16it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  98%|█████████▊| 983/1001 [03:06<00:03,  5.24it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  98%|█████████▊| 984/1001 [03:06<00:03,  5.27it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  98%|█████████▊| 985/1001 [03:07<00:03,  5.26it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  99%|█████████▊| 986/1001 [03:07<00:02,  5.31it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  99%|█████████▊| 987/1001 [03:07<00:02,  5.29it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  99%|█████████▊| 988/1001 [03:07<00:02,  5.31it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  99%|█████████▉| 989/1001 [03:07<00:02,  5.27it/s]\u001b[A\n","Iteration: 980 Loss: 3.842301845550537 Accuracy: 0.2741728961467743:  99%|█████████▉| 990/1001 [03:08<00:02,  5.22it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881:  99%|█████████▉| 990/1001 [03:08<00:02,  5.22it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881:  99%|█████████▉| 991/1001 [03:08<00:01,  5.20it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881:  99%|█████████▉| 992/1001 [03:08<00:01,  5.30it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881:  99%|█████████▉| 993/1001 [03:08<00:01,  5.27it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881:  99%|█████████▉| 994/1001 [03:08<00:01,  5.27it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881:  99%|█████████▉| 995/1001 [03:08<00:01,  5.29it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881: 100%|█████████▉| 996/1001 [03:09<00:00,  5.29it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881: 100%|█████████▉| 997/1001 [03:09<00:00,  5.27it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881: 100%|█████████▉| 998/1001 [03:09<00:00,  5.31it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881: 100%|█████████▉| 999/1001 [03:09<00:00,  5.31it/s]\u001b[A\n","Iteration: 990 Loss: 3.6605744123458863 Accuracy: 0.272500242292881: 100%|█████████▉| 1000/1001 [03:09<00:00,  5.34it/s]\u001b[A\n","Iteration: 1000 Loss: 3.767837476730347 Accuracy: 0.2785965889692307: 100%|█████████▉| 1000/1001 [03:10<00:00,  5.34it/s]\u001b[A\n","Iteration: 1000 Loss: 3.767837476730347 Accuracy: 0.2785965889692307: 100%|██████████| 1001/1001 [03:10<00:00,  5.26it/s]\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"id":"jo6AXe81FLrP"},"source":["# Using the Summarization model"]},{"cell_type":"markdown","metadata":{"id":"FsEOd0GXFLrQ"},"source":["Now that you have trained a Transformer to perform Summarization, we will use the model on news articles from the wild.\n","\n","The three subsections below explore what the model has learned."]},{"cell_type":"markdown","metadata":{"id":"tyO5f0ltFLrR"},"source":["## The validation loss"]},{"cell_type":"markdown","metadata":{"id":"RwMtIj6cFLrR"},"source":["Measure the validation loss of your model. This part could be used, as in our previous notebook, in deciding what is a likely, vs. unlikely summary for an article.\n","\n","We will use the code here with the unreleased test-set to evaluate your model."]},{"cell_type":"code","metadata":{"id":"M20e8ksXFLrS","executionInfo":{"status":"ok","timestamp":1617257075419,"user_tz":420,"elapsed":1030,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}}},"source":["gc.collect()\n","model_id = \"test1\"\n","save_dict = th.load(root_folder+'models/part2/'+f\"model_{model_id}.pt\", map_location='cpu')\n","model = Transformer(**save_dict['kwargs'])\n","model.load_state_dict(save_dict['model_state_dict'])\n","set_device('cpu')\n","model.eval()\n","trainer.model = model"],"execution_count":38,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"jpBA-TXJFLrT","executionInfo":{"status":"ok","timestamp":1617257095962,"user_tz":420,"elapsed":16483,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"7befb73a-ad3d-468f-ebe2-7922352ab3ad"},"source":["gc.collect()\n","losses = []\n","for i in tqdm(range(100)):\n","    batch = build_batch(d_valid, 1)\n","    # Build the feed-dict connecting placeholders and mini-batch\n","    batch_input, batch_input_mask, batch_output, batch_output_mask = [th.tensor(tensor) for tensor in batch]\n","    batch = {'source_sequence': batch_input, 'target_sequence': batch_output,\n","            'encoder_mask': batch_input_mask, 'decoder_mask': batch_output_mask}\n","    valid_loss, accuracy = trainer(batch,optimize=False)\n","    losses.append(float(valid_loss.cpu().item()))\n","print(\"Validation loss:\", np.mean(losses))"],"execution_count":39,"outputs":[{"output_type":"stream","text":["\n","  0%|          | 0/100 [00:00<?, ?it/s]\u001b[A\n","  1%|          | 1/100 [00:00<00:14,  6.64it/s]\u001b[A\n","  2%|▏         | 2/100 [00:00<00:15,  6.52it/s]\u001b[A\n","  3%|▎         | 3/100 [00:00<00:15,  6.42it/s]\u001b[A\n","  4%|▍         | 4/100 [00:00<00:15,  6.39it/s]\u001b[A\n","  5%|▌         | 5/100 [00:00<00:14,  6.35it/s]\u001b[A\n","  6%|▌         | 6/100 [00:00<00:14,  6.32it/s]\u001b[A\n","  7%|▋         | 7/100 [00:01<00:14,  6.28it/s]\u001b[A\n","  8%|▊         | 8/100 [00:01<00:14,  6.38it/s]\u001b[A\n","  9%|▉         | 9/100 [00:01<00:14,  6.50it/s]\u001b[A\n"," 10%|█         | 10/100 [00:01<00:14,  6.40it/s]\u001b[A\n"," 11%|█         | 11/100 [00:01<00:13,  6.46it/s]\u001b[A\n"," 12%|█▏        | 12/100 [00:01<00:13,  6.40it/s]\u001b[A\n"," 13%|█▎        | 13/100 [00:02<00:13,  6.48it/s]\u001b[A\n"," 14%|█▍        | 14/100 [00:02<00:13,  6.41it/s]\u001b[A\n"," 15%|█▌        | 15/100 [00:02<00:13,  6.47it/s]\u001b[A\n"," 16%|█▌        | 16/100 [00:02<00:13,  6.45it/s]\u001b[A\n"," 17%|█▋        | 17/100 [00:02<00:12,  6.50it/s]\u001b[A\n"," 18%|█▊        | 18/100 [00:02<00:12,  6.42it/s]\u001b[A\n"," 19%|█▉        | 19/100 [00:02<00:12,  6.50it/s]\u001b[A\n"," 20%|██        | 20/100 [00:03<00:12,  6.44it/s]\u001b[A\n"," 21%|██        | 21/100 [00:03<00:12,  6.46it/s]\u001b[A\n"," 22%|██▏       | 22/100 [00:03<00:12,  6.48it/s]\u001b[A\n"," 23%|██▎       | 23/100 [00:03<00:11,  6.49it/s]\u001b[A\n"," 24%|██▍       | 24/100 [00:03<00:11,  6.54it/s]\u001b[A\n"," 25%|██▌       | 25/100 [00:03<00:11,  6.49it/s]\u001b[A\n"," 26%|██▌       | 26/100 [00:04<00:11,  6.52it/s]\u001b[A\n"," 27%|██▋       | 27/100 [00:04<00:11,  6.54it/s]\u001b[A\n"," 28%|██▊       | 28/100 [00:04<00:10,  6.57it/s]\u001b[A\n"," 29%|██▉       | 29/100 [00:04<00:10,  6.58it/s]\u001b[A\n"," 30%|███       | 30/100 [00:04<00:10,  6.60it/s]\u001b[A\n"," 31%|███       | 31/100 [00:04<00:10,  6.57it/s]\u001b[A\n"," 32%|███▏      | 32/100 [00:04<00:10,  6.52it/s]\u001b[A\n"," 33%|███▎      | 33/100 [00:05<00:10,  6.46it/s]\u001b[A\n"," 34%|███▍      | 34/100 [00:05<00:10,  6.44it/s]\u001b[A\n"," 35%|███▌      | 35/100 [00:05<00:09,  6.52it/s]\u001b[A\n"," 36%|███▌      | 36/100 [00:05<00:09,  6.55it/s]\u001b[A\n"," 37%|███▋      | 37/100 [00:05<00:09,  6.56it/s]\u001b[A\n"," 38%|███▊      | 38/100 [00:05<00:09,  6.48it/s]\u001b[A\n"," 39%|███▉      | 39/100 [00:06<00:09,  6.46it/s]\u001b[A\n"," 40%|████      | 40/100 [00:06<00:09,  6.50it/s]\u001b[A\n"," 41%|████      | 41/100 [00:06<00:08,  6.56it/s]\u001b[A\n"," 42%|████▏     | 42/100 [00:06<00:08,  6.58it/s]\u001b[A\n"," 43%|████▎     | 43/100 [00:06<00:08,  6.60it/s]\u001b[A\n"," 44%|████▍     | 44/100 [00:06<00:08,  6.58it/s]\u001b[A\n"," 45%|████▌     | 45/100 [00:06<00:08,  6.46it/s]\u001b[A\n"," 46%|████▌     | 46/100 [00:07<00:08,  6.50it/s]\u001b[A\n"," 47%|████▋     | 47/100 [00:07<00:08,  6.56it/s]\u001b[A\n"," 48%|████▊     | 48/100 [00:07<00:07,  6.54it/s]\u001b[A\n"," 49%|████▉     | 49/100 [00:07<00:07,  6.58it/s]\u001b[A\n"," 50%|█████     | 50/100 [00:07<00:07,  6.56it/s]\u001b[A\n"," 51%|█████     | 51/100 [00:07<00:07,  6.58it/s]\u001b[A\n"," 52%|█████▏    | 52/100 [00:08<00:07,  6.56it/s]\u001b[A\n"," 53%|█████▎    | 53/100 [00:08<00:07,  6.55it/s]\u001b[A\n"," 54%|█████▍    | 54/100 [00:08<00:07,  6.50it/s]\u001b[A\n"," 55%|█████▌    | 55/100 [00:08<00:07,  6.34it/s]\u001b[A\n"," 56%|█████▌    | 56/100 [00:08<00:06,  6.43it/s]\u001b[A\n"," 57%|█████▋    | 57/100 [00:08<00:06,  6.47it/s]\u001b[A\n"," 58%|█████▊    | 58/100 [00:08<00:06,  6.45it/s]\u001b[A\n"," 59%|█████▉    | 59/100 [00:09<00:06,  6.49it/s]\u001b[A\n"," 60%|██████    | 60/100 [00:09<00:06,  6.46it/s]\u001b[A\n"," 61%|██████    | 61/100 [00:09<00:05,  6.52it/s]\u001b[A\n"," 62%|██████▏   | 62/100 [00:09<00:05,  6.59it/s]\u001b[A\n"," 63%|██████▎   | 63/100 [00:09<00:05,  6.41it/s]\u001b[A\n"," 64%|██████▍   | 64/100 [00:09<00:05,  6.42it/s]\u001b[A\n"," 65%|██████▌   | 65/100 [00:10<00:05,  6.37it/s]\u001b[A\n"," 66%|██████▌   | 66/100 [00:10<00:05,  6.35it/s]\u001b[A\n"," 67%|██████▋   | 67/100 [00:10<00:05,  6.37it/s]\u001b[A\n"," 68%|██████▊   | 68/100 [00:10<00:05,  6.39it/s]\u001b[A\n"," 69%|██████▉   | 69/100 [00:10<00:04,  6.46it/s]\u001b[A\n"," 70%|███████   | 70/100 [00:10<00:04,  6.47it/s]\u001b[A\n"," 71%|███████   | 71/100 [00:10<00:04,  6.45it/s]\u001b[A\n"," 72%|███████▏  | 72/100 [00:11<00:04,  6.50it/s]\u001b[A\n"," 73%|███████▎  | 73/100 [00:11<00:04,  6.48it/s]\u001b[A\n"," 74%|███████▍  | 74/100 [00:11<00:03,  6.53it/s]\u001b[A\n"," 75%|███████▌  | 75/100 [00:11<00:03,  6.49it/s]\u001b[A\n"," 76%|███████▌  | 76/100 [00:11<00:03,  6.61it/s]\u001b[A\n"," 77%|███████▋  | 77/100 [00:11<00:03,  6.65it/s]\u001b[A\n"," 78%|███████▊  | 78/100 [00:12<00:03,  6.64it/s]\u001b[A\n"," 79%|███████▉  | 79/100 [00:12<00:03,  6.63it/s]\u001b[A\n"," 80%|████████  | 80/100 [00:12<00:03,  6.65it/s]\u001b[A\n"," 81%|████████  | 81/100 [00:12<00:02,  6.64it/s]\u001b[A\n"," 82%|████████▏ | 82/100 [00:12<00:02,  6.50it/s]\u001b[A\n"," 83%|████████▎ | 83/100 [00:12<00:02,  6.50it/s]\u001b[A\n"," 84%|████████▍ | 84/100 [00:12<00:02,  6.46it/s]\u001b[A\n"," 85%|████████▌ | 85/100 [00:13<00:02,  6.49it/s]\u001b[A\n"," 86%|████████▌ | 86/100 [00:13<00:02,  6.52it/s]\u001b[A\n"," 87%|████████▋ | 87/100 [00:13<00:01,  6.54it/s]\u001b[A\n"," 88%|████████▊ | 88/100 [00:13<00:01,  6.54it/s]\u001b[A\n"," 89%|████████▉ | 89/100 [00:13<00:01,  6.56it/s]\u001b[A\n"," 90%|█████████ | 90/100 [00:13<00:01,  6.56it/s]\u001b[A\n"," 91%|█████████ | 91/100 [00:14<00:01,  6.54it/s]\u001b[A\n"," 92%|█████████▏| 92/100 [00:14<00:01,  6.57it/s]\u001b[A\n"," 93%|█████████▎| 93/100 [00:14<00:01,  6.48it/s]\u001b[A\n"," 94%|█████████▍| 94/100 [00:14<00:00,  6.54it/s]\u001b[A\n"," 95%|█████████▌| 95/100 [00:14<00:00,  6.53it/s]\u001b[A\n"," 96%|█████████▌| 96/100 [00:14<00:00,  6.56it/s]\u001b[A\n"," 97%|█████████▋| 97/100 [00:14<00:00,  6.57it/s]\u001b[A\n"," 98%|█████████▊| 98/100 [00:15<00:00,  6.51it/s]\u001b[A\n"," 99%|█████████▉| 99/100 [00:15<00:00,  6.54it/s]\u001b[A\n","100%|██████████| 100/100 [00:15<00:00,  6.50it/s]"],"name":"stderr"},{"output_type":"stream","text":["Validation loss: 5.321825635433197\n"],"name":"stdout"},{"output_type":"stream","text":["\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"KUDdDx4VFLrU","executionInfo":{"status":"ok","timestamp":1617257100298,"user_tz":420,"elapsed":302,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}}},"source":["# Your best performing model should go here.\n","os.makedirs(root_folder+\"best_models\",exist_ok=True)\n","best_model_file = root_folder+\"best_models/part2_best_model.pt\"\n","th.save(save_dict,best_model_file)"],"execution_count":40,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"fT_BvxRqFLrV"},"source":["## Generating an article's summary"]},{"cell_type":"markdown","metadata":{"id":"s1225KszFLrW"},"source":["This model we have built is meant to be used to generate summaries for new articles we do not have summaries for.\n","We got a [news article](https://www.chicagotribune.com/news/local/breaking/ct-met-officer-shot-20190309-story.html) from the Chicago Tribune about a police shooting, and want to use our model to produce a summary.\n","\n","As you will see, our model is still limited in its ability, and will most likely not produce an interpretible summary, however, with more data and training, this model would be able to produce good summaries."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"3t9dSsv1FLrW","executionInfo":{"status":"ok","timestamp":1617257115791,"user_tz":420,"elapsed":12845,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"27f4a1e0-b4d7-48f3-a460-3fc9ea53c8be"},"source":["article_text = \"A 34-year-old Chicago police officer has been shot in the shoulder during the execution of a search warrant in the Humboldt Park neighborhood, police say. The alleged shooter, a 19-year-old woman, was in custody. The shooting happened about 7:20 p.m. in the 2700 block of West Potomac Avenue, police said. The officer, part of the Grand Central District tactical unit, was taken to Stroger Hospital. While officers were serving a \\\"typical\\\" search warrant for \\\"narcotics and illegal weapons\\\" and were attempting to reach a rear door, \\\"a shot was fired,\\\" striking the tactical officer in the shoulder, said Chicago police Superintendent Eddie Johnson during a news briefing outside the hospital. He said the officer, who has about four or five years on the job, was \\\"stable\\\" but in critical condition. \\\"His family is here,\\\" Johnson said. \\\"He’s talking a lot and just wants the ordeal to be over.\\\" He said this incident serves as just another reminder of how dangerous a police officer’s job is. At the scene of the shooting, crime tape closed Potomac from Washtenaw Avenue to California Avenue and encompassed the alley west of the brick apartment building, south of Potomac. Dozens of officers stood in the alley, while even more walked up and down the street. Neighbors gathered at the edge of the yellow tape on the sidewalk along California and watched them work. Standing next to a man, a woman talked to police in the crime scene, across the street. \\\"We're not under arrest? We can go?\\\" the woman checked with officers. They told her she could go, and she and the man walked underneath the yellow tape and out of the crime scene.\"\n","input_length = 400\n","output_length = 100\n","\n","# Process the capitalization with the preprocess_capitalization of the capita package.\n","article_text = capita.preprocess_capitalization(article_text)\n","\n","# Numerize the tokens of the processed text using the loaded sentencepiece model.\n","numerized = sp.EncodeAsIds(article_text)\n","# Pad the sequence and keep the mask of the input\n","padded, mask = pad_sequence(numerized, pad_index, input_length)\n","\n","# Making the news article into a batch of size one, to be fed to the neural network.\n","encoder_input = np.array([padded])\n","encoder_mask = np.array([mask])\n","\n","decoded_so_far = [0]\n","\n","for j in range(output_length):\n","    padded_decoder_input, decoder_mask = pad_sequence(decoded_so_far, pad_index, output_length)\n","    padded_decoder_input = [padded_decoder_input]\n","    decoder_mask = [decoder_mask]\n","    print(\"========================\")\n","    print(padded_decoder_input)\n","    # Use the model to find the distrbution over the vocabulary for the next word\n","    batch = (encoder_input,encoder_mask,padded_decoder_input,decoder_mask)\n","    batch_input, batch_input_mask, batch_output, batch_output_mask = [th.tensor(tensor) for tensor in batch]\n","    batch = {'source_sequence': batch_input, 'target_sequence': batch_output,\n","            'encoder_mask': batch_input_mask, 'decoder_mask': batch_output_mask}\n","    logits = trainer.model(**batch).cpu().detach().numpy()\n","\n","    chosen_words = np.argmax(logits, axis=2) # Take the argmax, getting the most likely next word\n","    decoded_so_far.append(int(chosen_words[0, j])) # We add it to the summary so far\n","\n","\n","print(\"The final summary:\")\n","print(\"\".join([vocab[i] for i in decoded_so_far]).replace(\"▁\", \" \"))"],"execution_count":41,"outputs":[{"output_type":"stream","text":["========================\n","[[0, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 9998, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 194, 9998, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 194, 4037, 9998, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 194, 4037, 4037, 9998, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 194, 4037, 4037, 3, 9998, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 194, 4037, 4037, 3, 3, 9998, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 194, 4037, 4037, 3, 3, 4, 9998, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 194, 4037, 4037, 3, 3, 4, 4, 9998]]\n","========================\n","[[0, 3, 3, 4, 4, 107, 107, 528, 194, 58, 4037, 398, 3, 7450, 4, 7450, 8409, 3, 10, 4, 4037, 4037, 3, 3, 4, 4, 3659, 3659, 12, 12, 10, 10, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 3884, 3, 8933, 4, 7, 107, 3, 194, 4, 6, 4037, 3, 3, 4, 4, 107, 4037, 194, 3, 10, 4, 3, 4037, 4, 3, 107, 4, 194, 7308, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 4037, 4037, 3, 3, 4, 4, 107, 107, 194, 194, 4037, 4037, 3, 3, 4, 4, 7308]]\n","The final summary:\n","<unk>  ↑↑ police police officers say wereks killed  flock↑ flock condolences  a↑ksks  ↑↑ occasion occasion of of a a  ↑↑ksks  ↑↑ks joseph  1972↑ , police  say↑ .ks  ↑↑ policeks say  a↑ ks↑  police↑ say excitementks  ↑↑ksks  ↑↑ksks  ↑↑ police police say sayksks  ↑↑ excitement excitement\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"4cQee2cnFLrY"},"source":["## Word vectors"]},{"cell_type":"markdown","metadata":{"id":"quAhOWx-FLrZ"},"source":["The model we train learns word representations for each word in our vocabulary. A word represention is a vector of **dim** size.\n","\n","It is common in NLP to inspect the word vectors, as some properties of language often appear in the embedding structure.\n","\n","\n","We are going to load the word embeddings learned by our model, and inspect it.\n","Because our network was not trained for long, we are going for the simplest patterns, but if we let the network train longer, it learns more complex, semantic patterns."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Sny1RT7cFLrZ","executionInfo":{"status":"ok","timestamp":1617257119996,"user_tz":420,"elapsed":308,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"01200b2a-36d5-4763-828b-9bc0e58cb548"},"source":["# We help you load the matrix, as it is hidden within the Transformer structure.\n","E = trainer.model.encoder.embedding_layer.embedding.weight.cpu().detach().numpy()\n","\n","print(\"The embedding matrix has shape:\", E.shape)\n","print(\"The vocabulary has length:\", len(vocab))"],"execution_count":42,"outputs":[{"output_type":"stream","text":["The embedding matrix has shape: (10000, 160)\n","The vocabulary has length: 10000\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"MR4SX887FLra"},"source":["Pronouns serve very similar purposes, therefore we should expect the representation of \"he\" and \"she\" to be similar, and have cosine similarity.\n","\n","- **TODO**:  Find the cosine similarity between the vectors that represent words \"she\" and \"he\".\n","- **TODO**:  Find the cosine similarity between the vectors that represent words \"more\" and \"less\".\n","\n","We can contrast that with the cosine similarity to a random, non-related word, like \"ball\", or \"gorilla\".\n","- **TODO**: Compute the cosine similarity between \"she\" and \"ball\".\n","- **TODO**: Compute the cosine similarity between \"more\" and \"protest\".\n","\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"4wR1tDGMFLrb","executionInfo":{"status":"ok","timestamp":1617257412019,"user_tz":420,"elapsed":286,"user":{"displayName":"Matthew Chorlian","photoUrl":"","userId":"16119957957818108860"}},"outputId":"21628ffb-078e-4988-aca8-86fad3131aba"},"source":["def cosine_sim(v1, v2):\n","    # TODO: Implement the cosine similarity of 2 vectors. Careful: the words might not have unit norm.\n","    return np.dot(v1, v2) / (np.sqrt(np.sum(v1**2) + np.sum(v2**2)))\n","\n","for w1, w2 in [(\"she\", \"he\"), (\"more\", \"less\"), (\"she\", \"ball\"), (\"more\", \"gorilla\")]:\n","    w1_index = vocab.index('▁'+w1) # The index of the first  word in our vocabulary\n","    w2_index = vocab.index('▁'+w2) # The index of the second word in our vocabulary\n","    w1_vec = E[w1_index] # Get the embedding vector of the first  word\n","    w2_vec = E[w2_index] # Get the embedding vector of the second word\n","    \n","    print(w1,\" vs. \", w2, \"similarity:\",cosine_sim(w1_vec, w2_vec))\n","validate_to_array(lambda f,i: (f(*i),i), (cosine_sim,tuple(20*np.random.random((2,1000))-1)),'cosine_sim', root_folder) "],"execution_count":46,"outputs":[{"output_type":"stream","text":["she  vs.  he similarity: 0.83835024\n","more  vs.  less similarity: 0.4609797\n","she  vs.  ball similarity: 0.35393313\n","more  vs.  gorilla similarity: -0.047419112\n"],"name":"stdout"},{"output_type":"stream","text":["/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order, subok=True)\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"id":"5kJaau6wFLrc"},"source":["These effects are unfortunately small, as we have only trained the network on a few hours on a few thousand articles.\n","However, the same model trained for longer on more data exhibits many interesting semantic and syntactic patterns, such as:\n","\n","- Words vectors with high cosine similarity usually represent words that have semantic similarity (such as duck and pigeon)\n","- Analogies can occur, a famous case is that of: woman - man + king ≈ queen. Or france - paris + rome ≈ italy.\n","\n","- Looking at top-k similar words can help find synonyms.\n","\n","To read examples of more complex patterns that appear in word embedding spaces, read [this blog](https://explosion.ai/blog/sense2vec-with-spacy). To play with a live demo and try similarities on rich word embeddings, [go here.](https://explosion.ai/demos/sense2vec)"]},{"cell_type":"code","metadata":{"id":"lHxM8eDmFLrd"},"source":[""],"execution_count":null,"outputs":[]}]}