16_nlp_with_rnns_and_attention.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Shakesperian Text using a Character RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start exploring NLP by creating an RNN that takes in a sequence of characters and tries to predict the next character in the sequence. Check this [cool article](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) that contains examples (including Algebraic Geometry)!\n",
    "\n",
    "We'll train our model on Shakespeare's text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "shakespeare_url = 'https://homl.info/shakespeare'\n",
    "filepath = keras.utils.get_file('shakespeare.txt', shakespeare_url)\n",
    "with open(filepath) as f:\n",
    "    shakespeare_text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'First Citizen:\\n'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shakespeare_text[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To process the text, we'll encode each character as an integer by using Keras' ```Tokenizer``` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = keras.preprocessing.text.Tokenizer(char_level=True)\n",
    "tokenizer.fit_on_texts(shakespeare_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can give the tokenizer a string and get back its encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[20, 6, 9, 8, 3, 1, 19, 6, 3, 6, 36, 2, 10, 24, 11]]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.texts_to_sequences([shakespeare_text[:15]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "39"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_id = len(tokenizer.word_index)  # number of distinct characters\n",
    "max_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1115394"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total number of characters\n",
    "dataset_size = tokenizer.document_count\n",
    "dataset_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encode the text (subtracting 1 to get ranges from 0 to 38)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "[encoded] = np.array(tokenizer.texts_to_sequences([shakespeare_text])) - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([19,  5,  8, ..., 20, 26, 10])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Splitting a sequential Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to split into training, validation and test set. But how we do it with characters? We can't just shuffle all the characters in the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With time series, we gnerally split on time: for example if we might take years 2000 to 2012 for training, 2013 to 2015 for validation and 2016 to 2018 for test.\n",
    "\n",
    "In some other case we can split among other dimensions such as an industry type or country.\n",
    "\n",
    "Splitting accross time is safe, however it implicitly assumes that patterns learned in the past will still exist in the future. I.e. we assume the time series is *stationary*. To make sure the time series is sufficiently stationary, we can plot the model's error on the validation set across time: if the model performes better on the first part of the validation set than on the last part, then the time series might not be stationary enough. Training the model on a shorter time span might be better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the Shakespeare example  we'll take the first 90% of the text for the training set keeping the rest for validation and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_size = dataset_size *90 // 100\n",
    "dataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Chopping data into windows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use ```dataset.window``` to convert the one long instance we have into many smaller windows of text. Every instance in training will be a fairly short substring of the full text and the RNN will be unrolled over the length of the substrings. This is called **truncated backpropagation through time**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_steps = 100 # tune-able parameter\n",
    "window_length = n_steps + 1 # target will be the input shifted 1 character ahead\n",
    "dataset = dataset.window(window_length, shift=1, drop_remainder=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using ```shift=1``` we create overlapping windows: the first window will contain characters 0 to 100, the second 1 to 101 and so on...\n",
    "\n",
    "```drop_remainder``` ensures all windows are 101 characters long and we don't have to do any padding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```window()``` creates a dataset containing windows, each of which is also a dataset. It is a *nested dataset*. This is useful when we want to transform each window by calling dataset methods but it cannot be used directly for training. The model expects tensors, not datasets. \n",
    "\n",
    "The ```flat_map``` dataset converts a nested dataset into a *flat dataset* (i.e. without nesting). It also takes a function as an argument which allows you to transform each dataset _before_ flattening.\n",
    "\n",
    "For example passing ```lambda ds: ds.batch(2)``` to ```flat_map``` the dataset $\\{\\{1, 2\\}, \\{3,4, 5,6\\}\\}$ becomes $\\{[1,2], [3,4], [5,6]\\}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.flat_map(lambda window: window.batch(window_length))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By calling passing ```window_length``` to batch, we get a single tensor for each batch (all windows have that length). That is, the dataset contains windows of 101 characters each. Now we can shuffle the tensors and separate inputs from targets. See Figure 16-1 for a pictorial representation of the process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "dataset = dataset.shuffle(10000).batch(batch_size)\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[24  5  9 ...  0  8 13]\n",
      " [11 11 23 ...  1 12 26]\n",
      " [ 0 16  6 ... 22  0  4]\n",
      " ...\n",
      " [ 0 19  3 ...  9 17  0]\n",
      " [19 19  1 ...  7 13  8]\n",
      " [ 9 20  0 ...  0  4  8]], shape=(32, 100), dtype=int32)\n",
      "tf.Tensor(\n",
      "[[ 5  9 20 ...  8 13  2]\n",
      " [11 23 10 ... 12 26  0]\n",
      " [16  6  3 ...  0  4  9]\n",
      " ...\n",
      " [19  3  8 ... 17  0 15]\n",
      " [19  1  8 ... 13  8  1]\n",
      " [20  0  3 ...  4  8  1]], shape=(32, 100), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "for X, y in dataset.take(1):\n",
    "    print(X)\n",
    "    print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll do one-hot-encoding since there are fairly few features (only 39 characters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.map(lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bulding and Training the Char-RNN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll create a model with GRU units that are fed into a Time-distributed Dense layer of 39 unites (max_id), since we want to output character probabilities for each possible character"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Running the code below takes a good while..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer gru will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "Epoch 1/3\n",
      "31368/31368 [==============================] - 17566s 560ms/step - loss: 1.4672\n",
      "Epoch 2/3\n",
      "31368/31368 [==============================] - 17568s 560ms/step - loss: 1.3607\n",
      "Epoch 3/3\n",
      "31368/31368 [==============================] - 17605s 561ms/step - loss: 1.3395\n"
     ]
    }
   ],
   "source": [
    "# model = keras.models.Sequential([\n",
    "#     keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id],\n",
    "#                      dropout=0.2, recurrent_dropout=0.2),\n",
    "#     keras.layers.GRU(128, return_sequences=True,\n",
    "#                      dropout=0.2, recurrent_dropout=0.2),\n",
    "#     keras.layers.TimeDistributed(keras.layers.Dense(max_id, activation='softmax'))\n",
    "# ])\n",
    "# model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')\n",
    "# history = model.fit(dataset, epochs=3)\n",
    "# model.save('saved_models/16_nlp_with_rnns/shakespeare.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer gru will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.load_model('saved_models/16_nlp_with_rnns/shakespeare.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the Char-RNN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To pass some inputs to our model, we need to apply the same preprocessing we did for training it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(texts):\n",
    "    X = np.array(tokenizer.texts_to_sequences(texts)) - 1\n",
    "    return tf.one_hot(X, max_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2, 14,  1,  3, 10,  2,  1, 16,  4, 14]], dtype=int64)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new = preprocess(['How are yo'])\n",
    "y_pred = np.argmax(model.predict(X_new), axis=-1) + 1 # Account for the 1 we removed during preprocessing\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model generate one predicion for each character in the input. Since we're only interested in the prediction for the last character we simply pick up the last prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'u'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.sequences_to_texts(y_pred)[0][-1] # 1st sentence, last char"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating new text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a model, we could start using it by feeding in some text, generate a prediction, amend the text, feed it in, get a new prediction, etc...\n",
    "\n",
    "While this approach does work, it often leads to the same words being predicted over and over again. Instead, we can sample from the distribution of possible characters using ```tf.random.categorical()```, this will generate more interesting text.\n",
    "\n",
    "```categorical``` samples random class indices, given class log probabilities. We can also introduce a notion of *temperature*, which will control how wild our model will deviate from the logits. We divide the logits by the temperature, when temperature is close to 0 it will favour high probability characters, when it is close to 1, it gives characters an equal probability.\n",
    "\n",
    "We'll define the ```next_char``` function to do this for us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def next_char(text, model, temperature=1):\n",
    "    X_new = preprocess([text])\n",
    "    y_proba = model.predict(X_new)[0, -1:, :]\n",
    "    rescaled_logits = tf.math.log(y_proba) / temperature\n",
    "    char_id = tf.random.categorical(rescaled_logits, num_samples=1) + 1\n",
    "    return tokenizer.sequences_to_texts(char_id.numpy())[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a helper function that will generate the text and fill it in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def complete_text(text, model, n_chars=50, temperature=1):\n",
    "    for _ in range(n_chars):\n",
    "        text+= next_char(text, model, temperature)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "that she have a suitors of my hand,\n",
      "i shall be gone\n"
     ]
    }
   ],
   "source": [
    "print(complete_text('t', model, temperature=0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test signior words,\n",
      "you may be marry me all her to be \n"
     ]
    }
   ],
   "source": [
    "print(complete_text('test', model, temperature=0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "adams:\n",
      "that phes throught i shall be gentleman,\n",
      "and se\n"
     ]
    }
   ],
   "source": [
    "print(complete_text('adam', model, temperature=0.8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ursta pray, stand as father.\n",
      "\n",
      "hortensio:\n",
      "poor told \n"
     ]
    }
   ],
   "source": [
    "print(complete_text('u', model, temperature=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kine soil.\n",
      "wellbit quotkwurs, heis lowl gentlly-hou\n"
     ]
    }
   ],
   "source": [
    "print(complete_text('k', model, temperature=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get better text we can add more layers, train for longer, add regularization. The limitation of this model is that it is incapable of learning patterns longer than n_steps=100 characters. Stateful RNNs can help with that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stateful RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The RNNs discussed so far only used hidden states at each epoch. If we preserve the state after the end of each step and use that as the starting point for the next step, the RNN willl learn long-term patterns despite only backpropagating through short sequences. This is called a *stateful RNN*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It has a catch though: training a stateful RNN only makes sense if each input sequence in a batch starts exactly where the corresponding sequence in the previous batch left off.\n",
    "\n",
    "Instead of using shuffled data, we will use sequential, non-overlapping sequences. To do this we'll create the dataset using ```shift=n_steps``` and naturally we won't ```shuffle``` the data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batching is where things get a bit harder. If we were to use ```batch(32)``` then the first batch would contain windows 1 through 32 and the next batch would be 33 to 64, so the first window of each batch are not consecutive. One approach to this is to use 'batches' containing a single window"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])\n",
    "dataset = dataset.window(window_length, shift=n_steps, drop_remainder=True)\n",
    "dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "dataset = dataset.batch(1)\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))\n",
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another approach would be to divide the corpus into 32 texts of equal length, create one dataset of consecutive input sequences for them and then use ```tf.data.Dataset.zip(dataset).map(lambda *windows: tf.stack(windows))``` to create proper batches, where the nth input sequence in a batch start off exactly where the nth input sequence ended in the previous batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "encoded_parts = np.array_split(encoded[:train_size], batch_size)\n",
    "datasets = []\n",
    "for encoded_part in encoded_parts:\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(encoded_part)\n",
    "    dataset = dataset.window(window_length, shift=n_steps, drop_remainder=True)\n",
    "    dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "    datasets.append(dataset)\n",
    "dataset = tf.data.Dataset.zip(tuple(datasets)).map(lambda *windows: tf.stack(windows))\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))\n",
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to create the RNN we must set its ```stateful``` parameter as wel as specify the batch_input_shape for the first layer. We leave the second dimension unspecified since the inputs can have any length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer gru_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "WARNING:tensorflow:Layer gru_3 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, stateful=True,\n",
    "                     dropout=0.2, recurrent_dropout=0.2,\n",
    "                     batch_input_shape=[batch_size, None, max_id]),\n",
    "    keras.layers.GRU(128, return_sequences=True, stateful=True,\n",
    "                     dropout=0.2, recurrent_dropout=0.2),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(max_id, activation='softmax'))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to add a little callback to ensure we reset the states at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResetStatesCallback(keras.callbacks.Callback):\n",
    "    def on_epoch_begin(self, epoch, logs):\n",
    "        self.model.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "313/313 [==============================] - 46s 146ms/step - loss: 2.6218\n",
      "Epoch 2/10\n",
      "313/313 [==============================] - 46s 146ms/step - loss: 2.1815\n",
      "Epoch 3/10\n",
      "313/313 [==============================] - 46s 146ms/step - loss: 2.5545\n",
      "Epoch 4/10\n",
      "313/313 [==============================] - 46s 146ms/step - loss: 2.7257\n",
      "Epoch 5/10\n",
      "313/313 [==============================] - 46s 146ms/step - loss: 2.4261\n",
      "Epoch 6/10\n",
      "313/313 [==============================] - 46s 146ms/step - loss: 2.2789\n",
      "Epoch 7/10\n",
      "313/313 [==============================] - 46s 147ms/step - loss: 2.3096\n",
      "Epoch 8/10\n",
      "313/313 [==============================] - 46s 147ms/step - loss: 2.2237\n",
      "Epoch 9/10\n",
      "313/313 [==============================] - 46s 146ms/step - loss: 2.1581\n",
      "Epoch 10/10\n",
      "313/313 [==============================] - 46s 147ms/step - loss: 2.1108\n"
     ]
    }
   ],
   "source": [
    "# model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')\n",
    "# history = model.fit(dataset, epochs=10, callbacks=[ResetStatesCallback()])\n",
    "# model.save('saved_models/16_nlp_with_rnns/stateful_shake.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer gru_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "WARNING:tensorflow:Layer gru_3 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.load_model('saved_models/16_nlp_with_rnns/stateful_shake.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that this model is trained, it can only make predictions of the same size as the batch size used during training. To avoid this restriction, we can create a statelss model and copy the stateful model weights' to this model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "stateless_model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id]),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(max_id, activation='softmax'))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "stateless_model.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then use it for some predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "um! by hee's bicknto dissald you buck.\n",
      "\n",
      "stains:\n",
      "thi\n"
     ]
    }
   ],
   "source": [
    "print(complete_text('u', stateless_model))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "adame wheld cood tis him lose.\n",
      "\n",
      "plisler:\n",
      "it with, had \n"
     ]
    }
   ],
   "source": [
    "print(complete_text('adam', stateless_model, temperature=0.8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this we'll use the IMDB reviews dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 3s 0us/step\n"
     ]
    }
   ],
   "source": [
    "(X_train, y_train), (X_test, y_test) = keras.datasets.imdb.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the description, the dataset has already been transformed, with each character being encoded by its character frequency rank."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The integers 0, 1, 2 are special and represent padding token, Start of Sequense (SoS) token and unknown words, respectively. Let's write a function to decode the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
      "1646592/1641221 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "word_index = keras.datasets.imdb.get_word_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode(text):\n",
    "    id_to_word = {id_+3: word for word, id_ in word_index.items()}\n",
    "    for id_, token in enumerate(('<pad>', '<SoS>', '<unk>')):\n",
    "        id_to_word[id_] = token\n",
    "    return \" \".join(id_to_word[c] for c in text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"<SoS> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert redford's is an amazing actor and now the same being director norman's father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for retail and would recommend it to everyone to watch and the fly fishing was\""
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decode(X_train[0][:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a real project we'd have to take care of preprocessing ourselves. Kera's Tokenizer works well for English and most languages but there are exceptions (see pg.535), for example words like 'San Francisco' or '#ILoveDeepLearning'. \n",
    "\n",
    "One solution came in a paper by [Taku Kudu](https://arxiv.org/abs/1804.10959) that used unsupervised learning techniques to tokenize and detokenize text at subword level in a language independent way. Google's [SentencePie](https://github.com/google/sentencepiece) is an implementation of this.\n",
    "\n",
    "[TF.Text](https://medium.com/tensorflow/introducing-tf-text-438c8552bd5e) contains various tokenization strategies, including [WordPiece](https://arxiv.org/abs/1609.08144).\n",
    "\n",
    "We'll explore some TF operations to do preprocessing by loading the original IMDB reviews dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Dl Completed...: 0 url [00:00, ? url/s]\n",
      "Dl Completed...:   0%|          | 0/1 [00:00<?, ? url/s]\n",
      "Dl Size...: 0 MiB [00:00, ? MiB/s]\u001b[A\u001b[1mDownloading and preparing dataset imdb_reviews/plain_text/1.0.0 (download: Unknown size, generated: Unknown size, total: Unknown size) to C:\\Users\\gamin\\tensorflow_datasets\\imdb_reviews\\plain_text\\1.0.0...\u001b[0m\n",
      "Dl Size...: 0 MiB [00:21, ? MiB/s]\n",
      "Dl Completed...:   0%|          | 0/1 [00:21<?, ? url/s]\n"
     ]
    },
    {
     "ename": "ConnectionError",
     "evalue": "HTTPConnectionPool(host='ai.stanford.edu', port=80): Max retries exceeded with url: /~amaas/data/sentiment/aclImdb_v1.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000205CCD939A0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTimeoutError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\connection.py\u001b[0m in \u001b[0;36m_new_conn\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    158\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 159\u001b[1;33m             conn = connection.create_connection(\n\u001b[0m\u001b[0;32m    160\u001b[0m                 \u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_dns_host\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mport\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mextra_kw\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\util\\connection.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[1;34m(address, timeout, source_address, socket_options)\u001b[0m\n\u001b[0;32m     83\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0merr\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 84\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     85\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\util\\connection.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[1;34m(address, timeout, source_address, socket_options)\u001b[0m\n\u001b[0;32m     73\u001b[0m                 \u001b[0msock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbind\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msource_address\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 74\u001b[1;33m             \u001b[0msock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msa\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     75\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0msock\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTimeoutError\u001b[0m: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mNewConnectionError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[1;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[0;32m    669\u001b[0m             \u001b[1;31m# Make the request on the httplib connection object.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 670\u001b[1;33m             httplib_response = self._make_request(\n\u001b[0m\u001b[0;32m    671\u001b[0m                 \u001b[0mconn\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[1;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[0;32m    391\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 392\u001b[1;33m             \u001b[0mconn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mhttplib_request_kw\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    393\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\http\\client.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[0;32m   1254\u001b[0m         \u001b[1;34m\"\"\"Send a complete request to the server.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1255\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_send_request\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1256\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\http\\client.py\u001b[0m in \u001b[0;36m_send_request\u001b[1;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[0;32m   1300\u001b[0m             \u001b[0mbody\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_encode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'body'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1301\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mendheaders\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1302\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\http\\client.py\u001b[0m in \u001b[0;36mendheaders\u001b[1;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[0;32m   1249\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mCannotSendHeader\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1250\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_send_output\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmessage_body\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1251\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\http\\client.py\u001b[0m in \u001b[0;36m_send_output\u001b[1;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[0;32m   1009\u001b[0m         \u001b[1;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_buffer\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1010\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1011\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\http\\client.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, data)\u001b[0m\n\u001b[0;32m    949\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mauto_open\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 950\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    951\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\connection.py\u001b[0m in \u001b[0;36mconnect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    186\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 187\u001b[1;33m         \u001b[0mconn\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_new_conn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    188\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_prepare_conn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mconn\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\connection.py\u001b[0m in \u001b[0;36m_new_conn\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    170\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mSocketError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 171\u001b[1;33m             raise NewConnectionError(\n\u001b[0m\u001b[0;32m    172\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"Failed to establish a new connection: %s\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNewConnectionError\u001b[0m: <urllib3.connection.HTTPConnection object at 0x00000205CCD939A0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mMaxRetryError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\requests\\adapters.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[0;32m    438\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mchunked\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 439\u001b[1;33m                 resp = conn.urlopen(\n\u001b[0m\u001b[0;32m    440\u001b[0m                     \u001b[0mmethod\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[1;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[0;32m    725\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 726\u001b[1;33m             retries = retries.increment(\n\u001b[0m\u001b[0;32m    727\u001b[0m                 \u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merror\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0m_pool\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0m_stacktrace\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\urllib3\\util\\retry.py\u001b[0m in \u001b[0;36mincrement\u001b[1;34m(self, method, url, response, error, _pool, _stacktrace)\u001b[0m\n\u001b[0;32m    445\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mnew_retry\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_exhausted\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 446\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mMaxRetryError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_pool\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merror\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mResponseError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcause\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    447\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mMaxRetryError\u001b[0m: HTTPConnectionPool(host='ai.stanford.edu', port=80): Max retries exceeded with url: /~amaas/data/sentiment/aclImdb_v1.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000205CCD939A0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mConnectionError\u001b[0m                           Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-64-85d62fb1cc40>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_datasets\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mtfds\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mdatasets\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minfo\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtfds\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mload\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'imdb_reviews'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mas_supervised\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwith_info\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[0mtrain_size\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0minfo\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msplits\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'train'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnum_examples\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\core\\load.py\u001b[0m in \u001b[0;36mload\u001b[1;34m(name, split, data_dir, batch_size, shuffle_files, download, as_supervised, decoders, read_config, with_info, builder_kwargs, download_and_prepare_kwargs, as_dataset_kwargs, try_gcs)\u001b[0m\n\u001b[0;32m    342\u001b[0m   \u001b[1;32mif\u001b[0m \u001b[0mdownload\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    343\u001b[0m     \u001b[0mdownload_and_prepare_kwargs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdownload_and_prepare_kwargs\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 344\u001b[1;33m     \u001b[0mdbuilder\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdownload_and_prepare\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m**\u001b[0m\u001b[0mdownload_and_prepare_kwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    345\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    346\u001b[0m   \u001b[1;32mif\u001b[0m \u001b[0mas_dataset_kwargs\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\core\\dataset_builder.py\u001b[0m in \u001b[0;36mdownload_and_prepare\u001b[1;34m(self, download_dir, download_config)\u001b[0m\n\u001b[0;32m    400\u001b[0m           \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread_from_directory\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_data_dir\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    401\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 402\u001b[1;33m           self._download_and_prepare(\n\u001b[0m\u001b[0;32m    403\u001b[0m               \u001b[0mdl_manager\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdl_manager\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    404\u001b[0m               \u001b[0mdownload_config\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdownload_config\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\core\\dataset_builder.py\u001b[0m in \u001b[0;36m_download_and_prepare\u001b[1;34m(self, dl_manager, download_config)\u001b[0m\n\u001b[0;32m   1079\u001b[0m       \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1080\u001b[0m         \u001b[0moptional_pipeline_kwargs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1081\u001b[1;33m       split_generators = self._split_generators(  # pylint: disable=unexpected-keyword-arg\n\u001b[0m\u001b[0;32m   1082\u001b[0m           \u001b[0mdl_manager\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0moptional_pipeline_kwargs\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1083\u001b[0m       )\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\text\\imdb.py\u001b[0m in \u001b[0;36m_split_generators\u001b[1;34m(self, dl_manager)\u001b[0m\n\u001b[0;32m    120\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    121\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0m_split_generators\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdl_manager\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 122\u001b[1;33m     \u001b[0march_path\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdl_manager\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdownload\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_DOWNLOAD_URL\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    123\u001b[0m     \u001b[0marchive\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mlambda\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mdl_manager\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0miter_archive\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0march_path\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    124\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\core\\download\\download_manager.py\u001b[0m in \u001b[0;36mdownload\u001b[1;34m(self, url_or_urls)\u001b[0m\n\u001b[0;32m    608\u001b[0m     \u001b[1;31m# Add progress bar to follow the download state\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    609\u001b[0m     \u001b[1;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_downloader\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtqdm\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 610\u001b[1;33m       \u001b[1;32mreturn\u001b[0m \u001b[0m_map_promise\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_download\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl_or_urls\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    611\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    612\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0miter_archive\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mresource\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\core\\download\\download_manager.py\u001b[0m in \u001b[0;36m_map_promise\u001b[1;34m(map_fn, all_inputs)\u001b[0m\n\u001b[0;32m    703\u001b[0m   \u001b[1;34m\"\"\"Map the function into each element and resolve the promise.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    704\u001b[0m   \u001b[0mall_promises\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmap_structure\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmap_fn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mall_inputs\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# Apply the function\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 705\u001b[1;33m   \u001b[0mres\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmap_structure\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mlambda\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mall_promises\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# Wait promises\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    706\u001b[0m   \u001b[1;32mreturn\u001b[0m \u001b[0mres\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\nest.py\u001b[0m in \u001b[0;36mmap_structure\u001b[1;34m(func, *structure, **kwargs)\u001b[0m\n\u001b[0;32m    633\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    634\u001b[0m   return pack_sequence_as(\n\u001b[1;32m--> 635\u001b[1;33m       \u001b[0mstructure\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mentries\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    636\u001b[0m       expand_composites=expand_composites)\n\u001b[0;32m    637\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\nest.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[1;34m(.0)\u001b[0m\n\u001b[0;32m    633\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    634\u001b[0m   return pack_sequence_as(\n\u001b[1;32m--> 635\u001b[1;33m       \u001b[0mstructure\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mentries\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    636\u001b[0m       expand_composites=expand_composites)\n\u001b[0;32m    637\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\core\\download\\download_manager.py\u001b[0m in \u001b[0;36m<lambda>\u001b[1;34m(p)\u001b[0m\n\u001b[0;32m    703\u001b[0m   \u001b[1;34m\"\"\"Map the function into each element and resolve the promise.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    704\u001b[0m   \u001b[0mall_promises\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmap_structure\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmap_fn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mall_inputs\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# Apply the function\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 705\u001b[1;33m   \u001b[0mres\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmap_structure\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mlambda\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mall_promises\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# Wait promises\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    706\u001b[0m   \u001b[1;32mreturn\u001b[0m \u001b[0mres\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\promise\\promise.py\u001b[0m in \u001b[0;36mget\u001b[1;34m(self, timeout)\u001b[0m\n\u001b[0;32m    510\u001b[0m         \u001b[0mtarget\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_target\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    511\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_wait\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtimeout\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mDEFAULT_TIMEOUT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 512\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_target_settled_value\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_raise\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    513\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    514\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_target_settled_value\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0m_raise\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\promise\\promise.py\u001b[0m in \u001b[0;36m_target_settled_value\u001b[1;34m(self, _raise)\u001b[0m\n\u001b[0;32m    514\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_target_settled_value\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0m_raise\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    515\u001b[0m         \u001b[1;31m# type: (bool) -> Any\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 516\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_target\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_settled_value\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_raise\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    517\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    518\u001b[0m     \u001b[0m_value\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_reason\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_target_settled_value\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\promise\\promise.py\u001b[0m in \u001b[0;36m_settled_value\u001b[1;34m(self, _raise)\u001b[0m\n\u001b[0;32m    224\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0m_raise\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    225\u001b[0m                 \u001b[0mraise_val\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_fulfillment_handler0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 226\u001b[1;33m                 \u001b[0mreraise\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mraise_val\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mraise_val\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_traceback\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    227\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_fulfillment_handler0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    228\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\six.py\u001b[0m in \u001b[0;36mreraise\u001b[1;34m(tp, value, tb)\u001b[0m\n\u001b[0;32m    701\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__traceback__\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mtb\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    702\u001b[0m                 \u001b[1;32mraise\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 703\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    704\u001b[0m         \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    705\u001b[0m             \u001b[0mvalue\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\promise\\promise.py\u001b[0m in \u001b[0;36mhandle_future_result\u001b[1;34m(future)\u001b[0m\n\u001b[0;32m    842\u001b[0m         \u001b[1;31m# type: (Any) -> None\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    843\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 844\u001b[1;33m             \u001b[0mresolve\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfuture\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    845\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    846\u001b[0m             \u001b[0mtb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mexc_info\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\concurrent\\futures\\_base.py\u001b[0m in \u001b[0;36mresult\u001b[1;34m(self, timeout)\u001b[0m\n\u001b[0;32m    430\u001b[0m                 \u001b[1;32mraise\u001b[0m \u001b[0mCancelledError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    431\u001b[0m             \u001b[1;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_state\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0mFINISHED\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 432\u001b[1;33m                 \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__get_result\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    433\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    434\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_condition\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\concurrent\\futures\\_base.py\u001b[0m in \u001b[0;36m__get_result\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    386\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m__get_result\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    387\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_exception\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 388\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_exception\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    389\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    390\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\concurrent\\futures\\thread.py\u001b[0m in \u001b[0;36mrun\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     55\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     56\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 57\u001b[1;33m             \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     58\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mBaseException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     59\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfuture\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mset_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\core\\download\\downloader.py\u001b[0m in \u001b[0;36m_sync_download\u001b[1;34m(self, url, destination_path, verify)\u001b[0m\n\u001b[0;32m    193\u001b[0m       \u001b[1;32mpass\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    194\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 195\u001b[1;33m     \u001b[1;32mwith\u001b[0m \u001b[0m_open_url\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mverify\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mverify\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0miter_content\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    196\u001b[0m       \u001b[0mfname\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_get_filename\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    197\u001b[0m       \u001b[0mpath\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mos\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdestination_path\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\contextlib.py\u001b[0m in \u001b[0;36m__enter__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    111\u001b[0m         \u001b[1;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwds\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    112\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 113\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    114\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    115\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"generator didn't yield\"\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\tensorflow_datasets\\core\\download\\downloader.py\u001b[0m in \u001b[0;36m_open_with_requests\u001b[1;34m(url, **kwargs)\u001b[0m\n\u001b[0;32m    251\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0m_DRIVE_URL\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmatch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    252\u001b[0m       \u001b[0murl\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_get_drive_url\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msession\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 253\u001b[1;33m     \u001b[1;32mwith\u001b[0m \u001b[0msession\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstream\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    254\u001b[0m       \u001b[0m_assert_status\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    255\u001b[0m       \u001b[1;32myield\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0miter_content\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mchunk_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mio\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDEFAULT_BUFFER_SIZE\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\requests\\sessions.py\u001b[0m in \u001b[0;36mget\u001b[1;34m(self, url, **kwargs)\u001b[0m\n\u001b[0;32m    541\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    542\u001b[0m         \u001b[0mkwargs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msetdefault\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'allow_redirects'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 543\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'GET'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    544\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    545\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0moptions\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\requests\\sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[0;32m    528\u001b[0m         }\n\u001b[0;32m    529\u001b[0m         \u001b[0msend_kwargs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 530\u001b[1;33m         \u001b[0mresp\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    531\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    532\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\requests\\sessions.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, request, **kwargs)\u001b[0m\n\u001b[0;32m    641\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    642\u001b[0m         \u001b[1;31m# Send the request\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 643\u001b[1;33m         \u001b[0mr\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    644\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    645\u001b[0m         \u001b[1;31m# Total elapsed time of the request (approximately)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\requests\\adapters.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[0;32m    514\u001b[0m                 \u001b[1;32mraise\u001b[0m \u001b[0mSSLError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    515\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 516\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    517\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    518\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mClosedPoolError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mConnectionError\u001b[0m: HTTPConnectionPool(host='ai.stanford.edu', port=80): Max retries exceeded with url: /~amaas/data/sentiment/aclImdb_v1.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000205CCD939A0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))"
     ]
    }
   ],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "datasets, info = tfds.load('imdb_reviews', as_supervised=True, with_info=True)\n",
    "train_size = info.splits['train'].num_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(X_batch, y_batch):\n",
    "    X_batch = tf.strings.substr(X_batch, 0, 300)  # Truncate, keep only first 300 chars -> speed up training\n",
    "    X_batch = tf.strings.regex_replace(X_batch, b\"<br\\\\s*/?>\", b\" \")  # Replaces <br /> with spaces\n",
    "    X_batch = tf.strings.regex_replace(X_batch, b\"[^a-zA-Z']\", b\" \")  # remove anything other than characters or ''\n",
    "    X_batch = tf.strings.split(X_batch)\n",
    "    return X_batch.to_tensor(default_value=b\"<pad>\"), y_batch\n",
    "                                       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to construct the vocabulary by going through the dataset and keeping track of word occurences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "vocabulary = Counter()\n",
    "for X_batch, y_batch in datasets['train'].batch(32).map(preprocess):\n",
    "    for review in X_batch:\n",
    "        vocabulary.update(list(review.numpy()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocabulary.most_common()[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll truncate the vocabulary to the 10,000 most common words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = 10_000\n",
    "truncated_vocabulary = [word for word, count in  vocabulary.most_common()[:vocab_size]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then replace each word with it's ID from the vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = tf.constant(truncated_vocabulary)\n",
    "word_ids = tf.range(len(truncated_vocabulary), dtype=tf.int64)\n",
    "vocab_init = tf.lookup.KeyValueTensorInitializer(words, word_ids)\n",
    "num_oov_buckets = 1000\n",
    "table = tf.lookup.StaticVocabularyTable(vocab_init, num_oov_buckets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This table will serve as our reference for a given input. In the example below, faaaantastic is an unseen word so it falls into a newly created bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "table.lookup(tf.constant([b\"This move was faaaaantastic\".split()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip:** check out TF-Transform, in particular ```tft.compute_and_apply_vocabulary```, it goes through the dataset to find vocabulary and generate TF operations required to encode each word with the vocab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to create the final training set we use the previously defined ```preprocess``` and encode them with the vocabulary we just built"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_words(X_batch, y_batch):\n",
    "    return table.lookup(X_batch), y_batch\n",
    "\n",
    "train_set = datasets['train'].batch(32).map(preprocess)\n",
    "train_set = train_set.map(encode_words).prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for X, y in train_set.take(1):\n",
    "    print(X)\n",
    "    print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_size = 128\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size,\n",
    "                           input_shape=[None]),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.GRU(128),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\",\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first layer we use is an Embedding layer that will convert word IDs into embeddings. It's input matrix has one row per id (vocab_size + num_oov_buckets) and a column per embedding dimension (used 128 but this is a hyperparameter). The embedding layer outputs a 3D tensor of shape **[batch_size, time_steps, embedding_size]** which feeds in perfectly into the rest of the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Masking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model will have to learn to ignore padding tokens (which have index zero). We can set the Embedding layer's ```mask_zero``` parameter to True to make it pad tokens whose ID is 0. (note: in this case, the pad token has index zero because it is the most common token, in general is a good idea to map padding to 0). With this, padding is ignored by the layers downstream!\n",
    "\n",
    "With this parameter set to true, the Embedding layer creates a *mask tensor* equal to K.not_equal(inputs, 0); a boolean tensor with entries equal to False where inputs==0, True otherwise. This mask is propagated downstream as long as the time dimension is preserved. For example, the first GRU layer outputs sequences so it preserves the mask. The second doesn't so the mask is not transmitted to the dense layer. Each layer behaves differently when encountering a masked time step, but in general it simple ignored masked time steps.\n",
    "\n",
    "For example, recurrent layers simply copies the output from previous time step when it encounters a mask. If the mask propagates all the way to the output (e.g. a vec-to-seq or seq-to-seq architecture), then the masked time steps will not contribute to the loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: LSTMs and GRUs are optimized for GPUs based on Nvidia's cuDNN library, however this implementation does not support masking. The optimized implementation requires you to use the default values for several hyperparameters: activation, recurrent_activation, etc.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All layers that receive the mask must support masking (else an exception is raised). All recurrent layers, as well as TimeDistributed and a few other layers support this. \n",
    "\n",
    "See pg 539 for more info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Masking works best for simple sequential models, it doesn't always work for more complex models (e.g. if you need to mix a Conv1D layer with recurrent layers). In such cases you will need to explicitly compute the mask. \n",
    "\n",
    "Below an example that is the same model defined as above, but done using the Functional API and handling masks manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "root_logdir = os.path.join(os.curdir, 'my_logs')\n",
    "\n",
    "def get_run_logdir():\n",
    "    import time\n",
    "    run_id = time.strftime(\"run_%Y_%m_%d-%H_%M_%S\")\n",
    "    return os.path.join(root_logdir, run_id)\n",
    "\n",
    "callbacks = [keras.callbacks.TensorBoard(get_run_logdir())]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = keras.backend\n",
    "inputs = keras.layers.Input(shape=[None])\n",
    "mask = keras.layers.Lambda(lambda inputs: K.not_equal(inputs, 0))(inputs)\n",
    "z = keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size)(inputs)\n",
    "z = keras.layers.GRU(128, return_sequences=True)(z, mask=mask)\n",
    "z = keras.layers.GRU(128)(z, mask=mask)\n",
    "outputs = keras.layers.Dense(1, activation='sigmoid')(z)\n",
    "\n",
    "model = keras.models.Model(inputs=[inputs], outputs=[outputs])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set, epochs=5, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_set = datasets['test'].batch(32).map(preprocess)\n",
    "test_set = test_set.map(encode_words).prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's generate predictions for 1 batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for batch_X, batch_y in test_set.take(1):\n",
    "    y_preds = np.round(model.predict(test_set.take(1))).flatten()\n",
    "    print('True values\\n', batch_y)\n",
    "    print('\\nPredictions\\n', y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not too shabby, now let's look at Pretrained Embeddings!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reusing Pretrained Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using pre-trained models is pretty easy with tensorflow_hub. For this example we'll use ```nnlm-en-dim50``` sentence embedding module version 1. \n",
    "\n",
    "This module is a sequence encoder, taking strings as inputs and encoding each as a single vector (a 50dim vector). It parses the strings and embeds each word using an embedding matrix trained on the Google News 7B corpus, it then compute the mean of all the word embeddings and the result is the sentence embedding.\n",
    "\n",
    "By default ```hub.KerasLayer``` is not trainable but we can set ```training=True``` to make it trainable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import tensorflow_hub as hub\n",
    "\n",
    "model = keras.Sequential([\n",
    "    hub.KerasLayer(\"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\",\n",
    "                   dtype=tf.string, input_shape=[], output_shape=[50]),\n",
    "    keras.layers.Dense(128, activation='relu'),\n",
    "    keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reload the dataset before training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, with_info=True)\n",
    "train_size = info.splits[\"train\"].num_examples\n",
    "batch_size = 32\n",
    "train_set = datasets[\"train\"].repeat().batch(batch_size).prefetch(1)\n",
    "history = model.fit(train_set, steps_per_epoch=train_size // batch_size, epochs=5,\n",
    "                    callbacks=[keras.callbacks.TensorBoard(get_run_logdir())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An Encoder-Decoder Network for Neural Machine Translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll explore text translation by building a simple [neural machine translation model](https://arxiv.org/abs/1409.3215). Picture and explanation at page 543 are the best way to understand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, we'll be inputting a sequence to the encoder and (during training) the correct translation (shifted back one step) to the decoder. We also pass the sequence to the encoder reversed, so that the begginning of the sentence is fed last to the encoder.\n",
    "\n",
    "At each time step the decoder output scores which are then turned to probabilities by the softmax function and the word with the highest probability is output. Training can be done using the ```sparse_caregorical_crossentropy``` loss.\n",
    "\n",
    "At inference, we don't have the target output, so at each time step, we input the output word of the previous time step.\n",
    "\n",
    "The book mentions a couple of extra notes before implementing the model as well as the following:\n",
    "- When the output vocabulary is large, outputting probabilities for every word is very slow (e.b. consider a vocabulary of 50,000 words. Computing softmax over this vector is computationally intensive.). A solution to this is using a *sampled softmax*, which picks the logits of the correct word plus a random sample of incorrect words and computs an approximation based on these logits. Use this for training and the usual softmax at inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See book for full code and explanation. No dataset was given for this so I'm skipping this bit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bidirectional RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recurrent layers only look at past and present inputs before generating its output. It doesn't look ahead to generate an output. This makes sense for time series, but for NLP it makes sense to look a head before encoding a given word. For example the word queen in the following sentences is defined by what comes after: \"Queen of the UK\", \"the queen of hearts\", \"the queen bee\".\n",
    "\n",
    "Bidirectional layers are simply two recurrent layers running on the same inputs. One reading from left to right and the other from right to left. Their outpus are combined at each time step (tipycally concatenating). We can wrap our recurrent cells with ```keras.layers.Bidirectional``` to make them bidirectional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.layers.Bidirectional(keras.layers.GRU(10, return_sequences=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the layer above has 10 units, this layer has to GRUs, so its outputs have 20 values per time step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Beam Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Page 547 explains Beam Search, which is a way to expand the space of possible output sentences. Best to read this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Mechanisms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attention is a way to tell the machine to focus on a specific word at a particular time step. To construct it, we will take the existing encoder-decoder architecture and make a slight modification.\n",
    "\n",
    "Instead of sending the encoder final hidden state to the decoder, we send all the outputs. To focus the decoder on the next word to be predicted, we give each of these outputs a weight, the one with the highest weight will be the focused target. To compute these weights we add an *alignment model* (or *attention layer*). It is composed of a time distributed dense layer which computes how well each output is aligned to the decoder's previous hidden state, an energy score for each encoder output. It is followed by a softmax which calculates the weights for each encoder output. All the weights add up to 1. (figure page 550)\n",
    "\n",
    "This mechanism is called [*Bahdanau attention*](https://arxiv.org/abs/1409.0473) or *concatenative (additive) attention*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[*Luong attention*](https://arxiv.org/abs/1508.04025) or *multiplicative attention* was proposed later and it modifies how the similarity between decoder previous hidden state and encoder outputs is calculated. The authors propose the dot product of the two vectors to be calculated, this often works as a good similarity measure and is quicker to compute. The dot product scores also go through a softmax layer to calculate the final weights.\n",
    "\n",
    "They also propose using the decoder hidden state at the current time step ($\\textbf{h}_t$ rather than previous time step $\\textbf{h}_{t-1}$) then use the output of the attention mechanism ($\\widetilde{h}_t$) to compute the decoder's predictions (rather than to compute decoder's current hidden state. \n",
    "\n",
    "Finally, they propose a variation on the dot product, where the encoder outputs first go through a linear transformation (a time distributed Dense layer without bias term) before the dot products are computed. This is called the general dot product approach. They found that the dot product variants performed better than concatenative attention. \n",
    "\n",
    "Below equations for each mechanism"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ \\widetilde{h}_{(t)} = \\sum_i\\alpha_{(t,i)} \\mathbf{y}_{(t)}$$\n",
    "\n",
    "with\n",
    "\n",
    "$$ \\alpha_{(t, i)} = \\frac{\\exp(e_{(t,i)})}{\\sum_{i'}\\exp(e_{(t,i')})}$$\n",
    "\n",
    "and \n",
    "\n",
    "$$ e_{(t,i)} = \\begin{cases}\n",
    "                \\textbf{h}_{(t)}^{\\intercal}\\textbf{y}_{(i)} & & dot  \\\\\n",
    "                \\textbf{h}_{(t)}^{\\intercal}\\mathbf{W}\\textbf{y}_{(i)} & & general \\\\\n",
    "                \\textbf{v}{(t)}^{\\intercal}\\text{tanh}(\\textbf{W}[\\textbf{h}_{(t)};\\textbf{y}_{(i)}]) & & concat\n",
    "                \\end{cases} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add attention mechanisms to an Encoder-Decoder model use Tensorflow Addons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "import tensorflow_addons as tfa\n",
    "\n",
    "attention_mechanism = tfa.seq2seq.attention_wrapper.LuongAttention(\n",
    "    n_units, encoder_outputs, memory_sequence_length=encoder_sequence_length)\n",
    "\n",
    "attention_decoder_cell = tfa.seq2seq.attention_wrapper.AttentionWrapper(\n",
    "    decoder_cell, attention_mechanism, attention_layer_size=n_units)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visual Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The attention mechanism was also adopted for generating image captions. Page 553 explains a litte bit of what happens there along with a nice pictorial representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explainability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An added bonus of attention is that they can help us understand what led the model to produce its output. [This paper from 2016](https://arxiv.org/pdf/1602.04938.pdf) suggest an approach by learning an interpretable mode locally around a classifier's prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention is all you need: the Transformer architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A [groundbreaking paper from 2017](https://arxiv.org/abs/1706.03762) laid the foundations to a new type of architecture called the *Transformer*, which obtained incredible results without recurrent or convolutional layers. Page 555 shows the transformer architecture, along with its explanation. I won't add it here as the picture draws the explanation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Positional Encodings\n",
    "\n",
    "One of the novel features of the Transformer architecture is the use of Positional Encodings. These are dense vectors that encode the position of a word within a sentence. The authors used a fixed positional encoding, its advantage over a learned positional encoding being that it can be extended to arbitrarily long sentences. \n",
    "\n",
    "The positional encoding $P_{p,i}$ for the *i*th component of the encoding for the word located at the *p*th position in the sentence is given by\n",
    "\n",
    "$$ \\text{PE}_{p,i} = \\begin{cases} \\sin(p/10000^{i/d}) & \\text{if i is even} \\\\ \\cos(p/10000^{(i-1)/d}) & \\text{if i is odd} \\end{cases}$$\n",
    "\n",
    "Explanation along with image on page 557"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can define our own class for Positional Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(keras.layers.Layer):\n",
    "    def __init__(self, max_steps, max_dims, dtype=tf.float32, **kwargs):\n",
    "        super().__init__(dtype=dtype, **kwargs)\n",
    "        if max_dims % 2 == 1: max_dims += 1 # max_dims must be even\n",
    "        p, i = np.meshgrid(np.arange(max_steps), np.arange(max_dims // 2))\n",
    "        pos_emb = np.empty((1, max_steps, max_dims))  # Adding an extra dimension to use broadcasting rules on call\n",
    "        pos_emb[0, :, ::2] = np.sin(p / 10000**(2 * i / max_dims)).T\n",
    "        pos_emb[0, :, 1::2] = np.cos(p / 10000**(2 * i / max_dims)).T\n",
    "        self.positional_encoding = tf.constant(pos_emb.astype(self.dtype))\n",
    "\n",
    "    def call(self, inputs):\n",
    "        shape = tf.shape(inputs)\n",
    "        return inputs + self.positional_encoding[:, :shape[-2], :shape[-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then create the first layers of the Transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_size = 512\n",
    "max_steps = 500\n",
    "vocab_size = 10000\n",
    "\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "embeddings = keras.layers.Embedding(vocab_size, embed_size)\n",
    "encoder_embeddings = embeddings(encoder_inputs)\n",
    "decoder_embeddings = embeddings(decoder_inputs)\n",
    "positional_encoding = PositionalEncoding(max_steps, max_dims=embed_size)\n",
    "encoder_in = positional_encoding(encoder_embeddings)\n",
    "decoder_in = positional_encoding(decoder_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-head attention\n",
    "\n",
    "Let's start by looking at the *Scaled Dot-Product Attention* layer. Suppose the decoder analyzed the sentence \"*They played chess*\" and it was able to understand that the word \"They\" is the subject and \"played\" is the verb. Suppose also that the decoder has already translated They and thinks it should translate the verb next, to do this, it needs to fetch the verb from the input sentence. This works like a dictionary retrieval where the decoder has {'subject':'They', 'verb':'played', ...} and the decoder has a corresponding dictionary. \n",
    "\n",
    "The model, however, doesn't have a discrete representation of word tokens to represent the keys. It has vectorized representations of these, so the key (also called *query*) will not perfectly match any key in the dictionary. To solve this, we compute a similarity measure between the query and each key in the dictionary, then use softmax to convert the similarities to weights that add up to 1. The key most similar to the query will have weight closest to 1. The model then computes a weighted sum of the values and the final score will be close to the representation of the word 'played' in this example. \n",
    "\n",
    "The similarity measure used is the dot product, like in Luong attention. The equation is show below\n",
    "\n",
    "$$ \\text{Attention}(\\mathbf{Q, K, V}) = \\text{softmax}\\Big( \\frac{\\mathbf{QK}^\\intercal}{\\sqrt{d_{keys}}}\\Big)\\mathbf{V} $$\n",
    "\n",
    "Where \n",
    "- $\\mathbf{Q}$ is a matrix with a row per query of shape \\[$n_{queries}, d_{keys}$\\] where $d_{values}$ is the number of dimensions of each query and each key\n",
    "- $\\mathbf{K}$ is a matrix with one row per key with shape \\[$n_{keys}, d_{keys}$\\]\n",
    "- $\\mathbf{V}$ os a matrix with one row per value of shape \\[$n_{keys}, d_{values}$\\] where $d_{values}$ is the number of dimensions of each value\n",
    "- The shape of $\\mathbf{QK}^\\intercal$ is [$n_{queries}, n_{keys}$], with one similarity score per query/key pair. After going through the softmax, the shape is the same but all rows sum up to 1. The final output has shape [$n_{queries}, d_{values}$]. One row per query with each row representing the query result (a weighted sum of the values).\n",
    "- The scaling factor adjusts the similarity scores to avoid saturating the softmax function and unstable gradients\n",
    "- A Masked version of the Multi-Head Attention works by adding a very large negative value (-inf) to the corresponding similarity score. This is useful to eliminate words which we haven't seen yet in the decoder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multi-head attention in the encoder is applied to every input sentence in the batch, with $\\textbf{Q}, \\textbf{K}$ and $\\textbf{V}$ all equal to the list of words in the input sequence (each word in the sentence is compared to every word in the same sentence, including itself). \n",
    "\n",
    "A similar comparison happens in the decoder, with $\\textbf{Q}, \\textbf{K}$ and $\\textbf{V}$ all equal to the list of words in the target sequence, but this time using a mask to prevent any word to prevent comparing itself to words located after it (at inference, the decoder only has access to words it already output, not to future words). \n",
    "\n",
    "In the upper attention layer of the decoder, the keys $\\textbf{K}$ and values $\\textbf{V}$ are the list of word encodings produced by the encoder, and the queries $\\textbf{Q}$ are the list of word encodings produced by the decoder. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simplified version of the Transformer, without Feed Forward blocks and skip connections can be implemented as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Z = encoder_in\n",
    "for N in range(6):\n",
    "    Z = keras.layers.Attention(use_scale=True)([Z, Z])\n",
    "\n",
    "encoder_outputs = Z\n",
    "Z = decoder_in\n",
    "for N in range(6):\n",
    "    Z = keras.layers.Attention(use_scale=True, causal=True)([Z, Z])\n",
    "    Z = keras.layers.Attention(use_scale=True)([Z, encoder_outputs])\n",
    "\n",
    "outputs = keras.layers.TimeDistributed(\n",
    "    keras.layers.Dense(vocab_size, activation=\"softmax\"))(Z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`use_scale=True` tells the model to learn the similarity scores, instead of using the standard factor $\\sqrt{d_{keys}}$. `causal=True` ensures the second attention layer only attends to previous output tokens, not future ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Multi-headed attention layer is simply a bunch of Scaled Dot-Product Attention layers preceeded by a linear transformation of Keys, Values and Queries. The intuition behind this?\n",
    "\n",
    "Consider the word \"played\", as discussed earlier. The encoder identified it is a verb, but the word representation also includes position in the text and it probably includes other features useful for translation, such as tense. Using a single scaled dot-product attention, means we are querying all of these characteristics in a single shot. This is why the multi-headed attention layer applies multiple linear transformations of query, values and keys. \n",
    "\n",
    "This allows the model to apply many differnet projections of the word representations into difference subspaces, each focusing on a subset of the word's characteristics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recent Innovations in Language Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the book for a great descriptions and links to groundbreaking papers on ELMo, GPT, BERT and more."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}