00_Generating_Text_with_RNNs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "00-Generating-Text-with-RNNs.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.5"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ovpZyIhNIgoq"
      },
      "source": [
        "___\n",
        "\n",
        "<a href='http://www.pieriandata.com'> <img src='../Pierian_Data_Logo.png' /></a>\n",
        "___\n",
        "# Text Generation with Neural Networks\n",
        "\n",
        "In this notebook we will create a network that can generate text, here we show it being done character by character. Very awesome write up on this here: http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
        "\n",
        "We organized the process into \"steps\" so you can easily follow along with your own data sets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "00mwlQwloO5_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# GOOGLE COLLAB USERS ONLY\n",
        "%tensorflow_version 2.x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WBd69MDEm4rF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sDbPtshnm4rL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# IGNORE THE CONTENT OF THIS CELL\n",
        "# tf.compat.v1.disable_eager_execution()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kycWuRI9oaSP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.__version__"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "apj1Chkdm4rS",
        "colab_type": "text"
      },
      "source": [
        "## Step 1: The Data\n",
        "\n",
        "You can grab any free text you want from here: https://www.gutenberg.org/\n",
        "\n",
        "We'll choose all of shakespeare's works (which we have already downloaded for you), mainly for two reasons:\n",
        "\n",
        "1. Its a large corpus of text, its usually recommended you have at least a source of 1 million characters total to get realistic text generation.\n",
        "\n",
        "2. It has a very distinctive style. Since the text data uses old style english and is formatted in the style of a stage play, it will be very obvious to us if the model is able to reproduce similar results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pD_55cOxLkAb",
        "colab": {}
      },
      "source": [
        "path_to_file = 'shakespeare.txt'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "aavnuByVymwK",
        "colab": {}
      },
      "source": [
        "text = open(path_to_file, 'r').read()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Duhg9NrUymwO",
        "colab": {}
      },
      "source": [
        "print(text[:500])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXUmR627m4rd",
        "colab_type": "text"
      },
      "source": [
        "### Understanding unique characters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IlCgQBRVymwR",
        "colab": {}
      },
      "source": [
        "# The unique characters in the file\n",
        "vocab = sorted(set(text))\n",
        "print(vocab)\n",
        "len(vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rNnrKn_lL-IJ"
      },
      "source": [
        "## Step 2: Text Processing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LFjSVAlWzf-N"
      },
      "source": [
        "### Text Vectorization\n",
        "\n",
        "We know a neural network can't take in the raw string data, we need to assign numbers to each character. Let's create two dictionaries that can go from numeric index to character and character to numeric index."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IalZLbvOzf-F",
        "colab": {}
      },
      "source": [
        "char_to_ind = {u:i for i, u in enumerate(vocab)}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fmmP5iCwm4rp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "char_to_ind"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "30ZYaWAOm4rt",
        "colab": {}
      },
      "source": [
        "ind_to_char = np.array(vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_6JPOWwJm4rz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ind_to_char"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3fhOqV0lm4r2",
        "colab": {}
      },
      "source": [
        "encoded_text = np.array([char_to_ind[c] for c in text])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "axOX7rFom4r5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "encoded_text"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tZfqhkYCymwX"
      },
      "source": [
        "We now have a mapping we can use to go back and forth from characters to numerics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tFs1Uza-m4r9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sample = text[:20]\n",
        "sample"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gIqUCK5Am4sB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "encoded_text[:20]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bbmsf23Bymwe"
      },
      "source": [
        "## Step 3: Creating Batches\n",
        "\n",
        "Overall what we are trying to achieve is to have the model predict the next highest probability character given a historical sequence of characters. Its up to us (the user) to choose how long that historic sequence. Too short a sequence and we don't have enough information (e.g. given the letter \"a\" , what is the next character) , too long a sequence and training will take too long and most likely overfit to sequence characters that are irrelevant to characters farther out. While there is no correct sequence length choice, you should consider the text itself, how long normal phrases are in it, and a reasonable idea of what characters/words are relevant to each other."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pAvUYFk7m4sF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(text[:500])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D45OYgOfm4sJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "line = \"From fairest creatures we desire increase\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7dKiEVN8m4sL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "len(line)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "olX67f6-m4sP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "part_stanza = \"\"\"From fairest creatures we desire increase,\n",
        "  That thereby beauty's rose might never die,\n",
        "  But as the riper should by time decease,\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qal7MQnqm4sQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "len(part_stanza)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hgsVvVxnymwf"
      },
      "source": [
        "### Training Sequences\n",
        "\n",
        "The actual text data will be the text sequence shifted one character forward. For example:\n",
        "\n",
        "Sequence In: \"Hello my nam\"\n",
        "Sequence Out: \"ello my name\"\n",
        "\n",
        "\n",
        "We can use the `tf.data.Dataset.from_tensor_slices` function to convert a text vector into a stream of character indices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "0UHJDA39zf-O",
        "colab": {}
      },
      "source": [
        "seq_len = 120"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7VRSK4cOm4sZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "total_num_seq = len(text)//(seq_len+1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xtW0jbbvm4sc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "total_num_seq"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ciatnowvm4se",
        "colab": {}
      },
      "source": [
        "# Create Training Sequences\n",
        "char_dataset = tf.data.Dataset.from_tensor_slices(encoded_text)\n",
        "\n",
        "for i in char_dataset.take(500):\n",
        "     print(ind_to_char[i.numpy()])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-ZSYAcQV8OGP"
      },
      "source": [
        "The **batch** method converts these individual character calls into sequences we can feed in as a batch. We use seq_len+1 because of zero indexing. Here is what drop_remainder means:\n",
        "\n",
        "drop_remainder: (Optional.) A `tf.bool` scalar `tf.Tensor`, representing\n",
        "    whether the last batch should be dropped in the case it has fewer than\n",
        "    `batch_size` elements; the default behavior is not to drop the smaller\n",
        "    batch.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "l4hkDU3i7ozi",
        "colab": {}
      },
      "source": [
        "sequences = char_dataset.batch(seq_len+1, drop_remainder=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "UbLcIPBj_mWZ"
      },
      "source": [
        "Now that we have our sequences, we will perform the following steps for each one to create our target text sequences:\n",
        "\n",
        "1. Grab the input text sequence\n",
        "2. Assign the target text sequence as the input text sequence shifted by one step forward\n",
        "3. Group them together as a tuple"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "9NGu-FkO_kYU",
        "colab": {}
      },
      "source": [
        "def create_seq_targets(seq):\n",
        "    input_txt = seq[:-1]\n",
        "    target_txt = seq[1:]\n",
        "    return input_txt, target_txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "HszljTg8m4so",
        "colab": {}
      },
      "source": [
        "dataset = sequences.map(create_seq_targets)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JkPa7AMrm4sq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for input_txt, target_txt in  dataset.take(1):\n",
        "    print(input_txt.numpy())\n",
        "    print(''.join(ind_to_char[input_txt.numpy()]))\n",
        "    print('\\n')\n",
        "    print(target_txt.numpy())\n",
        "    # There is an extra whitespace!\n",
        "    print(''.join(ind_to_char[target_txt.numpy()]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MJdfPmdqzf-R"
      },
      "source": [
        "### Generating training batches\n",
        "\n",
        "Now that we have the actual sequences, we will create the batches, we want to shuffle these sequences into a random order, so the model doesn't overfit to any section of the text, but can instead generate characters given any seed text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "p2pGotuNzf-S",
        "colab": {}
      },
      "source": [
        "# Batch size\n",
        "batch_size = 128\n",
        "\n",
        "# Buffer size to shuffle the dataset so it doesn't attempt to shuffle\n",
        "# the entire sequence in memory. Instead, it maintains a buffer in which it shuffles elements\n",
        "buffer_size = 10000\n",
        "\n",
        "dataset = dataset.shuffle(buffer_size).batch(batch_size, drop_remainder=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gmcCALymm4su",
        "colab": {}
      },
      "source": [
        "dataset"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "r6oUuElIMgVx"
      },
      "source": [
        "## Step 4: Creating the Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "m8gPwEjRzf-Z"
      },
      "source": [
        "We will use an LSTM based model with a few extra features, including an embedding layer to start off with and **two** LSTM layers. We based this model architecture off the [DeepMoji](https://deepmoji.mit.edu/) and the original source code can be found [here](https://github.com/bfelbo/DeepMoji).\n",
        "\n",
        "The embedding layer will serve as the input layer, which essentially creates a lookup table that maps the numbers indices of each character to a vector with \"embedding dim\" number of dimensions. As you can imagine, the larger this embedding size, the more complex the training. This is similar to the idea behind word2vec, where words are mapped to some n-dimensional space. Embedding before feeding straight into the LSTM usually leads to more realisitic results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "zHT8cLh7EAsg",
        "colab": {}
      },
      "source": [
        "# Length of the vocabulary in chars\n",
        "vocab_size = len(vocab)\n",
        "\n",
        "# The embedding dimension\n",
        "embed_dim = 64\n",
        "\n",
        "# Number of RNN units\n",
        "rnn_neurons = 1026"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Atb060h5m4s0",
        "colab_type": "text"
      },
      "source": [
        "Now let's create a function that easily adapts to different variables as shown above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YeRlEXgym4s1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import LSTM,Dense,Embedding,Dropout,GRU"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FcMbIy-xj-w-",
        "colab_type": "text"
      },
      "source": [
        "### Setting up Loss Function\n",
        "\n",
        "For our loss we will use sparse categorical crossentropy, which we can import from Keras. We will also set this as logits=True"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VoFVGKlNkJfW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.losses import sparse_categorical_crossentropy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sblCzZoslZKH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "help(sparse_categorical_crossentropy)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y5N4Qxbij5gY",
        "colab_type": "text"
      },
      "source": [
        "https://datascience.stackexchange.com/questions/41921/sparse-categorical-crossentropy-vs-categorical-crossentropy-keras-accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FrOOK61Olm1C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sparse_cat_loss(y_true,y_pred):\n",
        "  return sparse_categorical_crossentropy(y_true, y_pred, from_logits=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MtCrdfzEI2N0",
        "colab": {}
      },
      "source": [
        "def create_model(vocab_size, embed_dim, rnn_neurons, batch_size):\n",
        "    model = Sequential()\n",
        "    model.add(Embedding(vocab_size, embed_dim,batch_input_shape=[batch_size, None]))\n",
        "    model.add(GRU(rnn_neurons,return_sequences=True,stateful=True,recurrent_initializer='glorot_uniform'))\n",
        "    # Final Dense Layer to Predict\n",
        "    model.add(Dense(vocab_size))\n",
        "    model.compile(optimizer='adam', loss=sparse_cat_loss) \n",
        "    return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wwsrpOik5zhv",
        "colab": {}
      },
      "source": [
        "model = create_model(\n",
        "  vocab_size = vocab_size,\n",
        "  embed_dim=embed_dim,\n",
        "  rnn_neurons=rnn_neurons,\n",
        "  batch_size=batch_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "liXuTFYMm4s6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LJL0Q0YPY6Ee"
      },
      "source": [
        "## Step 5: Training the model\n",
        "\n",
        "Let's make sure everything is ok with our model before we spend too much time training! Let's pass in a batch to confirm the model currently predicts random characters without any training.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A4ygvfHn-wan",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for input_example_batch, target_example_batch in dataset.take(1):\n",
        "\n",
        "  # Predict off some random batch\n",
        "  example_batch_predictions = model(input_example_batch)\n",
        "\n",
        "  # Display the dimensions of the predictions\n",
        "  print(example_batch_predictions.shape, \" <=== (batch_size, sequence_length, vocab_size)\")\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5ld8z3LPBAuv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "example_batch_predictions"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_achqjT-BGyY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xWrPFk2nBJX4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sampled_indices"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wi80PQVtBLqj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Reformat to not be a lists of lists\n",
        "sampled_indices = tf.squeeze(sampled_indices,axis=-1).numpy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4qYkIg00-wjq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sampled_indices"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H9-P_XqQ_7wY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Given the input seq: \\n\")\n",
        "print(\"\".join(ind_to_char[input_example_batch[0]]))\n",
        "print('\\n')\n",
        "print(\"Next Char Predictions: \\n\")\n",
        "print(\"\".join(ind_to_char[sampled_indices ]))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DAOE4rzuBh7f",
        "colab_type": "text"
      },
      "source": [
        "After confirming the dimensions are working, let's train our network!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZYDQjKTlm4s8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs = 30"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_PJ4OVdBm4s8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.fit(dataset,epochs=epochs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kKkD5M6eoSiN"
      },
      "source": [
        "## Step 6: Generating text\n",
        "\n",
        "Currently our model only expects 128 sequences at a time. We can create a new model that only expects a batch_size=1. We can create a new model with this batch size, then load our saved models weights. Then call .build() on the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eYRNG57Govdc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.save('shakespeare_gen.h5') "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GCoJayFS8H4d",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.models import load_model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_iXG3VJvEXWM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = create_model(vocab_size, embed_dim, rnn_neurons, batch_size=1)\n",
        "\n",
        "model.load_weights('shakespeare_gen.h5')\n",
        "\n",
        "model.build(tf.TensorShape([1, None]))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LAX3p7_YEilU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WvuwZBX5Ogfd",
        "colab": {}
      },
      "source": [
        "def generate_text(model, start_seed,gen_size=100,temp=1.0):\n",
        "  '''\n",
        "  model: Trained Model to Generate Text\n",
        "  start_seed: Intial Seed text in string form\n",
        "  gen_size: Number of characters to generate\n",
        "\n",
        "  Basic idea behind this function is to take in some seed text, format it so\n",
        "  that it is in the correct shape for our network, then loop the sequence as\n",
        "  we keep adding our own predicted characters. Similar to our work in the RNN\n",
        "  time series problems.\n",
        "  '''\n",
        "\n",
        "  # Number of characters to generate\n",
        "  num_generate = gen_size\n",
        "\n",
        "  # Vecotrizing starting seed text\n",
        "  input_eval = [char_to_ind[s] for s in start_seed]\n",
        "\n",
        "  # Expand to match batch format shape\n",
        "  input_eval = tf.expand_dims(input_eval, 0)\n",
        "\n",
        "  # Empty list to hold resulting generated text\n",
        "  text_generated = []\n",
        "\n",
        "  # Temperature effects randomness in our resulting text\n",
        "  # The term is derived from entropy/thermodynamics.\n",
        "  # The temperature is used to effect probability of next characters.\n",
        "  # Higher probability == lesss surprising/ more expected\n",
        "  # Lower temperature == more surprising / less expected\n",
        " \n",
        "  temperature = temp\n",
        "\n",
        "  # Here batch size == 1\n",
        "  model.reset_states()\n",
        "\n",
        "  for i in range(num_generate):\n",
        "\n",
        "      # Generate Predictions\n",
        "      predictions = model(input_eval)\n",
        "\n",
        "      # Remove the batch shape dimension\n",
        "      predictions = tf.squeeze(predictions, 0)\n",
        "\n",
        "      # Use a cateogircal disitribution to select the next character\n",
        "      predictions = predictions / temperature\n",
        "      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()\n",
        "\n",
        "      # Pass the predicted charracter for the next input\n",
        "      input_eval = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "      # Transform back to character letter\n",
        "      text_generated.append(ind_to_char[predicted_id])\n",
        "\n",
        "  return (start_seed + ''.join(text_generated))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bS69SG5D5lwd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(generate_text(model,\"flower\",gen_size=1000))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dff5xCHIeEHO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}