Day6_rnn_keras.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "print(sys.version_info)\n",
    "import os\n",
    "os.environ['KERAS_BACKEND'] = 'tensorflow'\n",
    "\n",
    "import keras\n",
    "print('Keras version: {}'.format(keras.__version__))\n",
    "import tensorflow as tf\n",
    "print('TensorFlow version: {}'.format(tf.__version__))\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem: generate names\n",
    "\n",
    "* Struggle to find a name for the variable? Let's see how you'll come up with a name for your son/daughter. Surely no human has expertize over what is a good child name, so let us train NN instead;\n",
    "* The dataset contains ~8k earthling names from different cultures, all in latin transcript;\n",
    "* Objective: learn a generative model over names.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "start_token = \" \"\n",
    "\n",
    "with open(\"names\") as f:\n",
    "    names = f.read()[:-1].split('\\n')\n",
    "    names = [start_token+name for name in names]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('n samples = {}'.format(len(names)))\n",
    "for x in names[::1000]:\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#all unique characters go here\n",
    "tokens = #<all unique characters in the dataset>\n",
    "\n",
    "tokens = list(tokens)\n",
    "\n",
    "n_tokens = len(tokens)\n",
    "print('n_tokens = ',n_tokens)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.title('Sequence length distribution')\n",
    "plt.hist(list(map(len, names)), bins=25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cast everything from symbols into identifiers\n",
    "\n",
    "Tensorflow string manipulation is a bit tricky, so we'll work around it. \n",
    "We'll feed our recurrent neural network with ids of characters from our dictionary.\n",
    "\n",
    "To create such dictionary, let's assign "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_to_id = #<dictionary of symbol -> its identifier (index in tokens list)>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(tokens) == len(token_to_id), \"dictionaries must have same size\"\n",
    "\n",
    "for i in range(n_tokens):\n",
    "    assert token_to_id[tokens[i]] == i, \"token identifier must be it's position in tokens list\"\n",
    "\n",
    "print(\"Seems alright!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_matrix(names, max_len=None, pad=token_to_id[' '], dtype='int32'):\n",
    "    \"\"\"Casts a list of names into rnn-digestable matrix\"\"\"\n",
    "    \n",
    "    max_len = max_len or max(map(len, names))\n",
    "    names_ix = np.zeros([len(names), max_len], dtype) + pad\n",
    "\n",
    "    for i in range(len(names)):\n",
    "        name_ix = list(map(token_to_id.get,names[i]))\n",
    "        names_ix[i, :len(name_ix)] = name_ix\n",
    "\n",
    "    return names_ix.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example: cast 4 random names to matrices, pad with zeros\n",
    "print('\\n'.join(names[::2000]))\n",
    "print(to_matrix(names[::2000]).T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_matrix(names[::2000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recurrent neural network\n",
    "\n",
    "We can rewrite recurrent neural network as a consecutive application of dense layer to input $x_t$ and previous rnn state $h_t$. This is exactly what we're gonna do now.\n",
    "<img src=\"./rnn.png\" width=480>\n",
    "\n",
    "Since we're training a language model, there should also be:\n",
    "* An embedding layer that converts character id x_t to a vector.\n",
    "* An output layer that predicts probabilities of next phoneme"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Concatenate, Dense, Embedding\n",
    "\n",
    "rnn_num_units = 64\n",
    "embedding_size = 16\n",
    "\n",
    "#Let's create layers for our recurrent network\n",
    "embed_x = #<an embedding layer that converts character ids into embeddings>\n",
    "get_h_next = #<a dense layer that maps [x_t,h_t]->h_t+1>. \n",
    "get_probas = #<a dense layer that maps [h_t+1]->P(x_t+1|h_t+1). \n",
    "\n",
    "#Note: please set the correct activation to Dense layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rnn_one_step(x_t, h_t):\n",
    "    \"\"\"\n",
    "    Recurrent neural network step that produces next state and output\n",
    "    given prev input and previous state.\n",
    "    We'll call this method repeatedly to produce the whole sequence.\n",
    "    \n",
    "    Follow inline isntructions to complete the function.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Pay attention to the shapes, it may be messy!\n",
    "    x_t_emb = #<convert character id into embedding>\n",
    "    x_and_h = #<concatenate x embedding and previous h state>\n",
    "    \n",
    "    h_next =  #<compute next state given x_and_h>\n",
    "    \n",
    "    output_probas = #<get probabilities for language model P(x_next|h_next)>\n",
    "    \n",
    "    return output_probas,h_next"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN loop\n",
    "\n",
    "Once rnn_one_step is ready, let's apply it in a loop over name characters to get predictions.\n",
    "\n",
    "Let's assume that all names are at most length-16 for now, so we can simply iterate over them in a for loop.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LENGTH = 16\n",
    "\n",
    "input_sequence = keras.backend.placeholder(shape=(MAX_LENGTH, None), dtype='int32')\n",
    "batch_size = keras.backend.shape(input_sequence)[1]\n",
    "\n",
    "predicted_probas = []\n",
    "h_prev = keras.backend.zeros([batch_size, rnn_num_units]) #initial hidden state\n",
    "\n",
    "for t in range(MAX_LENGTH):\n",
    "    x_t = input_sequence[t]\n",
    "    probas_next,h_next = rnn_one_step(x_t, h_prev)\n",
    "    \n",
    "    h_prev = h_next\n",
    "    predicted_probas.append(probas_next)\n",
    "    \n",
    "predicted_probas = keras.backend.stack(predicted_probas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN: loss and gradients\n",
    "\n",
    "Let's gather a matrix of predictions for $P(x_{next}|h)$ and the corresponding correct answers.\n",
    "\n",
    "Our network can then be trained by minimizing crossentropy between predicted probabilities and those answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_matrix = keras.backend.reshape(predicted_probas[:-1],[-1, len(tokens)])\n",
    "answers_matrix = keras.backend.one_hot(keras.backend.flatten(input_sequence[1:]), n_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To optimize the loss we need a bit of TensorFlow, because we've gone too deep into the backend. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = #<define loss as categorical crossentropy>\n",
    "optimize = tf.train.AdamOptimizer().minimize(loss) # It's the way Adam optimizer is called in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The training loop\n",
    "Docs about tf.Session: https://www.tensorflow.org/api_docs/python/tf/Session\n",
    "Actually, our network is built in TensorFlow, but using high-level keras API. So we need some tf stuff.\n",
    "\n",
    "Calling tf.Session (or tf.InteractiveSession)  creates the session, which is equal to \"runtime\" in terms of tf. All the varaibles are created within this session and available in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "from random import sample\n",
    "\n",
    "sess = tf.InteractiveSession()\n",
    "keras.backend.set_session(sess) \n",
    "sess.run(tf.global_variables_initializer())\n",
    "history = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(2000):\n",
    "    batch = to_matrix(sample(names,32), max_len=MAX_LENGTH) #<sample a batch of names in a matrix format>\n",
    "    loss_i,_ = sess.run([loss, optimize], {input_sequence: batch}) #<compute loss and perform gradient descent on that batch>\n",
    "    \n",
    "    history.append(loss_i.mean())\n",
    "    if (i+1)%100==0:\n",
    "        clear_output(True)\n",
    "        plt.plot(history,label='loss')\n",
    "        plt.legend()\n",
    "        plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN: sampling\n",
    "Once we've trained our network a bit, let's get to actually generating stuff. All we need is the `rnn_one_step` function you have written above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_t = keras.backend.placeholder(shape=(None,), dtype='int32')\n",
    "h_t = keras.backend.variable(np.zeros([1,rnn_num_units],'float32'))\n",
    "\n",
    "next_probs,next_h = rnn_one_step(x_t,h_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_sample(seed_phrase=' ',max_length=MAX_LENGTH, sess=sess):\n",
    "    '''\n",
    "    The function generates text given a phrase of length at least SEQ_LENGTH.\n",
    "        \n",
    "    parameters:\n",
    "        The phrase is set using the variable seed_phrase\n",
    "        The optional input \"N\" is used to set the number of characters of text to predict.     \n",
    "    '''\n",
    "    x_sequence = [token_to_id[token] for token in seed_phrase]\n",
    "    sess.run(tf.assign(h_t,h_t.initial_value))\n",
    "    \n",
    "    #feed the seed phrase, if any\n",
    "    for ix in x_sequence[:-1]:\n",
    "         sess.run(tf.assign(h_t,next_h),{x_t:[ix]})\n",
    "    \n",
    "    #start generating\n",
    "    for _ in range(max_length-len(seed_phrase)):\n",
    "        x_probs,_ = sess.run([next_probs,tf.assign(h_t,next_h)],{x_t:[x_sequence[-1]]})\n",
    "        x_sequence.append(np.random.choice(n_tokens,p=x_probs[0]))\n",
    "        \n",
    "    return ''.join([tokens[ix] for ix in x_sequence])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in range(10):\n",
    "    print(generate_sample())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in range(50):\n",
    "    print(generate_sample(' Murz'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try it out!\n",
    "You've just implemented a recurrent language model that can be tasked with generating any kind of sequence, so there's plenty of data you can try it on:\n",
    "\n",
    "* Novels/poems/songs of your favorite author\n",
    "* News titles/clickbait titles\n",
    "* Source code of Linux or Tensorflow\n",
    "* Molecules in [smiles](https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system) format\n",
    "* Melody in notes/chords format\n",
    "* Ikea catalog titles\n",
    "* Pokemon names\n",
    "* Cards from Magic, the Gathering / Hearthstone\n",
    "\n",
    "If you're willing to give it a try, here's what you wanna look at:\n",
    "* Current data format is a sequence of lines, so a novel can be formatted as a list of sentences. Alternatively, you can change data preprocessing altogether.\n",
    "* While some datasets are readily available, others can only be scraped from the web. Try `Selenium` or `Scrapy` for that.\n",
    "* Make sure MAX_LENGTH is adjusted for longer datasets.\n",
    "* More complex tasks require larger RNN architecture, try more neurons or several layers. It would also require more training iterations.\n",
    "* Long-term dependencies in music, novels or molecules are better handled with LSTM or GRU\n",
    "\n",
    "__Good hunting!__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please, leave the http://bit.ly/feedback_ml_dl_cnn!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Py3 research env",
   "language": "python",
   "name": "py3_research"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}