week4-seq2seq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn to calculate with seq2seq model\n",
    "\n",
    "In this assignment, you will learn how to use neural networks to solve sequence-to-sequence prediction tasks. Seq2Seq models are very popular these days because they achieve great results in Machine Translation, Text Summarization, Conversational Modeling and more.\n",
    "\n",
    "Using sequence-to-sequence modeling you are going to build a calculator for evaluating arithmetic expressions, by taking an equation as an input to the neural network and producing an answer as it's output.\n",
    "\n",
    "The resulting solution for this problem will be based on state-of-the-art approaches for sequence-to-sequence learning and you should be able to easily adapt it to solve other tasks. However, if you want to train your own machine translation system or intellectual chat bot, it would be useful to have access to compute resources like GPU, and be patient, because training of such systems is usually time consuming. \n",
    "\n",
    "### Libraries\n",
    "\n",
    "For this task you will need the following libraries:\n",
    " - [TensorFlow](https://www.tensorflow.org) — an open-source software library for Machine Intelligence.\n",
    " - [scikit-learn](http://scikit-learn.org/stable/index.html) — a tool for data mining and data analysis.\n",
    " \n",
    "If you have never worked with TensorFlow, you will probably want to read some tutorials during your work on this assignment, e.g. [Neural Machine Translation](https://www.tensorflow.org/tutorials/seq2seq) tutorial deals with very similar task and can explain some concepts to you. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "\n",
    "One benefit of this task is that you don't need to download any data — you will generate it on your own! We will use two operators (addition and subtraction) and work with positive integer numbers in some range. Here are examples of correct inputs and outputs:\n",
    "\n",
    "    Input: '1+2'\n",
    "    Output: '3'\n",
    "    \n",
    "    Input: '0-99'\n",
    "    Output: '-99'\n",
    "\n",
    "*Note, that there are no spaces between operators and operands.*\n",
    "\n",
    "\n",
    "Now you need to implement the function *generate_equations*, which will be used to generate the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_equations(allowed_operators, dataset_size, min_value, max_value):\n",
    "    \"\"\"Generates pairs of equations and solutions to them.\n",
    "    \n",
    "       Each equation has a form of two integers with an operator in between.\n",
    "       Each solution is an integer with the result of the operaion.\n",
    "    \n",
    "        allowed_operators: list of strings, allowed operators.\n",
    "        dataset_size: an integer, number of equations to be generated.\n",
    "        min_value: an integer, min value of each operand.\n",
    "        max_value: an integer, max value of each operand.\n",
    "\n",
    "        result: a list of tuples of strings (equation, solution).\n",
    "    \"\"\"\n",
    "    sample = []\n",
    "    for _ in range(dataset_size):\n",
    "        opr = random.choice(allowed_operators)\n",
    "        no1 = random.randint(min_value, max_value)\n",
    "        no2 = random.randint(min_value, max_value)\n",
    "        eqn = str(no1)+opr+str(no2)\n",
    "        result = str(eval(eqn))\n",
    "        sample.append((eqn, result))\n",
    "                \n",
    "    return sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check the correctness of your implementation, use *test_generate_equations* function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_generate_equations():\n",
    "    allowed_operators = ['+', '-']\n",
    "    dataset_size = 10\n",
    "    for (input_, output_) in generate_equations(allowed_operators, dataset_size, 0, 100):\n",
    "        if not (type(input_) is str and type(output_) is str):\n",
    "            return \"Both parts should be strings.\"\n",
    "        if eval(input_) != int(output_):\n",
    "            return \"The (equation: {!r}, solution: {!r}) pair is incorrect.\".format(input_, output_)\n",
    "    return \"Tests passed.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests passed.\n"
     ]
    }
   ],
   "source": [
    "print(test_generate_equations())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we are ready to generate the train and test data for the neural network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "allowed_operators = ['+', '-']\n",
    "dataset_size = 100000\n",
    "data = generate_equations(allowed_operators, dataset_size, min_value=0, max_value=9999)\n",
    "\n",
    "train_set, test_set = train_test_split(data, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data for the neural network\n",
    "\n",
    "The next stage of data preparation is creating mappings of the characters to their indices in some vocabulary. Since in our task we already know which symbols will appear in the inputs and outputs, generating the vocabulary is a simple step.\n",
    "\n",
    "#### How to create dictionaries for other task\n",
    "\n",
    "First of all, you need to understand what is the basic unit of the sequence in your task. In our case, we operate on symbols and the basic unit is a symbol. The number of symbols is small, so we don't need to think about filtering/normalization steps. However, in other tasks, the basic unit is often a word, and in this case the mapping would be *word $\\to$ integer*. The number of words might be huge, so it would be reasonable to filter them, for example, by frequency and leave only the frequent ones. Other strategies that your should consider are: data normalization (lowercasing, tokenization, how to consider punctuation marks), separate vocabulary for input and for output (e.g. for machine translation), some specifics of the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'4': 8, '2': 6, '#': 2, '8': 12, '7': 11, '9': 13, '0': 14, '-': 4, '+': 3, '$': 1, '^': 0, '3': 7, '5': 9, '1': 5, '6': 10}\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "word2id = {symbol:i for i, symbol in enumerate('^$#+-1234567890')}\n",
    "id2word = {i:symbol for symbol, i in word2id.items()}\n",
    "print(word2id)\n",
    "print(len(word2id.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Special symbols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "start_symbol = '^'\n",
    "end_symbol = '$'\n",
    "padding_symbol = '#'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could notice that we have added 3 special symbols: '^', '\\$' and '#':\n",
    "- '^' symbol will be passed to the network to indicate the beginning of the decoding procedure. We will discuss this one later in more details.\n",
    "- '\\$' symbol will be used to indicate the *end of a string*, both for input and output sequences. \n",
    "- '#' symbol will be used as a *padding* character to make lengths of all strings equal within one training batch.\n",
    "\n",
    "People have a bit different habits when it comes to special symbols in encoder-decoder networks, so don't get too much confused if you come across other variants in tutorials you read. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Padding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When vocabularies are ready, we need to be able to convert a sentence to a list of vocabulary word indices and back. At the same time, let's care about padding. We are going to preprocess each sequence from the input (and output ground truth) in such a way that:\n",
    "- it has a predefined length *padded_len*\n",
    "- it is probably cut off or padded with the *padding symbol* '#'\n",
    "- it *always* ends with the *end symbol* '$'\n",
    "\n",
    "We will treat the original characters of the sequence **and the end symbol** as the valid part of the input. We will store *the actual length* of the sequence, which includes the end symbol, but does not include the padding symbols. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Now you need to implement the function *sentence_to_ids* that does the described job. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sentence_to_ids(sentence, word2id, padded_len):\n",
    "    \"\"\" Converts a sequence of symbols to a padded sequence of their ids.\n",
    "    \n",
    "      sentence: a string, input/output sequence of symbols.\n",
    "      word2id: a dict, a mapping from original symbols to ids.\n",
    "      padded_len: an integer, a desirable length of the sequence.\n",
    "\n",
    "      result: a tuple of (a list of ids, an actual length of sentence).\n",
    "    \"\"\"\n",
    "    \n",
    "    sent_ids = []\n",
    "    lensen = len(sentence)\n",
    "    if lensen < padded_len-1:\n",
    "        a = lensen\n",
    "        b = padded_len - lensen - 1\n",
    "    else:\n",
    "        a = padded_len - 1\n",
    "        b = 0\n",
    "    \n",
    "    for i in range(a):\n",
    "        sent_ids.append(word2id[sentence[i]])\n",
    "    sent_ids.append(word2id[\"$\"])\n",
    "    if b != 0:\n",
    "        for i in range(b):\n",
    "            sent_ids.append(word2id[\"#\"])\n",
    "    ##print(sent_ids)\n",
    "    sent_len = a + 1\n",
    "    return sent_ids, sent_len"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that your implementation is correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_sentence_to_ids():\n",
    "    sentences = [(\"123+123\", 7), (\"123+123\", 8), (\"123+123\", 10)]\n",
    "    expected_output = [([5, 6, 7, 3, 5, 6, 1], 7), \n",
    "                       ([5, 6, 7, 3, 5, 6, 7, 1], 8), \n",
    "                       ([5, 6, 7, 3, 5, 6, 7, 1, 2, 2], 8)] \n",
    "    for (sentence, padded_len), (sentence_ids, expected_length) in zip(sentences, expected_output):\n",
    "        output, length = sentence_to_ids(sentence, word2id, padded_len)\n",
    "        if output != sentence_ids:\n",
    "            return(\"Convertion of '{}' for padded_len={} to {} is incorrect.\".format(\n",
    "                sentence, padded_len, output))\n",
    "        if length != expected_length:\n",
    "            return(\"Convertion of '{}' for padded_len={} has incorrect actual length {}.\".format(\n",
    "                sentence, padded_len, length))\n",
    "    return(\"Tests passed.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests passed.\n"
     ]
    }
   ],
   "source": [
    "print(test_sentence_to_ids())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to be able to get back from indices to symbols:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ids_to_sentence(ids, id2word):\n",
    "    \"\"\" Converts a sequence of ids to a sequence of symbols.\n",
    "    \n",
    "          ids: a list, indices for the padded sequence.\n",
    "          id2word:  a dict, a mapping from ids to original symbols.\n",
    "\n",
    "          result: a list of symbols.\n",
    "    \"\"\"\n",
    " \n",
    "    return [id2word[i] for i in ids] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating batches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step of data preparation is a function that transforms a batch of sentences to a list of lists of indices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def batch_to_ids(sentences, word2id, max_len):\n",
    "    \"\"\"Prepares batches of indices. \n",
    "    \n",
    "       Sequences are padded to match the longest sequence in the batch,\n",
    "       if it's longer than max_len, then max_len is used instead.\n",
    "\n",
    "        sentences: a list of strings, original sequences.\n",
    "        word2id: a dict, a mapping from original symbols to ids.\n",
    "        max_len: an integer, max len of sequences allowed.\n",
    "\n",
    "        result: a list of lists of ids, a list of actual lengths.\n",
    "    \"\"\"\n",
    "    \n",
    "    max_len_in_batch = min(max(len(s) for s in sentences) + 1, max_len)\n",
    "    batch_ids, batch_ids_len = [], []\n",
    "    for sentence in sentences:\n",
    "        ids, ids_len = sentence_to_ids(sentence, word2id, max_len_in_batch)\n",
    "        batch_ids.append(ids)\n",
    "        batch_ids_len.append(ids_len)\n",
    "    return batch_ids, batch_ids_len"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function *generate_batches* will help to generate batches with defined size from given samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_batches(samples, batch_size=64):\n",
    "    X, Y = [], []\n",
    "    for i, (x, y) in enumerate(samples, 1):\n",
    "        X.append(x)\n",
    "        Y.append(y)\n",
    "        if i % batch_size == 0:\n",
    "            yield X, Y\n",
    "            X, Y = [], []\n",
    "    if X and Y:\n",
    "        yield X, Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate the result of the implemented functions, run the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: ('9634+6607', '16241')\n",
      "Ids: [[13, 10, 7, 8, 3, 10, 10, 14, 11, 1], [5, 10, 6, 8, 5, 1, 2, 2, 2, 2]]\n",
      "Sentences lengths: [10, 6]\n"
     ]
    }
   ],
   "source": [
    "sentences = train_set[0]\n",
    "ids, sent_lens = batch_to_ids(sentences, word2id, max_len=10)\n",
    "print('Input:', sentences)\n",
    "print('Ids: {}\\nSentences lengths: {}'.format(ids, sent_lens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder-Decoder architecture\n",
    "\n",
    "Encoder-Decoder is a successful architecture for Seq2Seq tasks with different lengths of input and output sequences. The main idea is to use two recurrent neural networks, where the first neural network *encodes* the input sequence into a real-valued vector and then the second neural network *decodes* this vector into the output sequence. While building the neural network, we will specify some particular characteristics of this architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us use TensorFlow building blocks to specify the network architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Seq2SeqModel(object):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to create [placeholders](https://www.tensorflow.org/api_guides/python/io_ops#Placeholders) to specify what data we are going to feed into the network during the execution time. For this task we will need:\n",
    " - *input_batch* — sequences of sentences (the shape will equal to [batch_size, max_sequence_len_in_batch]);\n",
    " - *input_batch_lengths* — lengths of not padded sequences (the shape equals to [batch_size]);\n",
    " - *ground_truth* — sequences of groundtruth (the shape will equal to [batch_size, max_sequence_len_in_batch]);\n",
    " - *ground_truth_lengths* — lengths of not padded groundtruth sequences (the shape equals to [batch_size]);\n",
    " - *dropout_ph* — dropout keep probability; this placeholder has a predifined value 1;\n",
    " - *learning_rate_ph* — learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def declare_placeholders(self):\n",
    "    \"\"\"Specifies placeholders for the model.\"\"\"\n",
    "    \n",
    "    # Placeholders for input and its actual lengths.\n",
    "    self.input_batch = tf.placeholder(shape=(None, None), dtype=tf.int32, name='input_batch')\n",
    "    self.input_batch_lengths = tf.placeholder(shape=(None, ), dtype=tf.int32, name='input_batch_lengths')\n",
    "    \n",
    "    # Placeholders for groundtruth and its actual lengths.\n",
    "    self.ground_truth = tf.placeholder(shape=(None, None), dtype=tf.int32, name='ground_truth')\n",
    "    self.ground_truth_lengths = tf.placeholder(shape=(None, ), dtype=tf.int32, name='ground_truth_lengths')\n",
    "        \n",
    "    self.dropout_ph = tf.placeholder_with_default(tf.cast(1.0, tf.float32), shape=[])\n",
    "    self.learning_rate_ph = tf.placeholder(shape=[], dtype=tf.float32, name='learning_rate_ph')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.__declare_placeholders = classmethod(declare_placeholders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us specify the layers of the neural network. First, we need to prepare an embedding matrix. Since we use the same vocabulary for input and output, we need only one such matrix. For tasks with different vocabularies there would be multiple embedding layers.\n",
    "- Create embeddings matrix with [tf.Variable](https://www.tensorflow.org/api_docs/python/tf/Variable). Specify its name, type (tf.float32), and initialize with random values.\n",
    "- Perform [embeddings lookup](https://www.tensorflow.org/api_docs/python/tf/nn/embedding_lookup) for a given input batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_embeddings(self, vocab_size, embeddings_size):\n",
    "    \"\"\"Specifies embeddings layer and embeds an input batch.\"\"\"\n",
    "     \n",
    "    random_initializer = tf.random_uniform((vocab_size, embeddings_size), -1.0, 1.0)\n",
    "    self.embeddings = tf.Variable(random_initializer, dtype=tf.float32, name=\"embeddings\")\n",
    " \n",
    "    # Perform embeddings lookup for self.input_batch. \n",
    "    self.input_batch_embedded = tf.nn.embedding_lookup(params=self.embeddings, ids=self.input_batch)\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.__create_embeddings = classmethod(create_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Encoder\n",
    "\n",
    "The first RNN of the current architecture is called an *encoder* and serves for encoding an input sequence to a real-valued vector. Input of this RNN is an embedded input batch. Since sentences in the same batch could have different actual lengths, we also provide input lengths to avoid unnecessary computations. The final encoder state will be passed to the second RNN (decoder), which we will create soon. \n",
    "\n",
    "- TensorFlow provides a number of [RNN cells](https://www.tensorflow.org/api_guides/python/contrib.rnn#Core_RNN_Cells_for_use_with_TensorFlow_s_core_RNN_methods) ready for use. We suggest that you use [GRU cell](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/GRUCell), but you can also experiment with other types. \n",
    "- Wrap your cells with [DropoutWrapper](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/DropoutWrapper). Dropout is an important regularization technique for neural networks. Specify input keep probability using the dropout placeholder that we created before.\n",
    "- Combine the defined encoder cells with [Dynamic RNN](https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn). Use the embedded input batches and their lengths here.\n",
    "- Use *dtype=tf.float32* everywhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_encoder(self, hidden_size):\n",
    "    \"\"\"Specifies encoder architecture and computes its output.\"\"\"\n",
    "    \n",
    "    # Create GRUCell with dropout.\n",
    "    encoder_cell = tf.contrib.rnn.DropoutWrapper(tf.contrib.rnn.GRUCell(num_units = hidden_size),\n",
    "                                                 dtype = tf.float32,\n",
    "                                                 input_keep_prob = self.dropout_ph)\n",
    "    \n",
    "    # Create RNN with the predefined cell.\n",
    "    _, self.final_encoder_state = tf.nn.dynamic_rnn(cell = encoder_cell, inputs = self.input_batch_embedded, \n",
    "                                                    sequence_length=self.input_batch_lengths, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.__build_encoder = classmethod(build_encoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decoder\n",
    "\n",
    "The second RNN is called a *decoder* and serves for generating the output sequence. In the simple seq2seq arcitecture, the input sequence is provided to the decoder only as the final state of the encoder. Obviously, it is a bottleneck and [Attention techniques](https://www.tensorflow.org/tutorials/seq2seq#background_on_the_attention_mechanism) can help to overcome it. So far, we do not need them to make our calculator work, but this would be a necessary ingredient for more advanced tasks. \n",
    "\n",
    "During training, decoder also uses information about the true output. It is feeded in as input symbol by symbol. However, during the prediction stage (which is called *inference* in this architecture), the decoder can only use its own generated output from the previous step to feed it in at the next step. Because of this difference (*training* vs *inference*), we will create two distinct instances, which will serve for the described scenarios.\n",
    "\n",
    "The picture below illustrates the point. It also shows our work with the special characters, e.g. look how the start symbol `^` is used. The transparent parts are ignored. In decoder, it is masked out in the loss computation. In encoder, the green state is considered as final and passed to the decoder. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"encoder-decoder-pic.png\" style=\"width: 500px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, it's time to implement the decoder:\n",
    " - First, we should create two [helpers](https://www.tensorflow.org/api_guides/python/contrib.seq2seq#Dynamic_Decoding). These classes help to determine the behaviour of the decoder. During the training time, we will use [TrainingHelper](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/TrainingHelper). For the inference we recommend to use [GreedyEmbeddingHelper](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/GreedyEmbeddingHelper).\n",
    " - To share all parameters during training and inference, we use one scope and set the flag 'reuse' to True at inference time. You might be interested to know more about how [variable scopes](https://www.tensorflow.org/programmers_guide/variables) work in TF. \n",
    " - To create the decoder itself, we will use [BasicDecoder](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/BasicDecoder) class. As previously, you should choose some RNN cell, e.g. GRU cell. To turn hidden states into logits, we will need a projection layer. One of the simple solutions is using [OutputProjectionWrapper](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/OutputProjectionWrapper).\n",
    " - For getting the predictions, it will be convinient to use [dynamic_decode](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/dynamic_decode). This function uses the provided decoder to perform decoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_decoder(self, hidden_size, vocab_size, max_iter, start_symbol_id, end_symbol_id):\n",
    "    \"\"\"Specifies decoder architecture and computes the output.\n",
    "    \n",
    "        Uses different helpers:\n",
    "          - for train: feeding ground truth\n",
    "          - for inference: feeding generated output\n",
    "\n",
    "        As a result, self.train_outputs and self.infer_outputs are created. \n",
    "        Each of them contains two fields:\n",
    "          rnn_output (predicted logits)\n",
    "          sample_id (predictions).\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    # Use start symbols as the decoder inputs at the first time step.\n",
    "    batch_size = tf.shape(self.input_batch)[0]\n",
    "    start_tokens = tf.fill([batch_size], start_symbol_id)\n",
    "    ground_truth_as_input = tf.concat([tf.expand_dims(start_tokens, 1), self.ground_truth], 1)\n",
    "    \n",
    "    # Use the embedding layer defined before to lookup embedings for ground_truth_as_input. \n",
    "    self.ground_truth_embedded = tf.nn.embedding_lookup(params=self.embeddings, ids=ground_truth_as_input)\n",
    "\n",
    "     \n",
    "    # Create TrainingHelper for the train stage.\n",
    "    train_helper = tf.contrib.seq2seq.TrainingHelper(self.ground_truth_embedded, \n",
    "                                                     self.ground_truth_lengths)\n",
    "    \n",
    "    # Create GreedyEmbeddingHelper for the inference stage.\n",
    "    # You should provide the embedding layer, start_tokens and index of the end symbol.\n",
    "    infer_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(self.embeddings, start_tokens, end_symbol_id)\n",
    "    \n",
    "    def decode(helper, scope, reuse=None):\n",
    "        \"\"\"Creates decoder and return the results of the decoding with a given helper.\"\"\"\n",
    "        \n",
    "        with tf.variable_scope(scope, reuse=reuse):\n",
    "            # Create GRUCell with dropout. Do not forget to set the reuse flag properly.\n",
    "            decoder_cell = tf.contrib.rnn.DropoutWrapper(tf.contrib.rnn.GRUCell(num_units = hidden_size, reuse=reuse), \n",
    "                                                 input_keep_prob = self.dropout_ph)\n",
    "            \n",
    "            # Create a projection wrapper.\n",
    "            decoder_cell = tf.contrib.rnn.OutputProjectionWrapper(decoder_cell, vocab_size, reuse=reuse)\n",
    "            \n",
    "            # Create BasicDecoder, pass the defined cell, a helper, and initial state.\n",
    "            # The initial state should be equal to the final state of the encoder!\n",
    "            decoder = tf.contrib.seq2seq.BasicDecoder(cell=decoder_cell, helper=helper,\n",
    "                                                      initial_state=self.final_encoder_state)\n",
    "            \n",
    "            # The first returning argument of dynamic_decode contains two fields:\n",
    "            #   rnn_output (predicted logits)\n",
    "            #   sample_id (predictions)\n",
    "            outputs, _, _ = tf.contrib.seq2seq.dynamic_decode(decoder=decoder, maximum_iterations=max_iter, \n",
    "                                                              output_time_major=False, impute_finished=True)\n",
    "\n",
    "            return outputs\n",
    "        \n",
    "    self.train_outputs = decode(train_helper, 'decode')\n",
    "    self.infer_outputs = decode(infer_helper, 'decode', reuse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.__build_decoder = classmethod(build_decoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this task we will use [sequence_loss](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/sequence_loss), which is a weighted cross-entropy loss for a sequence of logits. Take a moment to understand, what is your train logits and targets. Also note, that we do not want to take into account loss terms coming from padding symbols, so we will mask them out using weights.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_loss(self):\n",
    "    \"\"\"Computes sequence loss (masked cross-entopy loss with logits).\"\"\"\n",
    "    \n",
    "    weights = tf.cast(tf.sequence_mask(self.ground_truth_lengths), dtype=tf.float32)\n",
    "    \n",
    "    self.loss = tf.contrib.seq2seq.sequence_loss(logits = self.train_outputs.rnn_output, \n",
    "                                                 targets = self.ground_truth, weights = weights) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.__compute_loss = classmethod(compute_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing to specify is the optimization of the defined loss. \n",
    "We suggest that you use [optimize_loss](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/optimize_loss) with Adam optimizer and a learning rate from the corresponding placeholder. You might also need to pass global step (e.g. as tf.train.get_global_step()) and clip gradients by 1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def perform_optimization(self):\n",
    "    \"\"\"Specifies train_op that optimizes self.loss.\"\"\"\n",
    "    \n",
    "    self.train_op = tf.contrib.layers.optimize_loss(loss = self.loss, global_step = tf.train.get_global_step(),\n",
    "                                                    learning_rate = self.learning_rate_ph, optimizer =\"Adam\",\n",
    "                                                    clip_gradients = 1.0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.__perform_optimization = classmethod(perform_optimization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have specified all the parts of your network. You may have noticed, that we didn't deal with any real data yet, so what you have written is just recipies on how the network should function.\n",
    "Now we will put them to the constructor of our Seq2SeqModel class to use it in the next section. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def init_model(self, vocab_size, embeddings_size, hidden_size, \n",
    "               max_iter, start_symbol_id, end_symbol_id, padding_symbol_id):\n",
    "    \n",
    "    self.__declare_placeholders()\n",
    "    self.__create_embeddings(vocab_size, embeddings_size)\n",
    "    self.__build_encoder(hidden_size)\n",
    "    self.__build_decoder(hidden_size, vocab_size, max_iter, start_symbol_id, end_symbol_id)\n",
    "    \n",
    "    # Compute loss and back-propagate.\n",
    "    self.__compute_loss()\n",
    "    self.__perform_optimization()\n",
    "    \n",
    "    # Get predictions for evaluation.\n",
    "    self.train_predictions = self.train_outputs.sample_id\n",
    "    self.infer_predictions = self.infer_outputs.sample_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.__init__ = classmethod(init_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the network and predict output\n",
    "\n",
    "[Session.run](https://www.tensorflow.org/api_docs/python/tf/Session#run) is a point which initiates computations in the graph that we have defined. To train the network, we need to compute *self.train_op*. To predict output, we just need to compute *self.infer_predictions*. In any case, we need to feed actual data through the placeholders that we defined above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_on_batch(self, session, X, X_seq_len, Y, Y_seq_len, learning_rate, dropout_keep_probability):\n",
    "    feed_dict = {\n",
    "            self.input_batch: X,\n",
    "            self.input_batch_lengths: X_seq_len,\n",
    "            self.ground_truth: Y,\n",
    "            self.ground_truth_lengths: Y_seq_len,\n",
    "            self.learning_rate_ph: learning_rate,\n",
    "            self.dropout_ph: dropout_keep_probability\n",
    "        }\n",
    "    pred, loss, _ = session.run([\n",
    "            self.train_predictions,\n",
    "            self.loss,\n",
    "            self.train_op], feed_dict=feed_dict)\n",
    "    return pred, loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.train_on_batch = classmethod(train_on_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We implemented two prediction functions: *predict_for_batch* and *predict_for_batch_with_loss*. The first one allows only to predict output for some input sequence, while the second one could compute loss because we provide also ground truth values. Both these functions might be useful since the first one could be used for predicting only, and the second one is helpful for validating results on not-training data during the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def predict_for_batch(self, session, X, X_seq_len):\n",
    "    feed_dict = {\n",
    "            self.input_batch: X,\n",
    "            self.input_batch_lengths: X_seq_len\n",
    "        }\n",
    "    pred = session.run([\n",
    "            self.infer_predictions\n",
    "        ], feed_dict=feed_dict)[0]\n",
    "    return pred\n",
    "\n",
    "def predict_for_batch_with_loss(self, session, X, X_seq_len, Y, Y_seq_len):\n",
    "    feed_dict = {\n",
    "            self.input_batch: X,\n",
    "            self.input_batch_lengths: X_seq_len,\n",
    "            self.ground_truth: Y,\n",
    "            self.ground_truth_lengths: Y_seq_len\n",
    "        }\n",
    "    pred, loss = session.run([\n",
    "            self.infer_predictions,\n",
    "            self.loss,\n",
    "        ], feed_dict=feed_dict)\n",
    "    return pred, loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Seq2SeqModel.predict_for_batch = classmethod(predict_for_batch)\n",
    "Seq2SeqModel.predict_for_batch_with_loss = classmethod(predict_for_batch_with_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run your experiment\n",
    "\n",
    "Create *Seq2SeqModel* model with the following parameters:\n",
    " - *vocab_size* — number of tokens;\n",
    " - *embeddings_size* — dimension of embeddings, recommended value: 20;\n",
    " - *max_iter* — maximum number of steps in decoder, recommended value: 7;\n",
    " - *hidden_size* — size of hidden layers for RNN, recommended value: 512;\n",
    " - *start_symbol_id* — an index of the start token (`^`).\n",
    " - *end_symbol_id* — an index of the end token (`$`).\n",
    " - *padding_symbol_id* — an index of the padding token (`#`).\n",
    "\n",
    "Set hyperparameters. You might want to start with the following values and see how it works:\n",
    "- *batch_size*: 128;\n",
    "- at least 10 epochs;\n",
    "- value of *learning_rate*: 0.001\n",
    "- *dropout_keep_probability* equals to 0.5 for training (typical values for dropout probability are ranging from 0.1 to 1.0); larger values correspond smaler number of dropout units;\n",
    "- *max_len*: 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "model = Seq2SeqModel(vocab_size=len(word2id.keys()), embeddings_size=20, hidden_size=512, \n",
    "               max_iter=7, start_symbol_id=word2id[\"^\"], end_symbol_id=word2id[\"$\"], padding_symbol_id=word2id[\"#\"])\n",
    "    \n",
    "##vocabulary_size=len(token2idx), n_tags=len(tag2idx), embedding_dim=200, n_hidden_rnn=200, PAD_index=token2idx[\"<PAD>\"])\n",
    "\n",
    "batch_size = 128\n",
    "n_epochs = 10\n",
    "learning_rate = 0.001\n",
    "dropout_keep_probability = 0.5\n",
    "max_len = 20\n",
    "\n",
    "n_step = int(len(train_set) / batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we are ready to run the training! A good indicator that everything works fine is decreasing loss during the training. You should account on the loss value equal to approximately 2.7 at the beginning of the training and near 1 after the 10th epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training... \n",
      "\n",
      "Train: epoch 1\n",
      "Epoch: [1/10], step: [1/625], loss: 2.708761\n",
      "Epoch: [1/10], step: [201/625], loss: 1.804391\n",
      "Epoch: [1/10], step: [401/625], loss: 1.720117\n",
      "Epoch: [1/10], step: [601/625], loss: 1.618968\n",
      "Test: epoch 1 loss: 1.5940574\n",
      "X: 7231-8385$\n",
      "Y: -1154$\n",
      "O: -166$^\n",
      "\n",
      "X: 5359+9895$\n",
      "Y: 15254$\n",
      "O: 14448$\n",
      "\n",
      "X: 3540+7137$\n",
      "Y: 10677$\n",
      "O: 10688$\n",
      "\n",
      "Train: epoch 2\n",
      "Epoch: [2/10], step: [1/625], loss: 1.635399\n",
      "Epoch: [2/10], step: [201/625], loss: 1.556944\n",
      "Epoch: [2/10], step: [401/625], loss: 1.517232\n",
      "Epoch: [2/10], step: [601/625], loss: 1.498058\n",
      "Test: epoch 2 loss: 1.4041077\n",
      "X: 4337-4059$\n",
      "Y: 278$##\n",
      "O: -113$^\n",
      "\n",
      "X: 6045-2536$\n",
      "Y: 3509$#\n",
      "O: 3885$^\n",
      "\n",
      "X: 4070+3612$\n",
      "Y: 7682$#\n",
      "O: 8333$^\n",
      "\n",
      "Train: epoch 3\n",
      "Epoch: [3/10], step: [1/625], loss: 1.424415\n",
      "Epoch: [3/10], step: [201/625], loss: 1.427099\n",
      "Epoch: [3/10], step: [401/625], loss: 1.421379\n",
      "Epoch: [3/10], step: [601/625], loss: 1.410324\n",
      "Test: epoch 3 loss: 1.380612\n",
      "X: 5089-5203$\n",
      "Y: -114$#\n",
      "O: -43$^^\n",
      "\n",
      "X: 8009+2271$\n",
      "Y: 10280$\n",
      "O: 10006$\n",
      "\n",
      "X: 5019-3242$\n",
      "Y: 1777$#\n",
      "O: 1988$^\n",
      "\n",
      "Train: epoch 4\n",
      "Epoch: [4/10], step: [1/625], loss: 1.368297\n",
      "Epoch: [4/10], step: [201/625], loss: 1.355149\n",
      "Epoch: [4/10], step: [401/625], loss: 1.350883\n",
      "Epoch: [4/10], step: [601/625], loss: 1.363960\n",
      "Test: epoch 4 loss: 1.2717757\n",
      "X: 3377-4148$\n",
      "Y: -771$#\n",
      "O: -1055$\n",
      "\n",
      "X: 8753-1604$\n",
      "Y: 7149$#\n",
      "O: 6575$^\n",
      "\n",
      "X: 6098+4425$\n",
      "Y: 10523$\n",
      "O: 10477$\n",
      "\n",
      "Train: epoch 5\n",
      "Epoch: [5/10], step: [1/625], loss: 1.322987\n",
      "Epoch: [5/10], step: [201/625], loss: 1.333319\n",
      "Epoch: [5/10], step: [401/625], loss: 1.263655\n",
      "Epoch: [5/10], step: [601/625], loss: 1.292997\n",
      "Test: epoch 5 loss: 1.2709008\n",
      "X: 7837+9242$\n",
      "Y: 17079$\n",
      "O: 16799$\n",
      "\n",
      "X: 5434-1028$\n",
      "Y: 4406$#\n",
      "O: 4001$^\n",
      "\n",
      "X: 9009+5375$\n",
      "Y: 14384$\n",
      "O: 14099$\n",
      "\n",
      "Train: epoch 6\n",
      "Epoch: [6/10], step: [1/625], loss: 1.308101\n",
      "Epoch: [6/10], step: [201/625], loss: 1.318805\n",
      "Epoch: [6/10], step: [401/625], loss: 1.318024\n",
      "Epoch: [6/10], step: [601/625], loss: 1.247735\n",
      "Test: epoch 6 loss: 1.2111386\n",
      "X: 187+809$##\n",
      "Y: 996$##\n",
      "O: 1004$^\n",
      "\n",
      "X: 1905+3068$\n",
      "Y: 4973$#\n",
      "O: 5055$^\n",
      "\n",
      "X: 37-9812$##\n",
      "Y: -9775$\n",
      "O: -9554$\n",
      "\n",
      "Train: epoch 7\n",
      "Epoch: [7/10], step: [1/625], loss: 1.244310\n",
      "Epoch: [7/10], step: [201/625], loss: 1.212193\n",
      "Epoch: [7/10], step: [401/625], loss: 1.224875\n",
      "Epoch: [7/10], step: [601/625], loss: 1.203544\n",
      "Test: epoch 7 loss: 1.177344\n",
      "X: 4330+3342$\n",
      "Y: 7672$#\n",
      "O: 7711$^\n",
      "\n",
      "X: 2152-224$#\n",
      "Y: 1928$#\n",
      "O: 2001$^\n",
      "\n",
      "X: 7776+5328$\n",
      "Y: 13104$\n",
      "O: 12988$\n",
      "\n",
      "Train: epoch 8\n",
      "Epoch: [8/10], step: [1/625], loss: 1.208542\n",
      "Epoch: [8/10], step: [201/625], loss: 1.145003\n",
      "Epoch: [8/10], step: [401/625], loss: 1.134246\n",
      "Epoch: [8/10], step: [601/625], loss: 1.091770\n",
      "Test: epoch 8 loss: 1.0204325\n",
      "X: 9726+3871$\n",
      "Y: 13597$\n",
      "O: 13644$\n",
      "\n",
      "X: 6817-1994$\n",
      "Y: 4823$#\n",
      "O: 4888$^\n",
      "\n",
      "X: 3976+7503$\n",
      "Y: 11479$\n",
      "O: 11542$\n",
      "\n",
      "Train: epoch 9\n",
      "Epoch: [9/10], step: [1/625], loss: 1.116297\n",
      "Epoch: [9/10], step: [201/625], loss: 1.030602\n",
      "Epoch: [9/10], step: [401/625], loss: 1.045512\n",
      "Epoch: [9/10], step: [601/625], loss: 1.007370\n",
      "Test: epoch 9 loss: 0.98444676\n",
      "X: 4117-4529$\n",
      "Y: -412$#\n",
      "O: -363$^\n",
      "\n",
      "X: 7394+755$#\n",
      "Y: 8149$#\n",
      "O: 8112$^\n",
      "\n",
      "X: 148-4778$#\n",
      "Y: -4630$\n",
      "O: -4611$\n",
      "\n",
      "Train: epoch 10\n",
      "Epoch: [10/10], step: [1/625], loss: 1.015111\n",
      "Epoch: [10/10], step: [201/625], loss: 1.000079\n",
      "Epoch: [10/10], step: [401/625], loss: 1.050248\n",
      "Epoch: [10/10], step: [601/625], loss: 0.962487\n",
      "Test: epoch 10 loss: 0.91267735\n",
      "X: 2887-7651$\n",
      "Y: -4764$\n",
      "O: -4775$\n",
      "\n",
      "X: 1621-1116$\n",
      "Y: 505$##\n",
      "O: 516$^^\n",
      "\n",
      "X: 3091+7839$\n",
      "Y: 10930$\n",
      "O: 11917$\n",
      "\n",
      "\n",
      "...training finished.\n"
     ]
    }
   ],
   "source": [
    "session = tf.Session()\n",
    "session.run(tf.global_variables_initializer())\n",
    "            \n",
    "invalid_number_prediction_counts = []\n",
    "all_model_predictions = []\n",
    "all_ground_truth = []\n",
    "\n",
    "print('Start training... \\n')\n",
    "for epoch in range(n_epochs):  \n",
    "    random.shuffle(train_set)\n",
    "    random.shuffle(test_set)\n",
    "    \n",
    "    print('Train: epoch', epoch + 1)\n",
    "    for n_iter, (X_batch, Y_batch) in enumerate(generate_batches(train_set, batch_size=batch_size)):\n",
    "        X, X_seq_len = batch_to_ids(X_batch, word2id, max_len)\n",
    "        Y, Y_seq_len = batch_to_ids(Y_batch, word2id, max_len)\n",
    "        # prepare the data (X_batch and Y_batch) for training\n",
    "        # using function batch_to_ids\n",
    "        predictions, loss = model.train_on_batch(session, X, X_seq_len, Y, Y_seq_len, learning_rate, dropout_keep_probability)\n",
    "        \n",
    "        if n_iter % 200 == 0:\n",
    "            print(\"Epoch: [%d/%d], step: [%d/%d], loss: %f\" % (epoch + 1, n_epochs, n_iter + 1, n_step, loss))\n",
    "                \n",
    "    X_sent, Y_sent = next(generate_batches(test_set, batch_size=batch_size))\n",
    "    X, X_seq_len = batch_to_ids(X_sent, word2id, max_len)\n",
    "    Y, Y_seq_len = batch_to_ids(Y_sent, word2id, max_len)\n",
    "    # prepare test data (X_sent and Y_sent) for predicting \n",
    "    # quality and computing value of the loss function\n",
    "    # using function batch_to_ids\n",
    "    \n",
    "    predictions, loss = predict_for_batch_with_loss(model, session, X, X_seq_len, Y, Y_seq_len)\n",
    "    print('Test: epoch', epoch + 1, 'loss:', loss,)\n",
    "    for x, y, p  in list(zip(X, Y, predictions))[:3]:\n",
    "        print('X:',''.join(ids_to_sentence(x, id2word)))\n",
    "        print('Y:',''.join(ids_to_sentence(y, id2word)))\n",
    "        print('O:',''.join(ids_to_sentence(p, id2word)))\n",
    "        print('')\n",
    "\n",
    "    model_predictions = []\n",
    "    ground_truth = []\n",
    "    invalid_number_prediction_count = 0\n",
    "    # For the whole test set calculate ground-truth values (as integer numbers)\n",
    "    # and prediction values (also as integers) to calculate metrics.\n",
    "    # If generated by model number is not correct (e.g. '1-1'), \n",
    "    # increase invalid_number_prediction_count and don't append this and corresponding\n",
    "    # ground-truth value to the arrays.\n",
    "    for X_batch, Y_batch in generate_batches(test_set, batch_size=batch_size):\n",
    "        X, X_seq_len = batch_to_ids(X_batch, word2id, max_len)\n",
    "        Y, Y_seq_len = batch_to_ids(Y_batch, word2id, max_len)\n",
    "        predictions = predict_for_batch(model, session, X, X_seq_len)\n",
    "        for yid, pid in list(zip(Y, predictions)):\n",
    "            y = str(\"\".join(ids_to_sentence(yid, id2word)))\n",
    "            p = str(\"\".join(ids_to_sentence(pid, id2word)))\n",
    "            try:\n",
    "                modelp = int(p)\n",
    "                model_predictions.append(modelp)\n",
    "                y1 = int(y.split(\"$\")[0])\n",
    "                ground_truth.append(y1)\n",
    "            except ValueError:\n",
    "                try:\n",
    "                    modelp = int(p.split(\"$\")[0])\n",
    "                    ##if len(p) == len(p.split(\"$\"))+1:\n",
    "                    model_predictions.append(modelp)\n",
    "                    y1 = int(y.split(\"$\")[0])\n",
    "                    ground_truth.append(y1)\n",
    "                except:\n",
    "                    invalid_number_prediction_count = invalid_number_prediction_count + 1\n",
    "        \n",
    "    all_model_predictions.append(model_predictions)\n",
    "    all_ground_truth.append(ground_truth)\n",
    "    invalid_number_prediction_counts.append(invalid_number_prediction_count)\n",
    "            \n",
    "print('\\n...training finished.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate results\n",
    "\n",
    "Because our task is simple and the output is straight-forward, we will use [MAE](https://en.wikipedia.org/wiki/Mean_absolute_error) metric to evaluate the trained model during the epochs. Compute the value of the metric for the output from each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_absolute_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20000 20000\n",
      "Epoch: 1, MAE: 784.005900, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 2, MAE: 367.797450, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 3, MAE: 285.398700, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 4, MAE: 209.980900, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 5, MAE: 195.420800, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 6, MAE: 136.618750, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 7, MAE: 123.433450, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 8, MAE: 55.581850, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 9, MAE: 64.398450, Invalid numbers: 0\n",
      "20000 20000\n",
      "Epoch: 10, MAE: 40.827550, Invalid numbers: 0\n"
     ]
    }
   ],
   "source": [
    "for i, (gts, predictions, invalid_number_prediction_count) in enumerate(zip(all_ground_truth,\n",
    "                                                                            all_model_predictions,\n",
    "                                                                            invalid_number_prediction_counts), 1):\n",
    "    print(len(gts), len(predictions))\n",
    "    mae = mean_absolute_error(y_true=gts, y_pred=predictions, sample_weight=None, multioutput='uniform_average')\n",
    "    print(\"Epoch: %i, MAE: %f, Invalid numbers: %i\" % (i, mae, invalid_number_prediction_count))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}