01_LSTM_N2N.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling Stock Market Sentiment with LSTMs and TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will build a Long Short Term Memory (LSTM) Network to predict the stock market sentiment based on a comment about the market."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the following libraries for our analysis:\n",
    "\n",
    "* numpy - numerical computing library used to work with our data\n",
    "* pandas - data analysis library used to read in our data from csv\n",
    "* tensorflow - deep learning framework used for modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also be using the python Counter object for counting our vocabulary items and we have a util module that extracts away a lot of the details of our data processing. Please read through the util.py to get a better understanding of how to preprocess the data for analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import utils as utl\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Processing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train the model using messages tagged with SPY, the S&P 500 index fund, from [StockTwits.com](https://www.stocktwits.com). StockTwits is a social media network for traders and investors to share their views about the stock market. When a user posts a message, they tag the relevant stock ticker ($SPY in our case) and have the option to tag the messages with their sentiment – “bullish” if they believe the stock will go up and “bearish” if they believe the stock will go down.\n",
    "\n",
    "Our dataset consists of approximately 100,000 messages posted in 2017 that are tagged with $SPY where the user indicated their sentiment. Before we get to our LSTM Network we have to perform some processing on our data to get it ready for modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read and View Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we simply read in our data using pandas, pull out our message and sentiment data into numpy arrays. Let's also take a look at a few samples to get familiar with the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Messages: $SPY crazy day so far!... Sentiment: bearish\n",
      "Messages: $SPY Will make a new ATH this week. Watch it!... Sentiment: bullish\n",
      "Messages: $SPY $DJIA white elephant in room is $AAPL. Up 14% since election. Strong headwinds w/Trump trade & Strong dollar. How many 7's do you see?... Sentiment: bearish\n",
      "Messages: $SPY blocks above. We break above them We should push to double top... Sentiment: bullish\n",
      "Messages: $SPY Nothing happening in the market today, guess I'll go to the store and spend some $.... Sentiment: bearish\n",
      "Messages: $SPY What an easy call. Good jobs report: good economy, markets go up.  Bad jobs report: no more rate hikes, markets go up.  Win-win.... Sentiment: bullish\n",
      "Messages: $SPY BS market.... Sentiment: bullish\n",
      "Messages: $SPY this rally all the cheerleaders were screaming about this morning is pretty weak. I keep adding 2 my short at all spikes... Sentiment: bearish\n",
      "Messages: $SPY Dollar ripping higher!... Sentiment: bearish\n",
      "Messages: $SPY no reason to go down !... Sentiment: bullish\n"
     ]
    }
   ],
   "source": [
    "# read data from csv file\n",
    "data = pd.read_csv(\"data/StockTwits_SPY_Sentiment_2017.gz\",\n",
    "                   encoding=\"utf-8\",\n",
    "                   compression=\"gzip\",\n",
    "                   index_col=0)\n",
    "\n",
    "# get messages and sentiment labels\n",
    "messages = data.message.values\n",
    "labels = data.sentiment.values\n",
    "\n",
    "# View sample of messages with sentiment\n",
    "\n",
    "for i in range(10):\n",
    "    print(\"Messages: {}...\".format(messages[i]),\n",
    "          \"Sentiment: {}\".format(labels[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess Messages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with raw text data often requires preprocessing the text in some fashion to normalize for context. In our case we want to normalize for known unique \"entities\" that appear within messages that carry a similar contextual meaning when analyzing sentiment. This means we want to replace references to specific stock tickers, user names, url links or numbers with a special token identifying the \"entity\". Here we will also make everything lower case and remove punctuation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "messages = np.array([utl.preprocess_ST_message(message) for message in messages])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generate Vocab to Index Mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To work with raw text we need some encoding from words to numbers for our algorithm to work with the inputs. The first step of doing this is keeping a collection of our full vocabularly and creating a mapping of each word to a unique index. We will use this word to index mapping in a little bit to prep out messages for analysis. \n",
    "\n",
    "Note that in practice we may want to only include the vocabularly from our training set here to account for the fact that we will likely see new words when our model is out in the wild when we are assessing the results on our validation and test sets. Here, for simplicity and demonstration purposes, we will use our entire data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "full_lexicon = \" \".join(messages).split()\n",
    "vocab_to_int, int_to_vocab = utl.create_lookup_tables(full_lexicon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check Message Lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also want to get a sense of the distribution of the length of our inputs. We check for the longest and average messages. We will need to make our input length uniform to feed the data into our model so later we will have some decisions to make about possibly truncating some of the longer messages if they are too long. We also notice that one message has no content remaining after we preprocessed the data, so we will remove this message from our data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zero-length messages: 1\n",
      "Maximum message length: 244\n",
      "Average message length: 78.21856920395598\n"
     ]
    }
   ],
   "source": [
    "messages_lens = Counter([len(x) for x in messages])\n",
    "print(\"Zero-length messages: {}\".format(messages_lens[0]))\n",
    "print(\"Maximum message length: {}\".format(max(messages_lens)))\n",
    "print(\"Average message length: {}\".format(np.mean([len(x) for x in messages])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "messages, labels = utl.drop_empty_messages(messages, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Encode Messages and Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Earlier we mentioned that we need to \"translate\" our text to number for our algorithm to take in as inputs. We call this translation an encoding. We encode our messages to sequences of numbers where each nummber is the word index from the mapping we made earlier. The phrase \"I am bullish\" would now look something like [1, 234, 5345] where each number is the index for the respective word in the message. For our sentiment labels we will simply encode \"bearish\" as 0 and \"bullish\" as 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "messages = utl.encode_ST_messages(messages, vocab_to_int)\n",
    "labels = utl.encode_ST_labels(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pad Messages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing we need to do is make our message inputs the same length. In our case, the longest message is 244 words. LSTMs can usually handle sequence inputs up to 500 items in length so we won't truncate any of the messages here. We need to Zero Pad the rest of the messages that are shorter. We will use a left padding that will pad all of the messages that are shorter than 244 words with 0s at the beginning. So our encoded \"I am bullish\" messages goes from [1, 234, 5345] (length 3) to [0, 0, 0, 0, 0, 0, ... , 0, 0, 1, 234, 5345] (length 244)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "messages = utl.zero_pad_messages(messages, seq_len=244)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train, Test, Validation Split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing we do is split our data into tranining, validation and test sets and observe the size of each set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Set Size\n",
      "Train set: \t\t(77572, 244) \n",
      "Validation set: \t(9697, 244) \n",
      "Test set: \t\t(9697, 244)\n"
     ]
    }
   ],
   "source": [
    "train_x, val_x, test_x, train_y, val_y, test_y = utl.train_val_test_split(messages, labels, split_frac=0.80)\n",
    "\n",
    "print(\"Data Set Size\")\n",
    "print(\"Train set: \\t\\t{}\".format(train_x.shape), \n",
    "      \"\\nValidation set: \\t{}\".format(val_x.shape),\n",
    "      \"\\nTest set: \\t\\t{}\".format(test_x.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building and Training our LSTM Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will define a number of functions that will construct the items in our network. We will then use these functions to build and train our network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we simply define a function to build TensorFlow Placeholders for our message sequences, our labels and a variable called keep probability associated with drop out (we will talk more about this later). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def model_inputs():\n",
    "    \"\"\"\n",
    "    Create the model inputs\n",
    "    \"\"\"\n",
    "    inputs_ = tf.placeholder(tf.int32, [None, None], name='inputs')\n",
    "    labels_ = tf.placeholder(tf.int32, [None, None], name='labels')\n",
    "    keep_prob_ = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    \n",
    "    return inputs_, labels_, keep_prob_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Embedding Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TensorFlow the word embeddings are represented as a vocabulary size x embedding size matrix and will learn these weights during our training process. The embedding lookup is then just a simple lookup from our embedding matrix based on the index of the current word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_embedding_layer(inputs_, vocab_size, embed_size):\n",
    "    \"\"\"\n",
    "    Create the embedding layer\n",
    "    \"\"\"\n",
    "    embedding = tf.Variable(tf.random_uniform((vocab_size, embed_size), -1, 1))\n",
    "    embed = tf.nn.embedding_lookup(embedding, inputs_)\n",
    "    \n",
    "    return embed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### LSTM Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow makes it extremely easy to build LSTM Layers and stack them on top of each other. We represent each LSTM layer as a BasicLSTMCell and keep these in a list to stack them together later. Here we will define a list with our LSTM layer sizes and the number of layers. \n",
    "\n",
    "We then take each of these LSTM layers and wrap them in a Dropout Layer. Dropout is a regularization technique using in Neural Networks in which any individual node has a probability of “dropping out” of the network during a given iteration of learning. The makes the model more generalizable by ensuring that it is not too dependent on any given nodes. \n",
    "\n",
    "Finally, we stack these layers using a MultiRNNCell, generate a zero initial state and connect our stacked LSTM layer to our word embedding inputs using dynamic_rnn. Here we track the output and the final state of the LSTM cell, which we will need to pass between mini-batches during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_lstm_layers(lstm_sizes, embed, keep_prob_, batch_size):\n",
    "    \"\"\"\n",
    "    Create the LSTM layers\n",
    "    \"\"\"\n",
    "    lstms = [tf.contrib.rnn.BasicLSTMCell(size) for size in lstm_sizes]\n",
    "    # Add dropout to the cell\n",
    "    drops = [tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob_) for lstm in lstms]\n",
    "    # Stack up multiple LSTM layers, for deep learning\n",
    "    cell = tf.contrib.rnn.MultiRNNCell(drops)\n",
    "    # Getting an initial state of all zeros\n",
    "    initial_state = cell.zero_state(batch_size, tf.float32)\n",
    "    \n",
    "    lstm_outputs, final_state = tf.nn.dynamic_rnn(cell, embed, initial_state=initial_state)\n",
    "    \n",
    "    return initial_state, lstm_outputs, cell, final_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loss Function and Optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we get our predictions by passing the final output of the LSTM layers to a sigmoid activation function via a Tensorflow fully connected layer.  we only care to use the final output for making predictions so we pull this out using the [: , -1] indexing on our LSTM outputs and pass it through a sigmoid activation function to make the predictions. We pass then pass these predictions to our mean squared error loss function and use the Adadelta Optimizer to minimize the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_cost_fn_and_opt(lstm_outputs, labels_, learning_rate):\n",
    "    \"\"\"\n",
    "    Create the Loss function and Optimizer\n",
    "    \"\"\"\n",
    "    predictions = tf.contrib.layers.fully_connected(lstm_outputs[:, -1], 1, activation_fn=tf.sigmoid)\n",
    "    loss = tf.losses.mean_squared_error(labels_, predictions)\n",
    "    optimzer = tf.train.AdadeltaOptimizer(learning_rate).minimize(loss)\n",
    "    \n",
    "    return predictions, loss, optimzer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we define our accuracy metric for assessing the model performance across our training, validation and test sets. Even though accuracy is just a calculation based on results, everything in TensorFlow is part of a Computation Graph. Therefore, we need to define our loss and accuracy nodes in the context of the rest of our network graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_accuracy(predictions, labels_):\n",
    "    \"\"\"\n",
    "    Create accuracy\n",
    "    \"\"\"\n",
    "    correct_pred = tf.equal(tf.cast(tf.round(predictions), tf.int32), labels_)\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
    "    \n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are finally ready to build and train our LSTM Network! First, we call each of our each of the functions we have defined to construct the network. Then we define a Saver to be able to write our model to disk to load for future use. Finally, we call a Tensorflow Session to train the model over a predefined number of epochs using mini-batches. At the end of each epoch we will print the loss, training accuracy and validation accuracy to monitor the results as we train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_and_train_network(lstm_sizes, vocab_size, embed_size, epochs, batch_size,\n",
    "                            learning_rate, keep_prob, train_x, val_x, train_y, val_y):\n",
    "    \n",
    "    inputs_, labels_, keep_prob_ = model_inputs()\n",
    "    embed = build_embedding_layer(inputs_, vocab_size, embed_size)\n",
    "    initial_state, lstm_outputs, lstm_cell, final_state = build_lstm_layers(lstm_sizes, embed, keep_prob_, batch_size)\n",
    "    predictions, loss, optimizer = build_cost_fn_and_opt(lstm_outputs, labels_, learning_rate)\n",
    "    accuracy = build_accuracy(predictions, labels_)\n",
    "    \n",
    "    saver = tf.train.Saver()\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        \n",
    "        sess.run(tf.global_variables_initializer())\n",
    "        n_batches = len(train_x)//batch_size\n",
    "        for e in range(epochs):\n",
    "            state = sess.run(initial_state)\n",
    "            \n",
    "            train_acc = []\n",
    "            for ii, (x, y) in enumerate(utl.get_batches(train_x, train_y, batch_size), 1):\n",
    "                feed = {inputs_: x,\n",
    "                        labels_: y[:, None],\n",
    "                        keep_prob_: keep_prob,\n",
    "                        initial_state: state}\n",
    "                loss_, state, _,  batch_acc = sess.run([loss, final_state, optimizer, accuracy], feed_dict=feed)\n",
    "                train_acc.append(batch_acc)\n",
    "                \n",
    "                if (ii + 1) % n_batches == 0:\n",
    "                    \n",
    "                    val_acc = []\n",
    "                    val_state = sess.run(lstm_cell.zero_state(batch_size, tf.float32))\n",
    "                    for xx, yy in utl.get_batches(val_x, val_y, batch_size):\n",
    "                        feed = {inputs_: xx,\n",
    "                                labels_: yy[:, None],\n",
    "                                keep_prob_: 1,\n",
    "                                initial_state: val_state}\n",
    "                        val_batch_acc, val_state = sess.run([accuracy, final_state], feed_dict=feed)\n",
    "                        val_acc.append(val_batch_acc)\n",
    "                    \n",
    "                    print(\"Epoch: {}/{}...\".format(e+1, epochs),\n",
    "                          \"Batch: {}/{}...\".format(ii+1, n_batches),\n",
    "                          \"Train Loss: {:.3f}...\".format(loss_),\n",
    "                          \"Train Accruacy: {:.3f}...\".format(np.mean(train_acc)),\n",
    "                          \"Val Accuracy: {:.3f}\".format(np.mean(val_acc)))\n",
    "    \n",
    "        saver.save(sess, \"checkpoints/sentiment.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we define our model hyper parameters. We will build a 2 Layer LSTM Newtork with hidden layer sizes of 128 and 64 respectively. We will use an embedding size of 300 and train over 50 epochs with mini-batches of size 256. We will use an initial learning rate of 0.1, though our Adadelta Optimizer will adapt this over time, and a keep probability of 0.5. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define Inputs and Hyperparameters\n",
    "lstm_sizes = [128, 64]\n",
    "vocab_size = len(vocab_to_int) + 1 #add one for padding\n",
    "embed_size = 300\n",
    "epochs = 50\n",
    "batch_size = 256\n",
    "learning_rate = 0.1\n",
    "keep_prob = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and now we train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/50... Batch: 303/303... Train Loss: 0.247... Train Accruacy: 0.562... Val Accuracy: 0.578\n",
      "Epoch: 2/50... Batch: 303/303... Train Loss: 0.245... Train Accruacy: 0.583... Val Accuracy: 0.596\n",
      "Epoch: 3/50... Batch: 303/303... Train Loss: 0.247... Train Accruacy: 0.597... Val Accuracy: 0.617\n",
      "Epoch: 4/50... Batch: 303/303... Train Loss: 0.240... Train Accruacy: 0.610... Val Accuracy: 0.627\n",
      "Epoch: 5/50... Batch: 303/303... Train Loss: 0.238... Train Accruacy: 0.620... Val Accuracy: 0.632\n",
      "Epoch: 6/50... Batch: 303/303... Train Loss: 0.234... Train Accruacy: 0.632... Val Accuracy: 0.642\n",
      "Epoch: 7/50... Batch: 303/303... Train Loss: 0.230... Train Accruacy: 0.636... Val Accuracy: 0.648\n",
      "Epoch: 8/50... Batch: 303/303... Train Loss: 0.227... Train Accruacy: 0.641... Val Accuracy: 0.653\n",
      "Epoch: 9/50... Batch: 303/303... Train Loss: 0.223... Train Accruacy: 0.646... Val Accuracy: 0.656\n",
      "Epoch: 10/50... Batch: 303/303... Train Loss: 0.221... Train Accruacy: 0.652... Val Accuracy: 0.659\n",
      "Epoch: 11/50... Batch: 303/303... Train Loss: 0.225... Train Accruacy: 0.656... Val Accuracy: 0.663\n",
      "Epoch: 12/50... Batch: 303/303... Train Loss: 0.220... Train Accruacy: 0.661... Val Accuracy: 0.666\n",
      "Epoch: 13/50... Batch: 303/303... Train Loss: 0.215... Train Accruacy: 0.665... Val Accuracy: 0.668\n",
      "Epoch: 14/50... Batch: 303/303... Train Loss: 0.213... Train Accruacy: 0.668... Val Accuracy: 0.670\n",
      "Epoch: 15/50... Batch: 303/303... Train Loss: 0.210... Train Accruacy: 0.669... Val Accuracy: 0.673\n",
      "Epoch: 16/50... Batch: 303/303... Train Loss: 0.213... Train Accruacy: 0.673... Val Accuracy: 0.675\n",
      "Epoch: 17/50... Batch: 303/303... Train Loss: 0.212... Train Accruacy: 0.675... Val Accuracy: 0.676\n",
      "Epoch: 18/50... Batch: 303/303... Train Loss: 0.206... Train Accruacy: 0.681... Val Accuracy: 0.679\n",
      "Epoch: 19/50... Batch: 303/303... Train Loss: 0.208... Train Accruacy: 0.683... Val Accuracy: 0.681\n",
      "Epoch: 20/50... Batch: 303/303... Train Loss: 0.202... Train Accruacy: 0.684... Val Accuracy: 0.684\n",
      "Epoch: 21/50... Batch: 303/303... Train Loss: 0.206... Train Accruacy: 0.685... Val Accuracy: 0.686\n",
      "Epoch: 22/50... Batch: 303/303... Train Loss: 0.204... Train Accruacy: 0.689... Val Accuracy: 0.689\n",
      "Epoch: 23/50... Batch: 303/303... Train Loss: 0.195... Train Accruacy: 0.690... Val Accuracy: 0.691\n",
      "Epoch: 24/50... Batch: 303/303... Train Loss: 0.200... Train Accruacy: 0.695... Val Accuracy: 0.692\n",
      "Epoch: 25/50... Batch: 303/303... Train Loss: 0.195... Train Accruacy: 0.696... Val Accuracy: 0.694\n",
      "Epoch: 26/50... Batch: 303/303... Train Loss: 0.200... Train Accruacy: 0.698... Val Accuracy: 0.695\n",
      "Epoch: 27/50... Batch: 303/303... Train Loss: 0.197... Train Accruacy: 0.701... Val Accuracy: 0.695\n",
      "Epoch: 28/50... Batch: 303/303... Train Loss: 0.199... Train Accruacy: 0.703... Val Accuracy: 0.698\n",
      "Epoch: 29/50... Batch: 303/303... Train Loss: 0.187... Train Accruacy: 0.704... Val Accuracy: 0.698\n",
      "Epoch: 30/50... Batch: 303/303... Train Loss: 0.190... Train Accruacy: 0.708... Val Accuracy: 0.701\n",
      "Epoch: 31/50... Batch: 303/303... Train Loss: 0.189... Train Accruacy: 0.708... Val Accuracy: 0.702\n",
      "Epoch: 32/50... Batch: 303/303... Train Loss: 0.184... Train Accruacy: 0.710... Val Accuracy: 0.704\n",
      "Epoch: 33/50... Batch: 303/303... Train Loss: 0.195... Train Accruacy: 0.714... Val Accuracy: 0.704\n",
      "Epoch: 34/50... Batch: 303/303... Train Loss: 0.190... Train Accruacy: 0.715... Val Accuracy: 0.704\n",
      "Epoch: 35/50... Batch: 303/303... Train Loss: 0.186... Train Accruacy: 0.714... Val Accuracy: 0.707\n",
      "Epoch: 36/50... Batch: 303/303... Train Loss: 0.178... Train Accruacy: 0.717... Val Accuracy: 0.707\n",
      "Epoch: 37/50... Batch: 303/303... Train Loss: 0.183... Train Accruacy: 0.722... Val Accuracy: 0.707\n",
      "Epoch: 38/50... Batch: 303/303... Train Loss: 0.181... Train Accruacy: 0.721... Val Accuracy: 0.710\n",
      "Epoch: 39/50... Batch: 303/303... Train Loss: 0.181... Train Accruacy: 0.723... Val Accuracy: 0.712\n",
      "Epoch: 40/50... Batch: 303/303... Train Loss: 0.179... Train Accruacy: 0.726... Val Accuracy: 0.712\n",
      "Epoch: 41/50... Batch: 303/303... Train Loss: 0.180... Train Accruacy: 0.726... Val Accuracy: 0.713\n",
      "Epoch: 42/50... Batch: 303/303... Train Loss: 0.177... Train Accruacy: 0.729... Val Accuracy: 0.714\n",
      "Epoch: 43/50... Batch: 303/303... Train Loss: 0.176... Train Accruacy: 0.731... Val Accuracy: 0.714\n",
      "Epoch: 44/50... Batch: 303/303... Train Loss: 0.180... Train Accruacy: 0.732... Val Accuracy: 0.716\n",
      "Epoch: 45/50... Batch: 303/303... Train Loss: 0.169... Train Accruacy: 0.734... Val Accuracy: 0.716\n",
      "Epoch: 46/50... Batch: 303/303... Train Loss: 0.173... Train Accruacy: 0.735... Val Accuracy: 0.717\n",
      "Epoch: 47/50... Batch: 303/303... Train Loss: 0.170... Train Accruacy: 0.736... Val Accuracy: 0.717\n",
      "Epoch: 48/50... Batch: 303/303... Train Loss: 0.173... Train Accruacy: 0.739... Val Accuracy: 0.718\n",
      "Epoch: 49/50... Batch: 303/303... Train Loss: 0.175... Train Accruacy: 0.740... Val Accuracy: 0.717\n",
      "Epoch: 50/50... Batch: 303/303... Train Loss: 0.175... Train Accruacy: 0.745... Val Accuracy: 0.718\n"
     ]
    }
   ],
   "source": [
    "with tf.Graph().as_default():\n",
    "    build_and_train_network(lstm_sizes, vocab_size, embed_size, epochs, batch_size,\n",
    "                            learning_rate, keep_prob, train_x, val_x, train_y, val_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing our Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing we want to do is check the model accuracy on our testing data to make sure it is in line with expecations. We build the Computational Graph just like we did before, however, now instead of training we restore our saved model from our checkpoint directory and then run our test data through the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_network(model_dir, batch_size, test_x, test_y):\n",
    "    \n",
    "    inputs_, labels_, keep_prob_ = model_inputs()\n",
    "    embed = build_embedding_layer(inputs_, vocab_size, embed_size)\n",
    "    initial_state, lstm_outputs, lstm_cell, final_state = build_lstm_layers(lstm_sizes, embed, keep_prob_, batch_size)\n",
    "    predictions, loss, optimizer = build_cost_fn_and_opt(lstm_outputs, labels_, learning_rate)\n",
    "    accuracy = build_accuracy(predictions, labels_)\n",
    "    \n",
    "    saver = tf.train.Saver()\n",
    "    \n",
    "    test_acc = []\n",
    "    with tf.Session() as sess:\n",
    "        saver.restore(sess, tf.train.latest_checkpoint(model_dir))\n",
    "        test_state = sess.run(lstm_cell.zero_state(batch_size, tf.float32))\n",
    "        for ii, (x, y) in enumerate(utl.get_batches(test_x, test_y, batch_size), 1):\n",
    "            feed = {inputs_: x,\n",
    "                    labels_: y[:, None],\n",
    "                    keep_prob_: 1,\n",
    "                    initial_state: test_state}\n",
    "            batch_acc, test_state = sess.run([accuracy, final_state], feed_dict=feed)\n",
    "            test_acc.append(batch_acc)\n",
    "        print(\"Test Accuracy: {:.3f}\".format(np.mean(test_acc)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from checkpoints/sentiment.ckpt\n",
      "Test Accuracy: 0.717\n"
     ]
    }
   ],
   "source": [
    "with tf.Graph().as_default():\n",
    "    test_network('checkpoints', batch_size, test_x, test_y)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [TensorFlow]",
   "language": "python",
   "name": "Python [TensorFlow]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}