rossman.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rossmann Store Sales Forecast Using Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Rossmann](images/photo.png)\n",
    "\n",
    "The objective of this notebook is to forecast sales using store, promotion, and competitor data using deep neural network and explain the process to the C-Suite inin layman's terms. The data was obtained from [Kaggle](https://www.kaggle.com/c/rossmann-store-sales/data).\n",
    "\n",
    "This notebook is also available at https://git.io/v5oCA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rossmann operates over 3,000 drug stores in 7 European countries. Currently, Rossmann store managers are tasked with predicting their daily sales for up to six weeks in advance. Store sales are influenced by many factors, including promotions, competition, school and state holidays, seasonality, and locality. With thousands of individual managers predicting sales based on their unique circumstances, the accuracy of results can be quite varied.\n",
    "\n",
    "The objective of this project is to predict daily sales for 1,115 stores located across Germany. Reliable sales forecasts enable store managers to create effective staff schedules that increase productivity and motivation. By creating a robust prediction model, store managers can stay focused on their customers and their teams.\n",
    "\n",
    "The developed model achieved high accuracy with Root Mean Squared Percentage Error (RMSPE) 0.23. The formula is:\n",
    "\n",
    "$$\n",
    "\\textrm{RMSE} = \\sqrt{\\frac{1}{n} \\sum_{i=1}^{n} \\left(\\frac{\\hat{y}_i - y_i}{y_i}\\right)^2}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $\\hat{y}_i$ is the actual sales of one store on a particular day\n",
    "- $y_i$ is the corresponding sales predicted by the model\n",
    "- $n$ is the number of predictions made\n",
    "\n",
    "Below is a high-level overview of the steps taken to develop such robust model:\n",
    "1. Transform the raw data into format which is accepted by the machine learning algorithm.\n",
    "2. Build a predictive model by feeding the transformed data to the algorithm.\n",
    "3. Use a seperate set of raw data to evaluate the performnace of the derived model.\n",
    "\n",
    "In particular, the model we uses is deep nerual network, which is also called deep learning and is the underlying model of cutting-edge applications like Siri and AlphaGo. A simplified illustration of an nerual network:\n",
    "\n",
    "![a simplified illustration of neural network](images/neural_network.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the modules to be used by this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:02.850466Z",
     "start_time": "2017-09-09T12:29:01.195351Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import math, keras, datetime, pandas as pd, numpy as np, keras.backend as K\n",
    "import matplotlib.pyplot as plt, operator, random, pickle\n",
    "from utils2 import *\n",
    "from isoweek import Week\n",
    "from pandas_summary import DataFrameSummary\n",
    "from IPython.display import SVG\n",
    "from numpy import mean, sqrt, square, arange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the preferred options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:02.870306Z",
     "start_time": "2017-09-09T12:29:02.854567Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/terry/notebook/rossmann/data\n"
     ]
    }
   ],
   "source": [
    "np.set_printoptions(threshold=50, edgeitems=20)\n",
    "%matplotlib inline\n",
    "%cd data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following tables are available in the datasets:\n",
    "\n",
    "| Table        | Table Description                                                                                               |\n",
    "|--------------|-----------------------------------------------------------------------------------------------------------------|\n",
    "| train        | training set: store information on a daily basis, tracking things like sales, customers, whether that day was a holiday, etc. |\n",
    "| store        | general information about the store including competition, etc.                                                 |\n",
    "| store_states | maps store to state it is in                                                                                    |\n",
    "| state_names  | maps state abbreviations to names                                                                               |\n",
    "| googletrend  | trend data for particular week/state                                                                            |\n",
    "| weather      | weather conditions for each state                                                                               |\n",
    "\n",
    "Import the tables to the environemnts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:03.442382Z",
     "start_time": "2017-09-09T12:29:02.874344Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "table_names = ['train', 'store', 'store_states', 'state_names', 'googletrend', 'weather']\n",
    "train, store, store_states, state_names, googletrend, weather = [pd.read_csv(fname+'.csv', low_memory=False) for fname in table_names]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the first 5 rows and statsitics of each table to get a quick understanding of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.098431Z",
     "start_time": "2017-09-09T12:29:03.446701Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sales</th>\n",
       "      <th>Customers</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>5263</td>\n",
       "      <td>555</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>6064</td>\n",
       "      <td>625</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>8314</td>\n",
       "      <td>821</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>13995</td>\n",
       "      <td>1498</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>4822</td>\n",
       "      <td>559</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store  DayOfWeek        Date  Sales  Customers  Open  Promo StateHoliday  \\\n",
       "0      1          5  2015-07-31   5263        555     1      1            0   \n",
       "1      2          5  2015-07-31   6064        625     1      1            0   \n",
       "2      3          5  2015-07-31   8314        821     1      1            0   \n",
       "3      4          5  2015-07-31  13995       1498     1      1            0   \n",
       "4      5          5  2015-07-31   4822        559     1      1            0   \n",
       "\n",
       "   SchoolHoliday  \n",
       "0              1  \n",
       "1              1  \n",
       "2              1  \n",
       "3              1  \n",
       "4              1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sales</th>\n",
       "      <th>Customers</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1.01721e+06</td>\n",
       "      <td>1.01721e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.01721e+06</td>\n",
       "      <td>1.01721e+06</td>\n",
       "      <td>1.01721e+06</td>\n",
       "      <td>1.01721e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.01721e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>558.43</td>\n",
       "      <td>3.99834</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5773.82</td>\n",
       "      <td>633.146</td>\n",
       "      <td>0.830107</td>\n",
       "      <td>0.381515</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.178647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>321.909</td>\n",
       "      <td>1.99739</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3849.93</td>\n",
       "      <td>464.412</td>\n",
       "      <td>0.375539</td>\n",
       "      <td>0.485759</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.383056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>280</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3727</td>\n",
       "      <td>405</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>558</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5744</td>\n",
       "      <td>609</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>838</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7856</td>\n",
       "      <td>837</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1115</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>41551</td>\n",
       "      <td>7388</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counts</th>\n",
       "      <td>1017209</td>\n",
       "      <td>1017209</td>\n",
       "      <td>1017209</td>\n",
       "      <td>1017209</td>\n",
       "      <td>1017209</td>\n",
       "      <td>1017209</td>\n",
       "      <td>1017209</td>\n",
       "      <td>1017209</td>\n",
       "      <td>1017209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uniques</th>\n",
       "      <td>1115</td>\n",
       "      <td>7</td>\n",
       "      <td>942</td>\n",
       "      <td>21734</td>\n",
       "      <td>4086</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing_perc</th>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>types</th>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>categorical</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>bool</td>\n",
       "      <td>bool</td>\n",
       "      <td>categorical</td>\n",
       "      <td>bool</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Store    DayOfWeek         Date        Sales    Customers  \\\n",
       "count         1.01721e+06  1.01721e+06          NaN  1.01721e+06  1.01721e+06   \n",
       "mean               558.43      3.99834          NaN      5773.82      633.146   \n",
       "std               321.909      1.99739          NaN      3849.93      464.412   \n",
       "min                     1            1          NaN            0            0   \n",
       "25%                   280            2          NaN         3727          405   \n",
       "50%                   558            4          NaN         5744          609   \n",
       "75%                   838            6          NaN         7856          837   \n",
       "max                  1115            7          NaN        41551         7388   \n",
       "counts            1017209      1017209      1017209      1017209      1017209   \n",
       "uniques              1115            7          942        21734         4086   \n",
       "missing                 0            0            0            0            0   \n",
       "missing_perc           0%           0%           0%           0%           0%   \n",
       "types             numeric      numeric  categorical      numeric      numeric   \n",
       "\n",
       "                     Open        Promo StateHoliday SchoolHoliday  \n",
       "count         1.01721e+06  1.01721e+06          NaN   1.01721e+06  \n",
       "mean             0.830107     0.381515          NaN      0.178647  \n",
       "std              0.375539     0.485759          NaN      0.383056  \n",
       "min                     0            0          NaN             0  \n",
       "25%                     1            0          NaN             0  \n",
       "50%                     1            0          NaN             0  \n",
       "75%                     1            1          NaN             0  \n",
       "max                     1            1          NaN             1  \n",
       "counts            1017209      1017209      1017209       1017209  \n",
       "uniques                 2            2            4             2  \n",
       "missing                 0            0            0             0  \n",
       "missing_perc           0%           0%           0%            0%  \n",
       "types                bool         bool  categorical          bool  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(train.head())\n",
    "display(DataFrameSummary(train).summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.162444Z",
     "start_time": "2017-09-09T12:29:04.102414Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>StoreType</th>\n",
       "      <th>Assortment</th>\n",
       "      <th>CompetitionDistance</th>\n",
       "      <th>CompetitionOpenSinceMonth</th>\n",
       "      <th>CompetitionOpenSinceYear</th>\n",
       "      <th>Promo2</th>\n",
       "      <th>Promo2SinceWeek</th>\n",
       "      <th>Promo2SinceYear</th>\n",
       "      <th>PromoInterval</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "      <td>1270.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>570.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>1</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>14130.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2006.0</td>\n",
       "      <td>1</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>c</td>\n",
       "      <td>c</td>\n",
       "      <td>620.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>29910.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store StoreType Assortment  CompetitionDistance  CompetitionOpenSinceMonth  \\\n",
       "0      1         c          a               1270.0                        9.0   \n",
       "1      2         a          a                570.0                       11.0   \n",
       "2      3         a          a              14130.0                       12.0   \n",
       "3      4         c          c                620.0                        9.0   \n",
       "4      5         a          a              29910.0                        4.0   \n",
       "\n",
       "   CompetitionOpenSinceYear  Promo2  Promo2SinceWeek  Promo2SinceYear  \\\n",
       "0                    2008.0       0              NaN              NaN   \n",
       "1                    2007.0       1             13.0           2010.0   \n",
       "2                    2006.0       1             14.0           2011.0   \n",
       "3                    2009.0       0              NaN              NaN   \n",
       "4                    2015.0       0              NaN              NaN   \n",
       "\n",
       "     PromoInterval  \n",
       "0              NaN  \n",
       "1  Jan,Apr,Jul,Oct  \n",
       "2  Jan,Apr,Jul,Oct  \n",
       "3              NaN  \n",
       "4              NaN  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>StoreType</th>\n",
       "      <th>Assortment</th>\n",
       "      <th>CompetitionDistance</th>\n",
       "      <th>CompetitionOpenSinceMonth</th>\n",
       "      <th>CompetitionOpenSinceYear</th>\n",
       "      <th>Promo2</th>\n",
       "      <th>Promo2SinceWeek</th>\n",
       "      <th>Promo2SinceYear</th>\n",
       "      <th>PromoInterval</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1115</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1112</td>\n",
       "      <td>761</td>\n",
       "      <td>761</td>\n",
       "      <td>1115</td>\n",
       "      <td>571</td>\n",
       "      <td>571</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>558</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5404.9</td>\n",
       "      <td>7.2247</td>\n",
       "      <td>2008.67</td>\n",
       "      <td>0.512108</td>\n",
       "      <td>23.5954</td>\n",
       "      <td>2011.76</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>322.017</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7663.17</td>\n",
       "      <td>3.21235</td>\n",
       "      <td>6.19598</td>\n",
       "      <td>0.500078</td>\n",
       "      <td>14.142</td>\n",
       "      <td>1.67494</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>1900</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2009</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>279.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>717.5</td>\n",
       "      <td>4</td>\n",
       "      <td>2006</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>2011</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>558</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2325</td>\n",
       "      <td>8</td>\n",
       "      <td>2010</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>2012</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>836.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6882.5</td>\n",
       "      <td>10</td>\n",
       "      <td>2013</td>\n",
       "      <td>1</td>\n",
       "      <td>37</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1115</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>75860</td>\n",
       "      <td>12</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>2015</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counts</th>\n",
       "      <td>1115</td>\n",
       "      <td>1115</td>\n",
       "      <td>1115</td>\n",
       "      <td>1112</td>\n",
       "      <td>761</td>\n",
       "      <td>761</td>\n",
       "      <td>1115</td>\n",
       "      <td>571</td>\n",
       "      <td>571</td>\n",
       "      <td>571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uniques</th>\n",
       "      <td>1115</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>654</td>\n",
       "      <td>12</td>\n",
       "      <td>23</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>354</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>544</td>\n",
       "      <td>544</td>\n",
       "      <td>544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing_perc</th>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0.27%</td>\n",
       "      <td>31.75%</td>\n",
       "      <td>31.75%</td>\n",
       "      <td>0%</td>\n",
       "      <td>48.79%</td>\n",
       "      <td>48.79%</td>\n",
       "      <td>48.79%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>types</th>\n",
       "      <td>numeric</td>\n",
       "      <td>categorical</td>\n",
       "      <td>categorical</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>bool</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>categorical</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Store    StoreType   Assortment CompetitionDistance  \\\n",
       "count            1115          NaN          NaN                1112   \n",
       "mean              558          NaN          NaN              5404.9   \n",
       "std           322.017          NaN          NaN             7663.17   \n",
       "min                 1          NaN          NaN                  20   \n",
       "25%             279.5          NaN          NaN               717.5   \n",
       "50%               558          NaN          NaN                2325   \n",
       "75%             836.5          NaN          NaN              6882.5   \n",
       "max              1115          NaN          NaN               75860   \n",
       "counts           1115         1115         1115                1112   \n",
       "uniques          1115            4            3                 654   \n",
       "missing             0            0            0                   3   \n",
       "missing_perc       0%           0%           0%               0.27%   \n",
       "types         numeric  categorical  categorical             numeric   \n",
       "\n",
       "             CompetitionOpenSinceMonth CompetitionOpenSinceYear    Promo2  \\\n",
       "count                              761                      761      1115   \n",
       "mean                            7.2247                  2008.67  0.512108   \n",
       "std                            3.21235                  6.19598  0.500078   \n",
       "min                                  1                     1900         0   \n",
       "25%                                  4                     2006         0   \n",
       "50%                                  8                     2010         1   \n",
       "75%                                 10                     2013         1   \n",
       "max                                 12                     2015         1   \n",
       "counts                             761                      761      1115   \n",
       "uniques                             12                       23         2   \n",
       "missing                            354                      354         0   \n",
       "missing_perc                    31.75%                   31.75%        0%   \n",
       "types                          numeric                  numeric      bool   \n",
       "\n",
       "             Promo2SinceWeek Promo2SinceYear PromoInterval  \n",
       "count                    571             571           NaN  \n",
       "mean                 23.5954         2011.76           NaN  \n",
       "std                   14.142         1.67494           NaN  \n",
       "min                        1            2009           NaN  \n",
       "25%                       13            2011           NaN  \n",
       "50%                       22            2012           NaN  \n",
       "75%                       37            2013           NaN  \n",
       "max                       50            2015           NaN  \n",
       "counts                   571             571           571  \n",
       "uniques                   24               7             3  \n",
       "missing                  544             544           544  \n",
       "missing_perc          48.79%          48.79%        48.79%  \n",
       "types                numeric         numeric   categorical  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(store.head())\n",
    "display(DataFrameSummary(store).summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.202439Z",
     "start_time": "2017-09-09T12:29:04.166159Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>State</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>HE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>TH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>NW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>BE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>SN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store State\n",
       "0      1    HE\n",
       "1      2    TH\n",
       "2      3    NW\n",
       "3      4    BE\n",
       "4      5    SN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>State</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1115</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>558</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>322.017</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>279.5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>558</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>836.5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1115</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counts</th>\n",
       "      <td>1115</td>\n",
       "      <td>1115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uniques</th>\n",
       "      <td>1115</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing_perc</th>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>types</th>\n",
       "      <td>numeric</td>\n",
       "      <td>categorical</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Store        State\n",
       "count            1115          NaN\n",
       "mean              558          NaN\n",
       "std           322.017          NaN\n",
       "min                 1          NaN\n",
       "25%             279.5          NaN\n",
       "50%               558          NaN\n",
       "75%             836.5          NaN\n",
       "max              1115          NaN\n",
       "counts           1115         1115\n",
       "uniques          1115           12\n",
       "missing             0            0\n",
       "missing_perc       0%           0%\n",
       "types         numeric  categorical"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(store_states.head())\n",
    "display(DataFrameSummary(store_states).summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.238509Z",
     "start_time": "2017-09-09T12:29:04.206331Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StateName</th>\n",
       "      <th>State</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BadenWuerttemberg</td>\n",
       "      <td>BW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bayern</td>\n",
       "      <td>BY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Berlin</td>\n",
       "      <td>BE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brandenburg</td>\n",
       "      <td>BB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bremen</td>\n",
       "      <td>HB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           StateName State\n",
       "0  BadenWuerttemberg    BW\n",
       "1             Bayern    BY\n",
       "2             Berlin    BE\n",
       "3        Brandenburg    BB\n",
       "4             Bremen    HB"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StateName</th>\n",
       "      <th>State</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Hessen</td>\n",
       "      <td>HE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counts</th>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uniques</th>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing_perc</th>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>types</th>\n",
       "      <td>unique</td>\n",
       "      <td>unique</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             StateName   State\n",
       "count               16      16\n",
       "unique              16      16\n",
       "top             Hessen      HE\n",
       "freq                 1       1\n",
       "counts              16      16\n",
       "uniques             16      16\n",
       "missing              0       0\n",
       "missing_perc        0%      0%\n",
       "types           unique  unique"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(state_names.head())\n",
    "display(DataFrameSummary(state_names).summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is very representative of a typical industry dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following returns summarized aggregate information to each table accross each field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.278447Z",
     "start_time": "2017-09-09T12:29:04.242482Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>week</th>\n",
       "      <th>trend</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-02 - 2012-12-08</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-09 - 2012-12-15</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-16 - 2012-12-22</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-23 - 2012-12-29</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Rossmann_DE_SN</td>\n",
       "      <td>2012-12-30 - 2013-01-05</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             file                     week  trend\n",
       "0  Rossmann_DE_SN  2012-12-02 - 2012-12-08     96\n",
       "1  Rossmann_DE_SN  2012-12-09 - 2012-12-15     95\n",
       "2  Rossmann_DE_SN  2012-12-16 - 2012-12-22     91\n",
       "3  Rossmann_DE_SN  2012-12-23 - 2012-12-29     48\n",
       "4  Rossmann_DE_SN  2012-12-30 - 2013-01-05     67"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>week</th>\n",
       "      <th>trend</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>63.8142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.6502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counts</th>\n",
       "      <td>2072</td>\n",
       "      <td>2072</td>\n",
       "      <td>2072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uniques</th>\n",
       "      <td>14</td>\n",
       "      <td>148</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing_perc</th>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>types</th>\n",
       "      <td>categorical</td>\n",
       "      <td>categorical</td>\n",
       "      <td>numeric</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     file         week    trend\n",
       "count                 NaN          NaN     2072\n",
       "mean                  NaN          NaN  63.8142\n",
       "std                   NaN          NaN  12.6502\n",
       "min                   NaN          NaN        0\n",
       "25%                   NaN          NaN       55\n",
       "50%                   NaN          NaN       64\n",
       "75%                   NaN          NaN       72\n",
       "max                   NaN          NaN      100\n",
       "counts               2072         2072     2072\n",
       "uniques                14          148       68\n",
       "missing                 0            0        0\n",
       "missing_perc           0%           0%       0%\n",
       "types         categorical  categorical  numeric"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(googletrend.head())\n",
    "display(DataFrameSummary(googletrend).summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.410462Z",
     "start_time": "2017-09-09T12:29:04.282681Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>Date</th>\n",
       "      <th>Max_TemperatureC</th>\n",
       "      <th>Mean_TemperatureC</th>\n",
       "      <th>Min_TemperatureC</th>\n",
       "      <th>Dew_PointC</th>\n",
       "      <th>MeanDew_PointC</th>\n",
       "      <th>Min_DewpointC</th>\n",
       "      <th>Max_Humidity</th>\n",
       "      <th>Mean_Humidity</th>\n",
       "      <th>...</th>\n",
       "      <th>Max_VisibilityKm</th>\n",
       "      <th>Mean_VisibilityKm</th>\n",
       "      <th>Min_VisibilitykM</th>\n",
       "      <th>Max_Wind_SpeedKm_h</th>\n",
       "      <th>Mean_Wind_SpeedKm_h</th>\n",
       "      <th>Max_Gust_SpeedKm_h</th>\n",
       "      <th>Precipitationmm</th>\n",
       "      <th>CloudCover</th>\n",
       "      <th>Events</th>\n",
       "      <th>WindDirDegrees</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-01</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>94</td>\n",
       "      <td>87</td>\n",
       "      <td>...</td>\n",
       "      <td>31.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>39</td>\n",
       "      <td>26</td>\n",
       "      <td>58.0</td>\n",
       "      <td>5.08</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-02</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>93</td>\n",
       "      <td>85</td>\n",
       "      <td>...</td>\n",
       "      <td>31.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>24</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-03</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>100</td>\n",
       "      <td>93</td>\n",
       "      <td>...</td>\n",
       "      <td>31.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>26</td>\n",
       "      <td>21</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.02</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-04</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>100</td>\n",
       "      <td>94</td>\n",
       "      <td>...</td>\n",
       "      <td>11.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23</td>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.25</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NordrheinWestfalen</td>\n",
       "      <td>2013-01-05</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>100</td>\n",
       "      <td>94</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>16</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Rain</td>\n",
       "      <td>268</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 file        Date  Max_TemperatureC  Mean_TemperatureC  \\\n",
       "0  NordrheinWestfalen  2013-01-01                 8                  4   \n",
       "1  NordrheinWestfalen  2013-01-02                 7                  4   \n",
       "2  NordrheinWestfalen  2013-01-03                11                  8   \n",
       "3  NordrheinWestfalen  2013-01-04                 9                  9   \n",
       "4  NordrheinWestfalen  2013-01-05                 8                  8   \n",
       "\n",
       "   Min_TemperatureC  Dew_PointC  MeanDew_PointC  Min_DewpointC  Max_Humidity  \\\n",
       "0                 2           7               5              1            94   \n",
       "1                 1           5               3              2            93   \n",
       "2                 6          10               8              4           100   \n",
       "3                 8           9               9              8           100   \n",
       "4                 7           8               7              6           100   \n",
       "\n",
       "   Mean_Humidity       ...        Max_VisibilityKm  Mean_VisibilityKm  \\\n",
       "0             87       ...                    31.0               12.0   \n",
       "1             85       ...                    31.0               14.0   \n",
       "2             93       ...                    31.0                8.0   \n",
       "3             94       ...                    11.0                5.0   \n",
       "4             94       ...                    10.0                6.0   \n",
       "\n",
       "   Min_VisibilitykM  Max_Wind_SpeedKm_h  Mean_Wind_SpeedKm_h  \\\n",
       "0               4.0                  39                   26   \n",
       "1              10.0                  24                   16   \n",
       "2               2.0                  26                   21   \n",
       "3               2.0                  23                   14   \n",
       "4               3.0                  16                   10   \n",
       "\n",
       "   Max_Gust_SpeedKm_h  Precipitationmm  CloudCover  Events  WindDirDegrees  \n",
       "0                58.0             5.08         6.0    Rain             215  \n",
       "1                 NaN             0.00         6.0    Rain             225  \n",
       "2                 NaN             1.02         7.0    Rain             240  \n",
       "3                 NaN             0.25         7.0    Rain             263  \n",
       "4                 NaN             0.00         7.0    Rain             268  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>Date</th>\n",
       "      <th>Max_TemperatureC</th>\n",
       "      <th>Mean_TemperatureC</th>\n",
       "      <th>Min_TemperatureC</th>\n",
       "      <th>Dew_PointC</th>\n",
       "      <th>MeanDew_PointC</th>\n",
       "      <th>Min_DewpointC</th>\n",
       "      <th>Max_Humidity</th>\n",
       "      <th>Mean_Humidity</th>\n",
       "      <th>...</th>\n",
       "      <th>Max_VisibilityKm</th>\n",
       "      <th>Mean_VisibilityKm</th>\n",
       "      <th>Min_VisibilitykM</th>\n",
       "      <th>Max_Wind_SpeedKm_h</th>\n",
       "      <th>Mean_Wind_SpeedKm_h</th>\n",
       "      <th>Max_Gust_SpeedKm_h</th>\n",
       "      <th>Precipitationmm</th>\n",
       "      <th>CloudCover</th>\n",
       "      <th>Events</th>\n",
       "      <th>WindDirDegrees</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>...</td>\n",
       "      <td>15459</td>\n",
       "      <td>15459</td>\n",
       "      <td>15459</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>3604</td>\n",
       "      <td>15840</td>\n",
       "      <td>14667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.6441</td>\n",
       "      <td>10.389</td>\n",
       "      <td>6.19899</td>\n",
       "      <td>8.58782</td>\n",
       "      <td>6.20581</td>\n",
       "      <td>3.62614</td>\n",
       "      <td>93.6596</td>\n",
       "      <td>74.2829</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0576</td>\n",
       "      <td>12.2398</td>\n",
       "      <td>7.02516</td>\n",
       "      <td>22.7666</td>\n",
       "      <td>11.9722</td>\n",
       "      <td>48.8643</td>\n",
       "      <td>0.831718</td>\n",
       "      <td>5.55131</td>\n",
       "      <td>NaN</td>\n",
       "      <td>175.897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.64601</td>\n",
       "      <td>7.37926</td>\n",
       "      <td>6.52639</td>\n",
       "      <td>6.24478</td>\n",
       "      <td>6.08677</td>\n",
       "      <td>6.12839</td>\n",
       "      <td>7.67853</td>\n",
       "      <td>13.4866</td>\n",
       "      <td>...</td>\n",
       "      <td>8.9768</td>\n",
       "      <td>5.06794</td>\n",
       "      <td>4.9806</td>\n",
       "      <td>8.98862</td>\n",
       "      <td>5.87284</td>\n",
       "      <td>13.027</td>\n",
       "      <td>2.51351</td>\n",
       "      <td>1.68771</td>\n",
       "      <td>NaN</td>\n",
       "      <td>101.589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-11</td>\n",
       "      <td>-13</td>\n",
       "      <td>-15</td>\n",
       "      <td>-14</td>\n",
       "      <td>-15</td>\n",
       "      <td>-73</td>\n",
       "      <td>44</td>\n",
       "      <td>30</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>90.75</td>\n",
       "      <td>65</td>\n",
       "      <td>...</td>\n",
       "      <td>14</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>16</td>\n",
       "      <td>8</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15</td>\n",
       "      <td>11</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>94</td>\n",
       "      <td>76</td>\n",
       "      <td>...</td>\n",
       "      <td>31</td>\n",
       "      <td>11</td>\n",
       "      <td>7</td>\n",
       "      <td>21</td>\n",
       "      <td>11</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21</td>\n",
       "      <td>16</td>\n",
       "      <td>11</td>\n",
       "      <td>13</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>100</td>\n",
       "      <td>85</td>\n",
       "      <td>...</td>\n",
       "      <td>31</td>\n",
       "      <td>14</td>\n",
       "      <td>10</td>\n",
       "      <td>27</td>\n",
       "      <td>14</td>\n",
       "      <td>55</td>\n",
       "      <td>0.25</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>39</td>\n",
       "      <td>31</td>\n",
       "      <td>24</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>19</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>...</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>101</td>\n",
       "      <td>53</td>\n",
       "      <td>111</td>\n",
       "      <td>58.93</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counts</th>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>...</td>\n",
       "      <td>15459</td>\n",
       "      <td>15459</td>\n",
       "      <td>15459</td>\n",
       "      <td>15840</td>\n",
       "      <td>15840</td>\n",
       "      <td>3604</td>\n",
       "      <td>15840</td>\n",
       "      <td>14667</td>\n",
       "      <td>11889</td>\n",
       "      <td>15840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uniques</th>\n",
       "      <td>16</td>\n",
       "      <td>990</td>\n",
       "      <td>51</td>\n",
       "      <td>45</td>\n",
       "      <td>40</td>\n",
       "      <td>40</td>\n",
       "      <td>36</td>\n",
       "      <td>40</td>\n",
       "      <td>53</td>\n",
       "      <td>71</td>\n",
       "      <td>...</td>\n",
       "      <td>24</td>\n",
       "      <td>32</td>\n",
       "      <td>24</td>\n",
       "      <td>44</td>\n",
       "      <td>29</td>\n",
       "      <td>47</td>\n",
       "      <td>41</td>\n",
       "      <td>9</td>\n",
       "      <td>21</td>\n",
       "      <td>362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>381</td>\n",
       "      <td>381</td>\n",
       "      <td>381</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12236</td>\n",
       "      <td>0</td>\n",
       "      <td>1173</td>\n",
       "      <td>3951</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>missing_perc</th>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>...</td>\n",
       "      <td>2.41%</td>\n",
       "      <td>2.41%</td>\n",
       "      <td>2.41%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>77.25%</td>\n",
       "      <td>0%</td>\n",
       "      <td>7.41%</td>\n",
       "      <td>24.94%</td>\n",
       "      <td>0%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>types</th>\n",
       "      <td>categorical</td>\n",
       "      <td>categorical</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>...</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>numeric</td>\n",
       "      <td>categorical</td>\n",
       "      <td>numeric</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     file         Date Max_TemperatureC Mean_TemperatureC  \\\n",
       "count                 NaN          NaN            15840             15840   \n",
       "mean                  NaN          NaN          14.6441            10.389   \n",
       "std                   NaN          NaN          8.64601           7.37926   \n",
       "min                   NaN          NaN              -11               -13   \n",
       "25%                   NaN          NaN                8                 4   \n",
       "50%                   NaN          NaN               15                11   \n",
       "75%                   NaN          NaN               21                16   \n",
       "max                   NaN          NaN               39                31   \n",
       "counts              15840        15840            15840             15840   \n",
       "uniques                16          990               51                45   \n",
       "missing                 0            0                0                 0   \n",
       "missing_perc           0%           0%               0%                0%   \n",
       "types         categorical  categorical          numeric           numeric   \n",
       "\n",
       "             Min_TemperatureC Dew_PointC MeanDew_PointC Min_DewpointC  \\\n",
       "count                   15840      15840          15840         15840   \n",
       "mean                  6.19899    8.58782        6.20581       3.62614   \n",
       "std                   6.52639    6.24478        6.08677       6.12839   \n",
       "min                       -15        -14            -15           -73   \n",
       "25%                         1          4              2            -1   \n",
       "50%                         7          9              7             4   \n",
       "75%                        11         13             11             8   \n",
       "max                        24         25             20            19   \n",
       "counts                  15840      15840          15840         15840   \n",
       "uniques                    40         40             36            40   \n",
       "missing                     0          0              0             0   \n",
       "missing_perc               0%         0%             0%            0%   \n",
       "types                 numeric    numeric        numeric       numeric   \n",
       "\n",
       "             Max_Humidity Mean_Humidity      ...       Max_VisibilityKm  \\\n",
       "count               15840         15840      ...                  15459   \n",
       "mean              93.6596       74.2829      ...                24.0576   \n",
       "std               7.67853       13.4866      ...                 8.9768   \n",
       "min                    44            30      ...                      0   \n",
       "25%                 90.75            65      ...                     14   \n",
       "50%                    94            76      ...                     31   \n",
       "75%                   100            85      ...                     31   \n",
       "max                   100           100      ...                     31   \n",
       "counts              15840         15840      ...                  15459   \n",
       "uniques                53            71      ...                     24   \n",
       "missing                 0             0      ...                    381   \n",
       "missing_perc           0%            0%      ...                  2.41%   \n",
       "types             numeric       numeric      ...                numeric   \n",
       "\n",
       "             Mean_VisibilityKm Min_VisibilitykM Max_Wind_SpeedKm_h  \\\n",
       "count                    15459            15459              15840   \n",
       "mean                   12.2398          7.02516            22.7666   \n",
       "std                    5.06794           4.9806            8.98862   \n",
       "min                          0                0                  3   \n",
       "25%                         10                3                 16   \n",
       "50%                         11                7                 21   \n",
       "75%                         14               10                 27   \n",
       "max                         31               31                101   \n",
       "counts                   15459            15459              15840   \n",
       "uniques                     32               24                 44   \n",
       "missing                    381              381                  0   \n",
       "missing_perc             2.41%            2.41%                 0%   \n",
       "types                  numeric          numeric            numeric   \n",
       "\n",
       "             Mean_Wind_SpeedKm_h Max_Gust_SpeedKm_h Precipitationmm  \\\n",
       "count                      15840               3604           15840   \n",
       "mean                     11.9722            48.8643        0.831718   \n",
       "std                      5.87284             13.027         2.51351   \n",
       "min                            2                 21               0   \n",
       "25%                            8                 39               0   \n",
       "50%                           11                 48               0   \n",
       "75%                           14                 55            0.25   \n",
       "max                           53                111           58.93   \n",
       "counts                     15840               3604           15840   \n",
       "uniques                       29                 47              41   \n",
       "missing                        0              12236               0   \n",
       "missing_perc                  0%             77.25%              0%   \n",
       "types                    numeric            numeric         numeric   \n",
       "\n",
       "             CloudCover       Events WindDirDegrees  \n",
       "count             14667          NaN          15840  \n",
       "mean            5.55131          NaN        175.897  \n",
       "std             1.68771          NaN        101.589  \n",
       "min                   0          NaN             -1  \n",
       "25%                   5          NaN             80  \n",
       "50%                   6          NaN            202  \n",
       "75%                   7          NaN            256  \n",
       "max                   8          NaN            360  \n",
       "counts            14667        11889          15840  \n",
       "uniques               9           21            362  \n",
       "missing            1173         3951              0  \n",
       "missing_perc      7.41%       24.94%             0%  \n",
       "types           numeric  categorical        numeric  \n",
       "\n",
       "[13 rows x 24 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(weather.head())\n",
    "display(DataFrameSummary(weather).summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the fields are self-explanatory. The following are descriptions for those that aren't.\n",
    "\n",
    "- `Customers`: the number of customers on a given day\n",
    "- `Open`: an indicator for whether the store was open: 0 = closed, 1 = open\n",
    "- `StateHoliday`: indicates a state holiday. Normally all stores, with few exceptions, are closed on state holidays. Note that all schools are closed on public holidays and weekends. a = public holiday, b = Easter holiday, c = Christmas, 0 = None\n",
    "- `SchoolHoliday`: indicates if the (Store, Date) was affected by the closure of public schools\n",
    "- `StoreType`: differentiates between 4 different store models: a, b, c, d\n",
    "- `Assortment`: describes an assortment level: a = basic, b = extra, c = extended\n",
    "- `CompetitionDistance`: distance in meters to the nearest competitor store\n",
    "- `CompetitionOpenSince[Month/Year]`: gives the approximate year and month of the time the nearest competitor was opened\n",
    "- `Promo`: indicates whether a store is running a promo on that day\n",
    "- `Promo2`: Promo2 is a continuing and consecutive promotion for some stores: 0 = store is not participating, 1 = store is participating\n",
    "- `Promo2Since[Year/Week]`: describes the year and calendar week when the store started participating in Promo2\n",
    "- `PromoInterval`: describes the consecutive intervals Promo2 is started, naming the months the promotion is started anew. E.g. \"Feb,May,Aug,Nov\" means each round starts in February, May, August, November of any given year for that store\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning / Feature Engineering\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data cleaning is the process of transforming the data into formats that the model can understand.\n",
    "\n",
    "Feature engineering is the process of using domain knowledge of the data to create features (columns) that make machine learning algorithms work. More meaningful features (field) make it easier to \"learn\". For example, we will create a field / column called \"CompetitionOpenSince\" and \"CompetitionDaysOpen\" using raw columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turn state Holidays to Boolean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.486491Z",
     "start_time": "2017-09-09T12:29:04.414124Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train.StateHoliday = train.StateHoliday!='0'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Join weather/state names into a single table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.502496Z",
     "start_time": "2017-09-09T12:29:04.490763Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def join_df(left, right, left_on, right_on=None):\n",
    "    if right_on is None: right_on = left_on\n",
    "    return left.merge(right, how='left', left_on=left_on, right_on=right_on, \n",
    "                      suffixes=(\"\", \"_y\"))\n",
    "weather = join_df(weather, state_names, \"file\", \"StateName\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extracting dates and state names from the given data and adding those columns.\n",
    "\n",
    "Replace all instances of state name 'NI' with the usage in the rest of the table, 'HB,NI'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.530488Z",
     "start_time": "2017-09-09T12:29:04.506766Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "googletrend['Date'] = googletrend.week.str.split(' - ', expand=True)[0]\n",
    "googletrend['State'] = googletrend.file.str.split('_', expand=True)[2]\n",
    "googletrend.loc[googletrend.State=='NI', \"State\"] = 'HB,NI'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following extracts particular date fields from a complete datetime for the purpose of constructing categoricals.\n",
    "Without expanding your date-time into these additional fields, you can't capture any trend/cyclical behavior as a function of time at any of these granularities.\n",
    "We'll add to every table w/ a date field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.946837Z",
     "start_time": "2017-09-09T12:29:04.534787Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_datepart(df):\n",
    "    df.Date = pd.to_datetime(df.Date)\n",
    "    df[\"Year\"] = df.Date.dt.year\n",
    "    df[\"Month\"] = df.Date.dt.month\n",
    "    df[\"Week\"] = df.Date.dt.week\n",
    "    df[\"Day\"] = df.Date.dt.day\n",
    "add_datepart(weather)\n",
    "add_datepart(googletrend)\n",
    "add_datepart(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.958533Z",
     "start_time": "2017-09-09T12:29:04.952869Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trend_de = googletrend[googletrend.file == 'Rossmann_DE']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can outer join all of our data into a single dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:04.974532Z",
     "start_time": "2017-09-09T12:29:04.963091Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store = join_df(store, store_states, \"Store\")\n",
    "len(store[store.State.isnull()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:05.406573Z",
     "start_time": "2017-09-09T12:29:04.978780Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined = join_df(train, store, \"Store\")\n",
    "len(joined[joined.StoreType.isnull()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:05.894625Z",
     "start_time": "2017-09-09T12:29:05.410807Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined = join_df(joined, googletrend, [\"State\",\"Year\", \"Week\"])\n",
    "len(joined[joined.trend.isnull()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:06.618699Z",
     "start_time": "2017-09-09T12:29:05.898522Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined = joined.merge(trend_de, 'left', [\"Year\", \"Week\"], suffixes=('', '_DE'))\n",
    "len(joined[joined.trend_DE.isnull()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:08.122859Z",
     "start_time": "2017-09-09T12:29:06.622923Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined = join_df(joined, weather, [\"State\",\"Date\"])\n",
    "len(joined[joined.Mean_TemperatureC.isnull()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll fill in missing values to avoid complications w/ na's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:29:08.214883Z",
     "start_time": "2017-09-09T12:29:08.126803Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined.CompetitionOpenSinceYear = joined.CompetitionOpenSinceYear.fillna(1900).astype(np.int32)\n",
    "joined.CompetitionOpenSinceMonth = joined.CompetitionOpenSinceMonth.fillna(1).astype(np.int32)\n",
    "joined.Promo2SinceYear = joined.Promo2SinceYear.fillna(1900).astype(np.int32)\n",
    "joined.Promo2SinceWeek = joined.Promo2SinceWeek.fillna(1).astype(np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll extract features \"CompetitionOpenSince\" and \"CompetitionDaysOpen\". Note the use of `apply()` in mapping a function across dataframe values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:31:29.085813Z",
     "start_time": "2017-09-09T12:29:08.219195Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined[\"CompetitionOpenSince\"] = pd.to_datetime(joined.apply(lambda x: datetime.datetime(\n",
    "    x.CompetitionOpenSinceYear, x.CompetitionOpenSinceMonth, 15), axis=1).astype(pd.datetime))\n",
    "joined[\"CompetitionDaysOpen\"] = joined.Date.subtract(joined[\"CompetitionOpenSince\"]).dt.days"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll replace some erroneous / outlying data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:31:29.209823Z",
     "start_time": "2017-09-09T12:31:29.090145Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined.loc[joined.CompetitionDaysOpen<0, \"CompetitionDaysOpen\"] = 0\n",
    "joined.loc[joined.CompetitionOpenSinceYear<1990, \"CompetitionDaysOpen\"] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Added \"CompetitionMonthsOpen\" field, limit the maximum to 2 years to limit number of unique embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:31:29.337825Z",
     "start_time": "2017-09-09T12:31:29.214175Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([24,  3, 19,  9,  0, 16, 17,  7, 15, 22, 11, 13,  2, 23, 12,  4, 10,\n",
       "        1, 14, 20,  8, 18,  6, 21,  5])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined[\"CompetitionMonthsOpen\"] = joined[\"CompetitionDaysOpen\"]//30\n",
    "joined.loc[joined.CompetitionMonthsOpen>24, \"CompetitionMonthsOpen\"] = 24\n",
    "joined.CompetitionMonthsOpen.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same process for Promo dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:33:57.741676Z",
     "start_time": "2017-09-09T12:31:29.341763Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined[\"Promo2Since\"] = pd.to_datetime(joined.apply(lambda x: Week(\n",
    "    x.Promo2SinceYear, x.Promo2SinceWeek).monday(), axis=1).astype(pd.datetime))\n",
    "joined[\"Promo2Days\"] = joined.Date.subtract(joined[\"Promo2Since\"]).dt.days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:33:57.861678Z",
     "start_time": "2017-09-09T12:33:57.745966Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined.loc[joined.Promo2Days<0, \"Promo2Days\"] = 0\n",
    "joined.loc[joined.Promo2SinceYear<1990, \"Promo2Days\"] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:33:58.009680Z",
     "start_time": "2017-09-09T12:33:57.865983Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0, 25, 17,  8, 13, 24, 16,  7, 12, 23, 15,  6, 11, 22, 14,  5, 10,\n",
       "       21,  4,  9, 20,  3, 19,  2, 18,  1])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined[\"Promo2Weeks\"] = joined[\"Promo2Days\"]//7\n",
    "joined.loc[joined.Promo2Weeks<0, \"Promo2Weeks\"] = 0\n",
    "joined.loc[joined.Promo2Weeks>25, \"Promo2Weeks\"] = 25\n",
    "joined.Promo2Weeks.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Durations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is common when working with time series data to extract data that explains relationships across rows as opposed to columns, e.g.:\n",
    "* Running averages\n",
    "* Time until next event\n",
    "* Time since last event\n",
    "\n",
    "This is often difficult to do with most table manipulation frameworks, since they are designed to work with relationships across columns. As such, we've created a class to handle this type of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:33:58.017675Z",
     "start_time": "2017-09-09T12:33:58.013588Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "columns = [\"Date\", \"Store\", \"Promo\", \"StateHoliday\", \"SchoolHoliday\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've defined a class `elapsed` for cumulative counting across a sorted dataframe.\n",
    "\n",
    "Given a particular field `fld` to monitor, this object will start tracking time since the last occurrence of that field. When the field is seen again, the counter is set to zero.\n",
    "\n",
    "Upon initialization, this will result in datetime na's until the field is encountered. This is reset every time a new store is seen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:33:58.033681Z",
     "start_time": "2017-09-09T12:33:58.020905Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class elapsed(object):\n",
    "    def __init__(self, fld):\n",
    "        self.fld = fld\n",
    "        self.last = pd.to_datetime(np.nan)\n",
    "        self.last_store = 0\n",
    "        \n",
    "    def get(self, row):\n",
    "        if row.Store != self.last_store:\n",
    "            self.last = pd.to_datetime(np.nan)\n",
    "            self.last_store = row.Store\n",
    "        if (row[self.fld]): self.last = row.Date\n",
    "        return row.Date-self.last"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:33:58.073702Z",
     "start_time": "2017-09-09T12:33:58.038183Z"
    },
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = train[columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a function for applying said class across dataframe rows and adding values to a new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:33:58.085773Z",
     "start_time": "2017-09-09T12:33:58.078010Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_elapsed(fld, prefix):\n",
    "    sh_el = elapsed(fld)\n",
    "    df[prefix+fld] = df.apply(sh_el.get, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's walk through an example.\n",
    "\n",
    "Say we're looking at School Holiday. We'll first sort by Store, then Date, and then call `add_elapsed('SchoolHoliday', 'After')`:\n",
    "This will generate an instance of the `elapsed` class for School Holiday:\n",
    "* Instance applied to every row of the dataframe in order of store and date\n",
    "* Will add to the dataframe the days since seeing a School Holiday\n",
    "* If we sort in the other direction, this will count the days until another promotion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:38:56.318003Z",
     "start_time": "2017-09-09T12:33:58.089696Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fld = 'SchoolHoliday'\n",
    "df = df.sort_values(['Store', 'Date'])\n",
    "add_elapsed(fld, 'After')\n",
    "df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
    "add_elapsed(fld, 'Before')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll do this for two more fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:44:24.643414Z",
     "start_time": "2017-09-09T12:38:56.322386Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fld = 'StateHoliday'\n",
    "df = df.sort_values(['Store', 'Date'])\n",
    "add_elapsed(fld, 'After')\n",
    "df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
    "add_elapsed(fld, 'Before')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:50:39.561351Z",
     "start_time": "2017-09-09T12:44:24.647992Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fld = 'Promo'\n",
    "df = df.sort_values(['Store', 'Date'])\n",
    "add_elapsed(fld, 'After')\n",
    "df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
    "add_elapsed(fld, 'Before')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to set the active index to Date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:50:39.589349Z",
     "start_time": "2017-09-09T12:50:39.565636Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df.set_index(\"Date\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then set null values from elapsed field calculations to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:50:39.597331Z",
     "start_time": "2017-09-09T12:50:39.593766Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "columns = ['SchoolHoliday', 'StateHoliday', 'Promo']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:34.476290Z",
     "start_time": "2017-09-09T12:50:39.601239Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for o in ['Before', 'After']:\n",
    "    for p in columns:\n",
    "        a = o+p\n",
    "        df[a] = df[a].fillna(pd.Timedelta(0)).dt.days"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll demonstrate window functions in pandas to calculate rolling quantities.\n",
    "\n",
    "Here we're sorting by date (`sort_index()`) and counting the number of events of interest (`sum()`) defined in `columns` in the following week (`rolling()`), grouped by Store (`groupby()`). We do the same in the opposite direction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:36.660413Z",
     "start_time": "2017-09-09T12:51:34.480574Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bwd = df[['Store']+columns].sort_index().groupby(\"Store\").rolling(7, min_periods=1).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:38.828535Z",
     "start_time": "2017-09-09T12:51:36.664767Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fwd = df[['Store']+columns].sort_index(ascending=False\n",
    "                                      ).groupby(\"Store\").rolling(7, min_periods=1).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we want to drop the Store indices grouped together in the window function.\n",
    "\n",
    "Often in pandas, there is an option to do this in place. This is time and memory efficient when working with large datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:38.848541Z",
     "start_time": "2017-09-09T12:51:38.832896Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bwd.drop('Store',1,inplace=True)\n",
    "bwd.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:38.868544Z",
     "start_time": "2017-09-09T12:51:38.852784Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fwd.drop('Store',1,inplace=True)\n",
    "fwd.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:38.880520Z",
     "start_time": "2017-09-09T12:51:38.872814Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll merge these values onto the df."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:39.280562Z",
     "start_time": "2017-09-09T12:51:38.884558Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df.merge(bwd, 'left', ['Date', 'Store'], suffixes=['', '_bw'])\n",
    "df = df.merge(fwd, 'left', ['Date', 'Store'], suffixes=['', '_fw'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:39.336575Z",
     "start_time": "2017-09-09T12:51:39.284873Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.drop(columns,1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:39.360551Z",
     "start_time": "2017-09-09T12:51:39.341181Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Store</th>\n",
       "      <th>AfterSchoolHoliday</th>\n",
       "      <th>BeforeSchoolHoliday</th>\n",
       "      <th>AfterStateHoliday</th>\n",
       "      <th>BeforeStateHoliday</th>\n",
       "      <th>AfterPromo</th>\n",
       "      <th>BeforePromo</th>\n",
       "      <th>SchoolHoliday_bw</th>\n",
       "      <th>StateHoliday_bw</th>\n",
       "      <th>Promo_bw</th>\n",
       "      <th>SchoolHoliday_fw</th>\n",
       "      <th>StateHoliday_fw</th>\n",
       "      <th>Promo_fw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-07-30</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-07-29</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-07-28</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>54</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-07-27</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>53</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Date  Store  AfterSchoolHoliday  BeforeSchoolHoliday  \\\n",
       "0 2015-07-31      1                   0                    0   \n",
       "1 2015-07-30      1                   0                    0   \n",
       "2 2015-07-29      1                   0                    0   \n",
       "3 2015-07-28      1                   0                    0   \n",
       "4 2015-07-27      1                   0                    0   \n",
       "\n",
       "   AfterStateHoliday  BeforeStateHoliday  AfterPromo  BeforePromo  \\\n",
       "0                 57                   0           0            0   \n",
       "1                 56                   0           0            0   \n",
       "2                 55                   0           0            0   \n",
       "3                 54                   0           0            0   \n",
       "4                 53                   0           0            0   \n",
       "\n",
       "   SchoolHoliday_bw  StateHoliday_bw  Promo_bw  SchoolHoliday_fw  \\\n",
       "0               5.0              0.0       5.0               1.0   \n",
       "1               4.0              0.0       4.0               2.0   \n",
       "2               3.0              0.0       3.0               3.0   \n",
       "3               2.0              0.0       2.0               4.0   \n",
       "4               1.0              0.0       1.0               5.0   \n",
       "\n",
       "   StateHoliday_fw  Promo_fw  \n",
       "0              0.0       1.0  \n",
       "1              0.0       2.0  \n",
       "2              0.0       3.0  \n",
       "3              0.0       4.0  \n",
       "4              0.0       5.0  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's usually a good idea to back up large tables of extracted / wrangled features before you join them onto another one, that way you can go back to it easily if you need to make changes to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:54.357405Z",
     "start_time": "2017-09-09T12:51:39.364463Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.to_csv('df.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:55.193509Z",
     "start_time": "2017-09-09T12:51:54.361778Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/terry/anaconda3/lib/python3.6/site-packages/numpy/lib/arraysetops.py:463: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
      "  mask |= (ar1 == a)\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv('df.csv', index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:55.433461Z",
     "start_time": "2017-09-09T12:51:55.197365Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df[\"Date\"] = pd.to_datetime(df.Date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:55.445447Z",
     "start_time": "2017-09-09T12:51:55.437864Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Date', 'Store', 'AfterSchoolHoliday', 'BeforeSchoolHoliday',\n",
       "       'AfterStateHoliday', 'BeforeStateHoliday', 'AfterPromo', 'BeforePromo',\n",
       "       'SchoolHoliday_bw', 'StateHoliday_bw', 'Promo_bw', 'SchoolHoliday_fw',\n",
       "       'StateHoliday_fw', 'Promo_fw'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:51:57.961603Z",
     "start_time": "2017-09-09T12:51:55.448758Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined = join_df(joined, df, ['Store', 'Date'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll back this up as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:29.639007Z",
     "start_time": "2017-09-09T12:51:57.966181Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined.to_csv('joined.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have our final set of engineered features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:36.755403Z",
     "start_time": "2017-09-09T12:53:29.642794Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/terry/anaconda3/lib/python3.6/site-packages/numpy/lib/arraysetops.py:463: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
      "  mask |= (ar1 == a)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Index(['Store', 'DayOfWeek', 'Date', 'Sales', 'Customers', 'Open', 'Promo',\n",
       "       'StateHoliday', 'SchoolHoliday', 'Year', 'Month', 'Week', 'Day',\n",
       "       'StoreType', 'Assortment', 'CompetitionDistance',\n",
       "       'CompetitionOpenSinceMonth', 'CompetitionOpenSinceYear', 'Promo2',\n",
       "       'Promo2SinceWeek', 'Promo2SinceYear', 'PromoInterval', 'State', 'file',\n",
       "       'week', 'trend', 'Date_y', 'Month_y', 'Day_y', 'file_DE', 'week_DE',\n",
       "       'trend_DE', 'Date_DE', 'State_DE', 'Month_DE', 'Day_DE', 'file_y',\n",
       "       'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',\n",
       "       'Dew_PointC', 'MeanDew_PointC', 'Min_DewpointC', 'Max_Humidity',\n",
       "       'Mean_Humidity', 'Min_Humidity', 'Max_Sea_Level_PressurehPa',\n",
       "       'Mean_Sea_Level_PressurehPa', 'Min_Sea_Level_PressurehPa',\n",
       "       'Max_VisibilityKm', 'Mean_VisibilityKm', 'Min_VisibilitykM',\n",
       "       'Max_Wind_SpeedKm_h', 'Mean_Wind_SpeedKm_h', 'Max_Gust_SpeedKm_h',\n",
       "       'Precipitationmm', 'CloudCover', 'Events', 'WindDirDegrees',\n",
       "       'StateName', 'Year_y', 'Month_y.1', 'Week_y', 'Day_y.1',\n",
       "       'CompetitionOpenSince', 'CompetitionDaysOpen', 'CompetitionMonthsOpen',\n",
       "       'Promo2Since', 'Promo2Days', 'Promo2Weeks', 'AfterSchoolHoliday',\n",
       "       'BeforeSchoolHoliday', 'AfterStateHoliday', 'BeforeStateHoliday',\n",
       "       'AfterPromo', 'BeforePromo', 'SchoolHoliday_bw', 'StateHoliday_bw',\n",
       "       'Promo_bw', 'SchoolHoliday_fw', 'StateHoliday_fw', 'Promo_fw'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined = pd.read_csv('joined.csv', index_col=0)\n",
    "joined[\"Date\"] = pd.to_datetime(joined.Date)\n",
    "joined.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While these steps were explicitly outlined in the paper, these are all fairly typical feature engineering steps for dealing with time series data and are practical in any similar setting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Create features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've engineered all our features, we need to convert to input compatible with a neural network.\n",
    "\n",
    "This includes converting categorical variables into contiguous integers or one-hot encodings, normalizing continuous features to standard normal, etc..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:36.771411Z",
     "start_time": "2017-09-09T12:53:36.759311Z"
    },
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from sklearn_pandas import DataFrameMapper\n",
    "from sklearn.preprocessing import LabelEncoder, Imputer, StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dictionary maps categories to embedding dimensionality. In generally, categories we might expect to be conceptually more complex have larger dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:36.783653Z",
     "start_time": "2017-09-09T12:53:36.775980Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_var_dict = {'Store': 50, 'DayOfWeek': 6, 'Year': 2, 'Month': 6,\n",
    "'Day': 10, 'StateHoliday': 3, 'CompetitionMonthsOpen': 2,\n",
    "'Promo2Weeks': 1, 'StoreType': 2, 'Assortment': 3, 'PromoInterval': 3,\n",
    "'CompetitionOpenSinceYear': 4, 'Promo2SinceYear': 4, 'State': 6,\n",
    "'Week': 2, 'Events': 4, 'Promo_fw': 1,\n",
    "'Promo_bw': 1, 'StateHoliday_fw': 1,\n",
    "'StateHoliday_bw': 1, 'SchoolHoliday_fw': 1,\n",
    "'SchoolHoliday_bw': 1}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Name categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:36.791955Z",
     "start_time": "2017-09-09T12:53:36.787912Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_vars = [o[0] for o in \n",
    "            sorted(cat_var_dict.items(), key=operator.itemgetter(1), reverse=True)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise for continuous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:36.803411Z",
     "start_time": "2017-09-09T12:53:36.797476Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# mean/max wind; min temp; cloud; min/mean humid; \n",
    "contin_vars = ['CompetitionDistance', \n",
    "   'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',\n",
    "   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h', \n",
    "   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',\n",
    "   'AfterStateHoliday', 'BeforeStateHoliday', 'Promo', 'SchoolHoliday']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace nulls w/ 0 for continuous, \"\" for categorical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:40.183627Z",
     "start_time": "2017-09-09T12:53:36.807598Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for v in contin_vars: joined.loc[joined[v].isnull(), v] = 0\n",
    "for v in cat_vars: joined.loc[joined[v].isnull(), v] = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we create a list of tuples, each containing a variable and an instance of a transformer for that variable.\n",
    "\n",
    "For categoricals, we use a label encoder that maps categories to continuous integers. For continuous variables, we standardize them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:40.195634Z",
     "start_time": "2017-09-09T12:53:40.188549Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_maps = [(o, LabelEncoder()) for o in cat_vars]\n",
    "contin_maps = [([o], StandardScaler()) for o in contin_vars]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same instances need to be used for the test set as well, so values are mapped/standardized appropriately.\n",
    "\n",
    "DataFrame mapper will keep track of these variable-instance mappings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:44.299867Z",
     "start_time": "2017-09-09T12:53:40.200459Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_mapper = DataFrameMapper(cat_maps)\n",
    "cat_map_fit = cat_mapper.fit(joined)\n",
    "cat_cols = len(cat_map_fit.features)\n",
    "cat_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:44.427873Z",
     "start_time": "2017-09-09T12:53:44.303777Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/terry/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:444: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.\n",
      "  warnings.warn(msg, DataConversionWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "contin_mapper = DataFrameMapper(contin_maps)\n",
    "contin_map_fit = contin_mapper.fit(joined)\n",
    "contin_cols = len(contin_map_fit.features)\n",
    "contin_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of first five rows of zeroth column being transformed appropriately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:49.896223Z",
     "start_time": "2017-09-09T12:53:44.431083Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/terry/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:444: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.\n",
      "  warnings.warn(msg, DataConversionWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(array([ 0, 30,  4,  6,  4]),\n",
       " array([-0.53772351,  1.02444552,  0.81863226,  0.32504749,  0.61112004]))"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_map_fit.transform(joined)[0,:5], contin_map_fit.transform(joined)[0,:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also pickle these mappings, which is great for portability!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:49.912227Z",
     "start_time": "2017-09-09T12:53:49.900164Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pickle.dump(contin_map_fit, open('contin_maps.pickle', 'wb'))\n",
    "pickle.dump(cat_map_fit, open('cat_maps.pickle', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:49.924212Z",
     "start_time": "2017-09-09T12:53:49.916517Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1115, 31, 7, 12, 12, 23, 8, 22, 2, 3, 4, 3, 25, 4, 52, 26, 6, 6, 3, 3, 8, 8]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[len(o[1].classes_) for o in cat_map_fit.features]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, the authors removed all instances where the store had zero sale / was closed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.152274Z",
     "start_time": "2017-09-09T12:53:49.928389Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined_sales = joined[joined.Sales!=0]\n",
    "n = len(joined_sales)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We speculate that this may have cost them a higher standing in the competition. One reason this may be the case is that a little EDA reveals that there are often periods where stores are closed, typically for refurbishment. Before and after these periods, there are naturally spikes in sales that one might expect. Be ommitting this data from their training, the authors gave up the ability to leverage information about these periods to predict this otherwise volatile behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.168232Z",
     "start_time": "2017-09-09T12:53:50.160980Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "844338"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to run on a sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.272249Z",
     "start_time": "2017-09-09T12:53:50.171255Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "samp_size = 100000\n",
    "np.random.seed(42)\n",
    "idxs = sorted(np.random.choice(n, samp_size, replace=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.340262Z",
     "start_time": "2017-09-09T12:53:50.276883Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined_samp = joined_sales.iloc[idxs].set_index(\"Date\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.476312Z",
     "start_time": "2017-09-09T12:53:50.344532Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "samp_size = n\n",
    "joined_samp = joined_sales.set_index(\"Date\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In time series data, cross-validation is not random. Instead, our holdout data is always the most recent data, as it would be in real application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've taken the last 10% as our validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.484290Z",
     "start_time": "2017-09-09T12:53:50.480594Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_ratio = 0.9\n",
    "train_size = int(samp_size * train_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.496245Z",
     "start_time": "2017-09-09T12:53:50.487720Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "759904"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.508272Z",
     "start_time": "2017-09-09T12:53:50.500196Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(84434, 759904)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined_valid = joined_samp[train_size:]\n",
    "joined_train = joined_samp[:train_size]\n",
    "len(joined_valid), len(joined_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a preprocessor for our categoricals using our instance mapper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:50.516263Z",
     "start_time": "2017-09-09T12:53:50.511228Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cat_preproc(dat):\n",
    "    return cat_map_fit.transform(dat).astype(np.int64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:54.700529Z",
     "start_time": "2017-09-09T12:53:50.518755Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_map_train = cat_preproc(joined_train)\n",
    "cat_map_valid = cat_preproc(joined_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same for continuous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:54.712514Z",
     "start_time": "2017-09-09T12:53:54.704867Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def contin_preproc(dat):\n",
    "    return contin_map_fit.transform(dat).astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:54.940543Z",
     "start_time": "2017-09-09T12:53:54.716615Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/terry/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:444: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.\n",
      "  warnings.warn(msg, DataConversionWarning)\n"
     ]
    }
   ],
   "source": [
    "contin_map_train = contin_preproc(joined_train)\n",
    "contin_map_valid = contin_preproc(joined_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grab our targets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:54.952549Z",
     "start_time": "2017-09-09T12:53:54.945841Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train_orig = joined_train.Sales\n",
    "y_valid_orig = joined_valid.Sales"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the authors modified the target values by applying a logarithmic transformation and normalizing to unit scale by dividing by the maximum log value.\n",
    "\n",
    "Log transformations are used on this type of data frequently to attain a nicer shape. \n",
    "\n",
    "Further by scaling to the unit interval we can now use a sigmoid output in our neural network. Then we can multiply by the maximum log value to get the original log value and transform back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.016541Z",
     "start_time": "2017-09-09T12:53:54.957108Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "max_log_y = np.max(np.log(joined_samp.Sales))\n",
    "y_train = np.log(y_train_orig)/max_log_y\n",
    "y_valid = np.log(y_valid_orig)/max_log_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Some testing shows this doesn't make a big difference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Root-mean-squared percent error is the metric Kaggle used for this competition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These undo the target transformations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're ready to put together our models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.024522Z",
     "start_time": "2017-09-09T12:53:55.020826Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def split_cols(arr): return np.hsplit(arr,arr.shape[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.032524Z",
     "start_time": "2017-09-09T12:53:55.028651Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "map_train = split_cols(cat_map_train) + [contin_map_train]\n",
    "map_valid = split_cols(cat_map_valid) + [contin_map_valid]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper function for getting categorical name and dim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.040525Z",
     "start_time": "2017-09-09T12:53:55.036612Z"
    },
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def cat_map_info(feat): return feat[0], len(feat[1].classes_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example the dimension of the feature `Day` is 31: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.056542Z",
     "start_time": "2017-09-09T12:53:55.043506Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Day', 31)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_map_info(cat_map_fit.features[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.064684Z",
     "start_time": "2017-09-09T12:53:55.058976Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_init(scale):\n",
    "    return lambda shape, name=None: initializations.uniform(shape, scale=scale, name=name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.072527Z",
     "start_time": "2017-09-09T12:53:55.067237Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def emb_init(shape, name=None): \n",
    "    return initializations.uniform(shape, scale=2/(shape[1]+1), name=name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper function for constructing embeddings.\n",
    "\n",
    "Also, note we're flattening the embedding. Embeddings in Keras come out as an element of a sequence like we might use in a sequence of words; here we just want to concatenate them so we flatten the 1-vector sequence into a vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.084532Z",
     "start_time": "2017-09-09T12:53:55.074923Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_emb(feat):\n",
    "    name, c = cat_map_info(feat)\n",
    "    c2 = (c+1)//2\n",
    "    if c2>50: c2=50\n",
    "    inp = Input((1,), dtype='int64', name=name+'_in')\n",
    "    u = Flatten(name=name+'_flt')(Embedding(c, c2, input_length=1, init=emb_init)(inp))\n",
    "    return inp,u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper function for continuous inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.096565Z",
     "start_time": "2017-09-09T12:53:55.088961Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_contin(feat):\n",
    "    name = feat[0][0]\n",
    "    inp = Input((1,), name=name+'_in')\n",
    "    return inp, Dense(1, name=name+'_d', init=my_init(1.))(inp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.128559Z",
     "start_time": "2017-09-09T12:53:55.100545Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "contin_inp = Input((contin_cols,), name='contin')\n",
    "contin_out = Dense(contin_cols*10, activation='relu', name='contin_d')(contin_inp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can put them together. Given the inputs, continuous and categorical embeddings, we're going to concatenate all of them.\n",
    "\n",
    "Next, we're going to pass through some dropout, then two dense layers w/ ReLU activations, then dropout again, then the sigmoid activation we mentioned earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.136533Z",
     "start_time": "2017-09-09T12:53:55.132852Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def root_mean_squared_error(y_true, y_pred):\n",
    "    return K.sqrt(K.mean(K.square(y_pred - y_true), axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T12:53:55.416551Z",
     "start_time": "2017-09-09T12:53:55.141454Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "embs = [get_emb(feat) for feat in cat_map_fit.features]\n",
    "x = merge([emb for inp,emb in embs] + [contin_out], mode='concat')\n",
    "x = Dropout(0.02)(x)\n",
    "x = Dense(1000, activation='relu', init='uniform')(x)\n",
    "x = Dense(500, activation='relu', init='uniform')(x)\n",
    "x = Dropout(0.2)(x)\n",
    "x = Dense(1, activation='sigmoid')(x)\n",
    "\n",
    "model = Model([inp for inp,emb in embs] + [contin_inp], x)\n",
    "# model.compile(optimizer = \"adam\", loss = 'mean_squared_error')\n",
    "model.compile(optimizer = \"adam\", loss = 'mean_absolute_error')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:54.319408Z",
     "start_time": "2017-09-09T12:53:55.421172Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 759904 samples, validate on 84434 samples\n",
      "Epoch 1/25\n",
      "759904/759904 [==============================] - 118s - loss: 0.0116 - val_loss: 0.0115\n",
      "Epoch 2/25\n",
      "759904/759904 [==============================] - 119s - loss: 0.0082 - val_loss: 0.0107\n",
      "Epoch 3/25\n",
      "759904/759904 [==============================] - 100s - loss: 0.0072 - val_loss: 0.0109\n",
      "Epoch 4/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0068 - val_loss: 0.0102\n",
      "Epoch 5/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0066 - val_loss: 0.0102\n",
      "Epoch 6/25\n",
      "759904/759904 [==============================] - 100s - loss: 0.0064 - val_loss: 0.0104\n",
      "Epoch 7/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0063 - val_loss: 0.0103\n",
      "Epoch 8/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0063 - val_loss: 0.0103\n",
      "Epoch 9/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0062 - val_loss: 0.0102\n",
      "Epoch 10/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0061 - val_loss: 0.0100\n",
      "Epoch 11/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0061 - val_loss: 0.0101\n",
      "Epoch 12/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0061 - val_loss: 0.0103\n",
      "Epoch 13/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0060 - val_loss: 0.0104\n",
      "Epoch 14/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0060 - val_loss: 0.0103\n",
      "Epoch 15/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0060 - val_loss: 0.0102\n",
      "Epoch 16/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0060 - val_loss: 0.0103\n",
      "Epoch 17/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0059 - val_loss: 0.0103\n",
      "Epoch 18/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0059 - val_loss: 0.0102\n",
      "Epoch 19/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0059 - val_loss: 0.0102\n",
      "Epoch 20/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0059 - val_loss: 0.0106\n",
      "Epoch 21/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0059 - val_loss: 0.0102\n",
      "Epoch 22/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0059 - val_loss: 0.0102\n",
      "Epoch 23/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0058 - val_loss: 0.0100\n",
      "Epoch 24/25\n",
      "759904/759904 [==============================] - 99s - loss: 0.0058 - val_loss: 0.0102\n",
      "Epoch 25/25\n",
      "759904/759904 [==============================] - 98s - loss: 0.0058 - val_loss: 0.0102\n"
     ]
    }
   ],
   "source": [
    "hist = model.fit(map_train, y_train, batch_size=128, nb_epoch=25,\n",
    "                 verbose=1, validation_data=(map_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:54.331401Z",
     "start_time": "2017-09-09T13:35:54.323830Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'loss': [0.01162134316499382,\n",
       "  0.0081869323948954281,\n",
       "  0.0072003268104881314,\n",
       "  0.0067859471569262659,\n",
       "  0.006567331280002555,\n",
       "  0.0064336042863367389,\n",
       "  0.0063238474541785078,\n",
       "  0.0062556071352428072,\n",
       "  0.006193132654132693,\n",
       "  0.0061474901312981705,\n",
       "  0.0061090898499808616,\n",
       "  0.0060713283697473967,\n",
       "  0.0060466320593580373,\n",
       "  0.0060108835889448433,\n",
       "  0.005984828528138383,\n",
       "  0.005962159071379149,\n",
       "  0.0059363457965337287,\n",
       "  0.0059244019551490598,\n",
       "  0.0059050714438506216,\n",
       "  0.0058871330172541982,\n",
       "  0.0058756062647058504,\n",
       "  0.0058583983999589192,\n",
       "  0.0058498918361017142,\n",
       "  0.0058381564955732176,\n",
       "  0.005835342206274706],\n",
       " 'val_loss': [0.011480208250625611,\n",
       "  0.010687497730154687,\n",
       "  0.010873794549884247,\n",
       "  0.010220619868634042,\n",
       "  0.010169948833681598,\n",
       "  0.010375462150886881,\n",
       "  0.01029860381840259,\n",
       "  0.010281497902071713,\n",
       "  0.010245389129211944,\n",
       "  0.0099707640192427342,\n",
       "  0.010090286008184889,\n",
       "  0.010322464609138635,\n",
       "  0.010389837869003887,\n",
       "  0.010288577121845601,\n",
       "  0.010211942092153892,\n",
       "  0.010320948124746177,\n",
       "  0.010315869419813469,\n",
       "  0.010207908521663245,\n",
       "  0.010243805451099122,\n",
       "  0.010564551854073179,\n",
       "  0.010156116401596879,\n",
       "  0.010218167661902965,\n",
       "  0.010049049564647104,\n",
       "  0.010157778065229949,\n",
       "  0.010242302661496607]}"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist.history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:54.475406Z",
     "start_time": "2017-09-09T13:35:54.335462Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl81NW9//HXJ+skZCFsSSAoqCCbyibivuJuccGt1q2t\nVKu19nazvbetvW1v/fVat9alWmm1t9UqarUtFvd9Y9EiCAIiSlhC2LKS/fz+ON/AEEO2ycwkmffz\n8chjZr7bnG8C855zvuecrznnEBER6aqkeBdARER6NwWJiIhEREEiIiIRUZCIiEhEFCQiIhIRBYmI\niEREQSISRWb2RzP7eQe3XWtmJ0V6HJFYU5CIiEhEFCQiIhIRBYkkvKBJ6btmtsTMqszsATPLN7Nn\nzKzCzJ43s7yw7b9gZsvMbIeZvWxmY8PWTTKzxcF+fwVCLd7rTDN7P9j3TTM7uItlvsrMVpvZNjN7\n2syGBsvNzG4zs81mVm5mH5jZhGDd6Wb2YVC29Wb2nS79wkRaUJCIeOcBM4DRwFnAM8APgcH4/yfX\nA5jZaOBh4IZg3Tzg72aWZmZpwN+APwEDgMeC4xLsOwmYA3wNGAj8DnjazNI7U1AzOwH4JXABUAh8\nCjwSrD4ZOCY4j9xgm63BugeArznnsoEJwIudeV+RvVGQiHi/cc6VOOfWA68B7zjn3nPO1QBPApOC\n7S4E/umce845Vw/cAmQARwDTgVTgdudcvXNuLrAg7D1mA79zzr3jnGt0zj0I1Ab7dcYlwBzn3GLn\nXC3wA+BwMxsB1APZwBjAnHPLnXMbg/3qgXFmluOc2+6cW9zJ9xVplYJExCsJe76zlddZwfOh+BoA\nAM65JmAdMCxYt97tORPqp2HP9wW+HTRr7TCzHcDwYL/OaFmGSnytY5hz7kXgt8BdwGYzu8/McoJN\nzwNOBz41s1fM7PBOvq9IqxQkIp2zAR8IgL8mgQ+D9cBGYFiwrNk+Yc/XAb9wzvUP+8l0zj0cYRn6\n4ZvK1gM45+50zk0BxuGbuL4bLF/gnJsJDME3wT3ayfcVaZWCRKRzHgXOMLMTzSwV+Da+eepN4C2g\nAbjezFLN7FxgWti+9wNXm9lhwUXxfmZ2hplld7IMDwNXmtnE4PrK/+Cb4taa2aHB8VOBKqAGaAqu\n4VxiZrlBk1w50BTB70FkFwWJSCc45z4CvgT8BtiCvzB/lnOuzjlXB5wLXAFsw19PeSJs34XAVfim\np+3A6mDbzpbheeBHwOP4WtD+wEXB6hx8YG3HN39tBf43WHcpsNbMyoGr8ddaRCJmurGViIhEQjUS\nERGJiIJEREQioiAREZGIKEhERCQiKfEuQCwMGjTIjRgxIt7FEBHpVRYtWrTFOTe4ve0SIkhGjBjB\nwoUL410MEZFexcw+bX8rNW2JiEiEFCQiIhIRBYmIiEQkIa6RiEjfUV9fT3FxMTU1NfEuSp8RCoUo\nKioiNTW1S/srSESkVykuLiY7O5sRI0aw50TL0hXOObZu3UpxcTEjR47s0jHUtCUivUpNTQ0DBw5U\niHQTM2PgwIER1fAUJCLS6yhEulekv08FSRteXFHC3S+vjncxRER6NAVJG15ftZW7XlSQiMieduzY\nwd13393p/U4//XR27NgRhRLFl4KkDYW5IarqGqmoqY93UUSkB9lbkDQ0NLS537x58+jfv3+0ihU3\n6rXVhvzcEACbymrIDnWtW5yI9D033ngjH3/8MRMnTiQ1NZVQKEReXh4rVqxg5cqVnH322axbt46a\nmhq++c1vMnv2bGD3dE2VlZWcdtppHHXUUbz55psMGzaMp556ioyMjDifWdcoSNpQGATJxrIaRuV3\n9rbaIhJtP/37Mj7cUN6txxw3NIefnDW+zW1uvvlmli5dyvvvv8/LL7/MGWecwdKlS3d1n50zZw4D\nBgxg586dHHrooZx33nkMHDhwj2OsWrWKhx9+mPvvv58LLriAxx9/nC996Uvdei6xoqatNuz/2ePc\nmno3m8o18ElE9m7atGl7jMG48847OeSQQ5g+fTrr1q1j1apVn9tn5MiRTJw4EYApU6awdu3aWBW3\n26lG0oYcq+Lc5Nd5pHgxTB0e7+KISAvt1RxipV+/fruev/zyyzz//PO89dZbZGZmctxxx7U6RiM9\nPX3X8+TkZHbu3BmTskaDaiRtSDn0y1SQyfhP5sS7KCLSg2RnZ1NRUdHqurKyMvLy8sjMzGTFihW8\n/fbbMS5d7KlG0pZQDvMyzuT8ssdgy2oYdEC8SyQiPcDAgQM58sgjmTBhAhkZGeTn5+9ad+qpp3Lv\nvfcyduxYDjzwQKZPnx7HksaGOefiXYaomzp1quvqja3+Y86z/PKzS0iffBF84TfdXDIR6azly5cz\nduzYeBejz2nt92pmi5xzU9vbV01b7cgcUMhTHAfvPwzlG+JdHBGRHkdB0o7C3Ax+U3sazjXC250f\nySoi0tcpSNqRnxNincunetQXYOEfYOf2eBdJRKRHUZC0o3lQ4sejr4K6SljwQJxLJCLSsyhI2lEQ\nBMknKSPhgBnw9j1Q33v7e4uIdDcFSTsKcnZPk8JR34LqLfDe/8W5VCIiPYeCpB390lPIDqWwqawG\n9j0CiqbBm3dCY9uzfIqINMvKygJgw4YNzJo1q9VtjjvuONobpnD77bdTXV2963VPmZZeQdIBhbkh\nHyRmvlay4zNY9mS8iyUivczQoUOZO3dul/dvGSQ9ZVp6BUkH5OeE2Ng8cePoU2HwGHjjdkiAwZwi\n8nk33ngjd911167XN910Ez//+c858cQTmTx5MgcddBBPPfXU5/Zbu3YtEyZMAGDnzp1cdNFFjB07\nlnPOOWePubauueYapk6dyvjx4/nJT34C+IkgN2zYwPHHH8/xxx8P+Gnpt2zZAsCtt97KhAkTmDBh\nArfffvuu9xs7dixXXXUV48eP5+STT47KnF5RnSLFzE4F7gCSgd87525usd6C9acD1cAVzrnFwbo5\nwJnAZufchLB9zgduAsYC05xzXRuy3gmFuSFWlpT6F0lJcOQ34W/XwOrnYdSMaL+9iOzNMzfCpg+6\n95gFB8FpN7e5yYUXXsgNN9zAtddeC8Cjjz7K/Pnzuf7668nJyWHLli1Mnz6dL3zhC3u9H/o999xD\nZmYmy5cvZ8mSJUyePHnXul/84hcMGDCAxsZGTjzxRJYsWcL111/PrbfeyksvvcSgQYP2ONaiRYv4\nwx/+wDvvvINzjsMOO4xjjz2WvLy8mExXH7UaiZklA3cBpwHjgIvNbFyLzU4DRgU/s4F7wtb9ETi1\nlUMvBc4FXu3mIu9VQW4GpRW1NDQ2+QUTZkFOEbx+W6yKICI9yKRJk9i8eTMbNmzg3//+N3l5eRQU\nFPDDH/6Qgw8+mJNOOon169dTUlKy12O8+uqruz7QDz74YA4++OBd6x599FEmT57MpEmTWLZsGR9+\n+GGb5Xn99dc555xz6NevH1lZWZx77rm89tprQGymq49mjWQasNo5twbAzB4BZgLhv5GZwEPOT/j1\ntpn1N7NC59xG59yrZjai5UGdc8uD40Wx6HsqyAnR5KC0spbC3AxISYMjroN/3QifvQP7HBazsohI\nmHZqDtF0/vnnM3fuXDZt2sSFF17In//8Z0pLS1m0aBGpqamMGDGi1enj2/PJJ59wyy23sGDBAvLy\n8rjiiiu6dJxmsZiuPprXSIYB68JeFwfLOrtNl5jZbDNbaGYLS0tLIzpW+J0Sd5l8GWTk+WslIpJw\nLrzwQh555BHmzp3L+eefT1lZGUOGDCE1NZWXXnqJTz/9tM39jznmGP7yl78AsHTpUpYsWQJAeXk5\n/fr1Izc3l5KSEp555pld++xt+vqjjz6av/3tb1RXV1NVVcWTTz7J0Ucf3Y1n27Y+O428c+4+4D7w\ns/9GcqzmQYkl4UGS1g8Ouxpe/iVsXg5DNBupSCIZP348FRUVDBs2jMLCQi655BLOOussDjroIKZO\nncqYMWPa3P+aa67hyiuvZOzYsYwdO5YpU6YAcMghhzBp0iTGjBnD8OHDOfLII3ftM3v2bE499VSG\nDh3KSy+9tGv55MmTueKKK5g2bRoAX/3qV5k0aVLM7roYtWnkzexw4Cbn3CnB6x8AOOd+GbbN74CX\nnXMPB68/Ao5zzm0MXo8A/hF+sT1s35eB73TkYnsk08gDbK+qY9LPnuPHZ47jy0ftvp0m1dvgtvEw\nbiacc2+Xjy8iHadp5KOjp04jvwAYZWYjzSwNuAh4usU2TwOXmTcdKGsOkZ6kf2Yq6SlJn793e+YA\nmHIFfPAY7FjX6r4iIn1d1ILEOdcAXAfMB5YDjzrnlpnZ1WZ2dbDZPGANsBq4H/h68/5m9jDwFnCg\nmRWb2VeC5eeYWTFwOPBPM5sfrXMIK8vuQYktHe67//HWXZ9f157KUnj2R3DbBFi3ILJCiojESVSv\nkTjn5uHDInzZvWHPHXDtXva9eC/LnwRiPqw8P2cvQZJbBAddAIsfhGO+C/0Gtn+wqi3wxh2w4PfQ\nUAOpmTDvO3DVS36cioi0yTkX056bfV2klzj0qdVBhbkhNpbvpdvckd+E+mp49762D1K1BZ77Mdx+\nELz1WxhzJlz7LpxxK2x8H5b8tfsLLtLHhEIhtm7dGvGHn3jOObZu3UooFOryMfpsr63uVpCbQUnZ\npta/CQ0ZAweeAe/+Do74BqRn7bm+aquf6PHd+33gHDQLjvkeDB7t1w/YH965F174KYz7gu8RJiKt\nKioqori4mEi79ctuoVCIoqKiLu+vIOmggpx06hqb2FZVx8Cs9M9vcNS34IF/wuKH4PDgUk/VVnjr\nN/DOfWEB8l0YfOCe+yYlwak3w5yTfZPX8T+M/gmJ9FKpqamMHDmy/Q0lZhQkHVSQmwH4QYmtBsnw\nQ2Hfo3yT1YTzfA3j3fugrsq/PvZ7nw+QcPsc5rd7404/2DG3698ORERiSddIOqh5dHtJyy7A4Y76\nFpSvh9vG+Xm4Rp8CX38bZj3Qdog0O+kmwMHzP+2OIouIxIRqJB1U0No0KS0dcKIfnJiU4puwOjva\nvf8+cPh18NotcNjXoKjdcUAiInGnGkkHDcpKJznJWu8C3MwMLngIZs3p+pQpR30LsvL9hJDqlSIi\nvYCCpIOSk4z87PTPj27vbulZcOKPoXgBLH08uu8lItINFCSdkL+30e3d7ZAvQsHB8NxPoL77p3wW\nSRjbPtGsETGgIOmEwtwQG8ti8MGelASn/hLKi+HN30b//UT6oo9fgt8dA3NOgZVRn0kpoSlIOqEg\nJyM2NRKAEUfB2LN876/yHjePpUjPtvhP8OdZkDscCibAo5f7m9BJVChIOqEgN52qukYqaupj84Yz\n/hua6uHFn8Xm/UR6O+fghZ/B09fByGPgy/+CSx6HnEL4ywX+3kHS7RQkndA8KDFmtZIB+/mbZ73/\nF9jwXmzeU6S3aqiFx7/qu89Pvgy++CiEciBrMFz6JKSkw/+dB2XF8S5pn6Mg6YRWb7kbbcd8BzIH\nwr9+qO7AIntTvQ0emglL58KJP4Gz7oTk1N3r80bAlx6H2gr40zl+e+k2CpJOKMjxQRL1LsDhQrlw\nwn/CZ2/C8pb3BZNeyTn/rfijf/m2/C2r9SUhEls/ht+fBOsX+zFcR/+HH9PVUsFBcPHDsP1T+PP5\nfvoi6RYa2d4JQ3L8HFsxa9pqNukyePf3/iZYo06B1K5P99wh9TV+JuMFv4fsQhg6GYZN9o8D9tM9\nUzqjrhpKl0PJMti01D+WLIWaHXtul7sP7H887H+Cb9vPHBCf8vY2n70DD1/kn1/+NOwzve3tRxzl\npyx69DJ/Af7ih/esuUiXKEg6IT0lmUFZabFt2gJIToFTfgF/OttPBnnUDdF5n6YmWPaEn+ur7DMY\ncTQ0NcCiP8I79/ht0nNh6MTdwTJsMuQMa/0bYKKp2OSvZZUs3R0a2z4G1+TXp/aD/HEw/mzIn+B/\nMgfA2tfh4xdh2ZP+BmmWBEMn+VDZ/wQoOjT2H3Y7t/vHpBSwZEhKDp4n9Zy/9dLH4clr/ASnlzwG\nA/fv2H5jz4Izb4O/fxOeug7OvkdfjiKkIOmk/JxQ2xM3Rsv+x8Po0+DVW2DiFyFrSPcef+3r8Ox/\n+Q/CgoNg5lOw33F+XWMDlK6ADYt988GGxfDmb3zIAPQbsjtYhoz1FzjTsvx9VVIzdz9PSe85H0Ld\nqa4KXr7Z327ZNfpl/ff1v8cJ50L+eB8aeSNb/8AafCAc+hX/e16/yIfKxy/Ca7+GV//X//5GHL07\nWAbu3/2/x4Y6+PQNWPUsrPwXbFuz9233CJbm58mQMxQOmAGjTg7CL0ofL875bvEv/BT2ORwu+kvn\na3BTrvC3un7p59BvkP+iJl1miXCXsalTp7qFCxd2y7G++uAC1u+o4ZlvHt0tx+uULavh7sNg0pfg\nrDu66Zir/Aj6j/7paxYn/AgOvrD9b2j1Nf6bd3OwrF8MW1YCbfx7smQfKHv8ZEFGng/GrALIzt/z\nsd/g6H0gdYdVz8E//wN2fOb/LpMuhSHjfJhGaucOWPuaD5XVL8COT/3yrIKwGuEk/9iVprCKTUFw\nzIc1L0NdJSSnw8ijfXClpPsvC02N/tE17fl6j2UNULoSPnvLh2mov5/EdNQp/rHfoMh/HwCN9f73\nvfghmDALZt7V9aZe5+CZ7/nbPcz4GRx5ffeUsQ8xs0XOuXZnj+3B/0N7poLcEIs+3R6fNx90AEyb\n7Zu3Dr3KD7TqqspSeOVmWPgHX2s48ccw/euQmtGx/VNDfnbi8BmKa8r9N9m6Kn8jr7pK//xzP5W7\nt6mt9Pt8+ibsbK0njfkwyc73k1lmFUDuMJh8uX+Ml4pNfmLNZU/CoNFwxTwYcWT3vkdGf98MM/Ys\n/3rbGh8qn73jw/ujebu3zRvhA2XoJB8yhYdAevaex2tq8jXOVfN9eGx83y/PGQYHne9vezDymMju\n0LlzB6x5yQfsqmeD+eLM/zsZdTKMmgEFh3SsKck538RWWeJ/Kkrg3w/74x/9HTj+PyNrkjLzN5Sr\nKoXnfuT/nU28uOvH60maGv2YmXXvwNgv+C7QUaQaSSf99sVV3PLsSlb87FRCqcndcsxO2bkd7pwE\n2UNh/Dn+A2TASN9skjmg/SaP+p3w9t3w2m3+g3zql+HY70f9H1qHNNSFfWhsgspNULk5eF6y+7Gy\nxPdmO/seOPC02JaxqQkWzfHXkRpq/e0Cjrzef3uPtZ07fBhseC+oGb4HZeuCleabzIZO9k1rmz/0\nH+xVpf46R9Gh/oN99Cm+2S0aTY5NTb58q57z4bV+MeD8F4IDZsCok/yXmNb+vhXBY2PtnsdMToMz\nboXJl3ZfORtqfS+uta/7i++jT+nYflVb/Id16Qpf9vFn++bMeKit8BO9rnvXh0fxQqgt9+su+JO/\nhXcXdLRGoiDppLmLivnOY//mle8ex74D43Rv9aVPwPz/hIoNey5Pz9kzWMIfswvhg7l+lHz5en+P\n+Rk/hUGj4nIKEdn6MTx2BWxaAtOv9TcES0mL/vtuWgr/uMH/hx15rL9g29ELvLFSWeoDJfx6VlWp\nD94DZvgPyQNOik+vsMpSWP28D7SPX4Casj3XZwyA7IKg5pnfopkzeJ5TGFmNaW9qK+CPZ0LpR773\n1/Bpu9dVbfU975pDY/MK/1i95fPH2e94OOIb/lpWtK4HOuebOde9C5+97R83Lws6dZj/4jD8sOBn\nmv9M6GJZFCRhujNI3li9hUt+/w6PzJ7O9P0Gdssxu6x+p+8Tv/0TP8tp+OP2T/30Ks0syf9DGzoJ\nTv5F9zfDxFpDre8O/e7v/DnNmuO7JkdDXTW88v/8bZRDuXDKL+HgC3pHxwHnfK0uc2DPutbU2OAD\nz5KC62P5sfky0JbKUphzsh+seNAsHyqlK3wQN0vP8TW9wWN8x5LBY/xPakbQu/FeX5PKP8gHyoRz\nu6fH3daPfQB/+oYPjsoSvzwt2zcbDj/M36572NTuuT4XUJCE6c4gWb25kpNufYU7LprIzIlxbKNv\nT1Ojr3mEB0vBBBh3Tt/q6vjh035eJed8B4QJ53bv8fe4mH6pn/9MYzz6ru1r/Qj5qq0+MIaMgcFj\ndz/mDG37C0RDLXzwmO/VWLrCX3+afo2/pteZD/g9etHN993IwfcG3Ge6r2kMP8x37EiKXhO7giRM\ndwZJZW0DE34ynxtPG8PVx/awZo1Etf1TePwrvslpypV+Cv6OdhrYm7L1vjv0sif8xfQzb+/9tTjp\nmKYmHxaR1DibmmD1cz5Q1r7mazJTrvChkjO09X3a6kU36hTfUWHAyK6XqQt6RK8tMzsVuANIBn7v\nnLu5xXoL1p8OVANXOOcWB+vmAGcCm51zE8L2GQD8FRgBrAUucM7FrBtVVnoK2aGU2I9ul73L2xeu\nfMZf/3njDl/1P/8P/htlZ+z4DJb/w09F89nb/sLu8f8Vv4vpEh/dUWNPSvLXo0af4q9Vvfkb3zT6\n9t2+h9wR3/A1nA2LfXCsmg8b/+337c5edDEStRqJmSUDK4EZQDGwALjYOfdh2DanA9/AB8lhwB3O\nucOCdccAlcBDLYLkV8A259zNZnYjkOec+35bZenOGgnAjFtfYf/BWdx76ZRuO6Z0k1XPw5Nf8z3S\nTg8Gb7b1zXLLKh8cHz69uzts/gTf5fbgC2P+DVD6sO1r4a274b0/+X+foVzf4cCSoGgajD7Z1zzy\nx/eY6289oUYyDVjtnFsTFOgRYCbwYdg2M/FB4YC3zay/mRU65zY65141sxGtHHcmcFzw/EHgZaDN\nIOluBbkhNsZjdLu0b9RJcPXr8MRV8NTX4ZNX4Ixf7x5T4Rxs+gCW/90HSOkKv3zYVDjppz5AelpP\nLOkb8kbA6b+C426EhXP89cv9jvMDNnv5dbdoBskwYF3Y62J8raO9bYYBbd0SMN8517x+E5AfYTk7\nrTA3xMqS0vY3lPjIKYTLnvLTybxys+9Tf9JP/DWU5X/33wwtCfY90o+jGXNmfAc3SmLJHOBvD9GH\n9KD+gJ3nnHNm1mrbnJnNBmYD7LPPPt36vgU5IUoramlobCIluQ/1gOpLkpLhuO/7C+SPf9XP9pqU\nCvsdC0f9Bxx4es8YhCnSB0QzSNYDw8NeFwXLOrtNSyXNzV9mVghsbm0j59x9wH3gr5F0puDtKcjN\noMlBaWUthbkR9g6S6BpxFFzzpq+VDJ/mpx0RkW4Vza/TC4BRZjbSzNKAi4CWd2Z6GrjMvOlAWViz\n1d48DVwePL8ceKo7C90RcblTonRd5gB/IVMhIhIVUQsS51wDcB0wH1gOPOqcW2ZmV5vZ1cFm84A1\nwGrgfuDrzfub2cPAW8CBZlZsZl8JVt0MzDCzVcBJweuYyg/ulFiiIBERie41EufcPHxYhC+7N+y5\nA67dy76tTsPpnNsKnNiNxew01UhERHbTleIu6J+ZSnpKUmzv3S4i0kMpSLrAzCjIDWl0u4gICpIu\nK8hRkIiIgIKkywpzQ2ws3xnvYoiIxJ2CpIvyc0OUlNWSCLMni4i0RUHSRYU5Ieoam9hWVRfvooiI\nxJWCpIsKghHt6gIsIolOQdJFBcFYkhJ1ARaRBKcg6SINShQR8RQkXTQoK53kJFMXYBFJeAqSLkpO\nMoZkp2t0u4gkPAVJBDS6XUREQRKRwtwQG8s0KFFEEpuCJAL5OSFKymvjXQwRkbhSkESgMDdEZW0D\nFTX18S6KiEjcKEgi0DwoUddJRCSRKUgiUBDcKVE9t0QkkSlIIqBBiSIiCpKIDMlJB9S0JSKJTUES\ngfSUZAb2S1PTlogkNAVJhDQoUUQSnYIkQn5QooJERBKXgiRCflCigkREEpeCJEKFuSG2VdVRU98Y\n76KIiMSFgiRCzYMSVSsRkUSlIInQrkGJuk4iIgkqqkFiZqea2UdmttrMbmxlvZnZncH6JWY2ub19\nzewQM3vLzD4ws7+bWU40z6E9zbfcVRdgEUlUUQsSM0sG7gJOA8YBF5vZuBabnQaMCn5mA/d0YN/f\nAzc65w4CngS+G61z6IgCjW4XkQQXzRrJNGC1c26Nc64OeASY2WKbmcBDznsb6G9mhe3sOxp4NXj+\nHHBeFM+hXVnpKWSnp6hpS0QSVjSDZBiwLux1cbCsI9u0te8ydofK+cDw1t7czGab2UIzW1haWtql\nE+goDUoUkUTWGy+2fxn4upktArKButY2cs7d55yb6pybOnjw4KgWqCA3xEZdIxGRBJUSxWOvZ8/a\nQlGwrCPbpO5tX+fcCuBkADMbDZzRraXugoKcECtLolvrERHpqTpUIzGzb5pZTtDL6gEzW2xmJ7ez\n2wJglJmNNLM04CLg6RbbPA1cFhx3OlDmnNvY1r5mNiR4TAL+C7i3g+caNYW5IUoramlobIp3UURE\nYq6jTVtfds6V42sCecClwM1t7eCcawCuA+YDy4FHnXPLzOxqM7s62GwesAZYDdwPfL2tfYN9Ljaz\nlcAKYAPwhw6eQ9QU5GbQ5KC0UvdvF5HE09GmLQseTwf+FASCtbUDgHNuHj4swpfdG/bcAdd2dN9g\n+R3AHR0sd0wU5Pr7kmwsq6EwGOkuIpIoOlojWWRmz+KDZL6ZZQNqxwkU5ATTpKjnlogkoI7WSL4C\nTATWOOeqzWwAcGX0itW76Ja7IpLIOlojORz4yDm3w8y+hL/IXRa9YvUu/TNTSUtJ0jQpIpKQOhok\n9wDVZnYI8G3gY+ChqJWqlzEzCjUoUUQSVEeDpCG4MD4T+K1z7i78YEAJFOQoSEQkMXU0SCrM7Af4\nbr//DMZwpEavWL2PH92+M97FEBGJuY4GyYVALX48ySb8SPP/jVqpeqGC3BAlZbX4ipuISOLoUJAE\n4fFnINfMzgRqnHO6RhKmMCdEXWMT26panfpLRKTP6ugUKRcA7+Jn270AeMfMZkWzYL2N7ksiIomq\no+NI/hM41Dm3GcDMBgPPA3OjVbDeJvze7ROG5ca5NCIisdPRayRJzSES2NqJfROCBiWKSKLqaI3k\nX2Y2H3g4eH0hrcyDlcgGZaWTnGTqAiwiCadDQeKc+66ZnQccGSy6zzn3ZPSK1fskJxlDstM1ul1E\nEk6Hb2xXxGgiAAAVnElEQVTlnHsceDyKZen1dMtdEUlEbQaJmVUArQ2MMPws8DlRKVUv5e+UWBHv\nYoiIxFSbQeKc0zQonVCQG+K1VVviXQwRkZhSz6tuVJgborK2gYqa+ngXRUQkZhQk3Sg/x3cB1nUS\nEUkkCpJu1HybXfXcEpFEoiDpRhqUKCKJSEHSjYbkpANq2hKRxKIg6UbpKckMykrn49LKeBdFRCRm\nFCTd7KSxQ3juwxKqahviXRQRkZhQkHSzWVOKqK5rZN4HG+NdFBGRmFCQdLMp++YxclA/5i4qjndR\nRERiIqpBYmanmtlHZrbazG5sZb2Z2Z3B+iVmNrm9fc1sopm9bWbvm9lCM5sWzXPoLDNj1pQi3vlk\nG59trY53cUREoi5qQWJmycBdwGnAOOBiMxvXYrPTgFHBz2zgng7s+yvgp865icCPg9c9yjmThmEG\ncxerViIifV80ayTTgNXOuTXOuTrgEWBmi21mAg85722gv5kVtrOvA5oni8wFNkTxHLpkaP8Mjjpg\nEI8vKqapqbU5L0VE+o5oBskwYF3Y6+JgWUe2aWvfG4D/NbN1wC3AD1p7czObHTR9LSwtLe3ySXTV\nrClFrN+xk7c/2Rrz9xYRiaXeeLH9GuBbzrnhwLeAB1rbyDl3n3NuqnNu6uDBg2NaQIBTxheQnZ7C\n3IVq3hKRvi2aQbIeGB72uihY1pFt2tr3cuCJ4Plj+GawHieUmsyZhwxl3tKNmg1YRPq0aAbJAmCU\nmY00szTgIuDpFts8DVwW9N6aDpQ55za2s+8G4Njg+QnAqiieQ0TOn1pETX0Tz3ywKd5FERGJmg7f\nareznHMNZnYdMB9IBuY455aZ2dXB+nuBecDpwGqgGriyrX2DQ18F3GFmKUANvrdXjzRpeH/2G9yP\nxxat44JDh7e/g4hILxS1IAFwzs3Dh0X4snvDnjvg2o7uGyx/HZjSvSWNjuYxJb/610es3VLFiEH9\n4l0kEZFu1xsvtvcq504qIsngcY0pEZE+SkESZQW5IY4eNVhjSkSkz1KQxMCsKUVsKKvhzY81pkRE\n+h4FSQzMGJdPTiiFuYvWtb+xiEgvoyCJgVBqMl+YOJR/LdtEucaUiEgfoyCJkVlThlNT38Q/l+g+\nJSLStyhIYuSQolwOGJKl+5SISJ+jIIkRM+P8KUUs+nQ7a3RPdxHpQxQkMXTOpGEkGaqViEifoiCJ\noSE5IY4dPZgnFq+nUWNKRKSPUJDE2PlTh7OpvIY3Vm+Jd1FERLqFgiTGThw7hNyMVB5T85aI9BEK\nkhhLT0lm5sShzF+2ibKdGlMiIr2fgiQOZk0poq6hiX8s6XG3mxcR6TQFSRwcNCyXA/OzeUy34RWR\nPkBBEgfN9yl5f90OVm+uiHdxREQioiCJk5mThpKcZMxd1PI29iIivYuCJE6GZIc4/sDBPPlescaU\niEivpiCJo1lTiigpr+XVVaXxLoqISJcpSOLohDH55GWmasoUEenVFCRxlJaSxMyJw3huWQnbq+ri\nXRwRkS5RkMTZRdOG09DUxA1/fZ/6xqZ4F0dEpNMUJHE2piCH/znnIF5ZWcr3H1+Cc7rwLiK9S0q8\nCyBw0bR9KCmv5bbnV1KQE+J7p46Jd5FERDpMQdJDXH/iAWwqr+Hulz8mPyfE5UeMiHeRREQ6JKpN\nW2Z2qpl9ZGarzezGVtabmd0ZrF9iZpPb29fM/mpm7wc/a83s/WieQ6yYGT+bOZ6TxuZz09+XMe8D\n3dtdRHqHqAWJmSUDdwGnAeOAi81sXIvNTgNGBT+zgXva29c5d6FzbqJzbiLwOPBEtM4h1lKSk/jN\nxZOYvE8eNzzyPm+v2RrvIomItCuaNZJpwGrn3BrnXB3wCDCzxTYzgYec9zbQ38wKO7KvmRlwAfBw\nFM8h5jLSknng8qkMH5DBVQ8tZMWm8ngXSUSkTdEMkmHAurDXxcGyjmzTkX2PBkqcc6tae3Mzm21m\nC81sYWlp7xo53j8zjQe/PI3MtGSumLOADTt2xrtIIiJ71Zu7/15MG7UR59x9zrmpzrmpgwcPjmGx\nukdRXiZ/vHIaVbUNXDbnXXZUa8CiiPRM0QyS9cDwsNdFwbKObNPmvmaWApwL/LUby9vjjC3M4b7L\npvLZ1mq++uBCauob410kEZHPiWaQLABGmdlIM0sDLgKebrHN08BlQe+t6UCZc25jB/Y9CVjhnOvz\nk1Qdvv9Abr3wEBZ9tp3rH35PMwWLSI8TtSBxzjUA1wHzgeXAo865ZWZ2tZldHWw2D1gDrAbuB77e\n1r5hh7+IPnaRvS1nHjyUH585jmc/LOHHTy3V6HcR6VEsET6Upk6d6hYuXBjvYkTsl88s53evrOHb\nM0bzjRNHxbs4ItLHmdki59zU9rbTyPZe5PunjKG0vJZfP7eSQdnpXDxtn3gXSUSkV/faSjhJScb/\nm3Uwx4wezA+e+ICr/7SIdduq410sEUlwCpJeJjU5ifsvm8J3Th7NKytLOenWV7jtuZXq0SUicaMg\n6YXSU5K57oRRvPDtY5kxLp87XljFib9+hX8t3agL8SIScwqSXmxo/wx++8XJPHzVdLLSU7j6/xZz\n6QPvsnpzRbyLJiIJREHSBxy+/0D+ef1R3HTWOJYU7+DU21/j5//4kPKa+ngXTUQSgIKkj0hJTuKK\nI0fy0neO4/ypRTzwxieccMsrPLZwHU0axCgiUaQg6WMGZqXzy3MP5qlrj2T4gAy+O3cJ5937JkuK\nd8S7aCLSRylI+qiDi/rz+NVH8OvzD2Hdtp3MvOsNvvanhcxftonaBvXwEpHuowGJfVhSknHelCJO\nHp/P3S9/zGML1zF/WQm5GamceXAh50waxpR98/C3dhER6RpNkZJAGhqbeH31Fv723nrmLythZ30j\nwwdkcPbEYZw9aRj7D86KdxFFpAfp6BQpCpIEVVnbwLPLNvHke+t5Y/UWmhwcUpTL2ZOGcdYhQxmU\nlR7vIopInClIwihI2lZSXsPf/72BJ99bz7IN5SQnGUePGsQ5k4ZxwpghZIdS411EEYkDBUkYBUnH\nrSyp4Mn31vPUe+vZUFZDSpIxbeQAThgzhBPGDGE/NX+JJAwFSRgFSec1NTkWfbadF5Zv5sUVJaws\nqQRg5KB+u0Ll0BEDSEtRxz+RvkpBEkZBErl126p56aPNvLB8M2+t2UpdQxNZ6SkcM3oQxx84hOPH\nDNF1FZE+RkESRkHSvarrGnhj9VZeXFHCiys2U1JeixkcUtSfY0YPZvzQHMYV5lCUl6GuxSK9mG5s\nJVGTmZbCjHH5zBiXj3OOZRvKeXHFZl5csZnfvLiK5u8m2ekpjC3MYWxhdvCYw4EF2YRSk+N7AiLS\nrVQjkW5VXdfAik0VLN9YHvxUsGJjOVV1fjR9kvnrLM3BMq4whzGF2RTkhFR7EelhVCORuMhMS2Hy\nPnlM3idv17KmJse67dUs31jOhxt9yLy/bgf/WLJx1zbZ6SmMys9idH42o/KzGR08H5KdroAR6eFU\nI5G4Ka+pZ8XGCj7aVM7KkkpWllSwanMl26rqdm2TE0r5XLiMys9icJYCRiTaVCORHi8nlMq0kQOY\nNnLAHsu3VNb6UGkOl5JKnlm6kYff3X1/lYzUZIblZVCUl8Gw/hkU5WXuel3UP4NBWekkJSloRGJB\nQSI9zqCsdAZlpXPE/oN2LXPOUVpZy6qSSlaVVLBu+07Wb99J8Y5q/r1uB9ur97yJV1pKUhAwPmiG\n9c8gPzdEfk6I/Jx0CnJC5GakqlYj0g0UJNIrmBlDskMMyQ5x5AGDPre+qraB9Tt2Ury92gfM9p0U\n7/CPy5dvZktl7ef2SU9J2hUs/jFEQU6IIcHrgmBZRpp6mYm0RUEifUK/dH8tZXR+dqvra+obKa2o\npaS8hk3lNZSU++cl5TVsKqth2YZyXli+mZ31n79XS3YoZXfgZIcYskf4pPuAy0knPUWBI4kpqkFi\nZqcCdwDJwO+dcze3WG/B+tOBauAK59zi9vY1s28A1wKNwD+dc9+L5nlI7xdKTWb4gEyGD8jc6zbO\nOSpqG9hcXsOmsiBoKmrYHBY673yyjc0VNdQ3fr6TSl5mKnmZaeRkpJK7l5/wdf0z/U9mmr7PSe8W\ntX/BZpYM3AXMAIqBBWb2tHPuw7DNTgNGBT+HAfcAh7W1r5kdD8wEDnHO1ZrZkGidgyQWMyMnlEpO\nKJUDhrReswHfnXnHzno2lTUHze4azo6d9ZTvrGd7dR1rt1ZRFrxuaqNzZL+0ZIbkhBiclc7gnHSG\nZPtazuDs4HlOOoOz0snLTFMHAumRovlVaBqw2jm3BsDMHsEHQHiQzAQecr4P8ttm1t/MCoERbex7\nDXCzc64WwDm3OYrnIPI5SUnGgH5pDOiXxjhy2t2+qclRWddAWXX9rmDZsdM/315dR2lFLaUVtWyu\nqOXDDeW8XF6zawBnuJQkY3B2OgOz0uifkUZuZir9m2s2e7xOC5alkpuZqiY3ibpoBskwYF3Y62J8\nraO9bYa1s+9o4Ggz+wVQA3zHObeg5Zub2WxgNsA+++zT9bMQiVBS0u6azvAO7lNV27ArXPxjDZsr\natlcXsv26jp2VNexoWwnZdU+lBrbqPJkpCaTk5FCdiiVnFAKORmpu55nh1LJyUghJ5RKdrAuJ1iX\nm+mb4BRE0p7e2DibAgwApgOHAo+a2X6uxchK59x9wH3gByTGvJQiEeiXnkK/9BRGDOrX7rbOOSpr\nG9gR1Hh2VNezY2dd2Os6ync2UFFbT/nOBrZV1fHp1mrKd9ZTXlPf6vWecJlpyXte18lI2/U8N+x1\nViiFUEoS6anJpKckEQoew5+nJOu2A31RNINkPezxBawoWNaRbVLb2LcYeCIIjnfNrAkYBJR2X9FF\neg8zIzvkaxkdrfE0c85R29AUhEoD5TX1VNQ0UBY0vZVV1wXB1Py6njVbKnctq2to6tT7JSfZHsGS\nmZZMVhCaWcFPv/QUskLB87RkskKpZKUn7wrXrPSUXftlpqXonjg9QDSDZAEwysxG4kPgIuCLLbZ5\nGrguuAZyGFDmnNtoZqVt7Ps34HjgJTMbDaQBW6J4HiJ9lpkRSk0mlJrMkPYv93xOTX3jrhpQZU0D\ntQ1N1DY0UlvfRE3w2Lyspn73utqGJmrqG6mua6SytoHK2ga2VVVTWdtAVfC6vZpSs7TkJDLTk+mX\nlkK/9GQyg0f/OoWMtOaaUfCYGvY8rAa1a5vUJDJS/f7Nxw2lJmnwahuiFiTOuQYzuw6Yj+/CO8c5\nt8zMrg7W3wvMw3f9XY3v/ntlW/sGh54DzDGzpUAdcHnLZi0RiY1QajIFuckU5Ia6/di1DY1U1TZS\nVdtARU0DVXU+YKqDZVV1DbuCqLq2gaq65uX+cWtlNdV1jVTXNQdcU6drUM3MoF+aD6V+abvDKjPN\n146awyo1efdPWrKRsuu1kbbHeiMtuWWoJRNK3R1m4eHX03vradJGEUkYTU2OusbdtaS6IGBqm2tL\nQU2ppt6HWHWdD6bmoKoOwqvluuq6Ruoam2hobKK+0b9HV0OrNanJRnpKMinJRpIZSeZrkwZ7vE5K\nan5tmPnn/3POQZ+bz66jNGmjiEgLSUlGKCk5uLlaalTfyzlHY5OjvtFR39REfYMPmfrGpl1B40Os\ncVeNqbZFc2BzsDU/NjU5mhw0Of/onMO1eN38vClY1y89+r3uFCQiIlFgZqQkGynJkEHf7kKt7g4i\nIhIRBYmIiEREQSIiIhFRkIiISEQUJCIiEhEFiYiIRERBIiIiEVGQiIhIRBJiipRgEshPu7j7IBJ7\nUshEPn+de+JK5PMPP/d9nXOD29shIYIkEma2sCNzzfRViXz+OvfEPHdI7PPvyrmraUtERCKiIBER\nkYgoSNp3X7wLEGeJfP4698SVyOff6XPXNRIREYmIaiQiIhIRBYmIiEREQdIGMzvVzD4ys9VmdmO8\nyxNLZrbWzD4ws/fNrM/fp9jM5pjZZjNbGrZsgJk9Z2argse8eJYxWvZy7jeZ2frg7/++mZ0ezzJG\ni5kNN7OXzOxDM1tmZt8MlifK335v59+pv7+ukeyFmSUDK4EZQDGwALjYOfdhXAsWI2a2FpjqnEuI\nQVlmdgxQCTzknJsQLPsVsM05d3PwRSLPOff9eJYzGvZy7jcBlc65W+JZtmgzs0Kg0Dm32MyygUXA\n2cAVJMbffm/nfwGd+PurRrJ304DVzrk1zrk64BFgZpzLJFHinHsV2NZi8UzgweD5g/j/YH3OXs49\nITjnNjrnFgfPK4DlwDAS52+/t/PvFAXJ3g0D1oW9LqYLv+BezAHPm9kiM5sd78LESb5zbmPwfBOQ\nH8/CxME3zGxJ0PTVJ5t2wpnZCGAS8A4J+Ldvcf7Qib+/gkT25ijn3ETgNODaoPkjYTnfBpxI7cD3\nAPsBE4GNwK/jW5zoMrMs4HHgBudcefi6RPjbt3L+nfr7K0j2bj0wPOx1UbAsITjn1gePm4En8U19\niaYkaENubkveHOfyxIxzrsQ51+icawLupw///c0sFf8h+mfn3BPB4oT527d2/p39+ytI9m4BMMrM\nRppZGnAR8HScyxQTZtYvuPCGmfUDTgaWtr1Xn/Q0cHnw/HLgqTiWJaaaP0QD59BH//5mZsADwHLn\n3K1hqxLib7+38+/s31+9ttoQdHm7HUgG5jjnfhHnIsWEme2Hr4UApAB/6evnbmYPA8fhp9AuAX4C\n/A14FNgHfxuCC5xzfe6i9F7O/Th8s4YD1gJfC7tm0GeY2VHAa8AHQFOw+If46wSJ8Lff2/lfTCf+\n/goSERGJiJq2REQkIgoSERGJiIJEREQioiAREZGIKEhERCQiChKRHs7MjjOzf8S7HCJ7oyAREZGI\nKEhEuomZfcnM3g3u3/A7M0s2s0ozuy2418MLZjY42Haimb0dTIr3ZPOkeGZ2gJk9b2b/NrPFZrZ/\ncPgsM5trZivM7M/BiGSRHkFBItINzGwscCFwZDDZZSNwCdAPWOicGw+8gh81DvAQ8H3n3MH4UcXN\ny/8M3OWcOwQ4Aj9hHvhZWW8AxuEn0zsy6icl0kEp8S6ASB9xIjAFWBBUFjLwE/01AX8Ntvk/4Akz\nywX6O+deCZY/CDwWzG82zDn3JIBzrgYgON67zrni4PX7wAjg9eiflkj7FCQi3cOAB51zP9hjodmP\nWmzX1TmJasOeN6L/u9KDqGlLpHu8AMwysyGw657f++L/j80Ktvki8LpzrgzYbmZHB8svBV4J7lBX\nbGZnB8dIN7PMmJ6FSBfoW41IN3DOfWhm/wU8a2ZJQD1wLVAFTAvWbcZfRwE/Nfm9QVCsAa4Mll8K\n/M7M/js4xvkxPA2RLtHsvyJRZGaVzrmseJdDJJrUtCUiIhFRjURERCKiGomIiEREQSIiIhFRkIiI\nSEQUJCIiEhEFiYiIROT/A9nr/P+558eGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7feacac28a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_train(hist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:55.215709Z",
     "start_time": "2017-09-09T13:35:54.478862Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "80896/84434 [===========================>..] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "preds = np.squeeze(model.predict(map_valid, 1024, verbose=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:55.227457Z",
     "start_time": "2017-09-09T13:35:55.220149Z"
    },
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def log_max_inv(preds, mx = max_log_y):\n",
    "    return np.exp(preds * mx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:55.259467Z",
     "start_time": "2017-09-09T13:35:55.231530Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7590.77\n",
      "7956.0\n",
      "0.04590598634285401\n"
     ]
    }
   ],
   "source": [
    "print(log_max_inv(preds, max_log_y)[0])\n",
    "print(log_max_inv(y_valid, max_log_y)[0])\n",
    "print(math.sqrt(np.square(((log_max_inv(preds, max_log_y)[0] - log_max_inv(y_valid, max_log_y)[0])/log_max_inv(y_valid, max_log_y)[0]).mean())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:55.271594Z",
     "start_time": "2017-09-09T13:35:55.263430Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  7590.77197266   9131.375        4811.43310547   6522.05419922\n",
      "   6024.72216797   5465.66601562   6593.625        7859.96728516\n",
      "   5332.12841797   6896.484375    10246.453125     7406.88916016\n",
      "   6931.609375     5271.66113281   7346.73925781   5328.95117188\n",
      "   8538.99609375   5246.24365234   3525.50170898   6758.55517578 ...,\n",
      "   6318.23974609  22252.78515625   4690.26855469   5588.13476562\n",
      "   9250.89746094  25776.7578125    3335.5390625    3511.1340332\n",
      "   3764.37255859   8824.31347656   4547.98925781   3932.67749023\n",
      "   4162.41162109  17020.87890625   5405.68408203   4393.96875\n",
      "  14539.59570312   5936.38232422   4748.70263672   7483.52197266]\n"
     ]
    }
   ],
   "source": [
    "preds_log_max_inv = log_max_inv(preds, max_log_y)\n",
    "print(preds_log_max_inv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:55.283477Z",
     "start_time": "2017-09-09T13:35:55.275569Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  7956.   8958.   4676.   6913.   6157.   5010.   6723.   8033.   5196.\n",
      "   6413.  10012.   6619.   6806.   4765.   6973.   5080.  11710.   5088.\n",
      "   3612.   6362. ...,   6218.  20642.   3697.   4220.   6851.  17267.\n",
      "   3102.   2401.   3139.   9643.   3113.   2646.   2907.   8498.   3821.\n",
      "   3375.  10765.   5035.   4491.   5961.]\n"
     ]
    }
   ],
   "source": [
    "y_valid_array =  log_max_inv(np.array(y_valid), max_log_y)\n",
    "print(y_valid_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-09-09T13:35:55.295476Z",
     "start_time": "2017-09-09T13:35:55.287400Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.230133434029\n"
     ]
    }
   ],
   "source": [
    "print(sqrt(mean(square((preds_log_max_inv - y_valid_array)/y_valid_array))))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "toc": {
   "nav_menu": {
    "height": "133px",
    "width": "254px"
   },
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {
    "height": "906px",
    "left": "0px",
    "right": "1411px",
    "top": "76px",
    "width": "293px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}