10. Keras Wide and Deep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![UCI](http://mlr.cs.umass.edu/ml/assets/logo.gif)\n",
    "\n",
    "# Loading in the Data\n",
    "In this example, we are going to use crossed columns and embedding columns inside of a tensorflow object created with the contrib \"learn\" library.\n",
    "\n",
    "However, we will start the process by loading up a dataset with a mix of categorical data and numeric data. This dataset is quite old and has been used many times in machine learning examples: the census data from 1990's. We will use it to predict if a person will earn over or under 50k per year.\n",
    "\n",
    "- https://archive.ics.uci.edu/ml/datasets/Census-Income+(KDD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 32561 entries, 0 to 32560\n",
      "Data columns (total 15 columns):\n",
      "age                32561 non-null int64\n",
      "workclass          32561 non-null object\n",
      "fnlwgt             32561 non-null int64\n",
      "education          32561 non-null object\n",
      "edu_num            32561 non-null int64\n",
      "marital_status     32561 non-null object\n",
      "occupation         32561 non-null object\n",
      "relationship       32561 non-null object\n",
      "race               32561 non-null object\n",
      "sex                32561 non-null object\n",
      "cap_gain           32561 non-null int64\n",
      "cap_loss           32561 non-null int64\n",
      "work_hrs_weekly    32561 non-null int64\n",
      "country            32561 non-null object\n",
      "income             32561 non-null object\n",
      "dtypes: int64(6), object(9)\n",
      "memory usage: 3.7+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "headers = ['age','workclass','fnlwgt','education','edu_num','marital_status',\n",
    "           'occupation','relationship','race','sex','cap_gain','cap_loss','work_hrs_weekly','country','income']\n",
    "df_train_orig = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data',names=headers)\n",
    "df_test_orig = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test',names=headers)\n",
    "df_test_orig = df_test_orig.iloc[1:]\n",
    "print(df_train_orig.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from copy import deepcopy\n",
    "df_train = deepcopy(df_train_orig)\n",
    "df_test = deepcopy(df_test_orig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is organized as follows: \n",
    "\n",
    "|Variable | description|\n",
    "|----|--------|\n",
    "|age: | continuous|\n",
    "|workclass:      |Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, ...|\n",
    "|fnlwgt:         |continuous.|\n",
    "|education:      |Bachelors, Some-college, 11th, HS-grad, Prof-school, ...|\n",
    "|education-num:  |continuous.|\n",
    "|marital-status: |Married-civ-spouse, Divorced, Never-married, Separated, Widowed, ... |\n",
    "|occupation:     |Tech-support, Craft-repair, Other-service, ...|\n",
    "|relationship:   | Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.|\n",
    "|race:           |White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.|\n",
    "|sex:            |Female, Male.|\n",
    "|capital-gain:   |continuous.|\n",
    "|capital-loss:   |continuous.|\n",
    "|hours-per-week: |continuous.|\n",
    "|native-country: |United-States, Cambodia, England, ... |\n",
    "|**income (target):**|<50k, >=50k| "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>edu_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>cap_gain</th>\n",
       "      <th>cap_loss</th>\n",
       "      <th>work_hrs_weekly</th>\n",
       "      <th>country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25</td>\n",
       "      <td>Private</td>\n",
       "      <td>226802.0</td>\n",
       "      <td>11th</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>89814.0</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Farming-fishing</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>28</td>\n",
       "      <td>Local-gov</td>\n",
       "      <td>336951.0</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Protective-serv</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>44</td>\n",
       "      <td>Private</td>\n",
       "      <td>160323.0</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>7688.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>34</td>\n",
       "      <td>Private</td>\n",
       "      <td>198693.0</td>\n",
       "      <td>10th</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  age   workclass    fnlwgt      education  edu_num       marital_status  \\\n",
       "1  25     Private  226802.0           11th      7.0        Never-married   \n",
       "2  38     Private   89814.0        HS-grad      9.0   Married-civ-spouse   \n",
       "3  28   Local-gov  336951.0     Assoc-acdm     12.0   Married-civ-spouse   \n",
       "4  44     Private  160323.0   Some-college     10.0   Married-civ-spouse   \n",
       "6  34     Private  198693.0           10th      6.0        Never-married   \n",
       "\n",
       "           occupation    relationship    race    sex  cap_gain  cap_loss  \\\n",
       "1   Machine-op-inspct       Own-child   Black   Male       0.0       0.0   \n",
       "2     Farming-fishing         Husband   White   Male       0.0       0.0   \n",
       "3     Protective-serv         Husband   White   Male       0.0       0.0   \n",
       "4   Machine-op-inspct         Husband   Black   Male    7688.0       0.0   \n",
       "6       Other-service   Not-in-family   White   Male       0.0       0.0   \n",
       "\n",
       "   work_hrs_weekly         country   income  \n",
       "1             40.0   United-States   <=50K.  \n",
       "2             50.0   United-States   <=50K.  \n",
       "3             40.0   United-States    >50K.  \n",
       "4             40.0   United-States    >50K.  \n",
       "6             30.0   United-States   <=50K.  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "import numpy as np\n",
    "\n",
    "# let's just get rid of rows with any missing data\n",
    "# and then reset the indices of the dataframe so it corresponds to row number\n",
    "df_train.replace(to_replace=' ?',value=np.nan, inplace=True)\n",
    "df_train.dropna(inplace=True)\n",
    "df_train.reset_index()\n",
    "\n",
    "df_test.replace(to_replace=' ?',value=np.nan, inplace=True)\n",
    "df_test.dropna(inplace=True)\n",
    "df_test.reset_index()\n",
    "\n",
    "df_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing\n",
    "For preprocessing, we are going to fix a few issues in the dataset. \n",
    "\n",
    "- This first includes the use of \"50K.\" instead of \"50K\" in the test set. \n",
    "- Next, we will encode the categorical features as integers (later on we will encode one hot)\n",
    "- Finally, we will make certain all the continuous data is scaled properly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<=50K    11360\n",
      ">50K      3700\n",
      "Name: income, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>edu_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>...</th>\n",
       "      <th>country</th>\n",
       "      <th>income</th>\n",
       "      <th>workclass_int</th>\n",
       "      <th>education_int</th>\n",
       "      <th>marital_status_int</th>\n",
       "      <th>occupation_int</th>\n",
       "      <th>relationship_int</th>\n",
       "      <th>race_int</th>\n",
       "      <th>sex_int</th>\n",
       "      <th>country_int</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.023104</td>\n",
       "      <td>Private</td>\n",
       "      <td>226802.0</td>\n",
       "      <td>11th</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.033340</td>\n",
       "      <td>Private</td>\n",
       "      <td>89814.0</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Farming-fishing</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.794697</td>\n",
       "      <td>Local-gov</td>\n",
       "      <td>336951.0</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Protective-serv</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.423474</td>\n",
       "      <td>Private</td>\n",
       "      <td>160323.0</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.337883</td>\n",
       "      <td>Private</td>\n",
       "      <td>198693.0</td>\n",
       "      <td>10th</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        age  workclass    fnlwgt     education  edu_num      marital_status  \\\n",
       "1 -1.023104    Private  226802.0          11th      7.0       Never-married   \n",
       "2 -0.033340    Private   89814.0       HS-grad      9.0  Married-civ-spouse   \n",
       "3 -0.794697  Local-gov  336951.0    Assoc-acdm     12.0  Married-civ-spouse   \n",
       "4  0.423474    Private  160323.0  Some-college     10.0  Married-civ-spouse   \n",
       "6 -0.337883    Private  198693.0          10th      6.0       Never-married   \n",
       "\n",
       "          occupation   relationship   race   sex     ...             country  \\\n",
       "1  Machine-op-inspct      Own-child  Black  Male     ...       United-States   \n",
       "2    Farming-fishing        Husband  White  Male     ...       United-States   \n",
       "3    Protective-serv        Husband  White  Male     ...       United-States   \n",
       "4  Machine-op-inspct        Husband  Black  Male     ...       United-States   \n",
       "6      Other-service  Not-in-family  White  Male     ...       United-States   \n",
       "\n",
       "   income  workclass_int education_int  marital_status_int  occupation_int  \\\n",
       "1       0              2             1                   4               6   \n",
       "2       0              2            11                   2               4   \n",
       "3       1              1             7                   2              10   \n",
       "4       1              2            15                   2               6   \n",
       "6       0              2             0                   4               7   \n",
       "\n",
       "   relationship_int  race_int  sex_int  country_int  \n",
       "1                 3         2        1           38  \n",
       "2                 0         4        1           38  \n",
       "3                 0         4        1           38  \n",
       "4                 0         2        1           38  \n",
       "6                 1         4        1           38  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# replace strings\n",
    "if df_test.income.dtype=='object':\n",
    "    df_test.income.replace(to_replace=[' <=50K.',' >50K.'],value=['<=50K','>50K'],inplace=True)\n",
    "    print(df_test.income.value_counts())\n",
    "\n",
    "# define objects that can encode each variable as integer    \n",
    "encoders = dict() \n",
    "categorical_headers = ['workclass','education','marital_status',\n",
    "                       'occupation','relationship','race','sex','country']\n",
    "\n",
    "# train all encoders (special case the target 'income')\n",
    "for col in categorical_headers+['income']:\n",
    "    df_train[col] = df_train[col].str.strip()\n",
    "    df_test[col] = df_test[col].str.strip()\n",
    "    \n",
    "    if col==\"income\":\n",
    "        tmp = LabelEncoder()\n",
    "        df_train[col] = tmp.fit_transform(df_train[col])\n",
    "        df_test[col] = tmp.transform(df_test[col])\n",
    "    else:\n",
    "        # integer encoded variables\n",
    "        encoders[col] = LabelEncoder()\n",
    "        df_train[col+'_int'] = encoders[col].fit_transform(df_train[col])\n",
    "        df_test[col+'_int'] = encoders[col].transform(df_test[col])\n",
    "\n",
    "# scale the numeric, continuous variables\n",
    "numeric_headers = [\"age\", \"cap_gain\", \"cap_loss\",\"work_hrs_weekly\"]\n",
    "\n",
    "for col in numeric_headers:\n",
    "    df_train[col] = df_train[col].astype(np.float)\n",
    "    df_test[col] = df_test[col].astype(np.float)\n",
    "    \n",
    "    ss = StandardScaler()\n",
    "    df_train[col] = ss.fit_transform(df_train[col].values.reshape(-1, 1))\n",
    "    df_test[col] = ss.transform(df_test[col].values.reshape(-1, 1))\n",
    "    \n",
    "df_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['workclass_int', 'education_int', 'marital_status_int', 'occupation_int', 'relationship_int', 'race_int', 'sex_int', 'country_int', 'age', 'cap_gain', 'cap_loss', 'work_hrs_weekly']\n"
     ]
    }
   ],
   "source": [
    "# let's start as simply as possible, without any feature preprocessing\n",
    "categorical_headers_ints = [x+'_int' for x in categorical_headers]\n",
    "\n",
    "# we will forego one-hot encoding right now and instead just scale all inputs\n",
    "feature_columns = categorical_headers_ints+numeric_headers\n",
    "X_train =  ss.fit_transform(df_train[feature_columns].values).astype(np.float32)\n",
    "X_test =  ss.transform(df_test[feature_columns].values).astype(np.float32)\n",
    "\n",
    "y_train = df_train['income'].values.astype(np.int)\n",
    "y_test = df_test['income'].values.astype(np.int)\n",
    "\n",
    "print(feature_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![keras](https://blog.keras.io/img/keras-tensorflow-logo.jpg)\n",
    "\n",
    "\n",
    "# An example similar to Sklearn, Keras\n",
    "- We will start with creating a model that is similar to what we have seen in scikit-learn. \n",
    "- Let's not worry about pre-processing the data right now. Let's just get familiar with Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics as mt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, Input\n",
    "from keras.layers import Embedding, Flatten, Merge, concatenate\n",
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine the features into a single large matrix\n",
    "X_train = df_train[feature_columns].values\n",
    "X_test = df_test[feature_columns].values\n",
    "\n",
    "# This returns a tensor\n",
    "inputs = Input(shape=(X_train.shape[1],))\n",
    "\n",
    "# a layer instance is callable on a tensor, and returns a tensor\n",
    "x = Dense(units=10, activation='relu')(inputs)\n",
    "predictions = Dense(1,activation='sigmoid')(x)\n",
    "\n",
    "# This creates a model that includes\n",
    "# the Input layer and three Dense layers\n",
    "model = Model(inputs=inputs, outputs=predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Adding more customization\n",
    "For this particular optimization, we could go beyond SGD and use a different optimizer. There are many excellent explanations of different optimizers, for instance:\n",
    "- http://sebastianruder.com/optimizing-gradient-descent/\n",
    "\n",
    "Now that we have defined the model, its as simple as using it in a very familiar syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 12)                0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                130       \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 11        \n",
      "=================================================================\n",
      "Total params: 141.0\n",
      "Trainable params: 141.0\n",
      "Non-trainable params: 0.0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[11207   153]\n",
      " [ 2964   736]] 0.793027888446\n",
      "CPU times: user 10.3 s, sys: 3.56 s, total: 13.8 s\n",
      "Wall time: 9.41 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model.fit(X_train, y_train, epochs=10, batch_size=50, verbose=0)\n",
    "\n",
    "from sklearn import metrics as mt\n",
    "yhat = np.round(model.predict(X_test))\n",
    "print(mt.confusion_matrix(y_test,yhat),mt.accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! But we are not handling categorical data properly here... Let's fix that.\n",
    "\n",
    "## One Hot Encoding Categorical Data\n",
    "In order to add one-hot encoding, we need to convert the different columns into sparse arrays. To start, let's only create a model from the one hot encoded variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0  7 23 30 44 50 55 57 98]\n",
      "(30162, 98)\n",
      "<class 'scipy.sparse.csr.csr_matrix'> ========THIS IS SPARSE ENCODED=======\n"
     ]
    }
   ],
   "source": [
    "# but we were dealing with the data incorrectly because we didn't one hot encode the \n",
    "#   categorical features\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "# now let's encode the integer outputs as one hot encoded labels\n",
    "ohe = OneHotEncoder()\n",
    "X_train_ohe = ohe.fit_transform(df_train[categorical_headers_ints].values)\n",
    "X_test_ohe = ohe.transform(df_test[categorical_headers_ints].values)\n",
    "\n",
    "# the ohe instance will help us to organize our encoded matrix\n",
    "print(ohe.feature_indices_)\n",
    "print(X_train_ohe.shape)\n",
    "print(type(X_train_ohe), '========THIS IS SPARSE ENCODED=======')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[10573   787]\n",
      " [ 1869  1831]] 0.82363877822\n"
     ]
    }
   ],
   "source": [
    "# This returns a tensor\n",
    "# the input can only be a scipy.sparse array\n",
    "inputs = Input(shape=(X_train_ohe.shape[1],), sparse=True)\n",
    "\n",
    "# a layer instance is callable on a tensor, and returns a tensor\n",
    "x = Dense(units=10, activation='relu')(inputs)\n",
    "predictions = Dense(1,activation='sigmoid')(x)\n",
    "\n",
    "# This creates a model that includes\n",
    "# the Input layer and three Dense layers\n",
    "model = Model(inputs=inputs, outputs=predictions)\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train_ohe,y_train, epochs=10, batch_size=50, verbose=0)\n",
    "\n",
    "# test on the data\n",
    "yhat = np.round(model.predict(X_test_ohe))\n",
    "print(mt.confusion_matrix(y_test,yhat),mt.accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Modal Networks: Combine Branches, Sparse and Dense\n",
    "The keras functional API allows us to setup different branches of inputs. When we call the `fit` function, we can tell keras the input to each branch using a simple list (you can also use a dictionary to be more explicit).\n",
    "\n",
    "- Lets create two matrics: \n",
    " - one with the one-hot encoded data, `X_train_ohe`\n",
    " - and another with the numeric data, `X_train_num`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________________________________________________________________________________________\n",
      "Layer (type)                     Output Shape          Param #     Connected to                     \n",
      "====================================================================================================\n",
      "X_ohe (InputLayer)               (None, 98)            0                                            \n",
      "____________________________________________________________________________________________________\n",
      "X_Numeric (InputLayer)           (None, 4)             0                                            \n",
      "____________________________________________________________________________________________________\n",
      "ohe_1 (Dense)                    (None, 10)            990                                          \n",
      "____________________________________________________________________________________________________\n",
      "num_1 (Dense)                    (None, 10)            50                                           \n",
      "____________________________________________________________________________________________________\n",
      "concat (Concatenate)             (None, 20)            0                                            \n",
      "____________________________________________________________________________________________________\n",
      "combined (Dense)                 (None, 1)             21                                           \n",
      "====================================================================================================\n",
      "Total params: 1,061.0\n",
      "Trainable params: 1,061.0\n",
      "Non-trainable params: 0.0\n",
      "____________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# combine the features with two branches\n",
    "\n",
    "# let's encode the integer outputs as one hot encoded labels\n",
    "ohe = OneHotEncoder()\n",
    "X_train_ohe = ohe.fit_transform(df_train[categorical_headers_ints].values)\n",
    "X_test_ohe = ohe.transform(df_test[categorical_headers_ints].values)\n",
    "\n",
    "# and save off the numeric features\n",
    "X_train_num =  df_train[numeric_headers].values\n",
    "X_test_num = df_test[numeric_headers].values\n",
    "\n",
    "# create sparse input branch for ohe\n",
    "inputsSparse = Input(shape=(X_train_ohe.shape[1],),sparse=True, name='X_ohe')\n",
    "xSparse = Dense(units=10, activation='relu', name='ohe_1')(inputsSparse)\n",
    "\n",
    "# create dense input branch for numeric\n",
    "inputsDense = Input(shape=(X_train_num.shape[1],),sparse=False, name='X_Numeric')\n",
    "xDense = Dense(units=10, activation='relu',name='num_1')(inputsDense)\n",
    "\n",
    "x = concatenate([xSparse, xDense], name='concat')\n",
    "predictions = Dense(1,activation='sigmoid', name='combined')(x)\n",
    "\n",
    "# This creates a model that includes\n",
    "# the Input layer and Dense layers\n",
    "model = Model(inputs=[inputsSparse,inputsDense], outputs=predictions)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"268pt\" viewBox=\"0.00 0.00 301.00 268.00\" width=\"301pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 264)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-264 297,-264 297,4 -4,4\" stroke=\"none\"/>\n",
       "<!-- 4917915488 -->\n",
       "<g class=\"node\" id=\"node1\"><title>4917915488</title>\n",
       "<polygon fill=\"none\" points=\"-0.5,-223 -0.5,-259 122.5,-259 122.5,-223 -0.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"61\" y=\"-237.3\">X_ohe: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4918041568 -->\n",
       "<g class=\"node\" id=\"node3\"><title>4918041568</title>\n",
       "<polygon fill=\"none\" points=\"24.5,-149 24.5,-185 117.5,-185 117.5,-149 24.5,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"71\" y=\"-163.3\">ohe_1: Dense</text>\n",
       "</g>\n",
       "<!-- 4917915488&#45;&gt;4918041568 -->\n",
       "<g class=\"edge\" id=\"edge1\"><title>4917915488-&gt;4918041568</title>\n",
       "<path d=\"M63.3699,-222.937C64.5113,-214.719 65.9083,-204.66 67.1936,-195.406\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"70.6686,-195.827 68.5777,-185.441 63.7352,-194.864 70.6686,-195.827\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4916807216 -->\n",
       "<g class=\"node\" id=\"node2\"><title>4916807216</title>\n",
       "<polygon fill=\"none\" points=\"141,-223 141,-259 293,-259 293,-223 141,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"217\" y=\"-237.3\">X_Numeric: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4777154880 -->\n",
       "<g class=\"node\" id=\"node4\"><title>4777154880</title>\n",
       "<polygon fill=\"none\" points=\"157,-149 157,-185 255,-185 255,-149 157,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"206\" y=\"-163.3\">num_1: Dense</text>\n",
       "</g>\n",
       "<!-- 4916807216&#45;&gt;4777154880 -->\n",
       "<g class=\"edge\" id=\"edge2\"><title>4916807216-&gt;4777154880</title>\n",
       "<path d=\"M214.393,-222.937C213.138,-214.719 211.601,-204.66 210.187,-195.406\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"213.635,-194.797 208.665,-185.441 206.715,-195.855 213.635,-194.797\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4777154152 -->\n",
       "<g class=\"node\" id=\"node5\"><title>4777154152</title>\n",
       "<polygon fill=\"none\" points=\"73,-75 73,-111 203,-111 203,-75 73,-75\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"138\" y=\"-89.3\">concat: Concatenate</text>\n",
       "</g>\n",
       "<!-- 4918041568&#45;&gt;4777154152 -->\n",
       "<g class=\"edge\" id=\"edge3\"><title>4918041568-&gt;4777154152</title>\n",
       "<path d=\"M86.8781,-148.937C95.1834,-140.012 105.509,-128.916 114.701,-119.037\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"117.52,-121.146 121.77,-111.441 112.396,-116.377 117.52,-121.146\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4777154880&#45;&gt;4777154152 -->\n",
       "<g class=\"edge\" id=\"edge4\"><title>4777154880-&gt;4777154152</title>\n",
       "<path d=\"M189.885,-148.937C181.372,-139.924 170.768,-128.696 161.37,-118.744\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"163.883,-116.308 154.472,-111.441 158.793,-121.114 163.883,-116.308\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4777211832 -->\n",
       "<g class=\"node\" id=\"node6\"><title>4777211832</title>\n",
       "<polygon fill=\"none\" points=\"81,-1 81,-37 195,-37 195,-1 81,-1\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"138\" y=\"-15.3\">combined: Dense</text>\n",
       "</g>\n",
       "<!-- 4777154152&#45;&gt;4777211832 -->\n",
       "<g class=\"edge\" id=\"edge5\"><title>4777154152-&gt;4777211832</title>\n",
       "<path d=\"M138,-74.937C138,-66.8072 138,-56.8761 138,-47.7047\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"141.5,-47.4406 138,-37.4407 134.5,-47.4407 141.5,-47.4406\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "\n",
    "# you will need to install pydot properly on your machine to get this running\n",
    "SVG(model_to_dot(model).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[10615   745]\n",
      " [ 1921  1779]] 0.822974767596\n",
      "CPU times: user 16 s, sys: 4.86 s, total: 20.9 s\n",
      "Wall time: 15.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit([ X_train_ohe, X_train_num ], # inputs for each branch are a list\n",
    "          y_train, \n",
    "          epochs=10, \n",
    "          batch_size=50, \n",
    "          verbose=0)\n",
    "\n",
    "yhat = model.predict([X_test_ohe,\n",
    "                      X_test_num]) # each branch has an input\n",
    "\n",
    "yhat = np.round(yhat)\n",
    "print(mt.confusion_matrix(y_test,yhat),mt.accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the confusion matrix is doing pretty well! But we still are just using an MLP with one hidden layer. We really want to take advantage of the embeddings and crossed columns that are possible with tensorflow. \n",
    "\n",
    "\n",
    "** [Back to Slides] **\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Crossed Columns and Deep Embeddings\n",
    "Lets create the network below in steps. This network consists of a number of different paths for the input data to take.\n",
    "![asdfasfd](https://www.tensorflow.org/images/wide_n_deep.svg)\n",
    "\n",
    "## Step One: Using Dense embeddings \n",
    "First let's try a deeper architecture with dense embeddings for the categorical features, as described in lecture.\n",
    "\n",
    "- The first decision to make is about the size of the dense feature embeddings. This is essentially a dimensionality reduction step. \n",
    " - This can be difficult to set, but one common setting is $log_2(N)$ or $\\sqrt{N}$ where $N$ is the total number of uniques values.\n",
    "- When using an Embedding, we can leave the variables represented as integers. That is, keras will use the integer representation to figure out how to one-hot-encode the variable. This is great because it means we don't need to change the variable in memory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# we need to create separate sequential models for each embedding\n",
    "embed_branches = []\n",
    "X_ints_train = []\n",
    "X_ints_test = []\n",
    "all_inputs = []\n",
    "all_branch_outputs = []\n",
    "\n",
    "for col in categorical_headers_ints:\n",
    "    # encode as ints for the embedding\n",
    "    X_ints_train.append( df_train[col].values )\n",
    "    X_ints_test.append( df_test[col].values )\n",
    "    \n",
    "    # get the number of categories\n",
    "    N = max(X_ints_train[-1]+1) # same as the max(df_train[col])\n",
    "    \n",
    "    # create embedding branch from the number of categories\n",
    "    inputs = Input(shape=(1,),dtype='int32', name=col)\n",
    "    all_inputs.append( inputs )\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=int(np.sqrt(N)), \n",
    "                  input_length=1, name=col+'_embed')(inputs)\n",
    "    x = Flatten()(x)\n",
    "    all_branch_outputs.append(x)\n",
    "\n",
    "# also get a dense branch of the numeric features\n",
    "all_inputs.append(Input(shape=(X_train_num.shape[1],),sparse=False, name='numeric'))\n",
    "x = Dense(units=20, activation='relu',name='numeric_1')(all_inputs[-1])\n",
    "all_branch_outputs.append( Dense(units=10,activation='relu', name='numeric_2')(x) )\n",
    "\n",
    "# merge the branches together\n",
    "final_branch = concatenate(all_branch_outputs, name='concat_1')\n",
    "final_branch = Dense(units=1,activation='sigmoid', name='combined')(final_branch)\n",
    "\n",
    "model = Model(inputs=all_inputs, outputs=final_branch)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"342pt\" viewBox=\"0.00 0.00 1917.00 342.00\" width=\"1917pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 338)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-338 1913,-338 1913,4 -4,4\" stroke=\"none\"/>\n",
       "<!-- 4772467152 -->\n",
       "<g class=\"node\" id=\"node1\"><title>4772467152</title>\n",
       "<polygon fill=\"none\" points=\"23,-297 23,-333 187,-333 187,-297 23,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"105\" y=\"-311.3\">workclass_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4918150704 -->\n",
       "<g class=\"node\" id=\"node10\"><title>4918150704</title>\n",
       "<polygon fill=\"none\" points=\"-0.5,-223 -0.5,-259 210.5,-259 210.5,-223 -0.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"105\" y=\"-237.3\">workclass_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4772467152&#45;&gt;4918150704 -->\n",
       "<g class=\"edge\" id=\"edge1\"><title>4772467152-&gt;4918150704</title>\n",
       "<path d=\"M105,-296.937C105,-288.807 105,-278.876 105,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"108.5,-269.441 105,-259.441 101.5,-269.441 108.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772457048 -->\n",
       "<g class=\"node\" id=\"node2\"><title>4772457048</title>\n",
       "<polygon fill=\"none\" points=\"252,-297 252,-333 414,-333 414,-297 252,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"333\" y=\"-311.3\">education_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4772470456 -->\n",
       "<g class=\"node\" id=\"node11\"><title>4772470456</title>\n",
       "<polygon fill=\"none\" points=\"228.5,-223 228.5,-259 437.5,-259 437.5,-223 228.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"333\" y=\"-237.3\">education_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4772457048&#45;&gt;4772470456 -->\n",
       "<g class=\"edge\" id=\"edge2\"><title>4772457048-&gt;4772470456</title>\n",
       "<path d=\"M333,-296.937C333,-288.807 333,-278.876 333,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"336.5,-269.441 333,-259.441 329.5,-269.441 336.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772556472 -->\n",
       "<g class=\"node\" id=\"node3\"><title>4772556472</title>\n",
       "<polygon fill=\"none\" points=\"479,-297 479,-333 665,-333 665,-297 479,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"572\" y=\"-311.3\">marital_status_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4772603552 -->\n",
       "<g class=\"node\" id=\"node12\"><title>4772603552</title>\n",
       "<polygon fill=\"none\" points=\"455.5,-223 455.5,-259 688.5,-259 688.5,-223 455.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"572\" y=\"-237.3\">marital_status_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4772556472&#45;&gt;4772603552 -->\n",
       "<g class=\"edge\" id=\"edge3\"><title>4772556472-&gt;4772603552</title>\n",
       "<path d=\"M572,-296.937C572,-288.807 572,-278.876 572,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"575.5,-269.441 572,-259.441 568.5,-269.441 575.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4773347000 -->\n",
       "<g class=\"node\" id=\"node4\"><title>4773347000</title>\n",
       "<polygon fill=\"none\" points=\"730.5,-297 730.5,-333 899.5,-333 899.5,-297 730.5,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"815\" y=\"-311.3\">occupation_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4773305928 -->\n",
       "<g class=\"node\" id=\"node13\"><title>4773305928</title>\n",
       "<polygon fill=\"none\" points=\"707,-223 707,-259 923,-259 923,-223 707,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"815\" y=\"-237.3\">occupation_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4773347000&#45;&gt;4773305928 -->\n",
       "<g class=\"edge\" id=\"edge4\"><title>4773347000-&gt;4773305928</title>\n",
       "<path d=\"M815,-296.937C815,-288.807 815,-278.876 815,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"818.5,-269.441 815,-259.441 811.5,-269.441 818.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772739056 -->\n",
       "<g class=\"node\" id=\"node5\"><title>4772739056</title>\n",
       "<polygon fill=\"none\" points=\"965,-297 965,-333 1139,-333 1139,-297 965,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1052\" y=\"-311.3\">relationship_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4772695456 -->\n",
       "<g class=\"node\" id=\"node14\"><title>4772695456</title>\n",
       "<polygon fill=\"none\" points=\"941.5,-223 941.5,-259 1162.5,-259 1162.5,-223 941.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1052\" y=\"-237.3\">relationship_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4772739056&#45;&gt;4772695456 -->\n",
       "<g class=\"edge\" id=\"edge5\"><title>4772739056-&gt;4772695456</title>\n",
       "<path d=\"M1052,-296.937C1052,-288.807 1052,-278.876 1052,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1055.5,-269.441 1052,-259.441 1048.5,-269.441 1055.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772866032 -->\n",
       "<g class=\"node\" id=\"node6\"><title>4772866032</title>\n",
       "<polygon fill=\"none\" points=\"1204.5,-297 1204.5,-333 1335.5,-333 1335.5,-297 1204.5,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1270\" y=\"-311.3\">race_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4772994520 -->\n",
       "<g class=\"node\" id=\"node15\"><title>4772994520</title>\n",
       "<polygon fill=\"none\" points=\"1181,-223 1181,-259 1359,-259 1359,-223 1181,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1270\" y=\"-237.3\">race_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4772866032&#45;&gt;4772994520 -->\n",
       "<g class=\"edge\" id=\"edge6\"><title>4772866032-&gt;4772994520</title>\n",
       "<path d=\"M1270,-296.937C1270,-288.807 1270,-278.876 1270,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1273.5,-269.441 1270,-259.441 1266.5,-269.441 1273.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4773030544 -->\n",
       "<g class=\"node\" id=\"node7\"><title>4773030544</title>\n",
       "<polygon fill=\"none\" points=\"1400.5,-297 1400.5,-333 1527.5,-333 1527.5,-297 1400.5,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1464\" y=\"-311.3\">sex_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4773071448 -->\n",
       "<g class=\"node\" id=\"node16\"><title>4773071448</title>\n",
       "<polygon fill=\"none\" points=\"1377.5,-223 1377.5,-259 1550.5,-259 1550.5,-223 1377.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1464\" y=\"-237.3\">sex_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4773030544&#45;&gt;4773071448 -->\n",
       "<g class=\"edge\" id=\"edge7\"><title>4773030544-&gt;4773071448</title>\n",
       "<path d=\"M1464,-296.937C1464,-288.807 1464,-278.876 1464,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1467.5,-269.441 1464,-259.441 1460.5,-269.441 1467.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779305392 -->\n",
       "<g class=\"node\" id=\"node8\"><title>4779305392</title>\n",
       "<polygon fill=\"none\" points=\"1591.5,-297 1591.5,-333 1742.5,-333 1742.5,-297 1591.5,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1667\" y=\"-311.3\">country_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4779347360 -->\n",
       "<g class=\"node\" id=\"node17\"><title>4779347360</title>\n",
       "<polygon fill=\"none\" points=\"1568.5,-223 1568.5,-259 1765.5,-259 1765.5,-223 1568.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1667\" y=\"-237.3\">country_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4779305392&#45;&gt;4779347360 -->\n",
       "<g class=\"edge\" id=\"edge8\"><title>4779305392-&gt;4779347360</title>\n",
       "<path d=\"M1667,-296.937C1667,-288.807 1667,-278.876 1667,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1670.5,-269.441 1667,-259.441 1663.5,-269.441 1670.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772466760 -->\n",
       "<g class=\"node\" id=\"node9\"><title>4772466760</title>\n",
       "<polygon fill=\"none\" points=\"1777,-297 1777,-333 1909,-333 1909,-297 1777,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1843\" y=\"-311.3\">numeric: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4772466872 -->\n",
       "<g class=\"node\" id=\"node18\"><title>4772466872</title>\n",
       "<polygon fill=\"none\" points=\"1783.5,-223 1783.5,-259 1902.5,-259 1902.5,-223 1783.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1843\" y=\"-237.3\">numeric_1: Dense</text>\n",
       "</g>\n",
       "<!-- 4772466760&#45;&gt;4772466872 -->\n",
       "<g class=\"edge\" id=\"edge9\"><title>4772466760-&gt;4772466872</title>\n",
       "<path d=\"M1843,-296.937C1843,-288.807 1843,-278.876 1843,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1846.5,-269.441 1843,-259.441 1839.5,-269.441 1846.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772467824 -->\n",
       "<g class=\"node\" id=\"node19\"><title>4772467824</title>\n",
       "<polygon fill=\"none\" points=\"76,-149 76,-185 188,-185 188,-149 76,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"132\" y=\"-163.3\">flatten_1: Flatten</text>\n",
       "</g>\n",
       "<!-- 4918150704&#45;&gt;4772467824 -->\n",
       "<g class=\"edge\" id=\"edge10\"><title>4918150704-&gt;4772467824</title>\n",
       "<path d=\"M111.399,-222.937C114.547,-214.542 118.415,-204.227 121.946,-194.81\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"125.226,-196.033 125.46,-185.441 118.671,-193.575 125.226,-196.033\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772467936 -->\n",
       "<g class=\"node\" id=\"node20\"><title>4772467936</title>\n",
       "<polygon fill=\"none\" points=\"419,-149 419,-185 531,-185 531,-149 419,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"475\" y=\"-163.3\">flatten_2: Flatten</text>\n",
       "</g>\n",
       "<!-- 4772470456&#45;&gt;4772467936 -->\n",
       "<g class=\"edge\" id=\"edge11\"><title>4772470456-&gt;4772467936</title>\n",
       "<path d=\"M366.652,-222.937C386.227,-213.012 411.097,-200.401 432.072,-189.766\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"433.761,-192.834 441.097,-185.19 430.596,-186.591 433.761,-192.834\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4773206280 -->\n",
       "<g class=\"node\" id=\"node21\"><title>4773206280</title>\n",
       "<polygon fill=\"none\" points=\"656,-149 656,-185 768,-185 768,-149 656,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"712\" y=\"-163.3\">flatten_3: Flatten</text>\n",
       "</g>\n",
       "<!-- 4772603552&#45;&gt;4773206280 -->\n",
       "<g class=\"edge\" id=\"edge12\"><title>4772603552-&gt;4773206280</title>\n",
       "<path d=\"M605.178,-222.937C624.477,-213.012 648.997,-200.401 669.676,-189.766\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"671.283,-192.876 678.575,-185.19 668.081,-186.651 671.283,-192.876\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4773399912 -->\n",
       "<g class=\"node\" id=\"node22\"><title>4773399912</title>\n",
       "<polygon fill=\"none\" points=\"839,-149 839,-185 951,-185 951,-149 839,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"895\" y=\"-163.3\">flatten_4: Flatten</text>\n",
       "</g>\n",
       "<!-- 4773305928&#45;&gt;4773399912 -->\n",
       "<g class=\"edge\" id=\"edge13\"><title>4773305928-&gt;4773399912</title>\n",
       "<path d=\"M833.959,-222.937C844.245,-213.679 857.127,-202.086 868.402,-191.938\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"870.808,-194.481 875.9,-185.19 866.126,-189.278 870.808,-194.481\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772789608 -->\n",
       "<g class=\"node\" id=\"node23\"><title>4772789608</title>\n",
       "<polygon fill=\"none\" points=\"996,-149 996,-185 1108,-185 1108,-149 996,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1052\" y=\"-163.3\">flatten_5: Flatten</text>\n",
       "</g>\n",
       "<!-- 4772695456&#45;&gt;4772789608 -->\n",
       "<g class=\"edge\" id=\"edge14\"><title>4772695456-&gt;4772789608</title>\n",
       "<path d=\"M1052,-222.937C1052,-214.807 1052,-204.876 1052,-195.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1055.5,-195.441 1052,-185.441 1048.5,-195.441 1055.5,-195.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772994128 -->\n",
       "<g class=\"node\" id=\"node24\"><title>4772994128</title>\n",
       "<polygon fill=\"none\" points=\"1170,-149 1170,-185 1282,-185 1282,-149 1170,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1226\" y=\"-163.3\">flatten_6: Flatten</text>\n",
       "</g>\n",
       "<!-- 4772994520&#45;&gt;4772994128 -->\n",
       "<g class=\"edge\" id=\"edge15\"><title>4772994520-&gt;4772994128</title>\n",
       "<path d=\"M1259.57,-222.937C1254.33,-214.365 1247.87,-203.791 1242.02,-194.217\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1244.86,-192.148 1236.66,-185.441 1238.89,-195.799 1244.86,-192.148\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4773161224 -->\n",
       "<g class=\"node\" id=\"node25\"><title>4773161224</title>\n",
       "<polygon fill=\"none\" points=\"1354,-149 1354,-185 1466,-185 1466,-149 1354,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1410\" y=\"-163.3\">flatten_7: Flatten</text>\n",
       "</g>\n",
       "<!-- 4773071448&#45;&gt;4773161224 -->\n",
       "<g class=\"edge\" id=\"edge16\"><title>4773071448-&gt;4773161224</title>\n",
       "<path d=\"M1451.2,-222.937C1444.64,-214.189 1436.52,-203.354 1429.22,-193.626\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1431.88,-191.341 1423.08,-185.441 1426.28,-195.541 1431.88,-191.341\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779474224 -->\n",
       "<g class=\"node\" id=\"node26\"><title>4779474224</title>\n",
       "<polygon fill=\"none\" points=\"1547,-149 1547,-185 1659,-185 1659,-149 1547,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1603\" y=\"-163.3\">flatten_8: Flatten</text>\n",
       "</g>\n",
       "<!-- 4779347360&#45;&gt;4779474224 -->\n",
       "<g class=\"edge\" id=\"edge17\"><title>4779347360-&gt;4779474224</title>\n",
       "<path d=\"M1651.83,-222.937C1643.9,-214.012 1634.04,-202.916 1625.26,-193.037\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1627.76,-190.59 1618.5,-185.441 1622.53,-195.24 1627.76,-190.59\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779042856 -->\n",
       "<g class=\"node\" id=\"node27\"><title>4779042856</title>\n",
       "<polygon fill=\"none\" points=\"1730.5,-149 1730.5,-185 1849.5,-185 1849.5,-149 1730.5,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1790\" y=\"-163.3\">numeric_2: Dense</text>\n",
       "</g>\n",
       "<!-- 4772466872&#45;&gt;4779042856 -->\n",
       "<g class=\"edge\" id=\"edge18\"><title>4772466872-&gt;4779042856</title>\n",
       "<path d=\"M1830.44,-222.937C1824,-214.189 1816.02,-203.354 1808.86,-193.626\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1811.59,-191.419 1802.84,-185.441 1805.95,-195.569 1811.59,-191.419\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4778995384 -->\n",
       "<g class=\"node\" id=\"node28\"><title>4778995384</title>\n",
       "<polygon fill=\"none\" points=\"980,-75 980,-111 1124,-111 1124,-75 980,-75\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1052\" y=\"-89.3\">concat_1: Concatenate</text>\n",
       "</g>\n",
       "<!-- 4772467824&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge19\"><title>4772467824-&gt;4778995384</title>\n",
       "<path d=\"M188.165,-161.604C344.576,-149.364 785.704,-114.841 969.75,-100.437\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"970.213,-103.912 979.91,-99.6419 969.667,-96.9328 970.213,-103.912\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772467936&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge20\"><title>4772467936-&gt;4778995384</title>\n",
       "<path d=\"M531.112,-158.998C633.341,-146.242 850.341,-119.164 969.461,-104.3\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"970.132,-107.743 979.622,-103.032 969.265,-100.797 970.132,-107.743\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4773206280&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge21\"><title>4773206280-&gt;4778995384</title>\n",
       "<path d=\"M768.144,-154.111C823.108,-142.471 907.458,-124.609 969.939,-111.378\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"970.713,-114.791 979.771,-109.295 969.263,-107.943 970.713,-114.791\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4773399912&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge22\"><title>4773399912-&gt;4778995384</title>\n",
       "<path d=\"M932.207,-148.937C954.124,-138.886 982.044,-126.082 1005.42,-115.364\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1007.16,-118.415 1014.79,-111.065 1004.24,-112.052 1007.16,-118.415\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772789608&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge23\"><title>4772789608-&gt;4778995384</title>\n",
       "<path d=\"M1052,-148.937C1052,-140.807 1052,-130.876 1052,-121.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1055.5,-121.441 1052,-111.441 1048.5,-121.441 1055.5,-121.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4772994128&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge24\"><title>4772994128-&gt;4778995384</title>\n",
       "<path d=\"M1184.76,-148.937C1160.15,-138.752 1128.7,-125.74 1102.6,-114.936\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1103.82,-111.655 1093.24,-111.065 1101.14,-118.123 1103.82,-111.655\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4773161224&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge25\"><title>4773161224-&gt;4778995384</title>\n",
       "<path d=\"M1353.67,-154.671C1294.67,-142.805 1201.42,-124.05 1134.2,-110.532\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1134.56,-107.034 1124.07,-108.494 1133.18,-113.897 1134.56,-107.034\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779474224&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge26\"><title>4779474224-&gt;4778995384</title>\n",
       "<path d=\"M1546.95,-157.96C1524.57,-154.792 1498.58,-151.162 1475,-148 1356.72,-132.134 1219.68,-114.838 1134.4,-104.204\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1134.62,-100.704 1124.26,-102.941 1133.75,-107.651 1134.62,-100.704\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779042856&#45;&gt;4778995384 -->\n",
       "<g class=\"edge\" id=\"edge27\"><title>4779042856-&gt;4778995384</title>\n",
       "<path d=\"M1730.21,-156.46C1710.44,-153.497 1688.31,-150.386 1668,-148 1477.37,-125.602 1252.55,-108.15 1134.28,-99.661\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1134.44,-96.1634 1124.22,-98.9416 1133.94,-103.146 1134.44,-96.1634\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779123712 -->\n",
       "<g class=\"node\" id=\"node29\"><title>4779123712</title>\n",
       "<polygon fill=\"none\" points=\"995,-1 995,-37 1109,-37 1109,-1 995,-1\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1052\" y=\"-15.3\">combined: Dense</text>\n",
       "</g>\n",
       "<!-- 4778995384&#45;&gt;4779123712 -->\n",
       "<g class=\"edge\" id=\"edge28\"><title>4778995384-&gt;4779123712</title>\n",
       "<path d=\"M1052,-74.937C1052,-66.8072 1052,-56.8761 1052,-47.7047\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1055.5,-47.4406 1052,-37.4407 1048.5,-47.4407 1055.5,-47.4406\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVG(model_to_dot(model).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1959 - acc: 0.7274     \n",
      "Epoch 2/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1589 - acc: 0.7576     \n",
      "Epoch 3/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1447 - acc: 0.7750     \n",
      "Epoch 4/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1354 - acc: 0.7979     \n",
      "Epoch 5/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1290 - acc: 0.8137     \n",
      "Epoch 6/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1244 - acc: 0.8214     \n",
      "Epoch 7/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1209 - acc: 0.8273     \n",
      "Epoch 8/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1183 - acc: 0.8323     \n",
      "Epoch 9/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1162 - acc: 0.8357     \n",
      "Epoch 10/10\n",
      "30162/30162 [==============================] - 4s - loss: 0.1145 - acc: 0.8386     \n",
      "CPU times: user 39.8 s, sys: 22.8 s, total: 1min 2s\n",
      "Wall time: 43.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_ints_train + [X_train_num], # create a list of inputs for embeddings\n",
    "        y_train, epochs=10, batch_size=32, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[10767   593]\n",
      " [ 1821  1879]] 0.839707835325\n"
     ]
    }
   ],
   "source": [
    "yhat = np.round(model.predict(X_ints_test + [X_test_num]))\n",
    "print(mt.confusion_matrix(y_test,yhat),mt.accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That works fairly well using the embeddings. Now let's try to perform the same thing with crossed embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "___\n",
    "\n",
    "\n",
    "## Step Two: Making Crossed Columns\n",
    "For this example, we are going to make embeddings for crossed columns. We will perform this step using `sklearn` \n",
    "\n",
    "Let's start simple with \n",
    "- just a single layer classifier \n",
    "- with branches of crossed categorical features as input \n",
    " - one branch for each crossed column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'workclass','education','marital_status',\n",
    "# 'occupation','relationship','race',\n",
    "# 'sex','country'\n",
    "\n",
    "cross_columns = [['education','occupation'],\n",
    "                 ['sex', 'marital_status'],\n",
    "                 ['workclass','occupation']]\n",
    "\n",
    "# we will create separate sequential models for each embedding\n",
    "embed_branches = []\n",
    "X_ints_train = []\n",
    "X_ints_test = []\n",
    "all_inputs = []\n",
    "all_branch_outputs = []\n",
    "\n",
    "for cols in cross_columns:\n",
    "    # encode as ints for the embedding\n",
    "    enc = LabelEncoder()\n",
    "    \n",
    "    # 1. create crossed labels by join operation\n",
    "    X_crossed_train = df_train[cols].apply(lambda x: '_'.join(x), axis=1)\n",
    "    X_crossed_test = df_test[cols].apply(lambda x: '_'.join(x), axis=1)\n",
    "    \n",
    "    # 2. encode as integers\n",
    "    enc.fit(np.hstack((X_crossed_train.values,  X_crossed_test.values)))\n",
    "    X_crossed_train = enc.transform(X_crossed_train)\n",
    "    X_crossed_test = enc.transform(X_crossed_test)\n",
    "    \n",
    "    # 3. save encodings in a list for each branch\n",
    "    X_ints_train.append( X_crossed_train )\n",
    "    X_ints_test.append( X_crossed_test )\n",
    "    \n",
    "    # get the number of categories in the crossing\n",
    "    N = max(X_ints_train[-1]+1) # same as the max(df_train[col])\n",
    "    \n",
    "    # create embedding branch from the crossed encoding\n",
    "    inputs = Input(shape=(1,),dtype='int32',name='_'.join(cols))\n",
    "    all_inputs.append(inputs)\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=int(np.sqrt(N)), \n",
    "                  input_length=1, name='_'.join(cols)+'_embed')(inputs)\n",
    "    x = Flatten()(x)\n",
    "    all_branch_outputs.append(x)\n",
    "    \n",
    "# merge the branches together\n",
    "final_branch = concatenate(all_branch_outputs)\n",
    "# add a layer after merging\n",
    "final_branch = Dense(units=1,activation='sigmoid',name='combined')(final_branch)\n",
    "\n",
    "model = Model(inputs=all_inputs, outputs=final_branch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"342pt\" viewBox=\"0.00 0.00 794.00 342.00\" width=\"794pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 338)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-338 790,-338 790,4 -4,4\" stroke=\"none\"/>\n",
       "<!-- 4780792408 -->\n",
       "<g class=\"node\" id=\"node1\"><title>4780792408</title>\n",
       "<polygon fill=\"none\" points=\"23.5,-297 23.5,-333 232.5,-333 232.5,-297 23.5,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"128\" y=\"-311.3\">education_occupation: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4919566288 -->\n",
       "<g class=\"node\" id=\"node4\"><title>4919566288</title>\n",
       "<polygon fill=\"none\" points=\"0,-223 0,-259 256,-259 256,-223 0,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"128\" y=\"-237.3\">education_occupation_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4780792408&#45;&gt;4919566288 -->\n",
       "<g class=\"edge\" id=\"edge1\"><title>4780792408-&gt;4919566288</title>\n",
       "<path d=\"M128,-296.937C128,-288.807 128,-278.876 128,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"131.5,-269.441 128,-259.441 124.5,-269.441 131.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919565896 -->\n",
       "<g class=\"node\" id=\"node2\"><title>4919565896</title>\n",
       "<polygon fill=\"none\" points=\"298,-297 298,-333 488,-333 488,-297 298,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"393\" y=\"-311.3\">sex_marital_status: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4919813344 -->\n",
       "<g class=\"node\" id=\"node5\"><title>4919813344</title>\n",
       "<polygon fill=\"none\" points=\"274.5,-223 274.5,-259 511.5,-259 511.5,-223 274.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"393\" y=\"-237.3\">sex_marital_status_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4919565896&#45;&gt;4919813344 -->\n",
       "<g class=\"edge\" id=\"edge2\"><title>4919565896-&gt;4919813344</title>\n",
       "<path d=\"M393,-296.937C393,-288.807 393,-278.876 393,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"396.5,-269.441 393,-259.441 389.5,-269.441 396.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919566232 -->\n",
       "<g class=\"node\" id=\"node3\"><title>4919566232</title>\n",
       "<polygon fill=\"none\" points=\"553,-297 553,-333 763,-333 763,-297 553,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"658\" y=\"-311.3\">workclass_occupation: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4919782648 -->\n",
       "<g class=\"node\" id=\"node6\"><title>4919782648</title>\n",
       "<polygon fill=\"none\" points=\"529.5,-223 529.5,-259 786.5,-259 786.5,-223 529.5,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"658\" y=\"-237.3\">workclass_occupation_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4919566232&#45;&gt;4919782648 -->\n",
       "<g class=\"edge\" id=\"edge3\"><title>4919566232-&gt;4919782648</title>\n",
       "<path d=\"M658,-296.937C658,-288.807 658,-278.876 658,-269.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"661.5,-269.441 658,-259.441 654.5,-269.441 661.5,-269.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919566120 -->\n",
       "<g class=\"node\" id=\"node7\"><title>4919566120</title>\n",
       "<polygon fill=\"none\" points=\"138,-149 138,-185 250,-185 250,-149 138,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"194\" y=\"-163.3\">flatten_9: Flatten</text>\n",
       "</g>\n",
       "<!-- 4919566288&#45;&gt;4919566120 -->\n",
       "<g class=\"edge\" id=\"edge4\"><title>4919566288-&gt;4919566120</title>\n",
       "<path d=\"M143.641,-222.937C151.822,-214.012 161.994,-202.916 171.049,-193.037\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"173.835,-195.177 178.013,-185.441 168.675,-190.447 173.835,-195.177\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919814688 -->\n",
       "<g class=\"node\" id=\"node8\"><title>4919814688</title>\n",
       "<polygon fill=\"none\" points=\"333.5,-149 333.5,-185 452.5,-185 452.5,-149 333.5,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"393\" y=\"-163.3\">flatten_10: Flatten</text>\n",
       "</g>\n",
       "<!-- 4919813344&#45;&gt;4919814688 -->\n",
       "<g class=\"edge\" id=\"edge5\"><title>4919813344-&gt;4919814688</title>\n",
       "<path d=\"M393,-222.937C393,-214.807 393,-204.876 393,-195.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"396.5,-195.441 393,-185.441 389.5,-195.441 396.5,-195.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919713688 -->\n",
       "<g class=\"node\" id=\"node9\"><title>4919713688</title>\n",
       "<polygon fill=\"none\" points=\"534.5,-149 534.5,-185 653.5,-185 653.5,-149 534.5,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"594\" y=\"-163.3\">flatten_11: Flatten</text>\n",
       "</g>\n",
       "<!-- 4919782648&#45;&gt;4919713688 -->\n",
       "<g class=\"edge\" id=\"edge6\"><title>4919782648-&gt;4919713688</title>\n",
       "<path d=\"M642.833,-222.937C634.899,-214.012 625.036,-202.916 616.255,-193.037\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"618.762,-190.59 609.503,-185.441 613.531,-195.24 618.762,-190.59\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919565784 -->\n",
       "<g class=\"node\" id=\"node10\"><title>4919565784</title>\n",
       "<polygon fill=\"none\" points=\"306.5,-75 306.5,-111 479.5,-111 479.5,-75 306.5,-75\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"393\" y=\"-89.3\">concatenate_1: Concatenate</text>\n",
       "</g>\n",
       "<!-- 4919566120&#45;&gt;4919565784 -->\n",
       "<g class=\"edge\" id=\"edge7\"><title>4919566120-&gt;4919565784</title>\n",
       "<path d=\"M241.16,-148.937C269.681,-138.618 306.222,-125.397 336.31,-114.511\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"337.621,-117.759 345.834,-111.065 335.24,-111.176 337.621,-117.759\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919814688&#45;&gt;4919565784 -->\n",
       "<g class=\"edge\" id=\"edge8\"><title>4919814688-&gt;4919565784</title>\n",
       "<path d=\"M393,-148.937C393,-140.807 393,-130.876 393,-121.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"396.5,-121.441 393,-111.441 389.5,-121.441 396.5,-121.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919713688&#45;&gt;4919565784 -->\n",
       "<g class=\"edge\" id=\"edge9\"><title>4919713688-&gt;4919565784</title>\n",
       "<path d=\"M546.366,-148.937C517.559,-138.618 480.65,-125.397 450.26,-114.511\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"451.235,-111.142 440.64,-111.065 448.874,-117.732 451.235,-111.142\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4919712176 -->\n",
       "<g class=\"node\" id=\"node11\"><title>4919712176</title>\n",
       "<polygon fill=\"none\" points=\"336,-1 336,-37 450,-37 450,-1 336,-1\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"393\" y=\"-15.3\">combined: Dense</text>\n",
       "</g>\n",
       "<!-- 4919565784&#45;&gt;4919712176 -->\n",
       "<g class=\"edge\" id=\"edge10\"><title>4919565784-&gt;4919712176</title>\n",
       "<path d=\"M393,-74.937C393,-66.8072 393,-56.8761 393,-47.7047\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"396.5,-47.4406 393,-37.4407 389.5,-47.4407 396.5,-47.4406\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVG(model_to_dot(model).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.2021 - acc: 0.7483     \n",
      "Epoch 2/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1747 - acc: 0.7511     \n",
      "Epoch 3/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1640 - acc: 0.7511     \n",
      "Epoch 4/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1556 - acc: 0.7547     \n",
      "Epoch 5/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1496 - acc: 0.7725     \n",
      "Epoch 6/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1455 - acc: 0.7893     \n",
      "Epoch 7/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1425 - acc: 0.8009     \n",
      "Epoch 8/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1400 - acc: 0.8032     \n",
      "Epoch 9/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1380 - acc: 0.8056     \n",
      "Epoch 10/10\n",
      "30162/30162 [==============================] - 2s - loss: 0.1363 - acc: 0.8060     \n",
      "CPU times: user 24.9 s, sys: 10.1 s, total: 35 s\n",
      "Wall time: 24.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_ints_train,\n",
    "        y_train, epochs=10, batch_size=32, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[10852   508]\n",
      " [ 2376  1324]] 0.808499335989\n"
     ]
    }
   ],
   "source": [
    "yhat = np.round(model.predict(X_ints_test))\n",
    "print(mt.confusion_matrix(y_test,yhat),mt.accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! That is just using crossed columns and a one layer Linear Classifer! So memorization works fairly well here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "___\n",
    "\n",
    "\n",
    "## Step Three (Final): Combining Crossed Linear Classifier and Deep Embeddings\n",
    "Now its just a matter of setting the wide and deep columns for tensorflow. After which, we can use the combined classifier!\n",
    "\n",
    "Wide and deep models can have really interesting and useful properties so they are great to keep in mind when selecting an architecture. Some of the hyperparameters that are specific to this are:\n",
    "- which features to cross together, typically you only want to cross columns you think are important to be connected--they somehow might create new knowledge by combining.\n",
    "- the size of the dense feature embeddings. This can be difficult to set, but one common setting is $log_2(N)$ or $\\sqrt{N}$ where $N$ is the total number of uniques values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'workclass','education','marital_status',\n",
    "# 'occupation','relationship','race',\n",
    "# 'sex','country'\n",
    "\n",
    "cross_columns = [['education','occupation'],\n",
    "                 ['relationship','marital_status','sex'],\n",
    "                 ['race','sex']]\n",
    "\n",
    "#'workclass','education','marital_status','occupation','relationship','race','sex','country'\n",
    "\n",
    "# we need to create separate lists for each branch\n",
    "embed_branches = []\n",
    "X_ints_train = []\n",
    "X_ints_test = []\n",
    "all_inputs = []\n",
    "all_wide_branch_outputs = []\n",
    "\n",
    "for cols in cross_columns:\n",
    "    # encode crossed columns as ints for the embedding\n",
    "    enc = LabelEncoder()\n",
    "    \n",
    "    # create crossed labels\n",
    "    X_crossed_train = df_train[cols].apply(lambda x: '_'.join(x), axis=1)\n",
    "    X_crossed_test = df_test[cols].apply(lambda x: '_'.join(x), axis=1)\n",
    "    \n",
    "    enc.fit(np.hstack((X_crossed_train.values,  X_crossed_test.values)))\n",
    "    X_crossed_train = enc.transform(X_crossed_train)\n",
    "    X_crossed_test = enc.transform(X_crossed_test)\n",
    "    X_ints_train.append( X_crossed_train )\n",
    "    X_ints_test.append( X_crossed_test )\n",
    "    \n",
    "    # get the number of categories\n",
    "    N = max(X_ints_train[-1]+1) # same as the max(df_train[col])\n",
    "    \n",
    "    # create embedding branch from the number of categories\n",
    "    inputs = Input(shape=(1,),dtype='int32', name = '_'.join(cols))\n",
    "    all_inputs.append(inputs)\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=int(np.sqrt(N)), \n",
    "                  input_length=1, name = '_'.join(cols)+'_embed')(inputs)\n",
    "    x = Flatten()(x)\n",
    "    all_wide_branch_outputs.append(x)\n",
    "    \n",
    "# merge the branches together\n",
    "wide_branch = concatenate(all_wide_branch_outputs, name='wide_concat')\n",
    "wide_branch = Dense(units=1,activation='sigmoid',name='wide_combined')(wide_branch)\n",
    "\n",
    "# reset this input branch\n",
    "all_deep_branch_outputs = []\n",
    "# add in the embeddings\n",
    "for col in categorical_headers_ints:\n",
    "    # encode as ints for the embedding\n",
    "    X_ints_train.append( df_train[col].values )\n",
    "    X_ints_test.append( df_test[col].values )\n",
    "    \n",
    "    # get the number of categories\n",
    "    N = max(X_ints_train[-1]+1) # same as the max(df_train[col])\n",
    "    \n",
    "    # create embedding branch from the number of categories\n",
    "    inputs = Input(shape=(1,),dtype='int32', name=col)\n",
    "    all_inputs.append(inputs)\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=int(np.sqrt(N)), \n",
    "                  input_length=1, name=col+'_embed')(inputs)\n",
    "    x = Flatten()(x)\n",
    "    all_deep_branch_outputs.append(x)\n",
    "    \n",
    "# also get a dense branch of the numeric features\n",
    "all_inputs.append(Input(shape=(X_train_num.shape[1],),\n",
    "                        sparse=False,\n",
    "                        name='numeric_data'))\n",
    "\n",
    "x = Dense(units=20, activation='relu',name='numeric_1')(all_inputs[-1])\n",
    "all_deep_branch_outputs.append( x )\n",
    "\n",
    "# merge the deep branches together\n",
    "deep_branch = concatenate(all_deep_branch_outputs,name='concat_embeds')\n",
    "deep_branch = Dense(units=50,activation='relu', name='deep1')(deep_branch)\n",
    "deep_branch = Dense(units=25,activation='relu', name='deep2')(deep_branch)\n",
    "deep_branch = Dense(units=10,activation='relu', name='deep3')(deep_branch)\n",
    "    \n",
    "final_branch = concatenate([wide_branch, deep_branch],name='concat_deep_wide')\n",
    "final_branch = Dense(units=1,activation='sigmoid',name='combined')(final_branch)\n",
    "\n",
    "model = Model(inputs=all_inputs, outputs=final_branch)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"638pt\" viewBox=\"0.00 0.00 2356.00 638.00\" width=\"2356pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 634)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-634 2352,-634 2352,4 -4,4\" stroke=\"none\"/>\n",
       "<!-- 4923502320 -->\n",
       "<g class=\"node\" id=\"node1\"><title>4923502320</title>\n",
       "<polygon fill=\"none\" points=\"1683,-593 1683,-629 1847,-629 1847,-593 1683,-593\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1765\" y=\"-607.3\">workclass_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4921214680 -->\n",
       "<g class=\"node\" id=\"node9\"><title>4921214680</title>\n",
       "<polygon fill=\"none\" points=\"1659.5,-519 1659.5,-555 1870.5,-555 1870.5,-519 1659.5,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1765\" y=\"-533.3\">workclass_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4923502320&#45;&gt;4921214680 -->\n",
       "<g class=\"edge\" id=\"edge1\"><title>4923502320-&gt;4921214680</title>\n",
       "<path d=\"M1765,-592.937C1765,-584.807 1765,-574.876 1765,-565.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1768.5,-565.441 1765,-555.441 1761.5,-565.441 1768.5,-565.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921310568 -->\n",
       "<g class=\"node\" id=\"node2\"><title>4921310568</title>\n",
       "<polygon fill=\"none\" points=\"1912,-593 1912,-629 2074,-629 2074,-593 1912,-593\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1993\" y=\"-607.3\">education_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4921383288 -->\n",
       "<g class=\"node\" id=\"node10\"><title>4921383288</title>\n",
       "<polygon fill=\"none\" points=\"1888.5,-519 1888.5,-555 2097.5,-555 2097.5,-519 1888.5,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1993\" y=\"-533.3\">education_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4921310568&#45;&gt;4921383288 -->\n",
       "<g class=\"edge\" id=\"edge2\"><title>4921310568-&gt;4921383288</title>\n",
       "<path d=\"M1993,-592.937C1993,-584.807 1993,-574.876 1993,-565.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1996.5,-565.441 1993,-555.441 1989.5,-565.441 1996.5,-565.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921372456 -->\n",
       "<g class=\"node\" id=\"node3\"><title>4921372456</title>\n",
       "<polygon fill=\"none\" points=\"2139,-593 2139,-629 2325,-629 2325,-593 2139,-593\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"2232\" y=\"-607.3\">marital_status_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4779784400 -->\n",
       "<g class=\"node\" id=\"node11\"><title>4779784400</title>\n",
       "<polygon fill=\"none\" points=\"2115.5,-519 2115.5,-555 2348.5,-555 2348.5,-519 2115.5,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"2232\" y=\"-533.3\">marital_status_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4921372456&#45;&gt;4779784400 -->\n",
       "<g class=\"edge\" id=\"edge3\"><title>4921372456-&gt;4779784400</title>\n",
       "<path d=\"M2232,-592.937C2232,-584.807 2232,-574.876 2232,-565.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"2235.5,-565.441 2232,-555.441 2228.5,-565.441 2235.5,-565.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779827664 -->\n",
       "<g class=\"node\" id=\"node4\"><title>4779827664</title>\n",
       "<polygon fill=\"none\" points=\"425.5,-593 425.5,-629 594.5,-629 594.5,-593 425.5,-593\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"510\" y=\"-607.3\">occupation_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4926434496 -->\n",
       "<g class=\"node\" id=\"node12\"><title>4926434496</title>\n",
       "<polygon fill=\"none\" points=\"402,-519 402,-555 618,-555 618,-519 402,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"510\" y=\"-533.3\">occupation_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4779827664&#45;&gt;4926434496 -->\n",
       "<g class=\"edge\" id=\"edge4\"><title>4779827664-&gt;4926434496</title>\n",
       "<path d=\"M510,-592.937C510,-584.807 510,-574.876 510,-565.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"513.5,-565.441 510,-555.441 506.5,-565.441 513.5,-565.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4926371488 -->\n",
       "<g class=\"node\" id=\"node5\"><title>4926371488</title>\n",
       "<polygon fill=\"none\" points=\"660,-593 660,-629 834,-629 834,-593 660,-593\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"747\" y=\"-607.3\">relationship_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4926189184 -->\n",
       "<g class=\"node\" id=\"node13\"><title>4926189184</title>\n",
       "<polygon fill=\"none\" points=\"636.5,-519 636.5,-555 857.5,-555 857.5,-519 636.5,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"747\" y=\"-533.3\">relationship_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4926371488&#45;&gt;4926189184 -->\n",
       "<g class=\"edge\" id=\"edge5\"><title>4926371488-&gt;4926189184</title>\n",
       "<path d=\"M747,-592.937C747,-584.807 747,-574.876 747,-565.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"750.5,-565.441 747,-555.441 743.5,-565.441 750.5,-565.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4926267288 -->\n",
       "<g class=\"node\" id=\"node6\"><title>4926267288</title>\n",
       "<polygon fill=\"none\" points=\"899.5,-593 899.5,-629 1030.5,-629 1030.5,-593 899.5,-593\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"965\" y=\"-607.3\">race_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4924456632 -->\n",
       "<g class=\"node\" id=\"node14\"><title>4924456632</title>\n",
       "<polygon fill=\"none\" points=\"876,-519 876,-555 1054,-555 1054,-519 876,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"965\" y=\"-533.3\">race_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4926267288&#45;&gt;4924456632 -->\n",
       "<g class=\"edge\" id=\"edge6\"><title>4926267288-&gt;4924456632</title>\n",
       "<path d=\"M965,-592.937C965,-584.807 965,-574.876 965,-565.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"968.5,-565.441 965,-555.441 961.5,-565.441 968.5,-565.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4924456856 -->\n",
       "<g class=\"node\" id=\"node7\"><title>4924456856</title>\n",
       "<polygon fill=\"none\" points=\"1095.5,-593 1095.5,-629 1222.5,-629 1222.5,-593 1095.5,-593\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1159\" y=\"-607.3\">sex_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4924447880 -->\n",
       "<g class=\"node\" id=\"node15\"><title>4924447880</title>\n",
       "<polygon fill=\"none\" points=\"1072.5,-519 1072.5,-555 1245.5,-555 1245.5,-519 1072.5,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1159\" y=\"-533.3\">sex_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4924456856&#45;&gt;4924447880 -->\n",
       "<g class=\"edge\" id=\"edge7\"><title>4924456856-&gt;4924447880</title>\n",
       "<path d=\"M1159,-592.937C1159,-584.807 1159,-574.876 1159,-565.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1162.5,-565.441 1159,-555.441 1155.5,-565.441 1162.5,-565.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4924332352 -->\n",
       "<g class=\"node\" id=\"node8\"><title>4924332352</title>\n",
       "<polygon fill=\"none\" points=\"1286.5,-593 1286.5,-629 1437.5,-629 1437.5,-593 1286.5,-593\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1362\" y=\"-607.3\">country_int: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4920287072 -->\n",
       "<g class=\"node\" id=\"node16\"><title>4920287072</title>\n",
       "<polygon fill=\"none\" points=\"1263.5,-519 1263.5,-555 1460.5,-555 1460.5,-519 1263.5,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1362\" y=\"-533.3\">country_int_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4924332352&#45;&gt;4920287072 -->\n",
       "<g class=\"edge\" id=\"edge8\"><title>4924332352-&gt;4920287072</title>\n",
       "<path d=\"M1362,-592.937C1362,-584.807 1362,-574.876 1362,-565.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1365.5,-565.441 1362,-555.441 1358.5,-565.441 1365.5,-565.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4923452440 -->\n",
       "<g class=\"node\" id=\"node21\"><title>4923452440</title>\n",
       "<polygon fill=\"none\" points=\"1656.5,-445 1656.5,-481 1775.5,-481 1775.5,-445 1656.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1716\" y=\"-459.3\">flatten_15: Flatten</text>\n",
       "</g>\n",
       "<!-- 4921214680&#45;&gt;4923452440 -->\n",
       "<g class=\"edge\" id=\"edge9\"><title>4921214680-&gt;4923452440</title>\n",
       "<path d=\"M1753.39,-518.937C1747.49,-510.277 1740.21,-499.573 1733.64,-489.921\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1736.39,-487.739 1727.87,-481.441 1730.6,-491.677 1736.39,-487.739\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921343112 -->\n",
       "<g class=\"node\" id=\"node22\"><title>4921343112</title>\n",
       "<polygon fill=\"none\" points=\"1888.5,-445 1888.5,-481 2007.5,-481 2007.5,-445 1888.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1948\" y=\"-459.3\">flatten_16: Flatten</text>\n",
       "</g>\n",
       "<!-- 4921383288&#45;&gt;4921343112 -->\n",
       "<g class=\"edge\" id=\"edge10\"><title>4921383288-&gt;4921343112</title>\n",
       "<path d=\"M1982.34,-518.937C1976.92,-510.277 1970.23,-499.573 1964.2,-489.921\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1967.17,-488.066 1958.9,-481.441 1961.23,-491.776 1967.17,-488.066\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779736928 -->\n",
       "<g class=\"node\" id=\"node23\"><title>4779736928</title>\n",
       "<polygon fill=\"none\" points=\"2076.5,-445 2076.5,-481 2195.5,-481 2195.5,-445 2076.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"2136\" y=\"-459.3\">flatten_17: Flatten</text>\n",
       "</g>\n",
       "<!-- 4779784400&#45;&gt;4779736928 -->\n",
       "<g class=\"edge\" id=\"edge11\"><title>4779784400-&gt;4779736928</title>\n",
       "<path d=\"M2209.25,-518.937C2196.67,-509.501 2180.85,-497.639 2167.14,-487.354\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"2169.02,-484.39 2158.92,-481.19 2164.82,-489.99 2169.02,-484.39\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4926483928 -->\n",
       "<g class=\"node\" id=\"node24\"><title>4926483928</title>\n",
       "<polygon fill=\"none\" points=\"754.5,-445 754.5,-481 873.5,-481 873.5,-445 754.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"814\" y=\"-459.3\">flatten_18: Flatten</text>\n",
       "</g>\n",
       "<!-- 4926434496&#45;&gt;4926483928 -->\n",
       "<g class=\"edge\" id=\"edge12\"><title>4926434496-&gt;4926483928</title>\n",
       "<path d=\"M586.601,-518.994C630.176,-509.18 685.995,-496.374 744.41,-482.113\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"745.308,-485.496 754.188,-479.718 743.642,-478.697 745.308,-485.496\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4926266448 -->\n",
       "<g class=\"node\" id=\"node25\"><title>4926266448</title>\n",
       "<polygon fill=\"none\" points=\"891.5,-445 891.5,-481 1010.5,-481 1010.5,-445 891.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"951\" y=\"-459.3\">flatten_19: Flatten</text>\n",
       "</g>\n",
       "<!-- 4926189184&#45;&gt;4926266448 -->\n",
       "<g class=\"edge\" id=\"edge13\"><title>4926189184-&gt;4926266448</title>\n",
       "<path d=\"M795.345,-518.937C824.582,-508.618 862.042,-495.397 892.886,-484.511\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"894.384,-487.694 902.649,-481.065 892.054,-481.093 894.384,-487.694\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4924454504 -->\n",
       "<g class=\"node\" id=\"node26\"><title>4924454504</title>\n",
       "<polygon fill=\"none\" points=\"1028.5,-445 1028.5,-481 1147.5,-481 1147.5,-445 1028.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1088\" y=\"-459.3\">flatten_20: Flatten</text>\n",
       "</g>\n",
       "<!-- 4924456632&#45;&gt;4924454504 -->\n",
       "<g class=\"edge\" id=\"edge14\"><title>4924456632-&gt;4924454504</title>\n",
       "<path d=\"M994.149,-518.937C1010.73,-509.234 1031.68,-496.966 1049.59,-486.485\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1051.77,-489.262 1058.63,-481.19 1048.24,-483.221 1051.77,-489.262\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4924333864 -->\n",
       "<g class=\"node\" id=\"node27\"><title>4924333864</title>\n",
       "<polygon fill=\"none\" points=\"1165.5,-445 1165.5,-481 1284.5,-481 1284.5,-445 1165.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1225\" y=\"-459.3\">flatten_21: Flatten</text>\n",
       "</g>\n",
       "<!-- 4924447880&#45;&gt;4924333864 -->\n",
       "<g class=\"edge\" id=\"edge15\"><title>4924447880-&gt;4924333864</title>\n",
       "<path d=\"M1174.64,-518.937C1182.82,-510.012 1192.99,-498.916 1202.05,-489.037\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1204.84,-491.177 1209.01,-481.441 1199.68,-486.447 1204.84,-491.177\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4924350304 -->\n",
       "<g class=\"node\" id=\"node28\"><title>4924350304</title>\n",
       "<polygon fill=\"none\" points=\"1302.5,-445 1302.5,-481 1421.5,-481 1421.5,-445 1302.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1362\" y=\"-459.3\">flatten_22: Flatten</text>\n",
       "</g>\n",
       "<!-- 4920287072&#45;&gt;4924350304 -->\n",
       "<g class=\"edge\" id=\"edge16\"><title>4920287072-&gt;4924350304</title>\n",
       "<path d=\"M1362,-518.937C1362,-510.807 1362,-500.876 1362,-491.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1365.5,-491.441 1362,-481.441 1358.5,-491.441 1365.5,-491.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4923652864 -->\n",
       "<g class=\"node\" id=\"node17\"><title>4923652864</title>\n",
       "<polygon fill=\"none\" points=\"1479,-519 1479,-555 1641,-555 1641,-519 1479,-519\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1560\" y=\"-533.3\">numeric_data: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4923654096 -->\n",
       "<g class=\"node\" id=\"node29\"><title>4923654096</title>\n",
       "<polygon fill=\"none\" points=\"1469.5,-445 1469.5,-481 1588.5,-481 1588.5,-445 1469.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1529\" y=\"-459.3\">numeric_1: Dense</text>\n",
       "</g>\n",
       "<!-- 4923652864&#45;&gt;4923654096 -->\n",
       "<g class=\"edge\" id=\"edge17\"><title>4923652864-&gt;4923654096</title>\n",
       "<path d=\"M1552.65,-518.937C1549.04,-510.542 1544.6,-500.227 1540.54,-490.81\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1543.68,-489.241 1536.51,-481.441 1537.25,-492.01 1543.68,-489.241\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4781097480 -->\n",
       "<g class=\"node\" id=\"node18\"><title>4781097480</title>\n",
       "<polygon fill=\"none\" points=\"351.5,-445 351.5,-481 560.5,-481 560.5,-445 351.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"456\" y=\"-459.3\">education_occupation: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4922072928 -->\n",
       "<g class=\"node\" id=\"node30\"><title>4922072928</title>\n",
       "<polygon fill=\"none\" points=\"328,-371 328,-407 584,-407 584,-371 328,-371\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"456\" y=\"-385.3\">education_occupation_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4781097480&#45;&gt;4922072928 -->\n",
       "<g class=\"edge\" id=\"edge18\"><title>4781097480-&gt;4922072928</title>\n",
       "<path d=\"M456,-444.937C456,-436.807 456,-426.876 456,-417.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"459.5,-417.441 456,-407.441 452.5,-417.441 459.5,-417.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4922072704 -->\n",
       "<g class=\"node\" id=\"node19\"><title>4922072704</title>\n",
       "<polygon fill=\"none\" points=\"23.5,-445 23.5,-481 286.5,-481 286.5,-445 23.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"155\" y=\"-459.3\">relationship_marital_status_sex: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4712702080 -->\n",
       "<g class=\"node\" id=\"node31\"><title>4712702080</title>\n",
       "<polygon fill=\"none\" points=\"0,-371 0,-407 310,-407 310,-371 0,-371\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"155\" y=\"-385.3\">relationship_marital_status_sex_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4922072704&#45;&gt;4712702080 -->\n",
       "<g class=\"edge\" id=\"edge19\"><title>4922072704-&gt;4712702080</title>\n",
       "<path d=\"M155,-444.937C155,-436.807 155,-426.876 155,-417.705\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"158.5,-417.441 155,-407.441 151.5,-417.441 158.5,-417.441\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4922235928 -->\n",
       "<g class=\"node\" id=\"node20\"><title>4922235928</title>\n",
       "<polygon fill=\"none\" points=\"601.5,-445 601.5,-481 736.5,-481 736.5,-445 601.5,-445\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"669\" y=\"-459.3\">race_sex: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4922238288 -->\n",
       "<g class=\"node\" id=\"node32\"><title>4922238288</title>\n",
       "<polygon fill=\"none\" points=\"602,-371 602,-407 784,-407 784,-371 602,-371\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"693\" y=\"-385.3\">race_sex_embed: Embedding</text>\n",
       "</g>\n",
       "<!-- 4922235928&#45;&gt;4922238288 -->\n",
       "<g class=\"edge\" id=\"edge20\"><title>4922235928-&gt;4922238288</title>\n",
       "<path d=\"M674.688,-444.937C677.457,-436.63 680.852,-426.444 683.964,-417.108\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"687.344,-418.034 687.186,-407.441 680.704,-415.821 687.344,-418.034\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921020088 -->\n",
       "<g class=\"node\" id=\"node33\"><title>4921020088</title>\n",
       "<polygon fill=\"none\" points=\"1203.5,-371 1203.5,-407 1382.5,-407 1382.5,-371 1203.5,-371\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1293\" y=\"-385.3\">concat_embeds: Concatenate</text>\n",
       "</g>\n",
       "<!-- 4923452440&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge21\"><title>4923452440-&gt;4921020088</title>\n",
       "<path d=\"M1656.29,-451.837C1587.86,-440.19 1474.64,-420.917 1392.75,-406.979\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1393.12,-403.492 1382.68,-405.264 1391.95,-410.393 1393.12,-403.492\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921343112&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge22\"><title>4921343112-&gt;4921020088</title>\n",
       "<path d=\"M1888.43,-455.452C1775.9,-443.082 1531.02,-416.164 1392.92,-400.984\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1393.08,-397.48 1382.76,-399.867 1392.32,-404.438 1393.08,-397.48\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4779736928&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge23\"><title>4779736928-&gt;4921020088</title>\n",
       "<path d=\"M2076.3,-452.151C2057.09,-449.221 2035.68,-446.208 2016,-444 1794.05,-419.101 1532.45,-402.776 1392.78,-395.101\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1392.7,-391.591 1382.52,-394.541 1392.32,-398.581 1392.7,-391.591\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4926483928&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge24\"><title>4926483928-&gt;4921020088</title>\n",
       "<path d=\"M873.604,-445.76C876.435,-445.135 879.244,-444.543 882,-444 987.506,-423.194 1109.87,-408.279 1193.47,-399.466\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1193.89,-402.942 1203.47,-398.422 1193.16,-395.98 1193.89,-402.942\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4926266448&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge25\"><title>4926266448-&gt;4921020088</title>\n",
       "<path d=\"M1010.65,-445.979C1013.47,-445.294 1016.26,-444.631 1019,-444 1076.48,-430.743 1141.21,-417.891 1193.31,-408.053\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1194.04,-411.477 1203.22,-406.189 1192.75,-404.597 1194.04,-411.477\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4924454504&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge26\"><title>4924454504-&gt;4921020088</title>\n",
       "<path d=\"M1136.58,-444.937C1165.96,-434.618 1203.61,-421.397 1234.6,-410.511\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1236.14,-413.681 1244.41,-407.065 1233.82,-407.077 1236.14,-413.681\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4924333864&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge27\"><title>4924333864-&gt;4921020088</title>\n",
       "<path d=\"M1241.12,-444.937C1249.63,-435.924 1260.23,-424.696 1269.63,-414.744\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1272.21,-417.114 1276.53,-407.441 1267.12,-412.308 1272.21,-417.114\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4924350304&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge28\"><title>4924350304-&gt;4921020088</title>\n",
       "<path d=\"M1345.65,-444.937C1337.01,-435.924 1326.25,-424.696 1316.71,-414.744\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1319.16,-412.239 1309.71,-407.441 1314.11,-417.082 1319.16,-412.239\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4923654096&#45;&gt;4921020088 -->\n",
       "<g class=\"edge\" id=\"edge29\"><title>4923654096-&gt;4921020088</title>\n",
       "<path d=\"M1473.07,-444.937C1438.81,-434.484 1394.79,-421.053 1358.84,-410.088\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1359.52,-406.636 1348.94,-407.065 1357.48,-413.331 1359.52,-406.636\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4922072816 -->\n",
       "<g class=\"node\" id=\"node34\"><title>4922072816</title>\n",
       "<polygon fill=\"none\" points=\"421.5,-297 421.5,-333 540.5,-333 540.5,-297 421.5,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"481\" y=\"-311.3\">flatten_12: Flatten</text>\n",
       "</g>\n",
       "<!-- 4922072928&#45;&gt;4922072816 -->\n",
       "<g class=\"edge\" id=\"edge30\"><title>4922072928-&gt;4922072816</title>\n",
       "<path d=\"M461.925,-370.937C464.809,-362.63 468.346,-352.444 471.588,-343.108\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"474.97,-344.035 474.944,-333.441 468.358,-341.739 474.97,-344.035\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4922845840 -->\n",
       "<g class=\"node\" id=\"node35\"><title>4922845840</title>\n",
       "<polygon fill=\"none\" points=\"202.5,-297 202.5,-333 321.5,-333 321.5,-297 202.5,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"262\" y=\"-311.3\">flatten_13: Flatten</text>\n",
       "</g>\n",
       "<!-- 4712702080&#45;&gt;4922845840 -->\n",
       "<g class=\"edge\" id=\"edge31\"><title>4712702080-&gt;4922845840</title>\n",
       "<path d=\"M180.358,-370.937C194.645,-361.323 212.675,-349.19 228.156,-338.774\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"230.111,-341.677 236.454,-333.19 226.203,-335.869 230.111,-341.677\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4923555568 -->\n",
       "<g class=\"node\" id=\"node36\"><title>4923555568</title>\n",
       "<polygon fill=\"none\" points=\"608.5,-297 608.5,-333 727.5,-333 727.5,-297 608.5,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"668\" y=\"-311.3\">flatten_14: Flatten</text>\n",
       "</g>\n",
       "<!-- 4922238288&#45;&gt;4923555568 -->\n",
       "<g class=\"edge\" id=\"edge32\"><title>4922238288-&gt;4923555568</title>\n",
       "<path d=\"M687.075,-370.937C684.191,-362.63 680.654,-352.444 677.412,-343.108\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"680.642,-341.739 674.056,-333.441 674.03,-344.035 680.642,-341.739\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921020144 -->\n",
       "<g class=\"node\" id=\"node37\"><title>4921020144</title>\n",
       "<polygon fill=\"none\" points=\"1009,-297 1009,-333 1101,-333 1101,-297 1009,-297\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1055\" y=\"-311.3\">deep1: Dense</text>\n",
       "</g>\n",
       "<!-- 4921020088&#45;&gt;4921020144 -->\n",
       "<g class=\"edge\" id=\"edge33\"><title>4921020088-&gt;4921020144</title>\n",
       "<path d=\"M1236.6,-370.937C1198.62,-359.447 1148.74,-344.359 1110.87,-332.902\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1111.61,-329.469 1101.02,-329.923 1109.58,-336.169 1111.61,-329.469\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4922072200 -->\n",
       "<g class=\"node\" id=\"node38\"><title>4922072200</title>\n",
       "<polygon fill=\"none\" points=\"492,-223 492,-259 656,-259 656,-223 492,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"574\" y=\"-237.3\">wide_concat: Concatenate</text>\n",
       "</g>\n",
       "<!-- 4922072816&#45;&gt;4922072200 -->\n",
       "<g class=\"edge\" id=\"edge34\"><title>4922072816-&gt;4922072200</title>\n",
       "<path d=\"M503.04,-296.937C515.228,-287.501 530.55,-275.639 543.834,-265.354\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"546.032,-268.079 551.796,-259.19 541.746,-262.544 546.032,-268.079\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4922845840&#45;&gt;4922072200 -->\n",
       "<g class=\"edge\" id=\"edge35\"><title>4922845840-&gt;4922072200</title>\n",
       "<path d=\"M321.788,-300.203C369.594,-289.171 437.179,-273.574 490.356,-261.303\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"491.152,-264.711 500.109,-259.052 489.578,-257.89 491.152,-264.711\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4923555568&#45;&gt;4922072200 -->\n",
       "<g class=\"edge\" id=\"edge36\"><title>4923555568-&gt;4922072200</title>\n",
       "<path d=\"M645.723,-296.937C633.404,-287.501 617.917,-275.639 604.49,-265.354\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"606.51,-262.492 596.443,-259.19 602.253,-268.049 606.51,-262.492\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921100384 -->\n",
       "<g class=\"node\" id=\"node39\"><title>4921100384</title>\n",
       "<polygon fill=\"none\" points=\"888,-223 888,-259 980,-259 980,-223 888,-223\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"934\" y=\"-237.3\">deep2: Dense</text>\n",
       "</g>\n",
       "<!-- 4921020144&#45;&gt;4921100384 -->\n",
       "<g class=\"edge\" id=\"edge37\"><title>4921020144-&gt;4921100384</title>\n",
       "<path d=\"M1026.32,-296.937C1010.02,-287.234 989.4,-274.966 971.787,-264.485\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"973.272,-261.296 962.889,-259.19 969.693,-267.311 973.272,-261.296\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4923554392 -->\n",
       "<g class=\"node\" id=\"node40\"><title>4923554392</title>\n",
       "<polygon fill=\"none\" points=\"611,-149 611,-185 759,-185 759,-149 611,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"685\" y=\"-163.3\">wide_combined: Dense</text>\n",
       "</g>\n",
       "<!-- 4922072200&#45;&gt;4923554392 -->\n",
       "<g class=\"edge\" id=\"edge38\"><title>4922072200-&gt;4923554392</title>\n",
       "<path d=\"M600.305,-222.937C615.127,-213.323 633.831,-201.19 649.891,-190.774\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"652.014,-193.568 658.499,-185.19 648.204,-187.696 652.014,-193.568\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921030976 -->\n",
       "<g class=\"node\" id=\"node41\"><title>4921030976</title>\n",
       "<polygon fill=\"none\" points=\"832,-149 832,-185 924,-185 924,-149 832,-149\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"878\" y=\"-163.3\">deep3: Dense</text>\n",
       "</g>\n",
       "<!-- 4921100384&#45;&gt;4921030976 -->\n",
       "<g class=\"edge\" id=\"edge39\"><title>4921100384-&gt;4921030976</title>\n",
       "<path d=\"M920.729,-222.937C913.924,-214.189 905.498,-203.354 897.931,-193.626\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"900.467,-191.185 891.565,-185.441 894.942,-195.483 900.467,-191.185\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4923081560 -->\n",
       "<g class=\"node\" id=\"node42\"><title>4923081560</title>\n",
       "<polygon fill=\"none\" points=\"682.5,-75 682.5,-111 879.5,-111 879.5,-75 682.5,-75\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"781\" y=\"-89.3\">concat_deep_wide: Concatenate</text>\n",
       "</g>\n",
       "<!-- 4923554392&#45;&gt;4923081560 -->\n",
       "<g class=\"edge\" id=\"edge40\"><title>4923554392-&gt;4923081560</title>\n",
       "<path d=\"M707.751,-148.937C720.332,-139.501 736.148,-127.639 749.861,-117.354\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"752.18,-119.99 758.08,-111.19 747.98,-114.39 752.18,-119.99\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4921030976&#45;&gt;4923081560 -->\n",
       "<g class=\"edge\" id=\"edge41\"><title>4921030976-&gt;4923081560</title>\n",
       "<path d=\"M855.012,-148.937C842.3,-139.501 826.319,-127.639 812.463,-117.354\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"814.275,-114.34 804.159,-111.19 810.102,-119.961 814.275,-114.34\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 4923260768 -->\n",
       "<g class=\"node\" id=\"node43\"><title>4923260768</title>\n",
       "<polygon fill=\"none\" points=\"724,-1 724,-37 838,-37 838,-1 724,-1\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"781\" y=\"-15.3\">combined: Dense</text>\n",
       "</g>\n",
       "<!-- 4923081560&#45;&gt;4923260768 -->\n",
       "<g class=\"edge\" id=\"edge42\"><title>4923081560-&gt;4923260768</title>\n",
       "<path d=\"M781,-74.937C781,-66.8072 781,-56.8761 781,-47.7047\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"784.5,-47.4406 781,-37.4407 777.5,-47.4407 784.5,-47.4406\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "\n",
    "# you will need to install pydot properly on your machine to get this running\n",
    "SVG(model_to_dot(model).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 30162 samples, validate on 15060 samples\n",
      "Epoch 1/10\n",
      "30162/30162 [==============================] - 9s - loss: 0.1090 - acc: 0.8414 - val_loss: 0.1029 - val_acc: 0.8510\n",
      "Epoch 2/10\n",
      "30162/30162 [==============================] - 8s - loss: 0.1019 - acc: 0.8533 - val_loss: 0.1020 - val_acc: 0.8509\n",
      "Epoch 3/10\n",
      "30162/30162 [==============================] - 9s - loss: 0.1008 - acc: 0.8548 - val_loss: 0.1017 - val_acc: 0.8517\n",
      "Epoch 4/10\n",
      "30162/30162 [==============================] - 9s - loss: 0.1003 - acc: 0.8553 - val_loss: 0.1014 - val_acc: 0.8521\n",
      "Epoch 5/10\n",
      "30162/30162 [==============================] - 8s - loss: 0.0998 - acc: 0.8565 - val_loss: 0.1016 - val_acc: 0.8532\n",
      "Epoch 6/10\n",
      "30162/30162 [==============================] - 8s - loss: 0.0996 - acc: 0.8552 - val_loss: 0.1017 - val_acc: 0.8506\n",
      "Epoch 7/10\n",
      "30162/30162 [==============================] - 8s - loss: 0.0994 - acc: 0.8562 - val_loss: 0.1010 - val_acc: 0.8523\n",
      "Epoch 8/10\n",
      "30162/30162 [==============================] - 7s - loss: 0.0991 - acc: 0.8569 - val_loss: 0.1012 - val_acc: 0.8533\n",
      "Epoch 9/10\n",
      "30162/30162 [==============================] - 7s - loss: 0.0989 - acc: 0.8569 - val_loss: 0.1010 - val_acc: 0.8525\n",
      "Epoch 10/10\n",
      "30162/30162 [==============================] - 7s - loss: 0.0987 - acc: 0.8569 - val_loss: 0.1010 - val_acc: 0.8521\n",
      "CPU times: user 1min 20s, sys: 44.1 s, total: 2min 4s\n",
      "Wall time: 1min 28s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model.compile(optimizer='adagrad',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# lets also add the history variable to see how we are doing\n",
    "# and lets add a validation set to keep track of our progress\n",
    "history = model.fit(X_ints_train+ [X_train_num],\n",
    "                    y_train, \n",
    "                    epochs=10, \n",
    "                    batch_size=32, \n",
    "                    verbose=1, \n",
    "                    validation_data = (X_ints_test + [X_test_num], y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[10475   885]\n",
      " [ 1342  2358]] 0.852124833997\n"
     ]
    }
   ],
   "source": [
    "yhat = np.round(model.predict(X_ints_test + [X_test_num]))\n",
    "print(mt.confusion_matrix(y_test,yhat), mt.accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x12a743278>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnEAAAEWCAYAAAAJlMFHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8VfX9+PHXO5tsyABCCBvCkC1LRcGFitsq7rpt62hr\nrdra2lb9fa2rdbR11Yk4cYurggLKBtlhhZVFBhCSQPb798e9wTSF5GbcnHtv3s/H4z5y77nnnPs+\nEA7v+xnvj6gqxhhjjDHGvwQ5HYAxxhhjjGk+S+KMMcYYY/yQJXHGGGOMMX7IkjhjjDHGGD9kSZwx\nxhhjjB+yJM4YY4wxxg9ZEmcChogEi0ipiKS15b7GGNNSItJbRFREQtyvPxORqz3ZtwWf9TsReaE1\n8Rr/IlYnzjhFRErrvYwEKoAa9+ubVPX19o/KGGP+m4h8DixV1T822H4u8CyQqqrVRzm2N7AdCD3a\nPi3c9yRgpqqmenQRJiBZS5xxjKpG1z2AXcDZ9bb9TwLX0m+nxhjTSq8AV4iINNh+JfB6UwmXMd5i\nSZzxWSLygIi8JSJviEgJrpvoRBFZLCL7RSRXRJ4UkVD3/iHuroje7tcz3e9/JiIlIrJIRPo0d1/3\n+2eIyGYRKRaRp0TkOxH5afv+iRhjHPIBkACcULdBRDoD04FXReQsEVklIgdEZLeI/OloJxKRb0Tk\nevfzYBF5VEQKRSQTOKvBvteIyEb3PSlTRG5yb48CPgNS3MNCSkUkRUT+JCIz6x1/joisd98vvxGR\nwfXe2yEivxGRNe772lsiEtEWf1im/VgSZ3zd+cAsIA54C6gGbgcSgeOAacBNjRx/GfAHoAuu1r77\nm7uviCQDbwN3uj93OzCupRdkjPEvqnoI1z3gqnqbLwYyVHU1UOZ+Lx5XIvYzETnPg1PfgCsRHAWM\nBS5q8H6++/1Y4BrgbyIyWlXLgDOAnHq9Fzn1DxSRgcAbwC+BJGAO8LGIhDW4hmlAH2A48FMPYjY+\nxJI44+sWqurHqlqrqodUdZmqLlHValXNBJ4DTmzk+HdVdbmqVgGvAyNbsO904AdV/dD93t+AwtZf\nmjHGj7wCXFSvteoq9zZU9RtVXeu+T63BlTw1dl+qczHwd1Xdrap7gf+r/6aqfqqq29TlW+BL6rUG\nNuES4FNV/cp933oU6ARMqrfPk6qa4/7sj2n8/mh8kCVxxtftrv9CRNJF5FMRyRORA8BfcLWOHU1e\nvecHgegW7JtSPw51zQbK8iB2Y0yAUNWFuL68nSci/XC1xs8CEJHxIjJPRApEpBi4mcbvS3X+694C\n7Kz/pnsYx2IR2Ssi+4EzPTxv3bkPn09Va92f1aPePs25PxofZEmc8XUNp08/C6wD+qtqLPBHoOFg\n47aWCxyeAeYe3Nzj6LsbYwLUq7ha4K4AvlDVPe7ts4CPgJ6qGgc8g2f3pVygZ73Xh0seiUg4MBtX\nC1pXVY3H1SVad96mSkvkAL3qnU/cn5XtQVzGT1gSZ/xNDFAMlLkH6TY2Hq6tfAKMFpGz3TNkb8c1\nxsQY07G8CpyCayzbK/W2xwB7VbVcRMbhGl/ribeB20Qk1T1R4u5674UB4UABUC0iZwCn1Xt/D5Ag\nInGNnPssETnZPfnrDlxlnL73MDbjByyJM/7mDuBqoARXq9xb3v5A97ftS4DHgSKgH7AK1w3RGNNB\nqOoOXElQFK6Wtzo/B/7inkX/R1wJlCeeB74AVgMrgffqfVYJcJv7XPtwJYYf1Xs/A9fYu0z37NOU\nBrFuwtVi+BSubuCzcZVxqvQwNuMHrNivMc0kIsG4uiouUtUFTsdjjDGmY7KWOGM8ICLTRCTePU7l\nD0AVsNThsIwxxnRglsQZ45njgUxc41NOB85XVetONcYY4xjrTjXGGGOM8UPWEmeMMcYY44c6xILi\niYmJ2rt3b6fDMMa0oxUrVhSqqt+XgrH7lzEdj6f3rw6RxPXu3Zvly5c7HYYxph2JyM6m9/J9dv8y\npuPx9P5l3anGGGOMMX7IkjhjjDHGGD/UIbpTjQlktbVK3oFytheWkVlQSmZhGdsLy9hRWMbByhqn\nw2szg7rF8Np1450OwxjTDv69cDvzMvL5+4yRJEaHOx2Oz7Ikzhg/UXyoisyCUney5krUMt3J2qGq\nH5O1yLBg+iRGMbRHHLERgfNPvHtcJ6dDMMa0g4VbCnng0w2owoznFvP69ePpGhvhdFg+KXDu8CYg\nVVbXsmrXPhZlFhEaHMTkAUkMTYklKEicDs0rKqpr2FV0kG3uJG17YenhhK2o7MclD4ODhJ6dO9En\nMYpJ/RLokxhF36Qo+iZG0zU2HJHA/PMxxh9t3lPCB6uyueO0QQQH6L2rreQVl3P7m6sYkBzNPWcO\n5pbXV3LJs4t4/YYJ9Ii3L3INWRJnfEptrbIh9wDfbyvku61FLN2+l0NVNYiAKjzyxSYSosI4fkAi\nkwckccLARJJj/OsbmqqSW1zuTs5K6yVsZWTtO0htvfrbidHh9E2K4tQhXembFEWfxGj6JEaR1iWS\nsBAb0mqMrztYWc3Nr60gs7CMkwd3ZUyvzk6H5LOqamq59Y2VHKqq4Z+Xj6Z/cgyvXT+eq19cysXP\nLGLWDePplRDldJg+xZI44yhVZUfRQb7bWsj32wpZtK2IfQerAOiXFMVPxqZyXP9EJvRJoKKmhoVb\nCpm/uYAFWwr58IccAAZ3j2XywEROHJDEmN6dCQ8JdvKS/kfxoSpW797Pql37WbV7Hz/s3s9+9zXC\nj92fI3rGc96oHvR1t6r1TowiNiLUwciNMa11/ycb2F5URpDAvIx8S+Ia8eiXm1i2Yx9PzBhJ/+QY\nAEandeaNGyZw5b+XcPGzi3j9+gn0T452OFLf0SGW3Ro7dqxanSXfkX+gnO+3FbkTtyKy9x8CoHtc\nBJP6JXJc/wQm9UukW9zRW9jqWuy+3VzA/M0FrNi5j+paJTIsmAl9E5g8IJHJA5PokxjVrl2L1TW1\nbNpT4krYdu3nh9372FZQBoAIDEiOZlTPzgxLjaNfUhT9kqJJjrHuT28QkRWqOtbpOFrL7l/+6/N1\nedw8cwU3n9iPFTv3UlZRw5zbT3A6LJ/01YY93PDqci4fn8aD5x/zP+9vyivh8hcWAzDz+vGkd4tt\n7xDblaf3L0vijNcdKK9i8baiw4nblvxSAOI6hTKxbwLHDUjkOPe4rpYmM6UV1SzaVsT8zQXM31LA\nzqKDAKR27sTkgUlMHpDEpP4Jbd6ytedAOat27WOVu6VtbVbx4UkGCVFhjEqLZ1RaZ0b2jGd4ahwx\n1rLWblqbxInINOAJIBh4QVUfavB+HDATSMPVq/Goqr7kfm8HUALUANV1cYjI/cC5QC2QD/xUVXMa\ni8PuX/4pr7icaU/Mp2fnSGb/bBIvLMzk4c83sfiekxv9gtoR7d57kLOeXEBaQiTv3jyJiNAj96Zs\nKyjl8ueXUF5dw8zrxjOsR1w7R9p+LImrx26C7au8qoaVO/fxnXtc25qs/dQqRIQGcWzvLhzXP5Hj\n+iUyJCXWa4N8dxaVMX9zAd9uLmTRtkLKKmsIDhJGp8UzeUASkwcmcUyPuGZNkCivqmFddvGP3aK7\n9pNTXA5AaLAwNCWOkT3jGZUWz+i0zqR27mQtbA5qTRInIsHAZuBUIAtYBlyqqhvq7fM7IE5V7xKR\nJGAT0E1VK91J3FhVLWxw3lhVPeB+fhswRFVvbiwWu3/5n9pa5coXl7By534+ue14+iVFk5F3gGl/\nX8BDFxzDjHFpTofoMyqqa/jJM4vYXljGp7eeQFpCZKP77yo6yKXPL+ZAeRWvXDuO0WmB2T3t6f3L\nxsSZVqupVdZlF7PQPa5t+Y59VFTXEhwkjEiN4xdT+nNc/0RGpcW323i1XglRXDkxiisn9qayupaV\nu/YdbqV77KvNPPbVZjpHhnL8gKTDXa/1p7DXjdVbtcs1hm3Vrv1szD1AtXvWQc8unRjTuwvX94xn\nZFo8Q7rHHvXbo/FL44CtqpoJICJv4mpB21BvHwVixJWpRwN7gerGTlqXwLlFuc9hAswLCzP5bmsR\nD11wDP2SXOO3BnWNISUugrkZ+ZbE1fPgpxtZk1XMs1eOaTKBA0hLiOTtmydy+fOLufKFJbz402MZ\n3zehHSL1TV5N4rzUHfEn4AagwH2a36nqHG9eR0dXXVNLfkkFucWHyNlfTl5xOTnFh8jdX05u8SEy\nC8soKXf935XeLYbLx/fiuP4JjOvTxSe6D8NCgpjQN4EJfRP47bR0CksrDk+QmL+lkI9Xu3qz0rvF\ncGzvLuzed/C/Jh9EhQUzomc8N53Yl5E9XV2jSTFWfDLA9QB213udBTSsNPw08BGQA8QAl6hqrfs9\nBf4jIjXAs6r6XN1BIvIgcBVQDEw50oeLyI3AjQBpafYfvj9Zl13MI19s4vShXbnk2J6Ht4sIJ6Un\n88GqbCqqa3xuApYTPl6dw6uLdnLDCX04fWg3j4/rEd+Jt2+ayGUvLOHql5by/FVjOWFAk2vFBySv\ndad6sTviT0Cpqj7qaSzWHXF0NbVKYWkFOfsPkVtc7nq4n+cUHyKvuJw9B8r/q+wFuBKb7vGd6B4X\nQc8ukYzv04VJ/RL9LrmprVU25h1g/mZXUrdq9z56dYk63C06Kq0z/ZOjrbaTH2pld+pFwDRVvd79\n+kpgvKre0mCf44BfA/2Ar4ARqnpARHqoaraIJLu336qq8xt8xj1AhKre11gsdv/yH4cqa5j+1AJK\nK6r5/PbJdI4K+6/3/7NhD9e/upzXrhvXYZOOOtsKSjnnqYWkd4/lzRsnEBrc/JJJhaUVXPHCEjIL\nyvjXFaM5eXBXL0TqDF/oTvVKd4TxXG2tUlRWSW7xoQbJ2Y/P9xwoP9xFWCciNIiUuE50j3fNFk2J\nj6C7+3VKXCe6xUUQGxESEOO9goJcY9mGpsTxs5P6OR2O8R3ZQM96r1Pd2+q7BnhIXd+Et4rIdiAd\nWKqq2QCqmi8i7+O6H85vcPzrwByg0STO+I8HPt1AZmEZM68b/z8JHMCk/gmEhQQxNyO/Qydxhypr\n+PnMlYSHBvP0ZaNalMCBq47mmzdO4KoXl3LTayt48tJRnHlM9zaO1rd5M4nzWncEcKuIXAUsB+5Q\n1X0NP7yjdkdU19Ty3bYiPl6dwxfr8w53c9YJCwmie1wE3eMiGNeni+t5fCdS4tyJWlwE8ZGhAZGg\nGdMKy4ABItIHV/I2A7iswT67gJOBBSLSFRgEZIpIFBCkqiXu56cBfwEQkQGqusV9/LlAhvcvxbSH\nL9fn8fqSXdw0uS/H9U884j6RYSFM7JvAvIx87jt7aDtH6Dv++OE6NueX8Mo141q9nF58ZBgzrx/P\nNS8t45ZZK3n84pGcN6pHG0Xq+5ye2HA68AMwFXd3hIgscA/+Pb5+d4SIZLi7I/4F3I8rybsfeAy4\ntuGJ3Unfc+DqjmiXq3FIba2ybMdePl6Tw5y1eewtqyQmPITThnZjeGqcO2lztaQlRIVZgmZME1S1\nWkRuAb7ANab3RVVdLyI3u99/Btf952URWQsIcJeqFopIX+B997+zEGCWqn7uPvVDIjIIV4mRnUCj\nM1ONf8g/UM5ds9cwNCWWO04b1Oi+U9OTue+j9WQWlNI3qeMVrX17+W7eWZHFbVP7M3lg27RGxkaE\n8uq147j+leX86u0fqKiu4ZJjO0bjjTeTOK90R6jqnrqDReR54BMvXoPPUlXWZBXz8eocPlmTS96B\nciJCgzhlcFfOHpHCiQOTbLakMa3gnjA1p8G2Z+o9z8HVytbwuExgxFHOeWEbh2kcVlur3PHOag5V\n1fDEjFFNLodXl8TN21TQ4ZK4jbkH+MMH65jUL4HbTxnYpueOCg/hpWuO5abXVnDX7LVUVNdy1cTe\nbfoZvsibSZy3uiO6q2qu+/jzgXVevAafsymvhI9WZ/Px6lx27T1IaLBw4sBkfnfWYE5OTyYq3OnG\nVWOM6The/G47C7YU8uD5wzxaDqpnl0j6J0czLyOf647v0w4R+oaS8ip+/vpK4jqF8sSMUV6ZLBYR\nGsxzV43hllmr+OOH6ymvquHGyYE91tlr/+N7sTviYREZias7dQdwk7euwVfsKCzj49U5fLwmh817\nSgkOEib1S+CWqf05fUg34iKdL+NhjDEdzfqcYh7+fBOnDunKZc2o/TY1PZmXvttOaUU10R3gi7eq\ncvd7a9m19yCzrh/v1SoG4SHB/PPy0fzyrR/4f3MyKK+q5dap/QN2GJFXf3u81B1xZRuH6ZNy9h/i\n0zW5fLwmhzVZxQCM692F+88dyhnHdCcx2r9KeRhjTCA5VFnD7W/+QFxkKH+9cHizkoSTBiXx3PxM\nFm4pZNowz+uj+atXF+3k0zW53DUtvV0K84YGB/HkjFGEhwTx+FebKa+q4c7TBwVkIhf4XwH8SGFp\nBZ+tzeWj1Tks2+GacDs8NY7fnzmYs4Z3JyW+dbN4jDHGtI3/N2cjW/NLee26cXQ5QjmRxhzbuwsx\n4SHMy8gP+CRu9e79PPDpBk5OT+amyX3b7XODg4RHLxpBRGgw//xmG+VVtfxh+uCAS+Q8TuJEZALw\nJyAC+LuqfuCtoDqS4kNVfLE+j49X5/Dd1kJqFQZ2jeY3pw1k+vAUeidGOR2iMcaYer7euIfXFu/k\n+uP7tKjeW2hwECcMTGTepnxUNeASizr7D1by89dXkhwTwWMXj2jWWtVtIShIePC8YYSHBPHid9up\nqK7h/nOHtXsc3nTUJE5EuqlqXr1Nv8Y1kUCAJYAlcS1UXlXjTtxymb+5gMqaWtK6RPLzk/pz9ogU\nBnWLcTpEY4wxR5BfUs6d765hcPdY7pzWeDmRxkwZlMyctXmszznAsB5xbRihb6itVe54ezX5JeW8\nc/Mk4iOb11rZVkSEP04fQkRoMP/6ZhsV1bX89cLhAbMKT2Mtcc+IyErgYVUtB/YDF+Gqb3SgkeNM\nE+55by3vr8qmW2wEV03sxdkjUhieGhew38aMMSYQ1NYqv3lnDWUV1Tw5Y2Sr1j89aVAyAPMy8gMy\niXtuQSZfZ+Tzp7OHMLJnvKOxiAi/PX0QESHB/O0/rjFyf7tkZItXivAlR03iVPU8ETkb+EREXgV+\niatESCRwXjvFF3D2H6zk0zW5XDoujQfPC6xmXWOMCWQvf7+D+ZsLuP/coQzo2roek6SYcEakxjF3\nUz63njygjSL0DUsyi3jki02cdUx3rp7U2+lwAFcid/spA4gIDeL/PsugsrqWpy4b1apE3Bc0moaq\n6se4VlWIA94HNqvqk6pa0B7BBaKPV+dQWVPL5ePTLIEzxhg/sTH3AA99lsHJ6clcMaFXm5xzSnoy\nP+zeT1FpRZuczxcUlFRw6xurSOsSyUMXHuNzPUw3ndiPP58zlC837OGm11ZQXlXjdEitctQkTkTO\nEZF5wOe4CupeApwrIm+KSGBXz/Oid1dmk94thqEpsU6HYowxxgPlVTXc/uYqYjuF8teLmldOpDFT\n05NRhW83B0a7SE2t8su3VlF8qIp/Xj6amAjfrGF69aTePHTBMXy7uYBrX15GWUV10wf5qMZa4h4A\nzgAuBv6qqvtV9Q7gD8CD7RFcoNmaX8rq3fu5cHSqz307McYYc2QPfZbB5j2lPPqT4W1ao3NYShyJ\n0eHMzchvs3M66Ymvt/Dd1iLuP3cYg7v7dkPFjHFpPH7xCBZnFnHK49/yxtJdVNfUOh1WszWWxBUD\nFwAXAod/w1R1i6rO8HZggWj2yiyCg4RzR6U4HYoxxhgPzMvI5+Xvd3DNcb0PT0ZoK0FBwkmDkpi/\nucAvE4j65m8u4Km5W7hoTCoXH9uz6QN8wPmjUnnzxol0i4vgnvfWcurf5vPx6hxqa9Xp0DzWWBJ3\nPpCAa/JDwzVPTTPV1Crvr8xm8oBEkmMinA7HGGNMEwpKKrjz3dWkd4vhrmnpXvmMqenJHCivZsXO\nfV45f3vILT7EL9/6gYHJMdx/7jCnw2mWcX268N7PJvH8VWMJCw7i1jdWMf2phczLcNXw83VHTeJU\ntVBVn1LVZ1TVSoq00qJtReQdKOfCMalOh2KMMaYJqspv313NgfJqnpgxiohQ78xiPH5AIiFBwtxN\n/tmlWlVTyy2zVlFRVcM/rxhNpzD/m+0pIpw6pCtzbj+Bv18yktKKaq55eRkXP7uIpdv3Oh1eo/y/\nSIqfmL0yi9iIEE4Z3NXpUIwxxjTh1UU7mbepgN+fOdirBdhjI0I5tncXvsnwz8kND3+ewYqd+3jo\nwuH0S4p2OpxWCQ4SzhvVg//8+kQeOG8YO4sOcvGzi/jpS0tZl13sdHhHZElcOyitqObzdXlMH5Hi\ntW9zxhhj2samvBIenLORKYOSuGpi25QTaczU9GQ27Skhe/8hr39WW/pyfR7PL9jOlRNcResDRVhI\nEFdM6MW3d07h7jPSWbVrP9OfWsgts1aSWVDqdHj/pckkTkRuFZHO7RFMoJqzNpdDVTVcOLqH06EY\nY4xpxOFyIhEhPHzRiHapJDAl3TVhwp9mqe4qOsgd76zmmB5x3Dt9sNPheEWnsGBuPrEfC+6awq1T\n+zM3I59T/zafu2evIcdHEm5PWuK6AstE5G0RmSZWG6PZZq/Iok9iFKPTLBc2xhhf9tfPM8jIK+GR\ni0aQFNN25UQa0y8pirQukczzkySuvKqGn89agQD/vHy036960JTYiFDuOG0Q3945hasm9uK9ldmc\n9Og33P/JBscLNTeZxKnqvcAA4N/AT4EtIvL/rOCvZ3bvPciS7Xu5YFQPqw1njDE+7JtN+bz03Q5+\nOqn34dax9iAiTBmUxPfbCv1iBYEHPt3AuuwDPHbxSHp2iXQ6nHaTFBPOfWcPZe5vTuS8kSm89N12\nJj88j8e/2kxJeZUjMXk0Jk5d82zz3I9qoDPwrog87MXYAsJ7K7MBON+6Uo0xxmcVllbwm3fWMKhr\nDHef4Z1yIo2Zkp5MeVUti7YVtftnN8eSzCJmLt7FjZP7cuqQjjlRL7VzJA9fNIIvf3UiJw5K4smv\ntzD54Xk8Pz+z3ZNwT8bE3S4iK4CHge+AY1T1Z8AYXIWAzVGoKu+tymJi3wRSO3ecbyvGGONPVJW7\n3l3DgfIqnrh0pCMT0Cb0TaBTaLDPj4t7dn4mXaLC+PWpA50OxXH9k6P55+Vj+PiW4zkmNZ4H52zk\npEe+YdaSXVS1U/FmT1riugAXqOrpqvqOqlYBqGotMN2r0fm55Tv3sbPooNWGM8YYHzZz8U6+zsjn\n7mnppHdzZrmoiNBgjuufwLxNvltkdsueEuZm5HPVxF5WaaGeY1LjePXacbx54wRS4iP43ftrOfXx\nb/moHVZ/8CSJ+ww4XO1ORGJFZDyAqm5s7ED3RIhNIrJVRO4+wvtxIvKxiKwWkfUick2993aIyFoR\n+UFEltfb3kVEvhKRLe6fPjtbYPaKLCLDgjljWDenQzHGGHMEW/aU8MCnGzlxYBLXHNfb0VimpCeT\nte8QW/N9q4xFnecXZBIRGsRVE3s7HYpPmtA3gdk/m8S/rx5LRGgwt72xirOeWsjcjD1eS8w9SeL+\nBdT/jSp1b2uUiAQD/wDOAIYAl4rIkAa7/QLYoKojgJOAx0QkrN77U1R1pKqOrbftbuBrVR0AfO1+\n7XPKq2r4dE0u04Z1Iyo8xOlwjDHGNLA+p5hfzFpJdHgIj/xkuOOTz6YM8t1SI/kHyvlgVQ4/GdOT\nLlFhTR/QQYkIJw/uypzbTuCJGSM5WFnNtS8v5yfPLGKbF2rMeZLEidZLId3dqJ5kJeOAraqaqaqV\nwJvAuQ32USDGXbYkGleLX3UT5z0XeMX9/BXgPA9iaXdfbthDSUU1F422rlRjjPEl2wvLuGXWSs56\nciF7DlTwt0tG+sSa1inxnUjvFuOTSdxL3++gqraW647v43QofiEoSDh3pGv1hwfPH8a+g5XEdwpt\n88/xJBnLFJHb+LH17edApgfH9QB213udBYxvsM/TwEdADhADXOJOEsGV4P1HRGqAZ1X1Off2rqqa\n636eh6uOnc+ZvSKLlLgIJvRNcDoUY0wLiMg04AkgGHhBVR9q8H4cMBNIw3UvfVRVX3K/twMoAWqA\n6rreBBF5BDgbqAS2Adeo6v52uSBDbvEhnvx6C28vzyIsOIhbpvTnhsl9ifPCf64tNSU9mefmZ1J8\nqMpn4iqtqGbm4p1MG9qN3olRTofjV0KDg7h8fC8uG5fmlZZeT1ribgYmAdn8mIjd2EaffzrwA5AC\njASeFpG6UaXHq+pIXN2xvxCRyQ0PdrcQHrGjWURuFJHlIrK8oKB916Tbc6CcBVsKOH90D4KCrDac\nMf7Gi8NBvgKGqepwYDNwj7euwfyoqLSCBz7ZwImPfMO7K7K4ckIv5v92Cr85fZDPJEp1pqYnU1Or\nLNjiO2upvrVsNyXl1dw4ua/Tofgtb3XVN9kSp6r5wIwWnDsb6Fnvdap7W33XAA+5k7GtIrIdSAeW\nqmp23eeLyPu4umfnA3tEpLuq5opId+CI7c7ulrvnAMaOHduuU30+WJVNrcIF1pVqjL86PBwEQETq\nhoNsqLdPs4eDqOqX9V4uBi5qy6DNfyspr+KFBdt5YUEmh6pquGB0KrefPMCnC9SO6hlPXKdQ5mbk\nM3248+uRVtXU8uLC7Yzr3YVRtuqQz2kyiRORCOA6YChweNCAql7bxKHLgAEi0gdX8jYDuKzBPruA\nk4EFItIVGISr+zYKCFLVEvfz04C/uI/5CLgaeMj988OmrqE9qSqzV2YxKi2efknRTodjjGkZbw0H\nqe9a4K0jfbiI3Ii7xyMtLa2l19BhlVfVMHPxTv4xbyv7DlYxbWg37jhtIAO6xjgdWpNCgoM4cWAS\n324qoLZWHe/NmbM2l+z9h/jzOUMdjcMcmSfdqa8B3XB1fX6Lq0WtpKmDVLUauAX4AtgIvK2q60Xk\nZhG52b3b/cAkEVmLa6bpXapaiGuc20IRWQ0sBT5V1c/dxzwEnCoiW4BT3K99xrrsA2zeU8qF1gpn\nTKBr8XAQEfk9rla71490YlV9TlXHqurYpKQkr11AoKmuqeWNpbs46ZFveODTjQzrEceHvziOZ64c\n4xcJXJ0zwCtnAAAgAElEQVSp6ckUlVWyJrvY0ThUlWe/zaRfUhRT23EZMuM5TyY29FfVn4jIuar6\niojMAhZ4cnJVnQPMabDtmXrPc3C1sjU8LhMYcZRzFuFqvfNJs1dmERYSxNk+0AxujGkxbw0HQUR+\niqtQ+sn1Z/6blqutVT5Zm8vfvtrM9sIyRqXF8/glI5jUL9Hp0FrkxIFJBImr1MjInvGOxfH9tiI2\n5B7goQuOcbxF0ByZJy1xdau67heRYUAcYCn5EVRW1/LhD9mcOrgrcZG+NVjWGNMsh4eDuCcrzMDV\ndVpf3XAQGg4HEZEY9/a64SDr3K+nAb8FzlHVg+1yJQFMVZmXkc/0pxZy2xurCAsO4vmrxvLezyb5\nbQIH0DkqjFFpnZnncKmRZ+dnkhgdznmjbO1vX+VJS9xz7lUR7sV1E4sG/uDVqPzUN5vy2XewigvH\n2C+8Mf5MVatFpG44SDDwYt1wEPf7z+AaDvKyeziI4B4OIiJ9gffds9FCgFn1hoM8DYQDX7nfX6yq\nN2Oaben2vTzyRQbLduwjrUskf79kJGePSCE4QFqMpqYn88gXm8g/UE5ybPvXsNuYe4D5mwu48/RB\ntsSWD2s0iRORIOCAqu7D1RVg84sbMXtlFonR4UweYGNYjPF3XhoO0r+Nw+xw1mUX8+iXm/hmUwHJ\nMeHcf94wLhnbk7AQTzqW/MdJg5J45AvXdV58bM+mD2hjz8/PJDIsmMvH28QaX9ZoEqeqtSLyW+Dt\ndorHb+0rq2RuRj5XT+xNSHBg3UyMMcZpmQWlPPbVZj5dk0tcp1DuPiOdqyf2plNYYLYSDekeS7fY\nCOZm5Ld7Epez/xAfrc7higm9iI+0JbZ8mSfdqf8Rkd/gmgpfVrdRVfd6LSo/9NHqHKpq1GrDGWNM\nG8rZ71pl4Z0VWYSH+OYqC94gIkxJT+Lj1blUVte2a0vjy9/vQMGW2PIDniRxl7h//qLeNsW6Vv/L\n7JVZDO4ey5CU2KZ3NsYY06h9ZZU8PW8rry3eCQpXTujFL6b0Jykm3OnQ2s2UQcm8sXQ3y3fsZVL/\n9pmocaC8illLdnHmMd19uiiycfFkxQZLxZuwZU8Ja7KKufeswU6HYowxfi+/pJxLnl3MzqIyLhyd\nyu2nDCC1c8dLKI7rn0hYcBBzM/LbLYl7Y8kuSiuqucmW2PILnqzYcNWRtqvqq20fjn96d2UWwUHC\nuSNtVqoxxrTG/oOVXPXvpeQVl/PmjRMZ16eL0yE5Jio8hPF9uzB3Uz73Tm+4dG/bq6yu5aXvdjCp\nXwLDesR5/fNM63nSyX5svccJwJ+Ac7wYk1+pqVU+WJXNSQOTOlQzvzHGtLWS8iqufnEpmYVlvHD1\n2A6dwNWZmp5MZkEZO4vKmt65lT5anUPegXJusFY4v9FkEqeqt9Z73ACMxlUrzgALtxay50AFF46x\nCQ3GGNNShypruO6V5azPOcA/LxvNce3Ufejrpgxy1daf6+XCv6rK8/MzGdQ1hpMGWpksf9GS6S5l\ngI2Tc3tvZRZxnUI5ebAtYmGMMS1RUV3DTTNXsGzHXh6/ZCSnDOnqdEg+o3diFH0To7yexH27uYBN\ne0q4YXJf3IWojR/wZEzcx7hmo4Ir6RuC1Y0DXE3/X6zP46IxqYSHBGatImOM8abqmlpue2MV8zcX\n8PCFwzlnhK073dCU9GReW7STsopqosI9KSrRfM/Nz6RrbLj9+fsZT34bHq33vBrYqapZXorHr8xZ\nm0t5VS0XWm04Y4xpttpa5c531/DF+j3cd/YQR1Ym8AdT05P598LtfLe1kNOGdmvz86/LLub7bUXc\nfUZ6wK18Eeg8+dvaBSxR1W9V9TugSER6ezUqPzF7RTZ9k6IY2TPe6VCMMcavqCp/+HAd76/K5jen\nDeSa42yUztEc27sL0eEhzNtU4JXzPzc/k+jwEC6zJbb8jidJ3DtAbb3XNe5tHdquooMs3bGXC0en\n2vgBY4xpBlXl/z7L4PUlu/jZSf34xRRbUrYxYSFBHN8/kW825aOqTR/QDFn7DvLp2lwuHdeT2IjA\nXgUjEHmSxIWoamXdC/fzDr+Y2uyVWYjA+aOsNpwxxjTHk19v5bn5mVw9sRe/PX2QfRH2wNT0ZHKL\ny9mYW9Km5/33wu0IWEuon/IkiSsQkcN14UTkXKDQeyH5vtpa5b1VWUzql0BKfCenwzHGGL/xwoJM\n/vafzVw0JpX7zh5qCZyHTkp3lf2Yt6ntZqkWH6zirWW7OWdEiv1f5qc8SeJuBn4nIrtEZBdwF3CT\nd8Pybct27GX33kM2ocEYY5ph1pJdPPDpRs48phsPXXAMQUGWwHkqOSaCY3rEtWmpkZlLdnKwsobr\nT7Divv7Kk7VTtwETRCTa/brU61H5uNkrs4gKC2basLafJWSMMYHog1XZ/P6DtUwZlMTfLxlFSLDN\ngmyuKYOSeHreVvaVVdI5qnWjmiqqa3jpux2cMCCRISmxbRShaW9N/isSkf8nIvGqWqqqpSLSWUQe\naI/gfNGhyhrmrM3jjGO6ExnmnXo9xhgTSL5Yn8cd76xmQp8E/nXFGCtj0UJT0pOpVVdh3tb6YFU2\nhaUV3DS5XxtEZpziyb+kM1R1f90LVd0HnOnJyUVkmohsEpGtInL3Ed6PE5GPRWS1iKwXkWsavB8s\nIqtE5JN62/4kItki8oP74VEsbeXLDXmUVlRbV6oxxnhg/uYCbp21imN6xPH81WOJCLXC6C01IjWe\nhKiwVo+Lq61VnpufyZDusRzXP6GNojNO8CSJCxaRwyu7i0gnoMmV3kUkGPgHcAauVR4uFZEhDXb7\nBbBBVUcAJwGPiUj9NuLbgY1HOP3fVHWk+zHHg2toM++uyKJHfCfG28LMxhjTqKXb93Lja8vplxzN\nK9eMI9pLqw10FEFBwomDkvh2cwE1tS0vNTI3I59tBWXcdKItseXvPEniXge+FpHrROQ64CvgVQ+O\nGwdsVdVMd1mSN4FzG+yjQIy4fouigb24VoVARFKBs4AXPLqSdpBXXM53Wwu5cHQPG5BrjDGNWJO1\nn2tfXkaP+E68dt044iKtBllbmJqezP6DVazata/F53huQSYpcRGceUz3NozMOKHJJE5V/wo8AAx2\nP+53b2tKD2B3vddZ7m31Pe0+Zw6wFrhdVesKC/8d+C3/XWi4zq0iskZEXhSRzh7E0ibeX5VNrcL5\n1pVqjDFHtSmvhKteXEp8ZCgzrx9PYnSTnTfGQycMSCI4SFo8S/WH3ftZun0v1x7fh1CbXOL3PPob\nVNXPVfU3qvoboExE/tFGn3868AOQAowEnhaRWBGZDuSr6oojHPMvoK97/1zgsSOdWERuFJHlIrK8\noKD1g0BVldkrsxjTqzN9EqNafT5jjAlE2wvLuPyFJYSHBDHr+gl0j7P6Y20prlMoY3t1bnES99z8\nbcREhDBjnC2xFQg8SuJEZJSIPCwiO4D7gQwPDssG6q9mnOreVt81wHvqshXYDqQDxwHnuD/vTWCq\niMwEUNU9qlrjbrF7Hle37f9Q1edUdayqjk1KSvLkMhu1JquYrfmlNqHBGGOOImvfQS5/fjG1qrx+\n/XjSEiKdDikgTU1PJiOvhJz9h5p13M6iMj5fl8cVE3rZ+MQAcdQkTkQGish9IpIBPIWra1RUdYqq\nPuXBuZcBA0Skj3uywgzgowb77AJOdn9eV2AQkKmq96hqqqr2dh83V1WvcO9XvxP/fGCdJxfaWrNX\nZhEWEsRZw20MgTHGNJR/oJwrXlhCSUU1r147jv7JMU6HFLCmpCcDzV+94YUF2wkOEq6Z1NsLURkn\nNNYSlwFMBaar6vHuxK3G0xOrajVwC/AFrhmmb6vqehG5WURudu92PzBJRNYCXwN3qWpTS3o9LCJr\nRWQNMAX4lacxtVRFdQ0frc7htCFdietkg3ONMaa+fWWVXPHvJeSXVPDyNeMY1iPO6ZAC2oDkaHrE\nd2JeM7pU95ZV8s6K3Zw3sgfJsRFejM60p8baUy/A1Qo2T0Q+x9Wt2awpme7yH3MabHum3vMc4LQm\nzvEN8E2911c2J4a2MC8jn/0Hq7hwjHWlGtNRiMg04AkgGHhBVR9q8H4cMBNIw3UvfVRVX3K/twMo\nwfXFt1pVx7q3/wT4E64JXeNUdXm7XIwXlZRXcfVLS9lRdJCXf3osY3q121yzDktEmJqezLsrsiiv\nqvGo9t5ri3ZSXlXLjZNtia1ActSWOFX9QFVn4BqjNg/4JZAsIv8SkUYTr0Aze2U2STHhnNA/0elQ\njDHtoI3qXE5x17IcW2/bOlxfkOd7Lfh2dLCymmtfXsaGnAM8c8VoJtk9st1MTU/mUFUNS7bvbXLf\n8qoaXl20g6npyQzoat3cgcSTtVPLgFnALHc5j58AdwFfejk2n1BUWsG8jHyuPb6PrfVnTMdxuM4l\ngIjU1bncUG+fo9a5PBpV3eg+nzdi5um5WxARkqLDSYgOI7Hez7ZeKaGiuoabXlvBip37ePLSUUxN\n79qm5zeNm9gvgYjQIOZl5HPiwMYn7727IouiskprhQtAzZqe4l5y6zn3o0P4aHUO1bVqs1KN6ViO\nVOdyfIN9nsY1WSsHiAEuqVfnUoH/iEgN8KyqNuueKSI3AjcCpKV5Xgpi1pJd5BSXH/G96PAQEqPD\nSIgOJyEqjMSYcBLdPxOiwg+/lxQdTmynkEYTzaqaWm6dtYoFWwp5+KLhTB+e0pzLM20gIjSYSf0S\nmZuRz31nDznq31dNrfLvhdsZkRpnKw0FIJtj3ITZK7MYmhLLoG7WBG2M+S91dS6nAv2Ar0Rkgaoe\nAI5X1WwRSXZvz1BVj7tQ3UnfcwBjx471eH2l7+85mUOVNRSVVVBYWklRaQWFpXXPKyksraCorIKd\nRQdZuWsfe8sqOdLqTaHBQkKUqxUvIdqV4CW6fyZEhTNvUz5fbtjDn88ZysVje/7vCUy7mJKefHgJ\nrf7J0Ufc56sNe9heWMbTl42yJbYCkCVxjdiUV8K67AP8cXrDoTDGmADnaZ3Lh1RVga0iUlfncqmq\nZgOoar6IvI+re7ZdxsF1CgsmNSyS1M5N12irqVX2HXQnd+4kr37yV7dtW34phaUVVFT/uIDOnacP\n4morVeGoKYNc3ajzMvKPmsQ9N38bPbt0YtrQbu0ZmmknlsQ1YvbKLEKChHNHWleBMR3M4TqXuJK3\nGcBlDfapq3O5oH6dSxGJAoJUtcT9/DTgL+0XuueCg8Tdwtb0sliqSlllDYUlFSjYyjU+ILVzJAO7\nRjM3I58bjjDebfmOvazctZ8/nzPUxnQHKPtbPYrqmlreX5XNSYOSSbB1/4zpUFpZ57IrsFBEVgNL\ngU9V9XMAETlfRLKAicCnIvJF+15Zy4kI0eEh9E6MsgTOh0xJT2bZjr2UlFf9z3vPzs8kPjKUn4y1\nMd2BylrijmLB1kIKSiq4aEwPp0MxxjigpXUu3TNaRxzlnO8D77dtpKYjmzoomWe/zWThlkLOOObH\nFYW2FZTyn417uHVKfyLD7L/6QGUtcUcxe0UW8ZGhh5c3McYYY3zNmF6diY0IYW6D1RteWLCd0OAg\nrpzY25nATLuwJO4Iig9V8eWGPZwzIoXwkLatrWSMMca0lZDgICYPTGLepgJq3VONC0oqmL0yiwtH\np5IUY8OBApklcUcwZ20uldW1VhvOGGOMz5uankxhaQXrcooBeHXRDqpqarnhhD7OBma8zpK4I5i9\nIov+ydEMT7VFnI0xxvi2EwcmIQJzM/I5WFnNa4t3curgrvRNOnLZERM4bLRjAzsKy1i+cx93TUu3\nwojGGGN8XkJ0OCNS45mXkU/nyDD2H6ziphNtia2OwJK4Bt5bmYUInDfKasMZY4zxD1PTk3n8q83k\nHShndFo8Y3rZElsdgXWn1lNbq8xemc3x/RPpHtfJ6XCMMcYYj0x1V1LYc6CCGyf3czga014siatn\nyfa9ZO8/ZBMajDHG+JWhKbF0jQ2nT2IUpw7p6nQ4pp1Yd2o9vRIiue3kAZxua8wZY4zxIyLCM1eM\nITIshOAgG8/dUVgSV09KfCd+fepAp8Mwxhhjmm1UWmenQzDtzLpTjTHGGGP8kCVxxhhjjDF+SFTV\n6Ri8TkQKgJ0e7p4IFHoxnPYWSNdj1+KbfPVaeqlqktNBtFYz71/gu38fLWHX4rsC6Xp88Vo8un91\niCSuOURkuaqOdTqOthJI12PX4psC6VoCQSD9fdi1+K5Auh5/vhbrTjXGGGOM8UOWxBljjDHG+CFL\n4v7Xc04H0MYC6XrsWnxTIF1LIAikvw+7Ft8VSNfjt9diY+KMMcYYY/yQtcQZY4wxxvghS+KMMcYY\nY/yQJXH1iMg0EdkkIltF5G6n42kpEekpIvNEZIOIrBeR252OqbVEJFhEVonIJ07H0loiEi8i74pI\nhohsFJGJTsfUUiLyK/fv2DoReUNEIpyOqaMKlPsX2D3Ml9n9y7dYEucmIsHAP4AzgCHApSIyxNmo\nWqwauENVhwATgF/48bXUuR3Y6HQQbeQJ4HNVTQdG4KfXJSI9gNuAsao6DAgGZjgbVccUYPcvsHuY\nL7P7lw+xJO5H44CtqpqpqpXAm8C5DsfUIqqaq6or3c9LcP0j6+FsVC0nIqnAWcALTsfSWiISB0wG\n/g2gqpWqut/ZqFolBOgkIiFAJJDjcDwdVcDcv8DuYb7K7l++x5K4H/UAdtd7nYUf3zTqiEhvYBSw\nxNlIWuXvwG+BWqcDaQN9gALgJXfXygsiEuV0UC2hqtnAo8AuIBcoVtUvnY2qwwrI+xfYPczH2P3L\nx1gSF8BEJBqYDfxSVQ84HU9LiMh0IF9VVzgdSxsJAUYD/1LVUUAZ4Jfjl0SkM67Wnj5AChAlIlc4\nG5UJJHYP8zl2//IxlsT9KBvoWe91qnubXxKRUFw3v9dV9T2n42mF44BzRGQHri6iqSIy09mQWiUL\nyFLVulaFd3HdFP3RKcB2VS1Q1SrgPWCSwzF1VAF1/wK7h/kou3/5GEvifrQMGCAifUQkDNcAx48c\njqlFRERwjVnYqKqPOx1Pa6jqPaqaqqq9cf2dzFVVv/u2VEdV84DdIjLIvelkYIODIbXGLmCCiES6\nf+dOxk8HOQeAgLl/gd3DfJXdv3xPiNMB+ApVrRaRW4AvcM1SeVFV1zscVksdB1wJrBWRH9zbfqeq\ncxyMyfzoVuB193+2mcA1DsfTIqq6RETeBVbimk24Cj9evsafBdj9C+we5svs/uVDbNktY4wxxhg/\nZN2pxhhjjDF+yJI4Y4wxxhg/ZEmcMcYYY4wf6hATGxITE7V3795Oh2GMaUcrVqwoVNUkp+NoLbt/\nGdPxeHr/6hBJXO/evVm+fLnTYRhj2pGI7HQ6hrZg9y9jOh5P71/WnWqMMcYY44csiaunvKqGD3/I\nprrG35e3M8Z0RCt27mNbQanTYRhj2oklcfV8u7mA29/8gbkZ+U6HYowxzXKwsprrXlnG799fi9X/\nNKZjsCSunpPTk+kWG8HMJbucDsUYY5olMiyEO08fxOLMvXzwg18vm2qM8VCTSZyI/J+IxIpIiIh8\nISJ7ROSy9giuvYUEBzFjXE/mby5gZ1GZ0+EYY0yzXHpsGiN6xvPgpxspPlTldDjGGC/zpCXuDFU9\nAEwHcoB04C6vRuWgS8elERwkzLLWOGOMnwkKEh48bxh7yyp57MtNTodjjPEyT5K4ujIkZwLvqOo+\nIGAHXHSNjeC0IV15e/luyqtqnA7HGGOaZViPOK6a2JvXFu9kbVax0+EYY7zIkyTuMxFZB4wHvhKR\nRKDCu2E564oJvdh3sIo5a3OdDsUYY5rt16cNJCEqnHs/WEtNbcB+5zamw2syiVPVO4GpwBhVrQIO\nARd4OzAnTeqXQN/EKGYuDohaocaYDiY2IpQ/TB/M6qxiZi21oSHGBCpPJjZcABxS1WoRuRt4CfD7\npWwaIyJcPqEXK3ftZ32OdUcYY/zPOSNSmNQvgYc/z6CgJKA7T4zpsDzpTv2TqpaIyCRc4+JeB57x\nbljOu2h0KhGhQcxcbN9ijTH+R0T4y7nDKK+q4f8+2+h0OMYYL/Akiasb3T8deFZVPwTCvReSb4iL\nDOXs4Sl8+EM2JeU2Vd8Y43/6J0dz4+S+vLcym8WZRU6HY4xpY54kcbki8g9gBjBHRMI8PM7vXTmx\nFwcra3h/lRXONMb4p1umDKBHfCf+8ME6qmxJQWMCiifJ2MXAt8CZ7vIiicDdXo3KRwxPjWd4ahyv\nLdppy9gYY/xSp7Bg/nzOULbkl/LvhdudDscY04Y8mZ1aCqwHThKRm4HOqvqZ1yPzEVeM78WW/FKW\nbt/rdCjGGNMipwzpyqlDuvLEf7aQvf+Q0+EYY9qIJ7NTbwHeAdLcj7dF5OfeDsxXnD0ihdiIEFtP\n1Rjj1+47ewiK8peP1zsdijGmjXjSnXojME5Vf6eqv8NV9Pdm74blOzqFBXPRmJ58vi7XpukbY/xW\naudIbjt5AF+s38PcjD1Oh2OMaQOeJHECVNZ7XeXe1mFcPiGNqhrl7eW7nQ7FGGNa7Prj+9IvKYr7\nPlpvywoaEwA8SeJeA5aIyL0ici/wPfCqd8PyLf2SopnUL4FZS3bZEjbGGL8VFhLE/ecNY/feQ/xj\n3lanwzHGtJInExseBm4CDrofN6vqI94OzNdcOaEX2fsP8c2mfKdDMcaYFpvUL5HzR/Xg2W8z2VZQ\n6nQ4xphW8Kjem6ouVdXH3Y9lIpLp7cB8zSlDupIcE85rtp6qMcbP3XNmOuGhQdz34Xorn2SMH2tp\n0d7QNo3CD4QGBzFjXBrfbi5g996DTodjjDEtlhwTwZ2nD2Lh1kI+WZPrdDjGmBZqaRLXIb+6XTqu\nJ0EivG7lRowxfu7y8b0Y1iOW+z/ZYEsLGuOnQo72hojcdrS3gGjvhOPbusd14pTByby9fDe/OnUA\n4SHBTodkjDEtEhwkPHjeMZz3z+94/KvN3Hf2UKdDMsY0U2MtcUlHeSQC//B+aL7pigm92FtWyWdr\n85wOxRhjWmVEz3guH5/GK9/vYH1OsdPhGGOa6agtcar6h/YMxF8c1y+R3gmRzFy8k/NG9XA6HGOM\naZU7T0vns7V53PvBOmbfPImgoA5VBtQYv9bSMXEdVlCQcPn4XizfuY+NuQecDscYY1olLjKU3505\nmFW79ltBc2P8jCVxLXDRmFTCQ4KYaeVGjDEB4ILRPRjXuwsPfZ7B3rLKpg8wxvgES+JaoHNUGNOH\np/DBqmxKK6qdDscYY1pFRHjg/GGUllfz0GcbnQ7HGOOhJpM4EbntCI+rRWRYewToq66YkEZZZQ3v\nr8p2OhRjjGm1gV1juO6EPry9PIvlO/Y6HY4xxgOetMRNAm4H+rkftwLnAK+KyB1ejM2njewZz7Ae\nsby+eKdVPDfGBITbpg4gJS6Cez9YR3VNrdPhGGOa4EkS1x0Yqaq3q+rtwGhcZUaOB67zZnC+TES4\nYnwvMvJKWL5zn9PhGGNMq0WFh/DHs4eSkVfCy9/vcDocY0wTPEniugKH6r2uALqq6kH38w7rnJEp\nxESE2AQHY0zAOH1oV6YMSuJvX20mr7jc6XCMMY3wJIl7C1gkIr8Xkd8DC4C3RCQK2OTV6HxcZFgI\nF45O5bO1eRSVduh81hgTIESEP58zjOpa5f5PNjgdjjGmEU0mcap6H65xcOXux+2qep+qlqnqjMaO\nFZFpIrJJRLaKyN1HeD9dRBaJSIWI/MaTY0XkTyKSLSI/uB9nenqx3nDFhDQqa2p5e3mWk2EYY0yb\nSUuI5JYp/fl0bS7fbi5wOhxjzFF4WmJkCfAa8AawS0RSmjpARIJxLc91BjAEuFREhjTYbS9wG/Bo\nM4/9m6qOdD/meHgNXtE/OYYJfbvw+pKd1NTaBAdjTGC48cS+9EmM4r4P11FeVeN0OMaYI/CkxMjP\ngQJc3aj/Ab52/2zKOGCrqmaqaiXwJnBu/R1UNV9VlwFVzT3Wl1wxoRdZ+w4x376xGmMCRHhIMH85\ndyg7ig7y7LeZTodjjDkCT1rifg0MVtVBqjpEVQerasMWtSPpAdRfwyXLvc0TTR17q4isEZEXRaTz\nkU4gIjeKyHIRWV5Q4N3k6rQh3UiKCbcJDsaYgHLCgCSmD+/OP77Zys6iMqfDMcY04EkSl4Wr29NX\n/AvoC4wEcoHHjrSTqj6nqmNVdWxSUpJXAwoLCWLGsT2Zuymf3XsPevWzjDGmPf1h+hDCgoP444fr\nrSamMT7GkyRuKzBXRO6sv2qDB8dlAz3rvU51b/PEUY9V1T2qWqOqtcDzuLpeHXfpuDQEeGPpLqdD\nMcaYNtM1NoJfnzqQbzcX8Pm6PKfDMcbU40kSlwvMB2KBpHqPpiwDBohIHxEJA2YAH3kY11GPFZHu\n9fY7H1jn4Tm9KiW+E1PTu/L28t1UVlulc2P8nZdm1z8iIhnu4SDvi0h8e1xLa101sReDu8fy5483\n2HrRxvgQT0qM/OFIDw+OqwZuAb4ANgJvq+p6EblZRG4GEJFuIpKFa9zdvSKSJSKxRzvWfeqHRWSt\niKwBpgC/asF1e8WVE3tRWFrJ5+vt26ox/syLs+u/Aoap6nBgM3CP1y6iDYUEB/HAecPIO1DOk19v\ncTocY4xbyNHeEJHHVPUOEXkf+J+BEKp6QVMnd5f/mNNg2zP1nufh6ir16Fj39iub+lynnNA/kV4J\nkcxctJNzRjRZhcUY47sOz5AHEJG6GfKHq9+qaj6QLyJneXqsqn5Zb7/FwEXeu4S2NaZXZ2Yc25N/\nL9zOhaNTGdQtxumQjOnwGmuJe8v982lc3yobPkwDQUHCZePSWLpjL5vySpwOxxjTct6cXV/nWuCz\nI52gPWfXN8dd09KJjQjh3g/W2rARY3zAUZM4VV3q/vn1kR7tF6J/+cnYnoSFBPH6Eis3Yow5MvcS\nhtXA60d6vz1n1zdH56gw7jlzMMt27OP4v87lH/O2sv9gpdNhGdNheVLsd4KIfCYiG0Rks4hsEZHN\n7ZK2ifgAAB8rSURBVBGcP+oSFcb0Y7rz3spsymwAsDH+yiuz6wFE5KfAdOBy9cOaHReP/f/t3Xl4\nVfW18PHvyjyHJGQgI1MgzMhsGVScoCKiVlHEXu1ge1u11LZqp2u9HbS3fasdvNXW4apQ1CIOWGet\ngAhBiEFGCQIJSSAhJCQh87DeP85JCAgh8xmyPs9znrP3Pnvvs7Yhy5W9f0MKT39tGiMTwvndW59x\n/gPv8/OXd3CgxMaRM6avnbVNXBtPAXcDWwGbe6UDbpqRxupPCng5u4Cbpqe5OhxjTOe19pDHUYDd\nACzp7rEiMg9HPr1AVT12UMkLRsRywYhY9hyp4In1B3j+40Msz8zlklHxfGPWEKYNiUZEXB2mMV6v\nI0Vchaqu6fVIvMik1AGMGhTB8k15LJmWasnMGA+jqo0i0tJD3hd4sqV3vfPzR0UkAdiCY/ilZhFZ\nBoxW1YozHes89V+AQOAdZ17YpKrf7tOL60EZCRH87roJ/GjeSJZvzOXZTbm8s6uIcUmRfGP2EL48\nbhD+vh2dotsY01lyrrv5IvKAc3E1UNeyXVU/7cW4etSUKVN0y5Ytffqd/8jM4ycvbefF//wSk9PO\nODOYMaYXichWVZ3i6ji6yxX5q6tq6ptY/Uk+T3x4gP1Hq0iICOKWmYO5cWoqkSH+rg7PGI/R0fzV\nkTtxs057B8eQI3O6Elh/cdXERH7z+m6Wb8q1Is4Y0y8EB/hy0/Q0bpyaygd7i3l8/QEefGMPf3ov\nh+unpHDrzMGkxYS6OkxjvMY5izhVnd0XgXib0EA/rpmUxHObD/HzBaOJDg1wdUjGGNMnfHyEuRnx\nzM2IZ2dhOU98eIAVmbk8vfEgl42O55uzhzI5LcqamhjTTe0N9nujqq482zypqvqn3gvLOyydkcYz\nG3P555ZDfOuCYa4Oxxhj+tyYxEj+cP1E7pmXwTMbD7J8Ux5v7SxiQsoAvjFrCPPHJuBn7eaM6ZL2\nfnNangHGnuVlzmFEfDjThkSzIjOP5maPG0nAGGN6THxEED+6PIONP57LLxeNpaKmgTtWfsIFv/uA\nv6/bT0Vtg6tDNMbjnPVOnKr+r/P9nPOkmrNbOiONO1d+wrqco1w4Ms7V4RhjjEuFBPhx84w0bpqW\nynt7inl8/X5+/fpuHn53L4unpnLrzMGkRIe4OkxjPMI528SJSCBwCzAGCGrZrqq39V5Y3mPemAQG\nhgWwfFOeFXHGGOPk4yNcOjqeS0fHsz2/nCc+3M8zGw/yfx8dYN7YBL4+a6h1CjPmHDrSEOEZYDCO\nEcYzgWFAbS/G5FUC/Hy4fkoK7+8pouB4javDMcYYtzMuOZKHbziP9fdcxG1zhvFhTgnX/vUjlvx9\nE4dKPXZM5D6hqnx2pJLH1+/nlqc289cPPnd1SKYPdWSIkRGqulhErlDVJ0TkGWB9bwfmTZZMT+Wv\naz9nZWYeP7x8pKvDMcYYtzQoMph752dwx9zhPP/xIR56Zy/zHl7HzxaM5oapKdab1ankRB0b9pWw\nbm8J63OOUlzpGMI1OjSADz47yvjkSGYOH+jiKE1f6EgR19La9LiIjAKKAHsu2AnJUSHMHRnHcx8f\n4s6L0wnws55YxhhzNqGBfnxt1hAuH5vAj/65jR+v3s5bO4/w22vHEx8RdO4TeJm6xia25paxPsdR\ntO0oqABgQIg/s4YPZE56LLPSBxIVEsCCP6/nBy9s481lsxkQYkNbebuOFHFPiEgUcB+OaWRCgP/q\n1ai80NIZabz3fx/z9q4jLBif6OpwjDHG7SUNCGb516fzzMaDPPjmHi57aB2/XDSWhRO8O4eqKp8f\nrWJ9zlHW7T3Kpv2l1DQ04ecjTEqL4oeXjWB2eixjkyLx9Tn17uQfbziPRY9s4CcvbeeRJZPs7qWX\na7eIExFfoERVy4B/A6l9EpUXmjMilpToYJZvyrUizhhjOsjHR7hl5hDmjIjlrhe2cefKT3hr5xF+\nddVYorxoEPWyqno2fF7Ceucj0sJyR9PzIQNDuW5KMrPTYzl/WAxhge3fexmbFMldl43gf978jNVZ\nBVw7Obkvwjcu0u6/BlVtEpGfAC/2UTxey9dHWDItjd++uYecokrS48NdHZIxxniMobFhrPr2+Ty2\nbj8Pv7uXzQdK+e2145ibEe/q0LqkoamZT/KOO+625ZTwaf5xVCE8yI+ZwwZy+9xYZqcP7NJwK9+a\nM4wPPjvKfa/uZNqQaBuyxYuJavuD0IrIAzjawT0PVLVsV9WK3g2t57jLBNLHTtRx/gPvs2R6Kr9Y\nOMbV4Rjj1To6gbS7c5f85U52FVZw1wvZ7DlSyeIpKfxswSjCg/xdHVa7VJXcY9WszznK2r0lbNp/\njBN1jfgInJcaxez0gcxOj2VCcmSPzGCRX1bN/IfXMzIhnOdum2GzYniYjuavjrSJW+p8/wGOie/F\n+W6PVjspJiyQ+eMSeHFrPnfPG0lIQEf+8xtjjGlrdGIEr9w+k4ffzeGxtZ+z4fMSfveVCZw/LMbV\noX1BwfEantucx8vZBRwqdQwzlRwVzMKJicxJH8j5wwYSGdzzBWhyVAi/XDSWZc9n8+jaz7l9bnqP\nf4dxvfbmTp2hqptUNaUvA/J2N89I45XsQl7JLuTGaVYHG2NMVwT6+XLPvAwuGRXHD17Yxo1/38TX\nZg7h7nkjCfL3dWlsTc3K2r3FrNiUx78/K0aBOemxfHP2UOakx5IWE9InHQ6umpjIe3uKefjdHMdd\nvpQBvf6dpm+1dyvof4FJfRVIfzE5LYqMhHD+58095JVWc+2kZIbHhbk6LGOM8UiT06J5/XuzefCN\nPTy54QBr9xbzh+snuqRgKa6o5YUth1i5+RAFx2uIDQ/kuxcNZ/HUFJKj+r5dmojwq6vGsvVgKd9/\nPpvX7pxlT4C8zFnbxIlIlqp6RRHnbm1KdhSU84d39rJ271GampUJyZFcOzmZK8cnelVvK2NcydrE\n9T/rc45y96pPKa6s47sXDuOOi9Px7+W2YM3Nysb9x1iRmcvbO4tobFZmDR/ITdNTuWR0fK9/f0d8\n9HkJNz2eyY3TUvnN1eNcHY7pgI7mr/aKuOPAurMdqKoLux5e33LXJFhcWcur2YW8mFXA7sMV+PsK\nczPiuGZSMheNjLNBgY3pBivi+qfymgbuX7OT1VkFjE2K4A/XT2REL4wGUFZVz6qt+fxjcx4HSqoY\nEOLPdZOTWTI9jSEDQ3v8+7rrgdd389i6/Tz+1SlcMtoze/T2Jz1RxOUA3zjbgaq6tuvh9S1PSIK7\nCitYnZXPy9mFlJyoIyrEn4UTErl2cjLjkiJtwEZjOsmKuP7tzR1H+OlL26msa+SHl43g67OGfmFg\n3M5SVbLyyli+KY9/bT9MfWMzU9KiuGlGKvPHDnJ5W7z21DU2cfUjH1FUUcuby+YQGx7o6pBMO3qi\niLPHqS7Q2NTM+pwSVmXl886uIuobm0mPC+OaScksOi+RQZHBrg7RGI9gRZwpOVHHT1Zv5+1dRUwd\nHMX/u24iqTGdb5tWWdvAy58UsCIzjz1HKgkL9OOaSUksmZ5KRkJEL0TeO3KKKlnw5w/50rAYnrxl\nqt0ccGM9UcStVtVrejwyF/DUJFhe08C/Pj3M6qx8tuSWIQKzhg/kmklJXD4mwRqoGtMOK+IMOO6e\nrc4q4Bev7qRJlZ9eMYol01I7VMDsKChnRWYur2QXUl3fxJjECJbOSGPhhERCzzFzgrt6+qOD3Pfq\nTn65aCw3z0hzdTjmLLpdxHkTb0iCB0uqWP1JAauz8skvqyE0wJf54wZx7aRkpg+JxqebjwmM8TZW\nxJm2Co/XcPeqT/lwXwkXjIjlt9eOJyEy6Av71dQ3sWZbISsyc9mWX06Qvw8LJyRy0/Q0xid7ftMW\nVeWWpz4m88AxXrtjto2O4KasiGvDm5Jgc7Oy+WApq7PyeX37EU7UNZI0IJhrJiVxzaRkt2xQa4wr\nWBFnTtfcrCzPzOU3r+8mwNeHXy4ay8IJiYgIe4sq+UdmHi9m5VNZ20h6XBg3TU/l6knJvTIYrysV\nV9Qy74/rSRwQxOr/nGmd6NyQFXFteGsSrKlv4u1dR1i1NZ8N+0poVpiUOoBrJyezYFwikSHelXiM\n6Qwr4szZHCip4gcvZJOVd5xLRsVTUdvA5gOlBPj6MH9cAjdNT2Pq4CiPv+vWnrd2HuFbz27lOxcO\n4+55Ga4Ox5ymJ9rELVXV5c7lmaq6oc1nt6vqX3os2l7WH5LgkfJaXs4u4MWt+eQUnyDAz4dLRsUx\nY2gMw+PCGB4XRmxYoFcnJWPasiLOtKepWXls3ec89M5eBkUGs2R6KtdNTiYmrP/02rz3xU95fssh\nnvvmDKYPdb8py/qzHu2denpPVU/rudqfkqCqsqOgghez8lmzrZBjVfWtn0UG+zM8Lox0Z1E3PC6M\n9PhwEiODrLgzXseKONMRtQ1NBPj69Mt2xVV1jVzxp/U0NClvLJtNRJA9vXEXHc1f7XWvkbMsn2nd\nuAkRYVxyJOOSI7nvytEUV9aRU3SCnOJK9hWfIKf4BG/vKuK5jw+1HhMa4MuwlqIuLry10EuJDun2\nuErGGOPO3Hlst94WGujHQ4sn8pVHN3LfKzt5aPFEV4dkOqm9Ik7PsnymdeOGRIT4iCDiI4KYlT7w\nlM+OnahrLer2OV8f7TvG6qyC1n0C/HwYOjCU9PhwhseGkR7vKO7SYkKtIawxxniB81KjuHNuOg+9\nu5eLMuJYOCHR1SGZTmiviMsQkU9x3HUb5lzGuT601yMzvSomLJCYsMAvtIOoqG1oLer2FZ8gp6iS\n7ENlrNlW2LqPn4+QFhNy8q5dfBgj4sMZFhtmxZ0xxniY7140jLV7i/nZS9uZkhZF4gAbVN5TtFfE\njeruyUVkHvBHwBd4XFUfPO3zDOApYBLwU1X9/bmOFZFo4HlgMHAQuF5Vy7obq3GICPJnUmoUk1Kj\nTtleXd/I/qNVzrt3leQUnWBvcSXv7C6iqdlxY9bPRxgaG8rIhAgyEsIZER9ORkI4SQOC+2V7E2OM\n8QR+vj48tHgiX/7jeu56IZt/fGOG5WwPcdYiTlVz266LSAwwB8hT1a3nOrGI+AKPAJcC+cDHIvKq\nqu5qs1spcCewqBPH3gu8p6oPisi9zvV7znmlpltCAvwYmxTJ2KTIU7bXNTZxoKSKz45Utr6yck+9\ncxca4MuIhPDWwm5kQjgZCRFEhwb09WUYY4w5g7SYUO5bOIa7V33K4x/u57Y5w1wdkumAsxZxIvIa\ncK+q7hCRQUAWsAXHo9W/qerD5zj3NGCfqu53nu854CqgtYhT1WKgWESu6MSxVwEXOvd7GvgAK+Jc\nJtDPl4yEiC/MH1hZ28DeohPOwq6CPUcqeWPHEVZuPtmhIjY88LTCLpz0uHCCA/pvQ2NjjHGV6yYn\n8/7uYn731mfMHD6QMYmR5z7IuFR7j1OHqOoO5/KtwDuq+lURCQc2AOcq4pKAQ23W84HpHYyrvWPj\nVfWwc/kIEH+mE4jIbcBtAKmpqR38WtNTwoP8mZwWxeS0k49lVZWjlXXscd6x23Okkr1FlSzflEtd\nYzMAIjA4JpQR8WGtj2VHJoQzOCbUesoaY0wvEhEeuGYclz9cxrLnsllzx6w+771bVdfIy9kFPLsx\nl4LjNVw2OoErJwxi5vCB+Ptam+vTtVfENbRZvhj4O4CqVopIc69G1UGqqiJyxp6yqvo34G/gGGep\nTwMzZyQixEUEERcRxJwRsa3bm5qV3GNVpxR2nx2p5J1dRTib2xHo50PSgGCiQwOICg0gOiSA6DDH\ne1RoANGh/kSHBjrX/QkL9LOx74wxppOiQgP4/XUT+OqTm3nwjT38YuGYPvneHOcf9C9mFXCirpHR\ngyK4dFQ8b+86wotZ+USHBjB/bAJXTkhk6uBo+6Peqb0i7pCI3IHjLtgk4E0AEQkGOjIiYAGQ0mY9\n2bmtI9o7tkhEBqnqYedj3uIOntO4KV8fYWhsGENjw5g/blDr9tqGJnKKTvBZkeORbGF5LWVV9Rwq\nrWbboeOUVdfT0HTm+jzA14eoUH+iQgJaC7+Y0IB21v0J9LPHuMYYM2dELLfOHMxTGw5yUUYcF7T5\no7sn1Tc28/auIzy7MZdM57RnC8YPYun5aZyXMgARoa6xiXV7S1izrZDVWQWsyMwjPiKQK8YlcuWE\nQUx07tdftVfEfR34b+ASYLGqHndun4GjR+m5fAyki8gQHAXYDcCSDsbV3rGvAv8BPOh8f6WD5zQe\nJsjft3Xg4jNRVU7UNVJW1cCxqjrKqusprWqgrKqeY1X1lFXVU1rteN9dWEFpdT3HqxvOeC6AsEA/\nBoQ47uIFB/gSGtDy7ktwgB8hpy07XqcuBwf4EhroS4i/Y9mGXDHGeKJ75mWwYV8JP/znNt5aNqdH\nO6IdLq9hZWYeKz8+xNHKOlKig/nx/Ayum5Lyhe8J9PPl0tHxXDo6nur6Rt7fU8yr2YUs35TLkxsO\nkBIdzJXjE7lyQiIZCeH9rqA767RbPXJykS/jaDvnCzypqr8WkW8DqOqjIpKAo7NEBNAMnABGq2rF\nmY51njMGeAFIBXJxDDFS2l4cNm2NadHY1Ex5TQOlVfWUVtW3Fn6lVXWOArC6nqq6Rmoamqiqa6S6\nvsm53ERNfSPVDU105lfG31cI9vcl1FkYthR7EUF+xEUEER8eRFxEIPERgcSFOwZmjgkNsO79PcCm\n3TKme3YVVrDokQ1cODKWx26e3K0CqblZ2fB5Cc9uzOXd3UUoMHdkHEvPT+OC9NhO57yK2gbe3lnE\nmm2FfLivhKZmZXhcmLOgG8TQ2LAux+oOemLu1FfbO1BVF3Yxtj5nSdD0FFWltqGZ6npHgVdd30RV\nfSM1zuWW7VV1zm0NTVTXndy35fPymgaKK+sobTO3bQs/HyE2PNBZ5AU6ijxngeco+BzLUSH+/e6v\nzs6wIs6Y7vv7uv38+vXd/PbacSye2vlOguXVDfxz6yFWZOZxoKSK6NAAFk9NYcm0VFKiQ3okxtKq\net7YcZhXswvZfLAUVRiTGMGVExJZMH4QyVE98z19qSeKuKM4eoiuBDI5bb5UVV3bA3H2CUuCxl3V\nNzZz9EQdRRW1FFfUUlThWC6qqKO4spbiijqKKmvP+Bg4wNfHWey1FHnOwi/Csewo9AKIDPbvl491\nu1vE9dJg5dcBv8AxmPo0VT1nYrL8ZVypuVlZ+kQm2YeO8/qdsxk8MLRDx23PL+fZTQd5dVshtQ3N\nTEmLYumMNOaPS+jV9sdHymv51/bDrNlWSPYhRyuwyWlRXDl+EF8eP4i48KBe++6e1BNFnC+OwXZv\nBMYD/wJWqurOngy0L1gSNJ6utqGJo5WnFnhFFXWOwq/yZPFXWdt4xuOD/X2JDPZvfUW0WR4Q4n/W\nzzy5AOxOEefMf3tpM+A4cGPbwcpFJA5IwzFYeVlLEdfesSIyCkfTkceAH1oRZzzB4fIaLn9oHUNj\nw1j17fPxO8tQH7UNTazZ5mivti2/nJAAXxadl8TS6WmMTow44zG9Ke9YNa9tL2TNtsPsPlyBj8CM\noTFcOSGReWMSiHLjAec7mr/am7GhCUeP1DdFJBBHMfeBiNyvqn/puVCNMecS5O9LSnTIOR8/1NQ3\ntRZ4RRW1lFXXU17dQHnNqa/8smp2FTqWq+qb2j1nsL9va6F3eoHXUgQOCAloHd4l2tnrt6/Hl+ph\nvTJYuarudm7r/SswpocMigzmN9eM4/Z/fMJf/r2PZZeMOOXzgyVVrMjM5YUt+ZTXNJAeF8b9C8dw\n9aQkIoI6MphF70iNCeE7Fw7nOxcOJ6eokjWfOu7Q/Xj1dn7+8g7mjIhlwXhH+zkfAR/n76WPCD4+\nIAg+4vh9FefnPuLYLgI+PoJwcjut+5zcLj60rgf7+/Z4e+f2eqfiLN6uwFHADQb+BLzUoxEYY3pM\ncIAvaTGhpMV07JEHOB7pVtSeWuRVtCyfVgAer2ngUGk1O5zr1e0UgMH+vq3Dt0SFtBnOJSTg1OFf\n2qy7UeHXW4OVG+ORFoxP5P09xfz5/X3MGRHL+KRI3t9TzLObclmfU4Kfj3D52ARunpHG9CHRbveH\nSnp8OHddGs73L0lnZ2EFaz4t5LVth3l/T9+NUvbWsjmMTAjv0XO2N+3WM8BY4HXg/jazNxhjvEiA\nnw8DwwIZGBbY6WPrGx29fY9X11NW3dCmx69jaJey6obW9bzSasqq6qk4yyNfgJAA3zMXeSEBpMWE\nsOi8pO5cqsewGWeMO7p/4Rg2HyjlO8uz8BEoLK8lISKIuy4dwQ1TU4iLcP/2ZiLSOg/4PZdnsL2g\nnGNVdTQ3gwLNqqgqqtCsznUcndqaT9uO871ZQXG+t+7zxfXY8M7n2HNp707cUqAK+B5wZ5uqWhzX\no33/gNsY41YC/BydKzqTnBqamjnuLO7K2gzz0rJe2vreQF5pNaVV9VTWNjI+ObIvi7jeGqy8Q2zG\nGeOOwoP8eWjxRG5+IpOpg6P5ryvHcMmouLO2kXN3Pj7ChJQBrg6jW9prE+eZPxVjjFvz9+184Vff\n2EzNOdru9bDeGqzcGI82dXA0u+6fZ2NZuol228QZY4w7CPDz6dNesqraKCK3A29xcsDxne0NVi4i\nyzg5WPkXjgUQkauBPwOxwL9EJFtVL++zCzOmB1gB5z6siDPGmDNQ1ddxtAluu+3RNstHcDwq7dCx\nzu0vYZ3DjDE9xB6ZGmOMMcZ4oF6dO9VdOGefyO3g7gOBkl4Mp6950/XYtbgnd72WNFWNdXUQ3dXJ\n/AXu+/PoCrsW9+VN1+OO19Kh/NUvirjOEJEt3jDfYgtvuh67FvfkTdfiDbzp52HX4r686Xo8+Vrs\ncaoxxhhjjAeyIs4YY4wxxgNZEfdFf3N1AD3Mm67HrsU9edO1eANv+nnYtbgvb7oej70WaxNnjDHG\nGOOB7E6cMcYYY4wHsiLOGGOMMcYDWRHXhojME5HPRGSfiNzr6ni6SkRSROTfIrJLRHaKyPdcHVN3\niYiviHwiIq+5OpbuEpEBIrJKRPaIyG4ROd/VMXWViHzf+W9sh4isFJEgV8fUX3lL/gLLYe7M8pd7\nsSLOSUR8gUeA+cBo4EYRGe3aqLqsEfiBqo4GZgDf9eBrafE9YLerg+ghfwTeVNUMYAIeel0ikgTc\nCUxR1bE45gm9wbVR9U9elr/Acpg7s/zlRqyIO2kasE9V96tqPfAccJWLY+oSVT2sqlnO5Uocv2RJ\nro2q60QkGbgCeNzVsXSXiEQCc4AnAFS1XlWPuzaqbvEDgkXEDwgBCl0cT3/lNfkLLIe5K8tf7seK\nuJOSgENt1vPx4KTRQkQGA+cBma6NpFseBu4Gml0dSA8YAhwFnnI+WnlcREJdHVRXqGoB8HsgDzgM\nlKvq266Nqt/yyvwFlsPcjOUvN2NFnBcTkTDgRWCZqla4Op6uEJEFQLGqbnV1LD3ED5gE/FVVzwOq\nAI9svyQiUTju9gwBEoFQEVnq2qiMN7Ec5nYsf7kZK+JOKgBS2qwnO7d5JBHxx5H8VqjqalfH0w0z\ngYUichDHI6K5IrLctSF1Sz6Qr6otdxVW4UiKnugS4ICqHlXVBmA18CUXx9RfeVX+Asthbsryl5ux\nIu6kj4F0ERkiIgE4Gji+6uKYukREBEebhd2q+gdXx9MdqvpjVU1W1cE4fibvq6rH/bXUQlWPAIdE\nZKRz08XALheG1B15wAwRCXH+m7sYD23k7AW8Jn+B5TB3ZfnL/fi5OgB3oaqNInI78BaOXipPqupO\nF4fVVTOBm4HtIpLt3PYTVX3dhTGZk+4AVjj/Z7sfuNXF8XSJqmaKyCogC0dvwk/w4OlrPJmX5S+w\nHObOLH+5EZt2yxhjjDHGA9njVGOMMcYYD2RFnDHGGGOMB7IizhhjjDHGA1kRZ4wxxhjjgayIM8YY\nY4zxQFbEmX5JRC4UkddcHYcxxnSW5S/Twoo4Y4wxxhgPZEWccWsislRENotItog8JiK+InJCRB4S\nkZ0i8p6IxDr3nSgim0TkUxF5yTk3HiIyXETeFZFtIpIlIsOcpw8TkVUiskdEVjhH7UZEHhSRXc7z\n/N5Fl26M8XCWv0xvsyLOuC0RGQUsBmaq6kSgCbgJCAW2qOoYYC1wn/OQZ4B7VHU8sL3N9hXAI6o6\nAcfceIed288DlgGjgaHATBGJAa4GxjjP86vevUpjjDey/GX6ghVxxp1dDEwGPnZOvXMxjmTVDDzv\n3Gc5MEtEIoEBqrrWuf1pYI6IhANJqvoSgKrWqmq1c5/Nqpqvqs1ANjAYKAdqgSdE5BqgZV9jjOkM\ny1+m11kRZ9yZAE+r6kTna6Sq/uIM+3V17ri6NstNgJ+qNgLTgFXAAuDNLp7bGNO/Wf4yvc6KOOPO\n3gO+IiJxACISLSJpOP7dfsW5zxLgQ1UtB8pEZLZz+83AWlWtBPJFZJHzHIEiEnK2LxSRMCDSOdH2\n94EJvXFhxhivZ/nL9Do/VwdgzNmo6i4R+Rnwtoj4AA3Ad4EqYJrzs2Ic7U4A/gN41Jnk9gO3Orff\nDDwmIv/tPMd17XxtOPCKiATh+Ev6rh6+LGNMP2D5y/QFUe3qnVxjXENETqhqmKvjMMaYzrL8ZXqS\nPU41xhhjjPFAdifOGGOMMcYD2Z04Y4wxxhgPZEWcMcYYY4wHsiLOGGOMMcYDWRFnjDHGGOOBrIgz\nxhhjjPFA/x8IBW7LdK0/MwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x126639780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure(figsize=(10,4))\n",
    "plt.subplot(2,2,1)\n",
    "plt.plot(history.history['acc'])\n",
    "\n",
    "plt.ylabel('Accuracy %')\n",
    "plt.title('Training')\n",
    "plt.subplot(2,2,2)\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('Validation')\n",
    "\n",
    "plt.subplot(2,2,3)\n",
    "plt.plot(history.history['loss'])\n",
    "plt.ylabel('MSE Training Loss')\n",
    "plt.xlabel('epochs')\n",
    "\n",
    "plt.subplot(2,2,4)\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.xlabel('epochs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Generalization Better\n",
    "In what ways might we try to make the model found generalize more concretely?\n",
    "\n",
    "**Self Test:** What is the best method to make the results better?  \n",
    "- A. Add Dropout to deep network\n",
    "- B. Add Dropout to wide network\n",
    "- C. Perturb the Categorical Data during Training (augmentation)\n",
    "- D. Add L2 regularization to different layer's weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:MLEnv]",
   "language": "python",
   "name": "conda-env-MLEnv-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}