[Solution] digit_recognizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handwritten Digit Recognition\n",
    "### Using MNIST, Keras and OpenCV\n",
    "---\n",
    "\n",
    "**Dataset**: [MNIST](http://yann.lecun.com/exdb/mnist/)\n",
    "* `x_train, y_train`: contains  60 000 training images, and labels\n",
    "* `x_test, y_test`: contains  10 000 test images, and labels\n",
    "\n",
    "\n",
    "**Idea**: We will train a Fully Connected Neural Network to perform real time prediction on hand written images using your built in PC-webcamera.\n",
    "\n",
    "> A key part in achieving this is succesfully preprocessing the images fed  from our webcamera to be as equal to the images the model will be trained on. In other words, we will attempt to create a MNIST preprocessing pipeline.\n",
    "\n",
    "By [William Kvaale](https://github.com/wQuole)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import all the libraries and modules needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "# NumPy - math library for handling matrices and arrays\n",
    "import numpy as np\n",
    "\n",
    "# MNIST dataset containing over 60 000 training and 10 000 test examples\n",
    "from keras.datasets import mnist\n",
    "\n",
    "# OpenCV - Opensource Computer Vision library\n",
    "import cv2 as cv\n",
    "\n",
    "# Matplotlib's PyPlot - visualization library\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Pandas - Data Analysis library\n",
    "import pandas as pd\n",
    "\n",
    "#  Keras is Tensorflow's high level API for Deep Learning\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Conv2D\n",
    "from keras.layers import MaxPool2D\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Activation\n",
    "from keras.layers import Flatten\n",
    "from keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load and visualize the data\n",
    "<details><summary>Hint: </summary>\n",
    "<p>\n",
    "    \n",
    "https://keras.io/api/datasets/mnist/\n",
    "    \n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's visualize the distribution of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualize_distribution(data):\n",
    "    df = pd.DataFrame.from_records([data])\n",
    "    df = df.T\n",
    "    df.rename(columns = {0: 'target'}, inplace=True)\n",
    "    count = df['target'].value_counts().sort_index()\n",
    "    count.plot.bar(color=plt.cm.Paired(np.arange(len(count))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD2CAYAAAA6eVf+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUQ0lEQVR4nO3df5Bd9Xnf8fcHZEhspyCCqsESsehYsYOnAZMtkJJ6iKmFwIlFMw4DyQQNQ63OFP9qO60h/YMWQoqnnTowk3hGMXJFxjbBJB5wyhirYOpxO4AWhPklsGQMkRR+bCyBkzCxC376x/0qvpZ32V3p7tWG7/s1s3PPec73nPMcLf7cs+ece52qQpLUhyMOdwOSpPEx9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOjJr6Cd5e5KHhn6+m+RjSY5LsiXJjva6tI1PkhuS7EzycJLThra1vo3fkWT9Qh6YJOnHZT7P6Sc5EtgDnAFcDuytquuSXAEsraqPJzkf+DBwfht3fVWdkeQ4YBKYAAp4APiFqto30/6OP/74WrVq1cEdmSR16oEHHvjLqlo23bIl89zWOcC3quqZJOuAs1t9M3AP8HFgHXBTDd5N7k1ybJIT2tgtVbUXIMkWYC3w+Zl2tmrVKiYnJ+fZoiT1LckzMy2b7zX9i/hhSC+vqmfb9HPA8ja9Atg1tM7uVpupfmCzG5JMJpmcmpqaZ3uSpNcy59BPchTwfuALBy5rZ/Uj+T6HqtpYVRNVNbFs2bR/nUiSDtJ8zvTPAx6squfb/PPtsg3t9YVW3wOcOLTeylabqS5JGpP5hP7F/Oj199uB/U/grAduG6pf0p7iORN4qV0GuhNYk2Rpe9JnTatJksZkTjdyk7wJeC/wr4bK1wG3JLkMeAa4sNXvYPDkzk7gZeBSgKram+QaYGsbd/X+m7qSpPGY1yOb4zYxMVE+vSNJ85PkgaqamG6Zn8iVpI4Y+pLUkfl+OEvTWHXl/zzkbTz9X943gk4k6bV5pi9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOuJXK0taeP8pI9jG4v1/+fv7xDN9SeqIoS9JHTH0JakjXtPXSE1s/seHvI3J9Y+MoBNJ05nTmX6SY5PcmuSJJNuT/GKS45JsSbKjvS5tY5PkhiQ7kzyc5LSh7axv43ckWb9QByVJmt5cL+9cD3y5qt4BnAJsB64A7qqq1cBdbR7gPGB1+9kAfAogyXHAVcAZwOnAVfvfKCRJ4zFr6Cc5Bng3cCNAVX2/ql4E1gGb27DNwAVteh1wUw3cCxyb5ATgXGBLVe2tqn3AFmDtCI9FkjSLuZzpnwRMAZ9Jsi3Jp5O8CVheVc+2Mc8By9v0CmDX0Pq7W22m+o9IsiHJZJLJqamp+R2NJOk1zeVG7hLgNODDVXVfkuv54aUcAKqqkozkkxNVtRHYCDAxMeGnMebhS0//10Na/1dX/fsRdSJpsZpL6O8GdlfVfW3+Vgah/3ySE6rq2Xb55oW2fA9w4tD6K1ttD3D2AfV7Dr71gc9v231I61/8rpWH2oIkzdnl7/7DQ97G73/tgwe97qyhX1XPJdmV5O1V9SRwDvB4+1kPXNdeb2ur3A58KMnNDG7avtTeGO4Efnfo5u0a4MqD7lzSnPzgno8d0vpHnP17I+lDi8Ncn9P/MPDZJEcBTwGXMrgfcEuSy4BngAvb2DuA84GdwMttLFW1N8k1wNY27uqq2juSo5AkzcmcQr+qHgImpll0zjRjC7h8hu1sAjbNoz/poLzyhxsPaf0lH9wwok60mGz90vZDWv+f/OrPjaiTw8evYZCkjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSR+b6LZuS5mn3ihNnHzSLlXt2zT5ImgfP9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSNzCv0kTyd5JMlDSSZb7bgkW5LsaK9LWz1JbkiyM8nDSU4b2s76Nn5HkvULc0iSpJnM50z/l6vq1KqaaPNXAHdV1WrgrjYPcB6wuv1sAD4FgzcJ4CrgDOB04Kr9bxSSpPE4lMs764DNbXozcMFQ/aYauBc4NskJwLnAlqraW1X7gC3A2kPYvyRpnuYa+gV8JckDSTa02vKqerZNPwcsb9MrgOFvidrdajPVf0SSDUkmk0xOTU3NsT1J0lzM9Vs2f6mq9iT5h8CWJE8ML6yqSlKjaKiqNgIbASYmJkayTUnSwJzO9KtqT3t9Afgig2vyz7fLNrTXF9rwPcDwd8qubLWZ6pKkMZk19JO8KclP7Z8G1gCPArcD+5/AWQ/c1qZvBy5pT/GcCbzULgPdCaxJsrTdwF3TapKkMZnL5Z3lwBeT7B//uar6cpKtwC1JLgOeAS5s4+8Azgd2Ai8DlwJU1d4k1wBb27irq2rvyI5EkjSrWUO/qp4CTpmm/h3gnGnqBVw+w7Y2AZvm36YkaRT8RK4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSR+Yc+kmOTLItyZ+1+ZOS3JdkZ5I/TnJUqx/d5ne25auGtnFlqz+Z5NyRH40k6TXN50z/o8D2oflPAJ+sqrcB+4DLWv0yYF+rf7KNI8nJwEXAO4G1wB8kOfLQ2pckzcecQj/JSuB9wKfbfID3ALe2IZuBC9r0ujZPW35OG78OuLmqvldV3wZ2AqeP4BgkSXM01zP93wP+A/CDNv/TwItV9Uqb3w2saNMrgF0AbflLbfzf1adZR5I0BrOGfpJfAV6oqgfG0A9JNiSZTDI5NTU1jl1KUjfmcqZ/FvD+JE8DNzO4rHM9cGySJW3MSmBPm94DnAjQlh8DfGe4Ps06f6eqNlbVRFVNLFu2bN4HJEma2ayhX1VXVtXKqlrF4Ebs3VX1m8BXgQ+0YeuB29r07W2etvzuqqpWv6g93XMSsBq4f2RHIkma1ZLZh8zo48DNSX4H2Abc2Oo3An+UZCewl8EbBVX1WJJbgMeBV4DLq+rVQ9i/JGme5hX6VXUPcE+bfoppnr6pqr8Ffn2G9a8Frp1vk5Kk0fATuZLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOzhn6Sn0hyf5JvJHksyX9u9ZOS3JdkZ5I/TnJUqx/d5ne25auGtnVlqz+Z5NwFOypJ0rTmcqb/PeA9VXUKcCqwNsmZwCeAT1bV24B9wGVt/GXAvlb/ZBtHkpOBi4B3AmuBP0hy5AiPRZI0i1lDvwb+us2+of0U8B7g1lbfDFzQpte1edryc5Kk1W+uqu9V1beBncDpozgISdLczOmafpIjkzwEvABsAb4FvFhVr7Qhu4EVbXoFsAugLX8J+Onh+jTrDO9rQ5LJJJNTU1PzPiBJ0szmFPpV9WpVnQqsZHB2/o6FaqiqNlbVRFVNLFu2bKF2I0ldmtfTO1X1IvBV4BeBY5MsaYtWAnva9B7gRIC2/BjgO8P1adaRJI3BXJ7eWZbk2Db9k8B7ge0Mwv8Dbdh64LY2fXubpy2/u6qq1S9qT/ecBKwG7h/RcUiS5mDJ7EM4AdjcnrQ5Arilqv4syePAzUl+B9gG3NjG3wj8UZKdwF4GT+xQVY8luQV4HHgFuLyqXh3t4UiSXsusoV9VDwPvmqb+FNM8fVNVfwv8+gzbuha4dv5tSpJGwU/kSlJHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjswa+klOTPLVJI8neSzJR1v9uCRbkuxor0tbPUluSLIzycNJThva1vo2fkeS9Qt3WJKk6czlTP8V4N9V1cnAmcDlSU4GrgDuqqrVwF1tHuA8YHX72QB8CgZvEsBVwBnA6cBV+98oJEnjMWvoV9WzVfVgm/4rYDuwAlgHbG7DNgMXtOl1wE01cC9wbJITgHOBLVW1t6r2AVuAtaM8GEnSa5vXNf0kq4B3AfcBy6vq2bboOWB5m14B7BpabXerzVQ/cB8bkkwmmZyamppPe5KkWcw59JO8GfgT4GNV9d3hZVVVQI2ioaraWFUTVTWxbNmyUWxSktTMKfSTvIFB4H+2qv60lZ9vl21ory+0+h7gxKHVV7baTHVJ0pjM5emdADcC26vqvw8tuh3Y/wTOeuC2ofol7SmeM4GX2mWgO4E1SZa2G7hrWk2SNCZL5jDmLOC3gEeSPNRqvw1cB9yS5DLgGeDCtuwO4HxgJ/AycClAVe1Ncg2wtY27uqr2juIgJElzM2voV9XXgcyw+Jxpxhdw+Qzb2gRsmk+DkqTR8RO5ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR2ZNfSTbEryQpJHh2rHJdmSZEd7XdrqSXJDkp1JHk5y2tA669v4HUnWL8zhSJJey1zO9P8HsPaA2hXAXVW1GrirzQOcB6xuPxuAT8HgTQK4CjgDOB24av8bhSRpfGYN/ar6GrD3gPI6YHOb3gxcMFS/qQbuBY5NcgJwLrClqvZW1T5gCz/+RiJJWmAHe01/eVU926afA5a36RXArqFxu1ttprokaYwO+UZuVRVQI+gFgCQbkkwmmZyamhrVZiVJHHzoP98u29BeX2j1PcCJQ+NWttpM9R9TVRuraqKqJpYtW3aQ7UmSpnOwoX87sP8JnPXAbUP1S9pTPGcCL7XLQHcCa5IsbTdw17SaJGmMlsw2IMnngbOB45PsZvAUznXALUkuA54BLmzD7wDOB3YCLwOXAlTV3iTXAFvbuKur6sCbw5KkBTZr6FfVxTMsOmeasQVcPsN2NgGb5tWdJGmk/ESuJHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqyNhDP8naJE8m2ZnkinHvX5J6NtbQT3Ik8PvAecDJwMVJTh5nD5LUs3Gf6Z8O7Kyqp6rq+8DNwLox9yBJ3UpVjW9nyQeAtVX1L9v8bwFnVNWHhsZsADa02bcDTx7ibo8H/vIQtzEKi6GPxdADLI4+7OGHFkMfi6EHWBx9jKKHt1bVsukWLDnEDY9cVW0ENo5qe0kmq2piVNv7+9zHYuhhsfRhD4urj8XQw2LpY6F7GPflnT3AiUPzK1tNkjQG4w79rcDqJCclOQq4CLh9zD1IUrfGenmnql5J8iHgTuBIYFNVPbbAux3ZpaJDtBj6WAw9wOLowx5+aDH0sRh6gMXRx4L2MNYbuZKkw8tP5EpSRwx9SeqIoS9JHVl0z+kfqiTvYPAp3xWttAe4vaq2H76u+pXkdKCqamv7yo21wBNVdcdh7OmmqrrkcO1fi8PQE4R/UVX/K8lvAP8U2A5srKr/d1gbXCCvqxu5ST4OXMzg6x12t/JKBr/Ym6vqusPV2+HQ3gBXAPdV1V8P1ddW1ZfHsP+rGHzP0hJgC3AG8FXgvcCdVXXtGHo48JHgAL8M3A1QVe9f6B6mk+SXGHwtyaNV9ZUx7vcMYHtVfTfJTwJXAKcBjwO/W1UvjaGHjwBfrKpdC72vWfr4LIP/Nt8IvAi8GfhT4BwG2bh+TH38I+DXGHyG6VXgm8Dnquq7C7K/11nofxN454Hv0O0d/bGqWn14OvuRXi6tqs+MYT8fAS5ncNZyKvDRqrqtLXuwqk4bQw+PtH0fDTwHrBwKm/uq6ufH0MODDALt00AxCP3PMzgRoKr+90L30Pq4v6pOb9MfZPC7+SKwBvjSuE5IkjwGnNIen94IvAzcyiDoTqmqXxtDDy8BfwN8i8Hv4gtVNbXQ+52mj4er6ueTLGFwReAtVfVqkgDfGNN/nx8BfgX4GnA+sI3BG9C/AP51Vd0z8p1W1evmB3iCwXdOHFh/K/Dk4e6v9fLnY9rPI8Cb2/QqYJJB8ANsG1MP26abbvMPjamHI4B/w+AvjVNb7anD8Hsf/rfYCixr028CHhljH9uHph88TL+Tbe33sga4EZgCvgysB35qjP8WjwJHAUuBvwKOa/WfGP53WuAeHgGObNNvBO5p0z+zUP87fb1d0/8YcFeSHcD+Px1/Bngb8KGZVhq1JA/PtAhYPqY2jqh2Saeqnk5yNnBrkre2Psbh+0neWFUvA7+wv5jkGOAH42igqn4AfDLJF9rr8xyee1lHJFnKIOxS7cy2qv4myStj7OPRob82v5Fkoqomk/wsMK5r2NV+L18BvpLkDQwuA14M/Ddg2i8KWwA3MjhRPBL4j8AXkjwFnMngEvG4LGFwWedoBpeYqKo/b/8uI/e6urwDkOQIBtdKh2/kbq2qV8fYw/PAucC+AxcB/7eq3jKGHu4G/m1VPTRUWwJsAn6zqo4cQw9HV9X3pqkfD5xQVY8sdA/T7Pt9wFlV9dtj3u/TDN7owuAy01lV9WySNwNfr6pTx9THMcD1wD9j8E2OpzE4QdoFfKSqvjGGHrZV1btmWLb/JGEskrwFoKr+IsmxwD9n8Nf4/WPa/0eBy4D7GPxOPlFVn0myDPiTqnr3yPf5egv9xSDJjcBnqurr0yz7XFX9xhh6WAm8UlXPTbPsrKr6Pwvdg2aX5I3A8qr69pj3+w+AkxicZe6uqufHuO+frapvjmt/i12SdwI/x+Cm/hMLvj9DX5L64YezJKkjhr4kdcTQl6SOGPqS1JH/D2wD9f5B9h60AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_distribution(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD1CAYAAAC87SVQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQNUlEQVR4nO3de7BeVX3G8e8DARSogHAmg0k0dMQLraPSFGhplTaWAl5CHXXEjmQYNH8UC5aOBbUzdNppq1OnqDMOM9Fgw1TxQnWILaNSLnVsK3K4lFtQIgpJ5HJULlW0Gvn1j3elHNITIOc9Z+fI+n5mzrx7r7X2Xr9zyfPus979nqSqkCT1YY/dXYAkaTiGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRxbt7gKeyCGHHFLLly/f3WVI0i+U66677ntVNTFT34IO/eXLlzM5Obm7y5CkXyhJ7tpZn8s7ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4s6Ddn/aJY/u5/Gfsc3/nbV89BJZL0xLzSl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEf+7xKeRL3zn78Y6/rXL3zVHlUhaqAx9SfPvLzIH56jxz6EnX95JcmGS+5PcMq3t2UkuT3JHezyotSfJh5NsSnJTkiOnHbO6jb8jyer5+XQkSU/kqazp/wNwwg5t5wJXVNXhwBVtH+BE4PD2sQa4AEZPEsB5wNHAUcB5258oJEnDedLlnar6SpLlOzSvAo5r2+uBq4FzWvtFVVXA15IcmOTQNvbyqvoBQJLLGT2RXDz+p6CFZMX6l4x9jsnVN89BJdLCdMYrPjr2OT7ylbfP+tjZ3r2zuKruadv3Aovb9hJg87RxW1rbztolSQMa+5bNdlU/Z6+wJFmTZDLJ5NTU1FydVpLE7EP/vrZsQ3u8v7VvBZZNG7e0te2s/f+pqrVVtaKqVkxMTMyyPEnSTGYb+huA7XfgrAYundZ+aruL5xjgobYM9CXg+CQHtRdwj29tkqQBPekLuUkuZvRC7CFJtjC6C+d9wGeSnA7cBbypDb8MOAnYBDwCnAZQVT9I8lfAtW3cX25/UVeaD9s+unas4xe9fc0cVSItLE/l7p1TdtK1coaxBZyxk/NcCFy4S9U9BRffsGWs4095+dI5qkSSFj7fkSupG9d+YeNYx//6a188R5XsPv7BNUnqiKEvSR1xeUd6mnv06neOdfwex31wTurQwuCVviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BH/9o40T7YsWfbkg57E0q2b56AS6TFe6UtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI2OFfpI/SXJrkluSXJzkGUkOS3JNkk1JPp1k7zZ2n7a/qfUvn5PPQJL0lM069JMsAc4EVlTVrwJ7Am8G3g+cX1XPBx4ATm+HnA480NrPb+MkSQMad3lnEfDMJIuAfYF7gN8FLmn964GT2/aqtk/rX5kkY84vSdoFsw79qtoKfAC4m1HYPwRcBzxYVdvasC3Akra9BNjcjt3Wxh+843mTrEkymWRyampqtuVJkmYwzvLOQYyu3g8DngPsB5wwbkFVtbaqVlTViomJiXFPJ0maZpzlnVcB366qqar6GfA54FjgwLbcA7AU2Nq2twLLAFr/AcD3x5hfkrSLxgn9u4Fjkuzb1uZXArcBVwFvaGNWA5e27Q1tn9Z/ZVXVGPNLknbROGv61zB6QfZ64OZ2rrXAOcDZSTYxWrNf1w5ZBxzc2s8Gzh2jbknSLIz1H6NX1XnAeTs03wkcNcPYnwBvHGc+SdJ4fEeuJHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktSRsUI/yYFJLklye5KNSX4jybOTXJ7kjvZ4UBubJB9OsinJTUmOnJtPQZL0VI17pf8h4ItV9SLgpcBG4Fzgiqo6HLii7QOcCBzePtYAF4w5tyRpF8069JMcALwCWAdQVT+tqgeBVcD6Nmw9cHLbXgVcVCNfAw5Mcuhs55ck7bpxrvQPA6aAjye5IcnHkuwHLK6qe9qYe4HFbXsJsHna8VtamyRpIOOE/iLgSOCCqno58CMeW8oBoKoKqF05aZI1SSaTTE5NTY1RniRpR+OE/hZgS1Vd0/YvYfQkcN/2ZZv2eH/r3wosm3b80tb2OFW1tqpWVNWKiYmJMcqTJO1o1qFfVfcCm5O8sDWtBG4DNgCrW9tq4NK2vQE4td3Fcwzw0LRlIEnSABaNefwfA59IsjdwJ3AaoyeSzyQ5HbgLeFMbexlwErAJeKSNlSQNaKzQr6obgRUzdK2cYWwBZ4wznyRpPL4jV5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0ZO/ST7JnkhiT/3PYPS3JNkk1JPp1k79a+T9vf1PqXjzu3JGnXzMWV/lnAxmn77wfOr6rnAw8Ap7f204EHWvv5bZwkaUBjhX6SpcCrgY+1/QC/C1zShqwHTm7bq9o+rX9lGy9JGsi4V/ofBP4MeLTtHww8WFXb2v4WYEnbXgJsBmj9D7Xxj5NkTZLJJJNTU1NjlidJmm7WoZ/kNcD9VXXdHNZDVa2tqhVVtWJiYmIuTy1J3Vs0xrHHAq9LchLwDOBZwIeAA5MsalfzS4GtbfxWYBmwJcki4ADg+2PML0naRbO+0q+qd1fV0qpaDrwZuLKq/hC4CnhDG7YauLRtb2j7tP4rq6pmO78kadfNx3365wBnJ9nEaM1+XWtfBxzc2s8Gzp2HuSVJT2Cc5Z3/U1VXA1e37TuBo2YY8xPgjXMxnyRpdnxHriR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUkVmHfpJlSa5KcluSW5Oc1dqfneTyJHe0x4Nae5J8OMmmJDclOXKuPglJ0lMzzpX+NuBPq+oI4BjgjCRHAOcCV1TV4cAVbR/gRODw9rEGuGCMuSVJszDr0K+qe6rq+rb938BGYAmwCljfhq0HTm7bq4CLauRrwIFJDp3t/JKkXTcna/pJlgMvB64BFlfVPa3rXmBx214CbJ522JbWJkkayNihn2R/4J+Ad1bVw9P7qqqA2sXzrUkymWRyampq3PIkSdOMFfpJ9mIU+J+oqs+15vu2L9u0x/tb+1Zg2bTDl7a2x6mqtVW1oqpWTExMjFOeJGkH49y9E2AdsLGq/n5a1wZgddteDVw6rf3UdhfPMcBD05aBJEkDWDTGsccCbwVuTnJja3sP8D7gM0lOB+4C3tT6LgNOAjYBjwCnjTG3JGkWZh36VfVVIDvpXjnD+ALOmO18kqTx+Y5cSeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdWTw0E9yQpJvJNmU5Nyh55ekng0a+kn2BD4CnAgcAZyS5Igha5Ckng19pX8UsKmq7qyqnwKfAlYNXIMkdStVNdxkyRuAE6rqbW3/rcDRVfWOaWPWAGva7guBb4w57SHA98Y8x1xYCHUshBpgYdRhDY9ZCHUshBpgYdQxFzU8r6omZupYNOaJ51xVrQXWztX5kkxW1Yq5Ot8vch0LoYaFUoc1LKw6FkINC6WO+a5h6OWdrcCyaftLW5skaQBDh/61wOFJDkuyN/BmYMPANUhStwZd3qmqbUneAXwJ2BO4sKpunedp52ypaEwLoY6FUAMsjDqs4TELoY6FUAMsjDrmtYZBX8iVJO1eviNXkjpi6EtSRwx9SerIgrtPf1xJXsToXb5LWtNWYENVbdx9Ve0e7WuxBLimqn44rf2EqvriQDUcBVRVXdv+5MYJwO1VddkQ8++kpouq6tTdNX+r4bcYvUP9lqr68oDzHg1srKqHkzwTOBc4ErgN+JuqemiAGs4EPl9Vm+d7riepY/sdhN+tqn9N8hbgN4GNwNqq+tlAdfwy8HpGt7P/HPgm8Mmqenhe5ns6vZCb5BzgFEZ/3mFLa17K6Bv7qap63+6qbbskp1XVxweY50zgDEY/wC8DzqqqS1vf9VV15AA1nMfo7ywtAi4HjgauAn4P+FJV/fUANex4S3CA3wGuBKiq1813Da2Or1fVUW377Yy+N58Hjge+MNTPZpJbgZe2O+nWAo8AlwArW/vrB6jhIeBHwLeAi4HPVtXUfM87Qx2fYPSzuS/wILA/8DlGX4tU1eoBajgTeA3wFeAk4IZWyx8Af1RVV8/5pFX1tPlg9Ay51wztewN37O76Wi13DzTPzcD+bXs5MMko+AFuGLCGPRn9o3oYeFZrfyZw00A1XA/8I3Ac8Mr2eE/bfuWA3/cbpm1fC0y07f2AmwesY+P0r80OfTcO9bVgtLR8PLAOmAK+CKwGfmnAr8VN7XERcB+wZ9vPgD+fN0+bd1/g6rb93Pn6d/p0W955FHgOcNcO7Ye2vkEkuWlnXcDigcrYo9qSTlV9J8lxwCVJntfqGMK2qvo58EiSb1X7dbWqfpxkqO/HCuAs4L3Au6rqxiQ/rqp/G2j+7fZIchCjsEu1K9uq+lGSbQPWccu03zb/K8mKqppM8gJgkOUMRst9jwJfBr6cZC9GvxGeAnwAmPFvxsyDPdoSz36MAvcA4AfAPsBeA9UAoyedn7d59weoqrvb12VeJns6eSdwRZI7gO3rhc8Fng+8Y2cHzYPFwO8DD+zQHuA/BqrhviQvq6obAarqh0leA1wIvGSgGn6aZN+qegT4te2NSQ5goCfhFi7nJ/lse7yP3fNzfwBwHaOfgUpyaFXdk2R/hnsSBngb8KEkf87oj3r9Z5LNjP69vG2gGh73+dZo7XwDsCHJvgPVAKPfMm5n9Nvoe4HPJrkTOIbREvEQPgZcm+Qa4LeB9wMkmWD0BDTnnlZr+gBJ9mD0Atn0F3KvbVecQ9WwDvh4VX11hr5PVtVbBqhhKaMr7Xtn6Du2qv59gBr2qar/maH9EODQqrp5vmuYYe5XA8dW1XuGnnsmLeQWV9W3B573WcBhjJ4At1TVfQPO/YKq+uZQ8z2RJM8BqKrvJjkQeBWjJdivD1jDrwAvZvSi/u3zPt/TLfQlSTvnffqS1BFDX5I6YuhLUkcMfUnqiKEvSR35X8RkDLwSIVI9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_distribution(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discuss with your partner**:\n",
    "\n",
    "what _could_ be the result of different distribution in training- and test data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's visualize an example of every digit (0-9) from the training set  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADSCAYAAADpGRMOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABPQElEQVR4nO29eXRc133n+Xm1o/YFVSjs+0KQEneRoihbVKTItqKWZEt2O7bTPe540km6E890T3rS03MYzJaeMz3dJ91JjzOT3e22nbgjS25L3mSJFClRFDeRBAhiLex7VaF2oAr15g/o3QDcSRFVKPB+zuGxWCDg+x7e+97f/a2KqqpIJBKJpDDoir0AiUQieZCQoiuRSCQFRIquRCKRFBApuhKJRFJApOhKJBJJATHc6ouKojwQqQ2qqip3+m/lPbkx8r5cj7wn1yPvibR0JRKJpKBI0ZVIJJICIkVXIpFICogUXYlEIikgUnQlEomkgEjRlUgkkgJyy5QxiaSYKMpq1o1Op1v3v9d+ns/n1/1dVVVUVWVlZQVVVcXXJZLNgBRdyabEZDJht9ux2+1UVlZitVoJBoNYLBYcDgdlZWVUVlaiKApTU1PkcjkqKysxmUxMT08Tj8e5cOECk5OTzM7OEovFin1JEglQQqKrKMq6PxqaVaP92Upo16nX6wFYWVkB2HLXeSNMJhNOpxOv10t9fT0ej4eWlhasVivl5eXY7Xba2tpQFIW+vj6y2Sytra1YrVb6+/tZWFggHo+zsrJCPB7fcqJ77btwo3cCkFb+JmTTi65er8dkMrFz506CwaB4AU0mE3q9nrGxMWZmZuju7qanp6fYy71vmM1mampq8Pv9vPDCC+j1el599VWmpqaYnJwknU4Xe4kbgtlsxmq1sn//fr7yla9gt9txOp2YTCZcLhcGgwGz2YzRaMTlcqEoCmazmXw+j9PpxGAwYDQayWQyVFVVEQ6H+Yu/+AuOHTtGKpViaWmp2Jf4sVAUBZfLJd6DtrY2XC4XTU1NAExOTpJIJAiFQsRiMc6fP08kEnkgNupSYdOJ7todW1EU8ZK1t7fT0tLCnj17qK6upqysDJPJxMWLF+nv7ycajXLlypUt83CZTCb8fj9tbW18/vOfx2AwcPXqVfL5PPPz81tWdA0Gg7BiX375Zcxm822/x+FwiN+7oijY7XYAmpqayOVynDx5knPnzpHL5UpadBVFQa/XY7PZqK+vp7q6mk984hMEg0EOHDgAQE9PDwsLC5w7d46pqSkGBgaIRqNb4r1Yqw1Quie+TSG6ZWVlNDY24na72bFjB2VlZcDqTTaZTJjNZvbu3UswGCQQCGC32zEYDOj1ejo7OwkGg+Jl6+/v5/Lly0W+oo+PyWSitraW6upqjEYjDoeDZ599lu3btxMOh0mlUuRyuZJ98G6Gw+Ggvr4ev9+/7iXTrDfNxaKRy+WYnZ1leXkZQDwTPp8Pq9WKXq/nF37hF/D7/bz22mucOHGioNdzPzCbzdhsNpqamnjssccIBAJs27YNp9MpDJDl5WVUVcXj8WCz2XA6nUSjUSKRCIODg0J8c7lcSbkcNBfKoUOHOHToEHa7HYfDwdDQEOfPn2d2dpb+/v6P9R7Y7XZMJhN1dXV4PB7GxsaYnp5meXlZPFf3k00huhaLhaamJmpqanjuuefweDzA34mu0Wikvr4eh8Nx3fdqu34ul8NgMJDP57eE6BqNRgKBAH6/X7x0hw8fprm5mVdeeYWRkRERnd9KaAEzt9u9TnTT6TRDQ0PXWapLS0v09fWRSCSA1WfJ7XZjMpnEs/PII4+wbds2rl69WpKiq/m3t23bxuc//3n8fj+NjY0iWyOXyxGLxYSLRa/X09zcTCqVore3F6vVysLCAslkknw+X3Kiq9Pp2L17N1/5ylcoLy8nGAxy4sQJVFWlt7eXwcHB6zbju/n5ZWVl2Gw2WltbaWxsBCAej6Oq6tYSXUVR8Pv9HDp0SByPfD4fra2twtKFVctFr9eLY+by8jIrKyvo9XrxC9HpdPj9fjo6Oujt7cVms5HNZjfkhhULRVGwWCw4nU6am5tJJBIMDg6yuLhY7KXdVyKRCJcvX8br9dLd3c38/DynTp0iGo0yPj5+Q0t3fn6eXC4HrLoncrkcNTU1fOpTn6KtrY2ysjKxgZcSmpXX3NzMkSNH6OzspLq6GrvdjqIoJJNJhoeHmZub47333mNpaQmTyYTNZmP//v04HA62b99OXV0dJpOJkZERzp49y8TERLEv7a5QFAWbzUYgEMBqtaKqKjU1NTz11FMYjUaOHTt2z6Kr0+lobm6mrq6Oxx9/nI6ODpqampiYmOD48eO888479z1IX1TRDQQCPP/889TV1bFv3z7KysowGG69JM3kNxqNImii1+vx+/2YTCaqqqqwWq2kUqktJ7pWq5VsNktLSwuZTIaZmZktKbqRSASfz8fly5e5fPky//7f//s79mHr9Xrm5+cJBoM89NBDtLW1YbFYhJuqlNCMisbGRp5//nmCwSA1NTXiBJBIJOju7qa/v58/+7M/IxaLYbPZKC8vR1VVmpqa2L9/vwgwDg0NMTo6WlKiq2082nXBqi+3qqqK8vJywuEwer2ebDZ7Tz9fu7+7du3i8ccf56GHHmLv3r3E43ESiQQnT54kn8+XvuiazWZcLhdVVVW0tLRQUVGByWQSx6VryefzjI6Osri4SE9PD/Pz82zfvp3a2lr8fj9utxuz2YzT6cRut2M2m+/5l7BZMJvNNDQ0UF9fj9FoLPZyCk48HmdwcJDJycm7Og7n83nC4TCqqpZ8sFEr7Fj7R1VV5ubmOHfuHJOTk5w4cYLp6WlisRiZTEb8u2PHjtHb20sikaCiogK3283OnTtpa2sjEomwsLAgXDKbnWsDaBr3wwJVFIW6ujo6Oztxu90AYnPeqE26aKJbWVlJXV0d27dvFxd7M3K5HFevXiUUCvHaa6/R39/PF7/4RQ4dOoTJZMLtdmOxWETivMViKekoNawGF9vb26mvry85C+1+EI1GuXTpEjMzM3d1dFRVldnZWRKJBKlUagNXuPGsrazT/gCMj4/z3e9+l9HRUd5//32WlpbExpROp1lcXGRychKr1UokEqG5uZmvfOUrNDc3c+LECcLhMNlstiRE99q8fI37lZuvuRf27t0rYkaalpjN5psK/sehoKKrHZcCgQBPPPEEHR0dworTHiitEED7bGJigkgkwunTp+nv72dsbIxYLLalXAc3wmAw4HK5cDqdNz0BbGUSiQTDw8MiQHSn6HQ66urqKC8vx+VyAavBtuXl5ZI9/SQSCcbGxjCZTCK4Mzc3RyQSYWVl5ab3J5fLEQqFyGazxGIxcrmcCMKNjY0V+CruDS1jKRAIrPs8EokQCoWYmJi458Cglo6quZ5u59q8XxRcdA0GA01NTfzKr/wKfr8fi8WCqqrihdDpdGJ3WV5e5uLFiwwPD/PKK69w5coVstksOp2u5C3Z22EwGCgvL6e8vLxgD8NmIhwOE4lEgLvLxzQYDOzatYu2tjbxoqZSKeLxOJlMZkPWutFEIhG6u7sxm82oqkoqlWJ0dJSFhQURQLwRy8vLfPjhh4yMjDAzMyNOTqqq0t3dXcAruHeam5vZvXs39fX16z6fmprinXfe4cqVK/ckulpg2mq1UlZWhsViWWfwbSQFeZu1I4LH46GmpobGxkY8Hg92ux2dTkcqlWJgYAC9Xk9LSwtGo1HkGH744Yf09/cTDodFXuq1L6Em0lstferao002myWTyTA3N8f09HTJisidcre/T7PZjN1up6Ojg507d+LxeFBVlampKSYmJgiHwxu00o0lEonQ09NDNpvFaDTS09PD4uIi6XT6lvdobTqUFnjWsoFK5fSUyWRYXFy8zsjS/N33+s4riiKMGrvdLrKhCkFBRFen02EwGKipqeGZZ55h+/btVFZWCl9lNBrlzTffxGQyUVlZic1mY3BwkLGxMV555RUuX75ccknd95t8Pk86nSYWi9Hf309PT09J+OQKhU6nw+l04vP5eOqpp/jkJz+JXq9HVVV6eno4ffo0o6OjxV7mPTE2NsbExAQnTpzg1VdfFRvv7d4HnU6Hz+ejoqJCFABowlsqRKNRJiYm7nvvDM3Aa2xspLy8vKDB6oKIrl6vx2KxUF5ezrZt26irq0Ov15PP51laWiIWizE6OorFYiGZTKKqKn19fQwPDxONRslms1vOir0X8vk8KysrZDIZMpnMPecmbkUMBgONjY1UV1fjdrsxGAxks1mWlpaYmJigr6+PaDRa7GXeE1owTXtXstnsbQVXURSMRiM1NTXU1tZis9mA1ayQ+fn5kjklpVIpwuHwfc9E0eJLt7JwzWYzDoeDTCZzX4OyBRFdLcOgo6ODz372syIfV9uxR0dHeffdd7Hb7bz00kuYzWZeffVVent7mZmZkYLL6ouXy+VE1DkWi8n7sgaLxcIv/uIvsmPHDiorK1FVlXg8Tjwe54MPPuCNN964pf+zFNCCgXfye9d6NBw+fHhdqfzY2BiXL18umQ1IE9xirNftdlNXV8fs7Gzpia7H46Gzs5O6ujosFosw5TOZDKFQiNHRUeLxOLlcjv7+foxGIzMzM6JW/HY8aOKzFdtYfhyMRiNWq5WKigqqq6tFcHZ6epqpqSnC4fCWyXa5nQ/XaDRiMpmoqamhvLychoYGqqqqWF5eZn5+nunpaSYmJkomnW5lZUVUoa5FM+SsVus9+2L1ej1Go/Gm/u1sNks6nb7vWS8FEd0dO3bwT//pP6W6unqdP2l2dpbvf//7hEIhpqam0Ol0/NVf/RU6nU4EC+QRWnIr9Ho9brebQCDArl27eOSRR0Qp8LFjx0Sq4YOAXq/H6/VSXl7Ol770JZqamnj00Udxu91cvnyZqakpTp48yXvvvVcy6XOaZX+t8eXz+XjooYeYmpq6J9HVqty05lk3IhqNMjo6WlqiazKZhC+3srJSNDHJ5XJkMhmi0ShTU1PMzs6SzWZRFIXp6WkURbmpz1Lzw2g7uibimr9T83c9yEG3Bwmj0UhTUxP19fWi0Y3mg5uZmWFsbKxkrLp7xWg04vF4sFqtNDY2EggEaGxspKamBofDgclkEu9cJpMpuXTLfD5PLpdjeXl5XS8Wp9OJ2+3G5XKRTCbv6vesKAoOh4Py8vLrio+0FgKpVOqO/Od3y4aKbiAQoKmpie3bt9PS0iJKfWOxGIODg3R3d3P27FnC4TBLS0uoqsqVK1cAbrq7mM1mysrK8Hg8+P1+0RwnmUySSCRYXFwkmUxumeOk5NZ4PB5+67d+ix07dtDQ0MDKygrj4+PMzs5y+vRp3n333S3/LHg8Hp599llqa2v51Kc+hc/nIxAIiE5rKysrQrRKzRjRXGmJRILp6WkhtA6Hg+bmZsbHx9m7dy+Tk5N0d3ff8cnYYDCwY8cODh06JHo6aIyNjTEyMsL4+Ph977sAGyy6FosFv9+Py+US6SqwupNoF5VMJslkMuLm3uoFURQFp9OJy+XC5XJhtVrFz1xeXiYej4vdaau5JfL5PIuLi0QikZIPCN0PDAYDPp+P6upqqqurCQaDYoLE9PQ0o6OjRCKRku+/cCsMBgNOp1MYN9qkEbvdTiqVIpVKiXanBoMBh8OB1+vF5/ORSCRKwuLVdCEcDjM0NER1dTU2mw2dTidaXtbU1JDNZkVG1O1EUstc0Ioj1o7D0oYEhEIh0fy9pEQ3GAyyb98+Ghoa1qVmDA0N8Yd/+IfMzMywsLBwVxHZAwcO0NnZye7du6mpqRGiOzc3R29vrygT3mqim0qlePvttxkYGGBhYaHYyyk6Pp+PX/3VX6W5uZn29nZ8Ph86nY5kMsn3vvc9Tp48ydDQULGXuaEEAgGeeeYZmpqa+OIXv4jNZmNxcZHR0VHefvttotEoFRUVOBwOHnnkEVpbW5mZmaGhoYG3336b3t7eYl/CbdHS5d588036+vr4pV/6Jb70pS9ht9txuVzU1dXxuc99jtOnT3P+/HmSyeRtN5O1Zb9rK2C1zmJvvfUWP/jBD5icnNyQgPWGiK62kzgcDioqKnA6nSiKIm5gKpUSUeW7mX6gKIqwbjTrGRDHj5mZGVFjvtXI5/PEYjGRt/ygotfrsdvtlJeX09jYSENDA1arVbitwuEwExMTjI+Pb3lfrk6nw2w2Yzab0el05HI5pqammJubY3BwkGg0SjKZxO1288gjj2C320WxhMViKfby7wqtz8TMzAzJZFJkQGlN76uqqqiqqmJxcZGFhQXR4F8bcaRZxloqncvlwm63r8teWF5eJp1Os7CwwNTUFPF4fEOuZUNEV4sKdnR08MQTTwjR1ZK7o9EomUxGjBi5U3Q6He3t7Tz66KOirl7rvtTd3c1rr73G8PDwRlxSwdAeEm0X1nZiRVFkgBAoLy/n7/29v0djYyNPPPEEfr8fm81GMpnk29/+Nr29vVy4cOG2fQm2ApFIhLfffpvLly+LzmHHjh0jGo2K0542wPPAgQNUV1fjdDqpqKjAarUWe/l3RSqVIpPJMDY2xtWrV2loaMDv9+NwOGhpacHr9VJXV8fAwADf/e53SSQSLC8vYzAYREVea2srbrebpqYm3G4327Ztw+v1ikZCsViMmZkZZmdnmZ2d3bDT8oaIrjYyRUtf0XalXC5HMpkknU6Ty+Xu6qLKysqw2+14PB58Pp+IOGoZC4uLi0xMTJR8U2+dTofVasVisawTXLh/7exKEW3yg8PhoKmpSZRvOhwOkskk4XCYwcFB+vr6RNXWVkfLvdXy25eWlrh69ep1FloqlSKdTgvf7q16V29WNOMqFosxPT2N3W4nFothNBoxm814vV46OzsxGAzU19eLToQmk4lgMIjD4aC9vR2Px0Nra6soGdd0REtLW15eZmlpaUOfnw0R3ZaWFh599FExZFL7BWsTe4eHh1lcXCSVSt2RgJhMJp555hlaW1vZvXs3FRUVQsiTySSxWIzx8XGGhoZKPlLt9Xo5ePCgGDxoNBrXie2Daum6XC527NhBa2srzz77LBUVFcKH+Z3vfIfBwUF+9KMfMTU1dd/r9DcruVxOZOssLi6KDmS3otQ37UuXLhGJRGhpaWHv3r3U19fzyCOPYLFY8Hq97Nq1i9/5nd8RQTHNrWAwGLDZbBgMBhGAL1b3vg35f3U6nVRVVeF2u9cVQywvL7OwsCD8kndi6WpVNnV1dXR0dODz+db5o7QuRJoTvNQxmUxUVFSIlo7ahlXqL8vHQeuWVVtbS0NDA9XV1Xg8HlES3dfXR29vL1NTUw9UkFFriapVTt2MQnXPKgSRSIRMJkMul6OsrIyVlRUaGxtxuVzYbDasVisdHR3rrlkzVK59h669L1o+8EazYYG0G3V8n5+f5/Tp0wwMDNyR4Op0OlwuFx6Phz179nDgwIF1OXWqqnLp0iXef//9kojEfhzy+TyZTEYcEx8UNJdCZ2cnv/Zrv0ZlZSUOh4NEIsGpU6cYGRnhnXfeYXR09IGxcO8GrUCpkK0LNxKtWGF4eJiFhQXOnz/PBx98QDAYZM+ePQQCAXbu3InZbMZoNIrBpcvLyySTSVEkotPp2LlzJz6fD/i7iSPaqPqNZMPs6xv9gjOZDNPT00QikTuy2rQMCK/XKyKUWjGEFlSamZmht7d3S1k4WrRVy/gARA7z2tEsWx3tlONyuQgGgzz88MOiR+7S0hLDw8MMDQ0xOTnJ7OxssZe76dDunza8de2pqVTR3vvFxUWRqTA/P09NTQ0mk4m6ujpqamqwWq2YTCaWl5eZmZkRbVG1DAWdTicCcJpWJZPJguR2b8qRBNrD4vF4+NrXvsb27dvZvn278A+vrKwwMDCwrpa8VLom3Q6Hw8HDDz9MXV2d8Odq41bOnz/P5cuXHwiLzmQyYbfb6ezs5Mtf/jL19fVYrVaWlpaYnJwkFArx/e9/n+HhYTFhQrKete0uPR6PEKFEIrFlAo1LS0vrOpFZrVZ+/OMfC9ec1odaC7irqiqerfr6emw2Gw6HQ0yRHh4e3rBUMY1NK7paL8uDBw/yyCOPiHaQWhRzZmaGwcFBhoeHGRkZKendey1ms5mqqiqCweC6ShlNbMbHx4u8wsJgNBpxOBzU1tZy5MgRPB4PRqORVCrF3NwcExMTXL58uWRmfRUDvV6P3++nqqpKBJFyuZwQoa2AlvefSqXu6LSrVbV6PB7m5uZIJpOUlZWh1+uFpbvRvYY3lehqHaM8Hg/PPfccDQ0NNDc3C59UPp/nypUrTE1N8cMf/pBz584xPDy8ZQQXVsUmEAjg8/lKqsP//cJsNmO1Wtm5cyef/exnaWhoIBAIYDAYWFxcZHh4mD/6oz9idHS0ZMfvbDQ6nQ673Y7X6+XZZ5+ls7MTl8tFJBLhwoULvPfee0xNTRV7mUVBVVXS6bTYwIuxAW060XU4HASDQY4cOUJrayt+v3/dxOCJiQmuXr3K6dOnOXXqVJFXfP/R6um1+XEPGpqF29zczGc+8xnRZ2N5eZloNMrk5CQ//elPH1jRuBN0Oh02m02kUO3evZulpSUSiQShUIgrV65sGXfcvbC8vCw6rhWjCVBBRdfr9bJnzx7MZjPnz58Xn5vNZiorKwkEAjz33HPU1NSI9DBtAur8/DyxWIx3332XM2fOPDDH7AcFrXvcwYMHhYVbXl6OTqdjcXGRkZER/st/+S8MDw9vidTAW3Ft5s+dpgtqhTVer5cXXnhBnBJSqRRvvfUWw8PDdHd3E4lESj6f/eOgVXxq8+IKndVRUNF1Op10dHQQi8XWHZ21qpGWlhaef/550Qd0rU8zGo0yOzvLxYsXOXXqFMlkspBLl2wwWmrYrl27+OpXvyp+91oe9sjICK+88gpzc3NbvqeCVgquofWKvh2ahVteXs5TTz1FW1ubCD6ePn2aM2fOPBCb1u3QenKvzRIqJBsiulqPhWu7/dhsNmpraxkbG8Nms4nyvaqqKl588UXRmk4rgV075+r48eMMDAwwPDy8pQIBGlrFjFaFpr10yWSSU6dOMTQ0tCWzFlpaWmhubqaxsZH29nYefvjhdW4Vg8GAy+Wio6OD3/zN3xRBk1wuJ5pNa6WwVqsVo9EoCmi0r/f09DA+Ps709HRJpBY2NTXR2toq+pSEw+FbulOMRiPl5eX4fD6OHDlCTU0NDQ0NlJWVcf78eaampuju7n4gmgDdDr1eT3l5OX6/XxQhXdvEfKPZENHNZDLEYrHrooBWq5Xa2lqCwSB2ux2AxsZG2traePHFF/H7/etmHq2srIhcvHfeeYdz584xNjZWMpNM7waj0SiaM2uiqygKyWSSd999l/7+/g1PZSkGzc3N/OIv/iL79u3jE5/4xHVf1/rAOhwO2traxOda0rtWkZbJZPD5fFitVlpbW3G5XITDYRKJBN/73vd4//33WVpaKhnRffLJJ5mYmCAUCjE8PMz09PRNXQwmk4nKykoaGxt5+eWXxXy0dDrNhQsX6O7uFhV7DzraWPqqqioCgYAotiqkEbchojs9Pc2FCxeorKxk9+7d2O12MRbbarXS1NTEF77wBXQ6HcFgULR/XDt7Xqsj/9nPfsbAwAC9vb3Mz8+XROPle6GsrEw049Ysfc3C6e7uZnBwcEs15N6zZw/t7e3s37+fvXv3UlNTc1ffrx2ltXE92WxWJMRbLBZROqwoCi0tLWSzWaamphgYGNigK7p/6PV6LBaLGCevqiq9vb3XNYkym80EAgEqKip49tlnqaqqory8nHw+z4kTJ5idneXcuXOEQiHpjvsIVVXJZDIkEgnS6TSZTEa0iC0UGyK6Y2NjTE5OUlVVxYEDB6isrMTlcmEwGETC+6//+q+LbAW9Xn/dVM98Pk8ikeBv//Zvee+994jH41smoftGaPOtampqsNlsKIrC7Owso6OjnDlzhpGRkWIv8b6hKAqHDx/mC1/4ArW1tdTW1t71z9CqFQFRynktVquVsrIyduzYgc/nWxe83cyYzWZsNhvBYBCXyyUaa8N6i8xqtdLc3ExHRwdf/vKX8fl8Iuj8xhtv0N/fzwcffFAS1n2h0FLGEomEmKum+XgLxYaIrpaCMTk5ydmzZ3nooYdobGwUTmvtyKh1AFo7iFLrPjY0NCRGaGcymS1f+moymSgvL8ftdqPT6YhGo7z99tsMDg5uSSslm80K/2wikSAWi7GwsCBeiJsdpbUxK1p58O06ReXzeT788ENCoRDT09MbcSn3nXQ6TSQSwe12U1NTw969e/n85z9PIpFgYWFBpBU6nU5aWlrw+/2igfmFCxeYnp6mp6dHvDuSvyOfz5NMJjGbzUJ4tTqAQrEhoqtNiBgcHOSHP/whqqry+OOPi1QYrcRXI5fLEYvFSCaTDAwMMDMzw/e//33GxsYYGBh4IJz/ZWVl1NTUEAgEUBSFubk5/vzP/5yxsbEtWeaqlW3a7XZsNht9fX1cvHiR2dlZQqHQTb9PC5TZ7Xba2tpEL46boY166e7uLpkG9/F4nKmpKVpbW2lvb6e2tpbDhw8zPz/PwMAADodDTMwoLy8XfShCoRD/4T/8B8bGxkSTl61UOHQ/UFWVaDTKysoK4XCYxcVFnE5nQYNpG5oylkwmhR/t3Llz+P1+mpqaRHR6ZWWF5eVlIpEI77//PpFIRHT5GRkZEQ/Og4C2GaVSKc6dO8fQ0BCLi4uk0+kt9+KoqsrQ0BAnTpzA6/Xi9XqZnJxkbGyMxcVF5ubmbvq9er2e2dlZLBYL8/Pzt/XH5fN5+vv7mZmZKZnNe35+nt7eXlwuF4FAQDStWVpaEu0Ltb/39/cTiUQ4e/Ys4+PjorT1bsZgPYhovamLcY82VHQXFhaIRCIYDAaMRqMYUqmJrmbhDg4O8o1vfIPx8XFGR0dZWloSM44elAdHr9djMpmYn5/n+PHjTE9Pi3lQW/EeaI2KtNPP2pfgVter+f1v1Dr0ZmidqUrlPg4NDTE6Osr09DSTk5PU1NSwbds2ysrKcLlcIlC4sLDAsWPHCIVCvPLKKywuLj5QXehKlQ0vjsjn80SjUfr7+0WzCc1/ojWhnpiYYHJycl2D4geNxcVFLl26RDabJRQKEYlE7nhKcimiNS6SXI9mcMzPzzM4OEgsFiORSIgAm9FoxG63E41GuXz5MrOzs2IKrhTcu0ObjZZOp5mfn2d+fn7Ds4SU21gV9+WNX1sBsvY4qP1/a120ijWORlXVOy5JuV/35Fq0oCL83UtXzM3nbu4JbNx92WwU8lnR3ptrB5R+9LPFM6JNPCjWBr0Z3p+7xW638/u///s89thjIn3sG9/4Bj//+c9Jp9Mf2615q3tSkDJg7XiXzWZlNPUmaJMhJBINOf1549B6cpvNZjFpeGZmRrg2N5KCWLqbnVLcqTcaaeneGPmsXE8p3hNFUYSrRosjpNPp++bSK7qlK5FIJJsJVVWL1vjnwWvYKpFIJEVEiq5EIpEUkFv6dCUSiURyf5GWrkQikRQQKboSiURSQKToSiQSSQGRoiuRSCQFRIquRCKRFBApuhKJRFJApOhKJBJJAdlUZcBKl/IbwP8AVALdwNfVo+o7xV1VcVC6lE8A/xzYC1QB/416VP2Loi6qyChdyu8CnwXagSXgFPC76lH1clEXVkSULuU3gV8DGj76qBv439Sj6g+LtqhNxkfPzf8B/JF6VP0nxV7PprF0lS7lC8AfsHpzdgPvAm8oXUpdURdWPOzAZeC3ga0zBvjj8QTwH4FDwJNADviZ0qV4i7moIjMO/AtgD7AP+DnwfaVLebioq9okKF3KQeC/BS4Wey0am6YiTelS3gcuqkfVr635rB/4nnpU/d3iraz4KF1KAvgnD7qley1Kl2IHFoEX1KPqD4q9ns2C0qWEWT0B/HGx11JMlC7FBZwDfhU4ClyWlu5HKF2KidVj9E+u+dJPWLVqJJIb4WD1Gd56kzvvAaVL0Stdyt9n9ZT0brHXswn4f1k12t4q9kLWsll8uuWAHpi55vMZ4KnCL0dSIvwBcAF4r8jrKCpKl/IQq/fAAiSAF9Wj6qXirqq4KF3K14AW4MvFXsu1bApLVyK5W5Qu5d8Ch4HPqUfVB33Y2lVgF3AA+H+Av1S6lB1FXVERUbqUdlZjQ7+sHlWzxV7PtWwWS3ceWAEqrvm8Apgu/HIkmxmlS/l3wN8HjqhH1aFir6fYqEfVZWDgo7+eVbqU/cB/B/yj4q2qqDzK6um5W+kSAxz0wCeULuUfAzb1qLpUrMVtCkv3o4fmLPD0NV96GumbkqxB6VL+APgi8KR6VO0t9no2KTrAXOxFFJHvAw+xav1rf84A3/novz/e1MmPyWaxdAH+LfBNpUs5DZwE/jGr+anfKOqqisRHkfmWj/6qA+qULmUXEFaPqqNFW1gRUbqUPwK+ArwARJQuJfjRlxLqUbU4s1eKjNKl/Gvgh8AYq4HFX2Y1te7ZIi6rqKhH1SgQXfuZ0qUkWX13ip7TvSksXQD1qPpd4OvAv2I1OHIY+Ix6VB0p4rKKyT7g/Ed/yoCuj/77fynmoorMb7AqLG8CU2v+/PNiLqrIBIH/xKpf901gP/Bp9aj6RlFXJbkpmyZPVyKRSB4ENo2lK5FIJA8CUnQlEomkgEjRlUgkkgIiRVcikUgKiBRdiUQiKSBSdCUSiaSASNGVSCSSAiJFVyKRSAqIFF2JRCIpIFJ0JRKJpIBI0ZVIJJICIkVXIpFICogUXYlEIikgUnQlEomkgEjRlUgkkgIiRVcikUgKiBRdiUQiKSBSdCUSiaSASNGVSCSSAiJFVyKRSAqIFF2JRCIpIFJ0JRKJpIBI0ZVIJJICIkVXIpFICogUXYlEIikgUnQlEomkgEjRlUgkkgIiRVcikUgKiBRdiUQiKSBSdCUSiaSASNGVSCSSAiJFVyKRSAqIFF2JRCIpIFJ0JRKJpIBI0ZVIJJICIkVXIpFICogUXYlEIikgUnQlEomkgEjRlUgkkgIiRVcikUgKiBRdiUQiKSBSdCUSiaSAGG71RUVR1EItpJioqqrc6b+V9+TGyPtyPfKeXI+8J9LSlUgkkoIiRVcikUgKyC3dC5LSwGazYTKZ0Ov16PV6kskkmUyGfD5PPp8v9vIkEskapOiWMIqiYDKZePnll9m5cycNDQ14vV6+853v8NOf/pRwOEw4HC72MiUSyRqk6JYoOp0Os9mM1WqlubmZ3bt309bWRkVFBadPn8ZutxOPx4u9TIlEcg1SdEsMRVGwWq1YrVaeeuopmpubefLJJ2lra8NisZDNZslms+RyOVT1gQgUSyQlhRTdEkNRFMxmMzabjY6ODnbs2EFdXR0+n4/l5WUhuLlcTvpzJZJNiBTdEsNoNLJjxw6qq6s5ePAg27Ztw+v1oqoqoVCIqakp+vr6mJycZGlpqdjLlUgk1yBFt8QwGAxUVlbS1NREXV0dNTU1AKysrDA3N8fIyAizs7PEYrEir1SymdDpdCiKIv7cDFVVyefzqKq65d1TN7on2nWv/XO/kaK7QVitVsxmMysrK6ysrIij/72i1+vxeDz4fD4OHjxIZ2cnPp8PVVWZnJwkGo3y85//nFOnTtHX13cfr0RSyhgMBiwWC08//TQNDQ3U1tbicrlu+u9DoRB9fX0MDw9z8eJF4araKlwbE2loaCAYDGK324XIdnd3Mzg4yNWrVzfkXZKiu0FoflfNx5rP5z+W6Op0Oux2Oz6fj5aWFjo6OsSDEo1GmZqa4vLly5w+fZpkMnkfr2TzcTNLTbNKbmfJPSgoioLBYMBqtfLII4+wd+9eHn74YSoqKm76PWfPnuX48ePodDp6e3sBtpTo6nQ6LBYLTqeTQ4cOsWfPHtra2igvLxdW/k9+8hNOnTrF4uKiFN1SQVEUamtrqa6uJh6Pk0gkmJ6eJp1O3/XP0uv1uFwuvF4vL730Eg0NDbS3t+N2uwFIp9OcOXOGDz/8kP7+fpLJ5McS981ORUUFzz//PF6vF4/Hw8rKCoODgySTSRYXF1lZWcHpdGI0Gq/73kQiQSqVYnBwkFAoVPjFFwi9Xo/f78flcvHJT36SmpoaDh8+TG1tLTab7ZbfW1NTwyc/+UmMRiPT09MiRlDKQVlFUbBYLHR2duLxeNi+fTvl5eXs3buX2tparFbrhrsU1iJFdwNQFIVAIEBbWxuzs7NEo1ESicQ9/Sy9Xo/T6SQYDPLUU0/R3t5OeXk5ZrOZTCbD0tISPT09vPvuu0xMTJDJZO7z1WwuPB4PL774InV1dTQ0NJDNZjl27Bjz8/NMTk6SzWYJBoNYrdbrvnd2dpZwOEwmk9nyolteXk5lZSWf/vSn2bZtG9XV1UJwbyUqgUCAQCBAIpGgp6cHgP7+/oKse6PQrNuOjg7q6ur4xCc+QWVlJfX19TgcDhRFKZjgwiYTXc2qczqdPProo1itVs6fP8/8/Dxzc3P3ZCkWA1VVGRoaIh6Pi5LchYWFe/pZLpeLF198kaamJvGQAGQyGQYGBpidnWVgYIDx8fEt61ZQFAW328327dtpbW2lpqYGv9+P0WhEr9fT0dFBKpWira2NXC6H3W6/oaUbj8fJZDKMj49z8uTJIlzJxmK1Wnn44YcpLy9n9+7dBAIBWlpa8Pl8mEymu/pZXq+Xzs5OUqnULd01mx3NLef3+zl8+DAtLS3U19djt9vv+p7cLzaV6BoMBrxeL7W1tXzxi1/E5/PxZ3/2Z1y9epVEIlFyojs0NPSxf5bL5eKFF16gra0Nr9eL0WhkaWlJiO7w8DDDw8NMTk7eh5VvThRFwePxcOjQIVpaWq4LBrW1td3Rz9F6URw7dmyjllpUbDYbjz76KA0NDRw+fJjy8nICgQBms/muf5bX62XHjh1MT0+j05VuXyy9Xi9E99ChQ3R2dhZ7SZtDdMvKykSC/6FDh6iqqqKhoQGr1crOnTtxOp3Mz8+vOzqbzWYMBgN+vx+Px3Pdz3Q6nVRUVLCyskI2m2V8fJxTp06VTFDAbrezY8cOmpubKS8vx2azodfrWVlZYWZmhnA4zKlTp7hy5Qqzs7PFXu6GYTab8Xq91NfXc+DAAaqrq+9JRADC4TCRSIRoNHp/F7kJ0IJmLpcLt9uNy+USp6JsNksymWR5eZl4PC5cMFpc4EZop7NEIlHSwUft9Ox2uzEYNoXcbQ7RtVqt7Nq1i8bGRr70pS8RDAZxOp3k83keffRR6uvruXz5smjeoigKDocDm83Gzp07aW1tve5nNjQ0sGvXLrLZLIlEgnfffZdz586VjOi63W5+4Rd+gaamJiorK9e5FcbGxhgbG+NnP/sZ586dK/JKN5aysjKqq6tpb2/nySefvKVQ3I7Z2VmGhoaYn5+/fwvcJOh0OoxGIx6PB6/Xi9frxe12s7y8zPLyMvPz8yQSCcbGxojFYhw8ePCW9zKVSjEzM0M0Gi150fV6vfh8Pim6sFpd5XQ6qampYffu3dTW1uJ2u7FYLCJxuby8HKPRyKc//Wl27NghvtdqtWIymaipqSEQCFz3s/1+P16vV/j4PB5PSfimysrKqKyspLGxkYceemidZbeyssLS0hIDAwP09fVt6QIIs9mMw+GgqamJz3zmMzQ3N6+zcFVVZWlpieXlZSYmJlheXsblcgnL+Np/q50Q+vr67tm/vlkxmUwEg0Hq6upobGykvr5e5IjPzc0Ri8U4fvw409PT1NbW4vF4bitAuVyOVCrF8vJyga7i/qKdFD0eD3v27KGqqgq73V7sZQFFFl2LxUJtbS0dHR186lOfIhgM4vF40Ov1QiBra2upqalh27ZtN/wZNxPSaytvrly5UhK+KafTyb59++jo6ODJJ5/E6/Wi1+tRVVUcE0+fPs2HH3645cRjLTabjbq6Oh599FG+/vWvY7fb0ev14uv5fJ54PM7i4iLHjx8nFovR0tKCx+PhoYceWie6motpaGiIU6dOMTExUYxL2jCsVisdHR20traye/duqqqqUBSF5eVlcSr60z/9U3p6evhn/+yfUV9ff1vRzWazxOPxkomjXIvP5+Nzn/sctbW1wkXp9XqLvSygiKKr0+nwer089thjNDY2Cgv3WrHU/nvtC3cnaFVg8Xic6elphoeHWVlZua/XcD8pKysjEAhQW1vLrl27hLWi0+lQVZV0Os2HH37I5OQkoVCI2dnZkrVC7gatqERzC2n3Ih6Pc/r0aebm5jh79iyZTAan04nBYLjOhbSwsEA0GmV0dJSRkZEt59O1WCw0NzfT2NiI2WwW78zKygpTU1OMjo6Sy+Uwm81CfCwWyw1/lrZBRSIRRkZGmJ+fL0n3gsViWVeBV1ZWJt6lWCxGLpfD6XRiMpnE/dJOTmNjY1y6dGnDYiVFEV2dTofJZKK+vp6vfe1rVFRUCIvufrG0tMTi4iL9/f289dZb9Pb2bmp/rtfr5eDBg7S1tfHiiy/i8XhErunKygrRaJS//uu/pre3lzNnzrCwsFCSL8OdorkEtI1TURRR4TczM8Po6Ch//Md/TCgUYnx8HKPRSEVFBUajcd1mpDUC6u/v54MPPuDMmTNb7r7Z7XYOHz5MQ0MDZWVl4vNsNkt3dzfd3d3kcjm8Xi/V1dXU19fftEgik8mQTCYZGRnhgw8+IBKJlGRhhN1uZ//+/dTU1Kwz4rLZLBMTEyLFUDP0AJLJJLFYjDNnzvCDH/xgw667oKKr1YF7vV7a29vZsWMHbrcbq9WKTqcjl8sxNzfHysqKuBGaS2BpaYlcLieisOl0+paWXjqdJpFIMD4+Tm9vL2NjY5vy4TEYDJjNZvx+Pzt27KC+vh6n0yl25nQ6TW9vLxMTEyI1LJPJbDnhuJZsNsvi4iIjIyP8/Oc/x+Fw4PV6yWazTE5OMjU1xeTkJIlEAqfTidPppLq6murqamHFpVIpMpkM/f39XLhwgZmZmU35DHxcDAYDdrtduGDWul6MRiMul4v29nby+Tw+nw+j0Xidq03rsTA3NydcErFYjHQ6XTLPmk6nEzGRuro6TCYTOp2OfD5PLpcThTHaKCutGEJLJZycnGRiYoL5+fkNNdAKKrpWq5XKykr27dvHb//2b+Pz+QgGg8K/lEqlOHPmjJh4oDWnUBSFqakp4vE4Q0NDLCwsMDIycstRNCsrK+RyOTKZDIlEQvx9s1FWVkZFRQU7d+7kC1/4Ah6PB7fbLQKJCwsL/Mmf/AmDg4OcOXNGlLpudbSS3YmJCU6cOIHD4aCxsZHl5WVGRkbIZDKk02ksFgsHDx6kvr6eJ554gra2NsxmM6qqMjMzw/z8PK+//jqvv/56yfonb4eW366dFrPZLKFQiEgkgtvtpq2tjcrKSpxOJ21tbWKe3lpSqRSxWIzz58/z1ltv0d3dzdTUVMlsUlr2RmVlJc899xytra2UlZWJE1MymRTa4vf7Rd+SfD4vMjxOnz7NqVOnGBgY2NC1FkR0NT+tw+EQfpby8nKcTue6X75erxf5dNpOrPno5ubmhH8qEokwNzd3S9+cdrNzudym7Cur1+sxmUxUVFSwZ88e2tvbcblc4rijPSjhcFhYdZlM5oEQXA3NAslmsyL7IJfLEYvFyGazqKqKXq+nsrKSmpoa7HY7ZrNZvEgjIyOEQiGmp6dJpVJb7t7p9XrKysqw2+3YbDaR9QOrBs7KygrV1dUsLS0RCASw2WzYbLZ1Vq5m4U5PTzM2NsbAwAChUIi5uTlhDZYCBoMBh8NBeXk5ra2t1NbWYjAYhA7k83kURRHFEg6HQ3w9FosRj8eZmJhgdHR0w7OCCiK6mg+3sbGR559/nsbGRioqKtY5sWH1QTlw4AArKyviWBCNRolGo3zzm9/k9OnThMNh8QLdyS68WR8am82G3+/nk5/8JF//+tdxu934fD4URSGfz5NMJrly5Yo4Gk9MTGxKS71QrO2XoImnXq/HZrPxyCOPiFaXOp2OeDxOPB7n1Vdf5cSJE4RCoS3ZBKisrIyGhgYaGxtFkZBOp0Ov19PY2IiqqnR2dorNSROdtaIbi8VEStmbb77JwMAA3d3drKysbNp350bYbDZaWlrYtWsXzz33HG63W1i6y8vL5HI53G63SEN0uVzCFdPf308oFOLEiRO88847G/6eFUR0NYGprq4WdfMGg+G6dC9tuu3ahsKa6W8ymTAYDCK4UspoQaHKykqCwSB+v19YINlsVkzxvXLlCsPDw6RSqXWicW3zZa06T0Pr3XunG1MpoFksGkajEb/fT1VVFcFgUORzq6rK4uIiCwsLzM3NMTs7uyWbAGlNXLTnR/NfamjPw416UACiz7PWKGh0dJTx8XEWFhZK8n7pdDoMBgMmk4mysjLKyspEIxvta06nk5WVFSwWy7r3JZVKsbi4SCKRKMipuCCi29TUxKc//Wk6Ozt5/PHHMZvNd5SpoCgKTqcTnU5Ha2sr8XicVCpV0lNu11oin/nMZ9i+fTtut1u8HAsLC7z11lsMDAzwzW9+k4WFhXXHHa1NnfaAGQwGGhoa1pVCj42NMTs7SyqVIp1Ol9Qx8U5xOp08//zzNDU1sX//fiorKzEajSJirzWhnpyc3LJuhWAwyJEjR2hsbFyXtXAnJBIJEokEb7/9Nu+88w7d3d2bPsPnXtA2J21yNqzfiFRVJRwOMzk5WTCff0FEV8tY0DIVNMHN5/MsLS2tC3JpkVXNP7V2B9PyVksZq9Uq+kJovm3N6s/n82QyGWZmZpienmZubo7FxUVg9eGx2WyYzWbKy8uxWCxYrVaMRiP19fVCdFVVxWq14na7RT/UbDZb8qeDtWjj54PBoKg0MplMLC0tkUqlmJqaYmxsjEQisaVERAsW2e12gsEgDQ0NoiLzdkZMLpcTcYLl5WVmZ2dFdsj4+DiRSKQkLdw7QcvPvbarWDKZJJVKMTc3d8/9ru+Fgomuz+fD6XSu+zyRSNDX10cikWBqagqdTkdNTQ0Oh4PW1tYb9kQtdTo6OnjiiSfYs2cPzzzzjDgWrqyskMlkmJ+f5+zZs4yNjYmjjpbFcfjwYaqrqzly5AiVlZX4/X7Kysqui0ZrxQTf/va3+Zu/+RsRhNwK6HQ6rFYrXq+Xffv20dzcjM1mI5fLEQqFmJmZ4Yc//CHnz59nbm6u2Mu9rzgcDqqrq9m2bRtf+MIXqKioYNu2bVgslpsWO8CqcTM3N0c8Hufdd98lFArR29vL5OQk4+PjzM3NbalN+U7IZrOcP3+e0dFRXnvtNc6dO1ew1qgFEV3Nok0mkyLjQMubC4VCxONxJicnMRqN6+aKrW0unMvlyGazJeuj1Ov1ovlGfX09lZWVuN1ucY2ZTIbZ2VmR5hSNRsnn8yKjw+12i8yPpqYmqqqqhMWrca2PvLa2lkAgQCqVKvTlbhhGo5Hy8nKCwSBerxeXy7Uux3tycpK5uTkWFhY2ZdbKvaC9Fx6Ph5qaGurq6mhqasLn84my+WvRMj9isRhLS0tMTk6yuLjI8PAwAwMDDA0NMT09TTQa3RJ9mDWjRSvuMJvNwq8LrIsTaXm7mj97bm6OSCRSsLUWRHQnJyc5fvw4Xq+XS5cuiaKFSCRCb2+vCJa5XC6eeuop6uvr6ejoED1TVVVlenqaUChUsg9IeXk5fr+fvXv38sQTT4hMBS2XuKenh29961uMjY1x9uxZETzz+/189atfpaGhgUOHDuHxeHC5XMIvtby8LF6wawNq27dv56WXXuLNN98kFAqJf1fKVFZW8lu/9Vs0NjbS2tqKw+Egl8sRjUZ5/fXXhT83mUxuGT92XV0de/bsoaWlhSNHjlBeXk5LS4to4n4tWnFENBrlP//n/8zw8DBzc3Mkk0mRv5vJZNaVV5c68XicK1euAHD8+HHq6urYv38/ZWVl6555VVVF5dm5c+e4cOFCwU9EBRHdVCrF9PQ0yWSSpaUlYrGYqIEfHBwkl8uhKAqZTEZ0NtJeGG1n0nawUguKrM1RrqmpIRgMEggEROAjl8uRSCSYm5ujv7+fqakpUQBhsVhwu920tLTQ3NwsJkdoJwHNitHukZanajKZRCVSfX39uhLrUhZdLUVM6/6v9RheXFwkHA6LPMtkMlnS16k9M9rpyOPx0NjYSEtLCy0tLSIv91YDOjOZDPF4nIGBAa5cuUI0GiWdTjM3N7elTj4amr9aG9JqsViEVminSQ3t1D0/P1+U7JaCiO7CwgIXLlwQBQGadac597WXqby8nD179tDU1ITVaiWfz5NKpcTUiFIsDrBarZSVlfH000/z8ssvU1VVJSrOAGZmZnjvvff48MMPuXDhggj+eDweHn/8cZqamjh06BCVlZUi4X1sbIxIJMKPfvQjRkZGRGpYS0sLFRUVHDhwgO3bt1NdXS1OFw6Hg3Q6XbKWjcViwe/3U1dXR3NzM1VVVZhMJuLxOD/4wQ8YGhrizJkzjI6OlnzlmdVqFWmWlZWVPPbYY7z00ks4nU78fv+6Lnw3IpvNMjw8zNjYGD09PfT29grX3Fb33UajUY4fP87c3BxPPPEENptt3QasVetNTExw5coV+vr6Cv683BfR1VKXtB06n88LIdB+0bfyr2nBEafTSSAQEHm8moWrWb/aKPNSQus3UVVVxfbt20XGgZZ3GovFGB0dZWJigkgkwvLystiEGhsbaWpqIhAI4Ha7hW88EokwOztLb28vV69eXbcRaZYzIPIVtZlhpfzCmUwmfD6fqGS02Wyip+7IyAgDAwOiUXepor0/WqCwsrKShoYGGhoaaGpqwmg0CveRdrrRYh9r06BWVlaIRCLMz88TiUREBsxWR9OeWCwmSv/Xnpi1ftTz8/OiQXsx0k8/tujqdDqOHDnCQw89JAIboVBIRJLHx8eJxWK37NbvcDh49NFHaWpqYtu2bQSDQcxmM9lslosXLzI2NkYoFGJ+fr7kKou0dDebzSaqYGC1EmhmZoYzZ87wt3/7t+LanE4nHR0dtLS08LnPfU5MjVhZWRGThX/84x8zPDzMBx98wMzMjHApaHX2NxpfVKpo6WGtra18/etfp66uDq/Xi6qqInB26dIl+vr6Stbfr+FwOLDb7TzzzDM8/fTT+Hw+AoEAXq93XfGD1nktEolw5coVbDYbBw4cEBtRMpnk2LFj9Pb23rI/yVbCZDLhcrlobm7ml3/5l6mrq1vXSS2bzTIyMsLs7Cx/8zd/w9WrVxkfHy/KWj+26CqKQn19PXv27KGyspLy8nLRud9kMpFMJm95pNWS/evq6sScNC21TKsJ1/y/pZhHqCiKyK9cmye4vLxMLBZjbm6O4eFh0uk0iqKILkm1tbW0t7eL0latgbnW53RgYIBwOCx2ar1ej9VqxefzrctoKPVgkl6vx2w24/P52LdvH4FAAIvFQjabJRaLEQ6HmZub2xJpT2VlZcKHf/DgQVwu17oNdK21Fo1GmZmZob+/X6TPab/rbDbL2NgYQ0NDJe9quRO0SlaXy0UwGGTbtm2izScgTtvz8/NMTEzQ29vLlStXinZv7ovoNjc38+ijj4rKD5vNRkdHB+fOncNkMnHlyhXGxsauEwAt2FNbW8uhQ4eora0VfkvtZXrjjTe4dOkS09PTH3epmwqt41pbWxuHDx8mmUySSCTwer3s3LlTtKbTfHfacE3tPjidTjo7O8nlcuzYsYOqqioef/xxOjo6xFgSLflbaxBTav5wWL3Obdu20d7ejsfjwW63o9PpRH/hoaEhQqEQsVisJK8P/m5jPnDgAIcPH2b//v1ixPxawuEwg4OD9Pf389prr4nTUkdHB08//bSw7LQRRuPj41smbe5maGOd2tvbeemll6ipqaGlpUUUYS0tLYmmR9/61rcIhULCyCnW83JffLra2HQNzVKNxWIMDAwwNTV1XQRRm15qt9vxer00NDRQVVWF0WgUKS8LCwv09vZy6dKlkvPl3g5tZ/b7/TQ3N4s0FrfbTXV1tagy0kRX81UtLi4KH7jVasVgMLBnzx4R3a6oqBD/H1qmiBZAK8V7qA2mDAaDYi6eqqqkUik+/PBD+vv7RZ/UUkWrumxoaGD//v00NDRc12RcVVUSiQSjo6NcvnyZH/3oR6Lxk3YS0t4vLVaw1SZk3AhNQ6qrq3nsscfw+XzrhlBms1lmZ2cZHR3lgw8+EL1MirlB3xfRnZiYoLu7m2AwiM/nu+2/t1qtBAIB6urq+KVf+iVqamqora0VTZjj8TgnTpxgcHCwZMeFaGiZGrFYjIWFBSwWi0h1slgstLa28tJLL5HNZslms2KwotVqXTfny2KxiMnH27dvF75tRVEIBoPY7XYx3TWZTJJOpzl27BjHjx/n4sWLRX/Q7paysjJh9X/pS1+iqqoKi8VCKpVicHCQoaEhRkdHmZmZKTk//1oURWH//v20trby2GOP0d7eLk4qmjshEokwNTVFd3c3r7/+uqhWVFUVo9GI0WhcN+bKYDDg8/nw+/1Eo9GSvj+3o7y8nAMHDrBz504R/9Ca/8/MzDA1NcV3v/tdRkdHmZqaEg3Mi8nHFl1VVYlEIoyPj2O329eJ7tpqkLUPhcViIRAI0NzczFNPPSV2Jy3ItLS0RG9vL729vSQSiZIX3Ww2K475gJiUodPpCAQCN5xmfC0Gg4H6+vo7uheayHd3d/OTn/yEcDhccv5OzY/b0NDAY489JjIwkskkY2NjjI6Oisq9Uk2Dg1Urt7m5mYMHD9Le3k4wGBRf0yqnFhcXGR0dpa+vj7Nnz7K4uEg2mxVZC9d27FvbMzaRSGxZ0dXy31tbW0U+uhY30SoUR0dHee+99xgfH980z8p9Ed3JyUl6enrw+/3U19eLr1VWVnLo0CHq6+tpb28XguF2u2lqahKt+bTRNMlkkgsXLoijwMjISMlHpJeWlsjn87z99tuEw2EaGhpobW2lurpadPG/2w5RN0PLbnjnnXc4d+6cGK5XSkdvo9GIxWKhs7OTF154QUwA0I6L4XCY1157jVAoRDgcXnesLjW01LeOjg727dsnXENaOlh3dzdnzpwhFApx6dIlZmZmmJ2dFVaux+Nh3759dHZ2indIE+qFhQUWFhZKbrO9U7Zv384nPvEJmpqaOHjw4HVjiObn5/nLv/xLRkZGmJiYEClkm4H74l6Ym5tjaGiI3bt3r/tc6/PZ3NzMzp07xcvhcrlobGwUFTca6XSaixcvMjg4yJUrV7ZE8ExzG5w9e5bu7m62b9/Ovn372LVrF1VVVaLj/62S3e8ErUXd+Pg4b7/9Nj/84Q9Jp9MlF0gxGo3YbDbRDlTLhNFepsXFRY4dO8bY2JhoW1mKKIqC3W7H4/HQ0NBAZ2ensNK0/NuhoSF+/OMfMzg4yPnz56/bXBwOBzt37qSlpUVMAdZcEolEYkvn5zY1NYmUyra2tuu6D0YiEf7rf/2vRUsLuxX3xdIdHx8nn8/T2tqK1+uloqKCQCCAyWTC6XRisViEnwoQ/XS1CQnhcJgTJ04wPT3NyZMnmZubK+kk9xuRy+VIp9OMjo6KnMGenh4MBoNoxdjc3Cx8dGtbOK5Nf7kVc3NzhEIhFhYWSrb6zOVy0dbWRkNDA+Xl5aLcNRKJcPHiRXp6esSRuVQtXFh1K2zbtk2MltEyVXK5HN3d3fT09HDq1Cm6u7sJh8PXXatOp8Nut4uSaL1ej6qqZLNZcbraSmjvRVtbG7t27WLXrl1UVFSIoL1WiBWNRunp6aGnp2fTGhz3RXSnpqaIRCI0Nzfj9Xp5+OGHRcrLtT0s15LL5YjH4wwPD/Otb32LiYkJQqHQpnB232+0WVTapFWtJBpWX6D6+nqefvppysrKsNlsOBwOOjo68Hg84uh0K1RVZWFhgeHhYcLh8KZ94G7HWtHVrluz4o8dO8bQ0BDxeLzkRVev17Nt2zYOHDhAdXU1RqNRPCOXL1/m1Vdfpa+vj56enuu+d+2sr7Ul0WtH02y198doNGK1Wtm9ezdf+cpX8Pl8VFZWrjsdaLm4WhB+s7rV7ot7QXsBLl26RDqdFhVkBoPhpmKRz+dZXFzk6tWrogBCCxCU8st0p2hWiRZgXFhY4OzZs8LKNZvNdHd3i2IJ7T7e7N6oqkp3dzfj4+Ml2TtXq9jbtm0bjz/+OI2Njeh0OjKZDJFIhFAoxNmzZ5mcnBQ+zVJGURRqamro7OwUWSfT09PMzs5y9epV+vr6rut+ZTAYxPTovXv30tbWRk1NjejlkU6n+fDDD0VaVCliMBgwGAyi8lAjGAxSUVHB/v37RT7/2gk04XCYS5cuEQqFeP/995ment60/uz7Irpaa8YzZ85w5swZOjs72bZtmyh/vRatL8P09DTHjh3bEi/R3XJtm8WZmRlmZmbW/Zu79fOW8j10uVyiheGzzz6LxWIR6YNjY2NcvXqV48ePb5ncU51OR1NTE3v27AH+zk13+fJlLly4wMWLF6/7HpPJhMfjYfv27Xz1q18lGAzS2NgoNuRkMsmpU6cYHBwsWfecyWTCYrHw8MMP09nZKT7v6Oigra2NQCBAdXW16LOtMTs7y09/+lMGBgZ48803N3Ul3oZ0GYtEIgwODooWgzdCS+DO5XIlLRYbyYN0XwKBAHv37qWhoUFMwshms8zPz3Pu3DnRKUtDp9OJhj7Nzc2YzWYxan1+fn7THi01tAGas7OzOBwO0UWttbWV3bt3E4lEWFpaIpPJYDKZhOA2NDTQ3NxMTU0NTqcTvV7PysoK8XhcBLSHh4dL1r1kMBgwm800NjaKDQkQw0c1Iy6VShGJRJibm6Ovr4+RkRGRrbPZYxkbIrpTU1N3nHnwIAmL5OY0NjbyqU99SlRjaUUlY2NjvP7660xOTq47LmpTOPx+P5/97GfxeDycPHmSyclJzp8/XxKiOzs7y/DwMI2NjVgsFjEZxGQyEQgEiEajhMNhUbVZVVXFzp07cTgcVFRUiIh9Op0WTf7Pnj1bsqmW2mRrm83G7t27+fSnP73ua2tz/aPRKFevXuWDDz7gr/7qr0QF69p5i5uVDeunK8VUcid4PB5RBl5dXY3L5UJRFFKplAg6Tk1NsbCwsM4dU1ZWxu7du6murqa9vR2XyyXa9qXTaTH7a7OKj6qqzM/PMzY2JtqZaiIaCATo6OggmUwSj8dF9o/WxU9LMdRG1MzNzXHy5Ml1UyFKPZCm9aO4lnQ6TSqVYmBggFOnTokG7Vp/7lK47oI0MZdIbkZ9fT27d+9m//79dHZ2iiKIhYUFzpw5w/nz5+np6SGdTq/byN1uNy+//DKtra1s27YNq9XKrl27iMfjmEwmLl++zLvvvrtpRTefzzM0NITFYqG+vp6GhgaRldDU1ERDQ8N1vUqutfa0vgI9PT184xvfYHJysiTbn94NWpe948eP8+d//ueixWUpGXlSdCVFJRAI0NnZSTAYFI3rtdSfq1evMjY2dlO/vxaM1BrGWCwWVFWlqalJBGrz+TzhcHjTia+Wamk0GnnnnXeuC6LeCVpD+9HRURYWFkpynNVatJS3ZDLJuXPnbjgNfGFhgdnZWa5cuUI8HieTyZSU4IIUXUmR6ezs5LOf/Sxutxu9Xk86nSYSidDT08Nrr71GJBK5oeWmBWLXtnS0WCyYTCaOHDnC3r17RdrdyZMnGRgYKPSl3ZKVlRUxGPGNN9644YDJ26FF8LUJK9dG9EuRWCxGPB7nD//wD/njP/7j676ubbQrKyvrZimWElJ0JUVlampKWDU2m000Burp6SEajd4033RpaYn+/n4xNdnhcACrQhSPx0W++Pz8/KaN5GsBn82aT1oMtI0jnU5v6rSvj4Nyq51CUZTS20buAVVV7zghVt6TG3Ov96WsrAyr1Sp8lZrlpnVmy+fzN7RmtJ7Cer1etDZcs3YxP03r8na/jt3yWbkeeU+u51b3RIou8qG5EYUS3VJDPivXI+/J9dzqnlyfkyGRSCSSDUOKrkQikRSQW7oXJBKJRHJ/kZauRCKRFBApuhKJRFJApOhKJBJJAZGiK5FIJAVEiq5EIpEUECm6EolEUkCk6EokEkkB2RQNb5Qu5feAo9d8PKMeVYNFWM6mQelSKoF/DXwGcABDwK+rR9VjRV1YkVC6lBBQf4Mvva4eVZ8t8HI2BUqXogd+D/gyUAlMAd8Cfk89qm7uEQobjNKlOID/FXgRCADngd9Wj6ofFHNdm0J0P+Iq8MSav5duY9D7gNKluIGTwAngWWAOaAJmi7isYrMfWNsDsRI4C/x1cZazKfgXwG8C/wC4BDwM/CWwxKrgPMj8Cav34x8A46xuTD9TupRO9ag6UaxFbSbRzalH1TsbrPZg8DvAlHpU/ZU1nw0XazGbAfWoum4mudKl/CMgxoMtuoeAH6hH1R989PeQ0qW8Bhwo4pqKjtKllAGfAz6nHlXf/ujj31O6lOeAXwf+VbHWtplEt0npUiZZ3aHfB/6lelQdKvKaiskLwI+ULuW7wBFgktWd+4/Uo7J2W+lSFOAfAf9JPapuzcard8YJ4DeULqVDPar2Kl1KJ/Ak8PtFXlexMbB6Krp2QmkaOFz45fwdmyWQ9j7wD4FPAV8DgsC7SpfiK+aiikwT8Bus+nGfAf6AVf/ubxZzUZuIp4FG4P8r9kKKzP8JfBPoUbqULNAN/KV6VP2PxV1WcVGPqnHgPeBfKV1KtdKl6JUu5cvAo6y6pYrGprB01aPqG2v/rnQpp1gVm38A/NuiLKr46IAz6lH1dz/6+3mlS2llVXT/sHjL2jR8DfhAPap+WOyFFJkvAL8C/DKrgrsL+AOlSxlWj6p/WsyFbQK+AvwZq/7cFeAc8G1gbzEXtVks3XWoR9UEqw9Qa7HXUkSmgJ5rPrsC1BVhLZsKpUsJAM8jrVyA/wv4N+pR9TvqUfWSelT9JquGyu/e5vu2POpRdVA9qn4SsAO16lH1EcDIqkFXNDaFpXstSpdiATqAt4q9liJyEmi/5rM2YKQIa9ls/ENWff/fLvI6NgNWrs/0WWGTGlTFQD2qJoGk0qV4WHXV/U4x17MpRFfpUv4N8ANglNV8uv8ZsLGa+vKg8u9Y9Wv/T8B3gd3AbwH/sqirKjIfBdB+FfjORyeiB50fAP+j0qUMs3o63A3898BfFXVVmwClS3mG1c2nF2hh9VTQC/x5Mde1WXbDGlatlqvA37JqxRxUj6oPrFX3UQL3C8DngcvA/87qZvRAB0hYzeVuRboWNP4p8D1Wn4srwP/N6r35n4q5qE2Ci9X4Ry+rm9AJ4Bn1qJot5qLk5AiJRCIpIJvF0pVIJJIHAim6EolEUkCk6EokEkkBkaIrkUgkBUSKrkQikRQQKboSiURSQKToSiQSSQGRoiuRSCQF5P8H1GVNWnd/AfoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    ith_number = np.where(y_train == i)[0]\n",
    "    random_index_number = ith_number[np.random.randint(0, (len(ith_number)))]\n",
    "    plt.subplot(2, 5, i+1)\n",
    "    plt.title(f\"{y_train[random_index_number]}\", y=-0.4, fontdict={'fontsize':14, 'color': 'green'})\n",
    "    plt.axis('off')\n",
    "    plt.imshow(x_train[random_index_number], cmap='gray')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Any thoughts on the data?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Prepare the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": true
   },
   "source": [
    "**What does our data look like, raw?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train:\n",
      "5\n",
      "x_train:\n",
      "[[  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   3  18  18  18 126 136\n",
      "  175  26 166 255 247 127   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  30  36  94 154 170 253 253 253 253 253\n",
      "  225 172 253 242 195  64   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0  49 238 253 253 253 253 253 253 253 253 251\n",
      "   93  82  82  56  39   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0  18 219 253 253 253 253 253 198 182 247 241\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  80 156 107 253 253 205  11   0  43 154\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0  14   1 154 253  90   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0 139 253 190   2   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0  11 190 253  70   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0  35 241 225 160 108   1\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0  81 240 253 253 119\n",
      "   25   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0  45 186 253 253\n",
      "  150  27   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  16  93 252\n",
      "  253 187   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 249\n",
      "  253 249  64   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0  46 130 183 253\n",
      "  253 207   2   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0  39 148 229 253 253 253\n",
      "  250 182   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0  24 114 221 253 253 253 253 201\n",
      "   78   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  23  66 213 253 253 253 253 198  81   2\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0  18 171 219 253 253 253 253 195  80   9   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0  55 172 226 253 253 253 253 244 133  11   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0 136 253 253 253 212 135 132  16   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]]\n"
     ]
    }
   ],
   "source": [
    "print(f\"y_train:\\n{y_train[0]}\")\n",
    "\n",
    "print(f\"x_train:\\n{x_train[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary> Click here for a more clear example  </summary>\n",
    "<p>\n",
    "    \n",
    "<img src=\"images/mnist_array_example.jpeg\" width=\"500\"/>\n",
    "<a href=\"https://towardsdatascience.com/how-to-teach-a-computer-to-see-with-convolutional-neural-networks-96c120827cd1\">[Image Source]</a>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is the shape (dimensions) of our data?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We would like to add another dimension to our images\n",
    "In other words, converting the shape from (28, 28) to (28, 28, 1).\n",
    "\n",
    "This will describe that our image has _one_ channel, and is grayscale. \n",
    "> FYI: A colored (RGB) image has 3 channels (e.g. (28, 28, 3))\n",
    "\n",
    "We do this in order to be able to use **Keras' Conv2D layer** which expects the format of the data to be: `(batch_size, height, width, channels)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Hint</summary>\n",
    "<p>\n",
    "        \n",
    "[Link to Numpy docs (np.reshape)](https://numpy.org/doc/stable/reference/generated/numpy.reshape.html)\n",
    "    \n",
    "```python\n",
    "data.reshape(NUMBER_OF_IMAGES, HEIGHT, WIDTH, CHANNELS)\n",
    "```\n",
    "    \n",
    "</p>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.reshape(x_train.shape[0], 28, 28, 1)\n",
    "x_test = x_test.reshape(x_test.shape[0], 28, 28, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28, 1)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize the data\n",
    "The pixel intensity in a images is in the range between `[0, 255]`. We would like to normalize this to `[0.0, 1.0]`.\n",
    "> Artificial Neural Networks processes the input data by updating its so called weights. This process can be disrupted by large integer values, and the training process can be slowed down."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize(data):\n",
    "    data = data.astype('float32')\n",
    "    data /= 255\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = normalize(x_train)\n",
    "x_test = normalize(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-Hot Encoding\n",
    "Most machine learning algorithms are not able to train on labeled data, e.g. \"cat\", \"dog\". This needs to be converted into a numerical representation. This step is called **integer encoding**. However, by just converting \"cat\" to 1, and \"dog\" to 2, we are making an unjustified assumption that there exists a **natural ordinal ordering**. This would be the case if our data were, e.g.,  $\\{male,\\ female,\\ child\\} \\rightarrow \\{0, 1 , 2\\}$ as in the Titanic data set used in the [Intro to Data Science Course](https://github.com/BRAIN-NTNU/data-science-course/blob/master/%5BSolution%5D%20Data%20Science%20Course%20-%20Titanic.ipynb).\n",
    "\n",
    "Therefore, we need to **one-hot encode** our data. This way we can represent categorical data, in a numerical way, without assuming anything about its ordering.\n",
    "![image-2.png](images/one_hot_encoding.jpeg)\n",
    "[[Image source](https://medium.com/@michaeldelsole/what-is-one-hot-encoding-and-how-to-do-it-f0ae272f1179)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000,)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Hint: </summary>\n",
    "<p>\n",
    "    \n",
    "https://keras.io/api/utils/python_utils/#to_categorical-function\n",
    "    \n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Hint 2: </summary>\n",
    "<p>\n",
    "    \n",
    "```python\n",
    "    data = to_categorical(data, num_classes=NUM_CLASSES)\n",
    "```\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = to_categorical(y_train, num_classes=10)\n",
    "y_test = to_categorical(y_test, num_classes=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 10)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAAD5CAYAAADcKCLLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAG4UlEQVR4nO3dwYuVex3H8e9jN5DrQk0JEUKhFpcoE7yrGkhoEA0XRiVEC11Vi9wJsxEOZycIgYhzb7UYs1WEkH+AVyUYXLSOEUI08MJICxdCJLf5tXAWdpnzzNyZOXM+47xeu3O+nOf5gbz9zcxzznO61loBeXZNegHAysQJocQJocQJocQJocQJod6b9AIYv27YfauqLlXVh1X17ar6chu0brKrYjV2zp3hRFX9sKr+WVV/m/BaWCNx7gx/bIP2tTZoP6qqTya9GNZGnIG6Yffjbti1bth9Z4XZg27YPfoix2uDtrR5q2OriDPT3ar6tKp++faT3bD7oKq+X1UfLz9+0A27p1u+OraEOAO1Qfusqn5fVT/vht2et0a/qKqXVfWn5cf/rarPtnZ1bBVx5vpdVb1fVT+rquqG3e6qulBVt9ug/buqqg3aD9qgfWNyS2ScxBmqDdqn9ebH218tP/XTqvpKVf12YotiS4kz22xVneiG3Yl68/vnX9ug/X3Ca2KLiDNYG7RPqmqhqn5TVd+r5T8EsTN4h1C+j6rqelX9q6ruvD3oht29qjqy2u+d3bB7v968CaGq6oPl536y/PhpGzRvTAgkznx/rjdx3mqD9p/Pzb5Ua/s3/OrycT5/3KqqP1TVxY0skPEQZ76zVdVqhT8EtUE7uZYDtEF7WlXeS7vNiDNUN+y+WVVfr6phVf2lDdo/Jrwktpg4c81W1Xerar6qfj3htTABnbvvQSaXUiBU74+1XdfZVmHMWlv5g+92TgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgglTgj13qQXwLvjypUrvfPhcNg737Vr9F5x8uTJ3tc+fPiwd74d2TkhlDghlDghlDghlDghlDghlDghlOucrNnFixd75zMzM73zpaWldZ+7tbbu125Xdk4IJU4IJU4IJU4IJU4IJU4I5VIKa3bkyJHe+e7du7doJTuDnRNCiRNCiRNCiRNCiRNCiRNCiRNCuc7J/5menh45u3Tp0oaOvbCw0Ds/e/bsyNni4uKGzr0d2TkhlDghlDghlDghlDghlDghlDghlOucO8zU1FTvfG5ubuRs7969Gzr3tWvXeufPnj3b0PHfNXZOCCVOCCVOCCVOCCVOCCVOCCVOCOU65w5z4cKF3vnhw4fXfewHDx70zm/fvr3uY+9Edk4IJU4IJU4IJU4IJU4IJU4IJU4I1bXWRg+7bvSQSAcPHuydr3b/16WlpZGzly9f9r72/PnzvfP79+/3zneq1lq30vN2TgglTgglTgglTgglTgglTgjlI2PbzNGjR3vnd+7cGdu5b9y40Tt3qWRz2TkhlDghlDghlDghlDghlDghlDghlOuc28zp06d758eOHdvQ8e/duzdydv369Q0dmy/GzgmhxAmhxAmhxAmhxAmhxAmhxAmh3BozzLlz53rnt27d6p3v2bOndz4/P98777u95Wq31WR93BoTthlxQihxQihxQihxQihxQihxQiif55yAvnvPjvO+s1VVT5486Z27lpnDzgmhxAmhxAmhxAmhxAmhxAmhxAmhXOecgJmZmZGzpaWlsZ776tWrYz0+m8fOCaHECaHECaHECaHECaHECaFcShmD48eP985PnTo1tnPfvXu3d/748eOxnZvNZeeEUOKEUOKEUOKEUOKEUOKEUOKEUL4CcAxevHjRO9+/f/+6j/3o0aPe+ZkzZ3rnr169Wve5GQ9fAQjbjDghlDghlDghlDghlDghlDghlM9zjsGBAwd65xu5/eXs7Gzv3HXMd4edE0KJE0KJE0KJE0KJE0KJE0KJE0K5zrkOc3NzvfNdu8b3f978/PzYjk0WOyeEEieEEieEEieEEieEEieEcillBat9hd/09HTvfLWPhL1+/Xrk7ObNm72vXVxc7J3z7rBzQihxQihxQihxQihxQihxQihxQijXOVewb9++3vmhQ4c2dPznz5+PnF2+fHlDx+bdYeeEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUD7PuYKFhYXe+Wpfwzc1NbWZy2GHsnNCKHFCKHFCKHFCKHFCKHFCKHFCqK61NnrYdaOHwKZorXUrPW/nhFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFDihFC9t8YEJsfOCaHECaHECaHECaHECaHECaH+B1S/E1AerEUeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "random_number = np.random.randint(0, len(x_train))\n",
    "\n",
    "number = np.where(y_train[random_number] == 1)[0][0]\n",
    "\n",
    "plt.axis('off')\n",
    "plt.imshow(x_train[random_number], cmap='gray')\n",
    "plt.title(f\"y: {number}\", fontdict={'fontsize':16, 'color': 'green'})\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Let's define a model\n",
    "\n",
    "We will define a Vanilla Fully Connected Feed Forward Neural Network. This might sound very complicated, but in should not discourage the upcomming machine learning apprentice.\n",
    "\n",
    "In order for the network to find non-linear relationships in the data, we use **activation functions**. One activation function that is quite common these day are the ReLu function.\n",
    "$$ ReLu = max(0, x)$$\n",
    "\n",
    "\n",
    "Our network will output a `1x10 vector` representing the _probability_ for each number. This is done using **softmax**. \n",
    "$$ S(y_i) = \\frac{e^{y_i}}{\\sum_{i=0}^{n=9}{e^{y_i}}}$$\n",
    "\n",
    "All the probabilities inside the outputted vector will sum to 1.\n",
    "You can read more about Softmax [here](https://medium.com/data-science-bootcamp/understand-the-softmax-function-in-minutes-f3a59641e86d).\n",
    "\n",
    "### Our model\n",
    "Is composed using the [Sequential class from Keras](https://keras.io/api/models/sequential/), and we will add three layers:\n",
    "1. a [Flatten](https://keras.io/api/layers/reshaping_layers/flatten/) layer with input_shape=(28, 28, 1)\n",
    "2. a [Dense](https://keras.io/api/layers/core_layers/dense/) layer with 128 units and **relu** as our activation function\n",
    "3. another Dense layer, with 10 units and **softmax** activation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Hint 1: </summary>\n",
    "<p>\n",
    "    \n",
    "```python\n",
    "model.add(Flatten(input_shape=(28, 28, 1)))\n",
    "```\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Hint 2: </summary>\n",
    "<p>\n",
    "    \n",
    "```python\n",
    "model.add(Dense(128, activation='relu'))\n",
    "```\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Hint 3: </summary>\n",
    "<p>\n",
    "    \n",
    "```python\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "```\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential() \n",
    "\n",
    "model.add(Flatten(input_shape=(28, 28, 1)))\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>\n",
    "    Architecture Visualized\n",
    "</h3>\n",
    "<img src=\"fcnn.png\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Node-Link Visualization of:\n",
    "#### [Fully Connected Neural Networks](https://www.cs.ryerson.ca/~aharley/vis/fc/)\n",
    "\n",
    "#### [Convolutional Neural Networks](https://www.cs.ryerson.ca/~aharley/vis/conv/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compiling the model\n",
    "In order to use our defined model, we need to _compile_ it. This will create a [computational graph](https://medium.com/tebs-lab/deep-neural-networks-as-computational-graphs-867fcaa56c9). We will need to choose a optimizer, and a loss function. In our case, `adam` is a good fit. You can read more about the [Adam Optimizer here](https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c). As we have categorical data, we will choose `categorical_crossentropy`as our loss function. Read more on [loss functions here](https://towardsdatascience.com/what-is-loss-function-1e2605aeb904)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Hint: </summary>\n",
    "<p>\n",
    "    \n",
    "```python\n",
    "model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='categorical_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")\n",
    "```\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='categorical_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1875/1875 [==============================] - 1s 692us/step - loss: 0.2631 - accuracy: 0.9244\n",
      "Epoch 2/10\n",
      "1875/1875 [==============================] - 1s 784us/step - loss: 0.1171 - accuracy: 0.9659\n",
      "Epoch 3/10\n",
      "1875/1875 [==============================] - 1s 747us/step - loss: 0.0810 - accuracy: 0.9753\n",
      "Epoch 4/10\n",
      "1875/1875 [==============================] - 1s 760us/step - loss: 0.0596 - accuracy: 0.9822\n",
      "Epoch 5/10\n",
      "1875/1875 [==============================] - 2s 832us/step - loss: 0.0460 - accuracy: 0.9861\n",
      "Epoch 6/10\n",
      "1875/1875 [==============================] - 2s 862us/step - loss: 0.0364 - accuracy: 0.9888\n",
      "Epoch 7/10\n",
      "1875/1875 [==============================] - 1s 763us/step - loss: 0.0281 - accuracy: 0.9917\n",
      "Epoch 8/10\n",
      "1875/1875 [==============================] - 1s 725us/step - loss: 0.0234 - accuracy: 0.9929\n",
      "Epoch 9/10\n",
      "1875/1875 [==============================] - 1s 693us/step - loss: 0.0190 - accuracy: 0.9944\n",
      "Epoch 10/10\n",
      "1875/1875 [==============================] - 1s 689us/step - loss: 0.0168 - accuracy: 0.9947\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAld0lEQVR4nO3deZwU9Z3/8denu+eCGW4EZbhMiHIYUAYk62pMPEIk3q5KYgwm6m425+bYmM2uJia/XTe6ycasxniQVeMBQeOykXhgNIYVDQioeEVEkEHEAeRmzv78/qgapqenBxqYnpqZej8fj350VX2/Vf2ZVurddZu7IyIi8ZWIugAREYmWgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSCxYmb/bWY/yrPvGjM7tdA1iURNQSAiEnMKApFuyMxSUdcgPYeCQLqccJfMt83sRTPbZWZ3mNkQM/u9me0ws4Vm1j+j/1lm9rKZbTWzp8xsbEbbsWa2LJxvDlCa9VmfMrMV4bzPmNmH86xxhpktN7PtZrbOzL6f1f7X4fK2hu2zwullZvYfZrbWzLaZ2aJw2slmVp3jezg1HP6+mc0zs1+b2XZglplNNbPF4WdsMLP/MrPijPnHm9njZrbFzDaa2T+Z2VAz221mAzP6HWdmNWZWlM/fLj2PgkC6qvOB04APAWcCvwf+CRhM8P/tVwHM7EPAfcDXw7YFwP+aWXG4UnwIuBsYAPwmXC7hvMcCs4G/BQYCvwTmm1lJHvXtAi4F+gEzgC+a2TnhckeG9f48rGkSsCKc7wZgMvBXYU3/CKTz/E7OBuaFn3kP0AT8AzAI+AhwCvD3YQ0VwELgEeAI4IPAE+7+LvAUcGHGcj8L3O/uDXnWIT2MgkC6qp+7+0Z3Xw/8CXjO3Ze7ey3wW+DYsN9FwMPu/ni4IrsBKCNY0U4DioD/dPcGd58HLMn4jCuBX7r7c+7e5O53AnXhfPvk7k+5+0vunnb3FwnC6KNh86eBhe5+X/i5m919hZklgM8DX3P39eFnPuPudXl+J4vd/aHwM/e4+/Pu/qy7N7r7GoIga67hU8C77v4f7l7r7jvc/bmw7U7gEgAzSwIzCcJSYkpBIF3VxozhPTnGy8PhI4C1zQ3ungbWAcPCtvXe+s6KazOGRwLfDHetbDWzrcDwcL59MrPjzezJcJfKNuDvCH6ZEy7jzRyzDSLYNZWrLR/rsmr4kJn9zszeDXcX/WseNQD8DzDOzEYTbHVtc/c/H2RN0gMoCKS7e4dghQ6AmRnBSnA9sAEYFk5rNiJjeB3w/9y9X8arl7vfl8fn3gvMB4a7e1/gFqD5c9YBH8gxzyagtp22XUCvjL8jSbBbKVP2rYJ/AbwGjHH3PgS7zjJrODJX4eFW1VyCrYLPoq2B2FMQSHc3F5hhZqeEBzu/SbB75xlgMdAIfNXMiszsPGBqxry3AX8X/ro3M+sdHgSuyONzK4At7l5rZlMJdgc1uwc41cwuNLOUmQ00s0nh1sps4CdmdoSZJc3sI+Exib8ApeHnFwH/DOzvWEUFsB3YaWZHA1/MaPsdcLiZfd3MSsyswsyOz2i/C5gFnIWCIPYUBNKtufvrBL9sf07wi/tM4Ex3r3f3euA8ghXeFoLjCQ9mzLsUuAL4L+B9YFXYNx9/D1xrZjuAqwkCqXm5bwNnEITSFoIDxRPD5m8BLxEcq9gC/DuQcPdt4TJvJ9ia2QW0Oosoh28RBNAOglCbk1HDDoLdPmcC7wJvAB/LaP8/goPUy9w9c3eZxJDpwTQi8WRmfwDudffbo65FoqUgEIkhM5sCPE5wjGNH1PVItLRrSCRmzOxOgmsMvq4QENAWgYhI7GmLQEQk5rrdjasGDRrko0aNiroMEZFu5fnnn9/k7tnXpgDdMAhGjRrF0qVLoy5DRKRbMbN2TxPWriERkZhTEIiIxJyCQEQk5hQEIiIxV7AgMLPZZvaema1sp93M7EYzW2XBk6iOK1QtIiLSvkJuEfw3MH0f7Z8ExoSvKwluqSsiIp2sYEHg7k8T3F2xPWcDd3ngWaCfmR1eqHpERCS3KK8jGEbrJy5Vh9M2ZHc0sysJthoYMWJEdrOIdAHujjuk3UmH7y3jjgOebhlPezBPOrNPm/kz2tMty2za2z9sT7deTqv5022nt3y205TO8Vl7l9nSt+XvDN/D5wS1jO+7Pft7ymcep3WHU8YOYeLwfgf936g93eKCMne/FbgVoKqqSjdHki6jeeXRmE7TlHYa005Tk9PQPN7kLdPT3qpfY5O3ma8xo09Dk9OUTmf0dRqbWsab0mkampcbLqttn3B6xrJbhsPl51p2m5VwjpU0rVfkum1ZYZnBYX1Ke1wQrCd4pGCzynCaxIx7sNJraErT0JSmvikdjDdmjTelaWjMGm9KU9+Ybjt/Y8v4/pcdTGteGbZZcTdljresMJvHo1SUNJIJoyiRIJk0UokEqYSRShqpRNiWTJBMGKlk2JYwSosSe/u27tMyX8KaX2CthiFhFk4jZ59Ewvb2SxgYrceD9n3Mb0F/MyPZqj3snyCjvtbtyTbLbvns4O+iTXvr5WUtC8PCnejNzwFtfvppy3hzu7UaZx/t+11W9kIKKMogmA982czuB44neIB2m91CEi13p64xzZ76JvY0NLG7vona8H1PQxN76hvD9zS76xuz2nLMk9EW9A9WxoVgBsXJBMXJBEWpBEXJYIVXnExQlExQlArGi5KJ9leM4XgqXOGmEi3Tg2kt87R6z1jptt8vEX5Wy3giAUV75w1W7kV7a8hcyQfzinSEggWBmd0HnAwMMrNq4BqgCMDdbwEWEDzObxWwG7isULXESVPa2VnbyI66BnbUNrKzrpEdtcFw5vjO2kZ21WeurBvZ05DOWLG3tB3oj95UwigrTlJWlKRXcZLS8L2sOEn/XsVhW4JexSlKi5KUpBIUZ6yo966sU1njybBPKhhPZa/YM9qaV+Qisn8FCwJ3n7mfdge+VKjP727cnV31TcFKvLaBHXXhiru2kZ11LSvyHRnjO+sa2V7byM7alvHd9U37/axkwigvSVFekqI0XCGXFSXpV1bE4X1Kg5V3cZJeRcHKO3ulHgynKCtOUFaUoqy49Qq/KKnrFEW6k25xsLgn2Lq7nufe2sJzq7ewetPOcIXfvDJvYFddY16/vMtLUlSUpva+9y0rorJ/GRUlzdOKKC8N2iraGS8tSnTq/kcR6doUBAWyZVc9f35rM8+u3sKzqzfz+sYduENJKsGHhlTQpyzFwPJewYq6JEWf0lS4wg7Gy0vDaSVFwYq/NEV5cYqEdneISAdTEHSQzTvr+PNbwUr/2dVbeH1j8CjY0qIEVSMHMOOYw5n2gYF8uLIvJalkxNWKiLRQEBykTTvreC78tf/cW5v5y8adAJQVJaka1Z+zJh3BtCMHcMywfhSntM9cRLouBUGe3ttRy3Ort/BcuLtn1XvBir93cZLJowZwzrHDOH508ItfB0tFpDtRELRj4/ba8Nd+8Kt/dc0uIDhYWzWqPxdMruT40QOYMEwrfhHp3hQEoXe31Ya/9jeHZ/YEK/6KkhRTRg/g4inDOX70QMYf0YeUVvwi0oPENgje2bonWPG/GezuWbN5NwAVpSmOHz2AmVNHMO3IgYw7oo8uTBKRHi02QbBxey2L3ti0dx//21uCFX/fsiKmjh7AJdNGMu3IgYw9XCt+EYmX2ATBvOeruf7R1+nXq4ipowYw669GMe3IgRw9tELn5otIrMUmCM4/rpKPH30YRw3Ril9EJFNsgmBo31KG9i2NugwRkS5Hp7+IiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxV9AgMLPpZva6ma0ys6tytI8wsyfNbLmZvWhmZxSyHhERaatgQWBmSeAm4JPAOGCmmY3L6vbPwFx3Pxa4GLi5UPWIiEhuhdwimAqscvfV7l4P3A+cndXHgT7hcF/gnQLWIyIiOaQKuOxhwLqM8Wrg+Kw+3wceM7OvAL2BUwtYj4iI5BD1weKZwH+7eyVwBnC3mbWpycyuNLOlZra0pqam04sUEenJChkE64HhGeOV4bRMXwDmArj7YqAUGJS9IHe/1d2r3L1q8ODBBSpXRCSeChkES4AxZjbazIoJDgbPz+rzNnAKgJmNJQgC/eQXEelEBQsCd28Evgw8CrxKcHbQy2Z2rZmdFXb7JnCFmb0A3AfMcncvVE0iItJWIQ8W4+4LgAVZ067OGH4FOKGQNYiIyL5FfbBYREQipiAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYKGgRmNt3MXjezVWZ2VTt9LjSzV8zsZTO7t5D1iIhIW6lCLdjMksBNwGlANbDEzOa7+ysZfcYA3wVOcPf3zeywQtUjIiK55bVFYGYPmtkMMzuQLYipwCp3X+3u9cD9wNlZfa4AbnL39wHc/b0DWL6IiHSAfFfsNwOfBt4ws+vM7Kg85hkGrMsYrw6nZfoQ8CEz+z8ze9bMpudakJldaWZLzWxpTU1NniWLiEg+8goCd1/o7p8BjgPWAAvN7Bkzu8zMig7h81PAGOBkYCZwm5n1y/H5t7p7lbtXDR48+BA+TkREsuW9q8fMBgKzgMuB5cDPCILh8XZmWQ8MzxivDKdlqgbmu3uDu78F/IUgGEREpJPkdbDYzH4LHAXcDZzp7hvCpjlmtrSd2ZYAY8xsNEEAXEyweynTQwRbAr8ys0EEu4pWH9BfICKx0NDQQHV1NbW1tVGX0qWVlpZSWVlJUVH+O2vyPWvoRnd/MleDu1e1M73RzL4MPAokgdnu/rKZXQssdff5YdvpZvYK0AR829035129iMRGdXU1FRUVjBo1CjOLupwuyd3ZvHkz1dXVjB49Ou/58g2CcWa23N23AphZf2Cmu9+8n6IWAAuypl2dMezAN8KXiEi7amtrFQL7YWYMHDiQAz2pJt9jBFc0hwBAeLrnFQf0SSIih0ghsH8H8x3lGwRJy1h6eLFY8QF/mohIN7V161ZuvnmfO0FyOuOMM9i6des++1x99dUsXLjwICs7dPkGwSMEB4ZPMbNTgPvCaSIisdBeEDQ2Nu5zvgULFtCvX7999rn22ms59dRTD6W8Q5JvEHwHeBL4Yvh6AvjHQhUlItLVXHXVVbz55ptMmjSJKVOmcOKJJ3LWWWcxbtw4AM455xwmT57M+PHjufXWW/fON2rUKDZt2sSaNWsYO3YsV1xxBePHj+f0009nz549AMyaNYt58+bt7X/NNddw3HHHccwxx/Daa68BUFNTw2mnncb48eO5/PLLGTlyJJs2beqQvy2vg8XungZ+Eb5ERCL1g/99mVfe2d6hyxx3RB+uOXN8u+3XXXcdK1euZMWKFTz11FPMmDGDlStX7j07Z/bs2QwYMIA9e/YwZcoUzj//fAYOHNhqGW+88Qb33Xcft912GxdeeCEPPPAAl1xySZvPGjRoEMuWLePmm2/mhhtu4Pbbb+cHP/gBH//4x/nud7/LI488wh133NFhf3u+9xoaY2bzwruErm5+dVgVIiLdzNSpU1udonnjjTcyceJEpk2bxrp163jjjTfazDN69GgmTZoEwOTJk1mzZk3OZZ933nlt+ixatIiLL74YgOnTp9O/f/8O+1vyPX30V8A1wE+BjwGXoWcZiEhE9vXLvbP07t177/BTTz3FwoULWbx4Mb169eLkk0/OeeFbSUnJ3uFkMrl311B7/ZLJ5H6PQXSEfFfmZe7+BGDuvtbdvw/MKFxZIiJdS0VFBTt27MjZtm3bNvr370+vXr147bXXePbZZzv880844QTmzp0LwGOPPcb777/fYcvOd4ugLrwF9Rvh1cLrgfIOq0JEpIsbOHAgJ5xwAhMmTKCsrIwhQ4bsbZs+fTq33HILY8eO5aijjmLatGkd/vnXXHMNM2fO5O677+YjH/kIQ4cOpaKiokOWbcHFvfvpZDYFeBXoB/wQ6ANc7+4dH3v7UVVV5UuXtnd7IxHpqV599VXGjh0bdRmRqaurI5lMkkqlWLx4MV/84hdZsWJFzr65visze769WwLtd4sgvHjsInf/FrCT4PiAiIh0orfffpsLL7yQdDpNcXExt912W4cte79B4O5NZvbXHfaJIiJywMaMGcPy5csLsux8jxEsN7P5wG+AXc0T3f3BglQlIiKdJt8gKAU2Ax/PmOaAgkBEpJvL98piHRcQEemh8n1C2a8ItgBacffPd3hFIiLSqfLdNfS7jOFS4FzgnY4vR0Sk6yovL2fnzp1Rl9Hh8t019EDmuJndBywqSEUiItKpDvZ+QWOAwzqyEBGR7sLd+fa3v82ECRM45phjmDNnDgAbNmzgpJNOYtKkSUyYMIE//elPNDU1MWvWrL19f/rTn0ZcfVv5HiPYQetjBO8SPKNARKTz/f4qePeljl3m0GPgk9fl1fXBBx9kxYoVvPDCC2zatIkpU6Zw0kknce+99/KJT3yC733vezQ1NbF7925WrFjB+vXrWblyJcB+n1YWhXx3DXXMDS1ERHqARYsWMXPmTJLJJEOGDOGjH/0oS5YsYcqUKXz+85+noaGBc845h0mTJnHkkUeyevVqvvKVrzBjxgxOP/30qMtvI98tgnOBP7j7tnC8H3Cyuz9UuNJERNqR5y/3znbSSSfx9NNP8/DDDzNr1iy+8Y1vcOmll/LCCy/w6KOPcssttzB37lxmz54ddamt5HuM4JrmEABw960EzycQEYmdE088kTlz5tDU1ERNTQ1PP/00U6dOZe3atQwZMoQrrriCyy+/nGXLlrFp0ybS6TTnn38+P/rRj1i2bFnU5beR7+mjuQIj33lFRHqUc889l8WLFzNx4kTMjB//+McMHTqUO++8k+uvv56ioiLKy8u56667WL9+PZdddhnpdBqAf/u3f4u4+rbyvQ31bGArcFM46UvAAHefVbDK2qHbUIvEU9xvQ30gDvQ21PnuGvoKUA/MAe4HagnCQEREurl8zxraBVxV4FpERCQCeW0RmNnj4ZlCzeP9zezRglUlIiKdJt9dQ4PCM4UAcPf30ZXFItLJ8jmmGXcH8x3lGwRpMxvRPGJmo8hxN1IRkUIpLS1l8+bNCoN9cHc2b95MaWnpAc2X7ymg3wMWmdkfAQNOBK48sBJFRA5eZWUl1dXV1NTURF1Kl1ZaWkplZeUBzZPvweJHzKyKYOW/HHgI2LO/+cxsOvAzIAnc7u45Lwc0s/OBecAUd9e5oSLSRlFREaNHj466jB4p31tMXA58DagEVgDTgMW0fnRl9jxJgusOTgOqgSVmNt/dX8nqVxEu+7mDqF9ERA5RvscIvgZMAda6+8eAYwkuMNuXqcAqd1/t7vUE1x+cnaPfD4F/J7g2QUREOlm+QVDr7rUAZlbi7q8BR+1nnmHAuozx6nDaXmZ2HDDc3R/e14LM7EozW2pmS7V/UESkY+UbBNXhdQQPAY+b2f8Aaw/lg80sAfwE+Ob++rr7re5e5e5VgwcPPpSPFRGRLPkeLD43HPy+mT0J9AUe2c9s64HhGeOV4bRmFcAE4CkzAxgKzDezs3TAWESk8xzwHUTd/Y95dl0CjDGz0QQBcDHw6YzlbAMGNY+b2VPAtxQCIiKd62CfWbxf7t4IfBl4FHgVmOvuL5vZtWZ2VqE+V0REDkxBnyng7guABVnTrm6n78mFrEVERHIr2BaBiIh0DwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxF58gWL8MHrgC6ndHXYmISJcSnyB49yV46Tdw55mwsybqakREuoz4BMHkz8FFd8PGlXDHqbBpVdQViYh0CfEJAoCxZ8Ksh6FuZxAGbz8bdUUiIpGLVxAAVFbB5Y9D2QC48yx4+bdRVyQiEqn4BQHAgCPh8oVwxLHwm1nwfz8D96irEhGJRDyDAKDXALj0f2D8ufD41bDgW9DUGHVVIiKdLhV1AZEqKoXzZ0Pf4fDMjbCtGi6YDcW9o65MRKTTxHeLoFkiAaf/EGb8B7zxGPzqDNixMeqqREQ6jYKg2ZTL4eL7YNNf4PZToeb1qCsSEekUCoJMR00PTi9trIU7ToM1i6KuSESk4AoaBGY23cxeN7NVZnZVjvZvmNkrZvaimT1hZiMLWU9ehh0XnFFUPhTuOgdenBt1RSIiBVWwIDCzJHAT8ElgHDDTzMZldVsOVLn7h4F5wI8LVc8B6T8SvvAojJgGD14BT9+g00tFpMcq5BbBVGCVu69293rgfuDszA7u/qS7N98F7lmgsoD1HJiy/nDJA3DMhfCHH8L/fhWaGqKuSkSkwxXy9NFhwLqM8Wrg+H30/wLw+1wNZnYlcCXAiBEjOqq+/UuVwHm3Qr8R8KcbYNt6uPBOKKnovBpERAqsSxwsNrNLgCrg+lzt7n6ru1e5e9XgwYM7uzg45V/gzBth9VMw+5Ow/Z3OrUFEpIAKGQTrgeEZ45XhtFbM7FTge8BZ7l5XwHoOzeTPwWfmwvtvBaeXbnw56opERDpEIYNgCTDGzEabWTFwMTA/s4OZHQv8kiAE3itgLR3jg6fCZb8HT8Ps6fDmk1FXJCJyyAoWBO7eCHwZeBR4FZjr7i+b2bVmdlbY7XqgHPiNma0ws/ntLK7rOPzDwemlfYfDPRfA8nuirkhE5JCYd7PTIquqqnzp0qVRlwG122DupcFxg49eBSdfFRxPEBHpgszseXevytXWJQ4Wd0ulfeEz82DSZ+CP18FDfw+N9VFXJSJywOJ999FDlSyCs2+CfiPhqX+F7dVw0a+DkBAR6Sa0RXCozODk78A5v4C1z8Adn4Ct6/Y/n4hIF6Eg6CiTPh1cibx9fXB66YYXoq5IRCQvCoKOdOTJ8PlHIZEKnmvwxuNRVyQisl8Kgo42ZFxweumA0XDvRbD0V1FXJCKyTwqCQuhzeHDh2Qc+Br/7Oiz8AaTTUVclIpKTgqBQSipg5hyYPAsW/SS4nXVj172DhojEl04fLaRkCj71n8HppU/8AHZsCE4v7TUg6spERPbSFkGhmcGJ34DzbofqJTD7E7BlddRViYjspS2CzvLhv4E+R8D9M+HGY2HgB2FYFVSGryETggvUREQ6mYKgM406Af72aVj5AFQ/D2/+AV68P2hLlcLhE6FyCgybHIRD3+G6f5GIFJxuOhcld9i2DqqXBq/1S4ML0Rprg/byIeFWw+QgII44Vk9HE5GDsq+bzmmLIEpmwWMw+42ACecF0xrrYeNKWP98GBBL4PWHm2eAw8aGWwxTgq2GwUdDIhnZnyAi3Z+2CLqD3Vtg/bIgFNaHWw+1W4O24vJgS6GyquWYQ8XQSMsVka5HWwTdXa8BMObU4AXBLqUtq4NgaN6l9MzPId0YtPcd3nqr4fCJUFQWXf0i0qUpCLojMxj4geA18eJgWsMe2PBiuMWwJDgY/cpDQVsiBUPGhweiw62GAR+AhM4eFhEFQc9RVAYjjg9ezXZsDI81hLuUXpgDS24P2kr7wmHjg+MT/UcGF701v/c5QscdRGJEQdCTVQyBo88IXgDpJqh5veU4w6Y3YM2f4MU5QMaxokQK+la2DofM4fLDdFqrSA+iIIiTRDK4O+qQcXDcpS3TG+uD01i3roX31wbvW98Ohl//Peyqab2cVGl4ttPIrK2JcFpZfwWFSDeiIBBIFbccc8ilflfw1LVWQREOV/8Zare17l/Sp2047N2yGAEl5YX/m0QkbwoC2b/i3nDY0cErlz1bW29FNIfE5jeDq6cbdrfu32tg1q6mIdB7UDC9+b3XICgqLfifJiIKAukIZf2C1+ET27a5w65NYTisCcKiOSg2vACv/g7SDbmXW1zeNhx6N7/nmFZSoV1SIgdBQSCFZQblg4NXZY5rWdLp4OK43ZuDwNi9KWM4Y9rOjbDxlWC4+RYc2ZLF+wiM5kAZ1PJe1k9nR4mgIJCoJRLBBXO9BsCgMfvv7x4cs9gbGJuD4VYhEk57f20wXrc997IsERzY7hWGRWlfKO0THOMoqWgZLu0bjJf0yZjWB4ordC2G9AgKAulezIKDzSXl0H9UfvM01geBsDcwNrfe2ti1KbiNx/ZqeG97EBy128Gb9r/s4op2wiMcL2kvXJqn9QkO1otESEEgPV+qOHiOdJ/D85/HPbhauzkU6ra3Hq7dDnU7Moa3BeO7twTHQprbG/fkUV9p66AoLg9eJeF7ce+grdVw76w+4XBRb22lyAFTEIjkYgbFvYLXodzEr7E+DIwwKNoEybaMoAnb63cFWyd1O4Ph+p1tz7zal6LeYWCU5xEqufqUB393siQI0VRpMKyA6bEUBCKFlCqG1MDg4PWhSDe1hELdzuB97/AuqN/ROjjqdrTuv/M9qF8dTGueP/Nq8nwkUlnhUAypkpZpyZJgPFWS1ZY9LZy/zbSS1vNktyWLWw/rQH+HURCIdAeJZLDbqLRPxywvnQ52W7UJlebXbmiqg8bw1VTf+r2xtu20pvrwQP7m3G3N83QUS4ahUJQRHkUZwVTcOqSSRS1Bs3e4OEfQNPfPGE6kgv8GiWQwbOF7IpE1HvZpNZ4KTkxoM2+yy5zurCAQiaNEIjzO0BsY0nmf654jJHIFTl2wWy3zvak+Y7ghq1996+VmDtftgMZNGdPrs+arA0933neQyRJ5hkg4/tHvwDEXdHgZBQ0CM5sO/AxIAre7+3VZ7SXAXcBkYDNwkbuvKWRNIhIhs5ZdQF1JU2NLKOQKm3RT8LyP5ndvCraq9g43tzVljTcGIdNm3lx985i3rH9B/vyCBYGZJYGbgNOAamCJmc1391cyun0BeN/dP2hmFwP/DlxUqJpERHJKpoIXvaKuJBKFPA1gKrDK3Ve7ez1wP3B2Vp+zgTvD4XnAKWZdZKeZiEhMFDIIhgHrMsarw2k5+7h7I7ANaHN6hZldaWZLzWxpTU1NdrOIiByCbnFisLvf6u5V7l41ePDgqMsREelRChkE64HhGeOV4bScfcwsBfQlOGgsIiKdpJBBsAQYY2ajzawYuBiYn9VnPvC5cPgC4A/ufoBXuYiIyKEo2FlD7t5oZl8GHiU4fXS2u79sZtcCS919PnAHcLeZrQK2EISFiIh0ooJeR+DuC4AFWdOuzhiuBf6mkDWIiMi+dYuDxSIiUjjW3XbJm1kNsPYgZx8EbOrAcro7fR+t6ftooe+itZ7wfYx095ynXXa7IDgUZrbU3XM8LzGe9H20pu+jhb6L1nr696FdQyIiMacgEBGJubgFwa1RF9DF6PtoTd9HC30XrfXo7yNWxwhERKStuG0RiIhIFgWBiEjMxSYIzGy6mb1uZqvM7Kqo64mKmQ03syfN7BUze9nMvhZ1TV2BmSXNbLmZ/S7qWqJmZv3MbJ6ZvWZmr5rZR6KuKSpm9g/hv5OVZnafmZVGXVMhxCIIMp6W9klgHDDTzMZFW1VkGoFvuvs4YBrwpRh/F5m+BrwadRFdxM+AR9z9aGAiMf1ezGwY8FWgyt0nENwzrUfeDy0WQUB+T0uLBXff4O7LwuEdBP/Isx8YFCtmVgnMAG6PupaomVlf4CSCG0Li7vXuvjXSoqKVAsrC2+T3At6JuJ6CiEsQ5PO0tNgxs1HAscBzEZcStf8E/hFIR1xHVzAaqAF+Fe4qu93MekddVBTcfT1wA/A2sAHY5u6PRVtVYcQlCCSLmZUDDwBfd/ftUdcTFTP7FPCeuz8fdS1dRAo4DviFux8L7AJieUzNzPoT7DkYDRwB9DazS6KtqjDiEgT5PC0tNsysiCAE7nH3B6OuJ2InAGeZ2RqCXYYfN7NfR1tSpKqBandv3kqcRxAMcXQq8Ja717h7A/Ag8FcR11QQcQmCfJ6WFgtmZgT7f191959EXU/U3P277l7p7qMI/r/4g7v3yF99+XD3d4F1ZnZUOOkU4JUIS4rS28A0M+sV/rs5hR564LygD6bpKtp7WlrEZUXlBOCzwEtmtiKc9k/hQ4REAL4C3BP+aFoNXBZxPZFw9+fMbB6wjOBsu+X00FtN6BYTIiIxF5ddQyIi0g4FgYhIzCkIRERiTkEgIhJzCgIRkZhTEIh0IjM7WXc4la5GQSAiEnMKApEczOwSM/uzma0ws1+GzyvYaWY/De9P/4SZDQ77TjKzZ83sRTP7bXiPGszsg2a20MxeMLNlZvaBcPHlGff7vye8alUkMgoCkSxmNha4CDjB3ScBTcBngN7AUncfD/wRuCac5S7gO+7+YeCljOn3ADe5+0SCe9RsCKcfC3yd4NkYRxJc7S0SmVjcYkLkAJ0CTAaWhD/Wy4D3CG5TPSfs82vgwfD+/f3c/Y/h9DuB35hZBTDM3X8L4O61AOHy/uzu1eH4CmAUsKjgf5VIOxQEIm0ZcKe7f7fVRLN/yep3sPdnqcsYbkL/DiVi2jUk0tYTwAVmdhiAmQ0ws5EE/14uCPt8Gljk7tuA983sxHD6Z4E/hk9/qzazc8JllJhZr878I0TypV8iIlnc/RUz+2fgMTNLAA3Alwge0jI1bHuP4DgCwOeAW8IVfebdOj8L/NLMrg2X8Ted+GeI5E13HxXJk5ntdPfyqOsQ6WjaNSQiEnPaIhARiTltEYiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMz9f7nCMKnW9LCVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    epochs=10,\n",
    "    batch_size=32,\n",
    "    verbose=1\n",
    ")\n",
    "\n",
    "# Visualize accuracy and loss during training\n",
    "plt.plot(history.history['accuracy'])\n",
    "plt.plot(history.history['loss'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['training', 'loss'], loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_weights('models/my_first_model.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#model.load_weights('models/simple_fnn_30_epochs.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluate our model\n",
    "In order to evaluate our model, we evaluate it on the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 0s 473us/step - loss: 0.0809 - accuracy: 0.9772\n",
      "Test loss: 0.0809\n",
      "Test accuracy: 0.977\n"
     ]
    }
   ],
   "source": [
    "loss, accuracy  = model.evaluate(x_test, y_test, verbose=True)\n",
    "\n",
    "print(f'Test loss: {loss:.3}')\n",
    "print(f'Test accuracy: {accuracy:.3}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Hint: </summary>\n",
    "<p>\n",
    "    \n",
    "```python\n",
    "score = model.evaluate(X_TESTING_DATA, Y_TESTING_DATA)\n",
    "```\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  5. Let's have some fun\n",
    "\n",
    "Here we'll use OpenCV and our trained model to perform predictions on images fed from our webcamera, in real time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "code_folding": [
     0
    ],
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# MNIST PREPROCESSING\n",
    "# Inspired by Ole Kröger (https://bit.ly/3jPiBMY)\n",
    "import math\n",
    "from scipy import ndimage\n",
    "\n",
    "\n",
    "def get_best_shift(img):\n",
    "    center_y , center_x = ndimage.measurements.center_of_mass(img)\n",
    "\n",
    "    rows, cols = img.shape\n",
    "    shift_y = np.round(rows//2 - center_y)\n",
    "    shift_x = np.round(cols//2 - center_x)\n",
    "\n",
    "    return shift_x, shift_y\n",
    "\n",
    "\n",
    "def shift(img, sx, sy):\n",
    "    rows,cols = img.shape\n",
    "    M = np.float32([[1,0,sx],[0,1,sy]])\n",
    "    shifted = cv.warpAffine(img,M,(cols,rows))\n",
    "    return shifted\n",
    "\n",
    "\n",
    "def mnist_preprocessing(image, dim=28, box=20):\n",
    "    resized_image = cv.resize(image, dsize=(dim, dim), interpolation=cv.INTER_LINEAR)\n",
    "    gray = np.asarray(cv.cvtColor(resized_image, cv.COLOR_BGR2GRAY))\n",
    "    gray = 255 - gray\n",
    "    gray = cv.resize(gray, (dim, dim))\n",
    "    _, thresh = cv.threshold(gray, 128, 255, cv.THRESH_BINARY + cv.THRESH_OTSU)\n",
    "    \n",
    "    \n",
    "    while np.sum(gray[0]) == 0:\n",
    "        gray = gray[1:]\n",
    "\n",
    "    while np.sum(gray[:,0]) == 0:\n",
    "        gray = np.delete(gray, 0, 1)\n",
    "\n",
    "    while np.sum(gray[-1]) == 0:\n",
    "        gray = gray[:-1]\n",
    "\n",
    "    while np.sum(gray[:,-1]) == 0:\n",
    "        gray = np.delete(gray, -1, 1)\n",
    "\n",
    "    rows,cols = gray.shape\n",
    "\n",
    "    if rows > cols:\n",
    "        factor = box/rows\n",
    "        rows = box\n",
    "        cols = int(round(cols*factor))\n",
    "        gray = cv.resize(gray, (cols,rows))\n",
    "    else:\n",
    "        factor = box/cols\n",
    "        cols = box\n",
    "        rows = int(round(rows*factor))\n",
    "        gray = cv.resize(gray, (cols, rows))\n",
    "\n",
    "    colsPadding = (int(math.ceil((dim - cols)/2.0)),int(math.floor((dim - cols)/2.0)))\n",
    "    rowsPadding = (int(math.ceil((dim - rows)/2.0)),int(math.floor((dim - rows)/2.0)))\n",
    "    gray = np.pad(gray, (rowsPadding, colsPadding), 'constant')\n",
    "\n",
    "    shiftx,shifty = get_best_shift(gray)\n",
    "    shifted = shift(gray,shiftx,shifty)\n",
    "    \n",
    "    normalized_img = normalize(shifted)\n",
    "    \n",
    "    return shifted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def predict_single_image(model, img):\n",
    "    img = np.expand_dims(img, axis=0)     # --> (1, 28, 28)\n",
    "    img = np.reshape(img, [1, 28, 28, 1]) # --> (1, 28, 28, 1)\n",
    "    pred = model.predict(img)\n",
    "    label = np.argmax(pred[:1])\n",
    "    \n",
    "    return label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cap = cv.VideoCapture(0) \n",
    "\n",
    "init_frame_read_correctly, init_frame = cap.read()\n",
    "if not init_frame_read_correctly:\n",
    "        print(\"No frames to read...\")\n",
    "        \n",
    "\n",
    "bounding_box = (init_frame.shape[1]//4, init_frame.shape[0]//4,\n",
    "                init_frame.shape[1]//2, init_frame.shape[0]//2)    \n",
    "\n",
    "while(cap.isOpened()):\n",
    "    # Capture frame-by-frame\n",
    "    frame_read_correctly, frame = cap.read()\n",
    "    if not frame_read_correctly:\n",
    "        print(\"No frames to read...\")\n",
    "        break\n",
    "    \n",
    "    \n",
    "    # new_bounding_box = bounding_box\n",
    "    wh = (frame.shape[1]//frame.shape[0])*200\n",
    "    if frame_read_correctly:\n",
    "        point1 = (frame.shape[1]//2 - wh, frame.shape[0]//2 - wh)\n",
    "        point2 = (frame.shape[1]//2 + wh, frame.shape[0]//2 + wh)\n",
    "        cv.rectangle(frame, point1, point2, (255, 255, 0), 2)\n",
    "        \n",
    "\n",
    "    try:\n",
    "        # Crop out new ROI and process it\n",
    "        new_ROI = frame[frame.shape[0]//2 - wh :frame.shape[0]//2 + wh, frame.shape[1]//2 - wh:frame.shape[1]//2 + wh]\n",
    "        img = mnist_preprocessing(new_ROI)\n",
    "        label = predict_single_image(model, img)\n",
    "\n",
    "        cv.namedWindow('processed', 0)\n",
    "        cv.imshow('processed', img)\n",
    "        \n",
    "    except Exception as e:\n",
    "        print(f\"Exception occured:\\n{e}\")\n",
    "        pass\n",
    "\n",
    "    \n",
    "    # Display the resulting pred\n",
    "    cv.putText(frame,\n",
    "               f\"{label}\",\n",
    "               org=(point2[0] - 75, point2[1] + 75),\n",
    "               fontFace=cv.FONT_HERSHEY_PLAIN,\n",
    "               fontScale=5,\n",
    "               color=(255, 255, 0),\n",
    "               thickness=2,\n",
    "               lineType=cv.LINE_AA\n",
    "              )\n",
    "    \n",
    "    cv.namedWindow('frame', 0)\n",
    "    cv.resizeWindow('frame', 640, 480)\n",
    "    cv.imshow('frame', frame)\n",
    "    #cv.imshow('box', new_ROI)\n",
    "\n",
    "    # Listen for keypress\n",
    "    # SINCE OpenCV videoCapture inside a Jupyter Notebook Kernel is 'kinda broken'\n",
    "    # one needs to quit the OpenCV-windows hard..\n",
    "    # MacOS --> CMD+Q\n",
    "    # MacOS --> ALT+F4\n",
    "    if cv.waitKey(1) & 0xFF == ord('q'):\n",
    "        break\n",
    "        \n",
    "    \n",
    "# Release the capture afterward\n",
    "cap.release()\n",
    "cv.destroyAllWindows()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3.8.1 64-bit ('venv')",
   "language": "python",
   "name": "python38164bitvenv355e9b74feae425dba00ca460173104b"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}