chapter05_fundamentals-of-ml.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n",
    "\n",
    "**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n",
    "\n",
    "This notebook was generated for TensorFlow 2.6."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "# Fundamentals of machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Generalization: The goal of machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Underfitting and overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Noisy training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Ambiguous features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Rare features and spurious correlations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Adding white-noise channels or all-zeros channels to MNIST**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "import numpy as np\n",
    "\n",
    "(train_images, train_labels), _ = mnist.load_data()\n",
    "train_images = train_images.reshape((60000, 28 * 28))\n",
    "train_images = train_images.astype(\"float32\") / 255\n",
    "\n",
    "train_images_with_noise_channels = np.concatenate(\n",
    "    [train_images, np.random.random((len(train_images), 784))], axis=1)\n",
    "\n",
    "train_images_with_zeros_channels = np.concatenate(\n",
    "    [train_images, np.zeros((len(train_images), 784))], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Training the same model on MNIST data with noise channels or all-zero channels**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "375/375 [==============================] - 3s 6ms/step - loss: 0.6098 - accuracy: 0.8149 - val_loss: 0.3313 - val_accuracy: 0.8983\n",
      "Epoch 2/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.2426 - accuracy: 0.9252 - val_loss: 0.2690 - val_accuracy: 0.9123\n",
      "Epoch 3/10\n",
      "375/375 [==============================] - 2s 6ms/step - loss: 0.1603 - accuracy: 0.9499 - val_loss: 0.1783 - val_accuracy: 0.9457\n",
      "Epoch 4/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.1156 - accuracy: 0.9642 - val_loss: 0.1267 - val_accuracy: 0.9628\n",
      "Epoch 5/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0827 - accuracy: 0.9736 - val_loss: 0.1210 - val_accuracy: 0.9648\n",
      "Epoch 6/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0622 - accuracy: 0.9799 - val_loss: 0.1579 - val_accuracy: 0.9558\n",
      "Epoch 7/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0454 - accuracy: 0.9854 - val_loss: 0.1412 - val_accuracy: 0.9603\n",
      "Epoch 8/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0342 - accuracy: 0.9888 - val_loss: 0.1786 - val_accuracy: 0.9570\n",
      "Epoch 9/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0261 - accuracy: 0.9915 - val_loss: 0.1635 - val_accuracy: 0.9614\n",
      "Epoch 10/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0211 - accuracy: 0.9931 - val_loss: 0.1435 - val_accuracy: 0.9648\n",
      "Epoch 1/10\n",
      "375/375 [==============================] - 2s 6ms/step - loss: 0.2886 - accuracy: 0.9157 - val_loss: 0.1502 - val_accuracy: 0.9576\n",
      "Epoch 2/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.1196 - accuracy: 0.9643 - val_loss: 0.1094 - val_accuracy: 0.9680\n",
      "Epoch 3/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0777 - accuracy: 0.9777 - val_loss: 0.0892 - val_accuracy: 0.9742\n",
      "Epoch 4/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0573 - accuracy: 0.9824 - val_loss: 0.0831 - val_accuracy: 0.9758\n",
      "Epoch 5/10\n",
      "375/375 [==============================] - 2s 6ms/step - loss: 0.0415 - accuracy: 0.9878 - val_loss: 0.0936 - val_accuracy: 0.9724\n",
      "Epoch 6/10\n",
      "375/375 [==============================] - 2s 6ms/step - loss: 0.0310 - accuracy: 0.9907 - val_loss: 0.0860 - val_accuracy: 0.9758\n",
      "Epoch 7/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0240 - accuracy: 0.9929 - val_loss: 0.0768 - val_accuracy: 0.9793\n",
      "Epoch 8/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0187 - accuracy: 0.9946 - val_loss: 0.0872 - val_accuracy: 0.9768\n",
      "Epoch 9/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0142 - accuracy: 0.9959 - val_loss: 0.0869 - val_accuracy: 0.9783\n",
      "Epoch 10/10\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.0107 - accuracy: 0.9973 - val_loss: 0.0874 - val_accuracy: 0.9793\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "def get_model():\n",
    "    model = keras.Sequential([\n",
    "        layers.Dense(512, activation=\"relu\"),\n",
    "        layers.Dense(10, activation=\"softmax\")\n",
    "    ])\n",
    "    model.compile(optimizer=\"rmsprop\",\n",
    "                  loss=\"sparse_categorical_crossentropy\",\n",
    "                  metrics=[\"accuracy\"])\n",
    "    return model\n",
    "\n",
    "model = get_model()\n",
    "history_noise = model.fit(\n",
    "    train_images_with_noise_channels, train_labels,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)\n",
    "\n",
    "model = get_model()\n",
    "history_zeros = model.fit(\n",
    "    train_images_with_zeros_channels, train_labels,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Plotting a validation accuracy comparison**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fc15331d6d8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABHqElEQVR4nO3deXgUVdbA4d8h7IvKPkjYRBDZkkAAQTZxQ1EQRBYVxX1jHHV0hHFG/RgddWTGFURcQFEHRYXBAUVlR0CJAo6AbAISQED2VUhyvj9uddIJnaSB7lQnOe/z9JPqquqqU9WdOlX3Vt0rqooxxhiTUwm/AzDGGBObLEEYY4wJyRKEMcaYkCxBGGOMCckShDHGmJAsQRhjjAnJEkQUiMgTIvKriPzive8tIptE5ICIJPkYV1TiEJG63jLjIrXMfNY3TkSeKIh1nQgR2SAiF/kdx8kIjl1E/iwir4cz70msp5OIrDrZOE3BsgRxErx/kMPeQTHwetmbVhf4I9BUVX/nfWQEMERVK6rqklNYr4rI2acQekTiyElVf/aWmR6pZRr/qOrfVfXWSCwr529WVeep6jmRWLaJvpJ+B1CIXamqX4YYXxfYqarbg8bVA5YXTFh5ipU4jClSRKSkqqb5HUek2RVEBHmX3V8AZ3pXFf8WkQNAHLBMRNZ5850pIh+JyA4RWS8i9wYtI867xF8nIvtF5FsRqSMic71ZlnnL7h9i/SVE5C8islFEtovI2yJyuoiUCRVHiM+riNwpImtEZI+IjBQRyWvZ3rT63mdLeu8Hi8hPXvzrReS6oHXcLCIrRWS3iEwXkXp57M+OIrLAi2WTiAwOmlxZRKZ66/haRBoGfe4Fb/593v7rFDTtcRH5wIt/v4gsF5HkoOkbRORBEfleRPaKyPsiUjZo+hUistSLaYGItMwl9rYikuLFsE1E/pXHdt4mImtFZJeITBGRM8P5TnIs40zvqrZK0LgkcUWdpUSkoYjMFJGd3rh3ReSMXOJ5XETeCXo/yPved4rIIyG2c6EX21YReVlESnvTjvvNikhXEUkN+vy5IjLb+/xyEekZNG2ct70hv+cQcU8UkV+8722uiDQLmlZORP7pbcdeEZkvIuW8aSF/Z15ctwYtY7CIzM/x3dwjImuANd64vH57uf1vjxSRf+bYlikicn9u21pgVNVeJ/gCNgAX5TKtK5CaY5wCZ3vDJYBvgUeB0sBZwE/Apd70h4D/AecAAiQAVXMuJ5d13wys9ZZZEfgYGB8qjlw+r8B/gTNwV0I7gO75LRuo7322JFAB2Aec402rBTTzhnt5yzjXm/cvwIJcYqkH7AcGAqWAqkCiN20csBNo6y3nXWBC0Gev9+YviSvu+wUo6017HDgCXI5LmE8Bi3J8t98AZwJVgJXAnd60JGA70M777I3e/GVy/i6AhcAgb7gicF4u29kN+BVoBZQBXgLmhvOdhFjWTOC2oPfPAqO94bOBi711VAfmAs+H+k17++gdb7gpcADo7H32X0Ba0LytgfO8fV3f21/35fabI+j/w/te1wJ/xv0vdPO+88BvJ8/vOZfffyUvzueBpUHTRgKzgdred9fBmy+v39ls4NagZQwG5ufYti9wv5NyYfz2Qv5ve9u3BSjhzVcNOATU9P1Y53cAhfHl/TMdAPYEvW7L+Q+Q44cUSBDtgJ9zTB8GjPWGVwG9cllvfgf4GcDdQe/PAY4BJcP8vAIdg95/AAzNb9kcnyD2AFcH/mmCPvMpcEvQ+xLeP0K9ELEMAyblEuc44PWg95cDP+axXbuBBG/4ceDLoGlNgcM5vtvrg97/g6yD7CvA33IsexXQJeizgQPnXOD/gGr5/JbeAP4R9L6it1/r5/edhFjWrcBMb1iATUDnXOa9CliSY7tDJYhHyZ58KwBHyf0E6b7g7y3nb47sCaIT7gBaImj6v4HHT+Z7zhHHGd66T/d+Z4cDv4ET+J3NJv8E0S2fOIJ/e3n9b68ELvaGhwDTwtnOaL+siOnkXaWqZwS9Xgvzc/VwRVB7Ai/cGVRNb3odIGQRUBjOBDYGvd+IO2jXDD17SL8EDR/CHbDCXraqHgT6A3cCW73igSbe5HrAC0HbvQt3IKsdIo789kNuceIVEa30ihL24A4S1fL4bFnxisfyWXY94I85vrs6uH2T0y1AY+BHEVksIlfksh3Z9quqHsCdNQfvk1y3NYePgPYiUgt3xp8BzAMQkZoiMkFENovIPuAdsu+T3JyJSzSB+A568eEtt7GI/Ncr2tkH/D3M5WYuW1UzgsZt5CS23Su+edorvtmHS3h4sVQDyhL693Qq/28QtG+8OPL67eW1rrdwVx94f8efQkwRYwmi4G0C1udILpVU9fKg6bmWs+ZjC+4gFlAXVxyw7eTDPfFlq+p0Vb0YV7z0IxBInpuAO3JsezlVXRBifSe1H7wy3z8B/YDKqnoGsBeXiE7VJuDJHPGXV9V/55xRVdeo6kCgBvAM8KGIVAixzGz71ZunKrD5RINT1d3A57gEfS3uzF+9yX/HnfG2UNXTcAehcPbJVtyBLRBfeS++gFdw33Ejb7l/DnO54La9jogEH4fqchLbjtveXsBFuINy/UDIuCK8I4T+PeX1OzsIlA96/7sQ8wT2bzi/vbzW9Q7QS0QScEWwk3OZr0BZgih43wD7ReRhr+IsTkSai0gbb/rrwN9EpJE4LUUk8A+5DVcHkJt/A/eLSAMRqYg7KLyvkbm7Iqxle2eqvbwD3W+4orjAGeJoYFig8lBcBfo1uazvXeAiEeknIiVFpKqIJIYRZyVc4toBlBSRR4HTTmxTc/UacKeItPO+mwoi0kNEKuWcUUSuF5Hq3tnxHm90Rs75cPv1JhFJFJEyuP36tapuOMkY3wNuAPp6wwGVcN/FXhGpjSsPD8eHwBVeRW5pYDjZjxuVcHVOB7wrxbtyfD6v3+zXuKuCP4mrSO8KXAlMCDO2YJVwv7eduIP63wMTvO/gTeBf4irz40Skvbe/8/qdLQX6iEh5cbfq3hJGDHn99nL931bVVGAx7srhI1U9fBL7IOIsQZy8TyT7cxCTwvmQumcFrgASgfW4s5vXcWc94CoBP8CdCe7DlVGX86Y9DrzlFW/0C7H4N3E/sLneso8Avz/xTQsp3GWXAB7AnR3uArrgHTRUdRLubHqCVwzwA3BZqJWp6s+4Muc/estZiqvUy8904DNgNa644gg5igFOlqqmALcBL+PKltfiyqVD6Q4sF3f32AvAgFD/9Opulf4rrnhoK+4Mc8AphDkFaAT8oqrLgsb/H64ifC8wFXeTQb5UdTlwDy7ZbMVtd2rQLA/izt734xLo+zkW8Ti5/GZV9SguIVyG+z8YBdygqj+GE1sOb+O+783ACmBRjukP4iqIF+N+T8/g6j7y+p09h6tv2YYrAno3nxjy++3l9b+Nt44WxEjxEoBkXYEaY4zxi4h0xhU11dMYOTDbFYQxxvhMREoBf8DdtRUTyQEsQRhjjK9E5FxcPVUt3PMbMcOKmIwxxoRkVxDGGGNCKjKN9VWrVk3r16/vdxjGGFOofPvtt7+qavVQ04pMgqhfvz4pKSl+h2GMMYWKiGzMbZoVMRljjAnJEoQxxpiQLEEYY4wJKaoJQkS6i8gqcZ2hDA0xvZ6IzBDXOctsEYkPmvYPcR2IrBSRF0WO7yTFGGNM9EQtQYjrwH4krp2VpsBAEWmaY7YRwNuq2hLXCNhT3mc7AOcDLYHmQBtcmz7GGGMKSDSvINoCa1X1J69Rrgm45niDNcX1ggUwK2i64tpvL43r9akUkWmy2hhjTJiimSBqk70lw1SO7xhmGdDHG+4NVBKRqqq6EJcwtnqv6aq6MucKROR2cf3+puzYsSPiG2CMMcWZ35XUDwJdRGQJrghpM5Dutb1+LhCPSyrdJKjz7wBVHaOqyaqaXL16yOc8jDERcPgwzJgBzzwD774LP/zgd0TFz6FDkJoKu3e79zt3wquvwtNPQ7RaTIrmg3KbCeqJCnewz9ZTlKpuwbuC8DqhuVpV94jIbbiO5A940z4F2uN1n2iMia6jR6F0aTf80EPw4otuXEDDhrB2rRv+17/cAaplS/eqeSId3BZDGRlQogSkpcGCBe5Av3Mn7Nrl/nbpApdfDtu2wUUXufG7dsGRI+7z//oX3H8//Por3HmnG3fPPVDpuG6rTl00E8RioJGINMAlhgG4jkUyiUg1YJfX49MwXKc0AD8Dt4nIU7ju+roQY60cGlOU/PYbfPMNzJrlXl9/7c5Wq1RxB/1774ULLoB27WDrVnfACnj/fffZgBo14Prr4Z//dO9XrYL69aFMmQLdpAKxfXv2A/yuXVC7Nlx8sZs+YADs2JE9Adx8M7z0kksUXXLcelO6NJQv7xJExYpw9tlQtar7HqpUccMdOrh5GzbM+o7KlSMqopYgVDVNRIbgelmKA95U1eUiMhxIUdUpQFfgKRFRXE9l93gf/xDohusBSoHPVPWTaMVqTHFz7Bikp0PZsjB5Mlx7rStGEoHERLjrrqwrhkGDsn+2atXs77/+2p3N/u9/8P337lWrlpuWluYSTEYGNGmSdZVxySWQlBTtrTx1KSmwbp07EAdeZ53litrA7autW7N/pk+frATx88/uaqFuXbe9VapA585uWunSrtiucmW3T6tWdckhcEN/hQowKY9+KkuWdMkomopMc9/JyclqbTEZE1pamjvYzZ7trhDmz4dXXoEbbnBFRS+95K4QOnd2B7FIOXrUHeQCieP7791B8+9/h2HD3Bn4gAFZiSMhAZo2jd4ZMbiimrJl3fCnn8KSJbB5c1YCqFYNpk9309u2hcWL3XCFClCnDnTrBiNHunEffOCSX+DsvkoV9/loFPdEi4h8q6rJIadZgjCm6ElPhz173EFr1y6oVw8OHHDTmjVzyeDGGyE55GEhunbvdnUWVarAjz/C4MHu6uPQITe9RAl34L36and2/u23LnnUqZN1dh2Kqlv21q1uG8Et5/PP3YE/kARKlXKJCdw6Pv7YxVK7NsTHu88++6ybvnSpO9OvXRtOOy3v9RdWeSWIItOaqzHFWXo6LFuWdYUwd64r5vjwQ3fwu/ded3betaurI/BT5cpZw02awKJFLv6ffnJXGMuWuVjBHdwHD3bDZ5yRdaUxdKg7aL/7LrzxRtbZ/+HDbt7Dh91VwqJFMHWqO/CffbYr869TxyUTEXcX0PjxrmgnlMTEKO2EQsKuIIwphDIyYONGaNDAve/aFebMccONG7v3PXpAz55+RRgZBw5kFU0tW5Y1vHatu1vqjTdg7Niss//4eDfcq5erFA8kApM7K2IyhUpGhisCiIsDe7zFUYXly7OuEObMcWfJe/a4IpOJE115f9eu0a+49FvgNlETGVbEZGLK0aOuOGDjRldhuXGju/y/9lpXmXraaVlFBeec44oFrrsu6+6P4iBQZHTOOa5y9JlnXKUuuPqEK6909Qjp6S5BXHONv/EWJEsOBccShIm4AwfcQT84AdSo4R7uAXeb4ObN2T8zaJBLECVLuvLlKlVckpgzByZMcAfKzp3dlcWwYe5MuUsXd/tgUXD0qHuWYN48V3+wYAHs2+fusune3RUV1azptjtQrGRMtFkRkzlhO3fC+vVZSWDjRndgDzwY1a5d9genSpVyB7kpU9z7MWPc/PXquVd8fNZth6Gkp7sDaLly8NVX7uw50NxA/fruoPnnP0OjRtHY2ug4cAAWLnQH/ZYt3a2WrVq5ac2aQadOLiFecsnxzx0YE0lWB2FOyLZt7unX4KuAPXvcLYOQdWtgQMWK7iGguXPd+8mT3dl/IAH87neuPiFSMjLcbZFz5rjX3LnubpWGDV1Z/JQp7uqiSxdXdBULlZQZGfDf/7pY586F775zie+uu2DUKDf8ySfQsaO7j96YgmIJwuTr2DF3pg9w333wwgtZ02rUcGfqX33lzvznzcu6t75uXXfbop8H4YwMt34RePll+Nvfsu5zP/NMlyjeftvFXlA2b3b76fBhuOkmV8lct65rdqFtW3d10KkTtG/v6lyM8YslCJOr3btdkc9LL7mz7/bt3dn51q1ZCSCaT7VGg6p7ACtwhbF9u2vSAOCWW2D/fpc0unZ1T+1GKrlNmeKeGp47193TD+4+/5VeQ/WrVrl9mldxmjEFze5iMsdZu9ZdJbz5pnuC9aKLsq4gWrRwr8JKBM49170CrV0GlCvnHr6aONG9r1YN7rgDnnjCvQ/nvvmMDHfL6dy5rh2isWNdEdr06a6YqFMnGDLEXSUEHvgCV9FuTGFiVxDF0NGj7l75vXvd7aP33+8qSosLVVfJHrjCaNEC/vhHVxzUsKErAgrUYSQkZNWfzJ8P//iH+xuoJK9d2xW9BZqyKF/ebsM0hYtdQRRzx465JhcmT4Z//9u1LfPee9C8eVarm8WJiLvV9qyzXP1AwP79cNllLmn85z9uXKVKrqmGTp3g4EFXdNWnT1YdQv36WVccFSsW+KYYE1WWIIqwPXvgtddcZy+pqa6IY/Nm1xZNoDlik6VGDdd0A7j9FbjjKNDI3SWXwOrV/sVnTEGzBFFELV3qbpk8eNA1Tzx6tDs7tuKP8MTHuwf3rg3q4ioWbpc1piBZgigiVN3Tt7/84p5TaN4cbr3VtYRZ3FukNMacnKieT4pIdxFZJSJrRWRoiOn1RGSGiHwvIrNFJN4bf4GILA16HRGRq6IZa2GVlua6fDzvPHfF8Ne/umRRsiQ8/7wlB2PMyYtaghCROGAkcBnQFBgoIk1zzDYCeFtVWwLDgacAVHWWqiaqaiKu69FDwOfRirWw+s9/3F03Awa4u2pGjXK9X1lRiDEmEqJ5BdEWWKuqP6nqUWAC0CvHPE2Bmd7wrBDTAfoCn6rqoahFWoisX5/VB+5pp7k7caZMcXfX3HWXa/nTGGMiIZoJojawKeh9qjcu2DKgjzfcG6gkIjmbJhsA/DvUCkTkdhFJEZGUHTt2RCDk2LVwoWvS+eyz4emn3bgLLnB9A1x5pVU+G2Miz+/DyoNAFxFZAnQBNgPpgYkiUgtoAUwP9WFVHaOqyaqaXL2I9iwzaZJr/qJDB/jyS/jTn9zLGGOiLZp3MW0G6gS9j/fGZVLVLXhXECJSEbhaVfcEzdIPmKSqx6IYZ8w5eDCrqGjKFNfA20svuTuS7GEsY0xBieYVxGKgkYg0EJHSuKKiKcEziEg1EQnEMAx4M8cyBpJL8VJRtHGja/LhzDMh0GrI88+7Rt6GDLHkYIwpWFFLEKqaBgzBFQ+tBD5Q1eUiMlxEAl2pdwVWichqoCbwZODzIlIfdwUyJ1oxxoqMDHjqKXdH0gsvuM7mK1Vy004/PbJ9KRhjTLissT6fqboH2yZNgn794Nlni043msaY2GeN9cUwEXfFcMklrtlpe4bBGBMrLEH4ICMDRoxwjeYNHOg6sTHGmFjj922uxc7One65hYcfhi++8DsaY4zJnV1BFKAFC6B/f9cF5siR7slnY4yJVZYgCsiaNa6Hsrp1XaJo3drviIwxJm9WxBRl6d5z4Y0auT4ZvvvOkkNhoOoeUAw0oW5MLMrIgGXLXC+I0WBXEFG0aBHceCNMmABJSVYZHYsOH3ZXd6tXuwcSg/8G+p2uUMG1f3X33dbmlfFXenpWQpgzx/V4uHu361N+2bLIr88SRBSownPPuYro+Hj33vgnPR02bTo+AaxaBT//nH3e+Hho3Ng1od64MTRo4JpR//3vXaJ/4w3XdasxBSEtDZYscclg9myYPx/27nXTGjaE3r1d0XWXLtFZvyWICNu927WZNGWK+/LefBPOOMPvqIqHnTtDXwmsWQO//ZY132mnuYN8p07ub+PG7m+jRqGbS+/ZE95+G+6/HxIS4LHH4MEHoVSpgts2UzwcO+aa2QlcIXz1Fezf76Y1buwepg0khPj46MdjCSLCRo+GTz91bSjde689+BZpR47A2rWhE8HOnVnzlSzpzrAaN4bu3bMngho1Tux7EXFFhZde6trE+vOfYeJEdzWRlBT5bTTFx2+/uU6+AglhwQLXWCfAuefC9de7ZNC5M9SqVfDxWVMbEaAKmze7jJ6WBsuXuzNNc/KOHIF5845PBBs3Zi+yq1Ur+8E/8Ld+/eid4X/0EdxzD/z6q2t6/dFHoWzZ6KzLFC1HjsDXX2cVGS1c6MYBtGiRdXXQubM7kSkIeTW1YQniFO3eDTffDN98Az/8AJUrF3gIRc6RI64zpEWL3PuKFd2BP3DwDySCxo2zGjUsaLt2uZZ3x41z8bzxBpx/vj+xmNh16JBLAoErhK+/dlcNIu4kMpAQOnWCatX8idHaYoqSlBTXy1tqKjzzjNU1RIKqu1to0SJXXHflle4qIdaK6qpUgbFjXWX27be7f/AhQ+Dvf7dm2YuzAwdcMVEgIXzzjatXKFHCFUfecw907QodOxaOk0lLECdBFV5+2Z1B1qrlikLOO8/vqIqGl192B96//tU1XhjrLr3UXTn++c8u9ilTYMwY1/iiKfr27XMVyYEio2+/dcXMcXGQnOxubOjSxV1dnn6639GeOCtiOgkZGVn9QL/1ljubNKdu5kx3YO3RwzV/XtieOZg/H2691dWVDB4M//pX4ThLNCdm40b3+5w0yX3nGRmuvqtNm6wiow4d/Cv+PFG+1UGISHfgBSAOeF1Vn84xvR6uF7nqwC7gelVN9abVBV7HdRqkwOWquiG3dRVEgvjuO6he3bXCevAglC8fe0UfhdX69e4frEYNV7x02ml+R3RyjhyB4cPhH/9wv5WRI6FPH7+jMqdq5Ur4+GOXFL791o1r3tzdAn3BBa7f+FC3SBcGeSUIVDUqL1xSWAecBZQGlgFNc8wzEbjRG+4GjA+aNhu42BuuCJTPa32tW7fWaMnIUB05UrV0adW+faO2mmLrwAHVli1VzzhDdfVqv6OJjO++U01MVAXVq69W3brV74gi48AB9/9Q1GVkqC5erDpsmGqTJu57BNXzzlN95pmi8ztVVQVSNLfjeG4TTvUFtAemB70fBgzLMc9yoI43LMA+b7gpMP9E1hetBLF3r2q/fm5PXX656o4dUVlNsZWRoXrNNaolSqh++qnf0UTW0aOqf/+7apkyqpUrq44bVzgPrqtWqT71lGqbNu7/4Mwz3f/Eiy+6RJiW5neEkZGWpjp7tuq996rWqeO2NS5OtVs31ZdfVk1N9TvC6PArQfTFFSsF3g8CXs4xz3vAH7zhPriipKrAVcB/gY+BJcCzQFyIddwOpAApdevWjfiOW71atWFD9yN5+mnV9PSIr6LYe/JJ9yv8xz/8jiR6Vq5U7dDBbeell6pu2OB3RHnLyHAH/r/8RbVp06yz5zZtVB95RPXaa1Xr1s0aX6mS6iWXqA4frjpzpurBg35vQfiOHFGdOlX11ltVq1d321OmjOqVV6qOHav6669+Rxh9sZwgzgxKAi8AqcAZ3mf3esVTJYGPgFvyWl80riD27VO94ALVefMivmijqp98oiqiOnBg4TyzPhHp6aovvaRaoYJqxYrujDSWTjjS0tzv/P77VevXd0eGEiVUu3Z1Vwo//3z8ZzZuVH33XdW77lJt0cJ9l6BasqRqu3aqf/yj6qRJqtu3F/jm5Gn/ftUPPnC/u0qVspLcgAFu/P79fkdYsGK2iCnH/BWBVG/4PGBO0LRBwMi81hepBLFvnyt3PHQoIoszuVixwv1TtmpVuM44T9X69e5sG1Q7dlT98Uf/YvntN9XPPlO9/XbVGjVcTKVLq/boofr66yd+YN+1y52NDxum2qmTOxMPXGWcc47qLbe4s/I1awr+hODXX926e/bMiqtaNRfT1KnuSqK48itBlAR+AhoEVVI3yzFPNaCEN/wkMNwbjvPmr+69Hwvck9f6IpEgli5VbdTInTlNm3bKizO52L3b7efq1d1ZaHGTkeEOVmec4Q5WTz3l6isKwoEDqh99pHrddaqnn+6OABUquDqFCRNcnVukHDmi+tVXrlL3yitdPUwgYdSs6Srvn3tONSVF9dixyK03IDXVXaldeKErJgZXt3Dvva6uIRrrLIx8SRBuvVwOrPbuZnrEGzcc6OkN9wXWePO8DpQJ+uzFwPfA/4BxQOm81nUqCSIjQ3XMGNWyZVVr1VKdM+ekF2XykZametllrhhi7ly/o/HX1q2qffq4/8KkJNUlS6Kznl27VN9+W/Wqq1TLlXPrq1JF9aabVKdMUT18ODrrzSk9XfWHH1RHj1a9/vqsoqxAkrroItXHH1f98suTL+ZZs8YlpHbtsl+9DBvm7koq6kWZJ8O3BFGQr1NJEI8/7vbExRerbtt20osxYRg61O3rV17xO5LY8eGH7ow6Lk71z3+OzAF7yxa3jy++2CVjUK1dW3XIENUZM2Ln7HnTJnflMmSIuy04UI8RF6eanKx6331u//zyS+jPZ2S4K/9HH3X1IIGk0Lq16hNPuKJMk7e8EoQ9SY3rL+DDD10HP4Xt6d3CZMIEGDjQtV306qt+RxNbdu2CBx5wT+Y3aeIa/+vQ4cSW8dNP7kGujz92DcSpwtlnw9VXu75J2rSJ/d/33r3uQcl589xTyl9/ndXa6dlnuzavOnaEevVcs/qTJrntFnHTevd2r3r1/N2OwsRaczW+W7LEtUfTqpVrUqN0ab8jik3Tp7sEumlT/o3/qbp2oAJJIdDlZGKie3q7d29o1qxwP+1/9KhrwWD+/KxXoN+PUqXgoovcdvbsCTVr+htrYWUJwvhq+3Z39pqR4VrAtX/kvO3fn9X4X7168NprcPHFblpGhmshNJAU1q51CaBDh6yk0KCBv/FHk6pr62rdOnclURgbwIs1liCMb44dc2d533zjig2SQ7f4YkIIbvzvxhvdlcSkSbBli+sxr1s3lxR69YLf/c7vaE1hZf1BGN/cdx/MnQvvvGPJ4UR17AhLl2Y1/le6tOs+tU8f1+KttRRros0ShIma11+HUaPgwQfhuuv8jqZwKlvW1UPcd59rLbSwthhqCidLECYqFixwPcNdcgk8/XT+85u8FVT/xMYEi/Gb3kxhlJrqikHq1XO3tsbF+R2RMeZk2BWEiajDh92dNAcPwowZVk5uTGFmCcJEjKrrRzolBSZPdvfgG2MKLytiMhHz/PMwfjz83/+5Wy+NMYWbJQgTEV984e5W6t0b/vIXv6MxxkSCJQhzytatg/79oWlT15ZQrLf3Y4wJj/0rm1Oyf39WcdLkyVCpkq/hGGMiyCqpzUnLyHBNQKxc6RqZa9jQ74iMMZFkCcKctCeecG0D/etfrr0lY0zREtUiJhHpLiKrRGStiAwNMb2eiMwQke9FZLaIxAdNSxeRpd5rSjTjNCdu8mR47DG44QbXDIQxpuiJ2hWEiMQBI3Fdh6YCi0VkiqquCJptBPC2qr4lIt2Ap4BB3rTDqpoYrfjMyVu+HAYNck14v/pq4e5vwBiTu2heQbQF1qrqT6p6FJgA5Lw7vikw0xueFWK6iTG7drlK6UDT02XL+h2RMSZaopkgagObgt6neuOCLQP6eMO9gUoiUtV7X1ZEUkRkkYhcFWoFInK7N0/Kjh07Ihi6CSUtDQYMgJ9/ho8+gto5v01jTJHi922uDwJdRGQJ0AXYDKR70+p5nVhcCzwvIsfdI6OqY1Q1WVWTq1evXmBBF1dDh7oH4l555cT7SzbGFD7RvItpM1An6H28Ny6Tqm7Bu4IQkYrA1aq6x5u22fv7k4jMBpKAdVGM1+Th3Xfhn/+Ee+6BW27xOxpjTEGI5hXEYqCRiDQQkdLAACDb3UgiUk1EAjEMA970xlcWkTKBeYDzgeDKbVOAUlJc15ddusBzz/kdjTGmoEQtQahqGjAEmA6sBD5Q1eUiMlxEenqzdQVWichqoCbwpDf+XCBFRJbhKq+fznH3kykg27a59pVq1ICJE6FUKb8jMsYUFFFVv2OIiOTkZE1JSfE7jCLl6FHo1g2++871EJeY6HdExphIE5Fvvfre49iT1CZXv/89fPWV6xXOkoMxxU++RUwicmVQPYEpJkaPhjFj3J1L/fv7HY0xxg/hHPj7A2tE5B8i0iTaARn/zZ3rrh4uv9y1t2SMKZ7yTRCqej1Zt5iOE5GF3gNq1rBzEbRpE/TtC2edBe+9B3FxfkdkjPFLWEVHqroP+BDXXEYt3FPP34nI76MYm/HBE0+4Ph7+8x84/XS/ozHG+CmcOoieIjIJmA2UAtqq6mVAAvDH6IZnCtKePfDOO3DttdDEChONKfbCuYvpauA5VZ0bPFJVD4mIPVNbhLz9Nhw65J6WNsaYcBLE48DWwBsRKQfUVNUNqjojWoGZgqUKo0ZBu3bQqpXf0RhjYkE4dRATgYyg9+neOFOEzJoFq1bB3Xf7HYkxJlaEkyBKev05AOANl45eSMYPI0dC1arQr5/fkRhjYkU4CWJHUNtJiEgv4NfohWQKWmqqu2vpllusAyBjTJZw6iDuBN4VkZcBwXUCdENUozIF6rXXICMD7rzT70iMMbEk3wShquuA87z+GlDVA1GPyhSYo0ddkxqXXw4NGvgdjTEmloTVWJ+I9ACa4boBBUBVh0cxLlNAJk+GX36xymljzPHCeVBuNK49pt/jipiuAepFOS5TQEaNclcOl17qdyTGmFgTTiV1B1W9Aditqv8HtAcah7NwEekuIqtEZK2IDA0xvZ6IzBCR70VktojE55h+moikevUfJsJ++AHmzIG77rI2l4wxxwsnQRzx/h4SkTOBY7j2mPIkInHASOAyoCkwUESa5phtBPC2qrYEhgNP5Zj+N2AuJipeeQXKlIGbbvI7EmNMLAonQXwiImcAzwLfARuA98L4XFtgrar+5D07MQHolWOepsBMb3hW8HQRaY3rhvTzMNZlTtD+/a5pjf79oVo1v6MxxsSiPBOE11HQDFXdo6of4eoemqjqo2EsuzbultiAVG9csGVAH2+4N1BJRKp66/0n8GA+8d0uIikikrJjx44wQjIB48fDgQPW7pIxJnd5JghVzcAVEwXe/6aqeyO4/geBLiKyBOgCbMY15XE3ME1VU/OJb4yqJqtqcvXq1SMYVtEWaHepdWto08bvaIwxsSqc21xniMjVwMeqqiew7M1AnaD38d64TKq6Be8KwnvO4mpV3SMi7YFOInI3UBEoLSIHVPW4im5z4ubNg+XL4Y03wLtr2RhjjhNOgrgDeABIE5EjuFtdVVVPy+dzi4FGItIAlxgGANcGzyAi1YBd3pXKMOBN3MKvC5pnMJBsySFyRo6EypVhwAC/IzHGxLJwuhytpKolVLW0qp7mvc8vOaCqacAQYDqwEvhAVZeLyPCgtp26AqtEZDWuQvrJk94SE5atW+Hjj92dS+XL+x2NMSaW5XsFISKdQ43P2YFQLvNMA6blGPdo0PCHuK5M81rGOGBcfusy4Xn9dUhLs3aXjDH5C6eI6aGg4bK421e/BbpFJSITNWlp8OqrcMkl0KiR39EYY2JdOI31XRn8XkTqAM9HKyATPVOmwObN7g4mY4zJTzgPyuWUCpwb6UBM9I0aBXXrQo8efkdijCkMwqmDeAkI3N5aAkjEPVFtCpEff4QZM+DJJ63dJWNMeMKpg0gJGk4D/q2qX0UpHhMlr7wCpUrBrbf6HYkxprAIJ0F8CBxR1XRwjfCJSHlVPRTd0EykHDwI48bBNddAjRp+R2OMKSzCqYOYAZQLel8O+DI64ZhoeO892LfPOgUyxpyYcBJE2eBuRr1he8SqkFB1T04nJECHDn5HY4wpTMJJEAdFpFXgjdcM9+HohWQiaeFCWLbMXT1Yu0vGmBMRTh3EfcBEEdmCa4fpd7guSE0hMGoUnHYaXHtt/vMaY0ywcB6UWywiTYBzvFGrVPVYdMMykbB9O0yc6JrVqFjR72iMMYVNvkVMInIPUEFVf1DVH4CKXjPcJsa98QYcPer6nDbGmBMVTh3Ebaq6J/BGVXcDt0UtIhMR6ekwejR06wZNmvgdjTGmMAonQcSJZFVvikgcUDp6IZlImDoVfv7ZuhQ1xpy8cCqpPwPeF5FXvfd3AJ9GLyQTCaNGwZlnQs+e+c9rjDGhhHMF8TAwE7jTe/2P7A/O5UpEuovIKhFZKyLH9QgnIvVEZIaIfC8is0UkPmj8dyKyVESWi4j1XnAC1q6F6dPhjjugZDinAMYYE0I4PcplAF8DG3B9QXTD9RCXJ68oaiRwGdAUGCgiTXPMNgJ4W1VbAsOBp7zxW4H2qpoItAOGisiZYWyPwbW7VLIk3GY1RcaYU5Dr+aWINAYGeq9fgfcBVPWCMJfdFlirqj95y5sA9AJWBM3TFNffNcAsYLK3jqNB85Th5JolL5YOHYKxY6FPH6hVy+9ojDGFWV4H3h9xVwtXqGpHVX0JSD+BZdcGNgW9T/XGBVsG9PGGewOVRKQquI6JROR7bxnPqOqWE1h3sfX++7B7t7W7ZIw5dXkliD64op5ZIvKaiFyIe5I6kh4EuojIEqALsBkvCanqJq/o6WzgRhGpmfPDInK7iKSISMqOHTsiHFrhNGoUNGsGnUP2JG6MMeHLNUGo6mRVHQA0wRX/3AfUEJFXROSSMJa9GagT9D7eGxe8ji2q2kdVk4BHvHF7cs4D/AB0ChHjGFVNVtXk6tWrhxFS0fbNN5CSYu0uGWMiI5xK6oOq+p7XN3U8sAR3Z1N+FgONRKSBiJQGBgBTgmcQkWoiEohhGPCmNz5eRMp5w5WBjsCqMLep2Bo1yjWpcf31fkdijCkKTqjyV1V3e2ftF4YxbxowBJiOu+vpA1VdLiLDRSRwd35XYJWIrAZqAk96488FvhaRZcAcYISq/u9EYi1udu6ECRNg0CDXOJ8xxpyqqN4lr6rTgGk5xj0aNPwhrse6nJ/7AmgZzdiKmjffhN9+s8ppY0zk2O2jRUBGhnv2oXNnaN7c72iMMUWFJYgiYPp0WL/erh6MMZFlCaIIGDkSfvc76N3b70iMMUWJJYhCbv16mDbNNatR2trYNcZEkCWIQu7VV6FECbj9dr8jMcYUNZYgCrEjR+D116FXL4iP9zsaY0xRYwmiEJs40T3/YJXTxphosARRiI0aBeec47oVNcaYSLMEUUh99x0sWmTtLhljoscSRCE1ahSULw833OB3JMaYosoSRCG0eze89x5cdx2ccYbf0RhjiipLEIXQuHFw+LBVThtjossSRCETaHepQwdITPQ7GmNMUWYJopCZMQPWrLGrB2NM9FmCKGRGjYLq1aFvX78jMcYUdZYgCpGff4YpU+DWW6FMGb+jMcYUdVFNECLSXURWichaERkaYno9EZkhIt+LyGwRiffGJ4rIQhFZ7k3rH804C4sxY0AV7rjD70iMMcVB1BKEiMQBI4HLgKbAQBFpmmO2EcDbqtoSGA485Y0/BNygqs2A7sDzInJGtGItDI4ehddegyuugHr1/I7GGFMcRPMKoi2wVlV/UtWjwASgV455mgIzveFZgemqulpV13jDW4DtQPUoxhrzPvoItm+He+7xOxJjTHERzQRRG9gU9D7VGxdsGdDHG+4NVBKRqsEziEhboDSwLucKROR2EUkRkZQdO3ZELPBYNGoUNGwIF1/sdyTGmOLC70rqB4EuIrIE6AJsBtIDE0WkFjAeuElVM3J+WFXHqGqyqiZXr150LzC+/x7mz4e77nJ9PxhjTEEoGcVlbwbqBL2P98Zl8oqP+gCISEXgalXd470/DZgKPKKqi6IYZ8wbNQrKloWbbvI7EmNMcRLN89HFQCMRaSAipYEBwJTgGUSkmogEYhgGvOmNLw1MwlVgfxjFGGPe3r3wzjswcCBUqeJ3NMaY4iRqCUJV04AhwHRgJfCBqi4XkeEi0tObrSuwSkRWAzWBJ73x/YDOwGARWeq9EqMVaywbPx4OHrQnp40xBU9U1e8YIiI5OVlTUlL8DiOiVKFZM6hUCb7+2u9ojDFFkYh8q6rJoaZFsw7CnKLZs2HlStd6qzHGFDS7JyaGjRrl6h369fM7EmNMcWQJIkZt3gyTJsEtt0C5cn5HY4wpjixBxKjXXnN9P1i7S8YYv1iCiEHHjrmG+bp3d09PG2OMH6ySOgb95z+wdau7ijDGGL/YFUQMGjkS6td3VxDGGOMXSxAxZsUKd3vrnXdCXJzf0RhjijNLEDHmlVegdGm4+Wa/IzHGFHeWIGLI/v3w1lvQv7/rd9oYY/xkCSKGvPuuSxLW7pIxJhZYgogRqu7J6aQkaNfO72iMMcZuc40Z48bB//4Hr78OIn5HY4wxdgURE955xzWpcdFFcP31fkdjjDGOJQifvfce3HgjXHCBe0CuTBm/IzLGGCeqCUJEuovIKhFZKyJDQ0yvJyIzROR7EZktIvFB0z4TkT0i8t9oxuin99+HQYOgc2eYMgXKl/c7ImOMyRK1BCEiccBI4DKgKTBQRJrmmG0ErlvRlsBw4Kmgac8Cg6IVn98mToTrroOOHeG//4UKFfyOyBhjsovmFURbYK2q/qSqR4EJQK8c8zQFZnrDs4Knq+oMYH8U4/PNxx+7Pqbbt4epUy05GGNiUzQTRG1gU9D7VG9csGVAH2+4N1BJRKqGuwIRuV1EUkQkZceOHacUbEGZPNk9CNeuHUybBhUr+h2RMcaE5vdtrg8CL4vIYGAusBlID/fDqjoGGAOuT+poBBhJn3zieodLToZPP3V9TRd2x44dIzU1lSNHjvgdijEmD2XLliU+Pp5SpUqF/ZloJojNQJ2g9/HeuEyqugXvCkJEKgJXq+qeKMbkm//+F66+2j0I99lncNppfkcUGampqVSqVIn69esj9gCHMTFJVdm5cyepqak0aNAg7M9Fs4hpMdBIRBqISGlgADAleAYRqSYigRiGAW9GMR7fTJvmkkNCAkyfDqef7ndEkXPkyBGqVq1qycGYGCYiVK1a9YSv9KOWIFQ1DRgCTAdWAh+o6nIRGS4iPb3ZugKrRGQ1UBN4MvB5EZkHTAQuFJFUEbk0WrFG0/Tp0KcPNG8On38OZ5zhd0SRZ8nBmNh3Mv+nUa2DUNVpwLQc4x4NGv4Q+DCXz3aKZmwF4YsvoFcvOPdcN1y5st8RGWNM+OxJ6iiZMQN69oQmTeDLL6FKFb8jKpouuOACpk+fnm3c888/z1133ZXrZ7p27UpKSgoAl19+OXv27Dlunscff5wRI0bkue7JkyezYsWKzPePPvooX3755QlEX3wF9vuePXsYNWpU5vjZs2dzxRVXRHx9KSkp3HvvvRFfLoT3W4mmilG8FdISRBTMmgVXXgmNGrnkUDXsG3fNiRo4cCATJkzINm7ChAkMHDgwrM9PmzaNM06y3C9nghg+fDgXXXTRSS3LL+npYd80GFGB/Z4zQURLcnIyL774YtTXU9RYgoiwOXPgiivgrLPcVUS1an5HVHDuuw+6do3s67778l5n3759mTp1KkePHgVgw4YNbNmyhU6dOnHXXXeRnJxMs2bNeOyxx0J+vn79+vz6668APPnkkzRu3JiOHTuyatWqzHlee+012rRpQ0JCAldffTWHDh1iwYIFTJkyhYceeojExETWrVvH4MGD+fBDV2I6Y8YMkpKSaNGiBTfffDO//fZb5voee+wxWrVqRYsWLfjxxx+Pi2nDhg106tSJVq1a0apVKxYsWJA57ZlnnqFFixYkJCQwdKhrvWbt2rVcdNFFJCQk0KpVK9atW3fcmfiQIUMYN25cZgwPP/wwrVq1YuLEiSG3D2Dbtm307t2bhIQEEhISWLBgAY8++ijPP/985nIfeeQRXnjhhWzxP/vss5kH4/vvv59u3boBMHPmTK677rps+33o0KGsW7eOxMREHnroIQAOHDhA3759adKkCddddx2qx9/B3rVrVx5++GHatm1L48aNmTdvHuBumrjpppto0aIFSUlJzJo1C8h+ZTJnzhwSExNJTEwkKSmJ/fv3Z8bdpk0bWrZsmevv5bPPPqNVq1YkJCRw4YUXZo5fsWIFXbt25ayzzsqWiK666ipat25Ns2bNGDNmTOb4ihUr8sgjj5CQkMB5553Htm3bABg8eDD33nsvHTp04Kyzzsr8PYUT39atW+ncuTOJiYk0b948c5+cElUtEq/WrVur3+bOVa1QQfXcc1V/+cXvaArGihUrMof/8AfVLl0i+/rDH/KPoUePHjp58mRVVX3qqaf0j3/8o6qq7ty5U1VV09LStEuXLrps2TJVVe3SpYsuXrxYVVXr1aunO3bs0JSUFG3evLkePHhQ9+7dqw0bNtRnn31WVVV//fXXzHU98sgj+uKLL6qq6o033qgTJ07MnBZ4f/jwYY2Pj9dVq1apquqgQYP0ueeey1xf4PMjR47UW2655bjtOXjwoB4+fFhVVVevXq2B3/a0adO0ffv2evDgwWzb17ZtW/34449VVfXw4cN68OBBnTVrlvbo0SNzmffcc4+OHTs2M4Znnnkmc1pu29evX7/MuNPS0nTPnj26fv16TUpKUlXV9PR0Peuss7J9XlV14cKF2rdvX1VV7dixo7Zp00aPHj2qjz/+uI4ePTrbfl+/fr02a9Ys87OzZs3S0047TTdt2qTp6el63nnn6bx5847bR126dNEHHnhAVVWnTp2qF154oaqqjhgxQm+66SZVVV25cqXWqVNHDx8+nG1/XHHFFTp//nxVVd2/f78eO3ZMp0+frrfddptmZGRoenq69ujRQ+fMmZNtndu3b9f4+Hj96aefsu3/xx57TNu3b69HjhzRHTt2aJUqVfTo0aPZ5jl06JA2a9Ysc18BOmXKFFVVfeihh/Rvf/ubqrrfUN++fTU9PV2XL1+uDRs2VFXNM74KFSpkbvsTTzyR+X3t27fvuP0W/P8aAKRoLsdVvx+UKzK++gouuwzq1IGZM6FmTb8jKnhBJ5YFKlDM1KtXLyZMmMAbb7wBwAcffMCYMWNIS0tj69atrFixgpYtW4Zcxrx58+jduzflvRYTe/bsmTnthx9+4C9/+Qt79uzhwIEDXHpp3jfUrVq1igYNGtC4cWMAbrzxRkaOHMl93uVQnz6u8YDWrVvz8ccfH/f5Y8eOMWTIEJYuXUpcXByrV68G4Msvv+Smm27KjLFKlSrs37+fzZs307t3b8A9DBWO/v3757t9M2fO5O233wYgLi6O008/ndNPP52qVauyZMkStm3bRlJSElVzlKG2bt2ab7/9ln379lGmTBlatWpFSkoK8+bNC6uYp23btsTHu3Y7ExMT2bBhAx07djxuvuD9uGHDBgDmz5/P73//ewCaNGlCvXr1MvdfwPnnn88DDzzAddddR58+fYiPj+fzzz/n888/JykpCXBXMWvWrKFz586Zn1u0aBGdO3fOfI6gSlDFYo8ePShTpgxlypShRo0abNu2jfj4eF588UUmTZoEwKZNm1izZg1Vq1aldOnSmVc0rVu35osvvshc1lVXXUWJEiVo2rRp5pVFOPG1adOGm2++mWPHjnHVVVeRmJiY777OjyWICFi4ELp3h9q1XXL43e/8jqh46dWrF/fffz/fffcdhw4donXr1qxfv54RI0awePFiKleuzODBg0/6ae/BgwczefJkEhISGDduHLNnzz6leMt4bbrHxcWRlpZ23PTnnnuOmjVrsmzZMjIyMsI+6AcrWbIkGRkZme9zbnuFoAbATnT7br31VsaNG8cvv/zCzTfffNz0UqVK0aBBA8aNG0eHDh1o2bIls2bNYu3atZx77rn5xl4mqM373PZR8Hx5zRPK0KFD6dGjB9OmTeP8889n+vTpqCrDhg3jjjvuCHs5+cU8e/ZsvvzySxYuXEj58uXp2rVr5vdQqlSpzNtOc8YfvCz1itfCia9z587MnTuXqVOnMnjwYB544AFuuOGGk9qeAKuDOEVffw2XXuqSwsyZUKuW3xEVPxUrVuSCCy7g5ptvzqyc3rdvHxUqVOD0009n27ZtfPrpp3kuo3PnzkyePJnDhw+zf/9+Pvnkk8xp+/fvp1atWhw7dox33303c3ylSpUyy6+DnXPOOWzYsIG1a9cCMH78eLp06RL29uzdu5datWpRokQJxo8fn1mRfPHFFzN27NjMOoJdu3ZRqVIl4uPjmTx5MgC//fYbhw4dol69eqxYsYLffvuNPXv2MGPGjFzXl9v2XXjhhbzyyiuAq8zeu3cvAL179+azzz5j8eLFuV5NderUiREjRtC5c2c6derE6NGjSUpKOu5e/Nz24cnq1KlT5jasXr2an3/+mXPOOSfbPOvWraNFixY8/PDDtGnThh9//JFLL72UN998kwMHDgCwefNmtm/fnu1z5513HnPnzmX9+vWA2/952bt3L5UrV6Z8+fL8+OOPLFq06KS3K5z4Nm7cSM2aNbntttu49dZb+e677056fQGWIE7B4sVwySVQo4a7c6l2zqYITYEZOHAgy5Yty0wQCQkJJCUl0aRJE6699lrOP//8PD/fqlUr+vfvT0JCApdddhlt2rTJnPa3v/2Ndu3acf7559OkSZPM8QMGDODZZ58lKSmJdevWZY4vW7YsY8eO5ZprrqFFixaUKFGCO++8M+xtufvuu3nrrbdISEjgxx9/zDzb7969Oz179iQ5OZnExMTMWyvHjx/Piy++SMuWLenQoQO//PILderUoV+/fjRv3px+/fplFk2Ektv2vfDCC8yaNYsWLVrQunXrzDu2SpcuzQUXXEC/fv2Ii4sLucxOnTqxdetW2rdvT82aNSlbtiydOh3/aFPVqlU5//zzad68eWYl9am4++67ycjIoEWLFvTv359x48ZlOyMHdxt08+bNadmyJaVKleKyyy7jkksu4dprr6V9+/a0aNGCvn37Hpe4qlevzpgxY+jTpw8JCQnZiulC6d69O2lpaZx77rkMHTqU884776S3K5z4Zs+enfm7f//99/nDH/5w0usLkMAlTGGXnJysgXvbC8K337ouQqtUgdmzXd1DcbRy5cqwig1M0ZGRkZF5B1SjRo38DsecgFD/ryLyraomh5rfriBOwpIlcPHFrtmMWbOKb3Iwxc+KFSs4++yzufDCCy05FANWSX2Cli51Vw6VKrnkULeu3xEZU3CaNm3KTz/95HcYpoDYFcQJ+P57lxwqVHDFSvXr+x2RMcZEjyWIMP3wA1x4IZQr564cTqBJdWOMKZQsQYRh+XLo1g1Kl3bJoWFDvyMyxpjoswSRj5UrXXIoWdIlh7PP9jsiY4wpGJYg8rBqlUsOIu4hOK/lBBNDrLnvwqmgm/uOpuDfU0GL9v6KaoIQke4iskpE1orI0BDT64nIDBH5XkRmi0h80LQbRWSN97oxmnGGsno1XHABZGS4K4eg54dMDLHmvk9NcWnuO+BEmuQwUUwQIhIHjAQuA5oCA0WkaY7ZRgBvq2pLYDjwlPfZKsBjQDugLfCYiBRYf2xr1rjkkJbmkoM9Bxa+UE12B/7/Dx0KPd1rhZpffz1+Wn6sue/i19z3li1bMpvrTkxMJC4ujo0bN7Jjxw6uvvpq2rRpQ5s2bfjqq68AdzU4aNAgzj//fAYNGsSGDRvo1q0bLVu25MILL+Tnn38GYOLEiTRv3pyEhIRsjeAFC7X/A5/N2fR4bt/j7Nmz6dq1a8htzO33cfDgQW6++Wbatm1LUlIS//nPf46LLbdmzE9Jbs28nuoLaA9MD3o/DBiWY57lQB1vWIB93vBA4NWg+V4FBua1vkg19712rWrt2qrVqqn+738RWWSRlrP54FBNdo8c6aYdPBh6utcKte7Ycfy0cFhz38Wvue+Al19+Wa+55hpVVR04cGDmvBs3btQmTZqoqmuOu1WrVnro0CFVdc19jxs3TlVV33jjDe3Vq5eqqjZv3lxTU1NVVXX37t3HrSu3/Z9b0+O5fY95bWNuv49hw4bp+PHjM2Nr1KiRHjhwIN9mzHOKpea+awObgt6n4q4Igi0D+gAvAL2BSiJSNZfPHtfSkYjcDtwOUDcCT6z99JO7cjhyxNU5NG9+yossdvJqCLR8+bynV6uW9/TcWHPfxbO576+++orXXnuN+fPnZ+6f4CK/ffv2ZTZu17NnT8qVKwfAwoULM/f7oEGD+NOf/gS4ZsAHDx5Mv379Mr+jYKH2f0Copsdz+x7z28ZQv4/PP/+cKVOmZNaLHTlyJPPKJyBUM+anyu8nqR8EXhaRwcBcYDMQdqGoqo4BxoBri+lUAtmwwSWHgwddT3C5HEdMDLLmvo9X1Jv73rp1K7fccgtTpkzJ7JM5IyODRYsWhdxfwdubm9GjR/P1118zderUzCSXM/nlF3NwvHl9j3ltY6hlqSofffTRcS3TBvqLgNDNmDc5xcrTaFZSbwaCWymK98ZlUtUtqtpHVZOAR7xxe8L5bCRt3OiSw7598MUXEIF+NkwBsua+i1dz38eOHeOaa67hmWeeybxKA9fi6UsvvZT5funSpSE/36FDh8wbG959993MVmbXrVtHu3btGD58ONWrV2fTpk3ZPhdq/+clt+/xZFx66aW89NJLmXUVS5YsOW6eUM2Yn6poJojFQCMRaSAipYEBwJTgGUSkmogEYhgGvOkNTwcuEZHKXuX0Jd64iNu82SWH3bvhyy+hVatorMVEmzX3XXya+16wYAEpKSk89thjmZWyW7Zs4cUXXyQlJYWWLVvStGlTRo8eHfLzL730EmPHjqVly5aMHz8+s5L9oYceokWLFjRv3pwOHTqQkJCQ7XO57f/c5PY9noy//vWvHDt2jJYtW9KsWTP++te/HjdPqGbMT1lulROReAGXA6uBdcAj3rjhQE9vuC+wxpvndaBM0GdvBtZ6r5vyW9fJVlLv26d6xRWq33xzUh8v9kJVepmiLT09XRMSEnT16tV+h2JOUCxVUqOq04BpOcY9GjT8IfBhLp99k6wriqipVAmCShOMMXlYsWIFV1xxBb1797bmvosBvyupjTGFiDX3XbxYUxvmlGkR6ZXQmKLsZP5PLUGYU1K2bFl27txpScKYGKaq7Ny584RvmbYiJnNK4uPjSU1NZceOHX6HYozJQ9myZU/44TlLEOaUBB6KMsYUPVbEZIwxJiRLEMYYY0KyBGGMMSYkKSp3n4jIDmCj33GcomrAr34HEUNsf2Rn+yOL7YvsTmV/1FPV6qEmFJkEURSISIqqJvsdR6yw/ZGd7Y8sti+yi9b+sCImY4wxIVmCMMYYE5IliNgyxu8AYoztj+xsf2SxfZFdVPaH1UEYY4wJya4gjDHGhGQJwhhjTEiWIGKAiNQRkVkiskJElovIH/yOyW8iEiciS0Tkv37H4jcROUNEPhSRH0VkpYi09zsmP4nI/d7/yQ8i8m8RObEmSgs5EXlTRLaLyA9B46qIyBcissb7WzkS67IEERvSgD+qalPgPOAeEWnqc0x++wOw0u8gYsQLwGeq2gRIoBjvFxGpDdwLJKtqcyAO1999cTIO6J5j3FBghqo2AmZ470+ZJYgYoKpbVfU7b3g/7gBQ29+o/CMi8UAPXD/lxZqInA50Bt4AUNWjqrrH16D8VxIoJyIlgfLAFp/jKVCqOhfYlWN0L+Atb/gt4KpIrMsSRIwRkfpAEvC1z6H46XngT0CGz3HEggbADmCsV+T2uohU8Dsov6jqZmAE8DOwFdirqp/7G1VMqKmqW73hX4CakVioJYgYIiIVgY+A+1R1n9/x+EFErgC2q+q3fscSI0oCrYBXVDUJOEiEig8KI69svRcucZ4JVBCR6/2NKraoe3YhIs8vWIKIESJSCpcc3lXVj/2Ox0fnAz1FZAMwAegmIu/4G5KvUoFUVQ1cUX6ISxjF1UXAelXdoarHgI+BDj7HFAu2iUgtAO/v9kgs1BJEDBARwZUxr1TVf/kdj59UdZiqxqtqfVzl40xVLbZniKr6C7BJRM7xRl0IrPAxJL/9DJwnIuW9/5sLKcaV9kGmADd6wzcC/4nEQi1BxIbzgUG4s+Wl3utyv4MyMeP3wLsi8j2QCPzd33D8411JfQh8B/wPdwwrVs1uiMi/gYXAOSKSKiK3AE8DF4vIGtxV1tMRWZc1tWGMMSYUu4IwxhgTkiUIY4wxIVmCMMYYE5IlCGOMMSFZgjDGGBOSJQhj8iEi6UG3Hy8VkYg9ySwi9YNb5TQmlpT0OwBjCoHDqprodxDGFDS7gjDmJInIBhH5h4j8T0S+EZGzvfH1RWSmiHwvIjNEpK43vqaITBKRZd4r0EREnIi85vVx8LmIlPPmv9frI+R7EZng02aaYswShDH5K5ejiKl/0LS9qtoCeBnXCi3AS8BbqtoSeBd40Rv/IjBHVRNw7Skt98Y3AkaqajNgD3C1N34okOQt587obJoxubMnqY3Jh4gcUNWKIcZvALqp6k9eY4u/qGpVEfkVqKWqx7zxW1W1mojsAOJV9begZdQHvvA6ekFEHgZKqeoTIvIZcACYDExW1QNR3lRjsrErCGNOjeYyfCJ+CxpOJ6tusAcwEne1sdjrIMeYAmMJwphT0z/o70JveAFZ3WBeB8zzhmcAd0Fmn9un57ZQESkB1FHVWcDDwOnAcVcxxkSTnZEYk79yIrI06P1nqhq41bWy18rqb8BAb9zvcT3APYTrDe4mb/wfgDFe65vpuGSxldDigHe8JCLAi9bVqCloVgdhzEny6iCSVfVXv2MxJhqsiMkYY0xIdgVhjDEmJLuCMMYYE5IlCGOMMSFZgjDGGBOSJQhjjDEhWYIwxhgT0v8DzR61Emech/UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "val_acc_noise = history_noise.history[\"val_accuracy\"]\n",
    "val_acc_zeros = history_zeros.history[\"val_accuracy\"]\n",
    "epochs = range(1, 11)\n",
    "plt.plot(epochs, val_acc_noise, \"b-\",\n",
    "         label=\"Validation accuracy with noise channels\")\n",
    "plt.plot(epochs, val_acc_zeros, \"b--\",\n",
    "         label=\"Validation accuracy with zeros channels\")\n",
    "plt.title(\"Effect of noise channels on validation accuracy\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### The nature of generalization in deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Fitting a MNIST model with randomly shuffled labels**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 2.3166 - accuracy: 0.1024 - val_loss: 2.3064 - val_accuracy: 0.1047\n",
      "Epoch 2/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.2993 - accuracy: 0.1176 - val_loss: 2.3165 - val_accuracy: 0.1005\n",
      "Epoch 3/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.2911 - accuracy: 0.1289 - val_loss: 2.3214 - val_accuracy: 0.1000\n",
      "Epoch 4/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.2775 - accuracy: 0.1404 - val_loss: 2.3303 - val_accuracy: 0.0978\n",
      "Epoch 5/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.2613 - accuracy: 0.1526 - val_loss: 2.3459 - val_accuracy: 0.0963\n",
      "Epoch 6/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.2401 - accuracy: 0.1650 - val_loss: 2.3504 - val_accuracy: 0.0978\n",
      "Epoch 7/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.2175 - accuracy: 0.1814 - val_loss: 2.3687 - val_accuracy: 0.0988\n",
      "Epoch 8/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.1907 - accuracy: 0.1967 - val_loss: 2.3948 - val_accuracy: 0.0995\n",
      "Epoch 9/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.1632 - accuracy: 0.2123 - val_loss: 2.4109 - val_accuracy: 0.1015\n",
      "Epoch 10/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.1311 - accuracy: 0.2278 - val_loss: 2.4240 - val_accuracy: 0.1022\n",
      "Epoch 11/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.1002 - accuracy: 0.2433 - val_loss: 2.4552 - val_accuracy: 0.0966\n",
      "Epoch 12/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.0671 - accuracy: 0.2571 - val_loss: 2.4705 - val_accuracy: 0.1037\n",
      "Epoch 13/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.0319 - accuracy: 0.2725 - val_loss: 2.4846 - val_accuracy: 0.1021\n",
      "Epoch 14/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.9985 - accuracy: 0.2877 - val_loss: 2.5318 - val_accuracy: 0.0994\n",
      "Epoch 15/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.9630 - accuracy: 0.3014 - val_loss: 2.5566 - val_accuracy: 0.0997\n",
      "Epoch 16/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.9285 - accuracy: 0.3203 - val_loss: 2.5990 - val_accuracy: 0.1032\n",
      "Epoch 17/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.8944 - accuracy: 0.3323 - val_loss: 2.6311 - val_accuracy: 0.1003\n",
      "Epoch 18/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.8603 - accuracy: 0.3441 - val_loss: 2.6582 - val_accuracy: 0.0992\n",
      "Epoch 19/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.8274 - accuracy: 0.3593 - val_loss: 2.6914 - val_accuracy: 0.1028\n",
      "Epoch 20/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.7931 - accuracy: 0.3737 - val_loss: 2.7441 - val_accuracy: 0.0951\n",
      "Epoch 21/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.7608 - accuracy: 0.3828 - val_loss: 2.7779 - val_accuracy: 0.1016\n",
      "Epoch 22/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.7276 - accuracy: 0.3994 - val_loss: 2.8308 - val_accuracy: 0.1028\n",
      "Epoch 23/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.6972 - accuracy: 0.4100 - val_loss: 2.8713 - val_accuracy: 0.1013\n",
      "Epoch 24/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.6641 - accuracy: 0.4227 - val_loss: 2.9216 - val_accuracy: 0.1018\n",
      "Epoch 25/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.6353 - accuracy: 0.4331 - val_loss: 2.9670 - val_accuracy: 0.0983\n",
      "Epoch 26/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.6054 - accuracy: 0.4449 - val_loss: 2.9713 - val_accuracy: 0.1029\n",
      "Epoch 27/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.5781 - accuracy: 0.4556 - val_loss: 3.0303 - val_accuracy: 0.1028\n",
      "Epoch 28/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.5498 - accuracy: 0.4663 - val_loss: 3.0564 - val_accuracy: 0.1026\n",
      "Epoch 29/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.5226 - accuracy: 0.4777 - val_loss: 3.1116 - val_accuracy: 0.1040\n",
      "Epoch 30/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.4941 - accuracy: 0.4884 - val_loss: 3.1660 - val_accuracy: 0.1023\n",
      "Epoch 31/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.4680 - accuracy: 0.4973 - val_loss: 3.2570 - val_accuracy: 0.0995\n",
      "Epoch 32/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.4450 - accuracy: 0.5040 - val_loss: 3.2385 - val_accuracy: 0.1009\n",
      "Epoch 33/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.4187 - accuracy: 0.5171 - val_loss: 3.2980 - val_accuracy: 0.1013\n",
      "Epoch 34/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.3947 - accuracy: 0.5267 - val_loss: 3.3550 - val_accuracy: 0.0978\n",
      "Epoch 35/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.3725 - accuracy: 0.5322 - val_loss: 3.4100 - val_accuracy: 0.0997\n",
      "Epoch 36/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.3473 - accuracy: 0.5412 - val_loss: 3.4421 - val_accuracy: 0.1062\n",
      "Epoch 37/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.3244 - accuracy: 0.5516 - val_loss: 3.5479 - val_accuracy: 0.1003\n",
      "Epoch 38/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.3018 - accuracy: 0.5562 - val_loss: 3.5577 - val_accuracy: 0.1047\n",
      "Epoch 39/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.2808 - accuracy: 0.5645 - val_loss: 3.5910 - val_accuracy: 0.0995\n",
      "Epoch 40/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.2587 - accuracy: 0.5739 - val_loss: 3.6299 - val_accuracy: 0.1050\n",
      "Epoch 41/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.2392 - accuracy: 0.5789 - val_loss: 3.6809 - val_accuracy: 0.1018\n",
      "Epoch 42/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.2187 - accuracy: 0.5888 - val_loss: 3.7466 - val_accuracy: 0.1038\n",
      "Epoch 43/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.1970 - accuracy: 0.5972 - val_loss: 3.8097 - val_accuracy: 0.1001\n",
      "Epoch 44/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.1757 - accuracy: 0.6024 - val_loss: 3.8845 - val_accuracy: 0.0995\n",
      "Epoch 45/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.1590 - accuracy: 0.6080 - val_loss: 3.9297 - val_accuracy: 0.1000\n",
      "Epoch 46/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.1407 - accuracy: 0.6162 - val_loss: 3.9806 - val_accuracy: 0.1040\n",
      "Epoch 47/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.1203 - accuracy: 0.6249 - val_loss: 4.0239 - val_accuracy: 0.0997\n",
      "Epoch 48/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.1032 - accuracy: 0.6313 - val_loss: 4.0932 - val_accuracy: 0.1012\n",
      "Epoch 49/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.0841 - accuracy: 0.6375 - val_loss: 4.1457 - val_accuracy: 0.1012\n",
      "Epoch 50/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.0689 - accuracy: 0.6417 - val_loss: 4.1934 - val_accuracy: 0.0989\n",
      "Epoch 51/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.0506 - accuracy: 0.6484 - val_loss: 4.2415 - val_accuracy: 0.0980\n",
      "Epoch 52/100\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1.0353 - accuracy: 0.6546 - val_loss: 4.2852 - val_accuracy: 0.1018\n",
      "Epoch 53/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 1.0193 - accuracy: 0.6594 - val_loss: 4.3500 - val_accuracy: 0.1042\n",
      "Epoch 54/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 1.0027 - accuracy: 0.6651 - val_loss: 4.4077 - val_accuracy: 0.1047\n",
      "Epoch 55/100\n",
      "375/375 [==============================] - 2s 6ms/step - loss: 0.9875 - accuracy: 0.6706 - val_loss: 4.4866 - val_accuracy: 0.1013\n",
      "Epoch 56/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.9699 - accuracy: 0.6794 - val_loss: 4.5609 - val_accuracy: 0.1007\n",
      "Epoch 57/100\n",
      "375/375 [==============================] - 4s 12ms/step - loss: 0.9564 - accuracy: 0.6829 - val_loss: 4.5814 - val_accuracy: 0.1028\n",
      "Epoch 58/100\n",
      "375/375 [==============================] - 5s 14ms/step - loss: 0.9405 - accuracy: 0.6883 - val_loss: 4.6568 - val_accuracy: 0.1021\n",
      "Epoch 59/100\n",
      "375/375 [==============================] - 4s 12ms/step - loss: 0.9277 - accuracy: 0.6907 - val_loss: 4.7038 - val_accuracy: 0.1031\n",
      "Epoch 60/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.9146 - accuracy: 0.6969 - val_loss: 4.7646 - val_accuracy: 0.1016\n",
      "Epoch 61/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.8989 - accuracy: 0.7015 - val_loss: 4.7985 - val_accuracy: 0.0996\n",
      "Epoch 62/100\n",
      "375/375 [==============================] - 2s 6ms/step - loss: 0.8823 - accuracy: 0.7071 - val_loss: 4.8513 - val_accuracy: 0.1041\n",
      "Epoch 63/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.8715 - accuracy: 0.7122 - val_loss: 4.9590 - val_accuracy: 0.1023\n",
      "Epoch 64/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.8565 - accuracy: 0.7160 - val_loss: 5.0354 - val_accuracy: 0.0992\n",
      "Epoch 65/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.8428 - accuracy: 0.7221 - val_loss: 5.0515 - val_accuracy: 0.0997\n",
      "Epoch 66/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.8337 - accuracy: 0.7261 - val_loss: 5.1318 - val_accuracy: 0.1012\n",
      "Epoch 67/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.8181 - accuracy: 0.7318 - val_loss: 5.1931 - val_accuracy: 0.1009\n",
      "Epoch 68/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.8073 - accuracy: 0.7331 - val_loss: 5.2220 - val_accuracy: 0.1028\n",
      "Epoch 69/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.7941 - accuracy: 0.7398 - val_loss: 5.3088 - val_accuracy: 0.1004\n",
      "Epoch 70/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.7829 - accuracy: 0.7448 - val_loss: 5.3738 - val_accuracy: 0.1003\n",
      "Epoch 71/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.7699 - accuracy: 0.7480 - val_loss: 5.4765 - val_accuracy: 0.1016\n",
      "Epoch 72/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.7589 - accuracy: 0.7508 - val_loss: 5.5142 - val_accuracy: 0.1028\n",
      "Epoch 73/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.7469 - accuracy: 0.7548 - val_loss: 5.5635 - val_accuracy: 0.0997\n",
      "Epoch 74/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.7368 - accuracy: 0.7600 - val_loss: 5.5918 - val_accuracy: 0.0997\n",
      "Epoch 75/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.7261 - accuracy: 0.7650 - val_loss: 5.6725 - val_accuracy: 0.1014\n",
      "Epoch 76/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.7149 - accuracy: 0.7661 - val_loss: 5.7551 - val_accuracy: 0.0982\n",
      "Epoch 77/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.7045 - accuracy: 0.7705 - val_loss: 5.7896 - val_accuracy: 0.1013\n",
      "Epoch 78/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.6910 - accuracy: 0.7751 - val_loss: 5.8679 - val_accuracy: 0.0996\n",
      "Epoch 79/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.6853 - accuracy: 0.7767 - val_loss: 5.9911 - val_accuracy: 0.0997\n",
      "Epoch 80/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.6736 - accuracy: 0.7805 - val_loss: 5.9759 - val_accuracy: 0.1036\n",
      "Epoch 81/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.6646 - accuracy: 0.7826 - val_loss: 6.0790 - val_accuracy: 0.0991\n",
      "Epoch 82/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.6537 - accuracy: 0.7887 - val_loss: 6.1283 - val_accuracy: 0.1009\n",
      "Epoch 83/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.6463 - accuracy: 0.7906 - val_loss: 6.2121 - val_accuracy: 0.1021\n",
      "Epoch 84/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.6339 - accuracy: 0.7946 - val_loss: 6.3019 - val_accuracy: 0.0990\n",
      "Epoch 85/100\n",
      "375/375 [==============================] - 2s 5ms/step - loss: 0.6272 - accuracy: 0.7966 - val_loss: 6.3109 - val_accuracy: 0.1013\n",
      "Epoch 86/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.6190 - accuracy: 0.7988 - val_loss: 6.3926 - val_accuracy: 0.1015\n",
      "Epoch 87/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.6085 - accuracy: 0.8034 - val_loss: 6.4724 - val_accuracy: 0.1027\n",
      "Epoch 88/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.5995 - accuracy: 0.8077 - val_loss: 6.5479 - val_accuracy: 0.1000\n",
      "Epoch 89/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5921 - accuracy: 0.8070 - val_loss: 6.6154 - val_accuracy: 0.1021\n",
      "Epoch 90/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5811 - accuracy: 0.8124 - val_loss: 6.6705 - val_accuracy: 0.1000\n",
      "Epoch 91/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5752 - accuracy: 0.8148 - val_loss: 6.7685 - val_accuracy: 0.0992\n",
      "Epoch 92/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5671 - accuracy: 0.8166 - val_loss: 6.8105 - val_accuracy: 0.1023\n",
      "Epoch 93/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5571 - accuracy: 0.8217 - val_loss: 6.8937 - val_accuracy: 0.1005\n",
      "Epoch 94/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5505 - accuracy: 0.8227 - val_loss: 6.9668 - val_accuracy: 0.1004\n",
      "Epoch 95/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5414 - accuracy: 0.8267 - val_loss: 6.9899 - val_accuracy: 0.0994\n",
      "Epoch 96/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5359 - accuracy: 0.8274 - val_loss: 7.0765 - val_accuracy: 0.1013\n",
      "Epoch 97/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5280 - accuracy: 0.8299 - val_loss: 7.1295 - val_accuracy: 0.0974\n",
      "Epoch 98/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.5201 - accuracy: 0.8336 - val_loss: 7.1894 - val_accuracy: 0.1012\n",
      "Epoch 99/100\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 0.5142 - accuracy: 0.8363 - val_loss: 7.3118 - val_accuracy: 0.0993\n",
      "Epoch 100/100\n",
      "375/375 [==============================] - 2s 4ms/step - loss: 0.5055 - accuracy: 0.8375 - val_loss: 7.3680 - val_accuracy: 0.1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fc156a1ed68>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(train_images, train_labels), _ = mnist.load_data()\n",
    "train_images = train_images.reshape((60000, 28 * 28))\n",
    "train_images = train_images.astype(\"float32\") / 255\n",
    "\n",
    "random_train_labels = train_labels[:]\n",
    "np.random.shuffle(random_train_labels)\n",
    "\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(train_images, random_train_labels,\n",
    "          epochs=100,\n",
    "          batch_size=128,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### The manifold hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Interpolation as a source of generalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Why deep learning works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Training data is paramount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Evaluating machine-learning models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Training, validation, and test sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Simple hold-out validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### K-fold validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Iterated K-fold validation with shuffling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Beating a common-sense baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Things to keep in mind about model evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Improving model fit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Tuning key gradient descent parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Training a MNIST model with an incorrectly high learning rate**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 1154.7906 - accuracy: 0.4297 - val_loss: 2.0853 - val_accuracy: 0.3083\n",
      "Epoch 2/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 4.3056 - accuracy: 0.3195 - val_loss: 3.3618 - val_accuracy: 0.2767\n",
      "Epoch 3/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.9163 - accuracy: 0.3107 - val_loss: 2.2821 - val_accuracy: 0.3622\n",
      "Epoch 4/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.8953 - accuracy: 0.2950 - val_loss: 2.7679 - val_accuracy: 0.3620\n",
      "Epoch 5/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.9759 - accuracy: 0.2924 - val_loss: 1.9348 - val_accuracy: 0.2840\n",
      "Epoch 6/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.6346 - accuracy: 0.2918 - val_loss: 2.0249 - val_accuracy: 0.2534\n",
      "Epoch 7/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.6678 - accuracy: 0.2949 - val_loss: 1.9493 - val_accuracy: 0.2603\n",
      "Epoch 8/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.7168 - accuracy: 0.2840 - val_loss: 2.4809 - val_accuracy: 0.2940\n",
      "Epoch 9/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 2.5445 - accuracy: 0.2971 - val_loss: 2.1392 - val_accuracy: 0.3115\n",
      "Epoch 10/10\n",
      "375/375 [==============================] - 1s 4ms/step - loss: 2.5994 - accuracy: 0.2717 - val_loss: 2.3798 - val_accuracy: 0.2644\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fc1572e7e48>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(train_images, train_labels), _ = mnist.load_data()\n",
    "train_images = train_images.reshape((60000, 28 * 28))\n",
    "train_images = train_images.astype(\"float32\") / 255\n",
    "\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(1.),\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(train_images, train_labels,\n",
    "          epochs=10,\n",
    "          batch_size=128,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**The same model with a more appropriate learning rate**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "375/375 [==============================] - 2s 3ms/step - loss: 0.3626 - accuracy: 0.9126 - val_loss: 0.1781 - val_accuracy: 0.9550\n",
      "Epoch 2/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.1384 - accuracy: 0.9643 - val_loss: 0.1636 - val_accuracy: 0.9628\n",
      "Epoch 3/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.1155 - accuracy: 0.9728 - val_loss: 0.1628 - val_accuracy: 0.9710\n",
      "Epoch 4/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.0992 - accuracy: 0.9788 - val_loss: 0.1894 - val_accuracy: 0.9688\n",
      "Epoch 5/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.0871 - accuracy: 0.9813 - val_loss: 0.2148 - val_accuracy: 0.9688\n",
      "Epoch 6/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.0789 - accuracy: 0.9846 - val_loss: 0.2403 - val_accuracy: 0.9691\n",
      "Epoch 7/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.0715 - accuracy: 0.9863 - val_loss: 0.2534 - val_accuracy: 0.9692\n",
      "Epoch 8/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.0614 - accuracy: 0.9880 - val_loss: 0.2745 - val_accuracy: 0.9681\n",
      "Epoch 9/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.0629 - accuracy: 0.9892 - val_loss: 0.2923 - val_accuracy: 0.9703\n",
      "Epoch 10/10\n",
      "375/375 [==============================] - 1s 3ms/step - loss: 0.0603 - accuracy: 0.9900 - val_loss: 0.2647 - val_accuracy: 0.9743\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fc14fefdba8>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(1e-2),\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(train_images, train_labels,\n",
    "          epochs=10,\n",
    "          batch_size=128,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Leveraging better architecture priors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Increasing model capacity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**A simple logistic regression on MNIST**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.6620 - accuracy: 0.8367 - val_loss: 0.3574 - val_accuracy: 0.9020\n",
      "Epoch 2/20\n",
      "375/375 [==============================] - 0s 912us/step - loss: 0.3502 - accuracy: 0.9034 - val_loss: 0.3055 - val_accuracy: 0.9165\n",
      "Epoch 3/20\n",
      "375/375 [==============================] - 0s 863us/step - loss: 0.3152 - accuracy: 0.9121 - val_loss: 0.2904 - val_accuracy: 0.9197\n",
      "Epoch 4/20\n",
      "375/375 [==============================] - 0s 881us/step - loss: 0.2991 - accuracy: 0.9168 - val_loss: 0.2804 - val_accuracy: 0.9246\n",
      "Epoch 5/20\n",
      "375/375 [==============================] - 0s 917us/step - loss: 0.2896 - accuracy: 0.9189 - val_loss: 0.2765 - val_accuracy: 0.9250\n",
      "Epoch 6/20\n",
      "375/375 [==============================] - 0s 972us/step - loss: 0.2827 - accuracy: 0.9215 - val_loss: 0.2727 - val_accuracy: 0.9249\n",
      "Epoch 7/20\n",
      "375/375 [==============================] - 0s 858us/step - loss: 0.2781 - accuracy: 0.9223 - val_loss: 0.2687 - val_accuracy: 0.9272\n",
      "Epoch 8/20\n",
      "375/375 [==============================] - 0s 931us/step - loss: 0.2741 - accuracy: 0.9242 - val_loss: 0.2675 - val_accuracy: 0.9273\n",
      "Epoch 9/20\n",
      "375/375 [==============================] - 0s 877us/step - loss: 0.2710 - accuracy: 0.9245 - val_loss: 0.2652 - val_accuracy: 0.9275\n",
      "Epoch 10/20\n",
      "375/375 [==============================] - 0s 918us/step - loss: 0.2687 - accuracy: 0.9258 - val_loss: 0.2644 - val_accuracy: 0.9284\n",
      "Epoch 11/20\n",
      "375/375 [==============================] - 0s 915us/step - loss: 0.2665 - accuracy: 0.9261 - val_loss: 0.2643 - val_accuracy: 0.9273\n",
      "Epoch 12/20\n",
      "375/375 [==============================] - 0s 1ms/step - loss: 0.2649 - accuracy: 0.9267 - val_loss: 0.2628 - val_accuracy: 0.9301\n",
      "Epoch 13/20\n",
      "375/375 [==============================] - 0s 921us/step - loss: 0.2630 - accuracy: 0.9272 - val_loss: 0.2631 - val_accuracy: 0.9291\n",
      "Epoch 14/20\n",
      "375/375 [==============================] - 0s 941us/step - loss: 0.2614 - accuracy: 0.9275 - val_loss: 0.2626 - val_accuracy: 0.9301\n",
      "Epoch 15/20\n",
      "375/375 [==============================] - 0s 917us/step - loss: 0.2602 - accuracy: 0.9285 - val_loss: 0.2652 - val_accuracy: 0.9282\n",
      "Epoch 16/20\n",
      "375/375 [==============================] - 0s 945us/step - loss: 0.2592 - accuracy: 0.9290 - val_loss: 0.2607 - val_accuracy: 0.9305\n",
      "Epoch 17/20\n",
      "375/375 [==============================] - 0s 885us/step - loss: 0.2580 - accuracy: 0.9300 - val_loss: 0.2623 - val_accuracy: 0.9323\n",
      "Epoch 18/20\n",
      "375/375 [==============================] - 0s 886us/step - loss: 0.2573 - accuracy: 0.9302 - val_loss: 0.2613 - val_accuracy: 0.9298\n",
      "Epoch 19/20\n",
      "375/375 [==============================] - 0s 876us/step - loss: 0.2564 - accuracy: 0.9303 - val_loss: 0.2624 - val_accuracy: 0.9305\n",
      "Epoch 20/20\n",
      "375/375 [==============================] - 0s 876us/step - loss: 0.2552 - accuracy: 0.9303 - val_loss: 0.2609 - val_accuracy: 0.9312\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([layers.Dense(10, activation=\"softmax\")])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "history_small_model = model.fit(\n",
    "    train_images, train_labels,\n",
    "    epochs=20,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fc140d899b0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzxElEQVR4nO3dd5xU1fnH8c9DR0CUokG6yK6AVBc0tiDBxBbsCsaCWH4SSUOjJjGKNYkaNSTEHkuigpqIqBhiR2NjQZAiSBEUVKQpIIKU5/fHucMOw+zu7LIzd8v3/XrNa2dum2fu3L3PnHPPPcfcHRERkVS14g5AREQqJyUIERFJSwlCRETSUoIQEZG0lCBERCQtJQgREUmrxicIM7vBzFaa2efR65PM7BMzW29mvWOMq8Q4oun7ZjmGQ81sfvReJ5rZ3mY22czWmdmfzOw3ZnZfBtu5y8x+l81Yc8HMhprZGxku+6CZ3ZDtmCqCmT1vZufGHUdFMbP+ZrY06fVsM+ufybLleK+sHNtmNsrM/lnR2y2rOnEHkG1mthjYG9iaNPlBdx9hZu2AS4H27v5FNO9WYIS7P72L7+tAZ3dfUM5NlBiHuzcud3CZuw74q7v/GSD6R1gJ7O5luIHG3S+uiGCif/J/unubitieBO5+TOK5mQ0FLnD3w+KLqGK5e7eK2E66fVNRx3ZlVe0TRORH7v5imuntgFVJyQGgPTA7N2GVqDLEkRpDe2BOWZKDiFRh7l6tH8BiYGCa6QOBb4BtwHrgseivA18DC6Pl9gH+BawAPgJ+lrSN2sBvgIXAOmAq0BaYnLSd9cAZad6/FnAVsAT4AngYaArUTxdHmvUd2C96/iAwBnguiuMdoFM0z4Dbo/dYC8wEDojmvUr4RZTY5lDgjej5wmjffJO0fzYD30avBwKjCL/oE+sfBrwJfAl8AgxNiu+GpOWOB6ZHy70J9Ej5vi4D3ge+AsYBDYBGKd/XemCfNPvlQeBvwPPRMv8DvgPcAawB5gK9k5bvEu2HLwnJcFDSvObAhGi/vQtcn9g/0fz9gReA1cA84PSUOG5I991F8y8EPoi+rzlAn2j6lRQdT3OAk1K+n/8Bf432zVzg+0nzz0va5iLg/1Le84Rov6+N3uPo5OMg2hcbCaXt9dE+6QssB2onbedkYEYxn6sp4VheQTi2rwJqJR9fhNLxGsL/0zHFbOcK4MmUaX8GRpf2WYH+wNJ05wCgYfTdrIn2769Slk27/9Ptm2KO7QuBBdExMYGkY5TwP3sxMD/at2MAK+bzj2LH/61BhOPzy+j76pKyr5ZFMc9LHBNAP6Aw+r6XA7eV+fxZ0SfkyvagmASR7kBK+hITJ95ahJP+1UA9YN/oYPxhNP9XhBNuPuFE3BNonrqdYt57WHQg7Qs0Bv4N/CNdHMWsn5ogVkUHRB3gEWBsNO+H0WfYI4qxC9AqmvcqxSSIdPsuzT/D9oOYULpYBwwB6hJOrr1S1wN6E5LVQYQEe270PvWT3vNdQmJuRjgJXFzc95VmvzxIqAY7kJBYXiaciM6J3u8G4JVo2brRd/Cb6PsdEH2G/Gj+WOBxQnI6gPBPmEigjQhJ8Lxon/eO3rdrun2VEuNp0bb6Rt/JfoRqzsS8fQjH3hmEHwmtkr6fLcAvo9jPICSKZtH844BO0Ta/B2ygKPH0i5Y9Ktp2a2D/1OMg9RiIps0h6UQOPAVcWsxnexh4GmgCdAA+BM5P2vZmwkm0NjAc+JQ0J0nC8bQBaBK9rg18BhycwWfd4ThhxwTxB+B1wrHVFpiVsmxp+z9132z/ngnHz0qgD+GH3l+AySn/s88S/hfbEZLo0cXsx1EU/W/lRXEcFX3vlxOO23qEc88nRIko2ueJH4dvAWdHzxsn9l1ZHjXlIvV4M/sy6XFhhuv1BVq6+3Xu/q27LwLuBQZH8y8ArnL3eR7McPdVGW77x4SMvsjd1wO/BgabWXmr/Z5y93fdfQshQfSKpm8m/LPuT/hH/MDdPyvne5TkTOBFd3/M3Te7+yp3n55muYuAu939HXff6u4PAZuAg5OWGe3un7r7auCZpM+Sqafcfaq7bySczDa6+8PuvpVQIklc9D+Y8I/zh+j7fZnwDzzEzGoDpwBXu/vX7j4LeCjpPY4HFrv7A+6+xd3fI5Q0T8sgvguAm919SnTcLHD3JQDu/kT02be5+zjCr81+Set+AdwR7eNxhF+Mx0XrPufuC6Ntvgb8Fzg8Wu984O/u/kK07WXuPjfD/fkQcBaAmTUj/Oh4NHWhaJ8NBn7t7uvcfTHwJ+DspMWWuPu90XfxENCKcI1wB9H+mAacFE0aAGxw97cz+KwlOR240d1Xu/snwOiU9y1t/5fkx4R9PM3dNxH+p79rZh2SlvmDu3/p7h8Dr5DZsX0G8Fz03W0mlMAaAocQSjT1ga5mVtfdF7v7wmi9zcB+ZtbC3dcn9l1Z1JQEcaK775H0uDfD9doD+yQnF8KvzcQB3ZZQHC2PfQhF8IQlhF+iO/2zZOjzpOcbCCc+opPeXwnF2S/M7B4z272c71GSTPdFe+DSlH3alrA/EtJ+ljJYnvT8mzSvE9vbB/jE3bclzV9C+HXdkvB9fJIyL/lzHJTyOX5MqM4qTbH7yszOMbPpSds8AGiRtMgyj34SJsW0T7TuMWb2tpmtjtY9NmndXTlW/wn8yMwaEU6wrxfzI6MF4Rdu6nHdOun19u/W3TdET4v7fh8llEgh/ADZnpRK+awl2Yfiv9NM9n9p296+veiH3yqK+fxkfmynbndb9Blae2gE8wtCieMLMxtrZon/pfMJpY+5ZjbFzI7P8HNsV1MSRHl9AnyUklyauPuxSfM7lXPbnxJOMgntCNUHy9MvXn7uPtrdDwS6Eg6YX0WzvgZ2S1o0k5NbcTLdF58QfsEl79Pd3P2xDNb10hcpk0+BtmaW/H/QjlD9s4LwfbRNmZfwCfBayudo7O7DM3jftPvKzNoTSqgjCFWVexCqQCxpsdZmlvy6HfCpmdUnlGBuBfaO1p2YtG6m389O+9jdlxGqK04mlAb+Ucy6Kwm/WlOP62UZvG86TwD9zawNoSTxKEAGn7Ukn1HMd5rB/i/t+NvhfzpKqM0p/+cvbrtG+AzLANz9UQ8tq9pHMf4xmj7f3YcAe0XTnoxiypgSRMneBdaZ2RVm1tDMapvZAWbWN5p/H3C9mXW2oIeZNY/mLSdcXyjOY8AvzayjmTUGbgLGRVVEFcbM+prZQWZWl5AQNhIu9EK4YHmyme1mZvsRfnGU1yPAQDM73czqmFlzM+uVZrl7gYujmMzMGpnZcWbWJIP3WA40N7OmuxBnsncIv+IuN7O6UTPaHxGu32wlXBcaFe2froTrJQnPAnlmdna0bt1oX3fJ4H3vAy4zswOjfbBfdHJqRPgHXwFgZucRfsEm2wv4WfR+pxGuKU0k1EfXj9bdYmbHAD9IWu9+4Dwz+76Z1TKz1ma2f5rYlgNtzKxeyvSHCXXf3aP9spNonz0O3GhmTaLPNJJQAikzd19BuD7yAOGH2gfRrNI+a0keB35tZntGieenSfNK2//F7ZuExwj7uFeUxG4C3omq2nbF48Bx0XdXl9A0fxPwppnlm9mA6P02UtSQAzM7y8xaRiWOL6Ntbdt588WrKQniGQs3eyUeT2WyUnTAH0+oJ/yI8AvpPkJLDYDbCF/efwktBe4n1A1CKPI9FBVVT0+z+b8TfolNjra9kR0P1oqyO+GkvIZQTF0F3BLNu53QKmk5oT74kfK+SVSneizh4F1NSD490yxXSLhI+dcopgWEi3+ZvMdcwj/homi/7lPaOqVs71tCQjiG8N3+DTgnqW5+BKEK4HPCxcgHktZdRzgpDSb8wvuc8Cutfgbv+wRwI+EX8TpgPOFC8xxCnf1bhO+kO6HVUrJ3gM5RvDcCp3q43rMO+BnheFxDqJKZkPSe7xIuqN9OuFj9Gjv+0k94mdBa5nMzW5k0/alo+aeSqobS+Snhh8giQoulRwnHenk9Smgxt716qbTPWoprCf8HHxH+b7eXhjLY/8Xtm8T6LwK/I5RuPiOU2AanLldW7j6PcA3oL4Tv/UeEpvvfEo63P0TTPyf8gPh1tOrRwGwzW09oATbY3b8py3vbjtWZIlJZWcw3sZnZQkJz0nT3FEk1VFNKECKyC8zsFEL1y8txxyK5U1PupBaRcjKzVwkNHM5OafEl1ZyqmEREJC1VMYmISFrVpoqpRYsW3qFDh7jDEBGpUqZOnbrS3Vumm1dtEkSHDh0oLCyMOwwRkSrFzJYUNy+rVUxmdrSZzTOzBWZ2ZZr5F5vZzOjW9jeim5ES83qY2VsWBvuYaWYNshmriIjsKGsJwkLHXWMINyF1JXSA1jVlsUfdvbu79wJuJtx4hoUO6/5J6MWzG6F3xs3ZilVERHaWzRJEP2CBh95KvyV0nXxC8gLuvjbpZeI2dwh3qL7v7jOi5VZFdzWLiEiOZPMaRGt27DVxKWEMgB2Y2SWE/loS/fFD6FDOzWwSoVfNse5+c5p1LyJ0H027du1SZ4tIlm3evJmlS5eycePGuEORUjRo0IA2bdpQt27djNeJ/SK1u48BxpjZmYTRp84lxHUYYTyGDcBLZjbV3V9KWfce4B6AgoIC3dAhkmNLly6lSZMmdOjQgR07mZXKxN1ZtWoVS5cupWPHjhmvl80qpmXs2K1uG0ru9nYscGL0fClhJKaVUcdgEwmjNIlIJbJx40aaN2+u5FDJmRnNmzcvc0kvmwliCtA56s66HqFXwx16XDSzzkkvjyOM3gQwCegedbNchzCk4Jwsxioi5aTkUDWU53vKWhWTu28xsxGEk31twlB8s83sOqDQ3ScAI8xsIKGF0hqi/vbdfY2Z3UZIMg5MdPfnshWriIjsLKv3Qbj7RHfPc/dO7n5jNO3qKDng7j93927u3svdj3T32Unr/jOad4C7X56tGDdvhoED4e67s/UOIpItRx55JJMmTdph2h133MHw4cUP7Ne/f//tN9Uee+yxfPnllzstM2rUKG699dYS33v8+PHMmVNUsXH11Vfz4ou73hP6q6++yvHHl3l00Kyo8X0x1a0Ls2bBO+/EHYmIlNWQIUMYO3bsDtPGjh3LkCFDilljRxMnTmSPPfYo13unJojrrruOgQMHlmtblVWNTxAA+fnw4YdxRyEiZXXqqafy3HPP8e233wKwePFiPv30Uw4//HCGDx9OQUEB3bp145prrkm7focOHVi5MgwOd+ONN5KXl8dhhx3GvHnzti9z77330rdvX3r27Mkpp5zChg0bePPNN5kwYQK/+tWv6NWrFwsXLmTo0KE8+eSTALz00kv07t2b7t27M2zYMDZt2rT9/a655hr69OlD9+7dmTt37s5BJVm9ejUnnngiPXr04OCDD+b9998H4LXXXqNXr1706tWL3r17s27dOj777DOOOOIIevXqxQEHHMDrr7++azsXJQggJIik40FEyql//50ff/tbmLdhQ/r5Dz4Y5q9cufO80jRr1ox+/frx/PPPA6H0cPrpp2Nm3HjjjRQWFvL+++/z2muvbT+5pjN16lTGjh3L9OnTmThxIlOmTNk+7+STT2bKlCnMmDGDLl26cP/993PIIYcwaNAgbrnlFqZPn06nTp22L79x40aGDh3KuHHjmDlzJlu2bOHOO+/cPr9FixZMmzaN4cOHl1qNdc0119C7d2/ef/99brrpJs455xwAbr31VsaMGcP06dN5/fXXadiwIY8++ig//OEPmT59OjNmzKBXr16l78BSKEEAeXnh4Fy9Ou5IRKSskquZkquXHn/8cfr06UPv3r2ZPXv2DtVBqV5//XVOOukkdtttN3bffXcGDRq0fd6sWbM4/PDD6d69O4888gizZ88udjsA8+bNo2PHjuTl5QFw7rnnMnny5O3zTz75ZAAOPPBAFi9eXOK23njjDc4++2wABgwYwKpVq1i7di2HHnooI0eOZPTo0Xz55ZfUqVOHvn378sADDzBq1ChmzpxJkyZNStx2JmK/Ua4y6NMnXKheuxaaNYs7GpGq69VXi5+3224lz2/RouT5xTnhhBP45S9/ybRp09iwYQMHHnggH330EbfeeitTpkxhzz33ZOjQoeW+23vo0KGMHz+enj178uCDD/JqeYJMUr9+fQBq167Nli1byrWNK6+8kuOOO46JEydy6KGHMmnSJI444ggmT57Mc889x9ChQxk5cuT2Ekd5qQQBDBgAL7wAGk5CpOpp3LgxRx55JMOGDdteeli7di2NGjWiadOmLF++fHsVVHGOOOIIxo8fzzfffMO6det45plnts9bt24drVq1YvPmzTzyyCPbpzdp0oR169bttK38/HwWL17MggULAPjHP/7B9773vXJ9tsMPP3z7e7766qu0aNGC3XffnYULF9K9e3euuOIK+vbty9y5c1myZAl77703F154IRdccAHTpk0r13smUwkiiTvonh+RqmfIkCGcdNJJ26uaevbsSe/evdl///1p27Ythx56aInr9+nThzPOOIOePXuy11570bdv3+3zrr/+eg466CBatmzJQQcdtD0pDB48mAsvvJDRo0dvvzgNoc+jBx54gNNOO40tW7bQt29fLr744nJ9rlGjRjFs2DB69OjBbrvtxkMPPQSEpryvvPIKtWrVolu3bhxzzDGMHTuWW265hbp169K4cWMefvjhcr1nsmozJnVBQYHvyoBBxx4LTZvCY49VYFAi1dwHH3xAly5d4g5DMpTu+4r6uStIt7yqmCJ16oT7IUREJFCCiOTlwfz5sFWjToiIAEoQ2+Xnw6ZN8MknpS8rIkWqSzV1dVee70kJIpKfH/7qhjmRzDVo0IBVq1YpSVRyifEgGjRoUKb11Iop0qULnHceNG8edyQiVUebNm1YunQpK1asiDsUKUViRLmyUIKItGwJf/973FGIVC1169Yt0whlUrWoiimJu7rbEBFJUIJIcv750Lt33FGIiFQOShBJOnaEjz8OvU6KiNR0ShBJEi2Z5s8veTkRkZpACSJJ1DuvBg8SEUEJYgedO4e/uhdCRETNXHfQqBH88Y9QSsePIiI1ghJEissvjzsCEZHKQVVMKdauhXfeCfdEiIjUZEoQKR58EA4+GNRzgIjUdEoQKdRpn4hIoASRQk1dRUQCJYgU7dpB/foqQYiIKEGkqF0b9ttPCUJERM1c07j9dthjj7ijEBGJlxJEGkcdFXcEIiLxUxVTGqtWwZNPwsqVcUciIhIfJYg05s2D006Dt9+OOxIRkfhkNUGY2dFmNs/MFpjZlWnmX2xmM81supm9YWZdU+a3M7P1ZnZZNuNMlbgXQk1dRaQmy1qCMLPawBjgGKArMCQ1AQCPunt3d+8F3AzcljL/NuD5bMVYnObNoVkztWQSkZotmyWIfsACd1/k7t8CY4ETkhdw97VJLxsB23tAMrMTgY+A2VmMsVj5+SpBiEjNls0E0Rr4JOn10mjaDszsEjNbSChB/Cya1hi4Ari2pDcws4vMrNDMCldUcOdJ+fkqQYhIzRb7RWp3H+PunQgJ4apo8ijgdndfX8q697h7gbsXtGzZskLj+t3v4PXXK3STIiJVSjbvg1gGtE163SaaVpyxwJ3R84OAU83sZmAPYJuZbXT3v2Yj0HT23TdX7yQiUjllswQxBehsZh3NrB4wGJiQvICZdU56eRwwH8DdD3f3Du7eAbgDuCmXyQHg66/hjjvg3Xdz+a4iIpVH1hKEu28BRgCTgA+Ax919tpldZ2aDosVGmNlsM5sOjATOzVY8ZVWnDowcCc89F3ckIiLxyGpXG+4+EZiYMu3qpOc/z2Aboyo+stLVrw8dOqglk4jUXLFfpK7M1JJJRGoyJYgSJO6F0PjUIlITKUGUIC8PNmyAzz+POxIRkdxTgijBueeG1kytWsUdiYhI7mk8iBI0ahR3BCIi8VEJohS/+x3cd1/cUYiI5J4SRCmefRb+/e+4oxARyT0liFKoqauI1FRKEKXIy4PFi2HTprgjERHJLSWIUuTnw7ZtsHBh3JGIiOSWEkQp8vOhZUuo4OEmREQqPTVzLcWBB8IXX8QdhYhI7qkEUQqzuCMQEYmHEkQGrrsOhg2LOwoRkdxSgsjA0qXwzDNxRyEikltKEBnIz4eVK2H16rgjERHJHSWIDOTnh78aPEhEahIliAzk5YW/uqNaRGoSJYgMdOwIBQVhGFIRkZpC90FkoG5dmDIl7ihERHJLJQgREUlLCSJDf/kLtGkT+mUSEakJlCAy1KABLFsGH38cdyQiIrmhBJGhRFNXtWQSkZpCCSJDuhdCRGoaJYgM7bUX7L67ShAiUnOomWuGzODCC4tKEiIi1Z0SRBncemvcEYiI5I6qmMpowwbYujXuKEREsk8JogyefhoaNYJZs+KOREQk+5QgyqB9+/BXLZlEpCZQgiiDzp3DX7VkEpGaIKsJwsyONrN5ZrbAzK5MM/9iM5tpZtPN7A0z6xpNP8rMpkbzpprZgGzGmalGjUJ3G0oQIlITZC1BmFltYAxwDNAVGJJIAEkedffu7t4LuBm4LZq+EviRu3cHzgX+ka04yyo/X1VMIlIzZLOZaz9ggbsvAjCzscAJwJzEAu6+Nmn5RoBH099Lmj4baGhm9d19UxbjzciFF8K6dXFHISKSfdlMEK2BT5JeLwUOSl3IzC4BRgL1gHRVSacA09IlBzO7CLgIoF27dhUQcunOOCMnbyMiErvYL1K7+xh37wRcAVyVPM/MugF/BP6vmHXvcfcCdy9o2bJl9oMl3AMxfz6sWpWTtxMRiU02E8QyoG3S6zbRtOKMBU5MvDCzNsBTwDnuvjAbAZbHxx+HMaqfeiruSEREsiubCWIK0NnMOppZPWAwMCF5ATPrnPTyOGB+NH0P4DngSnf/XxZjLLN27cLY1GrJJCLVXdYShLtvAUYAk4APgMfdfbaZXWdmg6LFRpjZbDObTrgOcW5iOrAfcHXUBHa6me2VrVjLonZt2G8/tWQSkerP3D3uGCpEQUGBFxYW5uS9TjkFZs+GuXNz8nYiIlljZlPdvSDdvNgvUldFeXmwcCFs3hx3JCIi2aPuvsthyBDo2xeqSeFLRCQtJYhy6NEjPEREqjNVMZWDO7z8MsyYEXckIiLZowRRDmZw+unwt7/FHYmISPYoQZRTfr7uhRCR6k0JopzUq6uIVHdKEOWUlweffQZr15a+rIhIVZRRgjCzRmZWK3qeZ2aDzKxudkOr3PLzw1+VIkSkusq0BDEZaGBmrYH/AmcDD2YrqKqgf394913o1i3uSEREsiPTBGHuvgE4Gfibu58G1OhT4557hpvlGjaMOxIRkezIOEGY2XeBHxN6WQWonZ2Qqo6nn4Zx4+KOQkQkOzK9k/oXwK+Bp6IeWfcFXslaVFXE3XfD559rlDkRqZ4yShDu/hrwGkB0sXqlu/8sm4FVBXl5MHlyuLPaLO5oREQqVqatmB41s93NrBEwC5hjZr/KbmiVX34+fP01fPpp3JGIiFS8TK9BdHX3tYQhQZ8HOhJaMtVoeXnhr+6oFpHqKNMEUTe67+FEYIK7bwZqfGfXiXsh5s+PNw4RkWzI9CL13cBiYAYw2czaAzX+HuLWrUP10ne+E3ckIiIVL9OL1KOB0UmTlpjZkdkJqeowg1at4o5CRCQ7Mr1I3dTMbjOzwujxJ6BRlmOrEsaPh0svjTsKEZGKl+k1iL8D64DTo8da4IFsBVWVTJsGd9wBmzbFHYmISMXK9BpEJ3c/Jen1tWY2PQvxVDn5+bBtGyxcCF27xh2NiEjFybQE8Y2ZHZZ4YWaHAt9kJ6SqRU1dRaS6yrQEcTHwsJk1jV6vAc7NTkhVy/77Q7168NxzcNJJcUcjIlJxMipBuPsMd+8J9AB6uHtvYEBWI6simjSB//s/WL4ctm6NOxoRkYqTaQkCgOhu6oSRwB0VGk0VddttUKdMe1JEpPLblSFH1T1dJJEcFi+GRYtiDUVEpMLsSoKo8V1tJNu0KQwgdNllcUciIlIxSkwQZrbOzNameawD9slRjFVC/frw05/CU09BYWHc0YiI7LoSE4S7N3H33dM8mri7at1T/OIX0Lw5XHVV3JGIiOy6XalikhS77w5XXAGTJsHrr8cdjYjIrslqgjCzo81snpktMLMr08y/2Mxmmtl0M3vDzLomzft1tN48M/thNuOsSJdcAu3awdSpcUciIrJrslZNZGa1gTHAUcBSYIqZTXD3OUmLPerud0XLDwJuA46OEsVgoBvhWseLZpbn7pX+ToPddoO5c6Fhw7gjERHZNdksQfQDFrj7Inf/FhgLnJC8QMp9FY0oahl1AjDW3Te5+0fAgmh7VUIiObz3XhivWkSkKspmgmgNfJL0emk0bQdmdomZLQRuBn5WlnUrs0mToE8fePrpuCMRESmf2C9Su/sYd+8EXAGUqf2PmV2UGKNixYoV2QmwnL7//dCR3+9+py44RKRqymaCWAa0TXrdJppWnLGEMa8zXtfd73H3AncvaNmy5a5FW8Hq1IFrr4VZs2DcuLijEREpu2wmiClAZzPraGb1CBedJyQvYGadk14eB8yPnk8ABptZfTPrCHQG3s1irFlx+unQowdccw1s3hx3NCIiZZO1BOHuW4ARwCTgA+Bxd59tZtdFLZYARpjZ7GjwoZFEXYi7+2zgcWAO8B/gkqrQgilVrVpw/fWwYgXMnh13NCIiZWNeTZrZFBQUeGEl7OPCHdauhaZNS19WRCTXzGyquxekmxf7Rerqziwkh23bYMGCuKMREcmcEkSO/OQncNhh8PXXcUciIpIZJYgcOeecMOrcX/8adyQiIplRgsiRQw6BY4+FP/4Rvvoq7mhEREqnBJFDN9wAa9aEIUpFRCo7JYgc6t0bTj0VJkwIF61FRCozDfqTY3feGVo11VJqFpFKTqepHGvRAurWhQ0bQnWTiEhlpQQRg40boUsXDU0qIpWbEkQMGjSAY46Be++FxYvjjkZEJD0liJhcdVW4DnHttXFHIiKSnhJETNq0CXdXP/wwzJsXdzQiIjtTgojRlVeG4UkfeyzuSEREdqZmrjHaay+YMQP23TfuSEREdqYSRMw6dQo9vs6ZA+vXxx2NiEgRJYhKYPVq+O53YdiwMH6EiEhloARRCTRrFlo1PfEE/OlPcUcjIhIoQVQSl10Gp50GV1wBL78cdzQiIkoQlYYZ/P3vsP/+cMYZsHRp3BGJSE2nBFGJNG4MTz0VBhfaa6+4oxGRmk4JopLJywvXIerVC62adNFaROKiBFFJLV0K3buH/ppEROKgBFFJtWoVrkeMGAFvvx13NCJSEylBVFK1a8Mjj4Q+m049FZYvjzsiEalplCAqsWbNwkXr1atDy6YtW+KOSERqEiWISq5nT7jnntAMdt26uKMRkZpECaIKOOsseOkl2HPPuCMRkZpECaKKqFULVqyA44+HmTPjjkZEagIliCpkyxaYOhVOOgm+/DLuaESkulOCqEJatYInn4QlS0K107ZtcUckItWZEkQVc+ihcMcd8NxzcP31cUcjItWZEkQV9JOfhP6aHnoIvv467mhEpLrKaoIws6PNbJ6ZLTCzK9PMH2lmc8zsfTN7yczaJ8272cxmm9kHZjbazCybsVYlZnDXXTBlCjRqFHc0IlJdZS1BmFltYAxwDNAVGGJmXVMWew8ocPcewJPAzdG6hwCHAj2AA4C+wPeyFWtV1LAhNG8O334LN92k4UpFpOJlswTRD1jg7ovc/VtgLHBC8gLu/oq7b4hevg20ScwCGgD1gPpAXUCdTaQxZUoYja5LF7jzTti0Ke6IRKS6yGaCaA18kvR6aTStOOcDzwO4+1vAK8Bn0WOSu3+QuoKZXWRmhWZWuGLFigoLvCo59NAwAl27duHaROfOcPfdsHVr3JGJSFVXKS5Sm9lZQAFwS/R6P6ALoUTRGhhgZoenrufu97h7gbsXtGzZMpchVyr9+8Mbb8CkSdC6dShJJK7YaDwJESmvbCaIZUDbpNdtomk7MLOBwG+BQe6eqCA5CXjb3de7+3pCyeK7WYy1yjODH/wA3nwTXnwx3Hm9Zg306QP33w+bN8cdoYhUNdlMEFOAzmbW0czqAYOBCckLmFlv4G5CcvgiadbHwPfMrI6Z1SVcoN6pikl2ZgYtWoTny5dD3bpwwQWQnx/GvFaiEJFMZS1BuPsWYAQwiXByf9zdZ5vZdWY2KFrsFqAx8ISZTTezRAJ5ElgIzARmADPc/ZlsxVpd7b8/vPMOPPNM6Dr8/PPDxeyvvoo7MhGpCsyrSSV1QUGBFxYWxh1GpeUOzz4LkyfDLbeEaW+9BX37Qp068cYmIvExs6nuXpBuXqW4SC3ZZwY/+lFRcliyBA4/HLp1g3/+U62eRGRnShA1VNu28MQT0KABnH12qI66//5w452ICChB1Fi1aoVuw997D/71L9h9dxg+XGNfi0gRJYgarlYtOPlkKCwMyaJt1DD5zDPh97/XBW2RmkwJQoBwjaJbt/D8m2/CPRS/+Q20bx+68li5Mt74RCT3lCBkJw0bwvPPh1LFwIGhM8D27cO42CJScyhBSLEOPDCMYDd7Nvz4x6FJLIR7KxYujDc2Eck+JQgpVZcucM894UI2wCWXQF5eGPZ01qx4YxOR7FGCkDJ75hkYORLGj4fu3UNrqBkz4o5KRCqaEoSUWatW4Ya7JUvgmmvgtdeKEkQ1uTFfRFCCkF3QvDmMGgUffRSaxQLcfjucfjrMmRNraCJSAZQgZJc1bVrUn5N7aAF1wAHhGsWHH8Ybm4iUnxKEVKhLLw0lissvh6eeChe4E/0/iUjVogQhFa5FC/jDH2DRIvj5z6FfvzB9xQr4+ON4YxORzClBSNbsvTfcdht873vh9Y03wn77hWayy3YaW1BEKhslCMmZkSNh2LBwT0WnTvDLX8Lnn8cdlYgURwlCcqZdO7jrrnDh+swz4S9/gSuuCPM2bYK1a+ONT0R2pLHEJOc6dgzjY//611CvXpj2xhuh36fvfCeMn52XFx6DB0ObNvHGK1JTKUFIbDp3LnresWPoXvzDD2HevNACauVK6N8/JIhHHoFrrw1JI5FA8vPh4IPDoEciUvGUIKRS2HdfuPLKHaetXg1NmoTne+0FPXuGBPLSS7BxY5j+2Weh1DFxYmhe+6MfhaosEdl1ShBSaTVrVvT8qKPCA2DbNli6NCSLvfcO0/797zBk6ogR0KNHSBSDBhU1sRWRstNFaqlyatUKpYSBA8NARwD33Qdz54ab8vbYI9yHMWJE0Tpvvgnr18cSrkiVpRKEVBv5+eFx2WWheurTT8P0r7+GAQPC8wED4PjjQwkjMbyqiKSnEoRUS82ahf6gAOrXD/1D/eQnMH9+uFGvXTu4884wf+tW+Pbb+GKt6Vatgg0b4o5C0lGCkGqvTh048shwV/eHH8IHH4SqqCOPDPOffTa0hPrOd8IoeoMGhWSS6BZk+fKwju7TqFjTpoUSXcuWYUjbe+8NyVoqD1UxSY1iBvvvHx4JrVrB1VeH7j+WLQvjXPzvf+HObwhNbC+9NDxv3Bhatw6PsWPDyW369PQDJp19drhe8vbbO/dqW7t2GMYVwj0gmzdDnz6hZ9zqas2aMMhU585w2GGhhdqnn8JvfgOTJ8NFF4W77N94I5T6JH5KEFLj9etXcmunQYNC6SKRQJYtC62oGjcO8x96CO64Y+f1zjqraP5dd+04r2HDogRx110hCUG4v6OgAA45JFSFVXXr1sGECTBuHPznPyER/uQnIUF07hxKZmahm/hx48L454nk8NVX1TthVgXm1WQIsIKCAi8sLIw7DKmBVq8OJ7NUHTqEk9+KFeFEmcws3BwIoT+q6dOhsBCmTg1/99kH3nknzD/33PC3oCA8evaE3XbL1qfZdVu3hhIShCbHM2eGBgGnnw5nnBE+Q6L1WXHefRe+//1wt/3IkboZMpvMbKq7F6SdpwQhUvl8/TU0ahSen3FGGNZ1+fLwunZtGD489GUF8N57IWGsXx8S0fr1RV2VrF4NY8bsOG/dOrjwQjj2WJg1K7Tqqls3jBDYvHm4wD98eCjFfPEFvPxy0fTEMo0b73iS37gxlBDGjQtNihcsCNt89lnYc0/47ndDdVumliwJieHf/w43Ud5+e2h5VlpikbIrKUGoikmkEkokBwgnXfdQX58oYXTpEuatWROuXaS64Qb47W9DQrj66vALvHHjUO/fuHHRBfc99oAjjghVP6tWhSQ0Zw6cemqYP2MGDBmy8/YnTAgn7NdeC+8zc2bYZosWcMopIQk1axaST3m0bw//+he8+GIYU+SEE+Dkk+HJJys2SWzdGqq52rcvums/FxI3e86fH6rRCgqK4kmUvioDJQiRKsCs6OL4oEFF0+vXD7+yv/kmnPgTSSDR3UjbtuHkX6eY//Q2beDhh4t/30MPDdcFVq8OCWTVqvC8e/cw3z2UFE45JZR0BgwIryvKwIGh+i3RJDlxvWL9+vKd0FetCknt3XdDFV5hYdjWRx+F7f3tb6EjyX33DVWA++4bHv37l/1zuYfqw/Xri/odO+uskHQXLCjqLubMM8M1qG+/DY0e8vOLqhMPPBC6dq3YfVoWqmISkSrliSfCBfzf/x7OO6/4qqt160KJ65134MQTw4n38cdDIqtbF3r1goMOCg0Uzjwz/HIfOzY0Kli0CBYvDidts3Ayr1cPrrsOXn11xwTSqVNRI4fRo0MLuPnzw2P9+lBCe+21MP/EE0PpIS8vJI3OnUNpsFWrcB3rhhuKrkUlrlvddFO4FrNmDTz9dEgc++9ffNIvq9iqmMzsaODPQG3gPnf/Q8r8kcAFwBZgBTDM3ZdE89oB9wFtAQeOdffF2YxXRCq/zp3DCfaCC0ILsNGjwzUOCL/Yr7oqJIU5c8LJGIp+mR91VGh23LNn+gvfgweHB4Tqnk8/hU8+KeqWvlGjkCyefbbomlCrVkV37b/0Uihx5eXB4YeHWHv0KNr++PHFf66mTYvGb9+2LZQyCguLqhDffjskRAjXnHr1Csnipz8NIzVmQ9ZKEGZWG/gQOApYCkwBhrj7nKRljgTecfcNZjYc6O/uZ0TzXgVudPcXzKwxsM3di73fUiUIkZrDHR57DC6/PDQ7HjUKrrkm/Oreb79QNZMoHfTrFy6sV7Svvw6ljDVrQrPdRFzZupC+bVu4n6awsKiUMW0avPXWjkmorOIqQfQDFrj7oiiIscAJwPYE4e6vJC3/NnBWtGxXoI67vxAtp27WRGQ7s1AtNGgQ/PnP4eI4hOsIn3+em9ZOjRpBt247x5UttWoV3eSZuMdm69bsvmc2E0Rr4JOk10uBg0pY/nzg+eh5HvClmf0b6Ai8CFzp7jvciG9mFwEXAbTTIAAiNU7jxqEVVbKa1BQ22y2eKkVfTGZ2FlAARDVw1AEOBy4D+gL7AkNT13P3e9y9wN0LWrZsmaNoRURqhmwmiGWEC8wJbaJpOzCzgcBvgUHuvimavBSY7u6L3H0LMB5I09pbRESyJZsJYgrQ2cw6mlk9YDAwIXkBM+sN3E1IDl+krLuHmSWKBQNIunYhIiLZl7UEEf3yHwFMAj4AHnf32WZ2nZklbvW5BWgMPGFm081sQrTuVkL10ktmNhMw4N5sxSoiIjvTjXIiIjVYSc1cK8VFahERqXyUIEREJC0lCBERSavaXIMwsxXAkrjjKEELYGXcQZRA8e0axbdrFN+u2ZX42rt72hvJqk2CqOzMrLC4C0GVgeLbNYpv1yi+XZOt+FTFJCIiaSlBiIhIWkoQuXNP3AGUQvHtGsW3axTfrslKfLoGISIiaakEISIiaSlBiIhIWkoQFcTM2prZK2Y2x8xmm9nP0yzT38y+ijomnG5mV8cQ52Izmxm9/06dV1kw2swWmNn7ZpazbtbNLD9p30w3s7Vm9ouUZXK6D83s72b2hZnNSprWzMxeMLP50d89i1n33GiZ+WZ2bg7ju8XM5kbf31Nmtkcx65Z4LGQxvlFmtizpOzy2mHWPNrN50bF4ZQ7jG5cU22Izm17MurnYf2nPKzk7Bt1djwp4AK2APtHzJoTxuLumLNMfeDbmOBcDLUqYfyxhZD8DDiaMGR5HnLWBzwk38cS2D4EjCGORzEqadjNhhEOAK4E/plmvGbAo+rtn9HzPHMX3A8KQvQB/TBdfJsdCFuMbBVyWwfe/kDBYWD1gRur/U7biS5n/J+DqGPdf2vNKro5BlSAqiLt/5u7ToufrCF2ct443qnI5AXjYg7cJ43K0iiGO7wML3T3Wu+PdfTKwOmXyCcBD0fOHgBPTrPpD4AV3X+3ua4AXgKNzEZ+7/9dDd/sQxnpvU9Hvm6li9l8mto9p7+7fAokx7StUSfGZmQGnA49V9PtmqoTzSk6OQSWILDCzDkBv4J00s79rZjPM7Hkz65ZmfrY58F8zm2phTO9U6cYSjyPRDab4f8y49+He7v5Z9PxzYO80y1SW/TiMorHeU5V2LGTTiKgK7O/FVI9Uhv13OLDc3ecXMz+n+y/lvJKTY1AJooKZWWPgX8Av3H1tyuxphCqTnsBfCEOp5tph7t4HOAa4xMyOiCGGElkYgXAQ8ESa2ZVhH27noSxfKduKm9lvgS3AI8UsEtexcCfQCegFfEaoxqmMhlBy6SFn+6+k80o2j0EliApkZnUJX+Ij7v7v1Pnuvtbd10fPJwJ1zaxFLmN092XR3y+ApwhF+WQZjSWeZccA09x9eeqMyrAPgeWJarfo7xdplol1P5rZUOB44MfRCWQnGRwLWeHuy919q7tvI4wUme59495/dYCTgXHFLZOr/VfMeSUnx6ASRAWJ6ivvBz5w99uKWeY70XKYWT/C/l+VwxgbmVmTxHPCxcxZKYtNAM6x4GDgq6SibK4U+8st7n0YmQAkWoScCzydZplJwA/MbM+oCuUH0bSsM7OjgcsJY71vKGaZTI6FbMWXfE3rpGLet9Qx7bNsIDDX3Zemm5mr/VfCeSU3x2A2r8DXpAdwGKGY9z4wPXocC1wMXBwtMwKYTWiR8TZwSI5j3Dd67xlRHL+NpifHaMAYQguSmUBBjmNsRDjhN02aFts+JCSqz4DNhDrc84HmwEvAfOBFoFm0bAFwX9K6w4AF0eO8HMa3gFD3nDgO74qW3QeYWNKxkKP4/hEdW+8TTnStUuOLXh9LaLWzMJfxRdMfTBxzScvGsf+KO6/k5BhUVxsiIpKWqphERCQtJQgREUlLCUJERNJSghARkbSUIEREJC0lCJFSmNlW27GX2QrrWdTMOiT3JCpSmdSJOwCRKuAbd+8VdxAiuaYShEg5ReMB3ByNCfCume0XTe9gZi9HndG9ZGbtoul7WxifYUb0OCTaVG0zuzfq7/+/ZtYwWv5n0TgA75vZ2Jg+ptRgShAipWuYUsV0RtK8r9y9O/BX4I5o2l+Ah9y9B6GjvNHR9NHAax46GuxDuAMXoDMwxt27AV8Cp0TTrwR6R9u5ODsfTaR4upNapBRmtt7dG6eZvhgY4O6Log7VPnf35ma2ktB9xOZo+mfu3sLMVgBt3H1T0jY6EPrs7xy9vgKo6+43mNl/gPWEHmvHe9RJoUiuqAQhsmu8mOdlsSnp+VaKrg0eR+gXqw8wJephVCRnlCBEds0ZSX/fip6/Seh9FODHwOvR85eA4QBmVtvMmha3UTOrBbR191eAK4CmwE6lGJFs0i8SkdI1tB0Hrv+Puyeauu5pZu8TSgFDomk/BR4ws18BK4Dzouk/B+4xs/MJJYXhhJ5E06kN/DNKIgaMdvcvK+jziGRE1yBEyim6BlHg7ivjjkUkG1TFJCIiaakEISIiaakEISIiaSlBiIhIWkoQIiKSlhKEiIikpQQhIiJp/T+bZJ7ezwQA9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "val_loss = history_small_model.history[\"val_loss\"]\n",
    "epochs = range(1, 21)\n",
    "plt.plot(epochs, val_loss, \"b--\",\n",
    "         label=\"Validation loss\")\n",
    "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "375/375 [==============================] - 1s 2ms/step - loss: 0.3660 - accuracy: 0.8961 - val_loss: 0.1933 - val_accuracy: 0.9444\n",
      "Epoch 2/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.1633 - accuracy: 0.9510 - val_loss: 0.1460 - val_accuracy: 0.9569\n",
      "Epoch 3/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.1143 - accuracy: 0.9654 - val_loss: 0.1189 - val_accuracy: 0.9644\n",
      "Epoch 4/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0887 - accuracy: 0.9739 - val_loss: 0.1099 - val_accuracy: 0.9681\n",
      "Epoch 5/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0705 - accuracy: 0.9784 - val_loss: 0.0979 - val_accuracy: 0.9722\n",
      "Epoch 6/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0588 - accuracy: 0.9823 - val_loss: 0.0975 - val_accuracy: 0.9732\n",
      "Epoch 7/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0484 - accuracy: 0.9858 - val_loss: 0.0939 - val_accuracy: 0.9742\n",
      "Epoch 8/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0396 - accuracy: 0.9877 - val_loss: 0.0938 - val_accuracy: 0.9743\n",
      "Epoch 9/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0342 - accuracy: 0.9900 - val_loss: 0.1050 - val_accuracy: 0.9726\n",
      "Epoch 10/20\n",
      "375/375 [==============================] - 1s 2ms/step - loss: 0.0284 - accuracy: 0.9911 - val_loss: 0.1130 - val_accuracy: 0.9702\n",
      "Epoch 11/20\n",
      "375/375 [==============================] - 1s 2ms/step - loss: 0.0238 - accuracy: 0.9927 - val_loss: 0.0994 - val_accuracy: 0.9757\n",
      "Epoch 12/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0208 - accuracy: 0.9933 - val_loss: 0.1058 - val_accuracy: 0.9745\n",
      "Epoch 13/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0171 - accuracy: 0.9943 - val_loss: 0.1074 - val_accuracy: 0.9768\n",
      "Epoch 14/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0148 - accuracy: 0.9952 - val_loss: 0.1195 - val_accuracy: 0.9758\n",
      "Epoch 15/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0128 - accuracy: 0.9961 - val_loss: 0.1153 - val_accuracy: 0.9743\n",
      "Epoch 16/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0109 - accuracy: 0.9969 - val_loss: 0.1276 - val_accuracy: 0.9739\n",
      "Epoch 17/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0093 - accuracy: 0.9972 - val_loss: 0.1392 - val_accuracy: 0.9746\n",
      "Epoch 18/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0083 - accuracy: 0.9973 - val_loss: 0.1323 - val_accuracy: 0.9762\n",
      "Epoch 19/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0069 - accuracy: 0.9976 - val_loss: 0.1449 - val_accuracy: 0.9744\n",
      "Epoch 20/20\n",
      "375/375 [==============================] - 1s 1ms/step - loss: 0.0062 - accuracy: 0.9981 - val_loss: 0.1443 - val_accuracy: 0.9756\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(96, activation=\"relu\"),\n",
    "    layers.Dense(96, activation=\"relu\"),\n",
    "    layers.Dense(10, activation=\"softmax\"),\n",
    "])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "history_large_model = model.fit(\n",
    "    train_images, train_labels,\n",
    "    epochs=20,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fc141dea438>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA2YklEQVR4nO3deZwU1bXA8d9hQJBVBETZUVFh2BlwQRGNUVBEJS4Qn4K4QYImMRrR5ClxyTMBo+IWVxQDovEFHioGV8DEjQEBZVOWYd8XGQQUmPP+ODXQDD0zPTPdXT095/v59Ke7az1dXV2n69ate0VVcc455wqqFHYAzjnnUpMnCOecc1F5gnDOOReVJwjnnHNReYJwzjkXlScI55xzUVX4BCEiD4jIZhFZH7y/TERWichOEekUYlxFxhEMPz7BMXQXkW+DdV0qIg1FZIaI5IrIwyJyt4g8H8Ny/iYi/53IWJNBRAaJyL9jnPYlEXkg0THFg4i8IyIDw44jXkSkp4isjng/X0R6xjJtKdaVkH1bREaIyN/jvdySqhx2AIkmIjlAQ2B/xOCXVHWYiDQDfgs0V9WNwbhRwDBV/b8yrleBVqq6pJSLKDIOVa1Z6uBidx/whKo+BhD8EDYDtbUEN9Co6pB4BBP8yP+uqk3isTxnVLV3/msRGQTcoKpnhhdRfKlqZjyWE23bxGvfTlVpnyACF6vq+1GGNwO2RCQHgObA/OSEVaRUiKNgDM2BBSVJDs65ckxV0/oB5ADnRRl+HrAbyAN2Aq8Gzwp8DywNpmsE/C+wCVgO3BqxjAzgbmApkAvMApoCMyKWsxO4Ksr6KwF/AFYAG4GxQB2garQ4osyvwInB65eAJ4G3gzg+B04IxgnwSLCOHcBXQNtg3DTsH1H+MgcB/w5eLw22ze6I7bMX+DF4fx4wAvtHnz//mcAnwHZgFTAoIr4HIqbrA8wJpvsEaF/g+7odmAd8B7wGVANqFPi+dgKNomyXl4CngHeCaf4DHAs8CmwDFgGdIqZvHWyH7Vgy7Bsxrh4wOdhuXwD352+fYPwpwHvAVmAxcGWBOB6I9t0F428EFgbf1wKgczB8OAf3pwXAZQW+n/8ATwTbZhHwk4jx10Uscxlwc4F1XhJs9x3BOnpF7gfBttiDnW3vDLZJV2ADkBGxnH7A3EI+Vx1sX96E7dt/ACpF7l/Y2fE27PfUu5Dl3Am8UWDYY8Do4j4r0BNYHe0YABwZfDfbgu17R4Fpo27/aNumkH37RmBJsE9MJmIfxX6zQ4Bvg237JCCFfP4RHPrb6ovtn9uD76t1gW21Joh5cf4+AXQDsoPvewPw1xIfP+N9QE61B4UkiGg7UsSXmH/grYQd9O8BjgCOD3bGC4Lxd2AH3JOxA3EHoF7B5RSy7sHBjnQ8UBP4J/BKtDgKmb9ggtgS7BCVgXHAhGDcBcFnOCqIsTVwXDBuGoUkiGjbLsqP4cBOjJ1d5AIDgCrYwbVjwfmATliyOhVLsAOD9VSNWOcXWGI+GjsIDCns+4qyXV7CisG6YInlQ+xAdG2wvgeAj4JpqwTfwd3B93tu8BlODsZPAF7HklNb7EeYn0BrYEnwumCbdwrW2ybatioQ4xXBsroG38mJWDFn/rhG2L53FfYn4biI72cf8Jsg9quwRHF0MP4i4IRgmWcDuziYeLoF0/40WHZj4JSC+0HBfSAYtoCIAzkwEfhtIZ9tLPB/QC2gBfANcH3EsvdiB9EMYCiwligHSWx/2gXUCt5nAOuA02L4rIfsJxyaIB4CPsb2rabA1wWmLW77F9w2B75nbP/ZDHTG/ug9Dswo8Jt9C/stNsOSaK9CtuMIDv62Tgri+Gnwvf8O22+PwI49qwgSUbDN8/8cfgpcE7yumb/tSvKoKBepJ4nI9ojHjTHO1xVooKr3qeqPqroMeA7oH4y/AfiDqi5WM1dVt8S47KuxjL5MVXcCdwH9RaS0xX4TVfULVd2HJYiOwfC92I/1FOyHuFBV15VyHUX5OfC+qr6qqntVdYuqzoky3U3AM6r6uaruV9WXgR+A0yKmGa2qa1V1K/BmxGeJ1URVnaWqe7CD2R5VHauq+7EzkvyL/qdhP5yHgu/3Q+wHPEBEMoCfAfeo6veq+jXwcsQ6+gA5qjpGVfep6pfYmeYVMcR3A/AXVZ0Z7DdLVHUFgKr+I/jsear6GvZvs1vEvBuBR4Nt/Br2j/GiYN63VXVpsMzpwLvAWcF81wMvqup7wbLXqOqiGLfny8B/AYjI0difjvEFJwq2WX/gLlXNVdUc4GHgmojJVqjqc8F38TJwHHaN8BDB9pgNXBYMOhfYpaqfxfBZi3Il8KCqblXVVcDoAustbvsX5WpsG89W1R+w3/TpItIiYpqHVHW7qq4EPiK2ffsq4O3gu9uLnYEdCZyBndFUBdqISBVVzVHVpcF8e4ETRaS+qu7M33YlUVESxKWqelTE47kY52sONIpMLti/zfwduil2OloajbBT8HwrsH+ih/1YYrQ+4vUu7MBHcNB7Ajud3Sgiz4pI7VKuoyixbovmwG8LbNOm2PbIF/WzlMCGiNe7o7zPX14jYJWq5kWMX4H9u26AfR+rCoyL/BynFvgcV2PFWcUpdFuJyLUiMidimW2B+hGTrNHgL2FETI2CeXuLyGcisjWY98KIecuyr/4duFhEamAH2I8L+ZNRH/uHW3C/bhzx/sB3q6q7gpeFfb/jsTNSsD8gB5JSMZ+1KI0o/DuNZfsXt+wDywv++G2hkM9P7Pt2weXmBZ+hsVolmF9jZxwbRWSCiOT/lq7Hzj4WichMEekT4+c4oKIkiNJaBSwvkFxqqeqFEeNPKOWy12IHmXzNsOKDDdEnLz1VHa2qXYA22A5zRzDqe6B6xKSxHNwKE+u2WIX9g4vcptVV9dUY5tXiJymRtUBTEYn8HTTDin82Yd9H0wLj8q0Cphf4HDVVdWgM6426rUSkOXaGOgwrqjwKKwKRiMkai0jk+2bAWhGpip3BjAIaBvNOiZg31u/nsG2sqmuw4op+2NnAK4XMuxn711pwv14Tw3qj+QfQU0SaYGcS4wFi+KxFWUch32kM27+4/e+Q33SQUOtR+s9f2HIF+wxrAFR1vFrNquZBjH8Ohn+rqgOAY4JhbwQxxcwTRNG+AHJF5E4ROVJEMkSkrYh0DcY/D9wvIq3EtBeResG4Ddj1hcK8CvxGRFqKSE3gT8BrQRFR3IhIVxE5VUSqYAlhD3ahF+yCZT8RqS4iJ2L/OEprHHCeiFwpIpVFpJ6IdIwy3XPAkCAmEZEaInKRiNSKYR0bgHoiUqcMcUb6HPsX9zsRqRJUo70Yu36zH7suNCLYPm2w6yX53gJOEpFrgnmrBNu6dQzrfR64XUS6BNvgxODgVAP7gW8CEJHrsH+wkY4Bbg3WdwV2TWkKVh5dNZh3n4j0Bs6PmO8F4DoR+YmIVBKRxiJySpTYNgBNROSIAsPHYmXf7YLtcphgm70OPCgitYLPdBt2BlJiqroJuz4yBvujtjAYVdxnLcrrwF0iUjdIPLdEjCtu+xe2bfK9im3jjkES+xPweVDUVhavAxcF310VrGr+D8AnInKyiJwbrG8PBytyICL/JSINgjOO7cGy8g5ffOEqSoJ4U+xmr/zHxFhmCnb4Plg54XLsH9LzWE0NgL9iX967WE2BF7CyQbBTvpeDU9Uroyz+Reyf2Ixg2Xs4dGeNl9rYQXkbdpq6BRgZjHsEq5W0ASsPHlfalQRlqhdiO+9WLPl0iDJdNnaR8okgpiXYxb9Y1rEI+xEuC7Zro+LmKWZ5P2IJoTf23T4FXBtRNj8MKwJYj12MHBMxby52UOqP/cNbj/1LqxrDev8BPIj9I84FJmEXmhdgZfafYt9JO6zWUqTPgVZBvA8Cl6td78kFbsX2x21YkczkiHV+gV1QfwS7WD2dQ//p5/sQqy2zXkQ2RwyfGEw/MaJoKJpbsD8iy7AaS+Oxfb20xmM15g4ULxX3WYvxR+x3sBz73R44G4ph+xe2bfLnfx/4b+zsZh12xta/4HQlpaqLsWtAj2Pf+8VY1f0fsf3toWD4euwPxF3BrL2A+SKyE6sB1l9Vd5dk3XJocaZzLlVJyDexichSrDpptHuKXBqqKGcQzrkyEJGfYcUvH4Ydi0ueinIntXOulERkGlbB4ZoCNb5cmvMiJuecc1F5EZNzzrmo0qaIqX79+tqiRYuww3DOuXJl1qxZm1W1QbRxaZMgWrRoQXZ2dthhOOdcuSIiKwob50VMzjnnovIE4ZxzLipPEM4556JKm2sQzrnk27t3L6tXr2bPnj1hh+KKUa1aNZo0aUKVKlVinscThHOu1FavXk2tWrVo0aIFhzYy61KJqrJlyxZWr15Ny5YtY57Pi5icc6W2Z88e6tWr58khxYkI9erVK/GZnicI51yZeHIoH0rzPXmCcM45F1WFTxD79sFPfwpPPRV2JM65kjrnnHOYOnXqIcMeffRRhg4tvGO/nj17Hrip9sILL2T79u2HTTNixAhGjRpV5LonTZrEggULDry/5557eP/9sreEPm3aNPr0KXHvoAlR4RNE5crwzTfwySdhR+KcK6kBAwYwYcKEQ4ZNmDCBAQMGFDLHoaZMmcJRRx1VqnUXTBD33Xcf5513XqmWlaoqfIIAyMyEr78OOwrnXEldfvnlvP322/z4448A5OTksHbtWs466yyGDh1KVlYWmZmZ3HvvvVHnb9GiBZs3W+dwDz74ICeddBJnnnkmixcvPjDNc889R9euXenQoQM/+9nP2LVrF5988gmTJ0/mjjvuoGPHjixdupRBgwbxxhtvAPDBBx/QqVMn2rVrx+DBg/nhhx8OrO/ee++lc+fOtGvXjkWLFh0eVIStW7dy6aWX0r59e0477TTmzZsHwPTp0+nYsSMdO3akU6dO5Obmsm7dOnr06EHHjh1p27YtH3/8cdk2Lp4gAGjbFhYtsuIm51zp9ex5+CO/+HbXrujjX3rJxm/efPi44hx99NF069aNd955B7CzhyuvvBIR4cEHHyQ7O5t58+Yxffr0AwfXaGbNmsWECROYM2cOU6ZMYebMmQfG9evXj5kzZzJ37lxat27NCy+8wBlnnEHfvn0ZOXIkc+bM4YQTTjgw/Z49exg0aBCvvfYaX331Ffv27ePpp58+ML5+/frMnj2boUOHFluMde+999KpUyfmzZvHn/70J6699loARo0axZNPPsmcOXP4+OOPOfLIIxk/fjwXXHABc+bMYe7cuXTs2LH4DVgMTxBYgvjhB1i6NOxInHMlFVnMFFm89Prrr9O5c2c6derE/PnzDykOKujjjz/msssuo3r16tSuXZu+ffseGPf1119z1lln0a5dO8aNG8f8+fOLjGfx4sW0bNmSk046CYCBAwcyY8aMA+P79esHQJcuXcjJySlyWf/+97+55pprADj33HPZsmULO3bsoHv37tx2222MHj2a7du3U7lyZbp27cqYMWMYMWIEX331FbVq1Spy2bHwG+WATp3gvPPAbwZ1rmymTSt8XPXqRY+vX7/o8YW55JJL+M1vfsPs2bPZtWsXXbp0Yfny5YwaNYqZM2dSt25dBg0aVOq7vQcNGsSkSZPo0KEDL730EtNKE2SEqlWrApCRkcG+UhZbDB8+nIsuuogpU6bQvXt3pk6dSo8ePZgxYwZvv/02gwYN4rbbbjtwxlFafgYBtGsH770HHTqEHYlzrqRq1qzJOeecw+DBgw+cPezYsYMaNWpQp04dNmzYcKAIqjA9evRg0qRJ7N69m9zcXN58880D43JzcznuuOPYu3cv48aNOzC8Vq1a5ObmHrask08+mZycHJYsWQLAK6+8wtlnn12qz3bWWWcdWOe0adOoX78+tWvXZunSpbRr144777yTrl27smjRIlasWEHDhg258cYbueGGG5g9e3ap1hnJzyAi5OVBJU+ZzpU7AwYM4LLLLjtQ1NShQwc6derEKaecQtOmTenevXuR83fu3JmrrrqKDh06cMwxx9C1a9cD4+6//35OPfVUGjRowKmnnnogKfTv358bb7yR0aNHH7g4Ddbm0ZgxY7jiiivYt28fXbt2ZciQIaX6XCNGjGDw4MG0b9+e6tWr8/LLLwNWlfejjz6iUqVKZGZm0rt3byZMmMDIkSOpUqUKNWvWZOzYsaVaZ6S06ZM6KytLy9Jh0LBh8PHHMHduHINyLs0tXLiQ1q1bhx2Gi1G070tEZqlqVrTp/f9y4KijYMECu1jtnHPOE8QBmZlWzfWbb8KOxDnnUoMniEDbtvbsN8w5VzLpUkyd7krzPXmCCJx0EmRkQDFVnJ1zEapVq8aWLVs8SaS4/P4gqlWrVqL5vBZToGpVuP12yIp6qcY5F02TJk1YvXo1mzZtCjsUV4z8HuVKwhNEhIceCjsC58qXKlWqlKiHMle+eBFTBFVYtQr27g07EuecC58niAgTJ0KzZlBEm17OOVdheIKIkJlpz36h2jnnPEEc4oQT4IgjvKqrc86BJ4hDVK4MrVt7gnDOOfAEcZjMTC9ics458Gquh7npJujTx2o0iYQdjXPOhccTRAGlbLbdOefSjhcxFZCXB599BhF9ljvnXIWU0AQhIr1EZLGILBGR4VHG9xCR2SKyT0QuLzDuLyIyX0QWishokeQV+PzkJxDRx7hzzlVICUsQIpIBPAn0BtoAA0SkTYHJVgKDgPEF5j0D6A60B9oCXYGkFP5UquQXqp1zDhJ7BtENWKKqy1T1R2ACcEnkBKqao6rzgLwC8ypQDTgCqApUATYkMNZDZGZ6VVfnnEtkgmgMrIp4vzoYVixV/RT4CFgXPKaq6sKC04nITSKSLSLZ8WxNsm1bWL8etmyJ2yKdc67cScmL1CJyItAaaIIllXNF5KyC06nqs6qapapZDRo0iNv6vckN55xLbIJYAzSNeN8kGBaLy4DPVHWnqu4E3gFOj3N8hTr9dJg2DTp3TtYanXMu9SQyQcwEWolISxE5AugPTI5x3pXA2SJSWUSqYBeoDytiSpQ6dex+iJo1k7VG55xLPQlLEKq6DxgGTMUO7q+r6nwRuU9E+gKISFcRWQ1cATwjIvmFOm8AS4GvgLnAXFV9M1GxRjN9OowZk8w1OudcapF06Us2KytLs7Oz47a8m2+GN96AzZu9yQ3nXPoSkVmqGrWz5ZS8SJ0K2raFrVutNpNzzlVEniAK4TWZnHMVnSeIQrRta89+w5xzrqLyBFGIY46B+vVhYdLqTjnnXGrx5r6LMHcuNGwYdhTOORcOTxBFaNQo7Aiccy48XsRUhK+/th7m1q4NOxLnnEs+TxBF2L4dnnsO5swJOxLnnEs+TxBFyK/q6jWZnHMVkSeIItSta9ch/F4I51xF5AmiGG3b+hmEc65i8gRRjHbt4McfIU2arHLOuZh5gijGyJHw1VfeYJ9zruLxBFEMTwzOuYrKE0Qx9u+Hiy+Gp58OOxLnnEsuTxDFyMiwi9QzZoQdiXPOJZcniBh4TSbnXEXkCSIGmZmweDHs3Rt2JM45lzyeIGLQtq0lh2+/DTsS55xLHk8QMejYEXr0gD17wo7EOeeSx5v7jkHbtjB9ethROOdccvkZRAn43dTOuYrEE0SMbrvNipqcc66i8AQRo1q1rKqrX4dwzlUUniBilJkJeXmwaFHYkTjnXHJ4gohR27b27DfMOecqCk8QMWrVCqpU8c6DnHMVhyeIGFWpAr/6FXTqFHYkzjmXHH4fRAmMHBl2BM45lzx+BlFCGzdaD3POOZfuEpogRKSXiCwWkSUiMjzK+B4iMltE9onI5QXGNRORd0VkoYgsEJEWiYw1FpMnQ8OGMGdO2JE451ziJSxBiEgG8CTQG2gDDBCRNgUmWwkMAsZHWcRYYKSqtga6ARsTFWusWre2Z79Q7ZyrCBJ5DaIbsERVlwGIyATgEmBB/gSqmhOMy4ucMUgklVX1vWC6nQmMM2bHHw/VqnlVV+dcxZDIIqbGwKqI96uDYbE4CdguIv8UkS9FZGRwRnIIEblJRLJFJHvTpk1xCLloGRl2FuEJwjlXEaTqRerKwFnA7UBX4HisKOoQqvqsqmapalaDBg2SEljbtl7E5JyrGBJZxLQGaBrxvkkwLBargTkRxVOTgNOAF+IZYGkMHgznnmstu4qEHY1zziVOIhPETKCViLTEEkN/4OclmPcoEWmgqpuAc4HsxIRZMj17hh2Bc84lR8KKmFR1HzAMmAosBF5X1fkicp+I9AUQka4ishq4AnhGROYH8+7Hipc+EJGvAAGeS1SsJZGXB7Nne/ejzrn0J5omveBkZWVpdnbiTzJUoXZtGDQIHn884atzzrmEEpFZqpoVbVyqXqROWSLW9LdfqHbOpTtPEKXQtq1XdXXOpT9PEKWQmQmbNlm7TM45l648QZSCdx7knKsIPEGUQrdu8N57kBX1so5zzqUH7w+iFOrUgfPOCzsK55xLLD+DKKVPPoGxY8OOwjnnEscTRCn9/e9w6612X4RzzqUjTxCllJkJ330Ha2JtXco558oZTxCllF+TyW+Yc86lK08QpZSZac9e1dU5l648QZRS/frWP7WfQTjn0pVXcy2DL76A444LOwrnnEsMTxBl0KxZ2BE451zieBFTGSxcCLfcAqtXhx2Jc87FnyeIMti+HZ54Ar78MuxInHMu/jxBlEGbNvbsNZmcc+nIE0QZ1KkDTZt6TSbnXHryBFFG3nmQcy5dxZQgRKSGiFQKXp8kIn1FpEpiQysf2raF77/3Npmcc+kn1jOIGUA1EWkMvAtcA7yUqKDKk4cegm+/tb6qnXMuncSaIERVdwH9gKdU9QogM3FhlR+VvJDOOZemYk4QInI6cDXwdjAsIzEhlS+qMHAg/PGPYUfinHPxFWuC+DVwFzBRVeeLyPHARwmLqhwRgQ0bYPz4sCNxzpVHy5db/zLvvGPN96TS9cyYmtpQ1enAdIDgYvVmVb01kYGVJ3362B3V33wDJ50UdjTOufJi2zY4/XT7kwlQrRrs3m2vr78e3nrLGgatV88eLVrAI4/Y+GnTIDfXhnfrBpUT0HBSTIsUkfHAEGA/MBOoLSKPqerI+IdU/uQniLff9gThnIvdHXfA5s3wr3/ZfVW5uQfH9ehhB/0tW2yaJUsOJhKABx6ADz6w199/n5gEIRrD+YyIzFHVjiJyNdAZGA7MUtX28Q+pdLKysjQ7Ozu09bdrBw0awIcfhhaCc64cUYU//9nOGEpzDXP1ali/3hLIBReUPg4RmaWqWdHGxZpzqgT3PVwKPKGqe0UkhUrKwnfTTbBunX3pXuXVOVccERg+vPTzN2lij0SKNUE8A+QAc4EZItIc2JGooMqjW24JOwLnXHnx8MN2PeFnPws7kqLFVItJVUeramNVvVDNCuCcBMdW7uzda02AO+dcYebNszOHN98MO5LixdrURh0R+auIZAePh4EaCY6t3PnFL+DMM2H//rAjcc6lov374cYboW5dO4tIdbHeB/EikAtcGTx2AGOKm0lEeonIYhFZIiKHlbaJSA8RmS0i+0Tk8ijja4vIahF5IsY4Q3X++bB1K3z2WdiROOdS0ZNP2r0Ojz1m1VNTXawJ4gRVvVdVlwWPPwLHFzWDiGQATwK9gTbAABFpU2CylcAgoLDbzO7H2oEqF84/36qalYdTR+dccm3YAHffDb17Q//+YUcTm1gTxG4ROTP/jYh0B3YXM083YEmQUH4EJgCXRE6gqjmqOg/IKziziHQBGmKNA5YLdepY3eW33go7EudcqjnmGHjuOXj66fJT0zHWBDEEeFJEckQkB3gCuLmYeRoDqyLerw6GFSu4W/th4PZiprsp/7rIpk2bYll0wvXpYx0ILV8ediTOuVSxd68lhQEDoHnzsKOJXay1mOaqagegPdBeVTsB5yYwrl8AU1R1dTFxPauqWaqa1aBBgwSGE7v+/WHGDOtpzjlXPqxdazeeJcLWrXDyyfDaa4lZfiKVqLFqVd2hqvn3P9xWzORrgMjDZJNgWCxOB4YFZyujgGtF5KGSxBqW446Ds85KzG3vzrn4ysmxm1t//nM49VSYMyf+67j9dli5Ek45Jf7LTrSy9GZQXCnaTKCViLQUkSOA/sDkWBasqlerajNVbYEVM41V1TLcc5hc33wDt912aLsqzrnU8umn0KqV/bMfPdr6djnzTJgyJX7r+PBDGDPG2lzq0CF+y02WsiSIIpvaUNV9wDBgKrAQeD1oKvw+EekLICJdRWQ1cAXwjIjML0M8KWPtWmtx8f33w47EORfNzp1w7bXQuDFceCG0bw+ff26NbV58MTz1VNnXsXu3NcFz4olwzz1lX14YimysT0RyiZ4IBDhSVVOmICXsxvoi7d1rNRb69YMXXgg7GudcQUOGwLPPWpPZPXocHL5zpxU3LVkCs2bBkUeWfh2TJtkx4P334dxEXrEto1I31qeqtRITUnqrUgV69bLmv/PyvFtS51LJlCnwzDN2bSAyOQDUrAkTJ1oLqUceCXv22G+4evWSr+fSS2HxYivGKq/80JUgffrYjTEpclLjnAts2mQXpB94IPr4jAwrAQDrtOeccw7th6E4+/cfvNhdnpMDeIJImF69rH+IFSvCjsQ5F2ngQPjkE6hatfhpr7gCvvoKTjsNFiyIbfmPPw6dO8Ps2WWLMxV4gkiQevWsM48rrgg7EuccWG2lceOsWmusxb6XXmr3Ne3eDWecUXyHYCtWwB/+YBe+O3Uqc8ih8wSRQJUq2c7orbs6F66VK61G0d/+Zr/JksjKshpOTZrYndDffx99OlUYOtTumH7qqfLTnEZRPEEk0ObNVm3u+efDjsS5iisvz4qV8vJg7NjSVRpp3hz+8x+7wF2jhiWDgolmwgR45x148EFo1iw+sYfNE0QC1atnO6U33udceB591KqzPvootGxZ+uXUqQNdutjrhx6Ca66BH344OH7bNujZE375y9KvI9V4gkggEavN9P77sGtX2NE4V/GsWWNNbPftC4MHx3fZ48Yd7AMGrMOwDz6wWlDpwhNEgvXpY3Wpi7u45ZyLv8aN4dVXrZnteF4TuOsuGD/eOgerVw/++teSXfwuL9Ls46SeHj3s5hsvZnIuuTZvtufLLjt4X0M8DRhgZww1atiNd3v3xn8dYfMEkWBVq9oNORdfHHYkzlUc//mPXSieOjWx6znzTKva+umncMQRiV1XGFKmLaV09qtfhR2BcxVHbq41xNewIZx+euLXVx76li4tTxBJsmiR1XJIxg7rXEX2299aj47Tp0Pt2mFHU755gkiSQYPsItbnn4cdiXPp66237IL0735nHXe5svFrEEly8cXwxRfW/IZzLjEWLLAmLu67L+xI0oMniCTp08ee49lblXPuUL/7nVU9jaUhPlc8TxBJ0r49NG3q1V2dS4Q33jh4r1E61iYKiyeIJMm/q/qjj2DfvrCjcS595OTYXdL331/yhvhc0TxBJNEf/mC1Kyp71QDn4iIvD667zl6PGZMeLaimEj9UJVGjRmFH4Fx6eeYZa4jv+eehRYuwo0k/fgaRZJMmwVVX+amwc2W1dq1dlD7vvPg3xOeMJ4gk27QJXn899u4LnXPRHXusNbv97LNetJQoniCS7MIL7fnNN8ONw7nybP9+azn1l78sWx8PrmieIJKscWPr0NyruzpXOuvXwymnWO9tLrE8QYSgTx9r/TG/OWLnXOyGDYNVq+D448OOJP15gghB375w9tl2PcI5F7v//V97jBgBJ58cdjTpTzRNqtNkZWVpdnZ22GE45xJk61Zo08aKaT//3O8nihcRmaWqWdHG+SYO0datUKsWVKkSdiTOpb7XX4ctW6wTIE8OyeFFTCGZPh0aNIB//zvsSJwrH4YMgfnzoUOHsCOpODxBhKRLF/sX5NVdnSvajh2WGABOOincWCoaTxAhqVkTzj3Xq7s6V5w774SuXWHDhrAjqXgSmiBEpJeILBaRJSIyPMr4HiIyW0T2icjlEcM7isinIjJfROaJyFWJjDMsffrAt9/CN9+EHYlzqWnaNPjb32DoUOtj2iVXwhKEiGQATwK9gTbAABFpU2CylcAgYHyB4buAa1U1E+gFPCoiRyUq1rDkdyLkZxHOHW7XLrjhBjjhBGvK2yVfIusCdAOWqOoyABGZAFwCHGiFSFVzgnF5kTOq6jcRr9eKyEagAbA9gfEmXfPm1kRxz55hR+Jc6rnnHli61PpQqV497GgqpkQWMTUGVkW8Xx0MKxER6QYcASyNMu4mEckWkexN5fSus0GDvJliV36pWouqv/oVrFsX3+VWrWptLfkfqPCkdG1iETkOeAUYqKp5Bcer6rPAs2A3yiU5vLjYu9fqdzdoAOefH3Y0zpXMhAkwcqS9fvFFWLkS6tYt+3JF4MEHvVn8sCXyDGIN0DTifZNgWExEpDbwNvB7Vf0szrGljEqV4NFH4bLL4Isvwo7GuditX2/tIp16KixeDI88cjA5vPYa5OaWbrlPPAEffGCvvRnvcCUyQcwEWolISxE5AugPTI5lxmD6icBYVX0jgTGGLiPDLlIfeyxcdJHXaHLlx+LFUK0avPSS3Z9www02fOlS6N/fGtN7+GHYvTv2Zc6dC7/5jS3ThS9hCUJV9wHDgKnAQuB1VZ0vIveJSF8AEekqIquBK4BnRCS4HYYrgR7AIBGZEzw6JirWsDVsCP/6l73u1cv+mTmX6s4+G5Yts6a3I51wgp0Nd+4Mt98OJ55oVVV//LHo5e3bZz3DHX20nVW78HljfSnkiy/gnHNg1Cir9+1cKlq3zlpU/cUvrIi0KNOnw+9/b2cGy5dD/fqFT/vQQ3DXXfCPf8Dllxc+nYuvohrr8wSRYlauhGbNwo7CuehU4dJL4d13rfmLWPpkULUzjRNOsNfXXgv9+tly8q8xLFtmLbVedJElH5c8RSUIb2ojxeQnh5kz4dZbIe+wulvOhWfcOJg8GR54IPYOe0QsOYAVn86caQni1FMt0ahaVe9HH4Unn0xU5K40PEGkqGnT4PHHrR0a51LBunX2p+WMM+DXvy7dMo47Dr7+2m4Q3bgRLrgA/uu/7PrEkCFWWcOlDk8QKer2260K4ahR8Ne/hh2Nc3ZdbPduO7hnZJR+OZUr2w2iixfbn6Avv7QOgFzq8WsQKWz/fqsu+MYbMH48DBgQdkSuIps2DXJy7OAeT/nFqMVd8HaJ4T3KlVMZGfDKK9Z39RtvWLLwG4dcsuXl2cE7UU1eeGJIXZ4gUly1atap0JFHenJwyadqd/m3b+8tqlZEnrvLgVq1rNx2/Xq48EKrT+5Sw6ZN1vFTul4nGjvWai0dfXTYkbgweIIoR7Ztg88+s5ofmzeHHY0DGDjQmqO+6y7r/CmdrFljrbR27261l1zF4wmiHGnd2v7NrVplnQ19/33YEbnHH7eG6apWtYNpmtT5QBVuusmqn5a11pIrvzxBlDNnngmvvmo3G111lbVf45Jr3Tq7UUzVbgC78kr44x+tPa1588KOLj4WLLAWVf/nf6BVq7CjcWHxBFEOXXqp3XG6bBls2RJ2NBXL2rVWm+ehhw4tUho2zJJ2hw6hhRZXmZl2Q9stt4QdiQuTJ4hyasgQmD3bWoJNl2KNVLdmjSWHtWth6lRr4jpflSrQpYu93rgxlPDiQhVmzLDXJ57oVVArOv/6y7Fq1ayM+Oc/t9omLnFWrbLmrdevt+TQvXv06Z5/Hlq2tBvKyqOXX7bPOWVK2JG4VOAJopyrVMnKxG+5xQ5iLjEWLYIdO6xxuTPOKHy6Cy6w59tuS05c8bR6tV1o79HD+iVxzhNEOVe5svUFvH8/3HijFzfF25499vzTn9o1n9NOK3r6pk3hD3+AiRPtTKO8ULX9Z98+25+8aMmBJ4i0cPzx8Oc/2wHphRfCjiZ95OTYxdpXX7X3NWvGNt9tt1n5/a23Ft+LWqoYM8ZqYT300MGmuZ3zBJEmhg61C6j33w9794YdTfm3fLmVxW/bdujF6FhUrQqjR1uR36xZiYkv3mrXtj4afvnLsCNxqcRbc00jK1dakVOjRmFHUr4tXWpdv37/Pbz/PnTqVLrlbNwIxxwT39jiIS/PquhmZ1sCGzHCEoSrmLxHuQqiWTNLDnl5dqOTK7mtW+1MbNcuu1GstMkBLDmohtvXQV7ewZspP/nEPttRR8Epp1hHPU8/nX5NhLj48QSRhoYPt4upK1eGHUn5U7eu3fT24YfQsWPZl/fSS/ZdTJ9e9mUVRxWWLLGmP+64wxoRrFv3YB/PVavaRfdrrrEL0fPmQW7uwfs3nCvIi5jS0PLl0K6dVcecOtWbCY/F4sV28Iz3ndC7dkGbNtYi75dfWhFgPOXlwc6dVkS0YoX17QyWDDp0sIP/oEHQrVt81+vShxcxVTAtW8Jf/gLvvWc3brmiLVxoRS9XX32wd7N4qV4dHnnEmq146qn4LnvbNrj4Ymt6Zf9+K2J88UVLRLm5VrT11FOeHFzp+RlEmsrLg/POswuRX39tBw93uE2brChp/34rVmrTJv7rUIXeveHTT+Gbb6x5lLKaN8868lm1ympM3Xyznym60vEziAqoUiW7J6JZM9iwIexoUpMqXH+99a3xr38lJjmAHbgfe8w63Vm2rOzLGz/ermvs2WPXNoYM8eTgEsO7HE1jLVvCV1/5waMw48ZZd66PPhqfC9JFOflku4Bc1n4Vdu2Cu++Grl3tYvSxx8YnPuei8TOINCdi/zT/+7/tIqY76Ior4JlnktdbWkaG3Vmd3zRKSWzcaPNWrw7Tptn9GZ4cXKJ5gqgANmywC6XeVpPZvRu2b7eaPjfdlNwzrClTrFirJJUHPv3U7scYPtzet2hhzYs7l2ieICqA5s1h5Eir1fTcc2FHE7477oDOna2mT7Jdcok14XH33cV39qRqN7KdfbY17T5wYHJidC6fJ4gK4uab7cap3/62/PZVEA9vvmm98V12md2bkGwi1o/1d99Zq6+F2b0bBg+GX/ziYG20dOmtzpUfCU0QItJLRBaLyBIRGR5lfA8RmS0i+0Tk8gLjBorIt8HD/zuVUX6tJrCDTkW0di1cd50V1/zpT+HF0a6d3a39zDPWK2A0OTnwxhtwzz3w1lt2R7RzyZawWkwikgE8CfwUWA3MFJHJqhrZStBKYBBwe4F5jwbuBbIABWYF825LVLwVQYsW1mNYSVsnTQd5eVZEs3u3Nd9dtWq48YwYYZ0QFbRggVW3bd3aaj3F454J50orkWcQ3YAlqrpMVX8EJgCXRE6gqjmqOg8oeP/qBcB7qro1SArvAd7HVRz06wdt29rr/M5wKoLcXKs59NhjVuU0bEcdZfdedO5s71WtL4Z27WDCBBvmycGFLZH3QTQGIjvBXA2cWoZ5G8cpLofdXLVkiV24rgj3SdSpY1VDU+2zbt0K//M/1sT4xIlw1VXQp0/YUTlnyvVFahG5SUSyRSR706ZNYYdTrnTsaM1ZP/NM2JEkVm4u3HCD9dtdqVLqJYicHHj4YZg82Z5ffTX2nuucS7REJog1QNOI902CYXGbV1WfVdUsVc1q0KBBqQOtiG6+2WrH3HFHetdquvVWuzEtVfs86NzZLkbPmGFdlaZaAnMVWyITxEyglYi0FJEjgP7A5BjnnQqcLyJ1RaQucH4wzMWJiN2sJWL/sNPxBroJE6w/ht//Hnr0CDuawvXrZ02zO5dqEnYNQlX3icgw7MCeAbyoqvNF5D4gW1Uni0hXYCJQF7hYRP6oqpmqulVE7seSDMB9qro1UbFWVM2bw6hRdjPWjh1WTv/zn9ud1zVqHHx06WJ9XoNVld2/35p8yB/ftKnVukklOTl2lnT66XDvvWFH41z5lNDG+lR1CjClwLB7Il7PxIqPos37IvBiIuNz1vxGo0aWHMCacPjhB7t4umuX9cv8ww8HE8Tw4db6aaSrr4a//91ef/fdwWWF6c477XncuPh30uNcReE/nQpO5NBaMy+/XPT0S5ZY0oh85N/ENXeu/WMfONDK01u1Slzcxfnb36wfjJYtw4vBufKuXNdicslXp46dcbRqZTWhunc/2I9CnTp2NvHii3avQb9+1tBcMn37rbV6WrcunHVWctftXLrxBOHipkULawxwxQprjG7aNDj//OQ1irdtG/zkJ3DNNclZn3PpzhOEi7tjj4UHHoCVK61561q1rJZUfv8Lu3fHf52q1nT3unVw++3FT++cK54nCJcwNWseLObZsgWWL7c7uJs3h/vuO/xid1m8+KLdT/DAA9bbmnOu7DxBuKSoXx9mzoSPPrID+L33Wn/Z8bhGsXix3RB37rl2459zLj68FpNLGhHo2dMe8+dbcVN+Y3VDh8I//2nT5D8aNoQvv7TxN9wAH3546PiWLeHdd+2+jK5dYexYa07DORcfniBcKDIzYfTog+/zi4Xy8ux6girUrn3o9D/+eHBcXt7B1k7btLEL4s65+BJNkzYWsrKyNDs7O+wwnHOuXBGRWaqaFW2cn5A755yLyhOEc865qDxBOOeci8oThHPOuag8QTjnnIvKE4RzzrmoPEE455yLyhOEc865qNLmRjkR2QSsCDuOItQH4tg8Xdx5fGXj8ZWNx1c2ZYmvuao2iDYibRJEqhOR7MLuVkwFHl/ZeHxl4/GVTaLi8yIm55xzUXmCcM45F5UniOR5NuwAiuHxlY3HVzYeX9kkJD6/BuGccy4qP4NwzjkXlScI55xzUXmCiBMRaSoiH4nIAhGZLyK/ijJNTxH5TkTmBI97QogzR0S+CtZ/WA9LYkaLyBIRmScinZMY28kR22aOiOwQkV8XmCap21BEXhSRjSLydcSwo0XkPRH5NniuW8i8A4NpvhWRgUmMb6SILAq+v4kiclQh8xa5LyQwvhEisibiO7ywkHl7icjiYF8cnsT4XouILUdE5hQybzK2X9TjStL2QVX1RxwewHFA5+B1LeAboE2BaXoCb4UcZw5Qv4jxFwLvAAKcBnweUpwZwHrsJp7QtiHQA+gMfB0x7C/A8OD1cODPUeY7GlgWPNcNXtdNUnznA5WD13+OFl8s+0IC4xsB3B7D978UOB44Aphb8PeUqPgKjH8YuCfE7Rf1uJKsfdDPIOJEVdep6uzgdS6wEGgcblSlcgkwVs1nwFEiclwIcfwEWKqqod4dr6ozgK0FBl8CvBy8fhm4NMqsFwDvqepWVd0GvAf0SkZ8qvququ4L3n4GNIn3emNVyPaLRTdgiaouU9UfgQnYdo+rouITEQGuBF6N93pjVcRxJSn7oCeIBBCRFkAn4PMoo08Xkbki8o6IZCY3MgAUeFdEZonITVHGNwZWRbxfTTiJrj+F/zDD3oYNVXVd8Ho90DDKNKmyHQdjZ4TRFLcvJNKwoAjsxUKKR1Jh+50FbFDVbwsZn9TtV+C4kpR90BNEnIlITeB/gV+r6o4Co2djRSYdgMeBSUkOD+BMVe0M9AZ+KSI9QoihSCJyBNAX+EeU0amwDQ9QO5dPybriIvJ7YB8wrpBJwtoXngZOADoC67BinFQ0gKLPHpK2/Yo6riRyH/QEEUciUgX7Esep6j8LjlfVHaq6M3g9BagiIvWTGaOqrgmeNwITsVP5SGuAphHvmwTDkqk3MFtVNxQckQrbENiQX+wWPG+MMk2o21FEBgF9gKuDA8hhYtgXEkJVN6jqflXNA54rZL1hb7/KQD/gtcKmSdb2K+S4kpR90BNEnATllS8AC1X1r4VMc2wwHSLSDdv+W5IYYw0RqZX/GruY+XWBySYD14o5Dfgu4lQ2WQr95xb2NgxMBvJrhAwE/i/KNFOB80WkblCEcn4wLOFEpBfwO6Cvqu4qZJpY9oVExRd5TeuyQtY7E2glIi2DM8r+2HZPlvOARaq6OtrIZG2/Io4rydkHE3kFviI9gDOx07x5wJzgcSEwBBgSTDMMmI/VyPgMOCPJMR4frHtuEMfvg+GRMQrwJFaD5CsgK8kx1sAO+HUihoW2DbFEtQ7Yi5XhXg/UAz4AvgXeB44Ops0Cno+YdzCwJHhcl8T4lmBlz/n74d+CaRsBU4raF5IU3yvBvjUPO9AdVzC+4P2FWK2dpcmMLxj+Uv4+FzFtGNuvsONKUvZBb2rDOedcVF7E5JxzLipPEM4556LyBOGccy4qTxDOOeei8gThnHMuKk8QzhVDRPbLoa3Mxq1lURFpEdmSqHOppHLYAThXDuxW1Y5hB+FcsvkZhHOlFPQH8JegT4AvROTEYHgLEfkwaIzuAxFpFgxvKNY/w9zgcUawqAwReS5o7/9dETkymP7WoB+AeSIyIaSP6SowTxDOFe/IAkVMV0WM+05V2wFPAI8Gwx4HXlbV9lhDeaOD4aOB6WoNDXbG7sAFaAU8qaqZwHbgZ8Hw4UCnYDlDEvPRnCuc30ntXDFEZKeq1owyPAc4V1WXBQ2qrVfVeiKyGWs+Ym8wfJ2q1heRTUATVf0hYhktsDb7WwXv7wSqqOoDIvIvYCfWYu0kDRopdC5Z/AzCubLRQl6XxA8Rr/dz8NrgRVi7WJ2BmUELo84ljScI58rmqojnT4PXn2CtjwJcDXwcvP4AGAogIhkiUqewhYpIJaCpqn4E3AnUAQ47i3EukfwfiXPFO1IO7bj+X6qaX9W1rojMw84CBgTDbgHGiMgdwCbgumD4r4BnReR67ExhKNaSaDQZwN+DJCLAaFXdHqfP41xM/BqEc6UUXIPIUtXNYcfiXCJ4EZNzzrmo/AzCOedcVH4G4ZxzLipPEM4556LyBOGccy4qTxDOOeei8gThnHMuqv8HbcZ6FwmrFdQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "val_loss = history_large_model.history[\"val_loss\"]\n",
    "epochs = range(1, 21)\n",
    "plt.plot(epochs, val_loss, \"b--\",\n",
    "         label=\"Validation loss\")\n",
    "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Improving generalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Dataset curation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Using early stopping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Regularizing your model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Reducing the network's size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Original model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 1s 0us/step\n",
      "17473536/17464789 [==============================] - 1s 0us/step\n",
      "Epoch 1/20\n",
      "30/30 [==============================] - 3s 72ms/step - loss: 0.5036 - accuracy: 0.7873 - val_loss: 0.3730 - val_accuracy: 0.8739\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 0s 13ms/step - loss: 0.2927 - accuracy: 0.9059 - val_loss: 0.3030 - val_accuracy: 0.8849\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 0s 13ms/step - loss: 0.2124 - accuracy: 0.9311 - val_loss: 0.3260 - val_accuracy: 0.8678\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 0s 12ms/step - loss: 0.1669 - accuracy: 0.9450 - val_loss: 0.2917 - val_accuracy: 0.8821\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 0s 12ms/step - loss: 0.1345 - accuracy: 0.9583 - val_loss: 0.2867 - val_accuracy: 0.8886\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.1148 - accuracy: 0.9626 - val_loss: 0.3127 - val_accuracy: 0.8832\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 0s 12ms/step - loss: 0.0958 - accuracy: 0.9715 - val_loss: 0.3215 - val_accuracy: 0.8827\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0799 - accuracy: 0.9777 - val_loss: 0.3417 - val_accuracy: 0.8816\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0668 - accuracy: 0.9825 - val_loss: 0.3670 - val_accuracy: 0.8792\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0578 - accuracy: 0.9850 - val_loss: 0.4216 - val_accuracy: 0.8699\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0479 - accuracy: 0.9874 - val_loss: 0.4330 - val_accuracy: 0.8729\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0366 - accuracy: 0.9924 - val_loss: 0.4603 - val_accuracy: 0.8725\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0364 - accuracy: 0.9902 - val_loss: 0.4800 - val_accuracy: 0.8721\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0241 - accuracy: 0.9954 - val_loss: 0.5139 - val_accuracy: 0.8699\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0217 - accuracy: 0.9961 - val_loss: 0.5509 - val_accuracy: 0.8685\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0187 - accuracy: 0.9963 - val_loss: 0.6047 - val_accuracy: 0.8657\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 0s 14ms/step - loss: 0.0137 - accuracy: 0.9981 - val_loss: 0.6159 - val_accuracy: 0.8674\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0129 - accuracy: 0.9979 - val_loss: 0.6577 - val_accuracy: 0.8651\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0069 - accuracy: 0.9995 - val_loss: 0.7002 - val_accuracy: 0.8634\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.0086 - accuracy: 0.9990 - val_loss: 0.7484 - val_accuracy: 0.8618\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.datasets import imdb\n",
    "(train_data, train_labels), _ = imdb.load_data(num_words=10000)\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.\n",
    "    return results\n",
    "train_data = vectorize_sequences(train_data)\n",
    "\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(16, activation=\"relu\"),\n",
    "    layers.Dense(16, activation=\"relu\"),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "history_original = model.fit(train_data, train_labels,\n",
    "                             epochs=20, batch_size=512, validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Version of the model with lower capacity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "30/30 [==============================] - 3s 72ms/step - loss: 0.5507 - accuracy: 0.7831 - val_loss: 0.4519 - val_accuracy: 0.8543\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 0s 12ms/step - loss: 0.3840 - accuracy: 0.8879 - val_loss: 0.3659 - val_accuracy: 0.8762\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.3011 - accuracy: 0.9085 - val_loss: 0.3214 - val_accuracy: 0.8846\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.2474 - accuracy: 0.9249 - val_loss: 0.2948 - val_accuracy: 0.8871\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.2102 - accuracy: 0.9356 - val_loss: 0.2797 - val_accuracy: 0.8931\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.1826 - accuracy: 0.9444 - val_loss: 0.2759 - val_accuracy: 0.8908\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 0s 11ms/step - loss: 0.1604 - accuracy: 0.9503 - val_loss: 0.2728 - val_accuracy: 0.8927\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.1408 - accuracy: 0.9594 - val_loss: 0.2753 - val_accuracy: 0.8896\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.1259 - accuracy: 0.9638 - val_loss: 0.2836 - val_accuracy: 0.8862\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.1127 - accuracy: 0.9676 - val_loss: 0.2887 - val_accuracy: 0.8868\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 0s 14ms/step - loss: 0.1007 - accuracy: 0.9723 - val_loss: 0.2963 - val_accuracy: 0.8856\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0905 - accuracy: 0.9747 - val_loss: 0.3101 - val_accuracy: 0.8841\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0806 - accuracy: 0.9788 - val_loss: 0.3301 - val_accuracy: 0.8805\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0731 - accuracy: 0.9813 - val_loss: 0.3344 - val_accuracy: 0.8812\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0654 - accuracy: 0.9849 - val_loss: 0.3461 - val_accuracy: 0.8818\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0586 - accuracy: 0.9864 - val_loss: 0.3751 - val_accuracy: 0.8772\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0526 - accuracy: 0.9879 - val_loss: 0.3856 - val_accuracy: 0.8754\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0468 - accuracy: 0.9897 - val_loss: 0.3948 - val_accuracy: 0.8766\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0415 - accuracy: 0.9912 - val_loss: 0.4135 - val_accuracy: 0.8766\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.0371 - accuracy: 0.9922 - val_loss: 0.4381 - val_accuracy: 0.8730\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(4, activation=\"relu\"),\n",
    "    layers.Dense(4, activation=\"relu\"),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "history_smaller_model = model.fit(\n",
    "    train_data, train_labels,\n",
    "    epochs=20, batch_size=512, validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Version of the model with higher capacity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "history_larger_model = model.fit(\n",
    "    train_data, train_labels,\n",
    "    epochs=20, batch_size=512, validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Adding weight regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Adding L2 weight regularization to the model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import regularizers\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(16,\n",
    "                 kernel_regularizer=regularizers.l2(0.002),\n",
    "                 activation=\"relu\"),\n",
    "    layers.Dense(16,\n",
    "                 kernel_regularizer=regularizers.l2(0.002),\n",
    "                 activation=\"relu\"),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "history_l2_reg = model.fit(\n",
    "    train_data, train_labels,\n",
    "    epochs=20, batch_size=512, validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Different weight regularizers available in Keras**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import regularizers\n",
    "regularizers.l1(0.001)\n",
    "regularizers.l1_l2(l1=0.001, l2=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Adding dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Adding dropout to the IMDB model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(16, activation=\"relu\"),\n",
    "    layers.Dropout(0.5),\n",
    "    layers.Dense(16, activation=\"relu\"),\n",
    "    layers.Dropout(0.5),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "history_dropout = model.fit(\n",
    "    train_data, train_labels,\n",
    "    epochs=20, batch_size=512, validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Summary"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "chapter05_fundamentals-of-ml.i",
   "private_outputs": false,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "deeplearning",
   "language": "python",
   "name": "deeplearning"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}