CS231N_TensorFlow_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow 2.0 Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: This notebook is meant to teach you the latest version of Tensorflow 2.0. Most examples on the web today are still in 1.x, so be careful not to confuse the two when looking up documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Tensorflow 2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to run this notebook on your local machine, please make sure you follow steps to install the dependencies. You will need TensorFlow version 2.0, and here are some tips to get you started. Please note that Tensorflow 2.0 is still not in a fully 100% stable release, but it's still usable and more intuitive than TF 1.x.\n",
    "\n",
    "     1. Have the latest version of Anaconda installed on your machine.\n",
    "     2. Create a new conda environment starting from Python 3.7. In this setup example, we'll call it tf_20_env.\n",
    "     3. Run the command: source activate tf_20_env\n",
    "     4. Then pip install TF 2.0 as described here: https://www.tensorflow.org/install/pip\n",
    "     \n",
    "\n",
    "A guide on creating Anaconda enviornments: https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/\n",
    "\n",
    "This will give you an new enviornemnt to play in TF 2.0. Generally, if you plan to also use TensorFlow in your other projects, you might also want to keep a seperate Conda environment or virtualenv in Python 3.7 that has Tensorflow 1.9, so you can switch back and forth at will.\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "import os\n",
    "\n",
    "import tensorflow as tf\n",
    "# import tensorflow_datasets as tfds\n",
    "\n",
    "from tensorflow.keras.layers import Dense, Flatten, Conv2D\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import Model\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this part, we will work with the MNIST data. We will demonstrate how the built-in dataset works in TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist = tf.keras.datasets.mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAOZ0lEQVR4nO3dbYxc5XnG8euKbezamMQbB9chLjjgFAg0Jl0ZEBZQobgOqgSoCsSKIkJpnSY4Ca0rQWlV3IpWbpUQUUqRTHExFS+BBIQ/0CTUQpCowWWhBgwEDMY0NmaNWYENIX5Z3/2w42iBnWeXmTMv3vv/k1Yzc+45c24NXD5nznNmHkeEAIx/H+p0AwDag7ADSRB2IAnCDiRB2IEkJrZzY4d5ckzRtHZuEkjlV3pbe2OPR6o1FXbbiyVdJ2mCpH+LiJWl50/RNJ3qc5rZJICC9bGubq3hw3jbEyTdIOnzkk6UtMT2iY2+HoDWauYz+wJJL0TE5ojYK+lOSedV0xaAqjUT9qMk/WLY4621Ze9ie6ntPtt9+7Snic0BaEbLz8ZHxKqI6I2I3kma3OrNAaijmbBvkzRn2ONP1JYB6ELNhP1RSfNsz7V9mKQvSlpbTVsAqtbw0FtE7Le9TNKPNDT0tjoinq6sMwCVamqcPSLul3R/Rb0AaCEulwWSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJpmZxRffzxPJ/4gkfm9nS7T/3F8fUrQ1OPVBc9+hjdxTrU7/uYv3Vaw+rW3u893vFdXcOvl2sn3r38mL9uD9/pFjvhKbCbnuLpN2SBiXtj4jeKpoCUL0q9uy/FxE7K3gdAC3EZ3YgiWbDHpJ+bPsx20tHeoLtpbb7bPft054mNwegUc0exi+MiG22j5T0gO2fR8TDw58QEaskrZKkI9wTTW4PQIOa2rNHxLba7Q5J90paUEVTAKrXcNhtT7M9/eB9SYskbayqMQDVauYwfpake20ffJ3bI+KHlXQ1zkw4YV6xHpMnFeuvnPWRYv2d0+qPCfd8uDxe/JPPlMebO+k/fzm9WP/Hf1lcrK8/+fa6tZf2vVNcd2X/54r1j//k0PtE2nDYI2KzpM9U2AuAFmLoDUiCsANJEHYgCcIOJEHYgST4imsFBs/+bLF+7S03FOufmlT/q5jj2b4YLNb/5vqvFOsT3y4Pf51+97K6tenb9hfXnbyzPDQ3tW99sd6N2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs1dg8nOvFOuP/WpOsf6pSf1VtlOp5dtPK9Y3v1X+Kepbjv1+3dqbB8rj5LP++b+L9VY69L7AOjr27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQhCPaN6J4hHviVJ/Ttu11i4FLTi/Wdy0u/9zzhCcPL9af+Pr1H7ing67Z+TvF+qNnlcfRB994s1iP0+v/APGWbxZX1dwlT5SfgPdZH+u0KwZGnMuaPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4exeYMPOjxfrg6wPF+ku31x8rf/rM1cV1F/zDN4r1I2/o3HfK8cE1Nc5ue7XtHbY3DlvWY/sB25tqtzOqbBhA9cZyGH+LpPfOen+lpHURMU/SutpjAF1s1LBHxMOS3nsceZ6kNbX7aySdX3FfACrW6G/QzYqI7bX7r0qaVe+JtpdKWipJUzS1wc0BaFbTZ+Nj6Axf3bN8EbEqInojoneSJje7OQANajTs/bZnS1Ltdkd1LQFohUbDvlbSxbX7F0u6r5p2ALTKqJ/Zbd8h6WxJM21vlXS1pJWS7rJ9qaSXJV3YyibHu8Gdrze1/r5djc/v/ukvPVOsv3bjhPILHCjPsY7uMWrYI2JJnRJXxwCHEC6XBZIg7EAShB1IgrADSRB2IAmmbB4HTrji+bq1S04uD5r8+9HrivWzvnBZsT79e48U6+ge7NmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2ceB0rTJr3/thOK6/7f2nWL9ymtuLdb/8sILivX43w/Xrc35+58V11Ubf+Y8A/bsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEUzYnN/BHpxfrt1397WJ97sQpDW/707cuK9bn3bS9WN+/eUvD2x6vmpqyGcD4QNiBJAg7kARhB5Ig7EAShB1IgrADSTDOjqI4Y36xfsTKrcX6HZ/8UcPbPv7BPy7Wf/tv63+PX5IGN21ueNuHqqbG2W2vtr3D9sZhy1bY3mZ7Q+3v3CobBlC9sRzG3yJp8QjLvxsR82t/91fbFoCqjRr2iHhY0kAbegHQQs2coFtm+8naYf6Mek+yvdR2n+2+fdrTxOYANKPRsN8o6VhJ8yVtl/Sdek+MiFUR0RsRvZM0ucHNAWhWQ2GPiP6IGIyIA5JukrSg2rYAVK2hsNuePezhBZI21nsugO4w6ji77TsknS1ppqR+SVfXHs+XFJK2SPpqRJS/fCzG2cejCbOOLNZfuei4urX1V1xXXPdDo+yLvvTSomL9zYWvF+vjUWmcfdRJIiJiyQiLb266KwBtxeWyQBKEHUiCsANJEHYgCcIOJMFXXNExd20tT9k81YcV67+MvcX6H3zj8vqvfe/64rqHKn5KGgBhB7Ig7EAShB1IgrADSRB2IAnCDiQx6rfekNuBheWfkn7xC+Upm0+av6VubbRx9NFcP3BKsT71vr6mXn+8Yc8OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzj7OufekYv35b5bHum86Y02xfuaU8nfKm7En9hXrjwzMLb/AgVF/3TwV9uxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7IeAiXOPLtZfvOTjdWsrLrqzuO4fHr6zoZ6qcFV/b7H+0HWnFesz1pR/dx7vNuqe3fYc2w/afsb207a/VVveY/sB25tqtzNa3y6ARo3lMH6/pOURcaKk0yRdZvtESVdKWhcR8yStqz0G0KVGDXtEbI+Ix2v3d0t6VtJRks6TdPBayjWSzm9VkwCa94E+s9s+RtIpktZLmhURBy8+flXSrDrrLJW0VJKmaGqjfQJo0pjPxts+XNIPJF0eEbuG12JodsgRZ4iMiFUR0RsRvZM0ualmATRuTGG3PUlDQb8tIu6pLe63PbtWny1pR2taBFCFUQ/jbVvSzZKejYhrh5XWSrpY0sra7X0t6XAcmHjMbxXrb/7u7GL9or/7YbH+px+5p1hvpeXby8NjP/vX+sNrPbf8T3HdGQcYWqvSWD6znyHpy5Kesr2htuwqDYX8LtuXSnpZ0oWtaRFAFUYNe0T8VNKIk7tLOqfadgC0CpfLAkkQdiAJwg4kQdiBJAg7kARfcR2jibN/s25tYPW04rpfm/tQsb5ken9DPVVh2baFxfrjN5anbJ75/Y3Fes9uxsq7BXt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUgizTj73t8v/2zx3j8bKNavOu7+urVFv/F2Qz1VpX/wnbq1M9cuL657/F//vFjveaM8Tn6gWEU3Yc8OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0mkGWffcn7537XnT767Zdu+4Y1ji/XrHlpUrHuw3o/7Djn+mpfq1ub1ry+uO1isYjxhzw4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTgiyk+w50i6VdIsSSFpVURcZ3uFpD+R9FrtqVdFRP0vfUs6wj1xqpn4FWiV9bFOu2JgxAszxnJRzX5JyyPicdvTJT1m+4Fa7bsR8e2qGgXQOmOZn327pO21+7ttPyvpqFY3BqBaH+gzu+1jJJ0i6eA1mMtsP2l7te0ZddZZarvPdt8+7WmqWQCNG3PYbR8u6QeSLo+IXZJulHSspPka2vN/Z6T1ImJVRPRGRO8kTa6gZQCNGFPYbU/SUNBvi4h7JCki+iNiMCIOSLpJ0oLWtQmgWaOG3bYl3Szp2Yi4dtjy2cOedoGk8nSeADpqLGfjz5D0ZUlP2d5QW3aVpCW252toOG6LpK+2pEMAlRjL2fifShpp3K44pg6gu3AFHZAEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IIlRf0q60o3Zr0l6ediimZJ2tq2BD6Zbe+vWviR6a1SVvR0dER8bqdDWsL9v43ZfRPR2rIGCbu2tW/uS6K1R7eqNw3ggCcIOJNHpsK/q8PZLurW3bu1LordGtaW3jn5mB9A+nd6zA2gTwg4k0ZGw215s+znbL9i+shM91GN7i+2nbG+w3dfhXlbb3mF747BlPbYfsL2pdjviHHsd6m2F7W21926D7XM71Nsc2w/afsb207a/VVve0feu0Fdb3re2f2a3PUHS85I+J2mrpEclLYmIZ9raSB22t0jqjYiOX4Bh+0xJb0m6NSJOqi37J0kDEbGy9g/ljIi4okt6WyHprU5P412brWj28GnGJZ0v6Svq4HtX6OtCteF968SefYGkFyJic0TslXSnpPM60EfXi4iHJQ28Z/F5ktbU7q/R0P8sbVent64QEdsj4vHa/d2SDk4z3tH3rtBXW3Qi7EdJ+sWwx1vVXfO9h6Qf237M9tJONzOCWRGxvXb/VUmzOtnMCEadxrud3jPNeNe8d41Mf94sTtC938KI+Kykz0u6rHa42pVi6DNYN42djmka73YZYZrxX+vke9fo9OfN6kTYt0maM+zxJ2rLukJEbKvd7pB0r7pvKur+gzPo1m53dLifX+umabxHmmZcXfDedXL6806E/VFJ82zPtX2YpC9KWtuBPt7H9rTaiRPZniZpkbpvKuq1ki6u3b9Y0n0d7OVdumUa73rTjKvD713Hpz+PiLb/STpXQ2fkX5T0V53ooU5fn5T0RO3v6U73JukODR3W7dPQuY1LJX1U0jpJmyT9l6SeLurtPyQ9JelJDQVrdod6W6ihQ/QnJW2o/Z3b6feu0Fdb3jculwWS4AQdkARhB5Ig7EAShB1IgrADSRB2IAnCDiTx/65XcTNOWsh5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAOx0lEQVR4nO3df5DU9X3H8deb6wmI4EAMhBBSonKhxDQQLxgbE0ycOGBnis40JkzHEGLnMpNoMdo2ju1MnHSmQzMmNmkwKYlEzA+czKiR6VAjXplaE0M4kAiCBkOggidUsAV/4R337h/3NXPqfT+77H53v3v3fj5mbnb3+97vft+z+uK73+9nv/sxdxeA0W9M2Q0AaA7CDgRB2IEgCDsQBGEHgviDZm7sNBvr4zShmZsEQnlFL+pVP2HD1eoKu5ktkvQNSW2SvufuK1PPH6cJusAuqWeTABI2e3dureaP8WbWJmmVpMWS5kpaamZza309AI1VzzH7AklPufted39V0l2SlhTTFoCi1RP2GZKeHvL4QLbsdcysy8x6zKynTyfq2ByAejT8bLy7r3b3TnfvbNfYRm8OQI56wn5Q0swhj9+RLQPQguoJ+xZJs83sXWZ2mqRPSVpfTFsAilbz0Ju795vZNZJ+psGhtzXu/nhhnQEoVF3j7O6+QdKGgnoB0EB8XRYIgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIJo6ZTNGn/6PnZ+s934+f8qvX1+4Nrnu+x5Zlqy/fdVpyXrbpm3JejTs2YEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMbZkTSwcH6y/s0130rWz23P/19soMK2H73w+8n6k50nk/W/mfXBCluIpa6wm9k+ScclnZTU7+6dRTQFoHhF7Nk/6u7PFfA6ABqIY3YgiHrD7pIeMLOtZtY13BPMrMvMesysp0/535MG0Fj1foy/yN0PmtlUSRvN7Al3f2joE9x9taTVkjTJpnid2wNQo7r27O5+MLs9LOleSQuKaApA8WoOu5lNMLOJr92XdKmknUU1BqBY9XyMnybpXjN77XV+7O73F9IVmqbv0vRo6d/e9oNkvaM9fU35QGI0fW9fX3Ld/xsYm6zPT5d1YvEHcmvjN+1IrjvwyivpFx+Bag67u++V9L4CewHQQAy9AUEQdiAIwg4EQdiBIAg7EASXuI4CbZMm5dZe/Mic5LpfvPXHyfpHx79QYeu17y/ueP5PkvXu2y5M1n9+8zeT9Y3f+05ube4Pr0mue/aXHknWRyL27EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBOPso8CBO2fk1rZ8YFUTOzk1X5m6JVm//4z0OPzyfZcm62tnPZhbmzT3SHLd0Yg9OxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EwTj7CND/sfOT9XXz8qdNHqP0Tz1Xsnz/Jcl6z4N/lKzvuDq/t00vj0uuO7Xn5WT9qefT1+q3/+Om3NoYS646KrFnB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgzN2btrFJNsUvsPS4bUQDC+cn6/+89rZk/dz22r8u8WdPXJGst/35i8n60T99d7J+5Lz8Ae2OVU8n1+1/+kCyXsm/HdyaW+s9mR7D/+yyv0rW2zZtq6mnRtvs3TrmR4d90yvu2c1sjZkdNrOdQ5ZNMbONZrYnu51cZMMAilfNx/g7JC16w7IbJXW7+2xJ3dljAC2sYtjd/SFJR9+weImktdn9tZIuL7gvAAWr9WBvmrv3ZveflTQt74lm1iWpS5LG6fQaNwegXnWfjffBM3y5Z/ncfbW7d7p7Z7vG1rs5ADWqNeyHzGy6JGW3h4trCUAj1Br29ZKWZfeXSbqvmHYANErFY3YzWyfpYklnmdkBSV+WtFLST8zsakn7JV3ZyCZHOjv/Pcn6c9enx3w72tPXpG89kV/7jxfmJtc9ctfMZP0tz6fnKT/zh79M1xO1/uSajTWtLX1IeeS6l5L1qfmXyresimF396U5Jb4dA4wgfF0WCIKwA0EQdiAIwg4EQdiBIPgp6QKMOT39NeD+rx5L1n85555k/Xf9rybr1990Q25t8n/9d3LdqRPS34c6mayOXgum70/W9zWnjUKxZweCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIBhnL8DLC9OXsP5sTvqnoCv5yxVfTNYn/jT/MtMyLyNFa2HPDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBMM5egD/+h+3J+pgK/6Yu35/+od7xP/3VKfcEqd3acmt9FWYqb7PmTWXeLOzZgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIxtmr9L9XXZhb+/tptyTXHVCFKZcfSE+r/E79IlnH8Po8/1fvBzSQXPf+3en/JrO1raaeylRxz25ma8zssJntHLLsZjM7aGbbs7/LGtsmgHpV8zH+DkmLhll+q7vPy/42FNsWgKJVDLu7PyTpaBN6AdBA9Zygu8bMHss+5k/Oe5KZdZlZj5n19OlEHZsDUI9aw/5tSedImiepV9LX8p7o7qvdvdPdO9s1tsbNAahXTWF390PuftLdByR9V9KCYtsCULSawm5m04c8vELSzrznAmgNFcfZzWydpIslnWVmByR9WdLFZjZPkmtwqurPNbDHltA/Pr925pj0OPojr6QPX86+85n0tpPV0avSvPdP3HJehVfYmlv5i72Lk2vOWfG7ZH0kzltfMezuvnSYxbc3oBcADcTXZYEgCDsQBGEHgiDsQBCEHQiCS1yb4MjJM5L1/r37mtNIi6k0tPbkyvcm608s+Vay/u8vnZlbe2bVucl1Jz6fPw32SMWeHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCYJy9Cf76559I1jsSl2KOdAML5+fWDl//cnLd3Z3pcfRLdnwyWZ+waG9ubaJG3zh6JezZgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIxtmrZfmlMRX+zfzGReuS9VXqqKWjlrD/K/lTWUvS3Z/+em6toz39E9zv/9WyZP3tV+xK1vF67NmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjG2avl+aUBDSRXXTj+SLJ+3R3nJ+vnfD/9+u3PHs+tHVr41uS6Uz55IFm/9p3dyfri09PX4q9/cVpu7dM7FiXXPetfJyTrODUV9+xmNtPMNpnZLjN73MxWZMunmNlGM9uT3U5ufLsAalXNx/h+STe4+1xJH5T0BTObK+lGSd3uPltSd/YYQIuqGHZ373X3bdn945J2S5ohaYmktdnT1kq6vFFNAqjfKR2zm9ksSfMlbZY0zd17s9KzkoY9ODOzLkldkjRO6bm9ADRO1WfjzewMSXdLus7djw2tubsr5xSWu692905372zX2LqaBVC7qsJuZu0aDPqP3P2ebPEhM5ue1adLOtyYFgEUoeLHeDMzSbdL2u3uQ69XXC9pmaSV2e19DelwFBhn6bd598e/k6w//OFxyfqeE2/LrS0/c19y3XqteObDyfr9v5iXW5u9It7POZepmmP2D0m6StIOM9ueLbtJgyH/iZldLWm/pCsb0yKAIlQMu7s/rPyfbrik2HYANApflwWCIOxAEIQdCIKwA0EQdiAIG/zyW3NMsil+gY3ME/htHefk1jrW7U+u+09ve6SubVf6qepKl9imPHoi/dpL/7MrWe9YPnqnmx6JNnu3jvnRYUfP2LMDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBD8lHSVTv7mt7m1PZ+YlVx37rXXJuu7rvyXWlqqypwNn0/W333bS8l6x6OMo48W7NmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAiuZwdGEa5nB0DYgSgIOxAEYQeCIOxAEIQdCIKwA0FUDLuZzTSzTWa2y8weN7MV2fKbzeygmW3P/i5rfLsAalXNj1f0S7rB3beZ2URJW81sY1a71d1vaVx7AIpSzfzsvZJ6s/vHzWy3pBmNbgxAsU7pmN3MZkmaL2lztugaM3vMzNaY2eScdbrMrMfMevp0oq5mAdSu6rCb2RmS7pZ0nbsfk/RtSedImqfBPf/XhlvP3Ve7e6e7d7ZrbAEtA6hFVWE3s3YNBv1H7n6PJLn7IXc/6e4Dkr4raUHj2gRQr2rOxpuk2yXtdvevD1k+fcjTrpC0s/j2ABSlmrPxH5J0laQdZrY9W3aTpKVmNk+SS9on6XMN6RBAIao5G/+wpOGuj91QfDsAGoVv0AFBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Jo6pTNZvY/kvYPWXSWpOea1sCpadXeWrUvid5qVWRvf+jubx2u0NSwv2njZj3u3llaAwmt2lur9iXRW62a1Rsf44EgCDsQRNlhX13y9lNatbdW7Uuit1o1pbdSj9kBNE/Ze3YATULYgSBKCbuZLTKzJ83sKTO7sYwe8pjZPjPbkU1D3VNyL2vM7LCZ7RyybIqZbTSzPdntsHPsldRbS0zjnZhmvNT3ruzpz5t+zG5mbZJ+I+njkg5I2iJpqbvvamojOcxsn6ROdy/9Cxhm9hFJL0i6093Py5Z9VdJRd1+Z/UM52d2/1CK93SzphbKn8c5mK5o+dJpxSZdL+oxKfO8SfV2pJrxvZezZF0h6yt33uvurku6StKSEPlqeuz8k6egbFi+RtDa7v1aD/7M0XU5vLcHde919W3b/uKTXphkv9b1L9NUUZYR9hqSnhzw+oNaa790lPWBmW82sq+xmhjHN3Xuz+89KmlZmM8OoOI13M71hmvGWee9qmf68Xpyge7OL3P39khZL+kL2cbUl+eAxWCuNnVY1jXezDDPN+O+V+d7VOv15vcoI+0FJM4c8fke2rCW4+8Hs9rCke9V6U1Efem0G3ez2cMn9/F4rTeM93DTjaoH3rszpz8sI+xZJs83sXWZ2mqRPSVpfQh9vYmYTshMnMrMJki5V601FvV7Ssuz+Mkn3ldjL67TKNN5504yr5Peu9OnP3b3pf5Iu0+AZ+d9K+rsyesjp62xJv87+Hi+7N0nrNPixrk+D5zaulvQWSd2S9kh6UNKUFurtB5J2SHpMg8GaXlJvF2nwI/pjkrZnf5eV/d4l+mrK+8bXZYEgOEEHBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0H8Px6GUTt0IpTWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAANSklEQVR4nO3db4wc9X3H8c/Hx9mOnaD4TH29GAcowQ9opZrqMFX4UypSRFAqgxJZsZTElVAvD2IpSHkApa1ClQclURMatRHSBdw4VQpKlCD8gKQYCxWhRI4P4mIb00KoXewYn1MnsgnGf799cEN0wO3seWd2Z33f90ta3e58d3a+GvnjmZ3f7v4cEQIw981rugEAvUHYgSQIO5AEYQeSIOxAEhf0cmPzvSAWanEvNwmk8qZ+o5NxwjPVKoXd9i2Svi5pQNKDEXFf2fMXarGu8U1VNgmgxLbY2rLW8Wm87QFJ35D0UUlXSlpn+8pOXw9Ad1V5z75a0ssR8UpEnJT0iKQ19bQFoG5Vwr5c0qvTHu8vlr2N7THbE7YnTulEhc0BqKLrV+MjYjwiRiNidFALur05AC1UCfsBSSumPb64WAagD1UJ+3ZJV9i+zPZ8SZ+UtLmetgDUreOht4g4bXuDpH/X1NDbxojYXVtnAGpVaZw9Ih6X9HhNvQDoIj4uCyRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKVZnEF+tlvPnFNy9qXv/JA6bpfWvuZ0npM7OqopyZVCrvtvZKOSToj6XREjNbRFID61XFk/9OI+GUNrwOgi3jPDiRRNewh6Qnbz9oem+kJtsdsT9ieOKUTFTcHoFNVT+Ovi4gDtpdJ2mL7xYh4evoTImJc0rgkXeihqLg9AB2qdGSPiAPF30lJj0paXUdTAOrXcdhtL7b9vrfuS7pZ0vk3HgEkUeU0fljSo7bfep1/i4gf1dJVFxxfU37ScXzpQGl9aONP6mwHPTA52vpY9qW9f97DTvpDx2GPiFck/WGNvQDoIobegCQIO5AEYQeSIOxAEoQdSCLNV1x/cUP5/2uLLv91+QtsrLEZ1GNe+XBpfPB4y9pNy14sXXerP9xRS/2MIzuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJJFmnP3vPva90vqX99zco05Ql4HLLymtv/gnrT8cseqnnypd9wPbd3bUUz/jyA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSaQZZx/06aZbQM0uePCNjtc9/vMLa+zk/MCRHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSmDPj7GevW1Vav37hMz3qBL1y6eL/63jdFU+eqbGT80PbI7vtjbYnbe+atmzI9hbbLxV/l3S3TQBVzeY0/luSbnnHsrslbY2IKyRtLR4D6GNtwx4RT0s68o7FayRtKu5vknRbzX0BqFmn79mHI+Jgcf81ScOtnmh7TNKYJC3Uog43B6CqylfjIyIkRUl9PCJGI2J0UAuqbg5AhzoN+yHbI5JU/J2sryUA3dBp2DdLWl/cXy/psXraAdAtbd+z235Y0o2SLrK9X9IXJd0n6bu275C0T9LabjY5G/s+9p7S+rIBrhecby649IOl9U8Mbe74td/zP78qrc/FUfi2YY+IdS1KN9XcC4Au4uOyQBKEHUiCsANJEHYgCcIOJDFnvuJ6wYeOVVr/zRffX1MnqMur/7i4tH7tgrOl9YeOXty6+OujnbR0XuPIDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJzJlx9qqWTZSP2WJmAxctLa0f+vjKlrWhtftL1/2PlQ+12frC0uoD32j904jLDv24zWvPPRzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtkLx4fK/98r/2Z1NWevv6q0HgMurb/6kdYz7Zz8wKnSdefNL//R5Ceu/6fS+mB5a3rtTOve/vaV20vXPXK2/LMPi+aV9z68rfVvHLScwmgO48gOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0nMmXH2E28OltbPthlZ/Zd77i+tb96w6px7mq27lj5YWp+n8sHs43GyZe0XZ8rHov/58I2l9Y88eWdp/f0/m19aH3niUMua95V/n/3wnvJpuIcHyj9DENt3ltazaXtkt73R9qTtXdOW3Wv7gO0dxe3W7rYJoKrZnMZ/S9ItMyy/PyJWFbfH620LQN3ahj0inpZ0pAe9AOiiKhfoNth+vjjNX9LqSbbHbE/YnjilExU2B6CKTsP+gKTLJa2SdFDSV1s9MSLGI2I0IkYH1fpLEQC6q6OwR8ShiDgTEWclfVPS6nrbAlC3jsJue2Taw9sl7Wr1XAD9oe04u+2HJd0o6SLb+yV9UdKNtldp6mvBeyV9tos9zsqHPvWz0vrv//2G0vqKqw/U2c45eWqy9W+rS9LhH5bMMy5p6e7W483zf7S9zdbLx6pXaqLN+uXKRvkP3PXh0nWvXvCT0vojry/voKO82oY9ItbNsLjdr/cD6DN8XBZIgrADSRB2IAnCDiRB2IEk5sxXXNu57K/Kh3H62Yj+t+kWumLRDYcrrf83T328tL5SP630+nMNR3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSCLNODvmnkseyzjxcuc4sgNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASfJ8dfWvA5ceiX60cLK3/7g/r7Ob81/bIbnuF7adsv2B7t+3PF8uHbG+x/VLxd0n32wXQqdmcxp+W9IWIuFLSH0v6nO0rJd0taWtEXCFpa/EYQJ9qG/aIOBgRzxX3j0naI2m5pDWSNhVP2yTptm41CaC6c3rPbvtSSVdJ2iZpOCIOFqXXJA23WGdM0pgkLdSiTvsEUNGsr8bbfq+k70u6MyKOTq9FREia8df/ImI8IkYjYnRQCyo1C6Bzswq77UFNBf07EfGDYvEh2yNFfUTSZHdaBFCH2VyNt6SHJO2JiK9NK22WtL64v17SY/W3h8zOxNnSm+ap/Ia3mc179mslfVrSTts7imX3SLpP0ndt3yFpn6S13WkRQB3ahj0inpHkFuWb6m0HQLdwsgMkQdiBJAg7kARhB5Ig7EASfMUV5603rn6j6RbOKxzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtnRt9r9lDTODXsTSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnB2NOfHk75TWz6w626NOcuDIDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJOCLKn2CvkPRtScOSQtJ4RHzd9r2S/lLS4eKp90TE42WvdaGH4hoz8SvQLdtiq47GkRlnXZ7Nh2pOS/pCRDxn+32SnrW9pajdHxH/UFejALpnNvOzH5R0sLh/zPYeScu73RiAep3Te3bbl0q6StK2YtEG28/b3mh7SYt1xmxP2J44pROVmgXQuVmH3fZ7JX1f0p0RcVTSA5Iul7RKU0f+r860XkSMR8RoRIwOakENLQPoxKzCbntQU0H/TkT8QJIi4lBEnImIs5K+KWl199oEUFXbsNu2pIck7YmIr01bPjLtabdL2lV/ewDqMpur8ddK+rSknbZ3FMvukbTO9ipNDcftlfTZrnQIoBazuRr/jKSZxu1Kx9QB9Bc+QQckQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUii7U9J17ox+7CkfdMWXSTplz1r4Nz0a2/92pdEb52qs7dLImLGubB7GvZ3bdyeiIjRxhoo0a+99WtfEr11qle9cRoPJEHYgSSaDvt4w9sv06+99WtfEr11qie9NfqeHUDvNH1kB9AjhB1IopGw277F9n/Zftn23U300IrtvbZ32t5he6LhXjbanrS9a9qyIdtbbL9U/J1xjr2GervX9oFi3+2wfWtDva2w/ZTtF2zvtv35Ynmj+66kr57st56/Z7c9IOm/Jf2ZpP2StktaFxEv9LSRFmzvlTQaEY1/AMP2DZJel/TtiPiDYtlXJB2JiPuK/yiXRMRdfdLbvZJeb3oa72K2opHp04xLuk3SX6jBfVfS11r1YL81cWRfLenliHglIk5KekTSmgb66HsR8bSkI+9YvEbSpuL+Jk39Y+m5Fr31hYg4GBHPFfePSXprmvFG911JXz3RRNiXS3p12uP96q/53kPSE7aftT3WdDMzGI6Ig8X91yQNN9nMDNpO491L75hmvG/2XSfTn1fFBbp3uy4i/kjSRyV9rjhd7Usx9R6sn8ZOZzWNd6/MMM34bzW57zqd/ryqJsJ+QNKKaY8vLpb1hYg4UPydlPSo+m8q6kNvzaBb/J1suJ/f6qdpvGeaZlx9sO+anP68ibBvl3SF7ctsz5f0SUmbG+jjXWwvLi6cyPZiSTer/6ai3ixpfXF/vaTHGuzlbfplGu9W04yr4X3X+PTnEdHzm6RbNXVF/ueS/rqJHlr09XuS/rO47W66N0kPa+q07pSmrm3cIWmppK2SXpL0pKShPurtXyXtlPS8poI10lBv12nqFP15STuK261N77uSvnqy3/i4LJAEF+iAJAg7kARhB5Ig7EAShB1IgrADSRB2IIn/Bziw80r6zfkYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for image in x_train[:3]:\n",
    "    plt.figure()\n",
    "    plt.imshow(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting Simple: Using the Keras Sequential API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you will quickly see, there are a ton of ways in TensorFlow to build models and train them. TensorFlow has many APIs that you can use, and all these APIs can be confusing. We start this tutorial with the simplest way to build a model in TensorFlow 2.0: using the Keras Sequential API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first build the tf.keras.Sequential model by stacking layers.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/jloscalzo/.local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dropout(0.2),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 3s 54us/sample - loss: 0.2880 - acc: 0.9158\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 3s 54us/sample - loss: 0.1399 - acc: 0.9590\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 3s 53us/sample - loss: 0.1059 - acc: 0.9678\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 3s 53us/sample - loss: 0.0878 - acc: 0.9733\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 3s 54us/sample - loss: 0.0743 - acc: 0.9769\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7ffa361652e8>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 25us/sample - loss: 0.0745 - acc: 0.9760\n",
      "\n",
      "Test accuracy: 0.976\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = model.evaluate(x_test, y_test)\n",
    "\n",
    "print('\\nTest accuracy:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More advanced: Writing a custom training loop with Gradient Tape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we transition to a more advanced usage of TensorFlow. This API will probably be the one you will use the most, as it provides for more flexibility. Note that you can specify more unique architectures with the forward pass. We will use the same MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reload the dataset \n",
    "def convert_types(image, label):\n",
    "  image = tf.cast(image, tf.float32)\n",
    "  image /= 255\n",
    "  return image, label\n",
    "\n",
    "dataset, info = tfds.load('mnist', data_dir='gs://tfds-data/datasets', with_info=True, as_supervised=True)\n",
    "mnist_train, mnist_test = dataset['train'], dataset['test']\n",
    "mnist_train = mnist_train.map(convert_types).shuffle(10000).batch(32)\n",
    "mnist_test = mnist_test.map(convert_types).batch(32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will go ahead and build the model using tf.keras. Note how building the model this way allows for a greater degree of control over the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyModel(Model):\n",
    "  def __init__(self):\n",
    "    super(MyModel, self).__init__()\n",
    "    self.conv1 = Conv2D(32, 3, activation='relu')\n",
    "    self.flatten = Flatten()\n",
    "    self.d1 = Dense(128, activation='relu')\n",
    "    self.d2 = Dense(10, activation='softmax')\n",
    "\n",
    "  def call(self, x):\n",
    "    x = self.conv1(x)\n",
    "    x = self.flatten(x)\n",
    "    x = self.d1(x)\n",
    "    return self.d2(x)\n",
    "\n",
    "model = MyModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to specify an optimizer and loss function, along with metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_object = tf.keras.losses.SparseCategoricalCrossentropy()\n",
    "optimizer = tf.keras.optimizers.Adam()\n",
    "\n",
    "train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
    "train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')\n",
    "test_loss = tf.keras.metrics.Mean(name='test_loss')\n",
    "test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can write our train function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def train_step(image, label):\n",
    "  with tf.GradientTape() as tape:\n",
    "    predictions = model(image)\n",
    "    loss = loss_object(label, predictions)\n",
    "  gradients = tape.gradient(loss, model.trainable_variables)\n",
    "  optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "\n",
    "  train_loss(loss)\n",
    "  train_accuracy(label, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the tf.function annotation on the function above. This means that this function will be compiled into a graph in the backend, allowing it to run efficiently as TensorFlow can optimize the function for you. This automatic conversion of Python code to its graph representation is called AutoGraph, and this creates callable graphs from Python functions.\n",
    "\n",
    "What is this gradient tape?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.Variable(3.0)\n",
    "with tf.GradientTape() as g:\n",
    "  y = x * x\n",
    "dy_dx = g.gradient(y, x) # Will compute to 6.0\n",
    "print(dy_dx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trainable variables are automatically watched within the GradientTape (when trainable=True, as we will see this is not always the case later in the notebook). Variables are trainable when created by tf.Variable, but not when created via tf.constant. In those cases, if you want to compute the gradients of non-trainable variables, you can manually watch the tensors using the watch function, as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.constant(3.0)\n",
    "with tf.GradientTape() as g:\n",
    "  g.watch(x)\n",
    "  y = x * x\n",
    "dy_dx = g.gradient(y, x) # Will compute to 6.0\n",
    "print(dy_dx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define our test function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def test_step(image, label):\n",
    "  predictions = model(image)\n",
    "  t_loss = loss_object(label, predictions)\n",
    "\n",
    "  test_loss(t_loss)\n",
    "  test_accuracy(label, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to train the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "EPOCHS = 3\n",
    "\n",
    "for epoch in range(EPOCHS):\n",
    "  for image, label in mnist_train:\n",
    "    train_step(image, label)\n",
    "\n",
    "  for test_image, test_label in mnist_test:\n",
    "    test_step(test_image, test_label)\n",
    "\n",
    "  template = 'Epoch {}, Loss: {}, Accuracy: {}, Test Loss: {}, Test Accuracy: {}'\n",
    "  print (template.format(epoch+1,\n",
    "                         train_loss.result(),\n",
    "                         train_accuracy.result()*100,\n",
    "                         test_loss.result(),\n",
    "                         test_accuracy.result()*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save and Load Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "train_labels = train_labels[:1000]\n",
    "test_labels = test_labels[:1000]\n",
    "\n",
    "train_images = train_images[:1000].reshape(-1, 28 * 28) / 255.0\n",
    "test_images = test_images[:1000].reshape(-1, 28 * 28) / 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Returns a short sequential model\n",
    "def create_model():\n",
    "  model = tf.keras.models.Sequential([\n",
    "    keras.layers.Dense(512, activation='relu', input_shape=(784,)),\n",
    "    keras.layers.Dropout(0.2),\n",
    "    keras.layers.Dense(10, activation='softmax')\n",
    "  ])\n",
    "\n",
    "  model.compile(optimizer='adam',\n",
    "                loss='sparse_categorical_crossentropy',\n",
    "                metrics=['accuracy'])\n",
    "\n",
    "  return model\n",
    "\n",
    "\n",
    "# Create a basic model instance\n",
    "model = create_model()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_path = \"training_1/cp.ckpt\"\n",
    "checkpoint_dir = os.path.dirname(checkpoint_path)\n",
    "\n",
    "# Create checkpoint callback\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,\n",
    "                                                 save_weights_only=True,\n",
    "                                                 verbose=1)\n",
    "\n",
    "model = create_model()\n",
    "\n",
    "model.fit(train_images, train_labels,  epochs = 10,\n",
    "          validation_data = (test_images,test_labels),\n",
    "          callbacks = [cp_callback])  # pass callback to training\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_model()\n",
    "\n",
    "loss, acc = model.evaluate(test_images, test_labels)\n",
    "print(\"Untrained model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_weights(checkpoint_path)\n",
    "loss,acc = model.evaluate(test_images, test_labels)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the entire model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_model()\n",
    "\n",
    "model.fit(train_images, train_labels, epochs=5)\n",
    "\n",
    "# Save entire model to a HDF5 file\n",
    "model.save('my_model.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recreate the exact same model, including weights and optimizer.\n",
    "new_model = keras.models.load_model('my_model.h5')\n",
    "new_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss, acc = new_model.evaluate(test_images, test_labels)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all Together: Transfer Learning Using Pretrained Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we will train a model to distinguish between cats and dogs. We will leverage pretrained models for this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin my loading in the dataset, and quickly visualizing it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'tfds' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-11-285daea22dbc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mSPLIT_WEIGHTS\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m8\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0msplits\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtfds\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSplit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTRAIN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubsplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweighted\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mSPLIT_WEIGHTS\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m (raw_train, raw_validation, raw_test), metadata = tfds.load(\n\u001b[1;32m      5\u001b[0m     \u001b[0;34m'cats_vs_dogs'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msplit\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msplits\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'tfds' is not defined"
     ]
    }
   ],
   "source": [
    "SPLIT_WEIGHTS = (8, 1, 1)\n",
    "splits = tfds.Split.TRAIN.subsplit(weighted=SPLIT_WEIGHTS)\n",
    "\n",
    "(raw_train, raw_validation, raw_test), metadata = tfds.load(\n",
    "    'cats_vs_dogs', split=list(splits),\n",
    "    with_info=True, as_supervised=True)\n",
    "\n",
    "get_label_name = metadata.features['label'].int2str\n",
    "\n",
    "for image, label in raw_train.take(2):\n",
    "  plt.figure()\n",
    "  plt.imshow(image)\n",
    "  plt.title(get_label_name(label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will preprocess and shuffle the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMG_SIZE = 128 # All images will be resized to 160x160\n",
    "\n",
    "def format_example(image, label):\n",
    "  image = tf.cast(image, tf.float32)\n",
    "  image = (image/127.5) - 1\n",
    "  image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))\n",
    "  return image, label\n",
    "\n",
    "train = raw_train.map(format_example)\n",
    "validation = raw_validation.map(format_example)\n",
    "test = raw_test.map(format_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 16\n",
    "SHUFFLE_BUFFER_SIZE = 1000\n",
    "train_batches = train.shuffle(SHUFFLE_BUFFER_SIZE).batch(BATCH_SIZE)\n",
    "validation_batches = validation.batch(BATCH_SIZE)\n",
    "test_batches = test.batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have taken care of our data, we will obtain a pretrained model. For this, we will use the MobileNet model (https://arxiv.org/abs/1801.04381). Note that we are using both its architecture, as well as pretrain weights for the model, trained on imagenet. This architecture will act as a feature extractor for us, and that is why we call it \"base_model\". We will then feed the features it extracts into more layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://github.com/JonathanCMitchell/mobilenet_v2_keras/releases/download/v1.1/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_224_no_top.h5\n",
      "9412608/9406464 [==============================] - 7s 1us/step\n",
      "Model: \"mobilenetv2_1.00_224\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 224, 224, 3) 0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Conv1_pad (ZeroPadding2D)       (None, 225, 225, 3)  0           input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "Conv1 (Conv2D)                  (None, 112, 112, 32) 864         Conv1_pad[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "bn_Conv1 (BatchNormalization)   (None, 112, 112, 32) 128         Conv1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "Conv1_relu (ReLU)               (None, 112, 112, 32) 0           bn_Conv1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "expanded_conv_depthwise (Depthw (None, 112, 112, 32) 288         Conv1_relu[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "expanded_conv_depthwise_BN (Bat (None, 112, 112, 32) 128         expanded_conv_depthwise[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "expanded_conv_depthwise_relu (R (None, 112, 112, 32) 0           expanded_conv_depthwise_BN[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "expanded_conv_project (Conv2D)  (None, 112, 112, 16) 512         expanded_conv_depthwise_relu[0][0\n",
      "__________________________________________________________________________________________________\n",
      "expanded_conv_project_BN (Batch (None, 112, 112, 16) 64          expanded_conv_project[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "block_1_expand (Conv2D)         (None, 112, 112, 96) 1536        expanded_conv_project_BN[0][0]   \n",
      "__________________________________________________________________________________________________\n",
      "block_1_expand_BN (BatchNormali (None, 112, 112, 96) 384         block_1_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_1_expand_relu (ReLU)      (None, 112, 112, 96) 0           block_1_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_1_pad (ZeroPadding2D)     (None, 113, 113, 96) 0           block_1_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_1_depthwise (DepthwiseCon (None, 56, 56, 96)   864         block_1_pad[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_1_depthwise_BN (BatchNorm (None, 56, 56, 96)   384         block_1_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_1_depthwise_relu (ReLU)   (None, 56, 56, 96)   0           block_1_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_1_project (Conv2D)        (None, 56, 56, 24)   2304        block_1_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_1_project_BN (BatchNormal (None, 56, 56, 24)   96          block_1_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_2_expand (Conv2D)         (None, 56, 56, 144)  3456        block_1_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_2_expand_BN (BatchNormali (None, 56, 56, 144)  576         block_2_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_2_expand_relu (ReLU)      (None, 56, 56, 144)  0           block_2_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_2_depthwise (DepthwiseCon (None, 56, 56, 144)  1296        block_2_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_2_depthwise_BN (BatchNorm (None, 56, 56, 144)  576         block_2_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_2_depthwise_relu (ReLU)   (None, 56, 56, 144)  0           block_2_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_2_project (Conv2D)        (None, 56, 56, 24)   3456        block_2_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_2_project_BN (BatchNormal (None, 56, 56, 24)   96          block_2_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_2_add (Add)               (None, 56, 56, 24)   0           block_1_project_BN[0][0]         \n",
      "                                                                 block_2_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_3_expand (Conv2D)         (None, 56, 56, 144)  3456        block_2_add[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_3_expand_BN (BatchNormali (None, 56, 56, 144)  576         block_3_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_3_expand_relu (ReLU)      (None, 56, 56, 144)  0           block_3_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_3_pad (ZeroPadding2D)     (None, 57, 57, 144)  0           block_3_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_3_depthwise (DepthwiseCon (None, 28, 28, 144)  1296        block_3_pad[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_3_depthwise_BN (BatchNorm (None, 28, 28, 144)  576         block_3_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_3_depthwise_relu (ReLU)   (None, 28, 28, 144)  0           block_3_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_3_project (Conv2D)        (None, 28, 28, 32)   4608        block_3_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_3_project_BN (BatchNormal (None, 28, 28, 32)   128         block_3_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_4_expand (Conv2D)         (None, 28, 28, 192)  6144        block_3_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_4_expand_BN (BatchNormali (None, 28, 28, 192)  768         block_4_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_4_expand_relu (ReLU)      (None, 28, 28, 192)  0           block_4_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_4_depthwise (DepthwiseCon (None, 28, 28, 192)  1728        block_4_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_4_depthwise_BN (BatchNorm (None, 28, 28, 192)  768         block_4_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_4_depthwise_relu (ReLU)   (None, 28, 28, 192)  0           block_4_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_4_project (Conv2D)        (None, 28, 28, 32)   6144        block_4_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_4_project_BN (BatchNormal (None, 28, 28, 32)   128         block_4_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_4_add (Add)               (None, 28, 28, 32)   0           block_3_project_BN[0][0]         \n",
      "                                                                 block_4_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_5_expand (Conv2D)         (None, 28, 28, 192)  6144        block_4_add[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_5_expand_BN (BatchNormali (None, 28, 28, 192)  768         block_5_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_5_expand_relu (ReLU)      (None, 28, 28, 192)  0           block_5_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_5_depthwise (DepthwiseCon (None, 28, 28, 192)  1728        block_5_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_5_depthwise_BN (BatchNorm (None, 28, 28, 192)  768         block_5_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_5_depthwise_relu (ReLU)   (None, 28, 28, 192)  0           block_5_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_5_project (Conv2D)        (None, 28, 28, 32)   6144        block_5_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_5_project_BN (BatchNormal (None, 28, 28, 32)   128         block_5_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_5_add (Add)               (None, 28, 28, 32)   0           block_4_add[0][0]                \n",
      "                                                                 block_5_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_6_expand (Conv2D)         (None, 28, 28, 192)  6144        block_5_add[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_6_expand_BN (BatchNormali (None, 28, 28, 192)  768         block_6_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_6_expand_relu (ReLU)      (None, 28, 28, 192)  0           block_6_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_6_pad (ZeroPadding2D)     (None, 29, 29, 192)  0           block_6_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_6_depthwise (DepthwiseCon (None, 14, 14, 192)  1728        block_6_pad[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_6_depthwise_BN (BatchNorm (None, 14, 14, 192)  768         block_6_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_6_depthwise_relu (ReLU)   (None, 14, 14, 192)  0           block_6_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_6_project (Conv2D)        (None, 14, 14, 64)   12288       block_6_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_6_project_BN (BatchNormal (None, 14, 14, 64)   256         block_6_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_7_expand (Conv2D)         (None, 14, 14, 384)  24576       block_6_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_7_expand_BN (BatchNormali (None, 14, 14, 384)  1536        block_7_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_7_expand_relu (ReLU)      (None, 14, 14, 384)  0           block_7_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_7_depthwise (DepthwiseCon (None, 14, 14, 384)  3456        block_7_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_7_depthwise_BN (BatchNorm (None, 14, 14, 384)  1536        block_7_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_7_depthwise_relu (ReLU)   (None, 14, 14, 384)  0           block_7_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_7_project (Conv2D)        (None, 14, 14, 64)   24576       block_7_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_7_project_BN (BatchNormal (None, 14, 14, 64)   256         block_7_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_7_add (Add)               (None, 14, 14, 64)   0           block_6_project_BN[0][0]         \n",
      "                                                                 block_7_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_8_expand (Conv2D)         (None, 14, 14, 384)  24576       block_7_add[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_8_expand_BN (BatchNormali (None, 14, 14, 384)  1536        block_8_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_8_expand_relu (ReLU)      (None, 14, 14, 384)  0           block_8_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_8_depthwise (DepthwiseCon (None, 14, 14, 384)  3456        block_8_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_8_depthwise_BN (BatchNorm (None, 14, 14, 384)  1536        block_8_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_8_depthwise_relu (ReLU)   (None, 14, 14, 384)  0           block_8_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_8_project (Conv2D)        (None, 14, 14, 64)   24576       block_8_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_8_project_BN (BatchNormal (None, 14, 14, 64)   256         block_8_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_8_add (Add)               (None, 14, 14, 64)   0           block_7_add[0][0]                \n",
      "                                                                 block_8_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_9_expand (Conv2D)         (None, 14, 14, 384)  24576       block_8_add[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_9_expand_BN (BatchNormali (None, 14, 14, 384)  1536        block_9_expand[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "block_9_expand_relu (ReLU)      (None, 14, 14, 384)  0           block_9_expand_BN[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_9_depthwise (DepthwiseCon (None, 14, 14, 384)  3456        block_9_expand_relu[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_9_depthwise_BN (BatchNorm (None, 14, 14, 384)  1536        block_9_depthwise[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "block_9_depthwise_relu (ReLU)   (None, 14, 14, 384)  0           block_9_depthwise_BN[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_9_project (Conv2D)        (None, 14, 14, 64)   24576       block_9_depthwise_relu[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "block_9_project_BN (BatchNormal (None, 14, 14, 64)   256         block_9_project[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_9_add (Add)               (None, 14, 14, 64)   0           block_8_add[0][0]                \n",
      "                                                                 block_9_project_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_10_expand (Conv2D)        (None, 14, 14, 384)  24576       block_9_add[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "block_10_expand_BN (BatchNormal (None, 14, 14, 384)  1536        block_10_expand[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_10_expand_relu (ReLU)     (None, 14, 14, 384)  0           block_10_expand_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_10_depthwise (DepthwiseCo (None, 14, 14, 384)  3456        block_10_expand_relu[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_10_depthwise_BN (BatchNor (None, 14, 14, 384)  1536        block_10_depthwise[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_10_depthwise_relu (ReLU)  (None, 14, 14, 384)  0           block_10_depthwise_BN[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "block_10_project (Conv2D)       (None, 14, 14, 96)   36864       block_10_depthwise_relu[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "block_10_project_BN (BatchNorma (None, 14, 14, 96)   384         block_10_project[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "block_11_expand (Conv2D)        (None, 14, 14, 576)  55296       block_10_project_BN[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_11_expand_BN (BatchNormal (None, 14, 14, 576)  2304        block_11_expand[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_11_expand_relu (ReLU)     (None, 14, 14, 576)  0           block_11_expand_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_11_depthwise (DepthwiseCo (None, 14, 14, 576)  5184        block_11_expand_relu[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_11_depthwise_BN (BatchNor (None, 14, 14, 576)  2304        block_11_depthwise[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_11_depthwise_relu (ReLU)  (None, 14, 14, 576)  0           block_11_depthwise_BN[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "block_11_project (Conv2D)       (None, 14, 14, 96)   55296       block_11_depthwise_relu[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "block_11_project_BN (BatchNorma (None, 14, 14, 96)   384         block_11_project[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "block_11_add (Add)              (None, 14, 14, 96)   0           block_10_project_BN[0][0]        \n",
      "                                                                 block_11_project_BN[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_12_expand (Conv2D)        (None, 14, 14, 576)  55296       block_11_add[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "block_12_expand_BN (BatchNormal (None, 14, 14, 576)  2304        block_12_expand[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_12_expand_relu (ReLU)     (None, 14, 14, 576)  0           block_12_expand_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_12_depthwise (DepthwiseCo (None, 14, 14, 576)  5184        block_12_expand_relu[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_12_depthwise_BN (BatchNor (None, 14, 14, 576)  2304        block_12_depthwise[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_12_depthwise_relu (ReLU)  (None, 14, 14, 576)  0           block_12_depthwise_BN[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "block_12_project (Conv2D)       (None, 14, 14, 96)   55296       block_12_depthwise_relu[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "block_12_project_BN (BatchNorma (None, 14, 14, 96)   384         block_12_project[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "block_12_add (Add)              (None, 14, 14, 96)   0           block_11_add[0][0]               \n",
      "                                                                 block_12_project_BN[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_13_expand (Conv2D)        (None, 14, 14, 576)  55296       block_12_add[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "block_13_expand_BN (BatchNormal (None, 14, 14, 576)  2304        block_13_expand[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_13_expand_relu (ReLU)     (None, 14, 14, 576)  0           block_13_expand_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_13_pad (ZeroPadding2D)    (None, 15, 15, 576)  0           block_13_expand_relu[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_13_depthwise (DepthwiseCo (None, 7, 7, 576)    5184        block_13_pad[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "block_13_depthwise_BN (BatchNor (None, 7, 7, 576)    2304        block_13_depthwise[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_13_depthwise_relu (ReLU)  (None, 7, 7, 576)    0           block_13_depthwise_BN[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "block_13_project (Conv2D)       (None, 7, 7, 160)    92160       block_13_depthwise_relu[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "block_13_project_BN (BatchNorma (None, 7, 7, 160)    640         block_13_project[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "block_14_expand (Conv2D)        (None, 7, 7, 960)    153600      block_13_project_BN[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_14_expand_BN (BatchNormal (None, 7, 7, 960)    3840        block_14_expand[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_14_expand_relu (ReLU)     (None, 7, 7, 960)    0           block_14_expand_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_14_depthwise (DepthwiseCo (None, 7, 7, 960)    8640        block_14_expand_relu[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_14_depthwise_BN (BatchNor (None, 7, 7, 960)    3840        block_14_depthwise[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_14_depthwise_relu (ReLU)  (None, 7, 7, 960)    0           block_14_depthwise_BN[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "block_14_project (Conv2D)       (None, 7, 7, 160)    153600      block_14_depthwise_relu[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "block_14_project_BN (BatchNorma (None, 7, 7, 160)    640         block_14_project[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "block_14_add (Add)              (None, 7, 7, 160)    0           block_13_project_BN[0][0]        \n",
      "                                                                 block_14_project_BN[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_15_expand (Conv2D)        (None, 7, 7, 960)    153600      block_14_add[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "block_15_expand_BN (BatchNormal (None, 7, 7, 960)    3840        block_15_expand[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_15_expand_relu (ReLU)     (None, 7, 7, 960)    0           block_15_expand_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_15_depthwise (DepthwiseCo (None, 7, 7, 960)    8640        block_15_expand_relu[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_15_depthwise_BN (BatchNor (None, 7, 7, 960)    3840        block_15_depthwise[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_15_depthwise_relu (ReLU)  (None, 7, 7, 960)    0           block_15_depthwise_BN[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "block_15_project (Conv2D)       (None, 7, 7, 160)    153600      block_15_depthwise_relu[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "block_15_project_BN (BatchNorma (None, 7, 7, 160)    640         block_15_project[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "block_15_add (Add)              (None, 7, 7, 160)    0           block_14_add[0][0]               \n",
      "                                                                 block_15_project_BN[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "block_16_expand (Conv2D)        (None, 7, 7, 960)    153600      block_15_add[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "block_16_expand_BN (BatchNormal (None, 7, 7, 960)    3840        block_16_expand[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "block_16_expand_relu (ReLU)     (None, 7, 7, 960)    0           block_16_expand_BN[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_16_depthwise (DepthwiseCo (None, 7, 7, 960)    8640        block_16_expand_relu[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "block_16_depthwise_BN (BatchNor (None, 7, 7, 960)    3840        block_16_depthwise[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "block_16_depthwise_relu (ReLU)  (None, 7, 7, 960)    0           block_16_depthwise_BN[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "block_16_project (Conv2D)       (None, 7, 7, 320)    307200      block_16_depthwise_relu[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "block_16_project_BN (BatchNorma (None, 7, 7, 320)    1280        block_16_project[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "Conv_1 (Conv2D)                 (None, 7, 7, 1280)   409600      block_16_project_BN[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "Conv_1_bn (BatchNormalization)  (None, 7, 7, 1280)   5120        Conv_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "out_relu (ReLU)                 (None, 7, 7, 1280)   0           Conv_1_bn[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 2,257,984\n",
      "Trainable params: 2,223,872\n",
      "Non-trainable params: 34,112\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "IMG_SHAPE = (224, 224, 3)\n",
    "\n",
    "# Create the base model from the pre-trained model MobileNet V2\n",
    "base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,\n",
    "                                               include_top=False,\n",
    "                                               weights='imagenet')\n",
    "\n",
    "base_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to get an idea of this pretrained model, we will pass in a batch of images through it, and examine our output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for image_batch, label_batch in train_batches.take(1):\n",
    "    feature_batch = base_model(image_batch)\n",
    "    print(feature_batch.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the input was (16, 128, 128, 3), with batch size as the first dimension. Thus, this feature extractor we have defined converts each 128x128x3 image to a 5x5x128 feature block. Since we only want to use the pretrained model as a feature extractor, we can go ahead and freeze the base. This means that we do not modify the weights of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_model.trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we add in the last part of the model, which we will train. First, we will add some average pooling to convert features to a single 1280-element vector per image. After we have that, we will add a single dense layer that outputs the classifications of the images. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "global_average_layer = tf.keras.layers.GlobalAveragePooling2D()\n",
    "feature_batch_average = global_average_layer(feature_batch)\n",
    "print(feature_batch_average.shape)\n",
    "\n",
    "\n",
    "prediction_layer = keras.layers.Dense(1)\n",
    "prediction_batch = prediction_layer(feature_batch_average)\n",
    "print(prediction_batch.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([\n",
    "  base_model,\n",
    "  global_average_layer,\n",
    "  prediction_layer\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_learning_rate = 0.0001\n",
    "model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=base_learning_rate),\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "num_train, num_val, num_test = (\n",
    "  metadata.splits['train'].num_examples*weight/10\n",
    "  for weight in SPLIT_WEIGHTS\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "initial_epochs = 10\n",
    "steps_per_epoch = round(num_train)//BATCH_SIZE\n",
    "validation_steps = 20\n",
    "\n",
    "loss0,accuracy0 = model.evaluate(validation_batches, steps = validation_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(train_batches,\n",
    "                    epochs=initial_epochs,\n",
    "                    validation_data=validation_batches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".tf_2.0",
   "language": "python",
   "name": ".tf_2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}