Feature-Visualization.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "FeatureVis_A.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZLykWOpxtAf",
        "colab_type": "text"
      },
      "source": [
        "# **Feature Visualization** using **DeepDream** and **Tensorflow**\n",
        "\n",
        "A notebook exploring how neural networks construct their understanding of images.\n",
        "\n",
        "Sources and Acknowledgements:\n",
        "\n",
        "\n",
        "*   Distill.Pub's blogpost: https://distill.pub/2017/feature-visualization/\n",
        "*   Alex Mordvintsev's Notebook: https://github.com/krantirk/DeepDream/blob/master/deepdream.py\n",
        "*   Official DeepDream Tutorial: https://www.tensorflow.org/tutorials/generative/deepdream\n",
        "*   Tensorflow's Lucid Library: https://github.com/tensorflow/lucid\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T64NUtqYfQwA",
        "colab_type": "text"
      },
      "source": [
        "# [Feedback Form](https://docs.google.com/forms/d/e/1FAIpQLSe_A7TibnNTqZu1GOH53f2ebSUvXmxmAjw2Avszex_UEWfcVQ/viewform?usp=sf_link)\n",
        "\n",
        "All the feedback is anonymous, and really helps me improve my events in the future. As a bonus, if you fill it out you get an extra surprise code cell! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Gpqe1Zz_02b",
        "colab_type": "text"
      },
      "source": [
        "# Neural Networks are not Interpretable\n",
        "\n",
        "That's a big problem. If we're talking about filtering out spam in emails, a mistake isn't the end of the world. But as the neural networks start being used for self driving cars and making medical diagnoseses, mistakes are life and death matters.\n",
        "\n",
        "Ideally, we should **always know** why a neural network made one decision as opposed to another."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LI8Xp4Iqtr-L",
        "colab_type": "text"
      },
      "source": [
        "# Visualizing Smaller Networks \n",
        "\n",
        "Smaller networks are easier to visulaize, and its a good strategy to try and thoroughly understand simple networks before you move on to larger networks if you're seriously interested in learning this stuff.\n",
        "\n",
        "*   [Neural Network Zoo](https://www.asimovinstitute.org/neural-network-zoo/)\n",
        "*   [3D Visualization of a Convolutional Neural Network](https://www.cs.ryerson.ca/~aharley/vis/conv/)\n",
        "*   [Tensorflow's Neural Network Playground](https://playground.tensorflow.org/)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "20o5n4lQ-rvT",
        "colab_type": "text"
      },
      "source": [
        "# Diving Deeper: GoogleNet\n",
        "\n",
        "Let's talk briefly about what [GoogleNet](https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/googlenet.html) is, and its [architecture](https://miro.medium.com/max/2588/1*ZFPOSAted10TPd3hBQU8iQ.png). \n",
        "\n",
        "GoogleNet was the prize winning architecture for tne ImageNet Large Scale Visual Recognition Challenge in 2014, with 22 layers and ~ 7 million parameters/weights. \n",
        "\n",
        "It's perfect for this demo because of its incredibly minimal computation requirements compared to other networks of this caliber.\n",
        "\n",
        "The kind of questions we're trying to answer are: \n",
        "\n",
        "\n",
        "*   What is each neuron contributing to the final prediction?\n",
        "*   Why is it doing that as opposed to something else?\n",
        "*   What structure allows these neurons collectively produce accurate predictions?\n",
        "\n",
        "\n",
        "\n",
        "![alt text](https://miro.medium.com/max/2588/1*ZFPOSAted10TPd3hBQU8iQ.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bTq4qBnBFj0h",
        "colab_type": "text"
      },
      "source": [
        "# Baby Steps\n",
        "\n",
        "Say we were tasked with finding out the answers to those questions.\n",
        "\n",
        "Where do we start? \n",
        "\n",
        "How do we start?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEHgCUiZHOHz",
        "colab_type": "text"
      },
      "source": [
        "# Back to Basics\n",
        "\n",
        "We know that neural networks are made up of nodes, and connections between nodes, where each node represents its confidence of a certain pattern or \"idea\" being present in our input.\n",
        "\n",
        "When we pass in an image through our network, each neuron's activation or lack thereof is represented by a number between 0 and 1.\n",
        "\n",
        "Maybe we can get an idea of what that neuron is looking for by looking at what images in our dataset activated it the most.\n",
        "\n",
        "Here are the images that resulted in the top activations for 4 randomly chosen neurons. \n",
        "\n",
        "Labels from Left to Right: (Each neuron is represented by an 3x3 grid)\n",
        "\n",
        "\"mixed4a: Unit 6\", \"mixed4a: Unit 240\", \"mixed4a: Unit 453\", \"mixed4a: Unit 492\"\n",
        "\n",
        "<details><summary>Dataset Results</summary>\n",
        "<p>\n",
        "\n",
        "<img src=https://distill.pub/2017/feature-visualization/images/why_optimization_examples.jpg width=\"15000\">\n",
        "\n",
        "\n",
        "</p>\n",
        "</details>\n",
        "\n",
        "\n",
        "And sure enough, there seems to be a strong connection between images that most activate a certain neuron.\n",
        "\n",
        "\n",
        "This can be generalized further than simply images to anything that can be represented as an image. For example, a spectrograph of a song is an image that sufficiently well captures the features of a song.\n",
        "\n",
        "For more details check out this [blog post](https://benanne.github.io/2014/08/05/spotify-cnns.html) by Sander Dieleman \n",
        "\n",
        "This, however, is still an incomplete picture. Is the neuron on the right looking for the sky? or a specific type of dome structure?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C41ymdNfQdMY",
        "colab_type": "text"
      },
      "source": [
        "# Diving Deeper: More Networks\n",
        "\n",
        "Based on techniques in this [paper](https://arxiv.org/abs/1311.2901) by Zeiler & Fergus, we can use yet another neural network to optimize for specific input images that lead to a certain activation in our network.\n",
        "\n",
        "This way, we pick a neuron, and starting with a completely random image, make small changes in each pixel(using gradient descent) so that they would raise the activation of the neuron.\n",
        "\n",
        "Essentially: We are generating an image from scratch to maximally excite a certain neuron. This way we don't need to rely on any input images to see what a certain neuron is looking for.\n",
        "\n",
        "Here's what that looks like: \n",
        "\n",
        "<details><summary>Noise to Feature</summary>\n",
        "<p>\n",
        "\n",
        "<img src=https://distill.pub/2017/feature-visualization/images/opt_progress_mixed4a-11.png width=\"15000\">\n",
        "\n",
        "\n",
        "</p>\n",
        "</details>\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Note: This is an incredibly complicated process. Depending on how you configure your second network, you have to make various tradeoffs between avoiding noise and other problems that arise when working on this type of problem. More details in this amazing [blog post](https://distill.pub/2017/feature-visualization/) by Distill.Pub\n",
        "\n",
        "<details><summary>Obligatory Meme</summary>\n",
        "<p>\n",
        "\n",
        "![alt text](https://i.imgflip.com/3o1x2x.jpg)\n",
        "\n",
        "</p>\n",
        "</details>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vL0EjhT0WTcT",
        "colab_type": "text"
      },
      "source": [
        "# So, what's everyone looking for?\n",
        "\n",
        "Here we can see the output when we optimize images to activate the neurons in our previous dataset examples\n",
        "\n",
        "<img src=https://distill.pub/2017/feature-visualization/images/why_optimization_examples.jpg width=\"15000\">\n",
        "\n",
        "Neuron Labels from left to right:\n",
        "\n",
        "\"mixed4a, Unit 6\", \"mixed4a, Unit 240\", \"mixed4a, Unit 453\", \"mixed4a, Unit 492\"\n",
        "\n",
        "\n",
        "<details><summary>Visualization Results</summary>\n",
        "<p>\n",
        "\n",
        "<img src=https://distill.pub/2017/feature-visualization/images/why_optimization_neuron.png width=\"15000\">\n",
        "\n",
        "\n",
        "</p>\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yLoERIx4l9Dl",
        "colab_type": "text"
      },
      "source": [
        "# What if we didn't start with noise?\n",
        "\n",
        "An immediate question that arises is \"What would the resulting image be if we didn't start with noise?\"\n",
        "\n",
        "We can find out by running the same process on images of our own."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4MZjqYgsru-A",
        "colab_type": "text"
      },
      "source": [
        "# Practical: Exaggerating Features in our own Image\n",
        "\n",
        "Let's get our hands dirty and write some code"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N91sju7LW-Qo",
        "colab_type": "text"
      },
      "source": [
        "Here are the setup and import statements "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bgKl2hVtpHLs",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "source": [
        "# Capture the output, otherwise we get annoying warnings and errors about upgrading to tensorflow 2\n",
        "%%capture\n",
        "\n",
        "!pip install --quiet lucid==0.0.5\n",
        "#!pip install --quiet --upgrade-strategy=only-if-needed git+https://github.com/tensorflow/lucid.git\n",
        "\n",
        "import warnings # Otherwise we get a bunch of \"dont't use tensorflow 1 when there is 2 warnings\"\n",
        "warnings.simplefilter(\"ignore\")\n",
        "import logging # \n",
        "logging.getLogger('tensorflow').disabled = True\n",
        "\n",
        "from io import BytesIO\n",
        "from IPython.display import clear_output, Image, display\n",
        "import PIL.Image\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import scipy.ndimage as nd\n",
        "import tensorflow as tf\n",
        "\n",
        "import lucid.modelzoo.vision_models as models\n",
        "from lucid.misc.io import show\n",
        "import lucid.optvis.objectives as objectives\n",
        "import lucid.optvis.param as param\n",
        "import lucid.optvis.render as render\n",
        "import lucid.optvis.transform as transform"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eOc1Tl1qXD2i",
        "colab_type": "text"
      },
      "source": [
        "This code fetches our images and extracts our pre trained inception algorithm and GoogleNet graph into memory "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p7k_rmPmwozK",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "source": [
        "# Capture the output, otherwise we get annoying warnings and errors about upgrading to tensorflow 2\n",
        "%%capture\n",
        "\n",
        "# Get our algorithm \n",
        "!wget -nc --no-check-certificate https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip && unzip -n inception5h.zip\n",
        "# Get our image\n",
        "!wget -nc https://i.imgur.com/AoPDzOT.jpg # Gogh Higher Res \n",
        "!wget -nc https://i.imgur.com/gPvEBRn.jpg # Gogh Lower Res\n",
        "!wget -nc https://i.imgur.com/HgmXhmx.jpg # Mona Lisa Lower Res\n",
        "\n",
        "# Open the image\n",
        "with open(\"gPvEBRn.jpg\", 'rb') as f:\n",
        "  file_contents = f.read()\n",
        "\n",
        "with open(\"HgmXhmx.jpg\", 'rb') as f:\n",
        "  lisa_contents = f.read()\n",
        "\n",
        "# Creating a TensorFlow session and loading the model\n",
        "model_fn = 'tensorflow_inception_graph.pb'\n",
        "graph = tf.Graph()\n",
        "sess = tf.InteractiveSession(graph=graph)\n",
        "with tf.gfile.GFile(model_fn, 'rb') as f:\n",
        "    graph_def = tf.GraphDef()\n",
        "    graph_def.ParseFromString(f.read())\n",
        "t_input = tf.placeholder(np.float32, name='input') # define the input tensor\n",
        "imagenet_mean = 117.0\n",
        "t_preprocessed = tf.expand_dims(t_input-imagenet_mean, 0)\n",
        "tf.import_graph_def(graph_def, {'input':t_preprocessed})\n",
        "\n",
        "def T(layer):\n",
        "    '''This function can isolate layers of the GoogleNet network allowing us to \n",
        "     peak inside'''\n",
        "    return graph.get_tensor_by_name(\"import/%s:0\"%layer)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MMMca7oo3jg_",
        "colab_type": "text"
      },
      "source": [
        "Optional: Upload or link to your own image!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2hWzvWBIfZYk",
        "colab_type": "text"
      },
      "source": [
        "Use the two cells below to upload an image from your computer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hz1yFiQy3qez",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "user_image_flag = False # This way these cells won't run when you \"Run All\"\n",
        "if user_image_flag:\n",
        "  # Create file upload dialog\n",
        "  from google.colab import files\n",
        "  uploaded = files.upload()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "83SWuOdL3zbi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "if user_image_flag:\n",
        "  if type(uploaded) is not dict: uploaded = uploaded.files  ## Deal with filedit versions\n",
        "  file_contents = uploaded[uploaded.keys()[0]]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Iq3Iefxfi5g",
        "colab_type": "text"
      },
      "source": [
        "OR Use the cell below to link to an image "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w6FlKB70fUq7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "if user_image_flag:\n",
        "  !wget -nc https://link.to/image_name.jpg # Link to image\n",
        "  with open(\"image_name.jpg\", 'rb') as f:\n",
        "    file_contents = f.read()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XdkWHuF83-i4",
        "colab_type": "text"
      },
      "source": [
        "Display Source Image"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2X5WSWrN4F10",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def showarray(a, fmt='jpeg'):\n",
        "    a = np.uint8(np.clip(a, 0, 255))\n",
        "    f = BytesIO()\n",
        "    PIL.Image.fromarray(a).save(f, fmt)\n",
        "    display(Image(data=f.getvalue()))\n",
        "    \n",
        "img0 = sess.run(tf.image.decode_image(file_contents))\n",
        "lisa = sess.run(tf.image.decode_image(lisa_contents))\n",
        "showarray(img0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_PtDr804ZCC",
        "colab_type": "text"
      },
      "source": [
        "# Image Manipulation and Gradient Descent Code \n",
        "\n",
        "Now onto the code to actually manipulate our image, we will use gradient descent to exaggerate certain features in our image, we will also do some more complex manipulations\n",
        "\n",
        "<!-- Core Algorithm:\n",
        "\n",
        "\n",
        "*   Gradient Descent \n",
        "*   Rendering Image \n",
        "*   Helpers -->\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cyqbHX3K4uR-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Hyperparameters # \n",
        "\n",
        "octave_n = 4 # Number of times we scale the image, \n",
        "             # helps distribute the size of manipulations\n",
        "octave_scale = 1.4 # Ratio by which we scale each time \n",
        "\n",
        "iter_n = 10 # Number of times we update the weights or \"manipulate pixels\"\n",
        "strength = 200 # Effective learning rate\n",
        "\n",
        "# Helper function that uses TensorFlow to resize an image\n",
        "def resize(img, new_size):\n",
        "    return sess.run(tf.image.resize_bilinear(img[np.newaxis,:], new_size))[0]\n",
        "\n",
        "# Apply gradients to an image in a seires of tiles\n",
        "def calc_grad_tiled(img, t_grad, tile_size=256):\n",
        "    '''Random shifts are applied to the image to blur tile boundaries over\n",
        "    multiple iterations.\n",
        "    This helps reduce noise in our image'''\n",
        "    h, w = img.shape[:2]\n",
        "    sx, sy = np.random.randint(tile_size, size=2)\n",
        "    # We randomly roll the image in x and y to avoid seams between tiles.\n",
        "    img_shift = np.roll(np.roll(img, sx, 1), sy, 0)\n",
        "    grad = np.zeros_like(img)\n",
        "    for y in range(0, max(h-tile_size//2, tile_size),tile_size):\n",
        "        for x in range(0, max(w-tile_size//2, tile_size),tile_size):\n",
        "            sub = img_shift[y:y+tile_size,x:x+tile_size]\n",
        "            g = sess.run(t_grad, {t_input:sub})\n",
        "            grad[y:y+tile_size,x:x+tile_size] = g\n",
        "    imggrad = np.roll(np.roll(grad, -sx, 1), -sy, 0)\n",
        "    # Add the image gradient to the image and return the result\n",
        "    return img + imggrad*(strength * 0.01 / (np.abs(imggrad).mean()+1e-7))\n",
        "\n",
        "# Putting it all together\n",
        "# Applies deepdream at multiple scales\n",
        "def render_deepdream(t_obj, input_img, show_steps = True):\n",
        "    # Collapse the optimization objective to a single number (the loss)\n",
        "    t_score = tf.reduce_mean(t_obj)\n",
        "    # We need the gradient of the image with respect to the objective\n",
        "    t_grad = tf.gradients(t_score, t_input)[0]\n",
        "\n",
        "    # split the image into a number of octaves (laplacian pyramid)\n",
        "    img = input_img\n",
        "    octaves = []\n",
        "    for i in range(octave_n-1):\n",
        "        lo = resize(img, np.int32(np.float32(img.shape[:2])/octave_scale))\n",
        "        octaves.append(img-resize(lo, img.shape[:2]))\n",
        "        img = lo\n",
        "\n",
        "    # generate details octave by octave\n",
        "    for octave in range(octave_n):\n",
        "        if octave>0:\n",
        "            hi = octaves[-octave]\n",
        "            img = resize(img, hi.shape[:2])+hi\n",
        "        for i in range(iter_n):\n",
        "            img = calc_grad_tiled(img, t_grad)\n",
        "        if show_steps:\n",
        "            clear_output()\n",
        "            showarray(img)\n",
        "    return img"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BCOGUkiBY9SQ",
        "colab_type": "text"
      },
      "source": [
        "Let's try this on our image!\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SL4EruKianld",
        "colab_type": "text"
      },
      "source": [
        "# mixed4a, Unit 6: The baseball spiral"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "o2utIN2-caXe",
        "colab": {}
      },
      "source": [
        "feature_channel = 249 #@param {type:\"slider\", max: 512}\n",
        "layer = \"mixed4a\"  #@param [\"mixed4d_3x3_bottleneck_pre_relu\", \"mixed3a\", \"mixed3b\", \"mixed4a\", \"mixed4c\", \"mixed5a\"]\n",
        "if feature_channel >= T(layer).shape[3]:\n",
        "  print(\"Feature channel exceeds size of layer \", layer, \" feature space. \")\n",
        "  print(\"Choose a smaller channel number.\")\n",
        "else:\n",
        "  render_deepdream(T(layer)[:,:,:,feature_channel], img0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Ul1yrRjcgZ6",
        "colab_type": "text"
      },
      "source": [
        "# mixed4a, Unit 453: Clouds—or fluffiness?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YEra5joScn-3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "feature_channel = 304 #@param {type:\"slider\", max: 512}\n",
        "layer = \"mixed4a\"  #@param [\"mixed4d_3x3_bottleneck_pre_relu\", \"mixed3a\", \"mixed3b\", \"mixed4a\", \"mixed4c\", \"mixed5a\"]\n",
        "strength = 100 #@param {type:\"slider\", max: 1000}\n",
        "\n",
        "if feature_channel >= T(layer).shape[3]:\n",
        "  print(\"Feature channel exceeds size of layer \", layer, \" feature space. \")\n",
        "  print(\"Choose a smaller channel number.\")\n",
        "else:\n",
        "  render_deepdream(T(layer)[:,:,:,feature_channel], img0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "alx9KUbydX0j",
        "colab_type": "text"
      },
      "source": [
        "# mixed4a, Unit 492: Buildings or Sky?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_KFT8-4GdjaB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "feature_channel = 294 #@param {type:\"slider\", max: 512}\n",
        "layer = \"mixed4a\"  #@param [\"mixed4d_3x3_bottleneck_pre_relu\", \"mixed3a\", \"mixed3b\", \"mixed4a\", \"mixed4c\", \"mixed5a\"]\n",
        "if feature_channel >= T(layer).shape[3]:\n",
        "  print(\"Feature channel exceeds size of layer \", layer, \" feature space. \")\n",
        "  print(\"Choose a smaller channel number.\")\n",
        "else:\n",
        "  render_deepdream(T(layer)[:,:,:,feature_channel], img0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kQ651VRMm3Mc",
        "colab_type": "text"
      },
      "source": [
        "# Back to Theory: Even this is incomplete\n",
        "\n",
        "Usually We can find many different images that optimally excite a certain neuron.\n",
        "\n",
        "Examples: \n",
        "\n",
        "mixed4a, 97\n",
        "\n",
        "![alt text](https://distill.pub/2017/feature-visualization/images/diversity/mixed4a_97_diversity.png)\n",
        "\n",
        "mixed4a, 143\n",
        "\n",
        "![alt text](https://distill.pub/2017/feature-visualization/images/diversity/mixed4a_143_diversity.png)\n",
        "\n",
        "A lot more research is needed on this topic. We are barely scratching the surface of the potential that neural networks have, and understanding them is crucial to being able to harness that potential in a safe way.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qg_o_WCIpWOM",
        "colab_type": "text"
      },
      "source": [
        "# Just one?\n",
        "\n",
        "We can also gereralize our process to appease more than one neuron at a time.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RUvf_bd5ppog",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Capture the output, otherwise we get annoying warnings and errors about upgrading to tensorflow 2\n",
        "%%capture\n",
        "# Let's import a model from the Lucid modelzoo!\n",
        "model = models.InceptionV1()\n",
        "model.load_graphdef()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IWRjQOcwp_Zu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def interpolate_param_f():\n",
        "  unique = param.fft_image((6, 128, 128, 3))\n",
        "  shared = [\n",
        "    param.lowres_tensor((6, 128, 128, 3), (1, 128//2, 128//2, 3)),\n",
        "    param.lowres_tensor((6, 128, 128, 3), (1, 128//4, 128//4, 3)),\n",
        "    param.lowres_tensor((6, 128, 128, 3), (1, 128//8, 128//8, 3)),\n",
        "    param.lowres_tensor((6, 128, 128, 3), (2, 128//8, 128//8, 3)),\n",
        "    param.lowres_tensor((6, 128, 128, 3), (1, 128//16, 128//16, 3)),\n",
        "    param.lowres_tensor((6, 128, 128, 3), (2, 128//16, 128//16, 3)),\n",
        "  ]\n",
        "  return param.to_valid_rgb(unique + sum(shared), decorrelate=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZDMZ9TU5U2Zm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "obj = objectives.channel_interpolate(\"mixed4a_pre_relu\", 97, \"mixed4a_pre_relu\", 143)\n",
        "_ = render.render_vis(model, obj, interpolate_param_f)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8P5BhYXRU3_y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "obj = objectives.channel_interpolate(\"mixed4a_pre_relu\", 476, \"mixed4a_pre_relu\", 460)\n",
        "_ = render.render_vis(model, obj, interpolate_param_f)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v0q8zJlYVKiy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "obj = objectives.channel_interpolate(\"mixed4a_pre_relu\", 6, \"mixed4a_pre_relu\", 453)\n",
        "_ = render.render_vis(model, obj, interpolate_param_f)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B41KxxxLv8d5",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title\n",
        "# You can use `Clear Output` if the animation gets annoying.\n",
        "%%html\n",
        "<style> \n",
        "  #animation {\n",
        "    width: 128px;\n",
        "    height: 128px;\n",
        "    background: url('https://storage.googleapis.com/tensorflow-lucid/static/img/notebook-interpolation-example-run-4.png') left center;\n",
        "    animation: play 1.5s steps(6) infinite alternate;\n",
        "  }\n",
        "  @keyframes play {\n",
        "    100% { background-position: -768px; }\n",
        "  }\n",
        "</style><div id='animation'></div>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXN00iMb1waE",
        "colab_type": "text"
      },
      "source": [
        "This interpolation process is also fairly complicated. There are a bunch of alignment issues you run into. More details [here](https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/differentiable-parameterizations/aligned_interpolation.ipynb).\n",
        "\n",
        "The question we want to think about here is how the \"idea\" or \"pattern\" changes when we're considering more than one neuron.\n",
        "Biologically speaking, most ideas are represented by groups of neurons working together. \n",
        "\n",
        "So instead of trying to interpret single neurons, we should also attempt to interpret groups of neuron activations as a whole.\n",
        "Current research shows that random group activations are often interpretable, but at a lower rate than sigular neurons. Keep in mind that the amount of possible group activations are several orders of magnitude larger than single activations.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wGxUSibT86_P",
        "colab_type": "text"
      },
      "source": [
        "# Let's try to exaggerate multiple features in our image\n",
        "\n",
        "In fact, why not do a whole layer?\n",
        "\n",
        "Lets pick one of the layers in GoogleNet and have our model optimize for activations in that layer.\n",
        "\n",
        "We can start with the the most basic layer 'mixed3a'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vs0HklgO85zN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "octave_n = 4 #@param {type:\"slider\", max: 10}\n",
        "octave_scale = 1.4 #@param {type:\"number\"}\n",
        "iter_n = 25 #@param {type:\"slider\", max: 50}\n",
        "strength = 200 #@param {type:\"slider\", max: 1000}\n",
        "layer = \"mixed3a\"  #@param [\"mixed3a\", \"mixed3b\", \"mixed4a\", \"mixed4c\", \"mixed5a\", \"mixed5b\"]\n",
        "\n",
        "final = render_deepdream(tf.square(T(layer)), img0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kvo2CMzqyWUg",
        "colab_type": "text"
      },
      "source": [
        "How about a more complex layer?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J4E-5QnZyawj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "octave_n = 4 #@param {type:\"slider\", max: 10}\n",
        "octave_scale = 1.4 #@param {type:\"number\"}\n",
        "iter_n = 50 #@param {type:\"slider\", max: 50}\n",
        "strength = 150 #@param {type:\"slider\", max: 1000}\n",
        "layer = \"mixed4c\"  #@param [\"mixed3a\", \"mixed3b\", \"mixed4a\", \"mixed4c\", \"mixed5a\", \"mixed5b\"]\n",
        "\n",
        "final = render_deepdream(tf.square(T(layer)), img0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SCysHCwkgDt9",
        "colab_type": "text"
      },
      "source": [
        "# What type of structure can we notice, if any?\n",
        "\n",
        "It turns out that our neurons early on in the network (closer to the input) pick up the most basic features in the image. Lines, edges, angles, colours, that type of thing.\n",
        "\n",
        "As we progress further, the features get more complex. You start to have shapes, some complicated pattterns.\n",
        "\n",
        "And eventually the neurons/layers close to the output are starting to capture more and more complex patterns and ideas, such as whole objects, animals, etc.\n",
        "\n",
        "Column labels from Left to Right: \n",
        "\n",
        "Edges (layer conv2d0), Textures (layer mixed3a), Patterns (layer mixed4a), Parts (layers mixed4b & mixed4c), Objects (layers mixed4d & mixed4e)\n",
        "\n",
        "<details><summary>Final Structure</summary>\n",
        "<p>\n",
        "\n",
        "![alt text](https://distill.pub/2017/feature-visualization/images/sprite_hero.png)\n",
        "\n",
        "\n",
        "\n",
        "</p>\n",
        "</details>\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pBEsqkRUnTce",
        "colab_type": "text"
      },
      "source": [
        "# Style Transfer\n",
        "\n",
        "Instead of individually picking the neurons we want to optimize for, can we instead optimize for another image? Yes!\n",
        "\n",
        "As usual, this is fairly complicated it. There are many different models for style transfer, and most of them are too computationally intensive to run on google collab, we will be using the old and basic TF-Hub and even then we will have to scale down our images.\n",
        "\n",
        "There's different notions of \"style\" you can prioritize when you train a network for style transfer. How much focus do you place on colour? Texture? Geometric Composition? \n",
        "\n",
        "You can learn more about style transfer [here](https://github.com/MacgyverCode/Style-Transfer-Colab)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EF0y3q-Bv2f_",
        "colab_type": "text"
      },
      "source": [
        "We need a different version of tensorflow to use the Tensorflow Hub module, so we should also restart our runtime before proceeding "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AdxIgwCjv_mT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "import PIL.Image\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "acibr6DRsNYQ",
        "colab_type": "text"
      },
      "source": [
        "We need a few helper functions here to make our lives easier. \n",
        "\n",
        "*  We need to limit the maximum dimension of our images to 512 pixels, otherwise we run out of resources. We can do this by scaling the images as we load them.\n",
        "*  We want to display the two images at once to see the styles being transfered.\n",
        "*  We want to display the returned tensor as an image.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wrWFPPbIshhz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def load_img(path_to_img):\n",
        "  max_dim = 512\n",
        "  img = tf.io.read_file(path_to_img)\n",
        "  img = tf.image.decode_image(img, channels=3)\n",
        "  img = tf.image.convert_image_dtype(img, tf.float32)\n",
        "\n",
        "  shape = tf.cast(tf.shape(img)[:-1], tf.float32)\n",
        "  long_dim = max(shape)\n",
        "  scale = max_dim / long_dim\n",
        "\n",
        "  new_shape = tf.cast(shape * scale, tf.int32)\n",
        "\n",
        "  img = tf.image.resize(img, new_shape)\n",
        "  img = img[tf.newaxis, :]\n",
        "  return img\n",
        "\n",
        "def imshow(image, title=None):\n",
        "  if len(image.shape) > 3:\n",
        "    image = tf.squeeze(image, axis=0)\n",
        "\n",
        "  plt.imshow(image)\n",
        "  if title:\n",
        "    plt.title(title)\n",
        "    \n",
        "def display_compare(content_image, style_image):\n",
        "  plt.subplot(1, 2, 1)\n",
        "  imshow(content_image, 'Content Image')\n",
        "\n",
        "  plt.subplot(1, 2, 2)\n",
        "  imshow(style_image, 'Style Image')\n",
        "\n",
        "def tensor_to_image(tensor):\n",
        "  tensor = tensor*255\n",
        "  tensor = np.array(tensor, dtype=np.uint8)\n",
        "  if np.ndim(tensor)>3:\n",
        "    assert tensor.shape[0] == 1\n",
        "    tensor = tensor[0]\n",
        "  return PIL.Image.fromarray(tensor)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nZVmCTOVq-6P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow_hub as hub\n",
        "\n",
        "content_path = tf.keras.utils.get_file('Gogh.jpg', 'https://i.imgur.com/gPvEBRn.jpg')\n",
        "style_path = tf.keras.utils.get_file('Waves.jpg','https://i.imgur.com/TDSIlPY.jpg')\n",
        "\n",
        "content_image = load_img(content_path)\n",
        "style_image = load_img(style_path)\n",
        "\n",
        "# content_image = load_img(style_path)\n",
        "# style_image = load_img(content_path)\n",
        "\n",
        "display_compare(content_image, style_image)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_AYGljAFxTfX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "hub_module = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/1')\n",
        "stylized_image = hub_module(tf.constant(content_image), tf.constant(style_image))[0]\n",
        "tensor_to_image(stylized_image)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eKGLzj4Ni4O_",
        "colab_type": "text"
      },
      "source": [
        "# Finally: Keep On Dreaming\n",
        "\n",
        "One thing that makes DeepDream so fascinating is that there are virtually no limits to what you can do with it. Just put everything in a loop and let it run.\n",
        "It will keep modifying the image in new and interesting ways, ad infiniteum. The result is kind of like Alice falling through the rabbit hole, discovering a strange new world along the way."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4ZICkwSKjOaw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "layer = \"mixed4c\"  #@param [\"mixed4d_3x3_bottleneck_pre_relu\", \"mixed3a\", \"mixed3b\", \"mixed4a\", \"mixed4c\", \"mixed5a\"]\n",
        "iter_n = 20 #@param {type:\"slider\", max: 50}\n",
        "strength = 200 #@param {type:\"slider\", max: 1000}\n",
        "zooming_steps = 20 #@param {type:\"slider\", max: 512}\n",
        "zoom_factor = 1.4 #@param {type:\"number\"}\n",
        " \n",
        "frame = img0\n",
        " \n",
        "# newsize = np.int32(np.float32(frame.shape[:2])*zoom_factor)\n",
        "# frame = resize(frame, newsize)\n",
        " \n",
        "img_y, img_x, _ = img0.shape\n",
        "for i in range(zooming_steps):\n",
        "  frame = render_deepdream(tf.square(T(layer)), frame, False)\n",
        "  # clear_output()\n",
        "  showarray(frame)\n",
        "  newsize = np.int32(np.float32(frame.shape[:2])*zoom_factor)\n",
        "  frame = resize(frame, newsize)\n",
        "  frame = frame[(newsize[0]-img_y)//2:(newsize[0]-img_y)//2+img_y,\n",
        "                (newsize[1]-img_x)//2:(newsize[1]-img_x)//2+img_x,:] "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GrFEnFbzmgaz"
      },
      "source": [
        "# [Feedback Form](https://docs.google.com/forms/d/e/1FAIpQLSe_A7TibnNTqZu1GOH53f2ebSUvXmxmAjw2Avszex_UEWfcVQ/viewform?usp=sf_link)\n",
        "\n",
        "All the feedback is anonymous, and really helps me improve my events in the future. As a bonus, if you fill it out you get an extra surprise code cell! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ATe0uguS0RF2",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D7JuEFDt0Slb",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    }
  ]
}