04 Neural Networks and PyTorch Design.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 01Build PyTorch CNN - Object Oriented Neural Networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Lizard: #class declaration\n",
    "    def __init__(self, name): #class constructor (code)\n",
    "        self.name = name #attribute (data)\n",
    "\n",
    "    def set_name(self, name): #method declaration (code)\n",
    "        self.name = name #method implementation (code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first line declares the class and specifies the class name, which in this case is Lizard.\n",
    "\n",
    "The second line defines a special method called the class constructor. Class constructors are called when a new instance of the class is created. As parameters, we have self and name.\n",
    "\n",
    "The self parameter gives us the ability to create attribute values that are stored or encapsulated within the object. When we call this constructor or any of the other methods, we don't pass the self parameter. Python does this for us automatically.\n",
    "\n",
    "Argument values for any other parameter are arbitrarily passed by the caller, and these passed values that come in to the method can be used in a calculation or saved and accessed later using self.\n",
    "\n",
    "After we're done with the constructor, we can create any number of specialized methods like this one here that allows a caller to change the name value that was stored in self. All we have to do here is call the method and pass a new value for the name. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building A Neural Network In PyTorch\n",
    "We now have enough information to provide an outline for building neural networks in PyTorch. The steps are as follows:\n",
    "\n",
    "Short version:\n",
    "\n",
    "1.Extend the nn.Module base class.\n",
    "2.Define layers as class attributes.\n",
    "3.Implement the forward() method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#let’s create a simple class to represent a neural network.\n",
    "class Network:\n",
    "    def __init__(self):\n",
    "        self.layer = None\n",
    "    \n",
    "    def forward(self,t):\n",
    "        t = self.layer(t)\n",
    "        return t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a simple network class that has a single dummy layer inside the constructor and a dummy implementation for the forward function.\n",
    "\n",
    "The implementation for the <code>forward()</code> function takes in a tensor <code>t</code> and transforms it using the dummy layer. After the tensor is transformed, the new tensor is returned.\n",
    "\n",
    "This is a good start, but the class hasn’t yet extended the <code>nn.Module</code> class. To make our <code>Network</code> class extend <code>nn.Module</code>, we must do two additional things:\n",
    "\n",
    "1.Specify the <code>nn.Module</code> class in parentheses on line 1.<br/>\n",
    "2.Insert a call to the super class constructor on line 3 inside the constructor.\n",
    "This gives us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.layer = None\n",
    "    def forward(self,t):\n",
    "        t = self.layer(t)\n",
    "        return t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These changes transform our simple neural network into a PyTorch neural network because we are now extending PyTorch's <code>nn.Module</code> base class.\n",
    "\n",
    "With this, we are done! Now we have a <code>Network</code> class that has all of the functionality of the PyTorch <code>nn.Module</code> class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define The Network’s Layers As Class Attributes\n",
    "At the moment, our Network class has a single dummy layer as an attribute. Let’s replace this now with some real layers that come pre-built for us from PyTorch's <code>nn</code> library. We’re building a CNN, so the two types of layers we'll use are linear layers and convolutional layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(in_channels=1,out_channels=6,kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(in_channels=6,out_channels=12,kernel_size = 5)\n",
    "        \n",
    "        self.fc1 = nn.Linear(in_features = 12*4*4,out_features = 120)\n",
    "        self.fc2 = nn.Linear(in_features = 120,out_features = 60)\n",
    "        self.out = nn.Linear(in_features = 60,out_features = 10)\n",
    "        \n",
    "    def forward(self,t):\n",
    "        # implement the forward pass\n",
    "        return t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Network(\n",
       "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
       "  (conv2): Conv2d(6, 12, kernel_size=(5, 5), stride=(1, 1))\n",
       "  (fc1): Linear(in_features=192, out_features=120, bias=True)\n",
       "  (fc2): Linear(in_features=120, out_features=60, bias=True)\n",
       "  (out): Linear(in_features=60, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network =Network()\n",
    "network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz 01\n",
    "Q1:When building a basic convolutional neural network, the goal is to model or approximate a function that maps image inputs to the correct ____________<br/>\n",
    "A1:output class\n",
    "\n",
    "Q2:The words _______________ and network mean the same thing.<br/>\n",
    "A2:model\n",
    "\n",
    "Q3:Neural networks and layers in PyTorch extend the _______________ class. This means that we must extend the _______________ class when building a new layer or neural network in PyTorch.<br/>\n",
    "A3:nn.Module\n",
    "\n",
    "Q4:When we pass a tensor to our network as input, the tensor flows _______________ though each layer transformation until the tensor reaches the output layer. This process of a tensor flowing _______________ though the network is known as a forward pass.<br/>\n",
    "A4:forward\n",
    "\n",
    "Q5:In the code below, we used the abbreviation fc in fc1 and fc2 because linear layers are also called fully connected layers. Linear layers also have a third name that is sometimes used. This name is _______________ layers.\n",
    "\n",
    "<code>class Network(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)\n",
    "        \n",
    "        self.fc1 = nn.Linear(in_features=12 * 4 * 4, out_features=120)\n",
    "        self.fc2 = nn.Linear(in_features=120, out_features=60)\n",
    "        self.out = nn.Linear(in_features=60, out_features=10)\n",
    "        \n",
    "    def forward(self, t):\n",
    "        # implement the forward pass\n",
    "        return t</code>\n",
    "A5:dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 02 CNN Layers - Deep Neural Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz02\n",
    "Q1:Parameters are used in function definitions as place-holders while _______________ are the actual values that are passed to the function.<br/>\n",
    "A1:arguments\n",
    "\n",
    "Q2:In general, _______________ are parameters whose values are chosen manually and arbitrarily.<br/>\n",
    "A2:hyperparameters\n",
    "\n",
    "Q3:As neural network programmers, we choose hyperparameter values mainly based on _______________ and increasingly by utilizing values that have proven to work well in the past.<br/>\n",
    "A3:trial and error\n",
    "\n",
    "Q4:In neural network programming, the words kernel and _______________ are interchangeable.<br/>\n",
    "A4:filter\n",
    "\n",
    "Q5:With a basic CNN, the <code>out_channels</code> parameter of a convolutional layer sets the number of ____________<br/>\n",
    "A5:filters\n",
    "\n",
    "Q6:Data dependent hyperparameters are parameters whose values are dependent on data. Two data dependent hyperparameters that stick out with basic CNNs are the <code>in_channels</code> of the first convolutional layer, and the <code>out_features</code> of the ____________<br/>\n",
    "A6:output layer\n",
    "    \n",
    "Q7:In a basic CNN, the <code>in_channels</code> of the first convolutional layer depend on the number of ____________<br/>\n",
    "A7:images in the training set\n",
    "    \n",
    "Q8:In a basic CNN, the <code>out_features</code> coming from the output layer depend on the number of<br/>\n",
    "A8:classes in the training set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 03 CNN Weights - Learnable Parameters in Neural Networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)\n",
    "\n",
    "        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)\n",
    "        self.fc2 = nn.Linear(in_features=120, out_features=60)\n",
    "        self.out = nn.Linear(in_features=60, out_features=10)\n",
    "\n",
    "    def forward(self, t):\n",
    "        # implement the forward pass\n",
    "        return t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learnable Parameters\n",
    "Learnable parameters are parameters whose values are learned during the training process.\n",
    "\n",
    "With learnable parameters, we typically start out with a set of arbitrary values, and these values then get updated in an iterative fashion as the network learns.\n",
    "\n",
    "In fact, when we say that a network is learning, we specifically mean that the network is learning the appropriate values for the learnable parameters. Appropriate values are values that minimize the loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "network = Network()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 12, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=192, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=60, bias=True)\n",
      "  (out): Linear(in_features=60, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <code>print()</code> function prints to the console a string representation of our network. With a sharp eye, we can notice that the printed output here is detailing our network’s architecture listing out our network’s layers, and showing the values that were passed to the layer constructors.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network():\n",
    "    def __init__(self):\n",
    "        #super().__init__()\n",
    "        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)\n",
    "\n",
    "        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)\n",
    "        self.fc2 = nn.Linear(in_features=120, out_features=60)\n",
    "        self.out = nn.Linear(in_features=60, out_features=10)\n",
    "\n",
    "    def forward(self, t):\n",
    "        # implement the forward pass\n",
    "        return t\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All Python classes automatically extend the object class. If we want to provide a custom string representation for our object, we can do it, but we need to introduce another object oriented concept called overriding.\n",
    "\n",
    "When we extend a class, we get all of its functionality, and to complement this, we can add additional functionality. However, we can also override existing functionality by changing it to behave differently.\n",
    "\n",
    "We can override Python’s default string representation using the <code> __repr__  </code>function. This name is short for representation.\n",
    "\n",
    "同理 pytorch中的 __repr__  是定义在module.py中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<__main__.Network object at 0x000001DA2B4B65B0>\n"
     ]
    }
   ],
   "source": [
    "network = Network()\n",
    "print(network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network():\n",
    "    def __init__(self):\n",
    "        #super().__init__()\n",
    "        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)\n",
    "\n",
    "        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)\n",
    "        self.fc2 = nn.Linear(in_features=120, out_features=60)\n",
    "        self.out = nn.Linear(in_features=60, out_features=10)\n",
    "\n",
    "    def forward(self, t):\n",
    "        # implement the forward pass\n",
    "        return t\n",
    "    def __repr__(self):\n",
    "        return \"lizardnet\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lizardnet\n"
     ]
    }
   ],
   "source": [
    "network = Network()\n",
    "print(network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 12, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=192, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=60, bias=True)\n",
      "  (out): Linear(in_features=60, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "class Network(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)\n",
    "\n",
    "        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)\n",
    "        self.fc2 = nn.Linear(in_features=120, out_features=60)\n",
    "        self.out = nn.Linear(in_features=60, out_features=10)\n",
    "\n",
    "    def forward(self, t):\n",
    "        # implement the forward pass\n",
    "        return t\n",
    "    \n",
    "network = Network()\n",
    "print(network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing The Network's Layers\n",
    "In Python and many other programming languages, we access attributes and methods of objects using dot notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.conv1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Conv2d(6, 12, kernel_size=(5, 5), stride=(1, 1))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.conv2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=192, out_features=120, bias=True)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.fc1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=120, out_features=60, bias=True)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.fc2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=60, out_features=10, bias=True)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Network(\n",
       "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
       "  (conv2): Conv2d(6, 12, kernel_size=(5, 5), stride=(1, 1))\n",
       "  (fc1): Linear(in_features=192, out_features=120, bias=True)\n",
       "  (fc2): Linear(in_features=120, out_features=60, bias=True)\n",
       "  (out): Linear(in_features=60, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[[[ 0.0516,  0.0247,  0.0753,  0.1139,  0.0294],\n",
       "          [ 0.0286, -0.1143,  0.0187, -0.1602,  0.0889],\n",
       "          [-0.0862,  0.1164, -0.0797, -0.0881, -0.0659],\n",
       "          [-0.0644,  0.0318, -0.0891,  0.1878, -0.1816],\n",
       "          [-0.1455,  0.1939,  0.0116, -0.0983,  0.0895]]],\n",
       "\n",
       "\n",
       "        [[[ 0.0798, -0.0607,  0.1528,  0.0290,  0.1208],\n",
       "          [-0.0843,  0.0786,  0.1338, -0.0950,  0.0589],\n",
       "          [-0.0068,  0.1232,  0.1197, -0.0473, -0.0312],\n",
       "          [ 0.0098, -0.1048, -0.1724,  0.0827,  0.1873],\n",
       "          [-0.0016,  0.0097,  0.1008, -0.1015,  0.0546]]],\n",
       "\n",
       "\n",
       "        [[[ 0.0906,  0.1924,  0.1485, -0.1521,  0.1248],\n",
       "          [-0.1287,  0.1772, -0.1020,  0.1167, -0.0005],\n",
       "          [ 0.0478, -0.1350,  0.0539, -0.1415, -0.0640],\n",
       "          [ 0.1415, -0.1464,  0.1800,  0.0145,  0.1695],\n",
       "          [ 0.1828, -0.0337,  0.1776,  0.0602,  0.0546]]],\n",
       "\n",
       "\n",
       "        [[[ 0.0084,  0.1880,  0.0287,  0.0334,  0.0222],\n",
       "          [-0.1886, -0.0883, -0.1730,  0.1352,  0.1244],\n",
       "          [ 0.0792, -0.0206,  0.1903,  0.1101, -0.1232],\n",
       "          [-0.1648,  0.0147,  0.1054,  0.0958,  0.0176],\n",
       "          [ 0.1973,  0.1517,  0.1526,  0.0231,  0.1599]]],\n",
       "\n",
       "\n",
       "        [[[-0.1514,  0.1091,  0.1485, -0.0666, -0.0892],\n",
       "          [ 0.0814, -0.0659, -0.0778,  0.0326, -0.0998],\n",
       "          [-0.1646, -0.0007,  0.0068,  0.1060,  0.1884],\n",
       "          [-0.1901, -0.1512, -0.0391,  0.0842, -0.0493],\n",
       "          [ 0.1678,  0.0322, -0.0140, -0.1655,  0.1093]]],\n",
       "\n",
       "\n",
       "        [[[ 0.0819, -0.0524, -0.0804,  0.0218, -0.0266],\n",
       "          [ 0.0141, -0.1374, -0.0204, -0.1324, -0.1389],\n",
       "          [-0.0548,  0.0213,  0.0975, -0.0895,  0.1128],\n",
       "          [-0.0515, -0.0335, -0.0820,  0.1069,  0.1054],\n",
       "          [-0.1208,  0.1867, -0.1600, -0.0279, -0.1977]]]], requires_grad=True)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.conv1.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch Parameter Class\n",
    "To keep track of all the weight tensors inside the network. PyTorch has a special class called <code>Parameter. </code>The <code>Parameter</code> class extends the tensor class, and so the weight tensor inside every layer is an instance of this <code>Parameter</code> class. This is why we see the <code>Parameter containing</code> text at the top of the string representation output.\n",
    "\n",
    "We can see in the Pytorch source code that the <code>Parameter</code> class is overriding the <code>__repr__ </code>function by prepending the text parameter containing to the regular tensor class representation output.\n",
    "\n",
    "<code>\n",
    "def __repr__(self):\n",
    "    return 'Parameter containing:\\n' + super(Parameter, self).__repr__()\n",
    "</code>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weight Tensor Shape\n",
    "Remember, the shape of a tensor really encodes all the information we need to know about the tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.conv1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 1, 5, 5])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.conv1.weight.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape of the weight tensor for the first convolutional layer shows us that we have a rank-4 weight tensor. The first axis has a length of 6, and this accounts for the 6 filters.\n",
    "\n",
    "The second axis has a length of 1 which accounts for the single input channel, and the last two axes account for the height and width of the filter.\n",
    "\n",
    "The way to think about this is as if we are packaging all of our filters into a single tensor.\n",
    "\n",
    "Now, the second conv layer has 12 filters, and instead of convolving a single input channel, there are 6 input channels coming from the previous layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([12, 6, 5, 5])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.conv2.weight.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Think of this value of 6 here as giving each of the filters some depth. Instead of having a filter that convolves all of the channels iteratively, our filter has a depth that matches the number of channels.\n",
    "\n",
    "The two main takeaways about these convolutional layers is that our filters are represented using a single tensor and that each filter inside the tensor also has a depth that accounts for the input channels that are being convolved.\n",
    "\n",
    "1.All filters are represented using a single tensor.<br/>\n",
    "2.Filters have depth that accounts for the input channels.<br/>\n",
    "\n",
    "Our tensors are rank-4 tensors. The first axis represents the number of filters. The second axis represents the depth of each filter which corresponds to the number of input channels being convolved.\n",
    "\n",
    "The last two axes represent the height and width of each filter. We can pull out any single filter by indexing into the weight tensor’s first axis.\n",
    "\n",
    "<code>(Number of filters, Depth, Height, Width)</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=192, out_features=120, bias=True)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.fc1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.0409, -0.0097,  0.0448,  ..., -0.0019,  0.0157,  0.0264],\n",
       "        [-0.0116, -0.0508,  0.0406,  ...,  0.0037,  0.0210, -0.0232],\n",
       "        [-0.0150,  0.0057, -0.0048,  ...,  0.0416,  0.0090,  0.0353],\n",
       "        ...,\n",
       "        [-0.0205,  0.0179,  0.0074,  ..., -0.0415,  0.0231, -0.0640],\n",
       "        [-0.0280, -0.0567, -0.0046,  ..., -0.0166,  0.0321,  0.0230],\n",
       "        [-0.0591, -0.0495, -0.0082,  ...,  0.0347, -0.0485,  0.0498]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.fc1.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([120, 192])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.fc1.weight.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([60, 120])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.fc2.weight.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 60])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.out.weight.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing The Networks Parameters\n",
    "\n",
    "The first example is the most common way, and we’ll use this to iterate over our weights when we update them during the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 1, 5, 5])\n",
      "torch.Size([6])\n",
      "torch.Size([12, 6, 5, 5])\n",
      "torch.Size([12])\n",
      "torch.Size([120, 192])\n",
      "torch.Size([120])\n",
      "torch.Size([60, 120])\n",
      "torch.Size([60])\n",
      "torch.Size([10, 60])\n",
      "torch.Size([10])\n"
     ]
    }
   ],
   "source": [
    "for param in network.parameters():\n",
    "    print(param.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second way is just to show how we can see the name as well. This reveals something that we won’t cover in detail, the bias is also a learnable parameter. Each layer has a bias by default, so for each layer we have a weight tensor and a bias tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv1.weight \t\t torch.Size([6, 1, 5, 5])\n",
      "conv1.bias \t\t torch.Size([6])\n",
      "conv2.weight \t\t torch.Size([12, 6, 5, 5])\n",
      "conv2.bias \t\t torch.Size([12])\n",
      "fc1.weight \t\t torch.Size([120, 192])\n",
      "fc1.bias \t\t torch.Size([120])\n",
      "fc2.weight \t\t torch.Size([60, 120])\n",
      "fc2.bias \t\t torch.Size([60])\n",
      "out.weight \t\t torch.Size([10, 60])\n",
      "out.bias \t\t torch.Size([10])\n"
     ]
    }
   ],
   "source": [
    "for name, param in network.named_parameters():\n",
    "    print(name, '\\t\\t', param.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz 03\n",
    "\n",
    "Q1:Hyperparameter values are chosen arbitrarily.<br/>\n",
    "A1:**True**\n",
    "\n",
    "Q2:Learnable parameters are parameters whose values are learned during the training process.<br/>\n",
    "A2:**True**\n",
    "\n",
    "Q3:With _________ we typically start out with a set of arbitrary values, and these values then get updated in an iterative fashion as the network learns.<br/>\n",
    "A3:**learnable parameters**\n",
    "\n",
    "Q4:Weights are learnable parameters that live inside each layer of the network.<br/>\n",
    "A4:**True**\n",
    "\n",
    "Q5:As a network is trained, the weight values inside the network are updated in such a way that the loss function output is minimized.<br/>\n",
    "A5:**True**\n",
    "\n",
    "Q6:For a convolutional layer, the weight values live inside the filters, and when implemented in code, the filters are actually represented using ____________<br/>\n",
    "A6:**a single tensor**\n",
    "\n",
    "Q7:In neural network programming, all convolutional filters for a given conv layer are represented using a single tensor.<br/>\n",
    "A7:**True**\n",
    "\n",
    "Q8:In neural network programming, the filter tensor, a.k.a. weight tensor, of a given conv layer has a depth that accounts for the input channels.<br/>\n",
    "A8:**True**\n",
    "\n",
    "Q9:Inside a convolutional layer, the depth dimension of the convolutional filter corresponds to the number of _______________ coming into the layer.<br/>\n",
    "A9:**input channels**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 04 Callable Neural Networks - Linear Layers in Depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_features = torch.tensor([1,2,3,4], dtype=torch.float32)\n",
    "\n",
    "weight_matrix = torch.tensor([\n",
    "    [1,2,3,4],\n",
    "    [2,3,4,5],\n",
    "    [3,4,5,6]\n",
    "], dtype=torch.float32)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([30., 40., 50.])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weight_matrix.matmul(in_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc = nn.Linear(in_features=4,out_features=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.4863, -2.2903,  0.7294], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fc(in_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can call the object instance like this because PyTorch neural network modules are <code>callable Python objects</code>. We'll look at this important detail more closely in a minute, but first, check out this output. We did indeed get a 1-dimensional tensor with three elements. However, different values were produced.\n",
    "\n",
    "This is because PyTorch creates a weight matrix and initializes it with random values. This means that the linear functions from the two examples are different, so we are using different function to produce these outputs.\n",
    "\n",
    "**Remember the values inside the weight matrix define the linear function.** This demonstrates how the network's mapping changes as the weights are updated during the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc.weight = nn.Parameter(weight_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([30.3760, 40.4606, 49.9037], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fc(in_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we are much closer to the <code>30, 40, and 50 </code>values. **However**, we're exact. Why is this? We'll, this is not exact because the linear layer is adding a <code>bias </code>tensor to the output. Watch what happens when we turn the bias off. We do this by passing a False flag to the constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc = nn.Linear(in_features=4, out_features=3, bias=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc.weight = nn.Parameter(weight_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([30., 40., 50.], grad_fn=<SqueezeBackward3>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fc(in_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What makes this possible is that PyTorch module classes implement another special Python function called <code>__call__()</code>. If a class implements the <code>__call__()</code> method, the special call method will be invoked anytime the object instance is called.\n",
    "\n",
    "This fact is an important PyTorch concept because of the way the <code>__call__() </code>method interacts with the <code>forward()</code> method for our layers and networks.\n",
    "\n",
    "Instead of calling the <code>forward()</code> method directly, we call the object instance. After the object instance is called, the <code>__call__()</code> method is invoked under the hood, and the <code> __call__() </code> in turn invokes the <code>forward()</code> method. This applies to all PyTorch neural network modules, namely, networks and layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>\n",
    "# torch/nn/modules/module.py (version 1.0.1)\n",
    "\n",
    "def __call__(self, *input, **kwargs):\n",
    "    for hook in self._forward_pre_hooks.values():\n",
    "        hook(self, input)\n",
    "    if torch._C._get_tracing_state():\n",
    "        result = self._slow_forward(*input, **kwargs)\n",
    "    else:\n",
    "        result = self.forward(*input, **kwargs)\n",
    "    for hook in self._forward_hooks.values():\n",
    "        hook_result = hook(self, input, result)\n",
    "        if hook_result is not None:\n",
    "            raise RuntimeError(\n",
    "                \"forward hooks should never return any values, but '{}'\"\n",
    "                \"didn't return None\".format(hook))\n",
    "    if len(self._backward_hooks) > 0:\n",
    "        var = result\n",
    "        while not isinstance(var, torch.Tensor):\n",
    "            if isinstance(var, dict):\n",
    "                var = next((v for v in var.values() if isinstance(v, torch.Tensor)))\n",
    "            else:\n",
    "                var = var[0]\n",
    "        grad_fn = var.grad_fn\n",
    "        if grad_fn is not None:\n",
    "            for hook in self._backward_hooks.values():\n",
    "</code>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The extra code that PyTorch runs inside the <code>__call__() </code>method is why we never invoke the <code>forward()</code> method directly. If we did, the additional PyTorch code would not be executed. As a result, any time we want to invoke our <code>forward()</code> method, we call the object instance. This applies to both layers, and networks because they are both PyTorch neural network modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz 04\n",
    "Q1:When input features are received by a linear layer, they are received in the form of a flattened 1-dimensional tensor and are then multiplied by______ <br/>\n",
    "A1:the weight matrix\n",
    "\n",
    "Q2:Suppose we have the code below. What will be the output of the following tensor operation?<br/>\n",
    "A2:tensor([30., 40., 50.])\n",
    "\n",
    "Q3:Linear layers map an <code>in_feature</code> space to an <code>out_feature</code> space using a weight matrix.<br/>\n",
    "A3:True\n",
    "\n",
    "Q4:he weight matrix inside a linear layer is initialized with random values.<br/>\n",
    "A4:True\n",
    "\n",
    "Q5:The linear layer operation can be expressed mathematically as $ y = Ax + b $ <br/>In this equation, which symbol represents the weight matrix?<br/>\n",
    "A5:A\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}