FFNetworkUsingInbuiltLibraries_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "FFNetworkUsingInbuiltLibraries_2.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "9UIJP9qdjnQf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rOIvo66ZjwUq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch.nn.functional as F"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JZMxYa4a02B3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch.nn as nn"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JGHcbkF61ND8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.datasets import make_blobs\n",
        "from sklearn.model_selection import train_test_split\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0tK5bqpw7AhA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GDiCq5jk7AtQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rq1DUUPZ6_QK",
        "colab_type": "text"
      },
      "source": [
        "Here we further simplify the code by using optim library and thus making the fit function very compact and concise"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GVWvzBQF6_G7",
        "colab_type": "text"
      },
      "source": [
        "The PyTorch module torch.optim provides many optimizers.\n",
        "\n",
        "An optimizer has an internal state to keep quantities such as moving averages, and operates on an iterator over Parameters.\n",
        "\n",
        "• Values specific to the optimizer can be specified to its constructor, and\n",
        "\n",
        "• its step method updates the internal state according to the grad attributes of the Parameters, and updates the latter according to the internal state.\n",
        "\n",
        "https://fleuret.org/ee559/ee559-slides-5-3-optim.pdf\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "niiLpbVH7kK_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y8gFoIvn7kgD",
        "colab_type": "text"
      },
      "source": [
        "<h3> Data Generation </h3>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aEifKfOA1R7l",
        "colab_type": "code",
        "outputId": "c1858129-a657-44d1-e140-9bcc84ebece5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data, labels = make_blobs(n_samples=1000, centers=4, n_features=2, random_state=0)\n",
        "print(data.shape, labels.shape)\n"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1000, 2) (1000,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "81ONTF3J_ggT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nwS0LSZQ_gRq",
        "colab_type": "code",
        "outputId": "d2f379df-7cfa-41e2-ec8e-104897b468e3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "X_train, X_val, Y_train, Y_val = train_test_split(data, labels, stratify=labels, random_state=0)\n",
        "print(X_train.shape, X_val.shape, labels.shape)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(750, 2) (250, 2) (1000,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E4q5QovJ_rPS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X_train, Y_train, X_val, Y_val = map(torch.tensor, (X_train, Y_train, X_val, Y_val))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sG4qdVhD_uBw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#https://stackoverflow.com/questions/56741087/how-to-fix-runtimeerror-expected-object-of-scalar-type-float-but-got-scalar-typ/56741419\n",
        "X_train = X_train.float()\n",
        "Y_train = Y_train.long()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bh4IPH10_yvZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dqD8Y7mV_yhv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xiGuBRsT7o-b",
        "colab_type": "text"
      },
      "source": [
        "<h3> Neural Network Class </h3>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TvNl7Mmr1Rt-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TwoLayerNetwork(nn.Module):\n",
        "  def __init__(self):\n",
        "    super().__init__()\n",
        "    torch.manual_seed(0)\n",
        "    self.net = nn.Sequential(\n",
        "        nn.Linear(2,2),\n",
        "        nn.Sigmoid(),\n",
        "        nn.Linear(2,4),\n",
        "        nn.Softmax()\n",
        "    )\n",
        "  \n",
        "  def forward(self,X):\n",
        "    return self.net(X)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8L24AD1m1YDp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pPFuC-Lh7uY2",
        "colab_type": "text"
      },
      "source": [
        "<h3> Using Optim </h3>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hOow_IOb-8Mx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from torch import optim"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4BjrmfzBAW10",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def accuracy(y_pred,y):\n",
        "  pred = torch.argmax(y_pred, dim=1)\n",
        "  return (pred == y).float().mean()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X0hGj831-8ko",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def fit_v1(epochs = 1000, learning_rate = 1):\n",
        "  loss_arr = []\n",
        "  acc_arr = []\n",
        "  opt = optim.SGD(fn.parameters(), lr=learning_rate)\n",
        "  \n",
        "  for epoch in range(epochs):\n",
        "    Y_pred = fn(X_train)\n",
        "    loss = F.cross_entropy(Y_pred, Y_train)\n",
        "    loss_arr.append(loss.item())\n",
        "    acc_arr.append(accuracy(Y_pred, Y_train))\n",
        "\n",
        "    loss.backward()\n",
        "    opt.step() # Update all the parameters\n",
        "    opt.zero_grad() # Make all gradients zero\n",
        "        \n",
        "  plt.plot(loss_arr, 'r-')\n",
        "  plt.plot(acc_arr, 'b-')\n",
        "  plt.show()      \n",
        "  print('Loss before training', loss_arr[0])\n",
        "  print('Loss after training', loss_arr[-1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wWJmmquQ_69L",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "fn = TwoLayerNetwork()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QJJUpnSyABtI",
        "colab_type": "code",
        "outputId": "6cc47e66-e3d9-4a67-b341-86c86968debd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 352
        }
      },
      "source": [
        "fit_v1()"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/container.py:92: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
            "  input = module(input)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfzElEQVR4nO3deZxWdd3/8deHYd8EZBQbwAFDwTXx\nEgg3zDQ0BStLydQMI27XjFtF7b4tzS3L3NAkNf3hr6jcIsRMDK1ckMEFQRZHZN9GFgUBYeBz//G9\nxrkYhpkL5po5c871fj4e53FdZ+G6PocD7/nO95zzPebuiIhI/DWJugAREckNBbqISEIo0EVEEkKB\nLiKSEAp0EZGEaBrVF3fu3NmLi4uj+noRkViaPn36R+5eWN26yAK9uLiYkpKSqL5eRCSWzGzhrtap\ny0VEJCEU6CIiCaFAFxFJCAW6iEhC1BroZvaIma0ys5m1bHe0mZWb2Vm5K09ERLKVTQv9UWBwTRuY\nWQFwO/CPHNQkIiJ7oNZAd/d/AWtq2ewy4ElgVS6KEhGR3VfnPnQzKwK+ATyQxbYjzKzEzErKysr2\n7AtXrICrroKlS/fsz4uIJFQuToreBVzj7ttr29Ddx7p7yt1ThYXV3uhUuylT4De/gZ49YfhweO01\n0JjuIiI5CfQUMN7MFgBnAfeb2Zk5+NzqDRsG8+aFMB8/HgYOhN694ZZbYNGievtaEZHGrs6B7u49\n3L3Y3YuBJ4CL3f2ZOldWk5494f77Q/fLI4/AfvvB9dfD/vvDoEHw8MPw8cf1WoKISGOTzWWLfwRe\nAw4ysyVmNtzMRprZyPovrxbt2sGFF8JLL8H8+XDTTbB8OVx0EXTpAmefDRMnwtatUVcqIlLvLKpn\niqZSKa+XwbncYdo0GDcudMl89BF07QqjRsEPfwht2uT+O0VEGoiZTXf3VHXrknenqBn06wf33gvL\nlsFf/woHHABXXgnFxaE7Znut529FRGIneYGeqVkzGDIkdMm88ko4eXrRRXDiiSHsRUQSJNmBnmng\nQPjXv8JJ1OnTIZUCjccuIgmSP4EOoTvmwgvh9dehRQs4+WR4882oqxIRyYn8CvQKhx4aumH22gu+\n/nV1v4hIIuRnoEO4Zv3ZZ2H9ejjrLCgvj7oiEZE6yd9ABzjkEHjooTB8wB13RF2NiEid5HegA5xz\nTmih/+xnMLPGId9FRBo1BTqEYQTatw83HukadRGJKQU6QGFhGMHx9dfht7+NuhoRkT2iQK9w7rnh\nMsbRozXWuojEkgK9gllonZeXw2WXRV2NiMhuU6Bn6tkznBx9+ukwiYjEiAK9qiuvhCOOgEsvhU8+\niboaEZGsKdCratYMfve7MK76dddFXY2ISNYU6NU5+mi4/HIYMyY8IENEJAYU6Lty663Qt2+4+mXu\n3KirERGplQJ9V1q1CidGW7SAwYNh8eKoKxIRqZECvSbdu8Nzz8GaNXDSSbBgQdQViYjskgK9Nkcd\nFUK9rAz694epU6OuSESkWgr0bAwcGEZkbNMGjj8+jMy4bVvUVYmI7ECBnq3eveGNN8IDMa6+Gr7y\nFY3OKCKNigJ9d3TuDE8+CY8+Cu++G25AuuSS0B0jIhKxWgPdzB4xs1VmVm1z1MzONbMZZvaumb1q\nZkfkvsxGxAwuuADefx8uvhgefBCKi+Gqq2DlyqirE5E8lk0L/VFgcA3rPwROcPfDgJuAsTmoq/Hb\ne2+4997Q7fLNb8Kdd0KPHvDjH8PChVFXJyJ5qNZAd/d/AWtqWP+qu69Nz74OdM1RbfHQuzeMGwez\nZ8O3vx3uLj3ggPAkpJKSqKsTkTyS6z704cBzOf7MeDjwQHjsMZg/Pwzw9dxzYQiBE06ACRP0JCQR\nqXc5C3QzO5EQ6NfUsM0IMysxs5KypJ5I7NYtXNa4eHHohlmwAIYOhT59Qn/7pk1RVygiCZWTQDez\nw4GHgKHuvnpX27n7WHdPuXuqsLAwF1/deLVvH1rqpaXwhz9A27YwciTsvz/84hewdm3tnyEishvq\nHOhm1h14CjjP3efVvaSEadYMhg0L/elTpkAqBf/zP6ElP2oULFkSdYUikhDZXLb4R+A14CAzW2Jm\nw81spJmNTG/yv8DewP1m9raZ6Uxgdcxg0CCYNAneeQfOPBPuvjs8JenCC8NJVRGROjB3j+SLU6mU\nl+T7VSALFoR+9oceCn3rQ4bANdeEoQZERKphZtPdPVXdOt0pGqXiYrjnHli0CG64Af7zHzjmmDCs\nwMsvR12diMSMAr0x6Nw5PJx60aLQYp89O3TPDBoU+t0j+i1KROJFgd6YtGkTroyZPz/0r8+bF1rr\nJ5wAL76oYBeRGinQG6NWrcIzTefPD8MLzJ8PX/0qHHccvPCCgl1EqqVAb8xatoRLLw3Xso8ZE8aI\nOeWU0GJ/9dWoqxORRkaBHgctW4aRHSuC/f33w8nToUM1JruIfE6BHictWlQG+y23hCthDj88DOer\nER5F8p4CPY7atIFrr4UPPoD//m/405/C4GCjR8P69VFXJyIRUaDH2d57wy9/GVrsw4bB7bfDQQfB\n44/rxKlIHlKgJ0HXruGxeK+9BkVFcN55cOyxYYgBEckbCvQkGTAApk6Fhx8OJ05TKfjpT2Hz5qgr\nE5EGoEBPmiZN4Ac/gDlz4Nxz4eab4cgjdZmjSB5QoCdVp06hG+bvf4eNG0MXzKhR8NlnUVcmIvVE\ngZ50X/tauFb9Rz8K48T07w/vvRd1VSJSDxTo+aBdO3jgAfjb32DZMjjqKLj/fl0JI5IwCvR8cvrp\nMGNGGMXxkkvgjDNg1aqoqxKRHFGg55suXcJTk+6+GyZPhiOOgH/8I+qqRCQHFOj5yCyM5vjGG+Hk\n6de+BlddBVu2RF2ZiNSBAj2fHX44TJsGI0fCr34FX/5yGINdRGJJgZ7vWrcOJ0yffjo847RvX/j9\n73XCVCSGFOgSnHlmGCrg6KPDjUnDhsG6dVFXJSK7QYEulbp2DSdKb7kFnngCvvQleOWVqKsSkSwp\n0GVHBQVhaN5XXgnvjz8ebrwRysujrkxEaqFAl+r17w9vvQXf/S7ccAMMHKjRG0UauVoD3cweMbNV\nZlbts84suMfMSs1shpn1zX2ZEon27WHcOBg/PpwwTaVC633TpqgrE5FqZNNCfxQYXMP6U4Fe6WkE\n8EDdy5JG5eyzYfZs+N734LbbwuWOzz8fdVUiUkWtge7u/wLW1LDJUOD/efA60MHM9stVgdJI7L13\nuJxx8uQwP3gwnHZaCHoRaRRy0YdeBCzOmF+SXrYTMxthZiVmVlJWVpaDr5YGd9JJYfTGX/0qnDg9\n7DC47DJYsSLqykTyXoOeFHX3se6ecvdUYWFhQ3615FKLFmFs9dJS+OEPw41JPXuGZStXRl2dSN7K\nRaAvBbplzHdNL5OkKywMYT5nDnz723DXXdCjB1x5JXz4YdTVieSdXAT6BOD89NUuA4CP3X15Dj5X\n4uKLX4THHqsM9vvuC8u+9S34z380jIBIA8nmssU/Aq8BB5nZEjMbbmYjzWxkepNJwHygFPgdcHG9\nVSuNW69eIdgXLIBrroGXXoLjjgvPNL37bvjoo6grFEk084haT6lUyktKSiL5bmkgGzeG69gfeghK\nSqBZs/BQjQsugFNOgZYto65QJHbMbLq7p6pbpztFpf60bh2eZTptWnhS0mWXwb//DUOHwj77hAHA\n/vIX2LAh6kpFEkEtdGlYW7fCP/8JTz4JzzwDZWWhpX7iiaHVfsop0KdPeAiHiOykpha6Al2is21b\nOGn61FPhztO5c8PyoqIQ7CecAMccAwccoIAXSVOgSzwsXAgvvBCecTp5MqxdG5bvs08YHOyYY2DA\ngPAc1Hbtoq1VJCIKdImf7dvDsAKvvBKmV18NNzJBaK1/8Yvh6pmK6YgjYN991ZKXxFOgSzKsXBlO\nsL71VuW0YEHl+g4doHfvMPXpU/m+Z09o2jSyskVySYEuybV2bRinfcaMcGNTxbQ84962ggLo1i3c\nxVoxFRdXvu/SBZrogi+Jh5oCXc0WibeOHWHQoDBlWrcunGSdMwfmzQtDEXz4IUyatPNAYs2awRe+\nEKaiop3fV7y2basuHWnUFOiSTB06hKcu9e+/87qNG8MJ2A8/DF02CxfCsmVhevfdcMXN+vU7/7mW\nLcP4NdlMnTqFGgoK6n1XRSoo0CX/tG4d+tj79Nn1NuvXV4b80qXhddWqcN18WVkYxmDu3PD+0093\n/Tnt2oXfIjp0CFPF+6qvFVO7dmFq2za8tm6t3wokawp0keq0awcHHRSm2mzaVBn0FdPatWFat67y\ndd268FtBxbLqfguoyqwy3DODvrr5tm3DD4CqU6tW1S9r3lw/LBJGgS67zb1uOVBxHt4MZs2CTz6B\nzZvhvfegvDw3NdamtBRWr87Vp7UCugPdKSiAQw4JeUmH9FS8859o0gQOPmgbrf3T8BewYQNsWA+f\nbgw/IDZu3PW0dhMsrZhfEl4/21xtZcvZj8U7jG6dwQyaNQ/B3rxZOJdQMd+sGTRrCk2bhSuEmjZN\nzzeFgqZhfcXyzKnW5QU7fmYOf6Bs3hz+PWXzb2jVqmhHeP7Rj+Dqq3P/uQp0ydrixXD99eGZ0Xvt\nFXosvvSlcMl4v35w9NFhSPQ336z5cxYtCvl12GEwdWrD1F5VkybhApdcX9xSVgaPP57t1gVA+/QU\nAQe2pKeE2KvFJvZp/Wn4QWGkX9MTlcsKCuDI7hto1XJ7+EdgTcI2TSy8b9Kk8r3ZjvOZ23z+Z9Kf\nn6X996+f/Vegyy598AHcfHMIqe7d4W9/C93JxxwTWjiLFoWxtlq2hDFjKv/cwIGhO3hXunULjdBp\n0+DCC+E73wn/H3r0COcTG0KLFqHnIde2bQsN7tqsWVN5n1R9atYs/MbQvHmOP3jbNvjss8ppy5bQ\nRM58zXb91q1hvrw8vG7dmjFtga3lNazfcdv2W9dQsH5zds30XD+Gp+pvJzXNH/0D4PIcF6BAl11Y\ntSoMZZ55OXfLliHUTzstzJeXh26Lzp3h738PQbbvvmGcrXztmi0oCOc5a9OxYxiiJr4KgNbpqRFy\nrwz6ih8C1b2Wl+84bd1a83yutqmpxVMHCnTZiXt4PsXKlaFLpF+/0FovLIT2Gb0DTZuGAAf4+tej\nqVWkWmbpcwPNoU2bqKtpMAp02cGSJaEbZPJkuO66EOYQ99akSH7Q/c4ChBb4GWeE/u3Jk+H22+Gm\nm6KuSkR2h1roeezjj8NNkatWhYcJQTihed99YQBDEYkXBXqeWrkS/uu/4Omnw3z79uFpcCeeGE7G\ni0j8KNDz1EUXwcSJYRjx0aPDpW2HHRZ1VSJSFwr0PLR+Pbz8MnzlK+HhQBo/SiQZdFI0D/361yHU\nb71VYS6SJFkFupkNNrO5ZlZqZqOrWd/dzKaY2VtmNsPMTst9qZILGzeG2/O/8Y3KSxJFJBlqDXQz\nKwDGAKcCBwPDzOzgKpv9FPizux8JnAPcn+tCJTeefDJc3XLFFVFXIiK5lk0LvR9Q6u7z3X0LMB4Y\nWmUbp3KEob2AZbkrUXJl5kw4//zwfOXjj4+6GhHJtWwCvQhYnDG/JL0s08+A75nZEmAScFl1H2Rm\nI8ysxMxKysrK9qBc2VPvvBO6WNq2hfvvz9+xVkSSLFcnRYcBj7p7V+A0YJyZ7fTZ7j7W3VPunips\nqGH1BAjXmH/2WRhz/OSTo65GROpDNoG+FHYYIb8rOw88ORz4M4C7vwa0BDrnokCpu2efDcPgnnRS\nuLVfRJIpm0CfBvQysx5m1pxw0nNClW0WAScBmFkfQqCrT6URWL4czj47PKFs7NioqxGR+lTrjUXu\nXm5mlwLPEwZBfsTdZ5nZjUCJu08ARgG/M7MrCSdIv+9e8aAxidKNN4ZnGE+cCMXFUVcjIvXJosrd\nVCrlJSUlkXx3vvj0UygqgsGDw2PjRCT+zGy6u6eqW6c7RRPsttvCNeeXXBJ1JSLSEBToCTVjBvzi\nF+F5ncceG3U1ItIQFOgJddtt4Zrz3/5W15yL5AsFegJt3RrGOT///OweWCwiyaBAT6DHH4fNm+G4\n46KuREQakgI9YdasgWuvhaOOCiMqikj+0AMuEmT7dvjWt2D16nB3aIsWUVckIg1JgZ4gDz0EL70U\nxjs/6qioqxGRhqYul4RwD6MoHnEEXH551NWISBQU6AkxdWoYInfkSF2mKJKvFOgJsH07/PSnYQCu\nc8+NuhoRiYoCPebmzQv95S++CHfeGUJdRPKTTorG2IoV0L8/NGkC99wDw4dHXZGIREmBHlMffxwu\nUfz00zBuS+/eUVckIlFTl0sMzZwZ7gJ99VW44w6FuYgECvQY+eAD+MlPIJWClSth0iS44oqoqxKR\nxkJdLjGxfTv07QuffAJDh4bHye2zT9RViUhjohZ6DKxcCYceGsL8zDPhmWcU5iKyM7XQG7nVq6Fb\ntzAk7vDh8MtfRl2RiDRWaqE3Ym+9BSefHML8nnvCWC2dOkVdlYg0VmqhN1KbNoXuleXLwwOezz47\n6opEpLFToDdSt94KixbByy/D8cdHXY2IxIG6XBohd3j4YTj9dIW5iGQvq0A3s8FmNtfMSs1s9C62\n+Y6ZvWdms8zsD7ktM7/MmQPLlsGQIVFXIiJxUmuXi5kVAGOAk4ElwDQzm+Du72Vs0wu4FjjG3dea\nmS6qq4N//CO8nnxytHWISLxk00LvB5S6+3x33wKMB4ZW2eaHwBh3Xwvg7qtyW2b+2LIFbr893M5f\nXBx1NSISJ9kEehGwOGN+SXpZpgOBA83sFTN73cwGV/dBZjbCzErMrKSsrGzPKk64l18OV7bcdFPU\nlYhI3OTqpGhToBcwCBgG/M7MOlTdyN3HunvK3VOFhYU5+upkmTo1vKq7RUR2VzaBvhToljHfNb0s\n0xJggrtvdfcPgXmEgJfdsGQJ3H03DBgAe+0VdTUiEjfZBPo0oJeZ9TCz5sA5wIQq2zxDaJ1jZp0J\nXTDzc1hn4n32WRg5cf16ePDBqKsRkTiqNdDdvRy4FHgemA382d1nmdmNZlZxYd3zwGozew+YAlzl\n7qvrq+gkOusseOopuO46OPzwqKsRkTgyd4/ki1OplJeUlETy3Y3N22/DkUeGUP/Tn8Ij5UREqmNm\n0909Vd06RUcj8POfQ+vWcO+9CnMR2XOKj4ht3AgvvQTf+Q506RJ1NSISZwr0iN12G6xbB+edF3Ul\nIhJ3CvQIbdgAd90FrVppEC4RqTsNnxuhcePCZYpTpkBTHQkRqSO10CP02GPh6pYTToi6EhFJAgV6\nRDZsgJISOPVUMIu6GhFJAgV6RCZMgG3b4Ljjoq5ERJJCgR6BFSvg6quhRw8FuojkTt6fitu2DS6/\nHKZPb7jvXLYM1qyByZOhTZuG+14RSbZEBPq4ceF1T67lHjUK7r8/XDbYqlVu69qVjh1h5EgYOLBh\nvk9E8kMixnKpOKm4ffvunWCcOBHOOAMuvhjuu08nJ0Wk8cubsVx25yFIGzeGMO/aNdzcozAXkbhL\nVKCvWJH9tv/+d3gdMQKaNaufekREGlLsA33btsr3y5dn/+fuuw8KC+HHP859TSIiUYh9oK9fX/k+\n20DfuhX++U84+2xo165+6hIRaWixD/Ty8sr32Qb6rFmhD11XmYhIksQ+0Ldvr3yfbR/6tGnh9eij\nc1+PiEhUEhXo2bbQ33gjXAt+wAH1U5OISBRiH+h7clJ02rTQOteliiKSJLEP9N1toW/cCDNnqrtF\nRJInMYHeqRMsXbpjwFfnrbdCq75fv/qvTUSkISUm0Pv0gc2bYeHCmrfXCVERSaqsAt3MBpvZXDMr\nNbPRNWz3LTNzM6t2nIH6UNGHfvjh4XXWrJq3nzYNiopgv/3qty4RkYZWa6CbWQEwBjgVOBgYZmYH\nV7NdO+AKYGqui6xJRQv90EPDa22BPmMG9O1bvzWJiEQhmxZ6P6DU3ee7+xZgPDC0mu1uAm4HNuew\nvlpl9qEXFdUc6O6wYAH07NkgpYmINKhsAr0IWJwxvyS97HNm1hfo5u7P1vRBZjbCzErMrKRsd4ZG\nrEFFoDdpAoccUnOgr1kTnuW5//45+WoRkUalzidFzawJcCcwqrZt3X2su6fcPVVYWFjXrwYq+9Ar\nAn327F1f6VJxwrS4OCdfLSLSqGQT6EuBbhnzXdPLKrQDDgVeMrMFwABgQkOdGK0I74KCEOibNsGc\nOdVvu2BBeFWgi0gSZRPo04BeZtbDzJoD5wATKla6+8fu3tndi929GHgdGOLuuXkcUS0yu1wGDw7B\nPn589dsuTnccdetW/XoRkTirNdDdvRy4FHgemA382d1nmdmNZjakvgusTWaXS1ERHHkkPPPMjkMC\nVFixIjzMolOnhq1RRKQhZPWQaHefBEyqsux/d7HtoLqXlb3MFjrA5ZfD+efD00/DWWftuO3KlbDP\nPpXbiogkSeyjLbMPHeC73w2XJT788M7brlgBXbo0XG0iIg0pMYFe0eouKIBjj4W33w7XnWdSoItI\nksU+0DP70Cscd1wI7xde2HHblSsV6CKSXLEP9KpdLgDnnQfdu8N111Wu375dgS4iyZaYQM9sobdo\nAT/7GUyfHvrT33wTVq8Orfl9942kTBGRepfIQAf4/vfhwQfD3aEjR1Y+b1QtdBFJqqwuW2zMqutD\nh/B4uREjYN48+PWv4cwzw3IFuogkVWJa6Jl96Jmuuy4Mlzt/fuhXrxg3XUQkaWLfQt9Vl0uFTp3g\njTfgnXdCmDeN/R6LiFQv9vFWW6BDaL3roRYiknSx73LZVR+6iEi+iX0M1taHLiKSLxIT6Gqhi0i+\ni30MKtBFRILYx6D60EVEgtjHoPrQRUSCxAS6Wugiku9iex36xo3wyCNw881hXoEuIvkutoF+/vnw\n5JPQvDmcdhp07Rp1RSIi0Yplu3bLFnj22fC4uc2bw/vmzaOuSkQkWrEM9LlzQ5CffnoYVVFERGIa\n6KWl4fXAA6OtQ0SkMYlloH/wQXg94IBo6xARaUyyCnQzG2xmc82s1MxGV7P+J2b2npnNMLMXzWz/\n3JdaaeFC6NAhTCIiEtQa6GZWAIwBTgUOBoaZ2cFVNnsLSLn74cATwC9zXWimBQtg/3r9kSEiEj/Z\ntND7AaXuPt/dtwDjgaGZG7j7FHffmJ59HajXiwgXLIDi4vr8BhGR+Mkm0IuAxRnzS9LLdmU48Fx1\nK8xshJmVmFlJWVlZ9lVmcA9dLgp0EZEd5fSkqJl9D0gBd1S33t3HunvK3VOFhYV79B0bNsD69VBU\n048UEZE8lM2dokuBbhnzXdPLdmBmXwWuB05w989yU97O1q0Lrx071tc3iIjEUzYt9GlALzPrYWbN\ngXOACZkbmNmRwIPAEHdflfsyK61dG14V6CIiO6o10N29HLgUeB6YDfzZ3WeZ2Y1mNiS92R1AW+Av\nZva2mU3YxcfVmQJdRKR6WQ3O5e6TgElVlv1vxvuv5riuXaroctE16CIiO4rdnaKFhfDNb0KXLlFX\nIiLSuMRu+NyBA8MkIiI7il0LXUREqqdAFxFJCAW6iEhCKNBFRBJCgS4ikhAKdBGRhFCgi4gkhAJd\nRCQhzN2j+WKzMmDhHv7xzsBHOSwnDrTP+UH7nB/qss/7u3u1449HFuh1YWYl7p6Kuo6GpH3OD9rn\n/FBf+6wuFxGRhFCgi4gkRFwDfWzUBURA+5wftM/5oV72OZZ96CIisrO4ttBFRKQKBbqISELELtDN\nbLCZzTWzUjMbHXU9uWJm3cxsipm9Z2azzOyK9PJOZvaCmb2ffu2YXm5mdk/672GGmfWNdg/2jJkV\nmNlbZjYxPd/DzKam9+tP6QeTY2Yt0vOl6fXFUdZdF2bWwcyeMLM5ZjbbzL6c5ONsZlem/03PNLM/\nmlnLJB5nM3vEzFaZ2cyMZbt9XM3sgvT275vZBbtTQ6wC3cwKgDHAqcDBwDAzOzjaqnKmHBjl7gcD\nA4BL0vs2GnjR3XsBL6bnIfwd9EpPI4AHGr7knLiC8PDxCrcDv3H3LwJrgeHp5cOBtenlv0lvF1d3\nA393997AEYT9T+RxNrMi4HIg5e6HAgXAOSTzOD8KDK6ybLeOq5l1Am4A+gP9gBsqfghkxd1jMwFf\nBp7PmL8WuDbquuppX/8KnAzMBfZLL9sPmJt+/yAwLGP7z7eLywR0Tf8j/wowETDC3XNNqx5v4Hng\ny+n3TdPbWdT7sAf7vBfwYdXak3qcgSJgMdApfdwmAl9L6nEGioGZe3pcgWHAgxnLd9iutilWLXQq\n/3FUWJJelijpXzOPBKYC+7r78vSqFcC+6fdJ+Lu4C7ga2J6e3xtY5+7l6fnMffp8f9PrP05vHzc9\ngDLg9+mupofMrA0JPc7uvhT4FbAIWE44btNJ/nGusLvHtU7HO26Bnnhm1hZ4Evixu3+Suc7Dj+xE\nXGdqZqcDq9x9etS1NLCmQF/gAXc/EviUyl/DgcQd547AUMIPsi8Abdi5WyIvNMRxjVugLwW6Zcx3\nTS9LBDNrRgjz/+/uT6UXrzSz/dLr9wNWpZfH/e/iGGCImS0AxhO6Xe4GOphZ0/Q2mfv0+f6m1+8F\nrG7IgnNkCbDE3aem558gBHxSj/NXgQ/dvczdtwJPEY590o9zhd09rnU63nEL9GlAr/QZ8uaEkysT\nIq4pJ8zMgIeB2e5+Z8aqCUDFme4LCH3rFcvPT58tHwB8nPGrXaPn7te6e1d3LyYcx3+6+7nAFOCs\n9GZV97fi7+Gs9Paxa8W6+wpgsZkdlF50EvAeCT3OhK6WAWbWOv1vvGJ/E32cM+zucX0eOMXMOqZ/\nuzklvSw7UZ9E2IOTDqcB84APgOujrieH+3Us4dexGcDb6ek0Qv/hi8D7wGSgU3p7I1zx8wHwLuEq\ngsj3Yw/3fRAwMf2+J/AGUAr8BWiRXt4yPV+aXt8z6rrrsL9fAkrSx/oZoGOSjzPwc2AOMBMYB7RI\n4nEG/kg4T7CV8JvY8D05rsAP0vtfCly4OzXo1n8RkYSIW5eLiIjsggJdRCQhFOgiIgmhQBcRSQgF\nuohIQijQRUQSQoEuIpIQ/wcNHTkf5decZQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Loss before training 1.395160436630249\n",
            "Loss after training 0.8738410472869873\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1pRlEhMRAUrI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WKHevP8gAhhX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def fit_v2(x,y,model,opt, loss_fn,epochs=5000):\n",
        "  for epoch in range(epochs):\n",
        "    loss = loss_fn(model(x),y)\n",
        "\n",
        "    loss.backward()\n",
        "    opt.step()\n",
        "    opt.zero_grad()\n",
        "    \n",
        "  return loss.item()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yyn5jPd3BBic",
        "colab_type": "code",
        "outputId": "9dbc0cc0-690d-4d1c-f025-50b5adc05f5c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "fn = TwoLayerNetwork()\n",
        "loss_fn = F.cross_entropy\n",
        "opt = optim.SGD(fn.parameters(), lr=1)\n",
        "fit_v2(X_train, Y_train, fn, opt, loss_fn)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/container.py:92: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
            "  input = module(input)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8406463861465454"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wIdrGART8Cle",
        "colab_type": "text"
      },
      "source": [
        "Instead of Stochastic gradient descent, lets try Adam"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FLExE656BKNC",
        "colab_type": "code",
        "outputId": "3e20aea7-a659-42cd-f02b-6bbd0096cfbd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "opt = optim.Adam(fn.parameters(), lr = 1)\n",
        "fit_v2(X_train, Y_train, fn, opt, loss_fn)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/container.py:92: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
            "  input = module(input)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8239385485649109"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vpyvtp57DdQj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}