Martin_Irungu_Solution_for_lesson_3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DaymondMartin/basic-ml-course/blob/master/Martin_Irungu_Solution_for_lesson_3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BVZK027eQ6DX"
      },
      "source": [
        "# Homework 3: Logistics Regression\n",
        "Implement everywhere that has  `...`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "3N4wa9IdQ6Da"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import sklearn.datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tn1OYHFWQ6Dd"
      },
      "source": [
        "### Load dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "HzjEepqyQ6De"
      },
      "outputs": [],
      "source": [
        "import sklearn\n",
        "from sklearn.datasets import load_iris\n",
        "data = load_iris()\n",
        "X, Y = data.data, data.target\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bkRKLkY7Q6Df"
      },
      "source": [
        "### Task 1: (1pt)\n",
        "Choose only the last 2 classes (class 1 and 2) from the dataset. You may refer to the code from the lecture video.\n",
        "\n",
        "Some numpy resource that might be useful\n",
        "* Indexing in numpy: https://aman.ai/primers/numpy/#indexing\n",
        "* Boolean mask: https://aman.ai/primers/numpy/#boolean-array-indexing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OQ3wW253Q6Df",
        "outputId": "49d34232-7004-4b73-f1f2-a8f5b5e7c792",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        }
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-3-286330491790>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# Your code here\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m150\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mY\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m150\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mY\u001b[0m \u001b[0;34m-=\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;31m#We need our label to be 0 and 1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'X' is not defined"
          ]
        }
      ],
      "source": [
        "# only classify class 1 vs. class 2\n",
        "# Your code here\n",
        "\n",
        "X = X[50:150]\n",
        "Y = Y[50:150]\n",
        "Y -= 1 #We need our label to be 0 and 1\n",
        "\n",
        "assert X.shape == (100, 4) \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "euyjAd0rSvDY",
        "outputId": "02462c21-2ff7-4360-af16-7cb1eb5bf0b6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(100,)"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "Y.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9XxfcqsdQ6Dg"
      },
      "source": [
        "### Task 2: Data normalization (1pt)\n",
        "Implement z-score normalization.\n",
        "\n",
        "Remember: normalization is done per feature, not all at once. So you need to calculate the mean and standard deviation for each feature seperately."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LLWh0TY5Q6Dg"
      },
      "outputs": [],
      "source": [
        "# Write your code here.\n",
        "\n",
        "x_means = X.mean()\n",
        "x_stdevs = X.std()\n",
        "X = (X - x_means) / x_stdevs\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Px6RgYyzQ6Dh"
      },
      "source": [
        "### Task 3: Construct the design matrix (1pt)\n",
        "We need to add an extra column of 1 that represents the bias/intercept term\n",
        "\n",
        "\n",
        "\\begin{bmatrix}\n",
        "1 & x_{11} & x_{12} & \\dots\\\\\n",
        "1 & x_{21} & x_{22} & \\dots\\\\\n",
        "\\vdots  & \\vdots &\\vdots & \\vdots\\\\\n",
        "1 & x_{n1} & x_{n2} & \\dots\\\\\n",
        "\\end{bmatrix}\n",
        "\n",
        "Helpful function: https://numpy.org/doc/stable/reference/generated/numpy.hstack.html"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ApQ1ccjFQ6Di"
      },
      "outputs": [],
      "source": [
        "# Your code here\n",
        "n,m = X.shape # gives you the dimensions of your matrix\n",
        "add_ones= np.ones((n,1))\n",
        "X = np.hstack((add_ones,X))\n",
        "\n",
        "# checking the shape of X\n",
        "assert X.shape == (100, 5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ysH-CvouQ6Di"
      },
      "source": [
        "### Task 4: Model initialization (1pt)\n",
        "Create parameters here. Initialize with zeros. Output should be a numpy array \n",
        "\n",
        "In case you forgot: $Y = \\sigma(X\\theta)$."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f3l6A8MNQ6Dj"
      },
      "outputs": [],
      "source": [
        "# Your code here\n",
        "weight = np.zeros((5,))\n",
        "assert weight.shape == (5, )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JLkIOFZ9Q6Dj"
      },
      "source": [
        "### Task 5: Sigmoid function (1pt)\n",
        "Implement the sigmoid function. You should use `np.exp` for this\n",
        "\n",
        "\n",
        "$$\\sigma(z) = \\frac{1}{1 + e^{-z}}$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_nJmRkmyQ6Dj"
      },
      "outputs": [],
      "source": [
        "def sigmoid(z):\n",
        "    # Your code here\n",
        "    return 1.0 / (1.0 + np.exp(-z))\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CyrOjvuWQ6Dk"
      },
      "source": [
        "### Task 6: Define evaluation metrics (2pt)\n",
        "Implement the accuracy score\n",
        "\n",
        "$$\\frac{\\text{\\# of correct prediction}}{\\text{\\# of datapoint}}$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7egO7NqtQ6Dk"
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "def accuracy(X, y, weight):\n",
        "    '''\n",
        "    Evaluate the model, represented by `weight`, with data (X, Y).\n",
        "    \n",
        "    Input:\n",
        "        X:      data features\n",
        "        Y:      data labels\n",
        "        weight: model weights\n",
        "    Ouput:\n",
        "        Model accuracy on input data.\n",
        "        \n",
        "    '''\n",
        "    # implement your code here. Below are only some suggestion. \n",
        "    # You don't necessary have to follow the exact step\n",
        "\n",
        "    #Step 1: get the model prediction\n",
        "\n",
        "    #Step 2: Threshold the probability value to get the class. \n",
        "\n",
        "    #Step 3: Compare your model predicted classes with the label y\n",
        "\n",
        "    #step 4: calculate the accuracy using the formula above\n",
        "   \n",
        "    datapoints = len(y)\n",
        "    y_predict = sigmoid(X@weight)\n",
        "\n",
        "    y_predict = (y_predict > 2).astype(int)\n",
        "    y_correct = 0\n",
        "    \n",
        "    for y_value in range(len(y_predict),len(y)):\n",
        "      if y_predict[y_value] == y[y_value]:\n",
        "           y_correct += 1 \n",
        "\n",
        "    accuracy_score = y_correct / datapoints \n",
        "    return accuracy_score\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7LkX3qT32Pp-",
        "outputId": "3f08ee8e-e98d-4934-85aa-89a09eceb642"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "accuracy(X,Y,weight)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z0oYiXcCQ6Dl"
      },
      "source": [
        "### Hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TzaBrDT2Q6Dl"
      },
      "outputs": [],
      "source": [
        "#Don't change these\n",
        "# learning rate\n",
        "alpha = 1e-2\n",
        "# epochs\n",
        "epoch = 10000"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g5TilBMaQ6Dl"
      },
      "source": [
        "### Task 7: Train the model. (3pt)\n",
        "Implement the model training loop:\n",
        "1. Calculate the model prediction\n",
        "$$h_\\theta(X) = \\sigma(X\\theta)$$\n",
        "2. Calculate the loss/cost function \n",
        "$$ J(\\theta) = \\frac{-1}{m} \\sum_{i=1}^{m} [y_i \\ln(h_\\theta(x_i)) +(1-y_i)\\ln(1-h_\\theta(x_i))]$$\n",
        "3. Calculate the gradient\n",
        "$$\n",
        "\\frac{\\partial J(\\theta)}{\\partial \\theta} = \\frac{1}{m} X^T(Y-h_\\theta(X))\n",
        "$$\n",
        "4. Update the model parameter \n",
        "$$\\theta = \\theta - \\alpha \\frac{\\partial J(\\theta)}{\\partial \\theta} $$\n",
        "5. Calcualte the model accuracy (use the function you implemented above)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a6CLWnAwQ6Dm"
      },
      "outputs": [],
      "source": [
        "# your code here\n",
        "for ep in range(epoch):\n",
        "  y_predict = sigmoid(X@weight)\n",
        "  \n",
        "  loss = -np.sum(Y * np.log(y_predict) + (1 - Y) * np.log(1 - y_predict)) / y_predict.size \n",
        "  m = y_predict.size\n",
        "  gradient = (1 / m) * X.T @ (y_predict-Y)\n",
        "  weight = weight - alpha  * gradient\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 183
        },
        "id": "cqYQyzMEQ6Dn",
        "outputId": "d99027a4-47c3-474f-a242-b1c9c4767f8b"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "AssertionError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-24-3b0ff20a62e8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;31m# just to double check\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mallclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0.12047504\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m0.44156746\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m0.89309501\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2.965364\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3.3427994\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m: "
          ]
        }
      ],
      "source": [
        "0# just to double check\n",
        "assert np.allclose(weight, np.array([0.12047504, -0.44156746, -0.89309501, 2.965364, 3.3427994]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c5k6yoNUQ6Dn",
        "outputId": "c2c52bbe-4e0e-4bec-fe40-53b8e58a304a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "# model accuracy\n",
        "accuracy(X, Y, weight)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wRMBtiW-Q6Do"
      },
      "source": [
        "### (Optional) How low can you go?\n",
        "Do anything you want to get the best performance out of the training set. For once, let's overfit to your heart's content."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mB28v2qrQ6Do"
      },
      "outputs": [],
      "source": [
        "# do some fun code here and try to match this :)\n",
        "..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZXmQVUKyQ6Dp",
        "outputId": "33ba561f-7684-4321-e0d4-ec90a10448d2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Weights: [ 0.19444499  0.67457184 -0.22200788  4.70341986  2.97482101]\n",
            "Loss: 0.21386730349359453\n",
            "Accuracy: 0.0\n"
          ]
        }
      ],
      "source": [
        "print('Weights:', weight)\n",
        "y = 1 / (1 + np.exp(-X @ weight))\n",
        "loss = -np.sum(Y * np.log(y) + (1 - Y) * np.log(1 - y)) / y.size\n",
        "print('Loss:', loss)\n",
        "print('Accuracy:', accuracy(X, Y, weight))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2hkXggRgN5m-"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "interpreter": {
      "hash": "0d31d8e9435eeb371278650612d4e0ff3f973ecc6a4199c9ae1c8c510f127393"
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}