Titanic neural network Numpy from scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Introduction**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Howdy, Welcome to the Titanic***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**To whom does this notebook appeal to?**<br><br>\n",
    "If you are just starting with WNNC's HELLO FOSS and want to do beginner task in ML, you can try this project!\n",
    "\n",
    "Data reading and analyzation has been done and a neural network is implemented from scratch in Numpy to predict whether or not they survived the sinking of the Titanic. The backward propogation part of the network has not been completed and we need you to code the rest of the function to return the gradients. We have also included `gender_submission.csv`, a set of predictions that assume all and only female passengers survive, as an example of what a submission file should look like.\n",
    "\n",
    "Head over to [this cell](#main) to implement back propogation of neural network and then add code for writing the output to file `predictions-ann.csv` [here](#main2). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://faithmag.com/sites/default/files/styles/article_full/public/2018-09/titanic2.jpg?h=6521bd5e&itok=H8td6QVv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Basic Imports**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./titanic\\gender_submission.csv\n",
      "./titanic\\test.csv\n",
      "./titanic\\train.csv\n"
     ]
    }
   ],
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
    "# For example, here's several helpful packages to load in \n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "# Input data files are available in the \"../input/\" directory.\n",
    "# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n",
    "\n",
    "import os\n",
    "for dirname, _, filenames in os.walk('./titanic'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))\n",
    "\n",
    "# Any results you write to the current directory are saved as output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"./titanic/train.csv\")\n",
    "#print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyze Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make a quick neural network using the data above,<br>\n",
    "we can easily create a neural network using the following the columns:<br>\n",
    "'**Age**', '**Sex**', '**Fare**', '**Pclass**', '**SibSp**', '**Parch**'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    1\n",
       "2    1\n",
       "3    1\n",
       "4    0\n",
       "Name: Survived, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets take out first the label\n",
    "train_y = df['Survived']\n",
    "train_y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to filter the age, sex, fare pclass, sibsp, parch columns\n",
    "def get_data(data):\n",
    "    # take only this specific column\n",
    "    data = data[['Age', 'Sex', 'Fare', 'Pclass', 'SibSp', 'Parch']]\n",
    "    \n",
    "    # replace male by 1, female by 0\n",
    "    data.replace({ 'male' : 1, 'female' : 0 }, inplace=True)\n",
    "    \n",
    "    # replace null/nan data by the mean (age and fare columns)\n",
    "    data['Fare'].fillna(int(data['Fare'].mean()), inplace=True)\n",
    "    data['Age'].fillna(int(data['Age'].mean()), inplace=True)\n",
    "    \n",
    "    # transform into a numpy array\n",
    "    data = data.to_numpy().astype(float)\n",
    "    \n",
    "    # normalize (make sure the data is between -1 and 1)\n",
    "    for i in range(data.shape[1]):\n",
    "        data[:,i] = (data[:,i] - data[:,i].mean()) / data[:,i].std()\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.58165904  0.73769513 -0.50244517  0.82737724  0.43279337 -0.47367361]\n",
      " [ 0.64932701 -1.35557354  0.78684529 -1.56610693  0.43279337 -0.47367361]\n",
      " [-0.27391253 -1.35557354 -0.48885426  0.82737724 -0.4745452  -0.47367361]\n",
      " ...\n",
      " [-0.04310264 -1.35557354 -0.17626324  0.82737724  0.43279337  2.00893337]\n",
      " [-0.27391253  0.73769513 -0.04438104 -1.56610693 -0.4745452  -0.47367361]\n",
      " [ 0.18770724  0.73769513 -0.49237783  0.82737724 -0.4745452  -0.47367361]]\n"
     ]
    }
   ],
   "source": [
    "train_x = get_data(df)\n",
    "print(train_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(891, 6)\n"
     ]
    }
   ],
   "source": [
    "print(train_x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shape will show us the number of rows and columns (891 and 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(891, 1)\n"
     ]
    }
   ],
   "source": [
    "# same for the labels (contains 0 - 1 if the victim survived or not)\n",
    "train_y = train_y.to_numpy()\n",
    "train_y = train_y.reshape(train_x.shape[0],1)\n",
    "print(train_y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the activation function and derivative of the action function\n",
    "def sigmoid(x):\n",
    "    return 1/(1+np.exp(-x))\n",
    "\n",
    "def dsigmoid(x):\n",
    "    return sigmoid(x) * (1 - sigmoid(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the loss function and its derivative\n",
    "def loss_fn(y, y_hat):\n",
    "    return 1/2 * (y - y_hat) ** 2\n",
    "\n",
    "def dloss_fn(y, y_hat):\n",
    "    return (y - y_hat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of rows\n",
    "instances = train_x.shape[0]\n",
    "\n",
    "# number oof columns\n",
    "attributes = train_x.shape[1]\n",
    "\n",
    "# number of hidden node for first layer \n",
    "hidden_nodes = 8\n",
    "\n",
    "# number of hidden node for second layer\n",
    "hidden_nodes_two = 4\n",
    "\n",
    "# number of output labels \n",
    "output_labels = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inititate the weights/biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "w1 = np.random.rand(attributes,hidden_nodes)\n",
    "b1 = np.random.randn(hidden_nodes)\n",
    "\n",
    "w2 = np.random.rand(hidden_nodes,hidden_nodes_two)\n",
    "b2 = np.random.randn(hidden_nodes_two)\n",
    "\n",
    "w3 = np.random.rand(hidden_nodes_two, output_labels)\n",
    "b3 = np.random.randn(output_labels)\n",
    "\n",
    "theta = w1, w2, w3, b1, b2, b3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network Forward Propogation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward(x, theta):\n",
    "    w1, w2, w3, b1, b2, b3 = theta\n",
    "    \n",
    "    k = np.dot(x, w1) + b1\n",
    "    l = sigmoid(k)\n",
    "    \n",
    "    m = np.dot(l, w2) + b2\n",
    "    n = sigmoid(m)\n",
    "    \n",
    "    o = np.dot(n, w3) + b3\n",
    "    p = sigmoid(o)\n",
    "    \n",
    "    return k, l, m, n, o, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network Backward Propogation\n",
    "<a id='main'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward(x, y, sigma, theta):\n",
    "    #complete this function\n",
    "       \n",
    "    return dw1, dw2, dw3, db1, db2, db3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the avg of the gradients for the derivative of each bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def avg_bias(grads):\n",
    "    dw1, dw2, dw3, db1, db2, db3 = grads\n",
    "    db1 = db1.mean(axis=0)\n",
    "    db2 = db2.mean(axis=0)\n",
    "    db3 = db3.mean(axis=0)\n",
    "    return dw1, dw2, dw3, db1, db2, db3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the SGD in order to optimize the weights and biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def optimize(theta, grads, lr=0.001):\n",
    "    dw1, dw2, dw3, db1, db2, db3 = grads\n",
    "    w1, w2, w3, b1, b2, b3 = theta\n",
    "    \n",
    "    w1 -= dw1 * lr\n",
    "    w2 -= dw2 * lr\n",
    "    w3 -= dw3 * lr\n",
    "    b1 -= db1 * lr\n",
    "    b2 -= db2 * lr\n",
    "    b3 -= db3 * lr\n",
    "    \n",
    "    return w1, w2, w3, b1, b2, b3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# return 1 if the prediction is higher than 0.5\n",
    "# return 0 if not\n",
    "def predict(x, theta):\n",
    "    predict = forward(x, theta)[-1]\n",
    "    return np.where(predict > 0.5, 1, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.26028048758523226\n",
      "0.11688264270387994\n",
      "0.11595864881174242\n",
      "0.11496660624169165\n",
      "0.1137825322711347\n",
      "0.11227591584390285\n",
      "0.11024756910914331\n",
      "0.10732953688698973\n",
      "0.10282167862923627\n",
      "0.09585455098103804\n"
     ]
    }
   ],
   "source": [
    "# time to train our model\n",
    "for epoch in range(1000):\n",
    "    \n",
    "    sigma = forward(train_x, theta)\n",
    "    grads = backward(train_x, train_y, sigma, theta)\n",
    "    theta = optimize(theta, avg_bias(grads))\n",
    "    \n",
    "    if(epoch % 100 == 0):\n",
    "        print(loss_fn(sigma[-1], train_y).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Print the accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training set:  79.2368125701459\n"
     ]
    }
   ],
   "source": [
    "pred_train = predict(train_x, theta)\n",
    "num = pred_train.shape[0]\n",
    "acc=0\n",
    "for i in range(num):\n",
    "     acc+=(pred_train[i,0]==train_y[i,0])\n",
    "print(\"Accuracy on training set: \", 100*acc/num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time to train the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = pd.read_csv(dirname + \"/test.csv\")\n",
    "test_x = get_data(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get test data predictions\n",
    "test_preds = predict(test_x, theta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add passengers ids to the test predictions\n",
    "passenger_ids = test_df['PassengerId'].to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Output the prediction to submission file\n",
    "<a id='main2'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine passenger ids with the predictions\n",
    "# convert array to dataframe\n",
    "# save the result to predictions-ann.csv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}