CS294Y_Project_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 1: Fully Connected Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instructions and assignment dates can be found on [github link](https://exception1984.github.io/CS294Y-2024/). \n",
    "Your submission will be this notebook, with all the outputs embedded in the notebook.\n",
    "We will grade only what we can see in the notebook.\n",
    "\n",
    "Main goal of this assignment is to get familiar with:\n",
    "- Basic Ibex \n",
    "- Basic PyTorch \n",
    "- Basic Tensor operations\n",
    "- Train a neural network\n",
    "\n",
    "You will train a autoencoder on fashion mnist dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IBEX is the compute cluster we use in KAUST. <br>\n",
    "IMPORTANT policy for unassociated students can be [found here](https://docs.hpc.kaust.edu.sa/policy/ibex.html#limits-on-unassociated-users), basically 1 GPU 1080TI or 2080TI. <br>\n",
    "Useful ibex documentation Quickstart can be [found here](https://docs.hpc.kaust.edu.sa/quickstart/ibex.html). <br>\n",
    "Ibex 101 slides can be [found here](https://drive.google.com/file/d/13tiL3HjCu16cJ3GP_gR37xrvZ4h7W7KH/view)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Submission note:** We grade the content in this notebook. Make sure outputs are present. You will package everything in a zip file and submit it with the following format:\n",
    "f\"P1_{LastName}_{FirstName}.V{version_number}.zip\" <br>e.g. \"P1_Smit_John_V1.zip\" - check more details on the announcement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TASK 1:  Setup (10 points)\n",
    "### Connecting to IBEX and SETUP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Complete the steps below, and leave the required output of the cells in the notebook <br>\n",
    "1. Read the [quickstart](https://docs.hpc.kaust.edu.sa/quickstart/ibex.html), and connect to ibex.\n",
    "2. Follow instructions on how to [setup miniconda](https://github.com/kaust-rccl/ibex-miniconda-install) ([full ibex guide](https://docs.hpc.kaust.edu.sa/soft_env/prog_env/python_package_management/conda/ibex.html) that covers same steps)\n",
    "3. Create a [new conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) named \"CS294Y\" that has:\n",
    "    * python, [pytorch](https://pytorch.org/get-started/locally/) and [jupyterlab](https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html) installed.\n",
    "4. Understand how to [submit jobs](https://docs.hpc.kaust.edu.sa/soft_env/job_schd/slurm/basic_jobscript.html), and follow [jupyterlab instructions to spin a server](https://docs.hpc.kaust.edu.sa/soft_env/job_schd/slurm/interactive_jobs/jupyter.html#job-on-ibex)\n",
    "5. Choose the front end of your choice:\n",
    "    * a) Connect to jupyterlab frontend as shown in the instructions. ([should see](https://jupyterlab.readthedocs.io/en/stable/_images/jupyterlab.png))\n",
    "    * b) Connect via VSCode jupyter server:\n",
    "        * You will get something like (http://gpu201-02-l:10009) in sbatch output, select kernel > use existing kernel > and use this HTTP to connect, and input the token when requested.  \n",
    "6. Execute the following two cells\n",
    "\n",
    "Important note: Future projects also need those steps. Setup for personal computers is similar,  **but we expect you to use IBEX.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi --query-gpu=name,utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Execute me in the correct environment\n",
    "import torch\n",
    "import os\n",
    "import sys\n",
    "import socket\n",
    "import time\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "cuda_tensor = torch.rand((2,3,256,256)).cuda()\n",
    "hostname = socket.gethostname()\n",
    "username = os.getlogin()\n",
    "\n",
    "print(f\"Hostname: {hostname}\", f\"Username: {username}\")\n",
    "print(torch.__version__, \"/\", torch.cuda.memory_allocated(0) / 1024**2)\n",
    "print(sys.executable, \"\\n\", sys.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TASK 2: Basic Pytorch and tensor manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2.1: Combining two tensors (2.5 points)\n",
    "Concatenate the two tensors below along the batch dimension and print the result. <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "a1 = torch.rand((2,3,256,256))\n",
    "b1 = torch.rand((3,256, 256))\n",
    "# TODO: Add/Modify your code below and print the shape\n",
    "r1 = None\n",
    "print(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2.2: Pytorch Gradient (2.5 points)\n",
    "Compute gradient of y = 3*x^2 + 2*x + 1 at x = 2 using PyTorch inbuild mehanism<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "# TODO: Add/Modify your code below and print the shape\n",
    "x = None\n",
    "y = None\n",
    "\n",
    "print(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2.3: Indexing (2.5 points)\n",
    "Select every other element in the batch, and reshape the last dimension to square"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "a1 = torch.rand((4,3,64 ** 2))\n",
    "# TODO: Add/Modify your code below and print the shape\n",
    "a1 = None\n",
    "print(None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2.4: Fix the code (7.5 points)\n",
    "What is wrong with the code below? \n",
    "Fix it and explain why it was wrong. <br>\n",
    "Hint: Think of the network in an optimization loop. <br> Can import extra classes if needed. <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SimpleNetwork, self).__init__()\n",
    "        self.layers = [\n",
    "            nn.Linear(10, 20),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(20, 5)\n",
    "        ]\n",
    "    \n",
    "    def forward(self, x):\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your interpretation and explanation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: Explain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TASK 3: Define data loaders (10 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define transforms, dataset, and dataloader for the [FashionMNIST](https://pytorch.org/vision/stable/datasets.html#fashion-mnist) dataset. <br> Finish the cell below.\n",
    "Use 50k / 10k validation split. <br>\n",
    "**Explain why and which batch size you selected.** <br> \n",
    "Hint: Use `torchvision.datasets.FashionMNIST` and `torch.utils.data.DataLoader` <br>\n",
    "\n",
    "Function load data returns a dictionary with the training phase as keys and the appropriate data <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Follow the link above, and define the following variables\n",
    "device = torch.device(\"cuda\")\n",
    "\n",
    "def load_data(batch_size, transform, data_dir= \"mnist_data/\"):\n",
    "    # TODO\n",
    "    pass\n",
    "      \n",
    "dataloaders, actual_datasets, dataset_sizes = None, None, None # TODO\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fully Connected Autoencoder\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK 4.1: Implement the autoencoder (15 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement a fully connected autoencoder with four encoder layers and decoder layers <br> \n",
    "At each layer, use a factor of 4 for increase/decrease. <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyNet(nn.Module):\n",
    "    def __init__(self, initial_size=None):\n",
    "        super(MyNet, self).__init__()\n",
    "        pass\n",
    "    def forward(self, x):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK 4.1: Training and test code (5 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Modify the code below to make use of the validation code. \n",
    "2. Note part of task 4.4: Modify the code below to return the loss on the train and validation set and plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_autoencoder(model, criterion, optimizer, dataloaders,\n",
    "                       dataset_sizes, device, num_epochs=25, \n",
    "                       save_path='saved_weight.pth', verbose=False):\n",
    "    since = time.time()\n",
    "\n",
    "    for epoch in range(num_epochs):\n",
    "        \n",
    "        epoch_str = 'Epoch {}/{}'.format(epoch, num_epochs - 1)\n",
    "        # Each epoch has a training phase\n",
    "        for phase in ['train']:\n",
    "            if phase == 'train': model.train()  # Set model to training mode\n",
    "\n",
    "            running_loss = 0.0\n",
    "\n",
    "            # Iterate over data.\n",
    "            for inputs, _ in dataloaders[phase]:  # Autoencoder doesn't need labels\n",
    "                inputs = inputs.to(device)\n",
    "\n",
    "                # zero the parameter gradients\n",
    "                optimizer.zero_grad()\n",
    "\n",
    "                # forward\n",
    "                # track history if only in train\n",
    "                with torch.set_grad_enabled(phase == 'train'):\n",
    "                    outputs = model(inputs)\n",
    "                    loss = criterion(outputs, inputs)  # Reconstruction loss\n",
    "\n",
    "                    # backward + optimize only if in training phase\n",
    "                    if phase == 'train':\n",
    "                        loss.backward()\n",
    "                        optimizer.step()\n",
    "\n",
    "                # statistics\n",
    "                running_loss += loss.item() * inputs.size(0)\n",
    "\n",
    "            epoch_loss = running_loss / dataset_sizes[phase]\n",
    "            if verbose:\n",
    "                print('[{}] {} Loss: {:.4f}'.format(epoch_str, phase, epoch_loss))\n",
    "    if verbose:\n",
    "        print()\n",
    "        time_elapsed = time.time() - since\n",
    "        print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))\n",
    "\n",
    "    torch.save(model.state_dict(), save_path)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_autoencoder(model, dataloaders, dataset_sizes, device, criterion, load_path='saved_weight.pth'):\n",
    "    # load the model weights\n",
    "    model.load_state_dict(torch.load(load_path))\n",
    "    \n",
    "    since = time.time()\n",
    "\n",
    "    for phase in ['test']:\n",
    "        if phase == 'test':\n",
    "            model.eval()   # Set model to evaluate mode\n",
    "\n",
    "        running_loss = 0.0\n",
    "\n",
    "        # Iterate over data.\n",
    "        for inputs, _ in dataloaders[phase]:  # Autoencoder doesn't need labels\n",
    "            inputs = inputs.to(device)\n",
    "\n",
    "            with torch.no_grad():\n",
    "                outputs = model(inputs)\n",
    "                loss = criterion(outputs, inputs)  # Reconstruction loss\n",
    "\n",
    "            # statistics\n",
    "            running_loss += loss.item() * inputs.size(0)\n",
    "\n",
    "        epoch_loss = running_loss / dataset_sizes[phase]\n",
    "\n",
    "        print('{} Loss: {:.4f}'.format(phase, epoch_loss))\n",
    "\n",
    "    time_elapsed = time.time() - since\n",
    "    print('Testing complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))\n",
    "\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4.3: Combine code together (15 points)\n",
    "Create a model, optimizer, loss function, and train the model. <br> \n",
    "In addition, print the model summary and check the performance on the test set. <br>\n",
    "**Explain why and which hyperparameters you selected. Which loss, etc, any design choice** <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: your explanation here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4: Visualize loss (5 points)\n",
    "Visualize a plot loss curve for the train/val set. <br>\n",
    "Note (**applies to 4.4 and 4.5**): Backup your arguments with multiple configurations visualized, backing up your explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.utils import make_grid\n",
    "from torchvision.transforms import ToPILImage\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# TODO: your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: your explanation here with plots above, explaining your findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK 4.5 Visualize Reconstruction (10 points)\n",
    "Implement input and output reconstruction from the first 10 test samples. <br>\n",
    "Expected to have ten images per row (input, output) <br>\n",
    "Note: You can use `torchvision.utils.make_grid` to make a grid of images. <br>\n",
    "**Explain the results of visualization. Did you get what you expected?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_images = 10\n",
    "data_input = actual_datasets['test']\n",
    "# TODO: your code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: your observation here about multiple reconstructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4.6: Reconstruction with Noise (5 points)\n",
    "Add noise to the input data and test output reconstruction of the same images. <br>\n",
    "Add Gaussian noise of different standard deviations. <br>\n",
    "Demonstrate different effects and explain the results. <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: your explanation and interpretation of the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK 4.7: Retrain the model with noise in mind (10 points)\n",
    "Retrain the model with noisy input augmentations. <br>\n",
    "Rerun training (task 4.3) and visualizations (4.5 and 4.6) with noisy input data. <br>\n",
    "**Explain what type of noise we can add and how you would validate your new results and your observations** <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: Explain what type of noise we would use and how we would validate the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ae",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}