Intro to Pandas 3DC Solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "603af0dd-3d2f-4f6d-9454-c3a867bd8b0b",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 3DC's DDW Clinic\n",
    "\n",
    "Part of SUTD Google DSC's Data Driven World Clinics 2022 for SUTD Students.\n",
    "\n",
    "\n",
    "Exercises by Thao Vy Dinh\n",
    "\n",
    "Notebook by Beckham Wee\n",
    "\n",
    "References: Quantopian Lecture Series, pandas documentation\n",
    "\n",
    "Dataset: Dr Ernest Chan\n",
    "\n",
    "## Introduction to Pandas\n",
    "pandas is a python library that provides tools to better manage data. We will be covering 'DataFrame' and 'Series' objects, both of which help in the process. This allows us to conduct statistical inferences/machine learning/data science much more easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2bbc8358-92a5-4b47-bc81-a805e892462a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a8681e2-73e1-4cfb-a19d-280f2772c0aa",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Series\n",
    "A pandas `Series` is a 1-dimensional array with labels that can contain any data type.\n",
    "\n",
    "#### Creating a `Series` \n",
    "as easy as calling `pandas.Series()` on a Python list or NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "eb4ae53d-98d6-4069-8dc3-1a6c93420cc9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    2.0\n",
       "2    NaN\n",
       "3    3.0\n",
       "4    4.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Called on a list or ndarray\n",
    "s = pd.Series([1, 2, np.nan, 3, 4])\n",
    "\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23fe0088-24cb-4b48-a5f0-fc5646f4902e",
   "metadata": {},
   "source": [
    "#### Series' name\n",
    "Every `Series` has a name. We can give the series a name as a parameter or we can define it afterwards by directly accessing the name attribute. For now, we haven't given our series a name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c4f95025-e5dc-4cdb-9597-956dc74f8f2e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print(s.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94f4d829-f5f7-4692-b402-e7be350aebf5",
   "metadata": {},
   "source": [
    "This name can be directly set (via their own setter method!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "92109c7e-2fa2-479c-a338-440cd729b479",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My exam results out of 100\n"
     ]
    }
   ],
   "source": [
    "s.name = \"My exam results out of 100\"\n",
    "\n",
    "print(s.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "103c1adb-43cd-4e18-ba95-830bed897b71",
   "metadata": {},
   "source": [
    "#### Index\n",
    "We call the collected axis labels of a `Series` its index. An index can either passed to a `Series` as a parameter or added later. \n",
    "- If we didn't set it, a `Series` will simply contain an index composed of integers, starting at $0$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6107a31e-f8ea-4964-8381-3b7c6fb8a2da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RangeIndex(start=0, stop=5, step=1)\n"
     ]
    }
   ],
   "source": [
    "print(s.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee8cf25e-5e46-4110-b78b-ab919992d999",
   "metadata": {},
   "source": [
    "We can set the index manually, again via their own setter method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "070f3f99-740b-41f9-8272-7cffce2e18b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Modelling Uncertaininty', 'Bio', 'DDW', 'HASS', 'FCP'], dtype='object')\n",
      "Modelling Uncertaininty    1.0\n",
      "Bio                        2.0\n",
      "DDW                        NaN\n",
      "HASS                       3.0\n",
      "FCP                        4.0\n",
      "Name: My exam results out of 100, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "new_index = ['Modelling Uncertaininty', 'Bio', 'DDW', 'HASS', 'FCP']\n",
    "s.index = new_index\n",
    "\n",
    "print(s.index)\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6848306d-b5d2-4329-8af0-430b2be8a35d",
   "metadata": {},
   "source": [
    "## DataFrames\n",
    "\n",
    "`DataFrames` are 2-dimensional objects, unlike `Series` which are 1-dimensional. In other words, DataFrames have more than one column.\n",
    "- Each `DataFrame` has an index and a columns attribute\n",
    "- The index attribute is the same as the index of a `Series`\n",
    "- Columns allow us to combine named columns (all `Series`), into a cohesive object. They will share the index."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe3871f-6816-482a-ae96-b30a71876424",
   "metadata": {},
   "source": [
    "#### Creating DataFrames\n",
    "\n",
    "We can create a `DataFrame` by calling `pandas.DataFrame()` on a dictionary or NumPy `ndarray`. We can also concatenate a group of pandas `Series` into a `DataFrame` using `pandas.concat()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "34563c5c-e9df-47a2-99a0-afd226a4d6ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dictionary\n",
    "dict_data = {\n",
    "    'a' : [1, 2, 3, 4, 5],\n",
    "    'b' : ['A', 'B', 'C', 'D', 'E'],\n",
    "    'c' : ['3', 'D', 'C', 'IS', 'COOL']\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f754b2-bb34-40ae-a2d9-92efd3b4413f",
   "metadata": {},
   "source": [
    "We can also set the index directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "625169fa-6f9c-47b0-a06c-32f883fe0f70",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1st</th>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2nd</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "      <td>D</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3rd</th>\n",
       "      <td>3</td>\n",
       "      <td>C</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4th</th>\n",
       "      <td>4</td>\n",
       "      <td>D</td>\n",
       "      <td>IS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5th</th>\n",
       "      <td>5</td>\n",
       "      <td>E</td>\n",
       "      <td>COOL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     a  b     c\n",
       "1st  1  A     3\n",
       "2nd  2  B     D\n",
       "3rd  3  C     C\n",
       "4th  4  D    IS\n",
       "5th  5  E  COOL"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Method 1: Creating DataFrame using dictionaries or nparrays\n",
    "df = pd.DataFrame(data=dict_data, index=['1st','2nd', '3rd', '4th', '5th'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf05883-bbba-42b7-bf3a-60ee801294d4",
   "metadata": {},
   "source": [
    "By concatenating, we have to specificy an axis.\n",
    "\n",
    "- `axis = 0` implies that we are adding the other Series as a row.\n",
    "- `axis = 1` implies that we are applying this operation to the columns.\n",
    "\n",
    "This is widely global for all methods in pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "010d34ac-5f1d-4464-87e6-1f24bd59b412",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Evens</th>\n",
       "      <th>Odds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Evens  Odds\n",
       "0      2     1\n",
       "1      4     3\n",
       "2      6     5\n",
       "3      8     7\n",
       "4     10     9"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Method 2: Creating by combining series\n",
    "s_1 = pd.Series([2, 4, 6, 8, 10], name='Evens')\n",
    "s_2 = pd.Series([1, 3, 5, 7, 9], name=\"Odds\")\n",
    "numbers = pd.concat([s_1, s_2], axis=1)\n",
    "\n",
    "numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d26bb435-3705-497d-bccf-821904c567db",
   "metadata": {},
   "source": [
    "#### DataFrame Columns\n",
    "Each `DataFrame` also has a `columns` attribute. These can either be assigned when we call `pandas.DataFrame` or they can be set.\n",
    "- However, `using pd.concat()` basically sets the names for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "d9a89533-589d-4d38-91bc-1470c980a9b8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DAMN ITS EVEN</th>\n",
       "      <th>DAMN ITS ODD</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   DAMN ITS EVEN  DAMN ITS ODD\n",
       "0              2             1\n",
       "1              4             3\n",
       "2              6             5\n",
       "3              8             7\n",
       "4             10             9"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can modify columns names just like that using a list\n",
    "numbers.columns = ['DAMN ITS EVEN', 'DAMN ITS ODD']\n",
    "\n",
    "numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01e63e47-7135-416d-b7c6-101322a601ed",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Indexing\n",
    "Refer to Series' indexing. Every column in a DataFrame shares the same index."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e32fc8b4-2457-4e1b-ac43-e92e70bb2cfc",
   "metadata": {},
   "source": [
    "#### HACKS: Getting the values out FAST\n",
    "You'll only know this if you attended the clinic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c3cb38f3-600c-431b-931b-e85f3bccc8d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2,  1],\n",
       "       [ 4,  3],\n",
       "       [ 6,  5],\n",
       "       [ 8,  7],\n",
       "       [10,  9]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# numbers.values returns a np array\n",
    "numbers.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "34e1fca5-2e3e-4c1b-bccd-d30f2738ae6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(numbers.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9ade050-949b-497b-860f-142416f9a6aa",
   "metadata": {},
   "source": [
    "#### Accessing elements in DataFrame\n",
    "In the real world, we need to be using DataFrames to get data, or manipulate them. Here's a practical example of stock data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8c00e3cb-0b55-42ae-89cf-9d420394c9f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading data here from excel\n",
    "PEP = pd.read_excel('PEP.xls')\n",
    "PEP.set_index('Date', inplace=True)\n",
    "\n",
    "PEP.sort_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "9d379253-0b50-4e95-b2b6-7129f3e197c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1977-01-03</th>\n",
       "      <td>79.00</td>\n",
       "      <td>80.13</td>\n",
       "      <td>79.00</td>\n",
       "      <td>80.00</td>\n",
       "      <td>1085400</td>\n",
       "      <td>0.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-04</th>\n",
       "      <td>80.00</td>\n",
       "      <td>80.25</td>\n",
       "      <td>78.88</td>\n",
       "      <td>79.00</td>\n",
       "      <td>3018600</td>\n",
       "      <td>0.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-05</th>\n",
       "      <td>78.75</td>\n",
       "      <td>78.75</td>\n",
       "      <td>77.88</td>\n",
       "      <td>78.38</td>\n",
       "      <td>2246400</td>\n",
       "      <td>0.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-06</th>\n",
       "      <td>78.63</td>\n",
       "      <td>79.00</td>\n",
       "      <td>78.63</td>\n",
       "      <td>78.75</td>\n",
       "      <td>1004400</td>\n",
       "      <td>0.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-07</th>\n",
       "      <td>78.75</td>\n",
       "      <td>78.75</td>\n",
       "      <td>78.25</td>\n",
       "      <td>78.50</td>\n",
       "      <td>950400</td>\n",
       "      <td>0.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-14</th>\n",
       "      <td>78.25</td>\n",
       "      <td>78.39</td>\n",
       "      <td>76.65</td>\n",
       "      <td>76.92</td>\n",
       "      <td>7762500</td>\n",
       "      <td>76.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-15</th>\n",
       "      <td>76.02</td>\n",
       "      <td>76.55</td>\n",
       "      <td>74.71</td>\n",
       "      <td>74.83</td>\n",
       "      <td>9342100</td>\n",
       "      <td>74.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-16</th>\n",
       "      <td>74.35</td>\n",
       "      <td>74.85</td>\n",
       "      <td>73.28</td>\n",
       "      <td>73.54</td>\n",
       "      <td>9636900</td>\n",
       "      <td>73.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-17</th>\n",
       "      <td>73.70</td>\n",
       "      <td>73.70</td>\n",
       "      <td>71.84</td>\n",
       "      <td>72.10</td>\n",
       "      <td>10016800</td>\n",
       "      <td>72.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-18</th>\n",
       "      <td>72.54</td>\n",
       "      <td>73.16</td>\n",
       "      <td>70.95</td>\n",
       "      <td>71.46</td>\n",
       "      <td>8986000</td>\n",
       "      <td>71.46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7835 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Open   High    Low  Close    Volume  Adj Close\n",
       "Date                                                       \n",
       "1977-01-03  79.00  80.13  79.00  80.00   1085400       0.66\n",
       "1977-01-04  80.00  80.25  78.88  79.00   3018600       0.65\n",
       "1977-01-05  78.75  78.75  77.88  78.38   2246400       0.65\n",
       "1977-01-06  78.63  79.00  78.63  78.75   1004400       0.65\n",
       "1977-01-07  78.75  78.75  78.25  78.50    950400       0.65\n",
       "...           ...    ...    ...    ...       ...        ...\n",
       "2008-01-14  78.25  78.39  76.65  76.92   7762500      76.92\n",
       "2008-01-15  76.02  76.55  74.71  74.83   9342100      74.83\n",
       "2008-01-16  74.35  74.85  73.28  73.54   9636900      73.54\n",
       "2008-01-17  73.70  73.70  71.84  72.10  10016800      72.10\n",
       "2008-01-18  72.54  73.16  70.95  71.46   8986000      71.46\n",
       "\n",
       "[7835 rows x 6 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PEP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4051d93d-4fe5-4c08-9213-5b429f0fedb2",
   "metadata": {},
   "source": [
    "That's a lot of data... sometimes we just need a small segment of that.\n",
    "\n",
    "We can do this either by directly accessing the attributes or by using `.iloc[]` and `.loc[]`.\n",
    "- They both take in indices for row and columns\n",
    "\n",
    "#### Method 1: iloc\n",
    "Let's look at `iloc[]` first. It's used when we are only using INTEGER INDICES, just like an array. For example, we want to rows 1 to 50, as well as columns `Close` and `Volume`\n",
    "- Rows 1 to 10 will be represented as `0:10`\n",
    "- Close has an index of 3, Volume has an index of 4, hence `3:5`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "c43fdf71-1d77-4cdc-8169-c50d293a9bce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1977-01-03</th>\n",
       "      <td>80.00</td>\n",
       "      <td>1085400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-04</th>\n",
       "      <td>79.00</td>\n",
       "      <td>3018600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-05</th>\n",
       "      <td>78.38</td>\n",
       "      <td>2246400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-06</th>\n",
       "      <td>78.75</td>\n",
       "      <td>1004400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-07</th>\n",
       "      <td>78.50</td>\n",
       "      <td>950400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-10</th>\n",
       "      <td>78.75</td>\n",
       "      <td>1641600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-11</th>\n",
       "      <td>77.75</td>\n",
       "      <td>1263600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-12</th>\n",
       "      <td>77.88</td>\n",
       "      <td>1063800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-13</th>\n",
       "      <td>78.75</td>\n",
       "      <td>766800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-14</th>\n",
       "      <td>78.00</td>\n",
       "      <td>394200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Close   Volume\n",
       "Date                      \n",
       "1977-01-03  80.00  1085400\n",
       "1977-01-04  79.00  3018600\n",
       "1977-01-05  78.38  2246400\n",
       "1977-01-06  78.75  1004400\n",
       "1977-01-07  78.50   950400\n",
       "1977-01-10  78.75  1641600\n",
       "1977-01-11  77.75  1263600\n",
       "1977-01-12  77.88  1063800\n",
       "1977-01-13  78.75   766800\n",
       "1977-01-14  78.00   394200"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example1 = PEP.iloc[0:10, 3:5]\n",
    "print(example1.shape)\n",
    "example1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23a4815b-6fb0-430f-bd97-e5dcfb28c137",
   "metadata": {},
   "source": [
    "To get all rows, but only specific columns, we can just leave the row index as `:`\n",
    "- we can do the same thing for columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "1e5c827b-508e-42a9-9859-bf9d304bdbf7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7835, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1977-01-03</th>\n",
       "      <td>80.00</td>\n",
       "      <td>1085400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-04</th>\n",
       "      <td>79.00</td>\n",
       "      <td>3018600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-05</th>\n",
       "      <td>78.38</td>\n",
       "      <td>2246400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-06</th>\n",
       "      <td>78.75</td>\n",
       "      <td>1004400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1977-01-07</th>\n",
       "      <td>78.50</td>\n",
       "      <td>950400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-14</th>\n",
       "      <td>76.92</td>\n",
       "      <td>7762500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-15</th>\n",
       "      <td>74.83</td>\n",
       "      <td>9342100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-16</th>\n",
       "      <td>73.54</td>\n",
       "      <td>9636900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-17</th>\n",
       "      <td>72.10</td>\n",
       "      <td>10016800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-18</th>\n",
       "      <td>71.46</td>\n",
       "      <td>8986000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7835 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Close    Volume\n",
       "Date                       \n",
       "1977-01-03  80.00   1085400\n",
       "1977-01-04  79.00   3018600\n",
       "1977-01-05  78.38   2246400\n",
       "1977-01-06  78.75   1004400\n",
       "1977-01-07  78.50    950400\n",
       "...           ...       ...\n",
       "2008-01-14  76.92   7762500\n",
       "2008-01-15  74.83   9342100\n",
       "2008-01-16  73.54   9636900\n",
       "2008-01-17  72.10  10016800\n",
       "2008-01-18  71.46   8986000\n",
       "\n",
       "[7835 rows x 2 columns]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example1 = PEP.iloc[:, 3:5]\n",
    "print(example1.shape)\n",
    "example1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cb534e0-33e8-495a-b654-57c650631747",
   "metadata": {},
   "source": [
    "#### Method 2: loc\n",
    "Lets look at `.loc[]` next. This is a special indexing method, whereby we can slice by column/row name, or even by boolean indexing.\n",
    "\n",
    "Lets try to get similar data to above, but using loc instead.\n",
    "- Rows are `1997-01-03` to `1997-01-14`\n",
    "- Columns are `Close` and `Volume`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "ca8e39b5-b3ac-42a9-a91f-51208a2e3f94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1997-01-02</th>\n",
       "      <td>29.50</td>\n",
       "      <td>4237700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-03</th>\n",
       "      <td>29.75</td>\n",
       "      <td>2795500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-06</th>\n",
       "      <td>29.25</td>\n",
       "      <td>5361400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-07</th>\n",
       "      <td>29.50</td>\n",
       "      <td>4311600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-08</th>\n",
       "      <td>29.62</td>\n",
       "      <td>3346800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-09</th>\n",
       "      <td>29.25</td>\n",
       "      <td>4212900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-10</th>\n",
       "      <td>29.12</td>\n",
       "      <td>5606200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-13</th>\n",
       "      <td>29.50</td>\n",
       "      <td>3874100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-14</th>\n",
       "      <td>29.75</td>\n",
       "      <td>3561100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Close   Volume\n",
       "Date                      \n",
       "1997-01-02  29.50  4237700\n",
       "1997-01-03  29.75  2795500\n",
       "1997-01-06  29.25  5361400\n",
       "1997-01-07  29.50  4311600\n",
       "1997-01-08  29.62  3346800\n",
       "1997-01-09  29.25  4212900\n",
       "1997-01-10  29.12  5606200\n",
       "1997-01-13  29.50  3874100\n",
       "1997-01-14  29.75  3561100"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using integers\n",
    "example2 = PEP.loc['1997-01-01':'1997-01-14', 'Close':'Volume']\n",
    "example2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b15cf5f5-1cd1-4b92-a35c-789447e201ad",
   "metadata": {},
   "source": [
    "Want more specific columns? We can store column names and put them in an array, before using them in loc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "1c4cd9d1-8eb1-4b31-bffd-e4c51fd0a025",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>Open</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1997-01-02</th>\n",
       "      <td>29.50</td>\n",
       "      <td>30.00</td>\n",
       "      <td>4237700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-03</th>\n",
       "      <td>29.75</td>\n",
       "      <td>29.75</td>\n",
       "      <td>2795500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-06</th>\n",
       "      <td>29.25</td>\n",
       "      <td>29.88</td>\n",
       "      <td>5361400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-07</th>\n",
       "      <td>29.50</td>\n",
       "      <td>29.12</td>\n",
       "      <td>4311600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-08</th>\n",
       "      <td>29.62</td>\n",
       "      <td>29.62</td>\n",
       "      <td>3346800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-09</th>\n",
       "      <td>29.25</td>\n",
       "      <td>29.62</td>\n",
       "      <td>4212900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-10</th>\n",
       "      <td>29.12</td>\n",
       "      <td>28.88</td>\n",
       "      <td>5606200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-13</th>\n",
       "      <td>29.50</td>\n",
       "      <td>29.12</td>\n",
       "      <td>3874100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-14</th>\n",
       "      <td>29.75</td>\n",
       "      <td>29.50</td>\n",
       "      <td>3561100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Close   Open   Volume\n",
       "Date                             \n",
       "1997-01-02  29.50  30.00  4237700\n",
       "1997-01-03  29.75  29.75  2795500\n",
       "1997-01-06  29.25  29.88  5361400\n",
       "1997-01-07  29.50  29.12  4311600\n",
       "1997-01-08  29.62  29.62  3346800\n",
       "1997-01-09  29.25  29.62  4212900\n",
       "1997-01-10  29.12  28.88  5606200\n",
       "1997-01-13  29.50  29.12  3874100\n",
       "1997-01-14  29.75  29.50  3561100"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colnames = ['Close', 'Open', 'Volume']\n",
    "example2 = PEP.loc['1997-01-01':'1997-01-14', colnames]\n",
    "example2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac13eee7-cb45-48c0-8475-3a46b4ee3d13",
   "metadata": {},
   "source": [
    "#### Boolean Indexing\n",
    "Sometimes, a small segment of data is great. But if you really want to get specific, we can use booleans in `.loc[]` to get VERY specific data, instead of just getting attributes via rows and columns.\n",
    "\n",
    "For example, lets first get the data for the year 2000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "57e0cae3-0e64-4e86-a78d-2b64cda3d7fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-01-03</th>\n",
       "      <td>36.88</td>\n",
       "      <td>5298400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-04</th>\n",
       "      <td>35.94</td>\n",
       "      <td>4555700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-05</th>\n",
       "      <td>35.06</td>\n",
       "      <td>5701900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-06</th>\n",
       "      <td>36.62</td>\n",
       "      <td>6863100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-07</th>\n",
       "      <td>37.62</td>\n",
       "      <td>10527800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-06</th>\n",
       "      <td>45.31</td>\n",
       "      <td>7443800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-07</th>\n",
       "      <td>46.94</td>\n",
       "      <td>6975800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-08</th>\n",
       "      <td>47.44</td>\n",
       "      <td>5436900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-11</th>\n",
       "      <td>47.06</td>\n",
       "      <td>4171700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-12</th>\n",
       "      <td>47.38</td>\n",
       "      <td>2542100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>240 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Close    Volume\n",
       "Date                       \n",
       "2000-01-03  36.88   5298400\n",
       "2000-01-04  35.94   4555700\n",
       "2000-01-05  35.06   5701900\n",
       "2000-01-06  36.62   6863100\n",
       "2000-01-07  37.62  10527800\n",
       "...           ...       ...\n",
       "2000-12-06  45.31   7443800\n",
       "2000-12-07  46.94   6975800\n",
       "2000-12-08  47.44   5436900\n",
       "2000-12-11  47.06   4171700\n",
       "2000-12-12  47.38   2542100\n",
       "\n",
       "[240 rows x 2 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = PEP.loc['2000-01-01':'2000-12-12', ['Close', 'Volume']]\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c12e11f-739c-4d19-aad3-81f5519206a9",
   "metadata": {},
   "source": [
    "We want to know the closing prices for the PEP stock on days whereby there is high volume. This is where boolean indexing comes in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "245173f8-8db8-47c2-8342-c705e10bdf77",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-01-06</th>\n",
       "      <td>36.62</td>\n",
       "      <td>6863100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-07</th>\n",
       "      <td>37.62</td>\n",
       "      <td>10527800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-10</th>\n",
       "      <td>36.88</td>\n",
       "      <td>6402500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-09</th>\n",
       "      <td>34.00</td>\n",
       "      <td>8432500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-03-07</th>\n",
       "      <td>30.62</td>\n",
       "      <td>7053200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Close    Volume\n",
       "Date                       \n",
       "2000-01-06  36.62   6863100\n",
       "2000-01-07  37.62  10527800\n",
       "2000-01-10  36.88   6402500\n",
       "2000-02-09  34.00   8432500\n",
       "2000-03-07  30.62   7053200"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# By leaving columns only with a ':', we get all columns in the data\n",
    "# As long as it fits our boolean\n",
    "booleaned = data.loc[data['Volume'] > 6000000, :]\n",
    "booleaned.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d43c9274-ca61-46f0-88f8-a24fffec7cea",
   "metadata": {},
   "source": [
    "## Common Operations that I think will help you with projects\n",
    "Quick hacks that only you would know if you came here today :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07c44e97-8db3-49bb-bc04-820fe7bbd79d",
   "metadata": {},
   "source": [
    "#### 1. mean\n",
    "`.mean(axis)`\n",
    "- axis=1 implies that we are finding the mean of a row\n",
    "- axis=0 implies that we are finding the mean of a column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "7b0dc4fb-4a49-45c4-94b7-9bd16cef9edf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2000-01-06    3431568.310\n",
       "2000-01-07    5263918.810\n",
       "2000-01-10    3201268.440\n",
       "2000-02-09    4216267.000\n",
       "2000-03-07    3526615.310\n",
       "2000-03-30    3425467.970\n",
       "2000-05-10    4392219.595\n",
       "2000-05-22    3212819.875\n",
       "2000-06-13    3309921.405\n",
       "2000-06-16    3022421.875\n",
       "2000-06-30    4129072.220\n",
       "2000-07-13    3081020.190\n",
       "2000-08-16    3338022.000\n",
       "2000-11-15    3445773.280\n",
       "2000-11-17    3106571.970\n",
       "2000-11-20    3324172.500\n",
       "2000-11-21    3136073.250\n",
       "2000-12-01    6050621.190\n",
       "2000-12-04    6186471.905\n",
       "2000-12-05    3309272.095\n",
       "2000-12-06    3721922.655\n",
       "2000-12-07    3487923.470\n",
       "dtype: float64"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mean by row\n",
    "data.mean(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9b496a3-9c84-41f3-9254-e312d8869b8a",
   "metadata": {},
   "source": [
    "Now lets find the mean by columns. In other words, this is the average price of PEP stock and average volume in the year 2000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "8e68497a-17cf-411a-a47e-64184a1607dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Close     4.061946e+01\n",
       "Volume    4.023941e+06\n",
       "dtype: float64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mean by column\n",
    "data.mean(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a5f10e6-9bd2-4035-b0b7-cf543e0fd416",
   "metadata": {},
   "source": [
    "#### 2. standard deviation\n",
    "`.std(axis)`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "147b1ec0-32d8-4734-a581-20c1ff3b2b02",
   "metadata": {},
   "source": [
    "Here we can get the standard deviation of a stock by columns. The standard deviation of a closing price also indicates its volatility!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "67d92ba3-1cad-4a94-ac3d-99b4bf0628ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Close     5.069496e+00\n",
       "Volume    1.613522e+06\n",
       "dtype: float64"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.std(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d9e8b75-279a-4a91-ad7f-9abf0a1c04d0",
   "metadata": {},
   "source": [
    "#### 3. Correlation Coefficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "d517cfe1-7449-4f03-b004-d61b1d6dc639",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Close</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.088497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Volume</th>\n",
       "      <td>-0.088497</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Close    Volume\n",
       "Close   1.000000 -0.088497\n",
       "Volume -0.088497  1.000000"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f11ce21d-a8f4-4380-ad39-8ed5f4ce18e1",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 4. Describing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "67b2c8e9-4610-4e1e-84c5-b77e6bfc4be4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>240.000000</td>\n",
       "      <td>2.400000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>40.619458</td>\n",
       "      <td>4.023941e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>5.069496</td>\n",
       "      <td>1.613522e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>30.500000</td>\n",
       "      <td>1.400100e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>36.250000</td>\n",
       "      <td>2.987375e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>41.955000</td>\n",
       "      <td>3.734950e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>44.637500</td>\n",
       "      <td>4.654575e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>49.120000</td>\n",
       "      <td>1.237290e+07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Close        Volume\n",
       "count  240.000000  2.400000e+02\n",
       "mean    40.619458  4.023941e+06\n",
       "std      5.069496  1.613522e+06\n",
       "min     30.500000  1.400100e+06\n",
       "25%     36.250000  2.987375e+06\n",
       "50%     41.955000  3.734950e+06\n",
       "75%     44.637500  4.654575e+06\n",
       "max     49.120000  1.237290e+07"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01553c4d-6d41-47ec-a651-0b12ebc38818",
   "metadata": {},
   "source": [
    "#### 5. Deleting a column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "82684a33-3273-41b3-9bfa-ef4a5d04edb2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-01-03</th>\n",
       "      <td>36.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-04</th>\n",
       "      <td>35.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-05</th>\n",
       "      <td>35.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-06</th>\n",
       "      <td>36.62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-07</th>\n",
       "      <td>37.62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-06</th>\n",
       "      <td>45.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-07</th>\n",
       "      <td>46.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-08</th>\n",
       "      <td>47.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-11</th>\n",
       "      <td>47.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-12</th>\n",
       "      <td>47.38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>240 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Close\n",
       "Date             \n",
       "2000-01-03  36.88\n",
       "2000-01-04  35.94\n",
       "2000-01-05  35.06\n",
       "2000-01-06  36.62\n",
       "2000-01-07  37.62\n",
       "...           ...\n",
       "2000-12-06  45.31\n",
       "2000-12-07  46.94\n",
       "2000-12-08  47.44\n",
       "2000-12-11  47.06\n",
       "2000-12-12  47.38\n",
       "\n",
       "[240 rows x 1 columns]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.drop(columns=['Volume'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03295f21-31f8-4338-9f12-160c38af6b1c",
   "metadata": {},
   "source": [
    "# END.\n",
    "We hope that you've taken away some new knowledge today! To learn more about pandas, refer to the pandas documentation:\n",
    "https://pandas.pydata.org/docs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e92ea028-b214-4c8a-a97a-6a0a856aa063",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}