lectures/06_column-transformer-text-feats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 6: `sklearn` `ColumnTransformer` and Text Features\n",
    "\n",
    "UBC 2021-22\n",
    "\n",
    "Instructor: Giulia Toti"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import HTML\n",
    "\n",
    "sys.path.append(\"code/.\")\n",
    "from plotting_functions import *\n",
    "from utils import *\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 200)\n",
    "\n",
    "from sklearn.compose import ColumnTransformer, make_column_transformer\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.model_selection import cross_val_score, cross_validate, train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lecture plan for today\n",
    "\n",
    "- Announcements (~2 mins)\n",
    "- Exercises on pre-watched video 6.1 (~15 mins) \n",
    "- Live explanation: More on `ColumnTransformer` (~20 mins) \n",
    "- Break (~5 mins)\n",
    "- Live explanation: ColumnTransformer on California housing dataset (~10 mins)\n",
    "- Exercises on pre-watched video 6.2 (~15 mins)\n",
    "- Summary and wrap up (~2 mins)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Announcements\n",
    "\n",
    "- HW3 is out and due next Wednesday, February 2nd at 11:59 PM. Please review the important update regarding this homework on [Piazza](https://piazza.com/class/ky0j51i4ud64t5?cid=110). **You are no longer allowed to submit in pairs for this homework.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "## Learning outcomes \n",
    "\n",
    "From this lecture, you will be able to \n",
    "\n",
    "- use `ColumnTransformer` to build all our transformations together into one object and use it with `sklearn` pipelines;  \n",
    "- define `ColumnTransformer` where transformers contain more than one steps;\n",
    "- explain `handle_unknown=\"ignore\"` hyperparameter of `scikit-learn`'s `OneHotEncoder`;\n",
    "- explain `drop=\"if_binary\"` argument of `OneHotEncoder`;\n",
    "- identify when it's appropriate to apply ordinal encoding vs one-hot encoding;\n",
    "- explain strategies to deal with categorical variables with too many categories; \n",
    "- explain why text data needs a different treatment than categorical variables;\n",
    "- use `scikit-learn`'s `CountVectorizer` to encode text data;\n",
    "- explain different hyperparameters of `CountVectorizer`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## sklearn's [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- In most applications, some features are categorical, some are continuous, some are binary, and some are ordinal. \n",
    "\n",
    "- When we want to develop supervised machine learning pipelines on real-world datasets, very often we want to apply different transformation on different columns. \n",
    "\n",
    "- Enter `sklearn`'s `ColumnTransformer`!! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "- Let's look at a toy example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>quiz2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Good</td>\n",
       "      <td>5</td>\n",
       "      <td>70</td>\n",
       "      <td>73.0</td>\n",
       "      <td>68</td>\n",
       "      <td>74</td>\n",
       "      <td>71</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>80</td>\n",
       "      <td>88.0</td>\n",
       "      <td>89</td>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>93.0</td>\n",
       "      <td>69</td>\n",
       "      <td>79</td>\n",
       "      <td>75</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>97</td>\n",
       "      <td>90.0</td>\n",
       "      <td>94</td>\n",
       "      <td>82</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Average</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>82.0</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>85</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>3</td>\n",
       "      <td>98</td>\n",
       "      <td>86.0</td>\n",
       "      <td>95</td>\n",
       "      <td>95</td>\n",
       "      <td>78</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Physics</td>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "      <td>95</td>\n",
       "      <td>88.0</td>\n",
       "      <td>93</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Physics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>2</td>\n",
       "      <td>98</td>\n",
       "      <td>96.0</td>\n",
       "      <td>96</td>\n",
       "      <td>99</td>\n",
       "      <td>100</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>94.0</td>\n",
       "      <td>96</td>\n",
       "      <td>95</td>\n",
       "      <td>100</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>90.0</td>\n",
       "      <td>93</td>\n",
       "      <td>95</td>\n",
       "      <td>70</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Good</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>85.0</td>\n",
       "      <td>67</td>\n",
       "      <td>94</td>\n",
       "      <td>92</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Average</td>\n",
       "      <td>5</td>\n",
       "      <td>75</td>\n",
       "      <td>91.0</td>\n",
       "      <td>93</td>\n",
       "      <td>86</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Average</td>\n",
       "      <td>3</td>\n",
       "      <td>86</td>\n",
       "      <td>89.0</td>\n",
       "      <td>65</td>\n",
       "      <td>86</td>\n",
       "      <td>87</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Biology</td>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>90</td>\n",
       "      <td>88</td>\n",
       "      <td>82</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Poor</td>\n",
       "      <td>2</td>\n",
       "      <td>77</td>\n",
       "      <td>94.0</td>\n",
       "      <td>87</td>\n",
       "      <td>81</td>\n",
       "      <td>89</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>96</td>\n",
       "      <td>92.0</td>\n",
       "      <td>92</td>\n",
       "      <td>96</td>\n",
       "      <td>87</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0           yes              1        Computer Science        Excellent   \n",
       "1           yes              1  Mechanical Engineering          Average   \n",
       "2           yes              0             Mathematics             Poor   \n",
       "3            no              0             Mathematics        Excellent   \n",
       "4           yes              0              Psychology             Good   \n",
       "5            no              1               Economics             Good   \n",
       "6           yes              1        Computer Science        Excellent   \n",
       "7            no              0  Mechanical Engineering             Poor   \n",
       "8            no              0             Linguistics          Average   \n",
       "9           yes              1             Mathematics          Average   \n",
       "10          yes              0              Psychology             Good   \n",
       "11          yes              1                 Physics          Average   \n",
       "12          yes              1                 Physics        Excellent   \n",
       "13          yes              0  Mechanical Engineering        Excellent   \n",
       "14           no              0             Mathematics             Poor   \n",
       "15           no              1        Computer Science             Good   \n",
       "16          yes              0        Computer Science          Average   \n",
       "17          yes              1               Economics          Average   \n",
       "18           no              1                 Biology             Good   \n",
       "19           no              0              Psychology             Poor   \n",
       "20          yes              1             Linguistics        Excellent   \n",
       "\n",
       "    university_years  lab1  lab2  lab3  lab4  quiz1   quiz2  \n",
       "0                  3    92  93.0    84    91     92      A+  \n",
       "1                  2    94  90.0    80    83     91  not A+  \n",
       "2                  3    78  85.0    83    80     80  not A+  \n",
       "3                  3    91   NaN    92    91     89      A+  \n",
       "4                  4    77  83.0    90    92     85      A+  \n",
       "5                  5    70  73.0    68    74     71  not A+  \n",
       "6                  4    80  88.0    89    88     91      A+  \n",
       "7                  3    95  93.0    69    79     75  not A+  \n",
       "8                  2    97  90.0    94    82     80  not A+  \n",
       "9                  4    95  82.0    94    94     85  not A+  \n",
       "10                 3    98  86.0    95    95     78      A+  \n",
       "11                 1    95  88.0    93    92     85      A+  \n",
       "12                 2    98  96.0    96    99    100      A+  \n",
       "13                 4    95  94.0    96    95    100      A+  \n",
       "14                 3    95  90.0    93    95     70  not A+  \n",
       "15                 3    92  85.0    67    94     92  not A+  \n",
       "16                 5    75  91.0    93    86     85      A+  \n",
       "17                 3    86  89.0    65    86     87  not A+  \n",
       "18                 2    91   NaN    90    88     82  not A+  \n",
       "19                 2    77  94.0    87    81     89  not A+  \n",
       "20                 4    96  92.0    92    96     87      A+  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/quiz2-grade-toy-col-transformer.csv\")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 21 entries, 0 to 20\n",
      "Data columns (total 11 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   enjoy_course      21 non-null     object \n",
      " 1   ml_experience     21 non-null     int64  \n",
      " 2   major             21 non-null     object \n",
      " 3   class_attendance  21 non-null     object \n",
      " 4   university_years  21 non-null     int64  \n",
      " 5   lab1              21 non-null     int64  \n",
      " 6   lab2              19 non-null     float64\n",
      " 7   lab3              21 non-null     int64  \n",
      " 8   lab4              21 non-null     int64  \n",
      " 9   quiz1             21 non-null     int64  \n",
      " 10  quiz2             21 non-null     object \n",
      "dtypes: float64(1), int64(6), object(4)\n",
      "memory usage: 1.9+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Transformations on the toy data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>quiz2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1   quiz2  \n",
       "0                 3    92  93.0    84    91     92      A+  \n",
       "1                 2    94  90.0    80    83     91  not A+  \n",
       "2                 3    78  85.0    83    80     80  not A+  \n",
       "3                 3    91   NaN    92    91     89      A+  \n",
       "4                 4    77  83.0    90    92     85      A+  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Scaling on numeric features\n",
    "- One-hot encoding on the categorical feature `major` and binary feature `enjoy_class`\n",
    "- Ordinal encoding on the ordinal feature `class_attendance`\n",
    "- Imputation on the `lab2` feature\n",
    "- None on the `ml_experience` feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `ColumnTransformer` example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['enjoy_course', 'ml_experience', 'major', 'class_attendance',\n",
       "       'university_years', 'lab1', 'lab2', 'lab3', 'lab4', 'quiz1'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df.drop(columns=[\"quiz2\"])\n",
    "y = df[\"quiz2\"]\n",
    "X.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Identify the transformations we want to apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_feats = [\"university_years\", \"lab1\", \"lab3\", \"lab4\", \"quiz1\"]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\n",
    "    \"lab2\",\n",
    "    \"class_attendance\",\n",
    "    \"enjoy_course\",\n",
    "]  # do not include these features in modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, let's only focus on scaling and one-hot encoding first. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Create a column transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Each transformation is specified by a name, a transformer object, and the columns this transformer should be applied to. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct = ColumnTransformer(\n",
    "    [\n",
    "        (\"scaling\", StandardScaler(), numeric_feats),\n",
    "        (\"onehot\", OneHotEncoder(sparse=False), categorical_feats),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Convenient `make_column_transformer` syntax\n",
    "\n",
    "- Similar to `make_pipeline` syntax, there is convenient `make_column_transformer` syntax. \n",
    "- The syntax automatically names each step based on its class. \n",
    "- We'll be mostly using this syntax. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "ct = make_column_transformer(\n",
    "    (StandardScaler(), numeric_feats),  # scaling on numeric features\n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ColumnTransformer(transformers=[('standardscaler', StandardScaler(),\n",
       "                                 ['university_years', 'lab1', 'lab3', 'lab4',\n",
       "                                  'quiz1']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop',\n",
       "                                 ['lab2', 'class_attendance', 'enjoy_course'])])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "source": [
    "- When we `fit_transform`, each transformer is applied to the specified columns and the result of the transformations are concatenated horizontally. \n",
    "- A big advantage here is that we build all our transformations together into one object, and that way we're sure we do the same operations to all splits of the data.\n",
    "- Otherwise we might, for example, do the OHE on both train and test but forget to scale the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Let's examine the transformed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(transformed[:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.09345386,  0.3589134 , -0.21733442,  0.36269995,  0.84002795,\n",
       "         0.        ,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [-1.07471942,  0.59082668, -0.61420598, -0.85597188,  0.71219761,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [-0.09345386, -1.26447953, -0.31655231, -1.31297381, -0.69393613,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  1.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386,  0.24295676,  0.57640869,  0.36269995,  0.45653693,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  1.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 0.8878117 , -1.38043616,  0.37797291,  0.51503393, -0.05478443,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  1.        ,  0.        ],\n",
       "       [ 1.86907725, -2.19213263, -1.80482065, -2.22697768, -1.84440919,\n",
       "         0.        ,  0.        ,  1.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [ 0.8878117 , -1.03256625,  0.27875502, -0.09430199,  0.71219761,\n",
       "         0.        ,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [-0.09345386,  0.70678332, -1.70560276, -1.46530779, -1.33308783,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-1.07471942,  0.93869659,  0.77484447, -1.00830586, -0.69393613,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 0.8878117 ,  0.70678332,  0.77484447,  0.81970188, -0.05478443,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  1.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [-0.09345386,  1.05465323,  0.87406235,  0.97203586, -0.94959681,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  1.        ,  0.        ],\n",
       "       [-2.05598498,  0.70678332,  0.67562658,  0.51503393, -0.05478443,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  1.        ,  0.        ,  1.        ],\n",
       "       [-1.07471942,  1.05465323,  0.97328024,  1.58137177,  1.86267067,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  1.        ,  0.        ,  1.        ],\n",
       "       [ 0.8878117 ,  0.70678332,  0.97328024,  0.97203586,  1.86267067,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386,  0.70678332,  0.67562658,  0.97203586, -1.97223953,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  1.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386,  0.3589134 , -1.90403853,  0.81970188,  0.84002795,\n",
       "         0.        ,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [ 1.86907725, -1.61234944,  0.67562658, -0.39896994, -0.05478443,\n",
       "         0.        ,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386, -0.33682642, -2.10247431, -0.39896994,  0.20087625,\n",
       "         0.        ,  0.        ,  1.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [-1.07471942,  0.24295676,  0.37797291, -0.09430199, -0.43827545,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [-1.07471942, -1.38043616,  0.08031924, -1.16063983,  0.45653693,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  1.        ,  0.        ],\n",
       "       [ 0.8878117 ,  0.82273995,  0.57640869,  1.12436984,  0.20087625,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "Note that the returned object is not a dataframe. So there are no column names. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Viewing the transformed data as a dataframe\n",
    "\n",
    "- How can we view our transformed data as a dataframe? \n",
    "- We are adding more columns. \n",
    "- So the original columns won't directly map to the transformed data. \n",
    "- Let's create column names for the transformed data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology',\n",
       " 'ml_experience']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    "    + passthrough_feats\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'standardscaler': StandardScaler(),\n",
       " 'onehotencoder': OneHotEncoder(),\n",
       " 'passthrough': 'passthrough',\n",
       " 'drop': 'drop'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct.named_transformers_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "Note that the order of the columns in the transformed data depends upon the order of the features we pass to the `ColumnTransformer` and can be different than the order of the features in the original dataframe.  \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "      <th>ml_experience</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-2.192133</td>\n",
       "      <td>-1.804821</td>\n",
       "      <td>-2.226978</td>\n",
       "      <td>-1.844409</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.032566</td>\n",
       "      <td>0.278755</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-1.705603</td>\n",
       "      <td>-1.465308</td>\n",
       "      <td>-1.333088</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.938697</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>-1.008306</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>0.874062</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-0.949597</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-2.055985</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>1.581372</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-1.972240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-1.904039</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-1.612349</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-0.336826</td>\n",
       "      <td>-2.102474</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>-0.438275</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>0.080319</td>\n",
       "      <td>-1.160640</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.822740</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>1.124370</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    university_years      lab1      lab3      lab4     quiz1  major_Biology  \\\n",
       "0          -0.093454  0.358913 -0.217334  0.362700  0.840028            0.0   \n",
       "1          -1.074719  0.590827 -0.614206 -0.855972  0.712198            0.0   \n",
       "2          -0.093454 -1.264480 -0.316552 -1.312974 -0.693936            0.0   \n",
       "3          -0.093454  0.242957  0.576409  0.362700  0.456537            0.0   \n",
       "4           0.887812 -1.380436  0.377973  0.515034 -0.054784            0.0   \n",
       "5           1.869077 -2.192133 -1.804821 -2.226978 -1.844409            0.0   \n",
       "6           0.887812 -1.032566  0.278755 -0.094302  0.712198            0.0   \n",
       "7          -0.093454  0.706783 -1.705603 -1.465308 -1.333088            0.0   \n",
       "8          -1.074719  0.938697  0.774844 -1.008306 -0.693936            0.0   \n",
       "9           0.887812  0.706783  0.774844  0.819702 -0.054784            0.0   \n",
       "10         -0.093454  1.054653  0.874062  0.972036 -0.949597            0.0   \n",
       "11         -2.055985  0.706783  0.675627  0.515034 -0.054784            0.0   \n",
       "12         -1.074719  1.054653  0.973280  1.581372  1.862671            0.0   \n",
       "13          0.887812  0.706783  0.973280  0.972036  1.862671            0.0   \n",
       "14         -0.093454  0.706783  0.675627  0.972036 -1.972240            0.0   \n",
       "15         -0.093454  0.358913 -1.904039  0.819702  0.840028            0.0   \n",
       "16          1.869077 -1.612349  0.675627 -0.398970 -0.054784            0.0   \n",
       "17         -0.093454 -0.336826 -2.102474 -0.398970  0.200876            0.0   \n",
       "18         -1.074719  0.242957  0.377973 -0.094302 -0.438275            1.0   \n",
       "19         -1.074719 -1.380436  0.080319 -1.160640  0.456537            0.0   \n",
       "20          0.887812  0.822740  0.576409  1.124370  0.200876            0.0   \n",
       "\n",
       "    major_Computer Science  major_Economics  major_Linguistics  \\\n",
       "0                      1.0              0.0                0.0   \n",
       "1                      0.0              0.0                0.0   \n",
       "2                      0.0              0.0                0.0   \n",
       "3                      0.0              0.0                0.0   \n",
       "4                      0.0              0.0                0.0   \n",
       "5                      0.0              1.0                0.0   \n",
       "6                      1.0              0.0                0.0   \n",
       "7                      0.0              0.0                0.0   \n",
       "8                      0.0              0.0                1.0   \n",
       "9                      0.0              0.0                0.0   \n",
       "10                     0.0              0.0                0.0   \n",
       "11                     0.0              0.0                0.0   \n",
       "12                     0.0              0.0                0.0   \n",
       "13                     0.0              0.0                0.0   \n",
       "14                     0.0              0.0                0.0   \n",
       "15                     1.0              0.0                0.0   \n",
       "16                     1.0              0.0                0.0   \n",
       "17                     0.0              1.0                0.0   \n",
       "18                     0.0              0.0                0.0   \n",
       "19                     0.0              0.0                0.0   \n",
       "20                     0.0              0.0                1.0   \n",
       "\n",
       "    major_Mathematics  major_Mechanical Engineering  major_Physics  \\\n",
       "0                 0.0                           0.0            0.0   \n",
       "1                 0.0                           1.0            0.0   \n",
       "2                 1.0                           0.0            0.0   \n",
       "3                 1.0                           0.0            0.0   \n",
       "4                 0.0                           0.0            0.0   \n",
       "5                 0.0                           0.0            0.0   \n",
       "6                 0.0                           0.0            0.0   \n",
       "7                 0.0                           1.0            0.0   \n",
       "8                 0.0                           0.0            0.0   \n",
       "9                 1.0                           0.0            0.0   \n",
       "10                0.0                           0.0            0.0   \n",
       "11                0.0                           0.0            1.0   \n",
       "12                0.0                           0.0            1.0   \n",
       "13                0.0                           1.0            0.0   \n",
       "14                1.0                           0.0            0.0   \n",
       "15                0.0                           0.0            0.0   \n",
       "16                0.0                           0.0            0.0   \n",
       "17                0.0                           0.0            0.0   \n",
       "18                0.0                           0.0            0.0   \n",
       "19                0.0                           0.0            0.0   \n",
       "20                0.0                           0.0            0.0   \n",
       "\n",
       "    major_Psychology  ml_experience  \n",
       "0                0.0            1.0  \n",
       "1                0.0            1.0  \n",
       "2                0.0            0.0  \n",
       "3                0.0            0.0  \n",
       "4                1.0            0.0  \n",
       "5                0.0            1.0  \n",
       "6                0.0            1.0  \n",
       "7                0.0            0.0  \n",
       "8                0.0            0.0  \n",
       "9                0.0            1.0  \n",
       "10               1.0            0.0  \n",
       "11               0.0            1.0  \n",
       "12               0.0            1.0  \n",
       "13               0.0            0.0  \n",
       "14               0.0            0.0  \n",
       "15               0.0            1.0  \n",
       "16               0.0            0.0  \n",
       "17               0.0            1.0  \n",
       "18               0.0            1.0  \n",
       "19               1.0            0.0  \n",
       "20               0.0            1.0  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(transformed, columns=column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### `ColumnTransformer`: Transformed data\n",
    "\n",
    "<br>\n",
    "\n",
    "<img src='./img/column-transformer.png' width=\"1500\">\n",
    "\n",
    "[Adapted from here.](https://amueller.github.io/COMS4995-s20/slides/aml-04-preprocessing/#37)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Training models with transformed data\n",
    "- We can now pass the `ColumnTransformer` object as a step in a pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['A+', 'not A+', 'not A+', 'A+', 'A+', 'not A+', 'A+', 'not A+',\n",
       "       'not A+', 'A+', 'A+', 'A+', 'A+', 'A+', 'not A+', 'not A+', 'A+',\n",
       "       'not A+', 'not A+', 'not A+', 'A+'], dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe = make_pipeline(ct, SVC())\n",
    "pipe.fit(X, y)\n",
    "pipe.predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### True/False: `ColumnTransformer` \n",
    "\n",
    "1. You could carry out cross-validation by passing a `ColumnTransformer` object to `cross_validate`. \n",
    "2. After applying column transformer, the order of the columns in the transformed data has to be the same as the order of the columns in the original data. \n",
    "3. After applying a column transformer, the transformed data is always going to be of different shape than the original data. \n",
    "4. When you call `fit_transform` on a `ColumnTransformer` object, you get a numpy ndarray. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### What transformations on what columns? \n",
    "Consider the feature columns below. \n",
    "\n",
    "- What transformations would you apply on each column? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| colour  | location  |  shape |  water_content | weight |\n",
    "|-----|-----|-----|-----|-----|\n",
    "|   red   |   canada  |    NaN  |       84  |        100 |\n",
    "| yellow  |   mexico  |   long  |       75  |        120 |\n",
    "| orange  |   spain   |    NaN  |       90  |        NaN |\n",
    "| magenta |    china  |    round|       NaN |        600 |\n",
    "| purple  |  austria  |    NaN  |       80  |        115 |\n",
    "| purple  |  turkey   |   oval  |       78  |        340 |\n",
    "| green   |  mexico   |   oval  |       83  |        NaN |\n",
    "| blue    | canada     | round  |      73   |       535  |\n",
    "| brown   |  china     |   NaN  |       NaN |       1743 | \n",
    "| yellow  |  mexico    |  oval  |       83  |        265 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## More on feature transformations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Multiple transformations in a transformer\n",
    "\n",
    "- Recall that `lab2` has missing values. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Good</td>\n",
       "      <td>5</td>\n",
       "      <td>70</td>\n",
       "      <td>73.0</td>\n",
       "      <td>68</td>\n",
       "      <td>74</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>80</td>\n",
       "      <td>88.0</td>\n",
       "      <td>89</td>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>93.0</td>\n",
       "      <td>69</td>\n",
       "      <td>79</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>97</td>\n",
       "      <td>90.0</td>\n",
       "      <td>94</td>\n",
       "      <td>82</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Average</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>82.0</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "5           no              1               Economics             Good   \n",
       "6          yes              1        Computer Science        Excellent   \n",
       "7           no              0  Mechanical Engineering             Poor   \n",
       "8           no              0             Linguistics          Average   \n",
       "9          yes              1             Mathematics          Average   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  \n",
       "5                 5    70  73.0    68    74     71  \n",
       "6                 4    80  88.0    89    88     91  \n",
       "7                 3    95  93.0    69    79     75  \n",
       "8                 2    97  90.0    94    82     80  \n",
       "9                 4    95  82.0    94    94     85  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- So we would like to apply more than one transformations on it: imputation and scaling.  \n",
    "- We can treat `lab2` separately, but we can also include it into `numeric_feats` and apply both transformations on all numeric columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\"class_attendance\", \"enjoy_course\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- To apply more than one transformations we can define a pipeline inside a column transformer to chain different transformations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab2',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology',\n",
       " 'ml_experience']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    "    + passthrough_feats\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "      <th>ml_experience</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>-1.103439</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-2.192133</td>\n",
       "      <td>-3.100139</td>\n",
       "      <td>-1.804821</td>\n",
       "      <td>-2.226978</td>\n",
       "      <td>-1.844409</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.032566</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.278755</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-1.705603</td>\n",
       "      <td>-1.465308</td>\n",
       "      <td>-1.333088</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.938697</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>-1.008306</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-1.303109</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>-0.504429</td>\n",
       "      <td>0.874062</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-0.949597</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-2.055985</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>1.492270</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>1.581372</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-1.972240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-1.904039</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-1.612349</td>\n",
       "      <td>0.493921</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-0.336826</td>\n",
       "      <td>0.094581</td>\n",
       "      <td>-2.102474</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>-0.438275</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.080319</td>\n",
       "      <td>-1.160640</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.822740</td>\n",
       "      <td>0.693590</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>1.124370</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    university_years      lab1      lab2      lab3      lab4     quiz1  \\\n",
       "0          -0.093454  0.358913  0.893260 -0.217334  0.362700  0.840028   \n",
       "1          -1.074719  0.590827  0.294251 -0.614206 -0.855972  0.712198   \n",
       "2          -0.093454 -1.264480 -0.704099 -0.316552 -1.312974 -0.693936   \n",
       "3          -0.093454  0.242957  0.000000  0.576409  0.362700  0.456537   \n",
       "4           0.887812 -1.380436 -1.103439  0.377973  0.515034 -0.054784   \n",
       "5           1.869077 -2.192133 -3.100139 -1.804821 -2.226978 -1.844409   \n",
       "6           0.887812 -1.032566 -0.105089  0.278755 -0.094302  0.712198   \n",
       "7          -0.093454  0.706783  0.893260 -1.705603 -1.465308 -1.333088   \n",
       "8          -1.074719  0.938697  0.294251  0.774844 -1.008306 -0.693936   \n",
       "9           0.887812  0.706783 -1.303109  0.774844  0.819702 -0.054784   \n",
       "10         -0.093454  1.054653 -0.504429  0.874062  0.972036 -0.949597   \n",
       "11         -2.055985  0.706783 -0.105089  0.675627  0.515034 -0.054784   \n",
       "12         -1.074719  1.054653  1.492270  0.973280  1.581372  1.862671   \n",
       "13          0.887812  0.706783  1.092930  0.973280  0.972036  1.862671   \n",
       "14         -0.093454  0.706783  0.294251  0.675627  0.972036 -1.972240   \n",
       "15         -0.093454  0.358913 -0.704099 -1.904039  0.819702  0.840028   \n",
       "16          1.869077 -1.612349  0.493921  0.675627 -0.398970 -0.054784   \n",
       "17         -0.093454 -0.336826  0.094581 -2.102474 -0.398970  0.200876   \n",
       "18         -1.074719  0.242957  0.000000  0.377973 -0.094302 -0.438275   \n",
       "19         -1.074719 -1.380436  1.092930  0.080319 -1.160640  0.456537   \n",
       "20          0.887812  0.822740  0.693590  0.576409  1.124370  0.200876   \n",
       "\n",
       "    major_Biology  major_Computer Science  major_Economics  major_Linguistics  \\\n",
       "0             0.0                     1.0              0.0                0.0   \n",
       "1             0.0                     0.0              0.0                0.0   \n",
       "2             0.0                     0.0              0.0                0.0   \n",
       "3             0.0                     0.0              0.0                0.0   \n",
       "4             0.0                     0.0              0.0                0.0   \n",
       "5             0.0                     0.0              1.0                0.0   \n",
       "6             0.0                     1.0              0.0                0.0   \n",
       "7             0.0                     0.0              0.0                0.0   \n",
       "8             0.0                     0.0              0.0                1.0   \n",
       "9             0.0                     0.0              0.0                0.0   \n",
       "10            0.0                     0.0              0.0                0.0   \n",
       "11            0.0                     0.0              0.0                0.0   \n",
       "12            0.0                     0.0              0.0                0.0   \n",
       "13            0.0                     0.0              0.0                0.0   \n",
       "14            0.0                     0.0              0.0                0.0   \n",
       "15            0.0                     1.0              0.0                0.0   \n",
       "16            0.0                     1.0              0.0                0.0   \n",
       "17            0.0                     0.0              1.0                0.0   \n",
       "18            1.0                     0.0              0.0                0.0   \n",
       "19            0.0                     0.0              0.0                0.0   \n",
       "20            0.0                     0.0              0.0                1.0   \n",
       "\n",
       "    major_Mathematics  major_Mechanical Engineering  major_Physics  \\\n",
       "0                 0.0                           0.0            0.0   \n",
       "1                 0.0                           1.0            0.0   \n",
       "2                 1.0                           0.0            0.0   \n",
       "3                 1.0                           0.0            0.0   \n",
       "4                 0.0                           0.0            0.0   \n",
       "5                 0.0                           0.0            0.0   \n",
       "6                 0.0                           0.0            0.0   \n",
       "7                 0.0                           1.0            0.0   \n",
       "8                 0.0                           0.0            0.0   \n",
       "9                 1.0                           0.0            0.0   \n",
       "10                0.0                           0.0            0.0   \n",
       "11                0.0                           0.0            1.0   \n",
       "12                0.0                           0.0            1.0   \n",
       "13                0.0                           1.0            0.0   \n",
       "14                1.0                           0.0            0.0   \n",
       "15                0.0                           0.0            0.0   \n",
       "16                0.0                           0.0            0.0   \n",
       "17                0.0                           0.0            0.0   \n",
       "18                0.0                           0.0            0.0   \n",
       "19                0.0                           0.0            0.0   \n",
       "20                0.0                           0.0            0.0   \n",
       "\n",
       "    major_Psychology  ml_experience  \n",
       "0                0.0            1.0  \n",
       "1                0.0            1.0  \n",
       "2                0.0            0.0  \n",
       "3                0.0            0.0  \n",
       "4                1.0            0.0  \n",
       "5                0.0            1.0  \n",
       "6                0.0            1.0  \n",
       "7                0.0            0.0  \n",
       "8                0.0            0.0  \n",
       "9                0.0            1.0  \n",
       "10               1.0            0.0  \n",
       "11               0.0            1.0  \n",
       "12               0.0            1.0  \n",
       "13               0.0            0.0  \n",
       "14               0.0            0.0  \n",
       "15               0.0            1.0  \n",
       "16               0.0            0.0  \n",
       "17               0.0            1.0  \n",
       "18               0.0            1.0  \n",
       "19               1.0            0.0  \n",
       "20               0.0            1.0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_transformed, columns=column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `sklearn` `set_config`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- With multiple transformations in a column transformer, it can get tricky to keep track of everything happening inside it.  \n",
    "- We can use `set_config` to display a diagram of this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "\n",
    "set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 {color: black;background-color: white;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 pre{padding: 0;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-toggleable {background-color: white;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-estimator:hover {background-color: #d4ebff;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-item {z-index: 1;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-parallel-item:only-child::after {width: 0;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-a0023e81-bc61-45c3-abc9-4ba4f833a911 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-a0023e81-bc61-45c3-abc9-4ba4f833a911\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4af259c0-f8c6-455a-a264-4560c985da79\" type=\"checkbox\" ><label for=\"4af259c0-f8c6-455a-a264-4560c985da79\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b8ad48b2-7fba-4f7e-a5e2-e6aac6d316d5\" type=\"checkbox\" ><label for=\"b8ad48b2-7fba-4f7e-a5e2-e6aac6d316d5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"ddfb25c7-ec16-4a39-be8a-ef5210cce7b6\" type=\"checkbox\" ><label for=\"ddfb25c7-ec16-4a39-be8a-ef5210cce7b6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e5091da0-b4f0-40fb-979e-f1df562fd473\" type=\"checkbox\" ><label for=\"e5091da0-b4f0-40fb-979e-f1df562fd473\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4ae535f2-74ef-4040-bfcb-51e0a885831a\" type=\"checkbox\" ><label for=\"4ae535f2-74ef-4040-bfcb-51e0a885831a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"026f084c-a5ef-4f6c-88dd-d232c851e446\" type=\"checkbox\" ><label for=\"026f084c-a5ef-4f6c-88dd-d232c851e446\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8737a45e-375f-4dfa-bf8e-e2616943a009\" type=\"checkbox\" ><label for=\"8737a45e-375f-4dfa-bf8e-e2616943a009\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b4a86bab-3865-4f83-a635-7578e7888dbb\" type=\"checkbox\" ><label for=\"b4a86bab-3865-4f83-a635-7578e7888dbb\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"87961e89-5d08-464b-8087-ef4677714e05\" type=\"checkbox\" ><label for=\"87961e89-5d08-464b-8087-ef4677714e05\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"5e9f8da4-820e-4bea-88da-f55bcb2ad75b\" type=\"checkbox\" ><label for=\"5e9f8da4-820e-4bea-88da-f55bcb2ad75b\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop',\n",
       "                                 ['class_attendance', 'enjoy_course'])])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ColumnTransformer(transformers=[('pipeline',\n",
      "                                 Pipeline(steps=[('simpleimputer',\n",
      "                                                  SimpleImputer()),\n",
      "                                                 ('standardscaler',\n",
      "                                                  StandardScaler())]),\n",
      "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
      "                                  'lab4', 'quiz1']),\n",
      "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
      "                                ('passthrough', 'passthrough',\n",
      "                                 ['ml_experience']),\n",
      "                                ('drop', 'drop',\n",
      "                                 ['class_attendance', 'enjoy_course'])])\n"
     ]
    }
   ],
   "source": [
    "print(ct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Incorporating ordinal feature `class_attendance` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `class_attendance` column is different than the `major` column in that there is some ordering of the values. \n",
    "    - Excellent > Good > Average > poor    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try applying `OrdinalEncoder` on this column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_toy = X[[\"class_attendance\"]]\n",
    "enc = OrdinalEncoder()\n",
    "enc.fit(X_toy)\n",
    "X_toy_ord = enc.transform(X_toy)\n",
    "df = pd.DataFrame(\n",
    "    data=X_toy_ord,\n",
    "    columns=[\"class_attendance_enc\"],\n",
    "    index=X_toy.index,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>class_attendance_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Poor</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Good</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Good</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Poor</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  class_attendance  class_attendance_enc\n",
       "0        Excellent                   1.0\n",
       "1          Average                   0.0\n",
       "2             Poor                   3.0\n",
       "3        Excellent                   1.0\n",
       "4             Good                   2.0\n",
       "5             Good                   2.0\n",
       "6        Excellent                   1.0\n",
       "7             Poor                   3.0\n",
       "8          Average                   0.0\n",
       "9          Average                   0.0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([X_toy, df], axis=1).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What's the problem here? \n",
    "    - The encoder doesn't know the order. \n",
    "- We can examine unique categories manually, order them based on our intuitions, and then provide this human knowledge to the transformer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What are the unique categories of `class_attendance`? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Excellent', 'Average', 'Poor', 'Good'], dtype=object)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_toy[\"class_attendance\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's order them manually. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_attendance_levels = [\"Poor\", \"Average\", \"Good\", \"Excellent\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "Note that if you use the reverse order of the categories, it wouldn't matter. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's make sure that we have included all categories in our manual ordering.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert set(class_attendance_levels) == set(X_toy[\"class_attendance\"].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array(['Poor', 'Average', 'Good', 'Excellent'], dtype=object)]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>class_attendance_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Poor</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Poor</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  class_attendance  class_attendance_enc\n",
       "0        Excellent                     3\n",
       "1          Average                     1\n",
       "2             Poor                     0\n",
       "3        Excellent                     3\n",
       "4             Good                     2\n",
       "5             Good                     2\n",
       "6        Excellent                     3\n",
       "7             Poor                     0\n",
       "8          Average                     1\n",
       "9          Average                     1"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "oe = OrdinalEncoder(categories=[class_attendance_levels], dtype=int)\n",
    "oe.fit(X_toy[[\"class_attendance\"]])\n",
    "ca_transformed = oe.transform(X_toy[[\"class_attendance\"]])\n",
    "df = pd.DataFrame(\n",
    "    data=ca_transformed, columns=[\"class_attendance_enc\"], index=X_toy.index\n",
    ")\n",
    "print(oe.categories_)\n",
    "pd.concat([X_toy, df], axis=1).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The encoded categories are looking better now! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### More than one ordinal columns?\n",
    "\n",
    "- We can pass the manually ordered categories when we create an `OrdinalEncoder` object as a list of lists. \n",
    "- If you have more than one ordinal columns\n",
    "    - manually create a list of ordered categories for each column\n",
    "    - pass a list of lists to `OrdinalEncoder`, where each inner list corresponds to manually created list of ordered categories for a corresponding ordinal column. \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let's incorporate ordinal encoding of `class_attendance` in our column transformer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "ordinal_feats = [\"class_attendance\"]  # apply ordinal encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\"enjoy_course\"]  # do not include these features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 {color: black;background-color: white;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 pre{padding: 0;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-toggleable {background-color: white;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-estimator:hover {background-color: #d4ebff;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-item {z-index: 1;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-parallel-item:only-child::after {width: 0;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-f5eb7815-b238-4dfc-a49e-1705146c2a00 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-f5eb7815-b238-4dfc-a49e-1705146c2a00\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d614c7d1-7a9c-46b6-8814-c04592a71ad4\" type=\"checkbox\" ><label for=\"d614c7d1-7a9c-46b6-8814-c04592a71ad4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"66767f47-51d9-45e1-a794-29583c9d9baf\" type=\"checkbox\" ><label for=\"66767f47-51d9-45e1-a794-29583c9d9baf\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"cd1745a2-d9d4-4b51-b5f2-8076af162f7d\" type=\"checkbox\" ><label for=\"cd1745a2-d9d4-4b51-b5f2-8076af162f7d\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"068632df-8258-41c4-aad1-ff41a80b74ba\" type=\"checkbox\" ><label for=\"068632df-8258-41c4-aad1-ff41a80b74ba\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"db775ba8-ca81-4f99-8537-4aa0c32f3ee4\" type=\"checkbox\" ><label for=\"db775ba8-ca81-4f99-8537-4aa0c32f3ee4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"10b65dc0-1277-48e0-b03d-fa7a2b8a3407\" type=\"checkbox\" ><label for=\"10b65dc0-1277-48e0-b03d-fa7a2b8a3407\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"835cdfb8-77e5-4dff-bf03-2b5043003494\" type=\"checkbox\" ><label for=\"835cdfb8-77e5-4dff-bf03-2b5043003494\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"9501d653-1f59-48d6-8af5-b7ebe249ba6d\" type=\"checkbox\" ><label for=\"9501d653-1f59-48d6-8af5-b7ebe249ba6d\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"c230806a-1daa-4131-9d0d-ca96cbe15f23\" type=\"checkbox\" ><label for=\"c230806a-1daa-4131-9d0d-ca96cbe15f23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d6ae229f-4ad2-45dc-b43c-95666eae57b7\" type=\"checkbox\" ><label for=\"d6ae229f-4ad2-45dc-b43c-95666eae57b7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"f8ae3bb3-1128-4196-a85a-d25381cbf28a\" type=\"checkbox\" ><label for=\"f8ae3bb3-1128-4196-a85a-d25381cbf28a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"482869bd-5f64-42aa-83ba-20ce019b0126\" type=\"checkbox\" ><label for=\"482869bd-5f64-42aa-83ba-20ce019b0126\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop', ['enjoy_course'])])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X_transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab2',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology',\n",
       " 'class_attendance',\n",
       " 'ml_experience']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    "    + ordinal_feats\n",
    "    + passthrough_feats\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>ml_experience</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>-1.103439</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-2.192133</td>\n",
       "      <td>-3.100139</td>\n",
       "      <td>-1.804821</td>\n",
       "      <td>-2.226978</td>\n",
       "      <td>-1.844409</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.032566</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.278755</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-1.705603</td>\n",
       "      <td>-1.465308</td>\n",
       "      <td>-1.333088</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.938697</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>-1.008306</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-1.303109</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>-0.504429</td>\n",
       "      <td>0.874062</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-0.949597</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-2.055985</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>1.492270</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>1.581372</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-1.972240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-1.904039</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-1.612349</td>\n",
       "      <td>0.493921</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-0.336826</td>\n",
       "      <td>0.094581</td>\n",
       "      <td>-2.102474</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>-0.438275</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.080319</td>\n",
       "      <td>-1.160640</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.822740</td>\n",
       "      <td>0.693590</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>1.124370</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    university_years      lab1      lab2      lab3      lab4     quiz1  \\\n",
       "0          -0.093454  0.358913  0.893260 -0.217334  0.362700  0.840028   \n",
       "1          -1.074719  0.590827  0.294251 -0.614206 -0.855972  0.712198   \n",
       "2          -0.093454 -1.264480 -0.704099 -0.316552 -1.312974 -0.693936   \n",
       "3          -0.093454  0.242957  0.000000  0.576409  0.362700  0.456537   \n",
       "4           0.887812 -1.380436 -1.103439  0.377973  0.515034 -0.054784   \n",
       "5           1.869077 -2.192133 -3.100139 -1.804821 -2.226978 -1.844409   \n",
       "6           0.887812 -1.032566 -0.105089  0.278755 -0.094302  0.712198   \n",
       "7          -0.093454  0.706783  0.893260 -1.705603 -1.465308 -1.333088   \n",
       "8          -1.074719  0.938697  0.294251  0.774844 -1.008306 -0.693936   \n",
       "9           0.887812  0.706783 -1.303109  0.774844  0.819702 -0.054784   \n",
       "10         -0.093454  1.054653 -0.504429  0.874062  0.972036 -0.949597   \n",
       "11         -2.055985  0.706783 -0.105089  0.675627  0.515034 -0.054784   \n",
       "12         -1.074719  1.054653  1.492270  0.973280  1.581372  1.862671   \n",
       "13          0.887812  0.706783  1.092930  0.973280  0.972036  1.862671   \n",
       "14         -0.093454  0.706783  0.294251  0.675627  0.972036 -1.972240   \n",
       "15         -0.093454  0.358913 -0.704099 -1.904039  0.819702  0.840028   \n",
       "16          1.869077 -1.612349  0.493921  0.675627 -0.398970 -0.054784   \n",
       "17         -0.093454 -0.336826  0.094581 -2.102474 -0.398970  0.200876   \n",
       "18         -1.074719  0.242957  0.000000  0.377973 -0.094302 -0.438275   \n",
       "19         -1.074719 -1.380436  1.092930  0.080319 -1.160640  0.456537   \n",
       "20          0.887812  0.822740  0.693590  0.576409  1.124370  0.200876   \n",
       "\n",
       "    major_Biology  major_Computer Science  major_Economics  major_Linguistics  \\\n",
       "0             0.0                     1.0              0.0                0.0   \n",
       "1             0.0                     0.0              0.0                0.0   \n",
       "2             0.0                     0.0              0.0                0.0   \n",
       "3             0.0                     0.0              0.0                0.0   \n",
       "4             0.0                     0.0              0.0                0.0   \n",
       "5             0.0                     0.0              1.0                0.0   \n",
       "6             0.0                     1.0              0.0                0.0   \n",
       "7             0.0                     0.0              0.0                0.0   \n",
       "8             0.0                     0.0              0.0                1.0   \n",
       "9             0.0                     0.0              0.0                0.0   \n",
       "10            0.0                     0.0              0.0                0.0   \n",
       "11            0.0                     0.0              0.0                0.0   \n",
       "12            0.0                     0.0              0.0                0.0   \n",
       "13            0.0                     0.0              0.0                0.0   \n",
       "14            0.0                     0.0              0.0                0.0   \n",
       "15            0.0                     1.0              0.0                0.0   \n",
       "16            0.0                     1.0              0.0                0.0   \n",
       "17            0.0                     0.0              1.0                0.0   \n",
       "18            1.0                     0.0              0.0                0.0   \n",
       "19            0.0                     0.0              0.0                0.0   \n",
       "20            0.0                     0.0              0.0                1.0   \n",
       "\n",
       "    major_Mathematics  major_Mechanical Engineering  major_Physics  \\\n",
       "0                 0.0                           0.0            0.0   \n",
       "1                 0.0                           1.0            0.0   \n",
       "2                 1.0                           0.0            0.0   \n",
       "3                 1.0                           0.0            0.0   \n",
       "4                 0.0                           0.0            0.0   \n",
       "5                 0.0                           0.0            0.0   \n",
       "6                 0.0                           0.0            0.0   \n",
       "7                 0.0                           1.0            0.0   \n",
       "8                 0.0                           0.0            0.0   \n",
       "9                 1.0                           0.0            0.0   \n",
       "10                0.0                           0.0            0.0   \n",
       "11                0.0                           0.0            1.0   \n",
       "12                0.0                           0.0            1.0   \n",
       "13                0.0                           1.0            0.0   \n",
       "14                1.0                           0.0            0.0   \n",
       "15                0.0                           0.0            0.0   \n",
       "16                0.0                           0.0            0.0   \n",
       "17                0.0                           0.0            0.0   \n",
       "18                0.0                           0.0            0.0   \n",
       "19                0.0                           0.0            0.0   \n",
       "20                0.0                           0.0            0.0   \n",
       "\n",
       "    major_Psychology  class_attendance  ml_experience  \n",
       "0                0.0               3.0            1.0  \n",
       "1                0.0               1.0            1.0  \n",
       "2                0.0               0.0            0.0  \n",
       "3                0.0               3.0            0.0  \n",
       "4                1.0               2.0            0.0  \n",
       "5                0.0               2.0            1.0  \n",
       "6                0.0               3.0            1.0  \n",
       "7                0.0               0.0            0.0  \n",
       "8                0.0               1.0            0.0  \n",
       "9                0.0               1.0            1.0  \n",
       "10               1.0               2.0            0.0  \n",
       "11               0.0               1.0            1.0  \n",
       "12               0.0               3.0            1.0  \n",
       "13               0.0               3.0            0.0  \n",
       "14               0.0               0.0            0.0  \n",
       "15               0.0               2.0            1.0  \n",
       "16               0.0               1.0            0.0  \n",
       "17               0.0               1.0            1.0  \n",
       "18               0.0               2.0            1.0  \n",
       "19               1.0               0.0            0.0  \n",
       "20               0.0               3.0            1.0  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_transformed, columns=column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Dealing with unknown categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's create a pipeline with the column transformer and pass it to `cross_validate`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "scores = cross_validate(pipe, X, y, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- What's going on here??\n",
    "- Let's look at the error message:\n",
    "`ValueError: Found unknown categories ['Biology'] in column 0 during transform\n",
    "`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Computer Science          4\n",
       "Mathematics               4\n",
       "Mechanical Engineering    3\n",
       "Psychology                3\n",
       "Economics                 2\n",
       "Linguistics               2\n",
       "Physics                   2\n",
       "Biology                   1\n",
       "Name: major, dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[\"major\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- There is only one instance of Biology.\n",
    "- During cross-validation, this is getting put into the validation split.\n",
    "- By default, `OneHotEncoder` throws an error because you might want to know about this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Simplest fix:\n",
    "- Pass `handle_unknown=\"ignore\"` argument to `OneHotEncoder`\n",
    "- It creates a row with all zeros. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (\n",
    "        OneHotEncoder(handle_unknown=\"ignore\"),\n",
    "        categorical_feats,\n",
    "    ),  # OHE on categorical features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-f0dd8f92-a0e9-4828-be91-336217b395ae {color: black;background-color: white;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae pre{padding: 0;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-toggleable {background-color: white;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-estimator:hover {background-color: #d4ebff;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-item {z-index: 1;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-parallel-item:only-child::after {width: 0;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-f0dd8f92-a0e9-4828-be91-336217b395ae div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-f0dd8f92-a0e9-4828-be91-336217b395ae\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dc28865a-ddfe-4f80-bc1f-a4c21444d843\" type=\"checkbox\" ><label for=\"dc28865a-ddfe-4f80-bc1f-a4c21444d843\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"1abde254-8bd8-449c-b095-1c64e7055aa1\" type=\"checkbox\" ><label for=\"1abde254-8bd8-449c-b095-1c64e7055aa1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d61a33af-ac6c-45bb-a8a1-cd339d50c781\" type=\"checkbox\" ><label for=\"d61a33af-ac6c-45bb-a8a1-cd339d50c781\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e1033f91-adb0-45de-a7a2-3adffae7e918\" type=\"checkbox\" ><label for=\"e1033f91-adb0-45de-a7a2-3adffae7e918\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"19efbaae-069f-4189-b7c8-7a4b18534b52\" type=\"checkbox\" ><label for=\"19efbaae-069f-4189-b7c8-7a4b18534b52\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"21d32d86-a5c2-4a7c-828f-b8b018d23724\" type=\"checkbox\" ><label for=\"21d32d86-a5c2-4a7c-828f-b8b018d23724\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7168a595-21e1-40f1-97e1-a87a7be6eef7\" type=\"checkbox\" ><label for=\"7168a595-21e1-40f1-97e1-a87a7be6eef7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"5dcf27ae-26a5-4bb4-9494-0d19e49f452e\" type=\"checkbox\" ><label for=\"5dcf27ae-26a5-4bb4-9494-0d19e49f452e\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"61047b72-d6ad-4fe9-97e0-84f43284de24\" type=\"checkbox\" ><label for=\"61047b72-d6ad-4fe9-97e0-84f43284de24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"235c55ef-718b-49bf-aadc-86ca426556fa\" type=\"checkbox\" ><label for=\"235c55ef-718b-49bf-aadc-86ca426556fa\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"3f4c40b7-011f-4579-b409-fea804350acd\" type=\"checkbox\" ><label for=\"3f4c40b7-011f-4579-b409-fea804350acd\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4f3fac62-db58-4277-b4ae-f94cc0c83717\" type=\"checkbox\" ><label for=\"4f3fac62-db58-4277-b4ae-f94cc0c83717\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['major']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop', ['enjoy_course'])])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.015854</td>\n",
       "      <td>0.005465</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.937500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.007426</td>\n",
       "      <td>0.003563</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.006512</td>\n",
       "      <td>0.003189</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.005595</td>\n",
       "      <td>0.003283</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.005206</td>\n",
       "      <td>0.002710</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.015854    0.005465        1.00     0.937500\n",
       "1  0.007426    0.003563        1.00     0.941176\n",
       "2  0.006512    0.003189        0.50     1.000000\n",
       "3  0.005595    0.003283        0.75     0.941176\n",
       "4  0.005206    0.002710        0.75     1.000000"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe, X, y, cv=5, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- With this approach, all unknown categories will be represented with all zeros and cross-validation is running OK now. \n",
    "\n",
    "Ask yourself the following questions when you work with categorical variables   \n",
    "- Do you want this behaviour? \n",
    "- Are you expecting to get many unknown categories? Do you want to be able to distinguish between them?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Cases where it's OK to break the golden rule \n",
    "\n",
    "- If it's some fix number of categories. For example, if it's something like provinces in Canada or majors taught at UBC. We know the categories in advance and this is one of the cases where it might be OK to violate the golden rule and get a list of all possible values for the categorical variable. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Categorical features with only two possible categories\n",
    "\n",
    "- Sometimes you have features with only two possible categories. \n",
    "- If we apply `OheHotEncoder` on such columns, it'll create two columns, which seems wasteful, as we could represent all information in the column in just one column with say 0's and 1's with presence of absence of one of one of the categories.\n",
    "- You can pass `drop=\"if_binary\"` argument to `OneHotEncoder` in order to create only one column in such scenario. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    yes\n",
       "1    yes\n",
       "2    yes\n",
       "3     no\n",
       "4    yes\n",
       "Name: enjoy_course, dtype: object"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[\"enjoy_course\"].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>enjoy_course_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  enjoy_course_enc\n",
       "0          yes                 1\n",
       "1          yes                 1\n",
       "2          yes                 1\n",
       "3           no                 0\n",
       "4          yes                 1\n",
       "5           no                 0\n",
       "6          yes                 1\n",
       "7           no                 0\n",
       "8           no                 0\n",
       "9          yes                 1"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe_enc = OneHotEncoder(drop=\"if_binary\", dtype=int, sparse=False)\n",
    "ohe_enc.fit(X[[\"enjoy_course\"]])\n",
    "transformed = ohe_enc.transform(X[[\"enjoy_course\"]])\n",
    "df = pd.DataFrame(data=transformed, columns=[\"enjoy_course_enc\"], index=X.index)\n",
    "pd.concat([X[[\"enjoy_course\"]], df], axis=1).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "ordinal_feats = [\"class_attendance\"]  # apply ordinal encoding\n",
    "binary_feats = [\"enjoy_course\"]  # apply one-hot encoding with drop=\"if_binary\"\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (\n",
    "        OneHotEncoder(handle_unknown=\"ignore\"),\n",
    "        categorical_feats,\n",
    "    ),  # OHE on categorical features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\n",
    "        OneHotEncoder(drop=\"if_binary\", dtype=int),\n",
    "        binary_feats,\n",
    "    ),  # OHE on categorical features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff {color: black;background-color: white;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff pre{padding: 0;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-toggleable {background-color: white;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-estimator:hover {background-color: #d4ebff;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-item {z-index: 1;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-parallel-item:only-child::after {width: 0;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-08c08a6a-08ad-4b21-8c6f-1ea25df10aff\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder-1&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;onehotencoder-2&#x27;,\n",
       "                                 OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                               dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;enjoy_course&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b6ae29e8-f24e-471d-bdbf-33e09c30c2f8\" type=\"checkbox\" ><label for=\"b6ae29e8-f24e-471d-bdbf-33e09c30c2f8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder-1&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;onehotencoder-2&#x27;,\n",
       "                                 OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                               dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;enjoy_course&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"bf54797d-3c66-462f-b40a-e74d468815c9\" type=\"checkbox\" ><label for=\"bf54797d-3c66-462f-b40a-e74d468815c9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"41fb0368-518c-47d7-90b5-1a6f15626e77\" type=\"checkbox\" ><label for=\"41fb0368-518c-47d7-90b5-1a6f15626e77\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a0d32a55-4cbe-4e31-b181-496a83eeeafb\" type=\"checkbox\" ><label for=\"a0d32a55-4cbe-4e31-b181-496a83eeeafb\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e90b4c76-1794-4762-9233-5896e62708e4\" type=\"checkbox\" ><label for=\"e90b4c76-1794-4762-9233-5896e62708e4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"2e4f7577-d0a9-42aa-9d8c-9b7a80589478\" type=\"checkbox\" ><label for=\"2e4f7577-d0a9-42aa-9d8c-9b7a80589478\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dc14470f-6030-443b-ba37-73e95ec0374d\" type=\"checkbox\" ><label for=\"dc14470f-6030-443b-ba37-73e95ec0374d\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"28e30183-8cab-4d90-8874-314bb8a83374\" type=\"checkbox\" ><label for=\"28e30183-8cab-4d90-8874-314bb8a83374\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"afeb0430-6ed3-4d71-ae8c-aa3c24d01130\" type=\"checkbox\" ><label for=\"afeb0430-6ed3-4d71-ae8c-aa3c24d01130\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"247f77a5-d41c-4c8e-9948-3f72772f762d\" type=\"checkbox\" ><label for=\"247f77a5-d41c-4c8e-9948-3f72772f762d\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(drop=&#x27;if_binary&#x27;, dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"c53b8b8f-cf6c-4123-8572-1dd2921eadf2\" type=\"checkbox\" ><label for=\"c53b8b8f-cf6c-4123-8572-1dd2921eadf2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e8bd7809-5bb5-4636-a665-da1e037f291f\" type=\"checkbox\" ><label for=\"e8bd7809-5bb5-4636-a665-da1e037f291f\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder-1',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['major']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('onehotencoder-2',\n",
       "                                 OneHotEncoder(drop='if_binary',\n",
       "                                               dtype=<class 'int'>),\n",
       "                                 ['enjoy_course']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience'])])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.017898</td>\n",
       "      <td>0.004811</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.005301</td>\n",
       "      <td>0.002933</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.004935</td>\n",
       "      <td>0.002855</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.004874</td>\n",
       "      <td>0.002952</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.004805</td>\n",
       "      <td>0.002855</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.017898    0.004811        1.00     1.000000\n",
       "1  0.005301    0.002933        1.00     0.941176\n",
       "2  0.004935    0.002855        0.50     1.000000\n",
       "3  0.004874    0.002952        1.00     0.941176\n",
       "4  0.004805    0.002855        0.75     1.000000"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe, X, y, cv=5, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "Do not read too much into the scores, as we are running cross-validation on a very small dataset with 21 examples. The main point here is to show you how can we use `ColumnTransformer` to apply different transformations on different columns.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Break (5 min)\n",
    "\n",
    "![](img/eva-coffee.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `ColumnTransformer` on the California housing dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2948.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>113600.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>137500.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3355.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>1457.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>170100.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1634.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>129300.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5566.0</td>\n",
       "      <td>1465.0</td>\n",
       "      <td>6303.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>205000.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "6051     -117.75     34.04                22.0       2948.0           636.0   \n",
       "20113    -119.57     37.94                17.0        346.0           130.0   \n",
       "14289    -117.13     32.74                46.0       3355.0           768.0   \n",
       "13665    -117.31     34.02                18.0       1634.0           274.0   \n",
       "14471    -117.23     32.88                18.0       5566.0          1465.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "6051       2600.0       602.0         3.1250            113600.0   \n",
       "20113        51.0        20.0         3.4861            137500.0   \n",
       "14289      1457.0       708.0         2.6604            170100.0   \n",
       "13665       899.0       285.0         5.2139            129300.0   \n",
       "14471      6303.0      1458.0         1.8580            205000.0   \n",
       "\n",
       "      ocean_proximity  \n",
       "6051           INLAND  \n",
       "20113          INLAND  \n",
       "14289      NEAR OCEAN  \n",
       "13665          INLAND  \n",
       "14471      NEAR OCEAN  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df = pd.read_csv(\"data/housing.csv\")\n",
    "train_df, test_df = train_test_split(housing_df, test_size=0.1, random_state=123)\n",
    "\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Some column values are mean/median but some are not. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's add some new features to the dataset which could help predicting the target: `median_house_value`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "train_df = train_df.assign(\n",
    "    rooms_per_household=train_df[\"total_rooms\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    rooms_per_household=test_df[\"total_rooms\"] / test_df[\"households\"]\n",
    ")\n",
    "\n",
    "train_df = train_df.assign(\n",
    "    bedrooms_per_household=train_df[\"total_bedrooms\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    bedrooms_per_household=test_df[\"total_bedrooms\"] / test_df[\"households\"]\n",
    ")\n",
    "\n",
    "train_df = train_df.assign(\n",
    "    population_per_household=train_df[\"population\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    population_per_household=test_df[\"population\"] / test_df[\"households\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2948.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>113600.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>4.897010</td>\n",
       "      <td>1.056478</td>\n",
       "      <td>4.318937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>137500.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>17.300000</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>2.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3355.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>1457.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>170100.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>4.738701</td>\n",
       "      <td>1.084746</td>\n",
       "      <td>2.057910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1634.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>129300.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0.961404</td>\n",
       "      <td>3.154386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5566.0</td>\n",
       "      <td>1465.0</td>\n",
       "      <td>6303.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>205000.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>3.817558</td>\n",
       "      <td>1.004801</td>\n",
       "      <td>4.323045</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "6051     -117.75     34.04                22.0       2948.0           636.0   \n",
       "20113    -119.57     37.94                17.0        346.0           130.0   \n",
       "14289    -117.13     32.74                46.0       3355.0           768.0   \n",
       "13665    -117.31     34.02                18.0       1634.0           274.0   \n",
       "14471    -117.23     32.88                18.0       5566.0          1465.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "6051       2600.0       602.0         3.1250            113600.0   \n",
       "20113        51.0        20.0         3.4861            137500.0   \n",
       "14289      1457.0       708.0         2.6604            170100.0   \n",
       "13665       899.0       285.0         5.2139            129300.0   \n",
       "14471      6303.0      1458.0         1.8580            205000.0   \n",
       "\n",
       "      ocean_proximity  rooms_per_household  bedrooms_per_household  \\\n",
       "6051           INLAND             4.897010                1.056478   \n",
       "20113          INLAND            17.300000                6.500000   \n",
       "14289      NEAR OCEAN             4.738701                1.084746   \n",
       "13665          INLAND             5.733333                0.961404   \n",
       "14471      NEAR OCEAN             3.817558                1.004801   \n",
       "\n",
       "       population_per_household  \n",
       "6051                   4.318937  \n",
       "20113                  2.550000  \n",
       "14289                  2.057910  \n",
       "13665                  3.154386  \n",
       "14471                  4.323045  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's keep both numeric and categorical columns in the data.\n",
    "X_train = train_df.drop(columns=[\"median_house_value\"])\n",
    "y_train = train_df[\"median_house_value\"]\n",
    "\n",
    "X_test = test_df.drop(columns=[\"median_house_value\"])\n",
    "y_test = test_df[\"median_house_value\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer, make_column_transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>ocean_proximity</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2948.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>4.897010</td>\n",
       "      <td>1.056478</td>\n",
       "      <td>4.318937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>17.300000</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>2.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3355.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>1457.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>4.738701</td>\n",
       "      <td>1.084746</td>\n",
       "      <td>2.057910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1634.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0.961404</td>\n",
       "      <td>3.154386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5566.0</td>\n",
       "      <td>1465.0</td>\n",
       "      <td>6303.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>3.817558</td>\n",
       "      <td>1.004801</td>\n",
       "      <td>4.323045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9730</th>\n",
       "      <td>-121.74</td>\n",
       "      <td>36.79</td>\n",
       "      <td>16.0</td>\n",
       "      <td>3841.0</td>\n",
       "      <td>620.0</td>\n",
       "      <td>1799.0</td>\n",
       "      <td>611.0</td>\n",
       "      <td>4.3814</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.286416</td>\n",
       "      <td>1.014730</td>\n",
       "      <td>2.944354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14690</th>\n",
       "      <td>-117.09</td>\n",
       "      <td>32.80</td>\n",
       "      <td>36.0</td>\n",
       "      <td>2163.0</td>\n",
       "      <td>367.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>4.7188</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>6.008333</td>\n",
       "      <td>1.019444</td>\n",
       "      <td>2.541667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7938</th>\n",
       "      <td>-118.11</td>\n",
       "      <td>33.86</td>\n",
       "      <td>33.0</td>\n",
       "      <td>2389.0</td>\n",
       "      <td>410.0</td>\n",
       "      <td>1229.0</td>\n",
       "      <td>393.0</td>\n",
       "      <td>5.3889</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.078880</td>\n",
       "      <td>1.043257</td>\n",
       "      <td>3.127226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18365</th>\n",
       "      <td>-122.12</td>\n",
       "      <td>37.28</td>\n",
       "      <td>21.0</td>\n",
       "      <td>349.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>149.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>5.8691</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.232143</td>\n",
       "      <td>1.142857</td>\n",
       "      <td>2.660714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10931</th>\n",
       "      <td>-117.91</td>\n",
       "      <td>33.74</td>\n",
       "      <td>25.0</td>\n",
       "      <td>4273.0</td>\n",
       "      <td>965.0</td>\n",
       "      <td>2946.0</td>\n",
       "      <td>922.0</td>\n",
       "      <td>2.9926</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>4.634490</td>\n",
       "      <td>1.046638</td>\n",
       "      <td>3.195228</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "6051     -117.75     34.04                22.0       2948.0           636.0   \n",
       "20113    -119.57     37.94                17.0        346.0           130.0   \n",
       "14289    -117.13     32.74                46.0       3355.0           768.0   \n",
       "13665    -117.31     34.02                18.0       1634.0           274.0   \n",
       "14471    -117.23     32.88                18.0       5566.0          1465.0   \n",
       "9730     -121.74     36.79                16.0       3841.0           620.0   \n",
       "14690    -117.09     32.80                36.0       2163.0           367.0   \n",
       "7938     -118.11     33.86                33.0       2389.0           410.0   \n",
       "18365    -122.12     37.28                21.0        349.0            64.0   \n",
       "10931    -117.91     33.74                25.0       4273.0           965.0   \n",
       "\n",
       "       population  households  median_income ocean_proximity  \\\n",
       "6051       2600.0       602.0         3.1250          INLAND   \n",
       "20113        51.0        20.0         3.4861          INLAND   \n",
       "14289      1457.0       708.0         2.6604      NEAR OCEAN   \n",
       "13665       899.0       285.0         5.2139          INLAND   \n",
       "14471      6303.0      1458.0         1.8580      NEAR OCEAN   \n",
       "9730       1799.0       611.0         4.3814       <1H OCEAN   \n",
       "14690       915.0       360.0         4.7188      NEAR OCEAN   \n",
       "7938       1229.0       393.0         5.3889       <1H OCEAN   \n",
       "18365       149.0        56.0         5.8691       <1H OCEAN   \n",
       "10931      2946.0       922.0         2.9926       <1H OCEAN   \n",
       "\n",
       "       rooms_per_household  bedrooms_per_household  population_per_household  \n",
       "6051              4.897010                1.056478                  4.318937  \n",
       "20113            17.300000                6.500000                  2.550000  \n",
       "14289             4.738701                1.084746                  2.057910  \n",
       "13665             5.733333                0.961404                  3.154386  \n",
       "14471             3.817558                1.004801                  4.323045  \n",
       "9730              6.286416                1.014730                  2.944354  \n",
       "14690             6.008333                1.019444                  2.541667  \n",
       "7938              6.078880                1.043257                  3.127226  \n",
       "18365             6.232143                1.142857                  2.660714  \n",
       "10931             4.634490                1.046638                  3.195228  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['longitude', 'latitude', 'housing_median_age', 'total_rooms',\n",
       "       'total_bedrooms', 'population', 'households', 'median_income',\n",
       "       'ocean_proximity', 'rooms_per_household', 'bedrooms_per_household',\n",
       "       'population_per_household'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Identify the categorical and numeric columns\n",
    "numeric_features = [\n",
    "    \"longitude\",\n",
    "    \"latitude\",\n",
    "    \"housing_median_age\",\n",
    "    \"total_rooms\",\n",
    "    \"total_bedrooms\",\n",
    "    \"population\",\n",
    "    \"households\",\n",
    "    \"median_income\",\n",
    "    \"rooms_per_household\",\n",
    "    \"bedrooms_per_household\",\n",
    "    \"population_per_household\",\n",
    "]\n",
    "\n",
    "categorical_features = [\"ocean_proximity\"]\n",
    "target = \"median_income\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's create a `ColumnTransformer` for our dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18576 entries, 6051 to 19966\n",
      "Data columns (total 12 columns):\n",
      " #   Column                    Non-Null Count  Dtype  \n",
      "---  ------                    --------------  -----  \n",
      " 0   longitude                 18576 non-null  float64\n",
      " 1   latitude                  18576 non-null  float64\n",
      " 2   housing_median_age        18576 non-null  float64\n",
      " 3   total_rooms               18576 non-null  float64\n",
      " 4   total_bedrooms            18391 non-null  float64\n",
      " 5   population                18576 non-null  float64\n",
      " 6   households                18576 non-null  float64\n",
      " 7   median_income             18576 non-null  float64\n",
      " 8   ocean_proximity           18576 non-null  object \n",
      " 9   rooms_per_household       18576 non-null  float64\n",
      " 10  bedrooms_per_household    18391 non-null  float64\n",
      " 11  population_per_household  18576 non-null  float64\n",
      "dtypes: float64(11), object(1)\n",
      "memory usage: 1.8+ MB\n"
     ]
    }
   ],
   "source": [
    "X_train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1H OCEAN     8221\n",
       "INLAND        5915\n",
       "NEAR OCEAN    2389\n",
       "NEAR BAY      2046\n",
       "ISLAND           5\n",
       "Name: ocean_proximity, dtype: int64"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[\"ocean_proximity\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "numeric_transformer = make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler())\n",
    "categorical_transformer = OneHotEncoder(handle_unknown=\"ignore\")\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (numeric_transformer, numeric_features),\n",
    "    (categorical_transformer, categorical_features),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 {color: black;background-color: white;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 pre{padding: 0;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-toggleable {background-color: white;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-item {z-index: 1;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-parallel-item:only-child::after {width: 0;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-a610b2ad-b4b3-404d-98b7-bb72fadf1dc5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"18245828-5d6e-4a4d-8c7a-2d6c422fb282\" type=\"checkbox\" ><label for=\"18245828-5d6e-4a4d-8c7a-2d6c422fb282\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b9e1b1db-96ad-4ff6-b84c-ca8d7e7b6b24\" type=\"checkbox\" ><label for=\"b9e1b1db-96ad-4ff6-b84c-ca8d7e7b6b24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"67fade1a-e27b-4845-92eb-9eeb0c25ba7a\" type=\"checkbox\" ><label for=\"67fade1a-e27b-4845-92eb-9eeb0c25ba7a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"149f7e5e-bca9-43e3-b22b-86b7505ccfe3\" type=\"checkbox\" ><label for=\"149f7e5e-bca9-43e3-b22b-86b7505ccfe3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"266fa603-ee2c-4ff2-ab67-0df7bf8ba49f\" type=\"checkbox\" ><label for=\"266fa603-ee2c-4ff2-ab67-0df7bf8ba49f\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4b476c28-da0d-460a-8b6f-a9080922fc93\" type=\"checkbox\" ><label for=\"4b476c28-da0d-460a-8b6f-a9080922fc93\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer(strategy='median')),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['longitude', 'latitude', 'housing_median_age',\n",
       "                                  'total_rooms', 'total_bedrooms', 'population',\n",
       "                                  'households', 'median_income',\n",
       "                                  'rooms_per_household',\n",
       "                                  'bedrooms_per_household',\n",
       "                                  'population_per_household']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['ocean_proximity'])])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X_train_pp = preprocessor.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- When we `fit` the preprocessor, it calls `fit` on _all_ the transformers\n",
    "- When we `transform` the preprocessor, it calls `transform` on _all_ the transformers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can get the new names of the columns that were generated by the one-hot encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-27359521-1567-4bce-9897-b35c56062727 {color: black;background-color: white;}#sk-27359521-1567-4bce-9897-b35c56062727 pre{padding: 0;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-toggleable {background-color: white;}#sk-27359521-1567-4bce-9897-b35c56062727 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-27359521-1567-4bce-9897-b35c56062727 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-27359521-1567-4bce-9897-b35c56062727 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-27359521-1567-4bce-9897-b35c56062727 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-27359521-1567-4bce-9897-b35c56062727 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-27359521-1567-4bce-9897-b35c56062727 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-estimator:hover {background-color: #d4ebff;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-item {z-index: 1;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-parallel-item:only-child::after {width: 0;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-27359521-1567-4bce-9897-b35c56062727 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-27359521-1567-4bce-9897-b35c56062727\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"9b70480d-2571-4798-b9a2-76eeca678a19\" type=\"checkbox\" ><label for=\"9b70480d-2571-4798-b9a2-76eeca678a19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"5224973a-4eae-4b4e-867d-5ce9bc5b77f5\" type=\"checkbox\" ><label for=\"5224973a-4eae-4b4e-867d-5ce9bc5b77f5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8321f1d3-58cc-4eef-863f-84ff15ae016f\" type=\"checkbox\" ><label for=\"8321f1d3-58cc-4eef-863f-84ff15ae016f\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"3a950cb1-64bd-4b6c-a973-5635bd36bf31\" type=\"checkbox\" ><label for=\"3a950cb1-64bd-4b6c-a973-5635bd36bf31\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"98420aa2-eae6-47bf-a495-c4aae9e64bc9\" type=\"checkbox\" ><label for=\"98420aa2-eae6-47bf-a495-c4aae9e64bc9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"6a0c7516-75dc-4944-a950-ef9ae586b88a\" type=\"checkbox\" ><label for=\"6a0c7516-75dc-4944-a950-ef9ae586b88a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer(strategy='median')),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['longitude', 'latitude', 'housing_median_age',\n",
       "                                  'total_rooms', 'total_bedrooms', 'population',\n",
       "                                  'households', 'median_income',\n",
       "                                  'rooms_per_household',\n",
       "                                  'bedrooms_per_household',\n",
       "                                  'population_per_household']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['ocean_proximity'])])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['ocean_proximity_<1H OCEAN', 'ocean_proximity_INLAND',\n",
       "       'ocean_proximity_ISLAND', 'ocean_proximity_NEAR BAY',\n",
       "       'ocean_proximity_NEAR OCEAN'], dtype=object)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor.named_transformers_[\"onehotencoder\"].get_feature_names_out(\n",
    "    categorical_features\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Combining this with the numeric feature names gives us all the column names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['longitude',\n",
       " 'latitude',\n",
       " 'housing_median_age',\n",
       " 'total_rooms',\n",
       " 'total_bedrooms',\n",
       " 'population',\n",
       " 'households',\n",
       " 'median_income',\n",
       " 'rooms_per_household',\n",
       " 'bedrooms_per_household',\n",
       " 'population_per_household',\n",
       " 'ocean_proximity_<1H OCEAN',\n",
       " 'ocean_proximity_INLAND',\n",
       " 'ocean_proximity_ISLAND',\n",
       " 'ocean_proximity_NEAR BAY',\n",
       " 'ocean_proximity_NEAR OCEAN']"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = numeric_features + list(\n",
    "    preprocessor.named_transformers_[\"onehotencoder\"].get_feature_names_out(\n",
    "        categorical_features\n",
    "    )\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's visualize the preprocessed training data as a dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "      <th>ocean_proximity_&lt;1H OCEAN</th>\n",
       "      <th>ocean_proximity_INLAND</th>\n",
       "      <th>ocean_proximity_ISLAND</th>\n",
       "      <th>ocean_proximity_NEAR BAY</th>\n",
       "      <th>ocean_proximity_NEAR OCEAN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.908140</td>\n",
       "      <td>-0.743917</td>\n",
       "      <td>-0.526078</td>\n",
       "      <td>0.143120</td>\n",
       "      <td>0.235339</td>\n",
       "      <td>1.026092</td>\n",
       "      <td>0.266135</td>\n",
       "      <td>-0.389736</td>\n",
       "      <td>-0.210591</td>\n",
       "      <td>-0.083813</td>\n",
       "      <td>0.126398</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.002057</td>\n",
       "      <td>1.083123</td>\n",
       "      <td>-0.923283</td>\n",
       "      <td>-1.049510</td>\n",
       "      <td>-0.969959</td>\n",
       "      <td>-1.206672</td>\n",
       "      <td>-1.253312</td>\n",
       "      <td>-0.198924</td>\n",
       "      <td>4.726412</td>\n",
       "      <td>11.166631</td>\n",
       "      <td>-0.050132</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.218207</td>\n",
       "      <td>-1.352930</td>\n",
       "      <td>1.380504</td>\n",
       "      <td>0.329670</td>\n",
       "      <td>0.549764</td>\n",
       "      <td>0.024896</td>\n",
       "      <td>0.542873</td>\n",
       "      <td>-0.635239</td>\n",
       "      <td>-0.273606</td>\n",
       "      <td>-0.025391</td>\n",
       "      <td>-0.099240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.128188</td>\n",
       "      <td>-0.753286</td>\n",
       "      <td>-0.843842</td>\n",
       "      <td>-0.459154</td>\n",
       "      <td>-0.626949</td>\n",
       "      <td>-0.463877</td>\n",
       "      <td>-0.561467</td>\n",
       "      <td>0.714077</td>\n",
       "      <td>0.122307</td>\n",
       "      <td>-0.280310</td>\n",
       "      <td>0.010183</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.168196</td>\n",
       "      <td>-1.287344</td>\n",
       "      <td>-0.843842</td>\n",
       "      <td>1.343085</td>\n",
       "      <td>2.210026</td>\n",
       "      <td>4.269688</td>\n",
       "      <td>2.500924</td>\n",
       "      <td>-1.059242</td>\n",
       "      <td>-0.640266</td>\n",
       "      <td>-0.190617</td>\n",
       "      <td>0.126808</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18571</th>\n",
       "      <td>0.733102</td>\n",
       "      <td>-0.804818</td>\n",
       "      <td>0.586095</td>\n",
       "      <td>-0.875337</td>\n",
       "      <td>-0.243446</td>\n",
       "      <td>-0.822136</td>\n",
       "      <td>-0.966131</td>\n",
       "      <td>-0.118182</td>\n",
       "      <td>0.063110</td>\n",
       "      <td>-0.099558</td>\n",
       "      <td>0.071541</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18572</th>\n",
       "      <td>1.163195</td>\n",
       "      <td>-1.057793</td>\n",
       "      <td>-1.161606</td>\n",
       "      <td>0.940194</td>\n",
       "      <td>0.609314</td>\n",
       "      <td>0.882438</td>\n",
       "      <td>0.728235</td>\n",
       "      <td>0.357500</td>\n",
       "      <td>0.235096</td>\n",
       "      <td>-0.163397</td>\n",
       "      <td>0.007458</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18573</th>\n",
       "      <td>-1.097293</td>\n",
       "      <td>0.797355</td>\n",
       "      <td>-1.876574</td>\n",
       "      <td>0.695434</td>\n",
       "      <td>0.433046</td>\n",
       "      <td>0.881563</td>\n",
       "      <td>0.514155</td>\n",
       "      <td>0.934269</td>\n",
       "      <td>0.211892</td>\n",
       "      <td>-0.135305</td>\n",
       "      <td>0.044029</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18574</th>\n",
       "      <td>-1.437367</td>\n",
       "      <td>1.008167</td>\n",
       "      <td>1.221622</td>\n",
       "      <td>-0.499947</td>\n",
       "      <td>-0.484029</td>\n",
       "      <td>-0.759944</td>\n",
       "      <td>-0.454427</td>\n",
       "      <td>0.006578</td>\n",
       "      <td>-0.273382</td>\n",
       "      <td>-0.149822</td>\n",
       "      <td>-0.132875</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18575</th>\n",
       "      <td>0.242996</td>\n",
       "      <td>0.272667</td>\n",
       "      <td>-0.684960</td>\n",
       "      <td>-0.332190</td>\n",
       "      <td>-0.353018</td>\n",
       "      <td>-0.164307</td>\n",
       "      <td>-0.396991</td>\n",
       "      <td>-0.711754</td>\n",
       "      <td>0.025998</td>\n",
       "      <td>0.042957</td>\n",
       "      <td>0.051269</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>18576 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0       0.908140 -0.743917           -0.526078     0.143120        0.235339   \n",
       "1      -0.002057  1.083123           -0.923283    -1.049510       -0.969959   \n",
       "2       1.218207 -1.352930            1.380504     0.329670        0.549764   \n",
       "3       1.128188 -0.753286           -0.843842    -0.459154       -0.626949   \n",
       "4       1.168196 -1.287344           -0.843842     1.343085        2.210026   \n",
       "...          ...       ...                 ...          ...             ...   \n",
       "18571   0.733102 -0.804818            0.586095    -0.875337       -0.243446   \n",
       "18572   1.163195 -1.057793           -1.161606     0.940194        0.609314   \n",
       "18573  -1.097293  0.797355           -1.876574     0.695434        0.433046   \n",
       "18574  -1.437367  1.008167            1.221622    -0.499947       -0.484029   \n",
       "18575   0.242996  0.272667           -0.684960    -0.332190       -0.353018   \n",
       "\n",
       "       population  households  median_income  rooms_per_household  \\\n",
       "0        1.026092    0.266135      -0.389736            -0.210591   \n",
       "1       -1.206672   -1.253312      -0.198924             4.726412   \n",
       "2        0.024896    0.542873      -0.635239            -0.273606   \n",
       "3       -0.463877   -0.561467       0.714077             0.122307   \n",
       "4        4.269688    2.500924      -1.059242            -0.640266   \n",
       "...           ...         ...            ...                  ...   \n",
       "18571   -0.822136   -0.966131      -0.118182             0.063110   \n",
       "18572    0.882438    0.728235       0.357500             0.235096   \n",
       "18573    0.881563    0.514155       0.934269             0.211892   \n",
       "18574   -0.759944   -0.454427       0.006578            -0.273382   \n",
       "18575   -0.164307   -0.396991      -0.711754             0.025998   \n",
       "\n",
       "       bedrooms_per_household  population_per_household  \\\n",
       "0                   -0.083813                  0.126398   \n",
       "1                   11.166631                 -0.050132   \n",
       "2                   -0.025391                 -0.099240   \n",
       "3                   -0.280310                  0.010183   \n",
       "4                   -0.190617                  0.126808   \n",
       "...                       ...                       ...   \n",
       "18571               -0.099558                  0.071541   \n",
       "18572               -0.163397                  0.007458   \n",
       "18573               -0.135305                  0.044029   \n",
       "18574               -0.149822                 -0.132875   \n",
       "18575                0.042957                  0.051269   \n",
       "\n",
       "       ocean_proximity_<1H OCEAN  ocean_proximity_INLAND  \\\n",
       "0                            0.0                     1.0   \n",
       "1                            0.0                     1.0   \n",
       "2                            0.0                     0.0   \n",
       "3                            0.0                     1.0   \n",
       "4                            0.0                     0.0   \n",
       "...                          ...                     ...   \n",
       "18571                        1.0                     0.0   \n",
       "18572                        1.0                     0.0   \n",
       "18573                        1.0                     0.0   \n",
       "18574                        0.0                     0.0   \n",
       "18575                        0.0                     1.0   \n",
       "\n",
       "       ocean_proximity_ISLAND  ocean_proximity_NEAR BAY  \\\n",
       "0                         0.0                       0.0   \n",
       "1                         0.0                       0.0   \n",
       "2                         0.0                       0.0   \n",
       "3                         0.0                       0.0   \n",
       "4                         0.0                       0.0   \n",
       "...                       ...                       ...   \n",
       "18571                     0.0                       0.0   \n",
       "18572                     0.0                       0.0   \n",
       "18573                     0.0                       0.0   \n",
       "18574                     0.0                       1.0   \n",
       "18575                     0.0                       0.0   \n",
       "\n",
       "       ocean_proximity_NEAR OCEAN  \n",
       "0                             0.0  \n",
       "1                             0.0  \n",
       "2                             1.0  \n",
       "3                             0.0  \n",
       "4                             1.0  \n",
       "...                           ...  \n",
       "18571                         0.0  \n",
       "18572                         0.0  \n",
       "18573                         0.0  \n",
       "18574                         0.0  \n",
       "18575                         0.0  \n",
       "\n",
       "[18576 rows x 16 columns]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_train_pp, columns=column_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.002 (+/- 0.002)</td>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                fit_time         score_time          test_score  \\\n",
       "dummy  0.002 (+/- 0.002)  0.001 (+/- 0.000)  -0.001 (+/- 0.001)   \n",
       "\n",
       "             train_score  \n",
       "dummy  0.000 (+/- 0.000)  "
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_dict = {}\n",
    "dummy = DummyRegressor()\n",
    "results_dict[\"dummy\"] = mean_std_cross_val_scores(\n",
    "    dummy, X_train, y_train, return_train_score=True\n",
    ")\n",
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVR\n",
    "\n",
    "knn_pipe = make_pipeline(preprocessor, KNeighborsRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 {color: black;background-color: white;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 pre{padding: 0;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-toggleable {background-color: white;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-estimator:hover {background-color: #d4ebff;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-item {z-index: 1;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-parallel-item:only-child::after {width: 0;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-f757cf89-8c05-45ef-9407-3e3630d2e090 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-f757cf89-8c05-45ef-9407-3e3630d2e090\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;standardscaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;longitude&#x27;, &#x27;latitude&#x27;,\n",
       "                                                   &#x27;housing_median_age&#x27;,\n",
       "                                                   &#x27;total_rooms&#x27;,\n",
       "                                                   &#x27;total_bedrooms&#x27;,\n",
       "                                                   &#x27;population&#x27;, &#x27;households&#x27;,\n",
       "                                                   &#x27;median_income&#x27;,\n",
       "                                                   &#x27;rooms_per_household&#x27;,\n",
       "                                                   &#x27;bedrooms_per_household&#x27;,\n",
       "                                                   &#x27;population_per_household&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                                  [&#x27;ocean_proximity&#x27;])])),\n",
       "                (&#x27;kneighborsregressor&#x27;, KNeighborsRegressor())])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d318e637-b237-4e4f-ae60-15c98afedb50\" type=\"checkbox\" ><label for=\"d318e637-b237-4e4f-ae60-15c98afedb50\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;standardscaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;longitude&#x27;, &#x27;latitude&#x27;,\n",
       "                                                   &#x27;housing_median_age&#x27;,\n",
       "                                                   &#x27;total_rooms&#x27;,\n",
       "                                                   &#x27;total_bedrooms&#x27;,\n",
       "                                                   &#x27;population&#x27;, &#x27;households&#x27;,\n",
       "                                                   &#x27;median_income&#x27;,\n",
       "                                                   &#x27;rooms_per_household&#x27;,\n",
       "                                                   &#x27;bedrooms_per_household&#x27;,\n",
       "                                                   &#x27;population_per_household&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                                  [&#x27;ocean_proximity&#x27;])])),\n",
       "                (&#x27;kneighborsregressor&#x27;, KNeighborsRegressor())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"81dac891-0fcc-4414-8788-4d8e4cc90c79\" type=\"checkbox\" ><label for=\"81dac891-0fcc-4414-8788-4d8e4cc90c79\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4b7aa975-2790-4fd1-841d-936f332d3534\" type=\"checkbox\" ><label for=\"4b7aa975-2790-4fd1-841d-936f332d3534\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a213603e-837b-46f3-9253-80d6d58b85cd\" type=\"checkbox\" ><label for=\"a213603e-837b-46f3-9253-80d6d58b85cd\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8348e4f5-7739-466c-b939-047b69f53c5a\" type=\"checkbox\" ><label for=\"8348e4f5-7739-466c-b939-047b69f53c5a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"bc143c1f-00f1-4605-9c0f-75bb19b6f5a9\" type=\"checkbox\" ><label for=\"bc143c1f-00f1-4605-9c0f-75bb19b6f5a9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"87d5e739-fe3c-4cb2-aa2f-9011a36599f6\" type=\"checkbox\" ><label for=\"87d5e739-fe3c-4cb2-aa2f-9011a36599f6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"eb8e8653-a1df-41ab-b1a9-3e7574d3171f\" type=\"checkbox\" ><label for=\"eb8e8653-a1df-41ab-b1a9-3e7574d3171f\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KNeighborsRegressor</label><div class=\"sk-toggleable__content\"><pre>KNeighborsRegressor()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('columntransformer',\n",
       "                 ColumnTransformer(transformers=[('pipeline',\n",
       "                                                  Pipeline(steps=[('simpleimputer',\n",
       "                                                                   SimpleImputer(strategy='median')),\n",
       "                                                                  ('standardscaler',\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  ['longitude', 'latitude',\n",
       "                                                   'housing_median_age',\n",
       "                                                   'total_rooms',\n",
       "                                                   'total_bedrooms',\n",
       "                                                   'population', 'households',\n",
       "                                                   'median_income',\n",
       "                                                   'rooms_per_household',\n",
       "                                                   'bedrooms_per_household',\n",
       "                                                   'population_per_household']),\n",
       "                                                 ('onehotencoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore'),\n",
       "                                                  ['ocean_proximity'])])),\n",
       "                ('kneighborsregressor', KNeighborsRegressor())])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dict[\"imp + scaling + ohe + KNN\"] = mean_std_cross_val_scores(\n",
    "    knn_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.021 (+/- 0.001)</td>\n",
       "      <td>0.765 (+/- 0.072)</td>\n",
       "      <td>0.721 (+/- 0.012)</td>\n",
       "      <td>0.816 (+/- 0.006)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    fit_time         score_time  \\\n",
       "dummy                      0.001 (+/- 0.000)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN  0.021 (+/- 0.001)  0.765 (+/- 0.072)   \n",
       "\n",
       "                                   test_score        train_score  \n",
       "dummy                      -0.001 (+/- 0.001)  0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN   0.721 (+/- 0.012)  0.816 (+/- 0.006)  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "svr_pipe = make_pipeline(preprocessor, SVR())\n",
    "results_dict[\"imp + scaling + ohe + SVR (default)\"] = mean_std_cross_val_scores(\n",
    "    svr_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.021 (+/- 0.001)</td>\n",
       "      <td>0.765 (+/- 0.072)</td>\n",
       "      <td>0.721 (+/- 0.012)</td>\n",
       "      <td>0.816 (+/- 0.006)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (default)</th>\n",
       "      <td>8.218 (+/- 0.029)</td>\n",
       "      <td>4.313 (+/- 0.041)</td>\n",
       "      <td>-0.049 (+/- 0.012)</td>\n",
       "      <td>-0.049 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              fit_time         score_time  \\\n",
       "dummy                                0.001 (+/- 0.000)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN            0.021 (+/- 0.001)  0.765 (+/- 0.072)   \n",
       "imp + scaling + ohe + SVR (default)  8.218 (+/- 0.029)  4.313 (+/- 0.041)   \n",
       "\n",
       "                                             test_score         train_score  \n",
       "dummy                                -0.001 (+/- 0.001)   0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN             0.721 (+/- 0.012)   0.816 (+/- 0.006)  \n",
       "imp + scaling + ohe + SVR (default)  -0.049 (+/- 0.012)  -0.049 (+/- 0.001)  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results with `scikit-learn`'s default SVR hyperparameters are pretty bad. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "svr_C_pipe = make_pipeline(preprocessor, SVR(C=10000))\n",
    "results_dict[\"imp + scaling + ohe + SVR (C=10000)\"] = mean_std_cross_val_scores(\n",
    "    svr_C_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.021 (+/- 0.001)</td>\n",
       "      <td>0.765 (+/- 0.072)</td>\n",
       "      <td>0.721 (+/- 0.012)</td>\n",
       "      <td>0.816 (+/- 0.006)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (default)</th>\n",
       "      <td>8.218 (+/- 0.029)</td>\n",
       "      <td>4.313 (+/- 0.041)</td>\n",
       "      <td>-0.049 (+/- 0.012)</td>\n",
       "      <td>-0.049 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (C=10000)</th>\n",
       "      <td>7.581 (+/- 0.049)</td>\n",
       "      <td>4.332 (+/- 0.011)</td>\n",
       "      <td>0.721 (+/- 0.007)</td>\n",
       "      <td>0.726 (+/- 0.007)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              fit_time         score_time  \\\n",
       "dummy                                0.001 (+/- 0.000)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN            0.021 (+/- 0.001)  0.765 (+/- 0.072)   \n",
       "imp + scaling + ohe + SVR (default)  8.218 (+/- 0.029)  4.313 (+/- 0.041)   \n",
       "imp + scaling + ohe + SVR (C=10000)  7.581 (+/- 0.049)  4.332 (+/- 0.011)   \n",
       "\n",
       "                                             test_score         train_score  \n",
       "dummy                                -0.001 (+/- 0.001)   0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN             0.721 (+/- 0.012)   0.816 (+/- 0.006)  \n",
       "imp + scaling + ohe + SVR (default)  -0.049 (+/- 0.012)  -0.049 (+/- 0.001)  \n",
       "imp + scaling + ohe + SVR (C=10000)   0.721 (+/- 0.007)   0.726 (+/- 0.007)  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a bigger value for `C` the results are much better. We need to carry out systematic hyperparameter optimization to get better results. (Coming up next week.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- Note that categorical features are different than free text features. Sometimes there are columns containing free text information and we we'll look at ways to deal with them in the later part of this lecture. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### OHE with many categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Do we have enough data for rare categories to learn anything meaningful? \n",
    "- How about grouping them into bigger categories?\n",
    "    - Example: country names into continents such as \"South America\" or \"Asia\"\n",
    "- Or having \"other\" category for rare cases? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Do we actually want to use certain features for prediction?\n",
    "\n",
    "- Do you want to use certain features such as **gender** or **race** in prediction?\n",
    "- Remember that the systems you build are going to be used in some applications. \n",
    "- It's extremely important to be mindful of the consequences of including certain features in your predictive model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preprocessing the targets?\n",
    "\n",
    "- Generally no need for this when doing classification. \n",
    "- In regression it makes sense in some cases. More on this later. \n",
    "- `sklearn` is fine with categorical labels ($y$-values) for classification problems. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "#### True/False: Categorical features\n",
    "\n",
    "1. `handle_unknown=\"ignore\"` would treat all unknown categories equally. \n",
    "2. Creating groups of rarely occurring categories might overfit the model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Encoding text data  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "toy_spam = [\n",
    "    [\n",
    "        \"URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\"Lol you are always so convincing.\", \"non spam\"],\n",
    "    [\"Nah I don't think he goes to usf, he lives around here though\", \"non spam\"],\n",
    "    [\n",
    "        \"URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\n",
    "        \"Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\"Congrats! I can't wait to see you!!\", \"non spam\"],\n",
    "]\n",
    "toy_df = pd.DataFrame(toy_spam, columns=[\"sms\", \"target\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Spam/non spam toy example "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- What if the feature is in the form of raw text?\n",
    "- The feature `sms` below is neither categorical nor ordinal. \n",
    "- How can we encode it so that we can pass it to the machine learning algorithms we have seen so far? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sms</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lol you are always so convincing.</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nah I don't think he goes to usf, he lives around here though</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Congrats! I can't wait to see you!!</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                          sms  \\\n",
       "0                                                                URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!   \n",
       "1                                                                                                                           Lol you are always so convincing.   \n",
       "2                                                                                               Nah I don't think he goes to usf, he lives around here though   \n",
       "3                                                                                 URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!   \n",
       "4  Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   \n",
       "5                                                                                                                         Congrats! I can't wait to see you!!   \n",
       "\n",
       "     target  \n",
       "0      spam  \n",
       "1  non spam  \n",
       "2  non spam  \n",
       "3      spam  \n",
       "4      spam  \n",
       "5  non spam  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What if we apply OHE? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Congrats! I can't wait to see you!!  \\\n",
       "0                                 0.0   \n",
       "1                                 0.0   \n",
       "2                                 0.0   \n",
       "3                                 0.0   \n",
       "4                                 0.0   \n",
       "5                                 1.0   \n",
       "\n",
       "  Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  \\\n",
       "0                                                                                                                                                        0.0   \n",
       "1                                                                                                                                                        0.0   \n",
       "2                                                                                                                                                        0.0   \n",
       "3                                                                                                                                                        0.0   \n",
       "4                                                                                                                                                        1.0   \n",
       "5                                                                                                                                                        0.0   \n",
       "\n",
       "  Lol you are always so convincing.  \\\n",
       "0                               0.0   \n",
       "1                               1.0   \n",
       "2                               0.0   \n",
       "3                               0.0   \n",
       "4                               0.0   \n",
       "5                               0.0   \n",
       "\n",
       "  Nah I don't think he goes to usf, he lives around here though  \\\n",
       "0                                                           0.0   \n",
       "1                                                           0.0   \n",
       "2                                                           1.0   \n",
       "3                                                           0.0   \n",
       "4                                                           0.0   \n",
       "5                                                           0.0   \n",
       "\n",
       "  URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!  \\\n",
       "0                                                                         0.0   \n",
       "1                                                                         0.0   \n",
       "2                                                                         0.0   \n",
       "3                                                                         1.0   \n",
       "4                                                                         0.0   \n",
       "5                                                                         0.0   \n",
       "\n",
       "  URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!  \n",
       "0                                                                                          1.0  \n",
       "1                                                                                          0.0  \n",
       "2                                                                                          0.0  \n",
       "3                                                                                          0.0  \n",
       "4                                                                                          0.0  \n",
       "5                                                                                          0.0  "
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### DO NOT DO THIS.\n",
    "enc = OneHotEncoder(sparse=False)\n",
    "transformed = enc.fit_transform(toy_df[[\"sms\"]])\n",
    "pd.DataFrame(transformed, columns=enc.categories_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- We do not have a fixed number of categories here. \n",
    "- Each \"category\" (feature value) is likely to occur only once in the training data and we won't learn anything meaningful if we apply one-hot encoding or ordinal encoding on this feature. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- How can we encode or represent raw text data into fixed number of features so that we can learn some useful patterns from it?  \n",
    "- This is a well studied problem in the field of Natural Language Processing (NLP), which is concerned with giving computers the ability to understand written and spoken language. \n",
    "- Some popular representations of raw text include: \n",
    "    - **Bag of words** \n",
    "    - TF-IDF\n",
    "    - Embedding representations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag of words (BOW) representation\n",
    "\n",
    "- One of the most popular representation of raw text \n",
    "- Ignores the syntax and word order\n",
    "- It has two components: \n",
    "    - The vocabulary (all unique words in all documents) \n",
    "    - A value indicating either the presence or absence or the count of each word in the document. \n",
    "        \n",
    "<center>\n",
    "<img src='./img/bag-of-words.png' width=\"600\">\n",
    "</center>\n",
    "\n",
    "[Source](https://web.stanford.edu/~jurafsky/slp3/4.pdf)       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Extracting BOW features using `scikit-learn`\n",
    "- `CountVectorizer`\n",
    "    - Converts a collection of text documents to a matrix of word counts.  \n",
    "    - Each row represents a \"document\" (e.g., a text message in our example). \n",
    "    - Each column represents a word in the vocabulary (the set of unique words) in the training data. \n",
    "    - Each cell represents how often the word occurs in the document.       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "In the Natural Language Processing (NLP) community text data  is referred to as a **corpus** (plural: corpora). ```   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vec = CountVectorizer()\n",
    "X_counts = vec.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec.get_feature_names(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(toy_df[\"sms\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{important}\n",
    "Note that unlike other transformers we are passing a `Series` object to `fit_transform`. For other transformers, you can define one transformer for more than one columns. But with `CountVectorizer` you need to define separate `CountVectorizer` transformers for each text column, if you have more than one text columns.    \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<6x61 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 71 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Why sparse matrices? \n",
    "\n",
    "- Most words do not appear in a given document.\n",
    "- We get massive computational savings if we only store the nonzero elements.\n",
    "- There is a bit of overhead, because we also need to store the locations:\n",
    "    - e.g. \"location (3,27): 1\".\n",
    "    \n",
    "- However, if the fraction of nonzero is small, this is a huge win."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The total number of elements:  366\n",
      "The number of non-zero elements:  71\n",
      "Proportion of non-zero elements: 0.1940\n",
      "The value at cell 3,27 is: 1\n"
     ]
    }
   ],
   "source": [
    "print(\"The total number of elements: \", np.prod(X_counts.shape))\n",
    "print(\"The number of non-zero elements: \", X_counts.nnz)\n",
    "print(\n",
    "    \"Proportion of non-zero elements: %0.4f\" % (X_counts.nnz / np.prod(X_counts.shape))\n",
    ")\n",
    "print(\n",
    "    \"The value at cell 3,%d is: %d\"\n",
    "    % (vec.vocabulary_[\"jackpot\"], X_counts[3, vec.vocabulary_[\"jackpot\"]])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "Question for you\n",
    "- What would happen if you apply `StandardScaler` on sparse data? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### `OneHotEncoder` and sparse features \n",
    "- By default, `OneHotEncoder` also creates sparse features. \n",
    "- You could set `sparse=False` to get a regular `numpy` array. \n",
    "- If there are a huge number of categories, it may be beneficial to keep them sparse.\n",
    "- For smaller number of categories, it doesn't matter much."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Important hyperparameters of `CountVectorizer` \n",
    "\n",
    "- `binary`\n",
    "    - whether to use absence/presence feature values or counts\n",
    "- `max_features`\n",
    "    - only consider top `max_features` ordered by frequency in the corpus\n",
    "- `max_df`\n",
    "    - ignore features which occur in more than `max_df` documents \n",
    "- `min_df` \n",
    "    - ignore features which occur in less than `min_df` documents \n",
    "- `ngram_range`\n",
    "    - consider word sequences in the given range "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at all features, i.e., words (along with their frequencies)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec = CountVectorizer()\n",
    "X_counts = vec.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec.get_feature_names(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When we use `binary=True`, the representation uses presence/absence of words instead of word counts.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec_binary = CountVectorizer(binary=True)\n",
    "X_counts = vec_binary.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec_binary.get_feature_names(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can control the size of X (the number of features) using `max_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>free</th>\n",
       "      <th>have</th>\n",
       "      <th>mobile</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>you</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            free  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            have  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            mobile  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            the  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            to  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          1   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            you  \n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1  \n",
       "Lol you are always so convincing.                                                                                                                             1  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0  \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8 = CountVectorizer(max_features=8)\n",
    "X_counts = vec8.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec8.get_feature_names(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "Notice that `vec8` and `vec8_binary` have different vocabularies, which is kind of unexpected behaviour and doesn't match the documentation of `scikit-learn`. \n",
    "\n",
    "[Here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L1206-L1225) is the code for `binary=True` condition in `scikit-learn`. As we can see, the binarization is done before limiting the features to `max_features`, and so now we are actually looking at the document counts (in how many documents it occurs) rather than term count. This is not explained anywhere in the documentation. \n",
    "\n",
    "The ties in counts between different words makes it even more confusing. I don't think it'll have a big impact on the results but this is good to know! Remember that `scikit-learn` developers are also humans who are prone to make mistakes. So it's always a good habit to question whatever tools we use every now and then. \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>you</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>free</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>have</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mobile</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>the</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>update</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>urgent</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        counts\n",
       "to           5\n",
       "you          4\n",
       "free         3\n",
       "have         2\n",
       "mobile       2\n",
       "the          2\n",
       "update       2\n",
       "urgent       2"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8 = CountVectorizer(max_features=8)\n",
    "X_counts = vec8.fit_transform(toy_df[\"sms\"])\n",
    "pd.DataFrame(\n",
    "    data=X_counts.sum(axis=0).tolist()[0],\n",
    "    index=vec8.get_feature_names(),\n",
    "    columns=[\"counts\"],\n",
    ").sort_values(\"counts\", ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>you</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>free</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>have</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>prize</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>urgent</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mobiles</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>months</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         counts\n",
       "to            4\n",
       "you           4\n",
       "free          2\n",
       "have          2\n",
       "prize         2\n",
       "urgent        2\n",
       "mobiles       1\n",
       "months        1"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8_binary = CountVectorizer(binary=True, max_features=8)\n",
    "X_counts = vec8_binary.fit_transform(toy_df[\"sms\"])\n",
    "pd.DataFrame(\n",
    "    data=X_counts.sum(axis=0).tolist()[0],\n",
    "    index=vec8_binary.get_feature_names(),\n",
    "    columns=[\"counts\"],\n",
    ").sort_values(\"counts\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preprocessing\n",
    "\n",
    "- Note that `CountVectorizer` is carrying out some preprocessing such as because of the default argument values \n",
    "    - Converting words to lowercase (`lowercase=True`)\n",
    "    - getting rid of punctuation and special characters (`token_pattern ='(?u)\\\\b\\\\w\\\\w+\\\\b'`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(CountVectorizer(), SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 {color: black;background-color: white;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 pre{padding: 0;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-toggleable {background-color: white;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-item {z-index: 1;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-parallel-item:only-child::after {width: 0;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-10221f93-aa41-41c5-95a8-b55672ccb2a4 div.sk-container {display: inline-block;position: relative;}</style><div id=\"sk-10221f93-aa41-41c5-95a8-b55672ccb2a4\" class\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"3b3bb630-2c0e-4d2c-a788-901b1467dd35\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"3b3bb630-2c0e-4d2c-a788-901b1467dd35\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('countvectorizer', CountVectorizer()), ('svc', SVC())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"25d6190a-9a25-4ec2-88ca-8dcb0bf87226\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"25d6190a-9a25-4ec2-88ca-8dcb0bf87226\">CountVectorizer</label><div class=\"sk-toggleable__content\"><pre>CountVectorizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"1fd55e6f-a4da-49d7-9675-5b873db773b5\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"1fd55e6f-a4da-49d7-9675-5b873db773b5\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('countvectorizer', CountVectorizer()), ('svc', SVC())])"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.fit(toy_df[\"sms\"], toy_df[\"target\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['spam', 'non spam', 'non spam', 'spam', 'spam', 'non spam'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.predict(toy_df[\"sms\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Is this a realistic representation of text data? \n",
    "\n",
    "- Of course this is not a great representation of language\n",
    "    - We are throwing out everything we know about language and losing a lot of information. \n",
    "    - It assumes that there is no syntax and compositional meaning in language.  \n",
    "- But it works surprisingly well for many tasks. \n",
    "- We will learn more expressive representations in the coming weeks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### `CountVectorizer`: True or False\n",
    "\n",
    "1. As you increase the value for `max_features` hyperparameter of `CountVectorizer` the training score is likely to go up. \n",
    "2. Suppose you are encoding text data using `CountVectorizer`. If you encounter a word in the validation or the test split that's not available in the training data, we'll get an error. \n",
    "3. `max_df` hyperparameter of `CountVectorizer` can be used to get rid of most frequently occurring words from the dictionary.\n",
    "4. In the code below, inside `cross_validate`, each fold might have slightly different number of features (columns) in the fold.\n",
    "\n",
    "```\n",
    "pipe = (CountVectorizer(), SVC())\n",
    "cross_validate(pipe, X_train, y_train)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Identify column transformations\n",
    "\n",
    "Consider the restaurant data from the survey you did a few weeks ago. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>north_america</th>\n",
       "      <th>eat_out_freq</th>\n",
       "      <th>age</th>\n",
       "      <th>n_people</th>\n",
       "      <th>price</th>\n",
       "      <th>food_type</th>\n",
       "      <th>noise_level</th>\n",
       "      <th>good_server</th>\n",
       "      <th>comments</th>\n",
       "      <th>restaurant_name</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Yes</td>\n",
       "      <td>3.0</td>\n",
       "      <td>29</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Other</td>\n",
       "      <td>no music</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Yes</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23</td>\n",
       "      <td>10</td>\n",
       "      <td>20</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>low</td>\n",
       "      <td>Yes</td>\n",
       "      <td>food tastes good</td>\n",
       "      <td>Midam</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yes</td>\n",
       "      <td>2.0</td>\n",
       "      <td>21</td>\n",
       "      <td>10</td>\n",
       "      <td>40</td>\n",
       "      <td>Other</td>\n",
       "      <td>low</td>\n",
       "      <td>Yes</td>\n",
       "      <td>good food</td>\n",
       "      <td>pear tree</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>No</td>\n",
       "      <td>2.0</td>\n",
       "      <td>24</td>\n",
       "      <td>15</td>\n",
       "      <td>40</td>\n",
       "      <td>Other</td>\n",
       "      <td>medium</td>\n",
       "      <td>Yes</td>\n",
       "      <td>My love for Korean food</td>\n",
       "      <td>Dami</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Yes</td>\n",
       "      <td>5.0</td>\n",
       "      <td>23</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Canadian/American</td>\n",
       "      <td>low</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Good food</td>\n",
       "      <td>NaN</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Yes</td>\n",
       "      <td>2.0</td>\n",
       "      <td>22</td>\n",
       "      <td>60</td>\n",
       "      <td>20</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>medium</td>\n",
       "      <td>Yes</td>\n",
       "      <td>food tasted great!</td>\n",
       "      <td>NaN</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Yes</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23</td>\n",
       "      <td>10000000</td>\n",
       "      <td>1000000</td>\n",
       "      <td>Canadian/American</td>\n",
       "      <td>crazy loud</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CACTUS CLUB CAFE</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Yes</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>40</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>high</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>21</td>\n",
       "      <td>40</td>\n",
       "      <td>60</td>\n",
       "      <td>Italian</td>\n",
       "      <td>high</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Frankie’s</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>No</td>\n",
       "      <td>2.0</td>\n",
       "      <td>20</td>\n",
       "      <td>50</td>\n",
       "      <td>45</td>\n",
       "      <td>Canadian/American</td>\n",
       "      <td>medium</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Steak was good, ambiance was nice and server was helpful.</td>\n",
       "      <td>Hy's steakhouse</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  north_america  eat_out_freq  age  n_people    price          food_type  \\\n",
       "0           Yes           3.0   29        20       10              Other   \n",
       "1           Yes           2.0   23        10       20            Chinese   \n",
       "2           Yes           2.0   21        10       40              Other   \n",
       "3            No           2.0   24        15       40              Other   \n",
       "4           Yes           5.0   23        20       10  Canadian/American   \n",
       "5           Yes           2.0   22        60       20            Chinese   \n",
       "6           Yes           2.0   23  10000000  1000000  Canadian/American   \n",
       "7           Yes           4.0   20        20       40            Chinese   \n",
       "8           Yes           1.0   21        40       60            Italian   \n",
       "9            No           2.0   20        50       45  Canadian/American   \n",
       "\n",
       "  noise_level good_server  \\\n",
       "0    no music         Yes   \n",
       "1         low         Yes   \n",
       "2         low         Yes   \n",
       "3      medium         Yes   \n",
       "4         low         Yes   \n",
       "5      medium         Yes   \n",
       "6  crazy loud         Yes   \n",
       "7        high         Yes   \n",
       "8        high         Yes   \n",
       "9      medium         Yes   \n",
       "\n",
       "                                                    comments  \\\n",
       "0                                                        NaN   \n",
       "1                                           food tastes good   \n",
       "2                                                  good food   \n",
       "3                                    My love for Korean food   \n",
       "4                                                  Good food   \n",
       "5                                         food tasted great!   \n",
       "6                                                        NaN   \n",
       "7                                                        NaN   \n",
       "8                                                        NaN   \n",
       "9  Steak was good, ambiance was nice and server was helpful.   \n",
       "\n",
       "    restaurant_name target  \n",
       "0               NaN   like  \n",
       "1             Midam   like  \n",
       "2         pear tree   like  \n",
       "3              Dami   like  \n",
       "4               NaN   like  \n",
       "5               NaN   like  \n",
       "6  CACTUS CLUB CAFE   like  \n",
       "7               NaN   like  \n",
       "8         Frankie’s   like  \n",
       "9   Hy's steakhouse   like  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "restaurant_data = pd.read_csv(\"data/cleaned_restaurant_data.csv\")\n",
    "restaurant_data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What all feature transformations you would apply on this dataset? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What did we learn today?\n",
    "\n",
    "- Motivation to use `ColumnTransformer`\n",
    "- `ColumnTransformer` syntax\n",
    "- Defining transformers with multiple transformations\n",
    "- How to visualize transformed features in a dataframe \n",
    "- More on ordinal features \n",
    "- Different arguments `OneHotEncoder`\n",
    "    - `handle_unknow=\"ignore\"`\n",
    "    - `if_binary`\n",
    "- Dealing with text features\n",
    "    - Bag of words representation: `CountVectorizer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/eva-talksoon.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330] *",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}