lectures/06_column-transformer-text-feats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 6: `sklearn` `ColumnTransformer` and Text Features\n",
    "\n",
    "UBC 2022 Summer\n",
    "\n",
    "Instructor: Mehrdad Oveisi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import HTML\n",
    "\n",
    "sys.path.append(\"code/.\")\n",
    "from plotting_functions import *\n",
    "from utils import *\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 200)\n",
    "\n",
    "from sklearn.compose import ColumnTransformer, make_column_transformer\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.model_selection import cross_val_score, cross_validate, train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "## Learning outcomes \n",
    "\n",
    "From this lecture, you will be able to \n",
    "\n",
    "- use `ColumnTransformer` to build all our transformations together into one object and use it with `sklearn` pipelines;  \n",
    "- define `ColumnTransformer` where transformers contain more than one steps;\n",
    "- explain `handle_unknown=\"ignore\"` hyperparameter of `scikit-learn`'s `OneHotEncoder`;\n",
    "- explain `drop=\"if_binary\"` argument of `OneHotEncoder`;\n",
    "- identify when it's appropriate to apply ordinal encoding vs one-hot encoding;\n",
    "- explain strategies to deal with categorical variables with too many categories; \n",
    "- explain why text data needs a different treatment than categorical variables;\n",
    "- use `scikit-learn`'s `CountVectorizer` to encode text data;\n",
    "- explain different hyperparameters of `CountVectorizer`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## sklearn's [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- In most applications, some features are categorical, some are continuous, some are binary, and some are ordinal. \n",
    "\n",
    "- When we want to develop supervised machine learning pipelines on real-world datasets, very often we want to apply different transformation on different columns. \n",
    "\n",
    "- Enter `sklearn`'s `ColumnTransformer`!! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "- Let's look at a toy example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>quiz2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Good</td>\n",
       "      <td>5</td>\n",
       "      <td>70</td>\n",
       "      <td>73.0</td>\n",
       "      <td>68</td>\n",
       "      <td>74</td>\n",
       "      <td>71</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>80</td>\n",
       "      <td>88.0</td>\n",
       "      <td>89</td>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>93.0</td>\n",
       "      <td>69</td>\n",
       "      <td>79</td>\n",
       "      <td>75</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>97</td>\n",
       "      <td>90.0</td>\n",
       "      <td>94</td>\n",
       "      <td>82</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Average</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>82.0</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>85</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>3</td>\n",
       "      <td>98</td>\n",
       "      <td>86.0</td>\n",
       "      <td>95</td>\n",
       "      <td>95</td>\n",
       "      <td>78</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Physics</td>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "      <td>95</td>\n",
       "      <td>88.0</td>\n",
       "      <td>93</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Physics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>2</td>\n",
       "      <td>98</td>\n",
       "      <td>96.0</td>\n",
       "      <td>96</td>\n",
       "      <td>99</td>\n",
       "      <td>100</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>94.0</td>\n",
       "      <td>96</td>\n",
       "      <td>95</td>\n",
       "      <td>100</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>90.0</td>\n",
       "      <td>93</td>\n",
       "      <td>95</td>\n",
       "      <td>70</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Good</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>85.0</td>\n",
       "      <td>67</td>\n",
       "      <td>94</td>\n",
       "      <td>92</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Average</td>\n",
       "      <td>5</td>\n",
       "      <td>75</td>\n",
       "      <td>91.0</td>\n",
       "      <td>93</td>\n",
       "      <td>86</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Average</td>\n",
       "      <td>3</td>\n",
       "      <td>86</td>\n",
       "      <td>89.0</td>\n",
       "      <td>65</td>\n",
       "      <td>86</td>\n",
       "      <td>87</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Biology</td>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>90</td>\n",
       "      <td>88</td>\n",
       "      <td>82</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Poor</td>\n",
       "      <td>2</td>\n",
       "      <td>77</td>\n",
       "      <td>94.0</td>\n",
       "      <td>87</td>\n",
       "      <td>81</td>\n",
       "      <td>89</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>96</td>\n",
       "      <td>92.0</td>\n",
       "      <td>92</td>\n",
       "      <td>96</td>\n",
       "      <td>87</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0           yes              1        Computer Science        Excellent   \n",
       "1           yes              1  Mechanical Engineering          Average   \n",
       "2           yes              0             Mathematics             Poor   \n",
       "3            no              0             Mathematics        Excellent   \n",
       "4           yes              0              Psychology             Good   \n",
       "5            no              1               Economics             Good   \n",
       "6           yes              1        Computer Science        Excellent   \n",
       "7            no              0  Mechanical Engineering             Poor   \n",
       "8            no              0             Linguistics          Average   \n",
       "9           yes              1             Mathematics          Average   \n",
       "10          yes              0              Psychology             Good   \n",
       "11          yes              1                 Physics          Average   \n",
       "12          yes              1                 Physics        Excellent   \n",
       "13          yes              0  Mechanical Engineering        Excellent   \n",
       "14           no              0             Mathematics             Poor   \n",
       "15           no              1        Computer Science             Good   \n",
       "16          yes              0        Computer Science          Average   \n",
       "17          yes              1               Economics          Average   \n",
       "18           no              1                 Biology             Good   \n",
       "19           no              0              Psychology             Poor   \n",
       "20          yes              1             Linguistics        Excellent   \n",
       "\n",
       "    university_years  lab1  lab2  lab3  lab4  quiz1   quiz2  \n",
       "0                  3    92  93.0    84    91     92      A+  \n",
       "1                  2    94  90.0    80    83     91  not A+  \n",
       "2                  3    78  85.0    83    80     80  not A+  \n",
       "3                  3    91   NaN    92    91     89      A+  \n",
       "4                  4    77  83.0    90    92     85      A+  \n",
       "5                  5    70  73.0    68    74     71  not A+  \n",
       "6                  4    80  88.0    89    88     91      A+  \n",
       "7                  3    95  93.0    69    79     75  not A+  \n",
       "8                  2    97  90.0    94    82     80  not A+  \n",
       "9                  4    95  82.0    94    94     85  not A+  \n",
       "10                 3    98  86.0    95    95     78      A+  \n",
       "11                 1    95  88.0    93    92     85      A+  \n",
       "12                 2    98  96.0    96    99    100      A+  \n",
       "13                 4    95  94.0    96    95    100      A+  \n",
       "14                 3    95  90.0    93    95     70  not A+  \n",
       "15                 3    92  85.0    67    94     92  not A+  \n",
       "16                 5    75  91.0    93    86     85      A+  \n",
       "17                 3    86  89.0    65    86     87  not A+  \n",
       "18                 2    91   NaN    90    88     82  not A+  \n",
       "19                 2    77  94.0    87    81     89  not A+  \n",
       "20                 4    96  92.0    92    96     87      A+  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/quiz2-grade-toy-col-transformer.csv\")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 21 entries, 0 to 20\n",
      "Data columns (total 11 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   enjoy_course      21 non-null     object \n",
      " 1   ml_experience     21 non-null     int64  \n",
      " 2   major             21 non-null     object \n",
      " 3   class_attendance  21 non-null     object \n",
      " 4   university_years  21 non-null     int64  \n",
      " 5   lab1              21 non-null     int64  \n",
      " 6   lab2              19 non-null     float64\n",
      " 7   lab3              21 non-null     int64  \n",
      " 8   lab4              21 non-null     int64  \n",
      " 9   quiz1             21 non-null     int64  \n",
      " 10  quiz2             21 non-null     object \n",
      "dtypes: float64(1), int64(6), object(4)\n",
      "memory usage: 1.9+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Transformations on the toy data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>quiz2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1   quiz2  \n",
       "0                 3    92  93.0    84    91     92      A+  \n",
       "1                 2    94  90.0    80    83     91  not A+  \n",
       "2                 3    78  85.0    83    80     80  not A+  \n",
       "3                 3    91   NaN    92    91     89      A+  \n",
       "4                 4    77  83.0    90    92     85      A+  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Scaling on numeric features\n",
    "- One-hot encoding on the categorical feature `major` and binary feature `enjoy_course`\n",
    "- Ordinal encoding on the ordinal feature `class_attendance`\n",
    "- Imputation on the `lab2` feature\n",
    "- None on the `ml_experience` feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `ColumnTransformer` example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['enjoy_course', 'ml_experience', 'major', 'class_attendance',\n",
       "       'university_years', 'lab1', 'lab2', 'lab3', 'lab4', 'quiz1'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df.drop(columns=[\"quiz2\"])\n",
    "y = df[\"quiz2\"]\n",
    "X.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Identify the transformations we want to apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_feats = [\"university_years\", \"lab1\", \"lab3\", \"lab4\", \"quiz1\"]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\n",
    "    \"lab2\",\n",
    "    \"class_attendance\",\n",
    "    \"enjoy_course\",\n",
    "]  # for now, do not include these features in modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, let's only focus on scaling and one-hot encoding first. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Create a column transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Each transformation is specified by a name, a transformer object, and the columns this transformer should be applied to. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct = ColumnTransformer(\n",
    "    [\n",
    "        (\"scaling\", StandardScaler(), numeric_feats),\n",
    "        (\"onehot\", OneHotEncoder(sparse=False), categorical_feats),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Convenient `make_column_transformer` syntax\n",
    "\n",
    "- Similar to `make_pipeline` syntax, there is convenient `make_column_transformer` syntax. \n",
    "- The syntax automatically names each step based on its class. \n",
    "- We'll be mostly using this syntax. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "ct = make_column_transformer(\n",
    "    (StandardScaler(), numeric_feats),  # scaling on numeric features\n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ColumnTransformer(transformers=[('standardscaler', StandardScaler(),\n",
       "                                 ['university_years', 'lab1', 'lab3', 'lab4',\n",
       "                                  'quiz1']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop',\n",
       "                                 ['lab2', 'class_attendance', 'enjoy_course'])])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "source": [
    "- When we `fit_transform`, each transformer is applied to the specified columns and the result of the transformations are concatenated horizontally. \n",
    "- A big advantage here is that we build all our transformations together into one object, and that way we're sure we do the same operations to all splits of the data.\n",
    "- Otherwise we might, for example, do the OHE on both train and test but forget to scale the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Let's examine the transformed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.09345386,  0.3589134 , -0.21733442,  0.36269995,  0.84002795,\n",
       "         0.        ,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [-1.07471942,  0.59082668, -0.61420598, -0.85597188,  0.71219761,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ,  0.        ,  0.        ,  1.        ]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformed[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(transformed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note*** that the returned object is not a dataframe. So there are no column names."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Viewing the transformed data as a dataframe\n",
    "\n",
    "- How can we view our transformed data as a dataframe? \n",
    "- We are adding more columns. \n",
    "- So the original columns won't directly map to the transformed data. \n",
    "- Let's create column names for the transformed data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'standardscaler': StandardScaler(),\n",
       " 'onehotencoder': OneHotEncoder(),\n",
       " 'passthrough': 'passthrough',\n",
       " 'drop': 'drop'}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct.named_transformers_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['major_Biology', 'major_Computer Science', 'major_Economics',\n",
       "       'major_Linguistics', 'major_Mathematics',\n",
       "       'major_Mechanical Engineering', 'major_Physics',\n",
       "       'major_Psychology'], dtype=object)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here are the new columns created by OneHotEncoder\n",
    "ct.named_transformers_[\"onehotencoder\"].get_feature_names_out()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology',\n",
       " 'ml_experience']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    "    + passthrough_feats\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note*** that the order of the columns in the transformed data depends upon the order of the features we pass to the `ColumnTransformer` and can be different than the order of the features in the original dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "      <th>ml_experience</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   university_years      lab1      lab3      lab4     quiz1  major_Biology  \\\n",
       "0         -0.093454  0.358913 -0.217334  0.362700  0.840028            0.0   \n",
       "1         -1.074719  0.590827 -0.614206 -0.855972  0.712198            0.0   \n",
       "2         -0.093454 -1.264480 -0.316552 -1.312974 -0.693936            0.0   \n",
       "3         -0.093454  0.242957  0.576409  0.362700  0.456537            0.0   \n",
       "4          0.887812 -1.380436  0.377973  0.515034 -0.054784            0.0   \n",
       "\n",
       "   major_Computer Science  major_Economics  major_Linguistics  \\\n",
       "0                     1.0              0.0                0.0   \n",
       "1                     0.0              0.0                0.0   \n",
       "2                     0.0              0.0                0.0   \n",
       "3                     0.0              0.0                0.0   \n",
       "4                     0.0              0.0                0.0   \n",
       "\n",
       "   major_Mathematics  major_Mechanical Engineering  major_Physics  \\\n",
       "0                0.0                           0.0            0.0   \n",
       "1                0.0                           1.0            0.0   \n",
       "2                1.0                           0.0            0.0   \n",
       "3                1.0                           0.0            0.0   \n",
       "4                0.0                           0.0            0.0   \n",
       "\n",
       "   major_Psychology  ml_experience  \n",
       "0               0.0            1.0  \n",
       "1               0.0            1.0  \n",
       "2               0.0            0.0  \n",
       "3               0.0            0.0  \n",
       "4               1.0            0.0  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(transformed, columns=column_names).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### `ColumnTransformer`: Transformed data\n",
    "\n",
    "<br>\n",
    "\n",
    "<img src='./img/column-transformer.png' width=\"1500\">\n",
    "\n",
    "[Adapted from here](https://amueller.github.io/COMS4995-s20/slides/aml-04-preprocessing/#37)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Training models with transformed data\n",
    "- We can now pass the `ColumnTransformer` object as a step in a pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['A+', 'not A+', 'not A+', 'A+', 'A+', 'not A+', 'A+', 'not A+',\n",
       "       'not A+', 'A+', 'A+', 'A+', 'A+', 'A+', 'not A+', 'not A+', 'A+',\n",
       "       'not A+', 'not A+', 'not A+', 'A+'], dtype=object)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a pipeline that applies ct to columns and then SVC to the resulting table\n",
    "pipe = make_pipeline(ct, SVC())\n",
    "pipe.fit(X, y)\n",
    "pipe.predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### True/False: `ColumnTransformer` \n",
    "\n",
    "1. You could carry out cross-validation by passing a `ColumnTransformer` object to `cross_validate`. \n",
    "2. After applying column transformer, the order of the columns in the transformed data has to be the same as the order of the columns in the original data. \n",
    "3. After applying a column transformer, the transformed data is always going to be of different shape than the original data. \n",
    "4. When you call `fit_transform` on a `ColumnTransformer` object, you get a numpy ndarray. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### What transformations on what columns? \n",
    "Consider the feature columns below. \n",
    "\n",
    "- What transformations would you apply on each column? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| colour  | location  |  shape |  water_content | weight |\n",
    "|-----|-----|-----|-----|-----|\n",
    "|   red   |   canada  |    NaN  |       84  |        100 |\n",
    "| yellow  |   mexico  |   long  |       75  |        120 |\n",
    "| orange  |   spain   |    NaN  |       90  |        NaN |\n",
    "| magenta |    china  |    round|       NaN |        600 |\n",
    "| purple  |  austria  |    NaN  |       80  |        115 |\n",
    "| purple  |  turkey   |   oval  |       78  |        340 |\n",
    "| green   |  mexico   |   oval  |       83  |        NaN |\n",
    "| blue    | canada     | round  |      73   |       535  |\n",
    "| brown   |  china     |   NaN  |       NaN |       1743 | \n",
    "| yellow  |  mexico    |  oval  |       83  |        265 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## More on feature transformations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Multiple transformations in a transformer\n",
    "\n",
    "- Recall that `lab2` has missing values. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- So we would like to apply more than one transformations on it: imputation and scaling.  \n",
    "- We can treat `lab2` separately, but we can also include it into `numeric_feats` and apply both transformations on all numeric columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\"class_attendance\", \"enjoy_course\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- To apply more than one transformations we can define a pipeline inside a column transformer to chain different transformations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab2',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology',\n",
       " 'ml_experience']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    "    + passthrough_feats\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "      <th>ml_experience</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>-1.103439</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   university_years      lab1      lab2      lab3      lab4     quiz1  \\\n",
       "0         -0.093454  0.358913  0.893260 -0.217334  0.362700  0.840028   \n",
       "1         -1.074719  0.590827  0.294251 -0.614206 -0.855972  0.712198   \n",
       "2         -0.093454 -1.264480 -0.704099 -0.316552 -1.312974 -0.693936   \n",
       "3         -0.093454  0.242957  0.000000  0.576409  0.362700  0.456537   \n",
       "4          0.887812 -1.380436 -1.103439  0.377973  0.515034 -0.054784   \n",
       "\n",
       "   major_Biology  major_Computer Science  major_Economics  major_Linguistics  \\\n",
       "0            0.0                     1.0              0.0                0.0   \n",
       "1            0.0                     0.0              0.0                0.0   \n",
       "2            0.0                     0.0              0.0                0.0   \n",
       "3            0.0                     0.0              0.0                0.0   \n",
       "4            0.0                     0.0              0.0                0.0   \n",
       "\n",
       "   major_Mathematics  major_Mechanical Engineering  major_Physics  \\\n",
       "0                0.0                           0.0            0.0   \n",
       "1                0.0                           1.0            0.0   \n",
       "2                1.0                           0.0            0.0   \n",
       "3                1.0                           0.0            0.0   \n",
       "4                0.0                           0.0            0.0   \n",
       "\n",
       "   major_Psychology  ml_experience  \n",
       "0               0.0            1.0  \n",
       "1               0.0            1.0  \n",
       "2               0.0            0.0  \n",
       "3               0.0            0.0  \n",
       "4               1.0            0.0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_transformed, columns=column_names).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `sklearn` `set_config`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- With multiple transformations in a column transformer, it can get tricky to keep track of everything happening inside it.  \n",
    "- We can use `set_config` to display a diagram of this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "\n",
    "set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 {color: black;background-color: white;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 pre{padding: 0;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-toggleable {background-color: white;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-estimator:hover {background-color: #d4ebff;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-item {z-index: 1;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-parallel-item:only-child::after {width: 0;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-fbe0acdb-5f65-46c0-8a37-aa6185006041 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-fbe0acdb-5f65-46c0-8a37-aa6185006041\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"1e254989-1e3d-4bf0-b3b1-4f146239af49\" type=\"checkbox\" ><label for=\"1e254989-1e3d-4bf0-b3b1-4f146239af49\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"5b0e7788-bf95-4ade-a9f2-9ae2eddbe90b\" type=\"checkbox\" ><label for=\"5b0e7788-bf95-4ade-a9f2-9ae2eddbe90b\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"56a73e41-fe95-48d0-b2ee-a1e7e539aaef\" type=\"checkbox\" ><label for=\"56a73e41-fe95-48d0-b2ee-a1e7e539aaef\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8e3d20df-c944-40be-be2c-e12e7984534f\" type=\"checkbox\" ><label for=\"8e3d20df-c944-40be-be2c-e12e7984534f\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"c4229816-e9c7-4297-816a-4b0b217d8385\" type=\"checkbox\" ><label for=\"c4229816-e9c7-4297-816a-4b0b217d8385\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dbeb4cf9-2854-4d2a-a932-2bae7e772ded\" type=\"checkbox\" ><label for=\"dbeb4cf9-2854-4d2a-a932-2bae7e772ded\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7e3b81a9-6eb3-4085-a50a-50ba66dc0de7\" type=\"checkbox\" ><label for=\"7e3b81a9-6eb3-4085-a50a-50ba66dc0de7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a05063f3-0caf-43d5-9d22-6d89897383a0\" type=\"checkbox\" ><label for=\"a05063f3-0caf-43d5-9d22-6d89897383a0\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e6cbd443-c739-4f8c-8867-a309ee3c6ed9\" type=\"checkbox\" ><label for=\"e6cbd443-c739-4f8c-8867-a309ee3c6ed9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"157fa3bc-6f15-4eb0-af25-89e633fa45e2\" type=\"checkbox\" ><label for=\"157fa3bc-6f15-4eb0-af25-89e633fa45e2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop',\n",
       "                                 ['class_attendance', 'enjoy_course'])])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ColumnTransformer(transformers=[('pipeline',\n",
      "                                 Pipeline(steps=[('simpleimputer',\n",
      "                                                  SimpleImputer()),\n",
      "                                                 ('standardscaler',\n",
      "                                                  StandardScaler())]),\n",
      "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
      "                                  'lab4', 'quiz1']),\n",
      "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
      "                                ('passthrough', 'passthrough',\n",
      "                                 ['ml_experience']),\n",
      "                                ('drop', 'drop',\n",
      "                                 ['class_attendance', 'enjoy_course'])])\n"
     ]
    }
   ],
   "source": [
    "print(ct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Incorporating ***ordinal*** feature `class_attendance` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `class_attendance` column is different than the `major` column in that there is some ordering of the values. \n",
    "    - Excellent > Good > Average > Poor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try applying `OrdinalEncoder` on `class_attendance` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_toy = X[[\"class_attendance\"]]\n",
    "enc = OrdinalEncoder()\n",
    "enc.fit(X_toy)\n",
    "X_toy_ord = enc.transform(X_toy)\n",
    "X_toy_ord_df = pd.DataFrame(\n",
    "    data=X_toy_ord,\n",
    "    columns=[\"class_attendance_ord\"],\n",
    "    index=X_toy.index,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>class_attendance_ord</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Poor</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Good</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Good</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Poor</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  class_attendance  class_attendance_ord\n",
       "0        Excellent                   1.0\n",
       "1          Average                   0.0\n",
       "2             Poor                   3.0\n",
       "3        Excellent                   1.0\n",
       "4             Good                   2.0\n",
       "5             Good                   2.0\n",
       "6        Excellent                   1.0\n",
       "7             Poor                   3.0\n",
       "8          Average                   0.0\n",
       "9          Average                   0.0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_toy.join(X_toy_ord_df).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What's the problem here? \n",
    "    - The encoder doesn't know the order. \n",
    "- We can examine unique categories **manually, order them based on our intuitions**, and then provide this human knowledge to the transformer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What are the unique categories of `class_attendance`? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Excellent', 'Average', 'Poor', 'Good'], dtype=object)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_toy[\"class_attendance\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's order them manually. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class_attendance_levels = [\"Poor\", \"Average\", \"Good\", \"Excellent\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***Note*** that if you use the reverse order of the categories, it wouldn't matter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's make sure that we have included all categories in our manual ordering.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert set(class_attendance_levels) == set(X_toy[\"class_attendance\"].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array(['Poor', 'Average', 'Good', 'Excellent'], dtype=object)]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>class_attendance_ord</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Poor</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Poor</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  class_attendance  class_attendance_ord\n",
       "0        Excellent                     3\n",
       "1          Average                     1\n",
       "2             Poor                     0\n",
       "3        Excellent                     3\n",
       "4             Good                     2\n",
       "5             Good                     2\n",
       "6        Excellent                     3\n",
       "7             Poor                     0\n",
       "8          Average                     1\n",
       "9          Average                     1"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "oe = OrdinalEncoder(categories=[class_attendance_levels], dtype=int)\n",
    "oe.fit(X_toy[[\"class_attendance\"]])\n",
    "ca_ord = oe.transform(X_toy[[\"class_attendance\"]])\n",
    "ca_ord_df = pd.DataFrame(\n",
    "    data=ca_ord, columns=[\"class_attendance_ord\"], index=X_toy.index\n",
    ")\n",
    "print(oe.categories_)\n",
    "X_toy.join(ca_ord_df).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The encoded categories are looking better now! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### More than one ordinal columns?\n",
    "\n",
    "- We can pass the manually ordered categories when we create an `OrdinalEncoder` object as a list of lists. \n",
    "- If you have more than one ordinal columns\n",
    "    - manually create a list of ordered categories for each column\n",
    "    - pass a list of lists to `OrdinalEncoder`, where each inner list corresponds to manually created list of ordered categories for a corresponding ordinal column. \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let's incorporate ordinal encoding of `class_attendance` in our column transformer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "ordinal_feats = [\"class_attendance\"]  # apply ordinal encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\"enjoy_course\"]  # do not include these features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 {color: black;background-color: white;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 pre{padding: 0;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-toggleable {background-color: white;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-estimator:hover {background-color: #d4ebff;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-item {z-index: 1;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-parallel-item:only-child::after {width: 0;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-5fcbf7b3-23c8-4a88-909f-bda096a32884 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-5fcbf7b3-23c8-4a88-909f-bda096a32884\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8c204ccc-d62b-4acb-ab5c-34670a1d27fc\" type=\"checkbox\" ><label for=\"8c204ccc-d62b-4acb-ab5c-34670a1d27fc\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"5a439966-70b3-4c18-8a08-6e4b31d69f5a\" type=\"checkbox\" ><label for=\"5a439966-70b3-4c18-8a08-6e4b31d69f5a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"fc373b34-166d-49a0-83c7-838fc849a4fb\" type=\"checkbox\" ><label for=\"fc373b34-166d-49a0-83c7-838fc849a4fb\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7bd0c237-f543-415a-9d33-c06223e56903\" type=\"checkbox\" ><label for=\"7bd0c237-f543-415a-9d33-c06223e56903\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"c20a86ca-85ec-4e33-b74d-30ae44c08fda\" type=\"checkbox\" ><label for=\"c20a86ca-85ec-4e33-b74d-30ae44c08fda\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7c41f74a-729d-417f-9124-ab638240af37\" type=\"checkbox\" ><label for=\"7c41f74a-729d-417f-9124-ab638240af37\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dee89fa4-7b5f-4aa2-9484-e858e124058e\" type=\"checkbox\" ><label for=\"dee89fa4-7b5f-4aa2-9484-e858e124058e\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"bf0facba-d91f-4f2e-92b2-3081ac07de63\" type=\"checkbox\" ><label for=\"bf0facba-d91f-4f2e-92b2-3081ac07de63\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"0fa2c61b-3de1-446e-95b2-01a34d5c7e99\" type=\"checkbox\" ><label for=\"0fa2c61b-3de1-446e-95b2-01a34d5c7e99\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4c48e7f4-fa6d-4704-8741-049934514252\" type=\"checkbox\" ><label for=\"4c48e7f4-fa6d-4704-8741-049934514252\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"041e1399-b402-4d4d-bb4b-015394c0caeb\" type=\"checkbox\" ><label for=\"041e1399-b402-4d4d-bb4b-015394c0caeb\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b34a5e62-41ee-4c5f-ae81-bf34ff09612b\" type=\"checkbox\" ><label for=\"b34a5e62-41ee-4c5f-ae81-bf34ff09612b\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop', ['enjoy_course'])])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X_transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab2',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology',\n",
       " 'class_attendance',\n",
       " 'ml_experience']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    "    + ordinal_feats\n",
    "    + passthrough_feats\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>ml_experience</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>-1.103439</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-2.192133</td>\n",
       "      <td>-3.100139</td>\n",
       "      <td>-1.804821</td>\n",
       "      <td>-2.226978</td>\n",
       "      <td>-1.844409</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.032566</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.278755</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-1.705603</td>\n",
       "      <td>-1.465308</td>\n",
       "      <td>-1.333088</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.938697</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>-1.008306</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-1.303109</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>-0.504429</td>\n",
       "      <td>0.874062</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-0.949597</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-2.055985</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>1.492270</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>1.581372</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-1.972240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-1.904039</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-1.612349</td>\n",
       "      <td>0.493921</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-0.336826</td>\n",
       "      <td>0.094581</td>\n",
       "      <td>-2.102474</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>-0.438275</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.080319</td>\n",
       "      <td>-1.160640</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.822740</td>\n",
       "      <td>0.693590</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>1.124370</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    university_years      lab1      lab2      lab3      lab4     quiz1  \\\n",
       "0          -0.093454  0.358913  0.893260 -0.217334  0.362700  0.840028   \n",
       "1          -1.074719  0.590827  0.294251 -0.614206 -0.855972  0.712198   \n",
       "2          -0.093454 -1.264480 -0.704099 -0.316552 -1.312974 -0.693936   \n",
       "3          -0.093454  0.242957  0.000000  0.576409  0.362700  0.456537   \n",
       "4           0.887812 -1.380436 -1.103439  0.377973  0.515034 -0.054784   \n",
       "5           1.869077 -2.192133 -3.100139 -1.804821 -2.226978 -1.844409   \n",
       "6           0.887812 -1.032566 -0.105089  0.278755 -0.094302  0.712198   \n",
       "7          -0.093454  0.706783  0.893260 -1.705603 -1.465308 -1.333088   \n",
       "8          -1.074719  0.938697  0.294251  0.774844 -1.008306 -0.693936   \n",
       "9           0.887812  0.706783 -1.303109  0.774844  0.819702 -0.054784   \n",
       "10         -0.093454  1.054653 -0.504429  0.874062  0.972036 -0.949597   \n",
       "11         -2.055985  0.706783 -0.105089  0.675627  0.515034 -0.054784   \n",
       "12         -1.074719  1.054653  1.492270  0.973280  1.581372  1.862671   \n",
       "13          0.887812  0.706783  1.092930  0.973280  0.972036  1.862671   \n",
       "14         -0.093454  0.706783  0.294251  0.675627  0.972036 -1.972240   \n",
       "15         -0.093454  0.358913 -0.704099 -1.904039  0.819702  0.840028   \n",
       "16          1.869077 -1.612349  0.493921  0.675627 -0.398970 -0.054784   \n",
       "17         -0.093454 -0.336826  0.094581 -2.102474 -0.398970  0.200876   \n",
       "18         -1.074719  0.242957  0.000000  0.377973 -0.094302 -0.438275   \n",
       "19         -1.074719 -1.380436  1.092930  0.080319 -1.160640  0.456537   \n",
       "20          0.887812  0.822740  0.693590  0.576409  1.124370  0.200876   \n",
       "\n",
       "    major_Biology  major_Computer Science  major_Economics  major_Linguistics  \\\n",
       "0             0.0                     1.0              0.0                0.0   \n",
       "1             0.0                     0.0              0.0                0.0   \n",
       "2             0.0                     0.0              0.0                0.0   \n",
       "3             0.0                     0.0              0.0                0.0   \n",
       "4             0.0                     0.0              0.0                0.0   \n",
       "5             0.0                     0.0              1.0                0.0   \n",
       "6             0.0                     1.0              0.0                0.0   \n",
       "7             0.0                     0.0              0.0                0.0   \n",
       "8             0.0                     0.0              0.0                1.0   \n",
       "9             0.0                     0.0              0.0                0.0   \n",
       "10            0.0                     0.0              0.0                0.0   \n",
       "11            0.0                     0.0              0.0                0.0   \n",
       "12            0.0                     0.0              0.0                0.0   \n",
       "13            0.0                     0.0              0.0                0.0   \n",
       "14            0.0                     0.0              0.0                0.0   \n",
       "15            0.0                     1.0              0.0                0.0   \n",
       "16            0.0                     1.0              0.0                0.0   \n",
       "17            0.0                     0.0              1.0                0.0   \n",
       "18            1.0                     0.0              0.0                0.0   \n",
       "19            0.0                     0.0              0.0                0.0   \n",
       "20            0.0                     0.0              0.0                1.0   \n",
       "\n",
       "    major_Mathematics  major_Mechanical Engineering  major_Physics  \\\n",
       "0                 0.0                           0.0            0.0   \n",
       "1                 0.0                           1.0            0.0   \n",
       "2                 1.0                           0.0            0.0   \n",
       "3                 1.0                           0.0            0.0   \n",
       "4                 0.0                           0.0            0.0   \n",
       "5                 0.0                           0.0            0.0   \n",
       "6                 0.0                           0.0            0.0   \n",
       "7                 0.0                           1.0            0.0   \n",
       "8                 0.0                           0.0            0.0   \n",
       "9                 1.0                           0.0            0.0   \n",
       "10                0.0                           0.0            0.0   \n",
       "11                0.0                           0.0            1.0   \n",
       "12                0.0                           0.0            1.0   \n",
       "13                0.0                           1.0            0.0   \n",
       "14                1.0                           0.0            0.0   \n",
       "15                0.0                           0.0            0.0   \n",
       "16                0.0                           0.0            0.0   \n",
       "17                0.0                           0.0            0.0   \n",
       "18                0.0                           0.0            0.0   \n",
       "19                0.0                           0.0            0.0   \n",
       "20                0.0                           0.0            0.0   \n",
       "\n",
       "    major_Psychology  class_attendance  ml_experience  \n",
       "0                0.0               3.0            1.0  \n",
       "1                0.0               1.0            1.0  \n",
       "2                0.0               0.0            0.0  \n",
       "3                0.0               3.0            0.0  \n",
       "4                1.0               2.0            0.0  \n",
       "5                0.0               2.0            1.0  \n",
       "6                0.0               3.0            1.0  \n",
       "7                0.0               0.0            0.0  \n",
       "8                0.0               1.0            0.0  \n",
       "9                0.0               1.0            1.0  \n",
       "10               1.0               2.0            0.0  \n",
       "11               0.0               1.0            1.0  \n",
       "12               0.0               3.0            1.0  \n",
       "13               0.0               3.0            0.0  \n",
       "14               0.0               0.0            0.0  \n",
       "15               0.0               2.0            1.0  \n",
       "16               0.0               1.0            0.0  \n",
       "17               0.0               1.0            1.0  \n",
       "18               0.0               2.0            1.0  \n",
       "19               1.0               0.0            0.0  \n",
       "20               0.0               3.0            1.0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_transformed, columns=column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Dealing with unknown categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How does `OneHotEncoder` deal with unknown categories? Let's see an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>science</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arts</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arts</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subject  group\n",
       "0  science     10\n",
       "1     arts     30\n",
       "2     arts     20"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_toy = [['science', 10], ['arts', 30], ['arts', 20]]\n",
    "columns=['subject', 'group']\n",
    "pd.DataFrame(X_toy, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe = OneHotEncoder(handle_unknown='error')  # default value for handle_unknown is 'error'\n",
    "ohe.fit(X_toy);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['subject_arts', 'subject_science', 'group_10', 'group_20', 'group_30']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns_ohe = ohe.get_feature_names_out(['subject', 'group']).tolist()\n",
    "columns_ohe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array(['arts', 'science'], dtype=object), array([10, 20, 30], dtype=object)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe.categories_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 1., 0., 0.],\n",
       "       [0., 1., 0., 0., 1.]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex1 = ohe.transform([['arts', 10], ['science', 30]]).toarray()\n",
    "ex1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject_arts</th>\n",
       "      <th>subject_science</th>\n",
       "      <th>group_10</th>\n",
       "      <th>group_20</th>\n",
       "      <th>group_30</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subject_arts  subject_science  group_10  group_20  group_30\n",
       "0           1.0              0.0       1.0       0.0       0.0\n",
       "1           0.0              1.0       0.0       0.0       1.0"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(ex1, columns=columns_ohe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ex2 = ohe.transform([['arts', 10], ['science', 4]]).toarray()\n",
    "\n",
    "# This would give an error:\n",
    "# ValueError: Found unknown categories [4] in column 1 during transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe = OneHotEncoder(handle_unknown='ignore')  # now use 'ignore' instead of 'error'\n",
    "ohe.fit(X_toy);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 1., 0., 0.],\n",
       "       [0., 1., 0., 0., 0.]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2 = ohe.transform([['arts', 10], ['science', 4]]).toarray()\n",
    "ex2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject_arts</th>\n",
       "      <th>subject_science</th>\n",
       "      <th>group_10</th>\n",
       "      <th>group_20</th>\n",
       "      <th>group_30</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subject_arts  subject_science  group_10  group_20  group_30\n",
       "0           1.0              0.0       1.0       0.0       0.0\n",
       "1           0.0              1.0       0.0       0.0       0.0"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(ex2, columns=columns_ohe)  # all \"group\" columns are 0 for value 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['science', 10],\n",
       "       ['arts', None]], dtype=object)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex3 = ohe.inverse_transform([[0, 1, 1, 0, 0], [1, 0, 0, 0, 0]])\n",
    "ex3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>science</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arts</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subject group\n",
       "0  science    10\n",
       "1     arts  None"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(ex3, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['science', 30],\n",
       "       [None, 20]], dtype=object)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex4 = ohe.inverse_transform([[0, 1, 0, 0, 1], [0, 0, 0, 1, 0]])\n",
    "ex4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>science</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>None</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subject group\n",
       "0  science    30\n",
       "1     None    20"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(ex4, columns=columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we know the possible categories beforehand? We can **specify categories** ahead of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe = OneHotEncoder(handle_unknown='error', categories=[['arts', 'science'], [10, 20, 30, 4]])\n",
    "ohe.fit(X_toy);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though `handle_unknown='error'`, `ex2` does not give error anymore because `categories` are known."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 1., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0., 1.]])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2 = ohe.transform([['arts', 10], ['science', 4]]).toarray()\n",
    "ex2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject_arts</th>\n",
       "      <th>subject_science</th>\n",
       "      <th>group_10</th>\n",
       "      <th>group_20</th>\n",
       "      <th>group_30</th>\n",
       "      <th>group_4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subject_arts  subject_science  group_10  group_20  group_30  group_4\n",
       "0           1.0              0.0       1.0       0.0       0.0      0.0\n",
       "1           0.0              1.0       0.0       0.0       0.0      1.0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns_ohe = ohe.get_feature_names_out(['subject', 'group']).tolist()\n",
    "pd.DataFrame(ex2, columns=columns_ohe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "####  Dealing with unknown categories in `cross_validate`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's create a pipeline with the column transformer and pass it to `cross_validate`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 {color: black;background-color: white;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 pre{padding: 0;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-toggleable {background-color: white;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-estimator:hover {background-color: #d4ebff;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-item {z-index: 1;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-parallel-item:only-child::after {width: 0;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-d7503e2f-1aee-4ab7-ab5d-c8388f980e65\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b5f25212-ab0b-4890-a2d1-e46a7de74f59\" type=\"checkbox\" ><label for=\"b5f25212-ab0b-4890-a2d1-e46a7de74f59\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"64830baf-47c1-4c1b-8a1f-fc80fa837199\" type=\"checkbox\" ><label for=\"64830baf-47c1-4c1b-8a1f-fc80fa837199\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"56d43469-c68d-4dda-9d8e-ebcdb8581d81\" type=\"checkbox\" ><label for=\"56d43469-c68d-4dda-9d8e-ebcdb8581d81\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"49840e5a-5b94-4d84-9934-7f3567b76af9\" type=\"checkbox\" ><label for=\"49840e5a-5b94-4d84-9934-7f3567b76af9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"ef4e8811-1944-4847-9c89-2c381cee4add\" type=\"checkbox\" ><label for=\"ef4e8811-1944-4847-9c89-2c381cee4add\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"55fce527-97a6-4509-83d4-486a4764b715\" type=\"checkbox\" ><label for=\"55fce527-97a6-4509-83d4-486a4764b715\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e24c6173-2896-47e9-9876-8acd02b45c8e\" type=\"checkbox\" ><label for=\"e24c6173-2896-47e9-9876-8acd02b45c8e\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"2c40185e-d972-48fc-ab38-7eedf89c578d\" type=\"checkbox\" ><label for=\"2c40185e-d972-48fc-ab38-7eedf89c578d\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"aab3ec6f-db0a-417a-9f7c-a731bda14330\" type=\"checkbox\" ><label for=\"aab3ec6f-db0a-417a-9f7c-a731bda14330\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d70a5f6d-4ffe-41c5-b0ca-2ea5996a1e4b\" type=\"checkbox\" ><label for=\"d70a5f6d-4ffe-41c5-b0ca-2ea5996a1e4b\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"676bad9c-8b3a-4225-bcf4-75bb1e5a98e2\" type=\"checkbox\" ><label for=\"676bad9c-8b3a-4225-bcf4-75bb1e5a98e2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d2708551-a893-49ad-a358-6da521cb8cc6\" type=\"checkbox\" ><label for=\"d2708551-a893-49ad-a358-6da521cb8cc6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop', ['enjoy_course'])])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:770: UserWarning: Scoring failed. The score on this train-test partition for these parameters will be set to nan. Details: \n",
      "Traceback (most recent call last):\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/model_selection/_validation.py\", line 761, in _score\n",
      "    scores = scorer(estimator, X_test, y_test)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/metrics/_scorer.py\", line 418, in _passthrough_scorer\n",
      "    return estimator.score(*args, **kwargs)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/metaestimators.py\", line 113, in <lambda>\n",
      "    out = lambda *args, **kwargs: self.fn(obj, *args, **kwargs)  # noqa\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/pipeline.py\", line 707, in score\n",
      "    Xt = transform.transform(Xt)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/compose/_column_transformer.py\", line 748, in transform\n",
      "    Xs = self._fit_transform(\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/compose/_column_transformer.py\", line 606, in _fit_transform\n",
      "    return Parallel(n_jobs=self.n_jobs)(\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 1044, in __call__\n",
      "    while self.dispatch_one_batch(iterator):\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 859, in dispatch_one_batch\n",
      "    self._dispatch(tasks)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 777, in _dispatch\n",
      "    job = self._backend.apply_async(batch, callback=cb)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/_parallel_backends.py\", line 208, in apply_async\n",
      "    result = ImmediateResult(func)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/_parallel_backends.py\", line 572, in __init__\n",
      "    self.results = batch()\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 262, in __call__\n",
      "    return [func(*args, **kwargs)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 262, in <listcomp>\n",
      "    return [func(*args, **kwargs)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/fixes.py\", line 216, in __call__\n",
      "    return self.function(*args, **kwargs)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/pipeline.py\", line 876, in _transform_one\n",
      "    res = transformer.transform(X)\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/preprocessing/_encoders.py\", line 509, in transform\n",
      "    X_int, X_mask = self._transform(\n",
      "  File \"/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/preprocessing/_encoders.py\", line 142, in _transform\n",
      "    raise ValueError(msg)\n",
      "ValueError: Found unknown categories ['Biology'] in column 0 during transform\n",
      "\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.008816</td>\n",
       "      <td>0.005662</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.937500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.008196</td>\n",
       "      <td>0.004937</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.007402</td>\n",
       "      <td>0.003597</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.006360</td>\n",
       "      <td>0.004390</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.006452</td>\n",
       "      <td>0.006439</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.008816    0.005662        1.00     0.937500\n",
       "1  0.008196    0.004937        1.00     0.941176\n",
       "2  0.007402    0.003597        0.50     1.000000\n",
       "3  0.006360    0.004390        0.75     0.941176\n",
       "4  0.006452    0.006439         NaN     1.000000"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe, X, y, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- What's going on here??\n",
    "- Let's look at the error message:\n",
    "  - `ValueError: Found unknown categories ['Biology'] in column 0 during transform`\n",
    "  - Same error that we got in the `OneHotEncoder` example above (`ex2`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Computer Science          4\n",
       "Mathematics               4\n",
       "Mechanical Engineering    3\n",
       "Psychology                3\n",
       "Economics                 2\n",
       "Linguistics               2\n",
       "Physics                   2\n",
       "Biology                   1\n",
       "Name: major, dtype: int64"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[\"major\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- There is only <u>one</u> instance of Biology.\n",
    "- During cross-validation, this is getting <u>put into the validation split</u>.\n",
    "- By default, `OneHotEncoder` throws an error because you might want to know about this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Simplest fix:\n",
    "- Pass `handle_unknown=\"ignore\"` argument to `OneHotEncoder`\n",
    "- It creates a row with all zeros (as we saw in the example above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (\n",
    "        OneHotEncoder(handle_unknown=\"ignore\"),\n",
    "        categorical_feats,\n",
    "    ),  # OHE on categorical features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 {color: black;background-color: white;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 pre{padding: 0;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-toggleable {background-color: white;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-estimator:hover {background-color: #d4ebff;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-item {z-index: 1;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-parallel-item:only-child::after {width: 0;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-f4b95e3a-3d09-4fc1-b519-6351540176a7 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-f4b95e3a-3d09-4fc1-b519-6351540176a7\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"269321c9-648f-4c8a-a5a0-eeb6e1f8305b\" type=\"checkbox\" ><label for=\"269321c9-648f-4c8a-a5a0-eeb6e1f8305b\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d8678312-fb18-4c0c-98c0-0db8b91f33cc\" type=\"checkbox\" ><label for=\"d8678312-fb18-4c0c-98c0-0db8b91f33cc\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"17ad51d9-8b0a-43db-b300-f8d699ad27c3\" type=\"checkbox\" ><label for=\"17ad51d9-8b0a-43db-b300-f8d699ad27c3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"37216e13-cb9a-4a2f-af8a-d4c16800d348\" type=\"checkbox\" ><label for=\"37216e13-cb9a-4a2f-af8a-d4c16800d348\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"fd55ceaa-db85-4fc1-bf5f-216506c7229c\" type=\"checkbox\" ><label for=\"fd55ceaa-db85-4fc1-bf5f-216506c7229c\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7410eb82-986d-4ef2-8ff7-5eaab6f56c31\" type=\"checkbox\" ><label for=\"7410eb82-986d-4ef2-8ff7-5eaab6f56c31\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e97ee7a7-f5e7-4dee-be2d-b72c5b4da9ec\" type=\"checkbox\" ><label for=\"e97ee7a7-f5e7-4dee-be2d-b72c5b4da9ec\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"98900c19-f50d-4f5f-9fa7-f3e3c7153fea\" type=\"checkbox\" ><label for=\"98900c19-f50d-4f5f-9fa7-f3e3c7153fea\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"acb54627-e5c7-4c2e-b9de-ebc5d2be6d94\" type=\"checkbox\" ><label for=\"acb54627-e5c7-4c2e-b9de-ebc5d2be6d94\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"985dfdc8-e8ee-4db3-adfe-0ec59d4a35ce\" type=\"checkbox\" ><label for=\"985dfdc8-e8ee-4db3-adfe-0ec59d4a35ce\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"89243d45-ef8e-4c0c-bbf6-b37b6157314d\" type=\"checkbox\" ><label for=\"89243d45-ef8e-4c0c-bbf6-b37b6157314d\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"c145f02f-1c0f-4ea5-b3e6-31b91873f36c\" type=\"checkbox\" ><label for=\"c145f02f-1c0f-4ea5-b3e6-31b91873f36c\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['major']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('drop', 'drop', ['enjoy_course'])])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.008285</td>\n",
       "      <td>0.006410</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.937500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.008545</td>\n",
       "      <td>0.004508</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.006607</td>\n",
       "      <td>0.003485</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.006461</td>\n",
       "      <td>0.005872</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.007575</td>\n",
       "      <td>0.004189</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.008285    0.006410        1.00     0.937500\n",
       "1  0.008545    0.004508        1.00     0.941176\n",
       "2  0.006607    0.003485        0.50     1.000000\n",
       "3  0.006461    0.005872        0.75     0.941176\n",
       "4  0.007575    0.004189        0.75     1.000000"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe, X, y, cv=5, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- With this approach, all unknown categories will be represented with all zeros and cross-validation is running OK now. \n",
    "\n",
    "Ask yourself the following questions when you work with categorical variables   \n",
    "- Do you want this behaviour? \n",
    "- Are you expecting to get many unknown categories? Do you want to be able to distinguish between them?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Cases where it's OK to break the golden rule \n",
    "\n",
    "We saw above that if we know categories beforehand we can specify them to `OneHotEncoder` to avoid errors during `cross_over` assessments. However, wouldn't tha be ***breaking the golden*** rule?\n",
    "\n",
    "When we know the categories in advance and this is one of the cases where it might be OK to violate the golden rule and get a list of all possible values for the categorical variable. \n",
    "\n",
    "For example, if it's some fix number of categories. E.g., if it's something like:\n",
    "  -  provinces in Canada or \n",
    "  -  majors taught at UBC. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Categorical features with only two possible categories\n",
    "\n",
    "- Sometimes you have features with only two possible categories. \n",
    "- If we apply `OheHotEncoder` on such columns, it'll **create two columns, which seems wasteful**, as we could represent all information in the column in just one column with say 0's and 1's with presence of absence of one of one of the categories.\n",
    "- You can pass `drop=\"if_binary\"` argument to `OneHotEncoder` in order to create only one column in such scenario. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    yes\n",
       "1    yes\n",
       "2    yes\n",
       "3     no\n",
       "4    yes\n",
       "Name: enjoy_course, dtype: object"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[\"enjoy_course\"].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>enjoy_course_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  enjoy_course_enc\n",
       "0          yes                 1\n",
       "1          yes                 1\n",
       "2          yes                 1\n",
       "3           no                 0\n",
       "4          yes                 1\n",
       "5           no                 0\n",
       "6          yes                 1\n",
       "7           no                 0\n",
       "8           no                 0\n",
       "9          yes                 1"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe_enc = OneHotEncoder(drop=\"if_binary\", dtype=int, sparse=False)\n",
    "ohe_enc.fit(X[[\"enjoy_course\"]])\n",
    "transformed = ohe_enc.transform(X[[\"enjoy_course\"]])\n",
    "df = pd.DataFrame(data=transformed, columns=[\"enjoy_course_enc\"], index=X.index)\n",
    "X[[\"enjoy_course\"]].join(df).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "ordinal_feats = [\"class_attendance\"]  # apply ordinal encoding\n",
    "binary_feats = [\"enjoy_course\"]  # apply one-hot encoding with drop=\"if_binary\"\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (\n",
    "        OneHotEncoder(handle_unknown=\"ignore\"),\n",
    "        categorical_feats,\n",
    "    ),  # OHE on categorical features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\n",
    "        OneHotEncoder(drop=\"if_binary\", dtype=int),\n",
    "        binary_feats,\n",
    "    ),  # OHE on categorical features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-d0a85361-ee37-4260-82d9-9b435813ff35 {color: black;background-color: white;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 pre{padding: 0;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-toggleable {background-color: white;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-estimator:hover {background-color: #d4ebff;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-item {z-index: 1;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-parallel-item:only-child::after {width: 0;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-d0a85361-ee37-4260-82d9-9b435813ff35 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-d0a85361-ee37-4260-82d9-9b435813ff35\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder-1&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;onehotencoder-2&#x27;,\n",
       "                                 OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                               dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;enjoy_course&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"12716b52-0b0f-4a7e-8062-0be4aba24af3\" type=\"checkbox\" ><label for=\"12716b52-0b0f-4a7e-8062-0be4aba24af3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;onehotencoder-1&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;onehotencoder-2&#x27;,\n",
       "                                 OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                               dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;enjoy_course&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"5a2d2e47-81f8-47ac-9de5-003fcfb5e1d2\" type=\"checkbox\" ><label for=\"5a2d2e47-81f8-47ac-9de5-003fcfb5e1d2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"64149c61-4f40-4bed-a3d4-ac52269486d2\" type=\"checkbox\" ><label for=\"64149c61-4f40-4bed-a3d4-ac52269486d2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7746ed10-7262-4285-8497-e0d176f8e4f1\" type=\"checkbox\" ><label for=\"7746ed10-7262-4285-8497-e0d176f8e4f1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"ca2fab8e-b4c9-4d08-ac79-de3c07386cc9\" type=\"checkbox\" ><label for=\"ca2fab8e-b4c9-4d08-ac79-de3c07386cc9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4f05c83c-66f4-4d09-8ca1-2dcd77fb50e6\" type=\"checkbox\" ><label for=\"4f05c83c-66f4-4d09-8ca1-2dcd77fb50e6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"65bd3541-ce00-4d32-9f90-680a750653d1\" type=\"checkbox\" ><label for=\"65bd3541-ce00-4d32-9f90-680a750653d1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"10c7a802-aaef-4ed8-b8c6-dd10d3502cff\" type=\"checkbox\" ><label for=\"10c7a802-aaef-4ed8-b8c6-dd10d3502cff\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d696b133-27dc-4636-b248-391e703ee1d1\" type=\"checkbox\" ><label for=\"d696b133-27dc-4636-b248-391e703ee1d1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"34008a62-0fb3-469d-8ce7-edb37d6bd005\" type=\"checkbox\" ><label for=\"34008a62-0fb3-469d-8ce7-edb37d6bd005\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(drop=&#x27;if_binary&#x27;, dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"931eba99-e945-4c21-a4b5-cc077d9f2da2\" type=\"checkbox\" ><label for=\"931eba99-e945-4c21-a4b5-cc077d9f2da2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7e8390dd-4c19-4a43-95f4-9213630408d5\" type=\"checkbox\" ><label for=\"7e8390dd-4c19-4a43-95f4-9213630408d5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('onehotencoder-1',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['major']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('onehotencoder-2',\n",
       "                                 OneHotEncoder(drop='if_binary',\n",
       "                                               dtype=<class 'int'>),\n",
       "                                 ['enjoy_course']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience'])])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.010822</td>\n",
       "      <td>0.005830</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.008248</td>\n",
       "      <td>0.004571</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.008762</td>\n",
       "      <td>0.005081</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.008046</td>\n",
       "      <td>0.005495</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.007485</td>\n",
       "      <td>0.004781</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.010822    0.005830        1.00     1.000000\n",
       "1  0.008248    0.004571        1.00     0.941176\n",
       "2  0.008762    0.005081        0.50     1.000000\n",
       "3  0.008046    0.005495        1.00     0.941176\n",
       "4  0.007485    0.004781        0.75     1.000000"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe, X, y, cv=5, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***Note***\n",
    "Do not read too much into the scores, as we are running cross-validation on a very small dataset with 21 examples. The main point here is to show you how can we use `ColumnTransformer` to apply different transformations on different columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Break (5 min)\n",
    "\n",
    "![](img/eva-coffee.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `ColumnTransformer` on the California housing dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2948.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>113600.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>137500.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3355.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>1457.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>170100.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1634.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>129300.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5566.0</td>\n",
       "      <td>1465.0</td>\n",
       "      <td>6303.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>205000.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "6051     -117.75     34.04                22.0       2948.0           636.0   \n",
       "20113    -119.57     37.94                17.0        346.0           130.0   \n",
       "14289    -117.13     32.74                46.0       3355.0           768.0   \n",
       "13665    -117.31     34.02                18.0       1634.0           274.0   \n",
       "14471    -117.23     32.88                18.0       5566.0          1465.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "6051       2600.0       602.0         3.1250            113600.0   \n",
       "20113        51.0        20.0         3.4861            137500.0   \n",
       "14289      1457.0       708.0         2.6604            170100.0   \n",
       "13665       899.0       285.0         5.2139            129300.0   \n",
       "14471      6303.0      1458.0         1.8580            205000.0   \n",
       "\n",
       "      ocean_proximity  \n",
       "6051           INLAND  \n",
       "20113          INLAND  \n",
       "14289      NEAR OCEAN  \n",
       "13665          INLAND  \n",
       "14471      NEAR OCEAN  "
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df = pd.read_csv(\"data/housing.csv\")\n",
    "train_df, test_df = train_test_split(housing_df, test_size=0.1, random_state=123)\n",
    "\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Some column values are mean/median but some are not. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's add some new features to the dataset which could help predicting the target: `median_house_value`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "train_df = train_df.assign(\n",
    "    rooms_per_household=train_df[\"total_rooms\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    rooms_per_household=test_df[\"total_rooms\"] / test_df[\"households\"]\n",
    ")\n",
    "\n",
    "train_df = train_df.assign(\n",
    "    bedrooms_per_household=train_df[\"total_bedrooms\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    bedrooms_per_household=test_df[\"total_bedrooms\"] / test_df[\"households\"]\n",
    ")\n",
    "\n",
    "train_df = train_df.assign(\n",
    "    population_per_household=train_df[\"population\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    population_per_household=test_df[\"population\"] / test_df[\"households\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2948.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>113600.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>4.897010</td>\n",
       "      <td>1.056478</td>\n",
       "      <td>4.318937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>137500.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>17.300000</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>2.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3355.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>1457.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>170100.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>4.738701</td>\n",
       "      <td>1.084746</td>\n",
       "      <td>2.057910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1634.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>129300.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0.961404</td>\n",
       "      <td>3.154386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5566.0</td>\n",
       "      <td>1465.0</td>\n",
       "      <td>6303.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>205000.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>3.817558</td>\n",
       "      <td>1.004801</td>\n",
       "      <td>4.323045</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "6051     -117.75     34.04                22.0       2948.0           636.0   \n",
       "20113    -119.57     37.94                17.0        346.0           130.0   \n",
       "14289    -117.13     32.74                46.0       3355.0           768.0   \n",
       "13665    -117.31     34.02                18.0       1634.0           274.0   \n",
       "14471    -117.23     32.88                18.0       5566.0          1465.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "6051       2600.0       602.0         3.1250            113600.0   \n",
       "20113        51.0        20.0         3.4861            137500.0   \n",
       "14289      1457.0       708.0         2.6604            170100.0   \n",
       "13665       899.0       285.0         5.2139            129300.0   \n",
       "14471      6303.0      1458.0         1.8580            205000.0   \n",
       "\n",
       "      ocean_proximity  rooms_per_household  bedrooms_per_household  \\\n",
       "6051           INLAND             4.897010                1.056478   \n",
       "20113          INLAND            17.300000                6.500000   \n",
       "14289      NEAR OCEAN             4.738701                1.084746   \n",
       "13665          INLAND             5.733333                0.961404   \n",
       "14471      NEAR OCEAN             3.817558                1.004801   \n",
       "\n",
       "       population_per_household  \n",
       "6051                   4.318937  \n",
       "20113                  2.550000  \n",
       "14289                  2.057910  \n",
       "13665                  3.154386  \n",
       "14471                  4.323045  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's keep both numeric and categorical columns in the data.\n",
    "X_train = train_df.drop(columns=[\"median_house_value\"])\n",
    "y_train = train_df[\"median_house_value\"]\n",
    "\n",
    "X_test = test_df.drop(columns=[\"median_house_value\"])\n",
    "y_test = test_df[\"median_house_value\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer, make_column_transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>ocean_proximity</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2948.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>4.897010</td>\n",
       "      <td>1.056478</td>\n",
       "      <td>4.318937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>17.300000</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>2.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3355.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>1457.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>4.738701</td>\n",
       "      <td>1.084746</td>\n",
       "      <td>2.057910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1634.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0.961404</td>\n",
       "      <td>3.154386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5566.0</td>\n",
       "      <td>1465.0</td>\n",
       "      <td>6303.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>3.817558</td>\n",
       "      <td>1.004801</td>\n",
       "      <td>4.323045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9730</th>\n",
       "      <td>-121.74</td>\n",
       "      <td>36.79</td>\n",
       "      <td>16.0</td>\n",
       "      <td>3841.0</td>\n",
       "      <td>620.0</td>\n",
       "      <td>1799.0</td>\n",
       "      <td>611.0</td>\n",
       "      <td>4.3814</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.286416</td>\n",
       "      <td>1.014730</td>\n",
       "      <td>2.944354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14690</th>\n",
       "      <td>-117.09</td>\n",
       "      <td>32.80</td>\n",
       "      <td>36.0</td>\n",
       "      <td>2163.0</td>\n",
       "      <td>367.0</td>\n",
       "      <td>915.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>4.7188</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>6.008333</td>\n",
       "      <td>1.019444</td>\n",
       "      <td>2.541667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7938</th>\n",
       "      <td>-118.11</td>\n",
       "      <td>33.86</td>\n",
       "      <td>33.0</td>\n",
       "      <td>2389.0</td>\n",
       "      <td>410.0</td>\n",
       "      <td>1229.0</td>\n",
       "      <td>393.0</td>\n",
       "      <td>5.3889</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.078880</td>\n",
       "      <td>1.043257</td>\n",
       "      <td>3.127226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18365</th>\n",
       "      <td>-122.12</td>\n",
       "      <td>37.28</td>\n",
       "      <td>21.0</td>\n",
       "      <td>349.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>149.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>5.8691</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.232143</td>\n",
       "      <td>1.142857</td>\n",
       "      <td>2.660714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10931</th>\n",
       "      <td>-117.91</td>\n",
       "      <td>33.74</td>\n",
       "      <td>25.0</td>\n",
       "      <td>4273.0</td>\n",
       "      <td>965.0</td>\n",
       "      <td>2946.0</td>\n",
       "      <td>922.0</td>\n",
       "      <td>2.9926</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>4.634490</td>\n",
       "      <td>1.046638</td>\n",
       "      <td>3.195228</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "6051     -117.75     34.04                22.0       2948.0           636.0   \n",
       "20113    -119.57     37.94                17.0        346.0           130.0   \n",
       "14289    -117.13     32.74                46.0       3355.0           768.0   \n",
       "13665    -117.31     34.02                18.0       1634.0           274.0   \n",
       "14471    -117.23     32.88                18.0       5566.0          1465.0   \n",
       "9730     -121.74     36.79                16.0       3841.0           620.0   \n",
       "14690    -117.09     32.80                36.0       2163.0           367.0   \n",
       "7938     -118.11     33.86                33.0       2389.0           410.0   \n",
       "18365    -122.12     37.28                21.0        349.0            64.0   \n",
       "10931    -117.91     33.74                25.0       4273.0           965.0   \n",
       "\n",
       "       population  households  median_income ocean_proximity  \\\n",
       "6051       2600.0       602.0         3.1250          INLAND   \n",
       "20113        51.0        20.0         3.4861          INLAND   \n",
       "14289      1457.0       708.0         2.6604      NEAR OCEAN   \n",
       "13665       899.0       285.0         5.2139          INLAND   \n",
       "14471      6303.0      1458.0         1.8580      NEAR OCEAN   \n",
       "9730       1799.0       611.0         4.3814       <1H OCEAN   \n",
       "14690       915.0       360.0         4.7188      NEAR OCEAN   \n",
       "7938       1229.0       393.0         5.3889       <1H OCEAN   \n",
       "18365       149.0        56.0         5.8691       <1H OCEAN   \n",
       "10931      2946.0       922.0         2.9926       <1H OCEAN   \n",
       "\n",
       "       rooms_per_household  bedrooms_per_household  population_per_household  \n",
       "6051              4.897010                1.056478                  4.318937  \n",
       "20113            17.300000                6.500000                  2.550000  \n",
       "14289             4.738701                1.084746                  2.057910  \n",
       "13665             5.733333                0.961404                  3.154386  \n",
       "14471             3.817558                1.004801                  4.323045  \n",
       "9730              6.286416                1.014730                  2.944354  \n",
       "14690             6.008333                1.019444                  2.541667  \n",
       "7938              6.078880                1.043257                  3.127226  \n",
       "18365             6.232143                1.142857                  2.660714  \n",
       "10931             4.634490                1.046638                  3.195228  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['longitude', 'latitude', 'housing_median_age', 'total_rooms',\n",
       "       'total_bedrooms', 'population', 'households', 'median_income',\n",
       "       'ocean_proximity', 'rooms_per_household', 'bedrooms_per_household',\n",
       "       'population_per_household'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Identify the categorical and numeric columns\n",
    "numeric_features = [\n",
    "    \"longitude\",\n",
    "    \"latitude\",\n",
    "    \"housing_median_age\",\n",
    "    \"total_rooms\",\n",
    "    \"total_bedrooms\",\n",
    "    \"population\",\n",
    "    \"households\",\n",
    "    \"median_income\",\n",
    "    \"rooms_per_household\",\n",
    "    \"bedrooms_per_household\",\n",
    "    \"population_per_household\",\n",
    "]\n",
    "\n",
    "categorical_features = [\"ocean_proximity\"]\n",
    "target = \"median_house_value\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's create a `ColumnTransformer` for our dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18576 entries, 6051 to 19966\n",
      "Data columns (total 12 columns):\n",
      " #   Column                    Non-Null Count  Dtype  \n",
      "---  ------                    --------------  -----  \n",
      " 0   longitude                 18576 non-null  float64\n",
      " 1   latitude                  18576 non-null  float64\n",
      " 2   housing_median_age        18576 non-null  float64\n",
      " 3   total_rooms               18576 non-null  float64\n",
      " 4   total_bedrooms            18391 non-null  float64\n",
      " 5   population                18576 non-null  float64\n",
      " 6   households                18576 non-null  float64\n",
      " 7   median_income             18576 non-null  float64\n",
      " 8   ocean_proximity           18576 non-null  object \n",
      " 9   rooms_per_household       18576 non-null  float64\n",
      " 10  bedrooms_per_household    18391 non-null  float64\n",
      " 11  population_per_household  18576 non-null  float64\n",
      "dtypes: float64(11), object(1)\n",
      "memory usage: 1.8+ MB\n"
     ]
    }
   ],
   "source": [
    "X_train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1H OCEAN     8221\n",
       "INLAND        5915\n",
       "NEAR OCEAN    2389\n",
       "NEAR BAY      2046\n",
       "ISLAND           5\n",
       "Name: ocean_proximity, dtype: int64"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[\"ocean_proximity\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "numeric_transformer = make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler())\n",
    "categorical_transformer = OneHotEncoder(handle_unknown=\"ignore\")\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (numeric_transformer, numeric_features),\n",
    "    (categorical_transformer, categorical_features),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 {color: black;background-color: white;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 pre{padding: 0;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-toggleable {background-color: white;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-estimator:hover {background-color: #d4ebff;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-item {z-index: 1;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-parallel-item:only-child::after {width: 0;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-abf39b9d-332e-400e-bc35-19c6c941ba44 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-abf39b9d-332e-400e-bc35-19c6c941ba44\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"97519d75-2540-4909-9586-bc75811970cd\" type=\"checkbox\" ><label for=\"97519d75-2540-4909-9586-bc75811970cd\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a913f5c8-2312-403b-91b1-69341c7a6c0f\" type=\"checkbox\" ><label for=\"a913f5c8-2312-403b-91b1-69341c7a6c0f\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"100d776f-f3a7-47cc-9754-d24e35e36945\" type=\"checkbox\" ><label for=\"100d776f-f3a7-47cc-9754-d24e35e36945\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dca4e2cc-3aa8-4fa3-9544-17fee383be08\" type=\"checkbox\" ><label for=\"dca4e2cc-3aa8-4fa3-9544-17fee383be08\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"976c84cc-c541-4a87-96c1-dc112f89d0c6\" type=\"checkbox\" ><label for=\"976c84cc-c541-4a87-96c1-dc112f89d0c6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"05c65ad3-15e8-4cc2-873c-e3a3bd02d0aa\" type=\"checkbox\" ><label for=\"05c65ad3-15e8-4cc2-873c-e3a3bd02d0aa\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer(strategy='median')),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['longitude', 'latitude', 'housing_median_age',\n",
       "                                  'total_rooms', 'total_bedrooms', 'population',\n",
       "                                  'households', 'median_income',\n",
       "                                  'rooms_per_household',\n",
       "                                  'bedrooms_per_household',\n",
       "                                  'population_per_household']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['ocean_proximity'])])"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X_train_pp = preprocessor.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- When we `fit` the preprocessor, it calls **`fit` on _all_** the transformers\n",
    "- When we `transform` the preprocessor, it calls **`transform` on _all_** the transformers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can get the new names of the columns that were generated by the one-hot encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-94067132-3c11-4288-a6b3-6585e4e4a56a {color: black;background-color: white;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a pre{padding: 0;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-toggleable {background-color: white;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-estimator:hover {background-color: #d4ebff;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-item {z-index: 1;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-parallel-item:only-child::after {width: 0;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-94067132-3c11-4288-a6b3-6585e4e4a56a div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-94067132-3c11-4288-a6b3-6585e4e4a56a\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"6c185003-4302-4b4a-8747-ab136f948c54\" type=\"checkbox\" ><label for=\"6c185003-4302-4b4a-8747-ab136f948c54\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"9ae1f2fa-117d-4d83-8ad9-35c077084c01\" type=\"checkbox\" ><label for=\"9ae1f2fa-117d-4d83-8ad9-35c077084c01\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"2923d3d7-1b44-40b9-8c00-7d96a3fc6808\" type=\"checkbox\" ><label for=\"2923d3d7-1b44-40b9-8c00-7d96a3fc6808\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d4d14b66-27db-4690-8ce7-0004a494716a\" type=\"checkbox\" ><label for=\"d4d14b66-27db-4690-8ce7-0004a494716a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"18ec54c8-8f2b-4ea0-8f98-baeaaf7ef97c\" type=\"checkbox\" ><label for=\"18ec54c8-8f2b-4ea0-8f98-baeaaf7ef97c\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8c19f124-11c0-45cc-b8dd-3dc61e291d22\" type=\"checkbox\" ><label for=\"8c19f124-11c0-45cc-b8dd-3dc61e291d22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer(strategy='median')),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['longitude', 'latitude', 'housing_median_age',\n",
       "                                  'total_rooms', 'total_bedrooms', 'population',\n",
       "                                  'households', 'median_income',\n",
       "                                  'rooms_per_household',\n",
       "                                  'bedrooms_per_household',\n",
       "                                  'population_per_household']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['ocean_proximity'])])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['ocean_proximity_<1H OCEAN', 'ocean_proximity_INLAND',\n",
       "       'ocean_proximity_ISLAND', 'ocean_proximity_NEAR BAY',\n",
       "       'ocean_proximity_NEAR OCEAN'], dtype=object)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor.named_transformers_[\"onehotencoder\"].get_feature_names_out(\n",
    "    categorical_features\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Combining this with the numeric feature names gives us all the column names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['longitude',\n",
       " 'latitude',\n",
       " 'housing_median_age',\n",
       " 'total_rooms',\n",
       " 'total_bedrooms',\n",
       " 'population',\n",
       " 'households',\n",
       " 'median_income',\n",
       " 'rooms_per_household',\n",
       " 'bedrooms_per_household',\n",
       " 'population_per_household',\n",
       " 'ocean_proximity_<1H OCEAN',\n",
       " 'ocean_proximity_INLAND',\n",
       " 'ocean_proximity_ISLAND',\n",
       " 'ocean_proximity_NEAR BAY',\n",
       " 'ocean_proximity_NEAR OCEAN']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = numeric_features + list(\n",
    "    preprocessor.named_transformers_[\"onehotencoder\"].get_feature_names_out(\n",
    "        categorical_features\n",
    "    )\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's visualize the preprocessed training data as a dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "      <th>ocean_proximity_&lt;1H OCEAN</th>\n",
       "      <th>ocean_proximity_INLAND</th>\n",
       "      <th>ocean_proximity_ISLAND</th>\n",
       "      <th>ocean_proximity_NEAR BAY</th>\n",
       "      <th>ocean_proximity_NEAR OCEAN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.908140</td>\n",
       "      <td>-0.743917</td>\n",
       "      <td>-0.526078</td>\n",
       "      <td>0.143120</td>\n",
       "      <td>0.235339</td>\n",
       "      <td>1.026092</td>\n",
       "      <td>0.266135</td>\n",
       "      <td>-0.389736</td>\n",
       "      <td>-0.210591</td>\n",
       "      <td>-0.083813</td>\n",
       "      <td>0.126398</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.002057</td>\n",
       "      <td>1.083123</td>\n",
       "      <td>-0.923283</td>\n",
       "      <td>-1.049510</td>\n",
       "      <td>-0.969959</td>\n",
       "      <td>-1.206672</td>\n",
       "      <td>-1.253312</td>\n",
       "      <td>-0.198924</td>\n",
       "      <td>4.726412</td>\n",
       "      <td>11.166631</td>\n",
       "      <td>-0.050132</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.218207</td>\n",
       "      <td>-1.352930</td>\n",
       "      <td>1.380504</td>\n",
       "      <td>0.329670</td>\n",
       "      <td>0.549764</td>\n",
       "      <td>0.024896</td>\n",
       "      <td>0.542873</td>\n",
       "      <td>-0.635239</td>\n",
       "      <td>-0.273606</td>\n",
       "      <td>-0.025391</td>\n",
       "      <td>-0.099240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.128188</td>\n",
       "      <td>-0.753286</td>\n",
       "      <td>-0.843842</td>\n",
       "      <td>-0.459154</td>\n",
       "      <td>-0.626949</td>\n",
       "      <td>-0.463877</td>\n",
       "      <td>-0.561467</td>\n",
       "      <td>0.714077</td>\n",
       "      <td>0.122307</td>\n",
       "      <td>-0.280310</td>\n",
       "      <td>0.010183</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.168196</td>\n",
       "      <td>-1.287344</td>\n",
       "      <td>-0.843842</td>\n",
       "      <td>1.343085</td>\n",
       "      <td>2.210026</td>\n",
       "      <td>4.269688</td>\n",
       "      <td>2.500924</td>\n",
       "      <td>-1.059242</td>\n",
       "      <td>-0.640266</td>\n",
       "      <td>-0.190617</td>\n",
       "      <td>0.126808</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18571</th>\n",
       "      <td>0.733102</td>\n",
       "      <td>-0.804818</td>\n",
       "      <td>0.586095</td>\n",
       "      <td>-0.875337</td>\n",
       "      <td>-0.243446</td>\n",
       "      <td>-0.822136</td>\n",
       "      <td>-0.966131</td>\n",
       "      <td>-0.118182</td>\n",
       "      <td>0.063110</td>\n",
       "      <td>-0.099558</td>\n",
       "      <td>0.071541</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18572</th>\n",
       "      <td>1.163195</td>\n",
       "      <td>-1.057793</td>\n",
       "      <td>-1.161606</td>\n",
       "      <td>0.940194</td>\n",
       "      <td>0.609314</td>\n",
       "      <td>0.882438</td>\n",
       "      <td>0.728235</td>\n",
       "      <td>0.357500</td>\n",
       "      <td>0.235096</td>\n",
       "      <td>-0.163397</td>\n",
       "      <td>0.007458</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18573</th>\n",
       "      <td>-1.097293</td>\n",
       "      <td>0.797355</td>\n",
       "      <td>-1.876574</td>\n",
       "      <td>0.695434</td>\n",
       "      <td>0.433046</td>\n",
       "      <td>0.881563</td>\n",
       "      <td>0.514155</td>\n",
       "      <td>0.934269</td>\n",
       "      <td>0.211892</td>\n",
       "      <td>-0.135305</td>\n",
       "      <td>0.044029</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18574</th>\n",
       "      <td>-1.437367</td>\n",
       "      <td>1.008167</td>\n",
       "      <td>1.221622</td>\n",
       "      <td>-0.499947</td>\n",
       "      <td>-0.484029</td>\n",
       "      <td>-0.759944</td>\n",
       "      <td>-0.454427</td>\n",
       "      <td>0.006578</td>\n",
       "      <td>-0.273382</td>\n",
       "      <td>-0.149822</td>\n",
       "      <td>-0.132875</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18575</th>\n",
       "      <td>0.242996</td>\n",
       "      <td>0.272667</td>\n",
       "      <td>-0.684960</td>\n",
       "      <td>-0.332190</td>\n",
       "      <td>-0.353018</td>\n",
       "      <td>-0.164307</td>\n",
       "      <td>-0.396991</td>\n",
       "      <td>-0.711754</td>\n",
       "      <td>0.025998</td>\n",
       "      <td>0.042957</td>\n",
       "      <td>0.051269</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>18576 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0       0.908140 -0.743917           -0.526078     0.143120        0.235339   \n",
       "1      -0.002057  1.083123           -0.923283    -1.049510       -0.969959   \n",
       "2       1.218207 -1.352930            1.380504     0.329670        0.549764   \n",
       "3       1.128188 -0.753286           -0.843842    -0.459154       -0.626949   \n",
       "4       1.168196 -1.287344           -0.843842     1.343085        2.210026   \n",
       "...          ...       ...                 ...          ...             ...   \n",
       "18571   0.733102 -0.804818            0.586095    -0.875337       -0.243446   \n",
       "18572   1.163195 -1.057793           -1.161606     0.940194        0.609314   \n",
       "18573  -1.097293  0.797355           -1.876574     0.695434        0.433046   \n",
       "18574  -1.437367  1.008167            1.221622    -0.499947       -0.484029   \n",
       "18575   0.242996  0.272667           -0.684960    -0.332190       -0.353018   \n",
       "\n",
       "       population  households  median_income  rooms_per_household  \\\n",
       "0        1.026092    0.266135      -0.389736            -0.210591   \n",
       "1       -1.206672   -1.253312      -0.198924             4.726412   \n",
       "2        0.024896    0.542873      -0.635239            -0.273606   \n",
       "3       -0.463877   -0.561467       0.714077             0.122307   \n",
       "4        4.269688    2.500924      -1.059242            -0.640266   \n",
       "...           ...         ...            ...                  ...   \n",
       "18571   -0.822136   -0.966131      -0.118182             0.063110   \n",
       "18572    0.882438    0.728235       0.357500             0.235096   \n",
       "18573    0.881563    0.514155       0.934269             0.211892   \n",
       "18574   -0.759944   -0.454427       0.006578            -0.273382   \n",
       "18575   -0.164307   -0.396991      -0.711754             0.025998   \n",
       "\n",
       "       bedrooms_per_household  population_per_household  \\\n",
       "0                   -0.083813                  0.126398   \n",
       "1                   11.166631                 -0.050132   \n",
       "2                   -0.025391                 -0.099240   \n",
       "3                   -0.280310                  0.010183   \n",
       "4                   -0.190617                  0.126808   \n",
       "...                       ...                       ...   \n",
       "18571               -0.099558                  0.071541   \n",
       "18572               -0.163397                  0.007458   \n",
       "18573               -0.135305                  0.044029   \n",
       "18574               -0.149822                 -0.132875   \n",
       "18575                0.042957                  0.051269   \n",
       "\n",
       "       ocean_proximity_<1H OCEAN  ocean_proximity_INLAND  \\\n",
       "0                            0.0                     1.0   \n",
       "1                            0.0                     1.0   \n",
       "2                            0.0                     0.0   \n",
       "3                            0.0                     1.0   \n",
       "4                            0.0                     0.0   \n",
       "...                          ...                     ...   \n",
       "18571                        1.0                     0.0   \n",
       "18572                        1.0                     0.0   \n",
       "18573                        1.0                     0.0   \n",
       "18574                        0.0                     0.0   \n",
       "18575                        0.0                     1.0   \n",
       "\n",
       "       ocean_proximity_ISLAND  ocean_proximity_NEAR BAY  \\\n",
       "0                         0.0                       0.0   \n",
       "1                         0.0                       0.0   \n",
       "2                         0.0                       0.0   \n",
       "3                         0.0                       0.0   \n",
       "4                         0.0                       0.0   \n",
       "...                       ...                       ...   \n",
       "18571                     0.0                       0.0   \n",
       "18572                     0.0                       0.0   \n",
       "18573                     0.0                       0.0   \n",
       "18574                     0.0                       1.0   \n",
       "18575                     0.0                       0.0   \n",
       "\n",
       "       ocean_proximity_NEAR OCEAN  \n",
       "0                             0.0  \n",
       "1                             0.0  \n",
       "2                             1.0  \n",
       "3                             0.0  \n",
       "4                             1.0  \n",
       "...                           ...  \n",
       "18571                         0.0  \n",
       "18572                         0.0  \n",
       "18573                         0.0  \n",
       "18574                         0.0  \n",
       "18575                         0.0  \n",
       "\n",
       "[18576 rows x 16 columns]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_train_pp, columns=column_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>median_house_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>113600.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>137500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>170100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>129300.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>205000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       median_house_value\n",
       "6051             113600.0\n",
       "20113            137500.0\n",
       "14289            170100.0\n",
       "13665            129300.0\n",
       "14471            205000.0"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.to_frame().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                fit_time         score_time          test_score  \\\n",
       "dummy  0.001 (+/- 0.000)  0.000 (+/- 0.000)  -0.001 (+/- 0.001)   \n",
       "\n",
       "             train_score  \n",
       "dummy  0.000 (+/- 0.000)  "
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_dict = {}\n",
    "dummy = DummyRegressor()\n",
    "results_dict[\"dummy\"] = mean_std_cross_val_scores(\n",
    "    dummy, X_train, y_train, return_train_score=True\n",
    ")\n",
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVR\n",
    "\n",
    "knn_pipe = make_pipeline(preprocessor, KNeighborsRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c {color: black;background-color: white;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c pre{padding: 0;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-toggleable {background-color: white;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-estimator:hover {background-color: #d4ebff;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-item {z-index: 1;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-parallel-item:only-child::after {width: 0;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-d8ef638d-0776-4291-9106-6a9d8b510d0c div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-d8ef638d-0776-4291-9106-6a9d8b510d0c\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;standardscaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;longitude&#x27;, &#x27;latitude&#x27;,\n",
       "                                                   &#x27;housing_median_age&#x27;,\n",
       "                                                   &#x27;total_rooms&#x27;,\n",
       "                                                   &#x27;total_bedrooms&#x27;,\n",
       "                                                   &#x27;population&#x27;, &#x27;households&#x27;,\n",
       "                                                   &#x27;median_income&#x27;,\n",
       "                                                   &#x27;rooms_per_household&#x27;,\n",
       "                                                   &#x27;bedrooms_per_household&#x27;,\n",
       "                                                   &#x27;population_per_household&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                                  [&#x27;ocean_proximity&#x27;])])),\n",
       "                (&#x27;kneighborsregressor&#x27;, KNeighborsRegressor())])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"c80243e5-1aac-4429-b447-4c90cb1a1de6\" type=\"checkbox\" ><label for=\"c80243e5-1aac-4429-b447-4c90cb1a1de6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;standardscaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;longitude&#x27;, &#x27;latitude&#x27;,\n",
       "                                                   &#x27;housing_median_age&#x27;,\n",
       "                                                   &#x27;total_rooms&#x27;,\n",
       "                                                   &#x27;total_bedrooms&#x27;,\n",
       "                                                   &#x27;population&#x27;, &#x27;households&#x27;,\n",
       "                                                   &#x27;median_income&#x27;,\n",
       "                                                   &#x27;rooms_per_household&#x27;,\n",
       "                                                   &#x27;bedrooms_per_household&#x27;,\n",
       "                                                   &#x27;population_per_household&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                                  [&#x27;ocean_proximity&#x27;])])),\n",
       "                (&#x27;kneighborsregressor&#x27;, KNeighborsRegressor())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7c7fecdb-d5c3-4a25-8b92-6447a42fa9ce\" type=\"checkbox\" ><label for=\"7c7fecdb-d5c3-4a25-8b92-6447a42fa9ce\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"0f39c0db-5008-40cb-83b2-57e5ab598feb\" type=\"checkbox\" ><label for=\"0f39c0db-5008-40cb-83b2-57e5ab598feb\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;total_rooms&#x27;, &#x27;total_bedrooms&#x27;, &#x27;population&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b5d0e94b-12b8-4634-8d34-30a11cd91e9b\" type=\"checkbox\" ><label for=\"b5d0e94b-12b8-4634-8d34-30a11cd91e9b\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"eb86e222-cf10-47a6-a3e9-c15072db9c40\" type=\"checkbox\" ><label for=\"eb86e222-cf10-47a6-a3e9-c15072db9c40\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"1c9318cf-83ce-4156-a1ad-5bf804b4686a\" type=\"checkbox\" ><label for=\"1c9318cf-83ce-4156-a1ad-5bf804b4686a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"5cc4e827-ad16-4137-8c18-6166a984ccc1\" type=\"checkbox\" ><label for=\"5cc4e827-ad16-4137-8c18-6166a984ccc1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"87384108-18fc-4838-bdb0-013f5cc03065\" type=\"checkbox\" ><label for=\"87384108-18fc-4838-bdb0-013f5cc03065\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KNeighborsRegressor</label><div class=\"sk-toggleable__content\"><pre>KNeighborsRegressor()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('columntransformer',\n",
       "                 ColumnTransformer(transformers=[('pipeline',\n",
       "                                                  Pipeline(steps=[('simpleimputer',\n",
       "                                                                   SimpleImputer(strategy='median')),\n",
       "                                                                  ('standardscaler',\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  ['longitude', 'latitude',\n",
       "                                                   'housing_median_age',\n",
       "                                                   'total_rooms',\n",
       "                                                   'total_bedrooms',\n",
       "                                                   'population', 'households',\n",
       "                                                   'median_income',\n",
       "                                                   'rooms_per_household',\n",
       "                                                   'bedrooms_per_household',\n",
       "                                                   'population_per_household']),\n",
       "                                                 ('onehotencoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore'),\n",
       "                                                  ['ocean_proximity'])])),\n",
       "                ('kneighborsregressor', KNeighborsRegressor())])"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dict[\"imp + scaling + ohe + KNN\"] = mean_std_cross_val_scores(\n",
    "    knn_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.024 (+/- 0.002)</td>\n",
       "      <td>0.807 (+/- 0.136)</td>\n",
       "      <td>0.721 (+/- 0.012)</td>\n",
       "      <td>0.816 (+/- 0.006)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    fit_time         score_time  \\\n",
       "dummy                      0.001 (+/- 0.000)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN  0.024 (+/- 0.002)  0.807 (+/- 0.136)   \n",
       "\n",
       "                                   test_score        train_score  \n",
       "dummy                      -0.001 (+/- 0.001)  0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN   0.721 (+/- 0.012)  0.816 (+/- 0.006)  "
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "svr_pipe = make_pipeline(preprocessor, SVR())\n",
    "results_dict[\"imp + scaling + ohe + SVR (default)\"] = mean_std_cross_val_scores(\n",
    "    svr_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.024 (+/- 0.002)</td>\n",
       "      <td>0.807 (+/- 0.136)</td>\n",
       "      <td>0.721 (+/- 0.012)</td>\n",
       "      <td>0.816 (+/- 0.006)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (default)</th>\n",
       "      <td>12.143 (+/- 1.702)</td>\n",
       "      <td>3.046 (+/- 0.437)</td>\n",
       "      <td>-0.049 (+/- 0.012)</td>\n",
       "      <td>-0.049 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               fit_time         score_time  \\\n",
       "dummy                                 0.001 (+/- 0.000)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN             0.024 (+/- 0.002)  0.807 (+/- 0.136)   \n",
       "imp + scaling + ohe + SVR (default)  12.143 (+/- 1.702)  3.046 (+/- 0.437)   \n",
       "\n",
       "                                             test_score         train_score  \n",
       "dummy                                -0.001 (+/- 0.001)   0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN             0.721 (+/- 0.012)   0.816 (+/- 0.006)  \n",
       "imp + scaling + ohe + SVR (default)  -0.049 (+/- 0.012)  -0.049 (+/- 0.001)  "
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results with `scikit-learn`'s default SVR hyperparameters are pretty bad. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "svr_C_pipe = make_pipeline(preprocessor, SVR(C=10000))\n",
    "results_dict[\"imp + scaling + ohe + SVR (C=10000)\"] = mean_std_cross_val_scores(\n",
    "    svr_C_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.024 (+/- 0.002)</td>\n",
       "      <td>0.807 (+/- 0.136)</td>\n",
       "      <td>0.721 (+/- 0.012)</td>\n",
       "      <td>0.816 (+/- 0.006)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (default)</th>\n",
       "      <td>12.143 (+/- 1.702)</td>\n",
       "      <td>3.046 (+/- 0.437)</td>\n",
       "      <td>-0.049 (+/- 0.012)</td>\n",
       "      <td>-0.049 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (C=10000)</th>\n",
       "      <td>11.154 (+/- 1.224)</td>\n",
       "      <td>3.165 (+/- 0.114)</td>\n",
       "      <td>0.721 (+/- 0.007)</td>\n",
       "      <td>0.726 (+/- 0.007)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               fit_time         score_time  \\\n",
       "dummy                                 0.001 (+/- 0.000)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN             0.024 (+/- 0.002)  0.807 (+/- 0.136)   \n",
       "imp + scaling + ohe + SVR (default)  12.143 (+/- 1.702)  3.046 (+/- 0.437)   \n",
       "imp + scaling + ohe + SVR (C=10000)  11.154 (+/- 1.224)  3.165 (+/- 0.114)   \n",
       "\n",
       "                                             test_score         train_score  \n",
       "dummy                                -0.001 (+/- 0.001)   0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN             0.721 (+/- 0.012)   0.816 (+/- 0.006)  \n",
       "imp + scaling + ohe + SVR (default)  -0.049 (+/- 0.012)  -0.049 (+/- 0.001)  \n",
       "imp + scaling + ohe + SVR (C=10000)   0.721 (+/- 0.007)   0.726 (+/- 0.007)  "
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a **bigger value for `C`** the results are much **better**. We need to carry out systematic **hyperparameter optimization** to get better results. (Coming up next week.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- Note that categorical features are different than free text features. Sometimes there are columns containing free text information and we we'll look at ways to deal with them in the later part of this lecture. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### OHE with many categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Do we have enough data for rare categories to learn anything meaningful? \n",
    "- How about grouping them into bigger categories?\n",
    "    - Example: country names into continents such as \"South America\" or \"Asia\"\n",
    "- Or having \"other\" category for rare cases? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Do we actually want to use certain features for prediction?\n",
    "\n",
    "- Do you ***want*** to use certain features such as **gender** or **race** in prediction?\n",
    "- Remember that the systems you build are going to be used in some applications. \n",
    "- It's extremely important to be mindful of the consequences of including certain features in your predictive model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preprocessing the targets?\n",
    "\n",
    "- Generally no need for this when doing classification. \n",
    "- In regression it makes sense in some cases. More on this later. \n",
    "- `sklearn` is fine with categorical labels ($y$-values) for classification problems. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "#### True/False: Categorical features\n",
    "\n",
    "1. `handle_unknown=\"ignore\"` would treat all unknown categories equally. \n",
    "2. Creating groups of rarely occurring categories might overfit the model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Encoding text data  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "toy_spam = [\n",
    "    [\n",
    "        \"URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\"Lol you are always so convincing.\", \"non spam\"],\n",
    "    [\"Nah I don't think he goes to usf, he lives around here though\", \"non spam\"],\n",
    "    [\n",
    "        \"URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\n",
    "        \"Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\"Congrats! I can't wait to see you!!\", \"non spam\"],\n",
    "]\n",
    "toy_df = pd.DataFrame(toy_spam, columns=[\"sms\", \"target\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Spam/non spam toy example "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- What if the feature is in the form of raw text?\n",
    "- The feature **`sms`** below is **neither categorical nor ordinal**. \n",
    "- How can we encode it so that we can pass it to the machine learning algorithms we have seen so far? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_8577d_row0_col0, #T_8577d_row0_col1, #T_8577d_row1_col0, #T_8577d_row1_col1, #T_8577d_row2_col0, #T_8577d_row2_col1, #T_8577d_row3_col0, #T_8577d_row3_col1, #T_8577d_row4_col0, #T_8577d_row4_col1, #T_8577d_row5_col0, #T_8577d_row5_col1 {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_8577d\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_8577d_level0_col0\" class=\"col_heading level0 col0\" >sms</th>\n",
       "      <th id=\"T_8577d_level0_col1\" class=\"col_heading level0 col1\" >target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_8577d_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_8577d_row0_col0\" class=\"data row0 col0\" >URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</td>\n",
       "      <td id=\"T_8577d_row0_col1\" class=\"data row0 col1\" >spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8577d_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_8577d_row1_col0\" class=\"data row1 col0\" >Lol you are always so convincing.</td>\n",
       "      <td id=\"T_8577d_row1_col1\" class=\"data row1 col1\" >non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8577d_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_8577d_row2_col0\" class=\"data row2 col0\" >Nah I don't think he goes to usf, he lives around here though</td>\n",
       "      <td id=\"T_8577d_row2_col1\" class=\"data row2 col1\" >non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8577d_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_8577d_row3_col0\" class=\"data row3 col0\" >URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</td>\n",
       "      <td id=\"T_8577d_row3_col1\" class=\"data row3 col1\" >spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8577d_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_8577d_row4_col0\" class=\"data row4 col0\" >Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</td>\n",
       "      <td id=\"T_8577d_row4_col1\" class=\"data row4 col1\" >spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8577d_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_8577d_row5_col0\" class=\"data row5 col0\" >Congrats! I can't wait to see you!!</td>\n",
       "      <td id=\"T_8577d_row5_col1\" class=\"data row5 col1\" >non spam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f7caae7fcd0>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_df.style.set_properties(**{\"text-align\": \"left\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What if we apply OHE? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Congrats! I can't wait to see you!!  \\\n",
       "0                                 0.0   \n",
       "1                                 0.0   \n",
       "2                                 0.0   \n",
       "3                                 0.0   \n",
       "4                                 0.0   \n",
       "5                                 1.0   \n",
       "\n",
       "  Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  \\\n",
       "0                                                                                                                                                        0.0   \n",
       "1                                                                                                                                                        0.0   \n",
       "2                                                                                                                                                        0.0   \n",
       "3                                                                                                                                                        0.0   \n",
       "4                                                                                                                                                        1.0   \n",
       "5                                                                                                                                                        0.0   \n",
       "\n",
       "  Lol you are always so convincing.  \\\n",
       "0                               0.0   \n",
       "1                               1.0   \n",
       "2                               0.0   \n",
       "3                               0.0   \n",
       "4                               0.0   \n",
       "5                               0.0   \n",
       "\n",
       "  Nah I don't think he goes to usf, he lives around here though  \\\n",
       "0                                                           0.0   \n",
       "1                                                           0.0   \n",
       "2                                                           1.0   \n",
       "3                                                           0.0   \n",
       "4                                                           0.0   \n",
       "5                                                           0.0   \n",
       "\n",
       "  URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!  \\\n",
       "0                                                                         0.0   \n",
       "1                                                                         0.0   \n",
       "2                                                                         0.0   \n",
       "3                                                                         1.0   \n",
       "4                                                                         0.0   \n",
       "5                                                                         0.0   \n",
       "\n",
       "  URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!  \n",
       "0                                                                                          1.0  \n",
       "1                                                                                          0.0  \n",
       "2                                                                                          0.0  \n",
       "3                                                                                          0.0  \n",
       "4                                                                                          0.0  \n",
       "5                                                                                          0.0  "
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### DO NOT DO THIS.\n",
    "enc = OneHotEncoder(sparse=False)\n",
    "transformed = enc.fit_transform(toy_df[[\"sms\"]])\n",
    "pd.DataFrame(transformed, columns=enc.categories_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- We do **not have a fixed number** of categories here. \n",
    "- Each \"category\" (feature value) is likely to **occur only once** in the training data and we won't learn anything meaningful if we apply one-hot encoding or ordinal encoding on this feature. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- How can we encode or represent **raw text data into fixed number of features** so that we can learn some useful patterns from it?  \n",
    "- This is a well studied problem in the field of ***Natural Language Processing (NLP)***, which is concerned with giving computers the ability to understand written and spoken language. \n",
    "- Some popular representations of raw text include: \n",
    "    - **Bag of words** \n",
    "    - TF-IDF\n",
    "    - Embedding representations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag of words (BOW) representation\n",
    "\n",
    "- One of the most popular representation of raw text \n",
    "- Ignores the syntax and word order\n",
    "- It has two components: \n",
    "    - The vocabulary (all unique words in all documents) \n",
    "    - A value indicating either the presence or absence or the count of each word in the document. \n",
    "        \n",
    "<center>\n",
    "<img src='./img/bag-of-words.png' width=\"600\">\n",
    "</center>\n",
    "\n",
    "[Source](https://web.stanford.edu/~jurafsky/slp3/4.pdf)       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Extracting BOW features using `scikit-learn`\n",
    "- `CountVectorizer`\n",
    "    - Converts a collection of text documents to a matrix of word counts.  \n",
    "    - Each row represents a \"document\" (e.g., a text message in our example). \n",
    "    - Each column represents a word in the vocabulary (the set of unique words) in the training data. \n",
    "    - Each cell represents how often the word occurs in the document.       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***Note***\n",
    "In the Natural Language Processing (NLP) community text data  is referred to as a **corpus** (plural: corpora)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vec = CountVectorizer()\n",
    "X_counts = vec.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec.get_feature_names_out(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input to `CountVectorizer.fit_transform`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(toy_df[\"sms\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***Important***\n",
    "Note that unlike other transformers we are **passing a `Series`** object to `fit_transform`. For other transformers, you can define one transformer for more than one columns. But with `CountVectorizer` you need to define **separate `CountVectorizer` transformers for each text column**, if you have more than one text columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output of `CountVectorizer.fit_transform`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fit_transform` has returned a sparse matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<6x61 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 71 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Why sparse matrices?\n",
    "\n",
    "- Most words do not appear in a given document.\n",
    "- We get massive computational savings if we **only store the nonzero elements**.\n",
    "- There is a bit of overhead, because we also need to store the locations:\n",
    "    - e.g. \"location (3,27): 1\".\n",
    "    \n",
    "- However, if the fraction of nonzero is small, this is a huge win."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of rows and columns:  6 61\n",
      "The total number of elements:  366\n",
      "The number of non-zero elements:  71\n",
      "Proportion of non-zero elements: 0.1940\n",
      "The value at cell (4,51) is: 2\n"
     ]
    }
   ],
   "source": [
    "print(\"The number of rows and columns: \", *X_counts.shape)\n",
    "print(\"The total number of elements: \", np.prod(X_counts.shape))\n",
    "print(\"The number of non-zero elements: \", X_counts.nnz)\n",
    "print(\n",
    "    \"Proportion of non-zero elements: %0.4f\" % (X_counts.nnz / np.prod(X_counts.shape))\n",
    ")\n",
    "print(\n",
    "    \"The value at cell (4,%d) is: %d\"\n",
    "    % (vec.vocabulary_[\"update\"], X_counts[4, vec.vocabulary_[\"update\"]])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "Question for you\n",
    "- What would happen if you apply `StandardScaler` on sparse data? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### `OneHotEncoder` and sparse features \n",
    "- By default, `OneHotEncoder` also creates sparse features. \n",
    "- You could set `sparse=False` to get a regular `numpy` array. \n",
    "- If there are a huge number of categories, it may be beneficial to keep them sparse.\n",
    "- For smaller number of categories, it doesn't matter much."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Important hyperparameters of `CountVectorizer` \n",
    "\n",
    "- `binary`\n",
    "    - whether to use absence/presence feature values or counts\n",
    "- `max_features`\n",
    "    - only consider top `max_features` ordered by frequency in the corpus\n",
    "- `max_df`\n",
    "    - max document frequency, ignore features which occur in more than `max_df` documents \n",
    "- `min_df` \n",
    "    - min document frequency, ignore features which occur in less than `min_df` documents \n",
    "- `ngram_range`\n",
    "    - consider word sequences in the given range "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at all features, i.e., words (along with their frequencies)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max value:  2\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec = CountVectorizer()\n",
    "X_counts = vec.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec.get_feature_names_out(), index=toy_df[\"sms\"]\n",
    ")\n",
    "print(\"Max value: \", bow_df.max().max())\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When we use `binary=True`, the representation uses presence/absence of words instead of word counts.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max value:  1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec_binary = CountVectorizer(binary=True)\n",
    "X_counts = vec_binary.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec_binary.get_feature_names_out(), index=toy_df[\"sms\"]\n",
    ")\n",
    "print(\"Max value: \", bow_df.max().max())\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can control the size of X (the number of features) using `max_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max value:  2\n",
      "free      2\n",
      "have      1\n",
      "mobile    2\n",
      "the       2\n",
      "to        2\n",
      "update    2\n",
      "urgent    1\n",
      "you       1\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>free</th>\n",
       "      <th>have</th>\n",
       "      <th>mobile</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>you</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            free  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            have  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            mobile  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            the  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            to  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          1   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            you  \n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1  \n",
       "Lol you are always so convincing.                                                                                                                             1  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0  \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1  "
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8 = CountVectorizer(max_features=8)\n",
    "X_counts = vec8.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec8.get_feature_names_out(), index=toy_df[\"sms\"]\n",
    ")\n",
    "print(\"Max value: \", bow_df.max().max())\n",
    "print(bow_df.max())\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***Note (Optional)*** \n",
    "\n",
    "> Notice that `vec8` and `vec8_binary` have different vocabularies, which is kind of unexpected behaviour and doesn't match the documentation of `scikit-learn`.\n",
    "> \n",
    "> [Here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L1206-L1225) is the code for `binary=True` condition in `scikit-learn`. As we can see, the binarization is done before limiting the features to `max_features`, and so now we are actually looking at the document counts (in how many documents it occurs) rather than term count. This is not explained anywhere in the documentation. \n",
    "> \n",
    "> The ties in counts between different words makes it even more confusing. I don't think it'll have a big impact on the results but this is good to know! Remember that `scikit-learn` developers are also humans who are prone to make mistakes. So it's always a good habit to question whatever tools we use every now and then."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>you</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>free</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>have</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mobile</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>the</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>update</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>urgent</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        counts\n",
       "to           5\n",
       "you          4\n",
       "free         3\n",
       "have         2\n",
       "mobile       2\n",
       "the          2\n",
       "update       2\n",
       "urgent       2"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8 = CountVectorizer(max_features=8)\n",
    "X_counts = vec8.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec8.get_feature_names_out(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df.sum().sort_values(ascending=False).rename(\"counts\").to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>you</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>free</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>have</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>prize</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>urgent</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mobiles</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>months</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         counts\n",
       "to            4\n",
       "you           4\n",
       "free          2\n",
       "have          2\n",
       "prize         2\n",
       "urgent        2\n",
       "mobiles       1\n",
       "months        1"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8_binary = CountVectorizer(binary=True, max_features=8)\n",
    "X_counts = vec8_binary.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec8_binary.get_feature_names_out(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df.sum().sort_values(ascending=False).rename(\"counts\").to_frame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preprocessing\n",
    "\n",
    "- Note that `CountVectorizer` is carrying out some preprocessing such as the following because of the default argument values:\n",
    "    - Converting words to lowercase (`lowercase=True`)\n",
    "    - getting rid of punctuation and special characters (`token_pattern ='(?u)\\\\b\\\\w\\\\w+\\\\b'`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(CountVectorizer(), SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e {color: black;background-color: white;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e pre{padding: 0;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-toggleable {background-color: white;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-estimator:hover {background-color: #d4ebff;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-item {z-index: 1;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-parallel-item:only-child::after {width: 0;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-4d958fdc-3f9e-48c3-b5a2-ca27d1e5c72e\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;countvectorizer&#x27;, CountVectorizer()), (&#x27;svc&#x27;, SVC())])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d6e1f66b-017c-4ca2-a143-a98a1a0ab591\" type=\"checkbox\" ><label for=\"d6e1f66b-017c-4ca2-a143-a98a1a0ab591\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;countvectorizer&#x27;, CountVectorizer()), (&#x27;svc&#x27;, SVC())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a6b142ad-7fca-45b0-af21-eef6fb775e49\" type=\"checkbox\" ><label for=\"a6b142ad-7fca-45b0-af21-eef6fb775e49\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CountVectorizer</label><div class=\"sk-toggleable__content\"><pre>CountVectorizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"9cff1ced-fbe3-49d5-89f1-b74aee0548c8\" type=\"checkbox\" ><label for=\"9cff1ced-fbe3-49d5-89f1-b74aee0548c8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('countvectorizer', CountVectorizer()), ('svc', SVC())])"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.fit(toy_df[\"sms\"], toy_df[\"target\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['spam', 'non spam', 'non spam', 'spam', 'spam', 'non spam']"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.predict(toy_df[\"sms\"]).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['spam', 'non spam', 'non spam', 'spam', 'spam', 'non spam']"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_df[\"target\"].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Is this a realistic representation of text data? \n",
    "\n",
    "- Of course this is not a great representation of language\n",
    "    - We are throwing out everything we know about language and losing a lot of information. \n",
    "    - It assumes that there is **no syntax and compositional meaning** in language.  \n",
    "- But it **works surprisingly well** for many tasks. \n",
    "- We will learn more expressive representations in the coming weeks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### `CountVectorizer`: True or False\n",
    "\n",
    "1. As you increase the value for `max_features` hyperparameter of `CountVectorizer` the training score is likely to go up. \n",
    "2. Suppose you are encoding text data using `CountVectorizer`. If you encounter a word in the validation or the test split that's not available in the training data, we'll get an error. \n",
    "3. `max_df` hyperparameter of `CountVectorizer` can be used to get rid of most frequently occurring words from the dictionary.\n",
    "4. In the code below, inside `cross_validate`, each fold might have slightly different number of features (columns) in the fold.\n",
    "\n",
    "```\n",
    "pipe = (CountVectorizer(), SVC())\n",
    "cross_validate(pipe, X_train, y_train)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Identify column transformations\n",
    "\n",
    "Consider the restaurant data from the survey you did a few weeks ago. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>north_america</th>\n",
       "      <th>eat_out_freq</th>\n",
       "      <th>age</th>\n",
       "      <th>n_people</th>\n",
       "      <th>price</th>\n",
       "      <th>food_type</th>\n",
       "      <th>noise_level</th>\n",
       "      <th>good_server</th>\n",
       "      <th>comments</th>\n",
       "      <th>restaurant_name</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Yes</td>\n",
       "      <td>3.0</td>\n",
       "      <td>29</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Other</td>\n",
       "      <td>no music</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Yes</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23</td>\n",
       "      <td>10</td>\n",
       "      <td>20</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>low</td>\n",
       "      <td>Yes</td>\n",
       "      <td>food tastes good</td>\n",
       "      <td>Midam</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yes</td>\n",
       "      <td>2.0</td>\n",
       "      <td>21</td>\n",
       "      <td>10</td>\n",
       "      <td>40</td>\n",
       "      <td>Other</td>\n",
       "      <td>low</td>\n",
       "      <td>Yes</td>\n",
       "      <td>good food</td>\n",
       "      <td>pear tree</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>No</td>\n",
       "      <td>2.0</td>\n",
       "      <td>24</td>\n",
       "      <td>15</td>\n",
       "      <td>40</td>\n",
       "      <td>Other</td>\n",
       "      <td>medium</td>\n",
       "      <td>Yes</td>\n",
       "      <td>My love for Korean food</td>\n",
       "      <td>Dami</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Yes</td>\n",
       "      <td>5.0</td>\n",
       "      <td>23</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Canadian/American</td>\n",
       "      <td>low</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Good food</td>\n",
       "      <td>NaN</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Yes</td>\n",
       "      <td>2.0</td>\n",
       "      <td>22</td>\n",
       "      <td>60</td>\n",
       "      <td>20</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>medium</td>\n",
       "      <td>Yes</td>\n",
       "      <td>food tasted great!</td>\n",
       "      <td>NaN</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Yes</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23</td>\n",
       "      <td>10000000</td>\n",
       "      <td>1000000</td>\n",
       "      <td>Canadian/American</td>\n",
       "      <td>crazy loud</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CACTUS CLUB CAFE</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Yes</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>40</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>high</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>21</td>\n",
       "      <td>40</td>\n",
       "      <td>60</td>\n",
       "      <td>Italian</td>\n",
       "      <td>high</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Frankie’s</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>No</td>\n",
       "      <td>2.0</td>\n",
       "      <td>20</td>\n",
       "      <td>50</td>\n",
       "      <td>45</td>\n",
       "      <td>Canadian/American</td>\n",
       "      <td>medium</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Steak was good, ambiance was nice and server was helpful.</td>\n",
       "      <td>Hy's steakhouse</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  north_america  eat_out_freq  age  n_people    price          food_type  \\\n",
       "0           Yes           3.0   29        20       10              Other   \n",
       "1           Yes           2.0   23        10       20            Chinese   \n",
       "2           Yes           2.0   21        10       40              Other   \n",
       "3            No           2.0   24        15       40              Other   \n",
       "4           Yes           5.0   23        20       10  Canadian/American   \n",
       "5           Yes           2.0   22        60       20            Chinese   \n",
       "6           Yes           2.0   23  10000000  1000000  Canadian/American   \n",
       "7           Yes           4.0   20        20       40            Chinese   \n",
       "8           Yes           1.0   21        40       60            Italian   \n",
       "9            No           2.0   20        50       45  Canadian/American   \n",
       "\n",
       "  noise_level good_server  \\\n",
       "0    no music         Yes   \n",
       "1         low         Yes   \n",
       "2         low         Yes   \n",
       "3      medium         Yes   \n",
       "4         low         Yes   \n",
       "5      medium         Yes   \n",
       "6  crazy loud         Yes   \n",
       "7        high         Yes   \n",
       "8        high         Yes   \n",
       "9      medium         Yes   \n",
       "\n",
       "                                                    comments  \\\n",
       "0                                                        NaN   \n",
       "1                                           food tastes good   \n",
       "2                                                  good food   \n",
       "3                                    My love for Korean food   \n",
       "4                                                  Good food   \n",
       "5                                         food tasted great!   \n",
       "6                                                        NaN   \n",
       "7                                                        NaN   \n",
       "8                                                        NaN   \n",
       "9  Steak was good, ambiance was nice and server was helpful.   \n",
       "\n",
       "    restaurant_name target  \n",
       "0               NaN   like  \n",
       "1             Midam   like  \n",
       "2         pear tree   like  \n",
       "3              Dami   like  \n",
       "4               NaN   like  \n",
       "5               NaN   like  \n",
       "6  CACTUS CLUB CAFE   like  \n",
       "7               NaN   like  \n",
       "8         Frankie’s   like  \n",
       "9   Hy's steakhouse   like  "
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "restaurant_data = pd.read_csv(\"data/cleaned_restaurant_data.csv\")\n",
    "restaurant_data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What all feature transformations you would apply on this dataset? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What did we learn today?\n",
    "\n",
    "- Motivation to use `ColumnTransformer`\n",
    "- `ColumnTransformer` syntax\n",
    "- Defining transformers with multiple transformations\n",
    "- How to visualize transformed features in a dataframe \n",
    "- More on ordinal features \n",
    "- Different arguments `OneHotEncoder`\n",
    "    - `handle_unknow=\"ignore\"`\n",
    "    - `if_binary`\n",
    "- Dealing with text features\n",
    "    - Bag of words representation: `CountVectorizer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/eva-talksoon.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}