lectures/13_feature-engineering-selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 13: Feature engineering and feature selection \n",
    "\n",
    "UBC 2022 Summer\n",
    "\n",
    "Instructor: Mehrdad Oveisi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import numpy.random as npr\n",
    "import pandas as pd\n",
    "from sklearn.compose import (\n",
    "    ColumnTransformer,\n",
    "    TransformedTargetRegressor,\n",
    "    make_column_transformer,\n",
    ")\n",
    "from sklearn.dummy import DummyRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge, RidgeCV\n",
    "from sklearn.metrics import make_scorer, mean_squared_error, r2_score\n",
    "from sklearn.model_selection import cross_val_score, cross_validate, train_test_split\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Learning outcomes \n",
    "\n",
    "From this lecture, students are expected to be able to:\n",
    "\n",
    "- Explain what feature engineering is and the importance of feature engineering in building machine learning models.  \n",
    "- Carry out preliminary feature engineering on text data. \n",
    "- Explain the general concept of feature selection. \n",
    "- Discuss and compare different feature selection methods at a high level. \n",
    "- Use `sklearn`'s implementation of recursive feature elimination (`RFE`) and forward and backward selection (`SequentialFeatureSelector`). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature engineering: Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is feature engineering?  \n",
    "\n",
    "- Better features: more flexibility, higher score, we can get by with simple and more interpretable models. \n",
    "- If your features, i.e., representation is bad, whatever fancier model you build is not going to help.\n",
    "\n",
    "<blockquote>\n",
    "<b>Feature engineering</b> is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in improved model accuracy on unseen data.<br> \n",
    "- Jason Brownlee    \n",
    "</blockquote>    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Some quotes on feature engineering \n",
    "\n",
    "A quote by Pedro Domingos [A Few Useful Things to Know About Machine Learning](https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf)\n",
    "\n",
    "<blockquote>\n",
    "... At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used. \n",
    "</blockquote>\n",
    "\n",
    "\n",
    "A quote by Andrew Ng, [Machine Learning and AI via Brain simulations](https://ai.stanford.edu/~ang/slides/DeepLearning-Mar2013.pptx)\n",
    "\n",
    "<blockquote>\n",
    "Coming up with features is difficult, time-consuming, requires expert knowledge. \"Applied machine learning\" is basically feature engineering.\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Better features usually help more than a better model.\n",
    "- Good features would ideally:\n",
    "    - capture **most important aspects** of the problem\n",
    "    - allow learning with **few examples** \n",
    "    - **generalize** to new scenarios.\n",
    " \n",
    "- There is a trade-off between **simple** and **expressive** features:\n",
    "    - With **simple features** overfitting risk is low, but **scores might be low**.\n",
    "    - With **complicated features** scores can be high, but so is **overfitting risk**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### The best features may be dependent on the model you use.\n",
    "\n",
    "- Examples:\n",
    "    - For counting-based methods like decision trees, separate relevant **groups of variable values**\n",
    "        - Discretization\n",
    "           - Enables using continuous features for algorithms requiring discrete features\n",
    "    - For distance-based methods like kNN, we want different class labels to be \"far\".\n",
    "        - Standardization \n",
    "           - Avoid dominance of wide-ranging features over other features with smaller ranges\n",
    "    - For regression-based methods like linear regression, we want targets to have a linear dependency on features.\n",
    "        - Transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Domain-specific transformations\n",
    "\n",
    "In some domains there are natural transformations to do:\n",
    "- Spectrograms (sound data)\n",
    "- Wavelets (image data)\n",
    "- Convolutions \n",
    "\n",
    "![](img/spectogram.png)\n",
    "\n",
    "<!-- <img src=\"img/spectogram.png\" width=\"800\" height=\"800\"> -->\n",
    "\n",
    "[Source](https://en.wikipedia.org/wiki/Spectrogram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Two example domains where feature engineering plays an important role:\n",
    "\n",
    "- Text data\n",
    "- Audio data (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Common features used in text classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag of words \n",
    "\n",
    "- So far for text data we have been using bag of word features. \n",
    "- They are good enough for many tasks. But ... \n",
    "- This encoding **throws out a lot** of things we know about language\n",
    "- It assumes that word order is not that important.   \n",
    "- So if you want to improve the scores further on text classification tasks you carry out **feature engineering**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at some examples from research papers.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: Label \"Personalized\" Important E-mails: \n",
    "- [The Learning Behind Gmail Priority Inbox](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36955.pdf)\n",
    "- **Features**: bag of words, trigrams, regular expressions, and so on.\n",
    "- There might be some \"**globally**\" important messages:\n",
    "    - \"This is your mother, something terrible happened, give me a call ASAP.\"\n",
    "- But your \"important\" message may be **unimportant to others**.\n",
    "     - Similar for spam: \"spam\" for one user could be \"not spam\" for another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- **Social** features (e.g., percentage of sender emails that is read by the recipient)\n",
    "- **Content** features (e.g., recent terms the user has been using in emails)\n",
    "- **Thread** features (e.g., whether the user has started the thread)\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### [The Learning Behind Gmail Priority Inbox](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36955.pdf)\n",
    "\n",
    "![](img/gmail_priority_inbox.png)\n",
    "\n",
    "<!-- <img src=\"img/gmail_priority_inbox.png\" width=\"1000\" height=\"1000\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Feature engineering examples: [Automatically Identifying Good Conversations Online](http://www.courtneynapoles.com/res/icwsm17-automatically.pdf)\n",
    "\n",
    "![](img/classifying_good_conversations_online.png)\n",
    "\n",
    "<!-- <img src=\"img/classifying_good_conversations_online.png\" width=\"800\" height=\"800\"> -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### (optional) Term weighing (TF-IDF) \n",
    "\n",
    "- A measure of **relatedness** between **words** and **documents**\n",
    "- Intuition: Meaningful words may occur repeatedly in related documents, but functional words (e.g., _make_, _the_) may be distributed evenly over all documents\n",
    "\n",
    "\n",
    "$$tf.idf(w_i,d_j) = (1+log(tf_{ij})) log\\frac{D}{df_i}$$\n",
    "\n",
    "\n",
    "where, \n",
    "- $tf_{ij}$ &rarr; number of occurrences of the term $w_i$ in document $d_j$\n",
    "- $D$ &rarr; number of documents\n",
    "- $df_i$ &rarr; number of documents in which $w_i$ occurs\n",
    "\n",
    "Check `TfidfVectorizer` from `sklearn`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### N-grams \n",
    "\n",
    "- Incorporating **more context** \n",
    "- A contiguous sequence of _n_ items (characters, tokens) in text.\n",
    "    <blockquote>\n",
    "        CPSC330 students are hard-working .\n",
    "    </blockquote>    \n",
    "\n",
    "- 2-grams (bigrams): a contiguous sequence of two words\n",
    "    * _CPSC330 students, students are, are hard-working, hard-working ._\n",
    "- 3-grams (trigrams): a contiguous sequence of three words\n",
    "    * _CPSC330 students are, students are hard-working, are hard-working ._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can extract ngram features using `CountVectorizer` by passing `ngram_range`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "X = [\n",
    "    \"URGENT!! As a valued network customer you have been selected to receive a $900 prize reward!\",\n",
    "    \"Lol you are always so convincing.\",\n",
    "    \"URGENT!! Call right away!!\",\n",
    "]\n",
    "vec = CountVectorizer(ngram_range=(1, 3))\n",
    "X_counts = vec.fit_transform(X)\n",
    "bow_df = pd.DataFrame(X_counts.toarray(), columns=vec.get_feature_names_out(), index=X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>900</th>\n",
       "      <th>900 prize</th>\n",
       "      <th>900 prize reward</th>\n",
       "      <th>always</th>\n",
       "      <th>always so</th>\n",
       "      <th>always so convincing</th>\n",
       "      <th>are</th>\n",
       "      <th>are always</th>\n",
       "      <th>are always so</th>\n",
       "      <th>as</th>\n",
       "      <th>...</th>\n",
       "      <th>urgent call</th>\n",
       "      <th>urgent call right</th>\n",
       "      <th>valued</th>\n",
       "      <th>valued network</th>\n",
       "      <th>valued network customer</th>\n",
       "      <th>you</th>\n",
       "      <th>you are</th>\n",
       "      <th>you are always</th>\n",
       "      <th>you have</th>\n",
       "      <th>you have been</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a $900 prize reward!</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT!! Call right away!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    900  900 prize  \\\n",
       "URGENT!! As a valued network customer you have ...    1          1   \n",
       "Lol you are always so convincing.                     0          0   \n",
       "URGENT!! Call right away!!                            0          0   \n",
       "\n",
       "                                                    900 prize reward  always  \\\n",
       "URGENT!! As a valued network customer you have ...                 1       0   \n",
       "Lol you are always so convincing.                                  0       1   \n",
       "URGENT!! Call right away!!                                         0       0   \n",
       "\n",
       "                                                    always so  \\\n",
       "URGENT!! As a valued network customer you have ...          0   \n",
       "Lol you are always so convincing.                           1   \n",
       "URGENT!! Call right away!!                                  0   \n",
       "\n",
       "                                                    always so convincing  are  \\\n",
       "URGENT!! As a valued network customer you have ...                     0    0   \n",
       "Lol you are always so convincing.                                      1    1   \n",
       "URGENT!! Call right away!!                                             0    0   \n",
       "\n",
       "                                                    are always  are always so  \\\n",
       "URGENT!! As a valued network customer you have ...           0              0   \n",
       "Lol you are always so convincing.                            1              1   \n",
       "URGENT!! Call right away!!                                   0              0   \n",
       "\n",
       "                                                    as  ...  urgent call  \\\n",
       "URGENT!! As a valued network customer you have ...   1  ...            0   \n",
       "Lol you are always so convincing.                    0  ...            0   \n",
       "URGENT!! Call right away!!                           0  ...            1   \n",
       "\n",
       "                                                    urgent call right  valued  \\\n",
       "URGENT!! As a valued network customer you have ...                  0       1   \n",
       "Lol you are always so convincing.                                   0       0   \n",
       "URGENT!! Call right away!!                                          1       0   \n",
       "\n",
       "                                                    valued network  \\\n",
       "URGENT!! As a valued network customer you have ...               1   \n",
       "Lol you are always so convincing.                                0   \n",
       "URGENT!! Call right away!!                                       0   \n",
       "\n",
       "                                                    valued network customer  \\\n",
       "URGENT!! As a valued network customer you have ...                        1   \n",
       "Lol you are always so convincing.                                         0   \n",
       "URGENT!! Call right away!!                                                0   \n",
       "\n",
       "                                                    you  you are  \\\n",
       "URGENT!! As a valued network customer you have ...    1        0   \n",
       "Lol you are always so convincing.                     1        1   \n",
       "URGENT!! Call right away!!                            0        0   \n",
       "\n",
       "                                                    you are always  you have  \\\n",
       "URGENT!! As a valued network customer you have ...               0         1   \n",
       "Lol you are always so convincing.                                1         0   \n",
       "URGENT!! Call right away!!                                       0         0   \n",
       "\n",
       "                                                    you have been  \n",
       "URGENT!! As a valued network customer you have ...              1  \n",
       "Lol you are always so convincing.                               0  \n",
       "URGENT!! Call right away!!                                      0  \n",
       "\n",
       "[3 rows x 61 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Aside: [Google n-gram viewer](https://books.google.com/ngrams)\n",
    " \n",
    "- All Our N-gram are Belong to You\n",
    "    - https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-toyou.html\n",
    "\n",
    "<blockquote>\n",
    "Here at Google Research we have been using word n-gram models for a variety\n",
    "of R&D projects, such as statistical machine translation, speech recognition,\n",
    "spelling correction, entity detection, information extraction, and others.\n",
    "That's why we decided to share this enormous dataset with everyone. We\n",
    "processed 1,024,908,267,229 words of running text and are publishing the\n",
    "counts for all 1,176,470,663 five-word sequences that appear at least 40\n",
    "times. There are 13,588,391 unique words, after discarding words that appear\n",
    "less than 200 times.”\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"800\"\n",
       "            src=\"https://books.google.com/ngrams/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f8aba32b760>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "url = \"https://books.google.com/ngrams/\"\n",
    "IFrame(src=url, width=1000, height=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Aside: [Google n-gram viewer](https://books.google.com/ngrams)\n",
    " \n",
    "- Count the occurrences of the bigram _smart women_ in the corpus from 1800 to 2000 \n",
    "\n",
    "![](img/ngram_viewer_smart_woman.png)\n",
    "\n",
    "<!-- <img src=\"img/ngram_viewer_smart_woman.png\" width=\"800\" height=\"800\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Aside: [Google n-gram viewer](https://books.google.com/ngrams)\n",
    " \n",
    "- Trends in the word _challenge_ used as a **noun** vs. **verb**\n",
    "\n",
    "![](img/ngram_viewer_challenge_NN_VB.png)\n",
    "\n",
    "<!-- <img src=\"img/ngram_viewer_challenge_NN_VB.png\" width=\"800\" height=\"800\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Part-of-speech features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Part-of-speech (POS) in English\n",
    "\n",
    "- Part-of-speech: A kind of syntactic category that tells you some of the grammatical properties of a word.\n",
    "    * Noun &rarr; water, sun, cat  \n",
    "    * Verb &rarr; run, eat, teach\n",
    "\n",
    "    \n",
    "<blockquote>\n",
    "The ____ was running. \n",
    "</blockquote>    \n",
    "\n",
    "- Only a noun fits here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Part-of-speech (POS) features\n",
    "\n",
    "- POS features use POS information for the words in text.  \n",
    "\n",
    "<blockquote>\n",
    "    CPSC330/<span style=\"color:green\">PROPER_NOUN</span> students/<span style=\"color:green\">NOUN</span> are/<span style=\"color:green\">VERB</span> hard-working/<span style=\"color:green\">ADJECTIVE</span>\n",
    "</blockquote>    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### An example from a project \n",
    "\n",
    "- Data: a bunch of documents \n",
    "- Task: identify texts with *permissions* and identify who is giving permission to whom. \n",
    "\n",
    "<blockquote>\n",
    "<b>You</b> may <b>disclose</b> Google confidential information when compelled to do so by law if <b>you</b> provide <b>us</b> reasonable prior notice, unless a court orders that <b>we</b> not receive notice.\n",
    "</blockquote>\n",
    "\n",
    "- A very simple solution\n",
    "    * Look for **pronouns** and **verbs**. \n",
    "    * Add **POS tags** as features in your model. \n",
    "    * Maybe look up words similar to **disclose**.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Penn Treebank part-of-speech tags (bonus)\n",
    "\n",
    "![](img/PTB_POS.png)\n",
    "\n",
    "<!-- <img src=\"img/PTB_POS.png\" width=\"900\" height=\"900\"/> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- How do we extract part-of-speech information?\n",
    "- We use **pre-trained models**! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- A couple of popular libraries which include such pre-trained models. \n",
    "- `nltk`\n",
    "```\n",
    "conda install -n cpsc330 -c conda-forge nltk\n",
    "```\n",
    "- spaCy\n",
    "```\n",
    "conda install -n cpsc330 -c conda-forge spacy\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "# Uncomment the following line, run it once, and comment it out again\n",
    "# nltk.download(\"punkt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You also need to download the language model which contains all the pre-trained models. For that run the following in your course `conda` environment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the following line, run it once, and comment it out again\n",
    "# !python -m spacy download en_core_web_md\n",
    "\n",
    "# Example output upon success:\n",
    "# ✔ Download and installation successful\n",
    "# You can now load the package via spacy.load('en_core_web_md')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [spaCy](https://spacy.io/) \n",
    "\n",
    "A useful package for text processing and feature extraction\n",
    "- Active development: https://github.com/explosion/spaCy\n",
    "- Interactive lessons by Ines Montani: https://course.spacy.io/en/\n",
    "- Good documentation, easy to use, and customizable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import en_core_web_md  # pre-trained model\n",
    "import spacy\n",
    "\n",
    "nlp = en_core_web_md.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "sample_text = \"\"\"Dolly Parton is a gift to us all. \n",
    "From writing all-time great songs like “Jolene” and “I Will Always Love You”, \n",
    "to great performances in films like 9 to 5, to helping fund a COVID-19 vaccine, \n",
    "she’s given us so much. Now, Netflix bring us Dolly Parton’s Christmas on the Square, \n",
    "an original musical that stars Christine Baranski as a Scrooge-like landowner \n",
    "who threatens to evict an entire town on Christmas Eve to make room for a new mall. \n",
    "Directed and choreographed by the legendary Debbie Allen and counting Jennifer Lewis \n",
    "and Parton herself amongst its cast, Christmas on the Square seems like the perfect movie\n",
    "to save Christmas 2020. 😻 👍🏿\"\"\"\n",
    "\n",
    "# [Adapted from here.](https://thepopbreak.com/2020/11/22/dolly-partons-christmas-on-the-square-review-not-quite-a-christmas-miracle/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Spacy extracts all interesting information from text with the following call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "doc = nlp(sample_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at part-of-speech tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(Dolly, 'PROPN'), (Parton, 'PROPN'), (is, 'AUX'), (a, 'DET'), (gift, 'NOUN'), (to, 'ADP'), (us, 'PRON'), (all, 'PRON'), (., 'PUNCT'), (\n",
      ", 'SPACE'), (From, 'ADP'), (writing, 'VERB'), (all, 'DET'), (-, 'PUNCT'), (time, 'NOUN'), (great, 'ADJ'), (songs, 'NOUN'), (like, 'ADP'), (“, 'PUNCT'), (Jolene, 'PROPN')]\n"
     ]
    }
   ],
   "source": [
    "print([(token, token.pos_) for token in doc][:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Often we want to know who did what to whom. \n",
    "- **Named entities** give you this information.  \n",
    "- What are named entities in the text? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Named entities:\n",
      " [('Dolly Parton', 'PERSON'), ('Jolene', 'PERSON'), ('9 to 5', 'DATE'), ('Netflix', 'ORG'), ('Dolly Parton’s', 'PERSON'), ('Christmas', 'DATE'), ('Square', 'ORG'), ('Christine Baranski', 'PERSON'), ('Christmas Eve', 'DATE'), ('Debbie Allen', 'PERSON'), ('Jennifer Lewis', 'PERSON'), ('Parton', 'PERSON'), ('Christmas', 'DATE'), ('Christmas 2020', 'DATE')]\n",
      "\n",
      "ORG means:  Companies, agencies, institutions, etc.\n",
      "\n",
      "PERSON means:  People, including fictional\n",
      "\n",
      "DATE means:  Absolute or relative dates or periods\n"
     ]
    }
   ],
   "source": [
    "print(\"Named entities:\\n\", [(ent.text, ent.label_) for ent in doc.ents])\n",
    "print(\"\\nORG means: \", spacy.explain(\"ORG\"))\n",
    "print(\"\\nPERSON means: \", spacy.explain(\"PERSON\"))\n",
    "print(\"\\nDATE means: \", spacy.explain(\"DATE\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Dolly Parton\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " is a gift to us all. </br>From writing all-time great songs like “\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Jolene\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       "” and “I Will Always Love You”, </br>to great performances in films like \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    9 to 5\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       ", to helping fund a COVID-19 vaccine, </br>she’s given us so much. Now, \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Netflix\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " bring us \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Dolly Parton’s\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christmas\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " on the \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Square\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       ", </br>an original musical that stars \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christine Baranski\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " as a Scrooge-like landowner </br>who threatens to evict an entire town on \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christmas Eve\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " to make room for a new mall. </br>Directed and choreographed by the legendary \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Debbie Allen\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " and counting \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Jennifer Lewis\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " </br>and \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Parton\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " herself amongst its cast, \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christmas\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " on the Square seems like the perfect movie</br>to save \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christmas 2020\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       ". 😻 👍🏿</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  An example from a project \n",
    "\n",
    "Goal: _Extract_ and _visualize_ **inter-corporate relationships** from disclosed annual 10-K reports of public companies. \n",
    "\n",
    "[Source for the text below.](https://www.bbc.com/news/business-39875417)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = (\n",
    "    \"Heavy hitters, including Microsoft and Google, \"\n",
    "    \"are competing for customers in cloud services with the likes of IBM and Salesforce.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">Heavy hitters, including \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Microsoft\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Google\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       ", are competing for customers in cloud services with the likes of \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    IBM\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Salesforce\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       ".</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Named entities:\n",
      " [('Microsoft', 'ORG'), ('Google', 'ORG'), ('IBM', 'ORG'), ('Salesforce', 'ORG')]\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(text)\n",
    "displacy.render(doc, style=\"ent\")\n",
    "print(\"Named entities:\\n\", [(ent.text, ent.label_) for ent in doc.ents])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want emoji identification support install [`spacymoji`](https://pypi.org/project/spacymoji/) in the course environment.\n",
    "\n",
    "The following `pip` command will install `spacymoji` in the cpsc330 environment (which should be the current environment in this notebook):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the following line, run it once, and comment it out again\n",
    "#!pip install spacymoji\n",
    "\n",
    "# Example successful output:\n",
    "# Successfully installed emoji-0.6.0 spacymoji-3.0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After installing `spacymoji`, if it's still complaining about module not found, my guess is that you do not have `pip` installed in your `conda` environment. \n",
    "\n",
    "First, uncomment the following line and run it once, to make sure you have `pip` properly installed in your cpsc330 environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip --version\n",
    "\n",
    "# Example successful output:\n",
    "# pip 22.0.4 from YOUR_MINICONDA_PATH/miniconda3/envs/cpsc330/lib/python3.10/site-packages/pip (python 3.10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `pip` is not found or it is not in your `cpsc330` environment, then go to your course `conda` environment, and install `pip` using the following command. Then, retry the lines above to install `spacymoji`.\n",
    "\n",
    "```\n",
    "conda install -n cpsc330 pip\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from spacymoji import Emoji\n",
    "\n",
    "nlp.add_pipe(\"emoji\", first=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Does the text have any emojis? If yes, extract the description. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('😻', 138, 'smiling cat face with heart-eyes'),\n",
       " ('👍🏿', 139, 'thumbs up dark skin tone')]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp(sample_text)\n",
    "doc._.emoji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Final remarks\n",
    "\n",
    "- If we want to go beyond bag-of-words and incorporate **human knowledge in models**, we carry out **feature engineering**. \n",
    "- Some **common** features include:\n",
    "    - ngram features\n",
    "    - part-of-speech features\n",
    "    - named entity features\n",
    "    - emoticons in text\n",
    "- These are usually extracted from pre-trained models using libraries such as `spaCy`.  \n",
    "- Now a lot of this **has moved to deep learning**.\n",
    "- But industries still rely on manual feature engineering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "---\n",
    "\n",
    "## (Optional) Classify music style from audio files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note***\n",
    "\n",
    "> This subsection is your reference and will be optional for the course.\n",
    "> \n",
    "> Currently, the dataset used in the subsection is not available. Thus, all the `Code` cells in the subsection has been disabled (set to `Raw`). To run them, you need to change the cell types to `Code`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Imagine you are asked to develop a music style classification system to help a song recommendation system. \n",
    "    - E.g., something similar to what Spotify does\n",
    "- Training data: audio files along with their music styles (e.g., classical, blues, pop)\n",
    "- Prediction task: Given a new raw audio file predict the music style of in the audio. \n",
    "- You can download the data from [MARSYAS](http://marsyas.info/downloads/datasets.html). \n",
    "    - Beware of the size; it is 1.2 GB.     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We'll be using `librosa` library for feature engineering of audio files.  \n",
    "- You can install `librosa` in your conda environment as follows: \n",
    "\n",
    "```\n",
    "conda install -n cpsc330 -c conda-forge librosa\n",
    "```"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "import librosa.display\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "y, sr = librosa.load(\n",
    "    \"/Users/kvarada/MDS/data/genres/classical/classical.00001.au\", duration=10\n",
    ")\n",
    "plt.figure()\n",
    "plt.subplot(3, 1, 1)\n",
    "librosa.display.waveplot(y, sr=sr)\n",
    "plt.title(\"Monophonic\")"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "import IPython.display\n",
    "\n",
    "IPython.display.Audio(y, rate=sr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- I have taken a subset of the above dataset with genres and stored it in `music_genres_small`\n",
    "    - blues\n",
    "    - classical\n",
    "    - pop\n",
    "    - rock\n",
    "\n",
    "\n",
    "- Let's extract some domain-specific features called [MFCC features](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum) from the above subset and store it as a CSV. I am pushing this CSV in the repo. \n",
    "- (You do not need to understand the feature extraction code below.)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "data = {}\n",
    "import glob\n",
    "import ntpath"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "data_dir = \"/Users/kvarada/MDS/data/music_genres_small/\"\n",
    "raw_audio_X = np.empty(shape=(400, 1))\n",
    "raw_audio_y = []\n",
    "row_ind = 0\n",
    "\n",
    "for audio_file in glob.glob(data_dir + r\"*/*.au\"):\n",
    "    y, sr = librosa.load(audio_file, mono=True, duration=30)\n",
    "    mfcc = librosa.feature.mfcc(y=y, sr=sr)\n",
    "    chroma_stft = librosa.feature.chroma_stft(y=y, sr=sr)\n",
    "    spec_cent = librosa.feature.spectral_centroid(y=y, sr=sr)\n",
    "    spec_bw = librosa.feature.spectral_bandwidth(y=y, sr=sr)\n",
    "    rolloff = librosa.feature.spectral_rolloff(y=y, sr=sr)\n",
    "    zcr = librosa.feature.zero_crossing_rate(y)\n",
    "    filename = ntpath.basename(audio_file)\n",
    "    label = filename.split(\".\")[0]\n",
    "\n",
    "    data.setdefault(\"audio_file\", []).append(filename)\n",
    "    data.setdefault(\"label\", []).append(label)\n",
    "    data.setdefault(\"chroma_stft\", []).append(np.mean(chroma_stft))\n",
    "    data.setdefault(\"spec_cent\", []).append(np.mean(spec_cent))\n",
    "    data.setdefault(\"spec_bw\", []).append(np.mean(spec_bw))\n",
    "    data.setdefault(\"rolloff\", []).append(np.mean(rolloff))\n",
    "    data.setdefault(\"zcr\", []).append(np.mean(zcr))\n",
    "\n",
    "    # Get mfcc features\n",
    "    feat_prefix = \"mfcc\"\n",
    "    ind = 1\n",
    "    for feat in mfcc:\n",
    "        key = feat_prefix + str(ind)\n",
    "        data.setdefault(key, []).append(np.mean(feat))\n",
    "        ind += 1\n",
    "    try:\n",
    "        raw_audio_X[row_ind] = np.mean(y)\n",
    "        raw_audio_y.append(label)\n",
    "    except:\n",
    "        print(\"Broadcasting problem with file %s\" % (audio_file))\n",
    "    row_ind += 1\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "df.to_csv(\"data/music_genre.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- If we just pass raw audio data to an ML model. \n",
    "- We are just taking the mean of the audio time series which won't be much meaningful in each case. "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "raw_audio_y = np.asarray(raw_audio_y)\n",
    "raw_audio_y.shape\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    raw_audio_X, raw_audio_y, test_size=0.20, random_state=111\n",
    ")\n",
    "lr = LogisticRegression(solver=\"liblinear\")\n",
    "pd.DataFrame(cross_validate(lr, X_train, y_train, return_train_score=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try it with standard domain-specific MFCC features."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "music_cvs_df = pd.read_csv(\"data/music_genre.csv\")\n",
    "music_cvs_df.columns\n",
    "X = music_cvs_df.drop(columns=[\"audio_file\", \"label\"])\n",
    "y = music_cvs_df[\"label\"]\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.20, random_state=111\n",
    ")\n",
    "lr = LogisticRegression(solver=\"liblinear\")\n",
    "pd.DataFrame(cross_validate(lr, X_train, y_train, return_train_score=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Much better results with domain-specific features!! \n",
    "- You could improve it further with more careful feature engineering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "[End of optional subsection]\n",
    "\n",
    "---\n",
    "\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Summary\n",
    "\n",
    "- Feature engineering is finding the useful representation of the data that can help us effectively solve our problem.  \n",
    "- We looked at commonly used features in text data\n",
    "    - Bag of word features \n",
    "    - N-gram features\n",
    "    - Part-of-speech features\n",
    "    - Classify music style from audio files\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Feature engineering \n",
    "\n",
    "- The best features are **application-dependent**.\n",
    "- It's hard to give general advice, but here are **some guidelines**\n",
    "    - Ask the **domain experts**.\n",
    "    - Go through **academic papers** in the discipline. \n",
    "    - Often have idea of right discretization/standardization/transformation.\n",
    "    - If no domain expert, **cross-validation** will help.\n",
    "- If you have **lots of data**, use **deep learning** methods. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<blockquote>\n",
    "    The algorithms we used are very standard for Kagglers ... We spent most of our efforts in feature engineering... <br>\n",
    "- Xavier Conort, on winning the Flight Quest challenge on Kaggle    \n",
    "</blockquote>    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Break (5 min)\n",
    "\n",
    "![](img/eva-coffee.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature selection: Introduction and motivation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "- With so many ways to add new features, we can increase dimensionality of the data. \n",
    "- More features means more complex models, which means increasing the chance of overfitting. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is feature selection?\n",
    "\n",
    "- Find the features\t(columns) in $X$ that are important for predicting\t$y$, and remove the features that aren't. \n",
    "\n",
    "- Given $X = \\begin{bmatrix}x_1 & x_2 & \\dots & x_n\\\\  \\\\  \\\\  \\end{bmatrix}$ and $y = \\begin{bmatrix}\\\\  \\\\  \\\\  \\end{bmatrix}$, find the columns $1 \\leq j \\leq n$ in $X$ that are important for predicting $y$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **Why** feature selection? \n",
    "\n",
    "- **Interpretability**: Models are more interpretable with fewer features. If you get the same performance with 10 features instead of 500 features, why not use the model with smaller number of features?     \n",
    "- **Computation**: Models fit/predict faster with fewer columns.\n",
    "- **Data collection**: What type of new data should I collect? It may be cheaper to collect fewer columns.\n",
    "- **Fundamental tradeoff**: Can I reduce overfitting by removing useless features?\n",
    "\n",
    "Feature selection can often result in better performing (less overfit), easier to understand, and faster model. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **How** do we carry out feature selection? \n",
    "\n",
    "- There are a number of ways. \n",
    "- You could use **domain knowledge** to **discard** features. \n",
    "- We are briefly going to look at some **automatic feature selection** methods from `sklearn`: \n",
    "    - Model-based selection \n",
    "    - Recursive feature elimination\n",
    "    - Forward selection \n",
    "- Very **related** to looking at **feature importances**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "cancer = load_breast_cancer()\n",
    "# rng = np.random.RandomState(42)\n",
    "# noise = rng.normal(size=(len(cancer.data), 50)) # Let's add some random noise in `sklearn`'s breast cancer dataset.\n",
    "# X_w_noise = np.hstack([cancer.data, noise])\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, random_state=0, test_size=0.5\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(284, 30)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.025724\n",
       "score_time     0.001315\n",
       "test_score     0.968233\n",
       "train_score    0.987681\n",
       "dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr_all_feats = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))\n",
    "pipe_lr_all_feats.fit(X_train, y_train)\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_lr_all_feats, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model-based selection\n",
    "\n",
    "- Use a supervised machine learning **model to judge the importance** of each feature.\n",
    "- Keep only the most important ones. \n",
    "- Supervised machine learning **model used for feature selection** can be **different** that the one used as the **final estimator**. \n",
    "- Use a model which has some way to calculate feature importances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- To use model-based selection, we use `SelectFromModel` transformer.\n",
    "- It selects features which have the feature importances greater than the provided threshold.\n",
    "- Below I'm using `RandomForestClassifier` for feature selection with threahold \"median\" of feature importances. \n",
    "- Approximately how many features will be selected? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "select_rf = SelectFromModel(\n",
    "    RandomForestClassifier(n_estimators=100, random_state=42), threshold=\"median\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can put the feature selection transformer in a pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.294820\n",
       "score_time     0.027555\n",
       "test_score     0.950564\n",
       "train_score    0.974480\n",
       "dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr_model_based = make_pipeline(\n",
    "    StandardScaler(), select_rf, LogisticRegression(max_iter=1000)\n",
    ")\n",
    "\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_lr_model_based, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(284, 15)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr_model_based.fit(X_train, y_train)\n",
    "pipe_lr_model_based.named_steps[\"selectfrommodel\"].transform(X_train).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar results with only 15 features instead of 30 features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Recursive feature elimination (RFE)\n",
    "\n",
    "- Build a series of models\n",
    "- At each iteration, discard the **least important feature** according to the model. \n",
    "- Computationally expensive\n",
    "- Basic idea\n",
    "    - fit model\n",
    "    - find least important feature\n",
    "    - remove\n",
    "    - iterate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### RFE algorithm \n",
    "\n",
    "1. Decide $k$, the number of features to select (to keep at the end)\n",
    "2. Assign importances to features, e.g. by fitting a model and looking at `coef_` or `feature_importances_`.\n",
    "3. Remove the least important feature.\n",
    "4. Repeat steps 2-3 until only $k$ features are remaining.\n",
    "\n",
    "Note that this is **not** the same as just removing all the less important features in one shot!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([16, 12, 19, 13, 23, 20, 10,  1,  9, 22,  2, 25,  5,  7, 15,  4, 26,\n",
       "       18, 21,  8,  1,  1,  1,  6, 14, 24,  3,  1, 17, 11])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_selection import RFE\n",
    "\n",
    "# create ranking of features\n",
    "rfe = RFE(LogisticRegression(), n_features_to_select=5)\n",
    "rfe.fit(X_train_scaled, y_train)\n",
    "rfe.ranking_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False False False False False False False  True False False False False\n",
      " False False False False False False False False  True  True  True False\n",
      " False False False  True False False]\n"
     ]
    }
   ],
   "source": [
    "print(rfe.support_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "print(rfe.support_.sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "selected features: \n",
      " ['mean concave points' 'worst radius' 'worst texture' 'worst perimeter'\n",
      " 'worst concave points']\n"
     ]
    }
   ],
   "source": [
    "print(\"selected features: \\n\", cancer.feature_names[rfe.support_])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How do we know what value to pass to `n_features_to_select`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Use `RFECV` which uses cross-validation to select number of features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17 \n",
      " [False  True False  True False False  True  True  True False  True False\n",
      "  True  True False  True False False False  True  True  True  True  True\n",
      " False False  True  True False  True]\n",
      "['mean texture' 'mean area' 'mean concavity' 'mean concave points'\n",
      " 'mean symmetry' 'radius error' 'perimeter error' 'area error'\n",
      " 'compactness error' 'fractal dimension error' 'worst radius'\n",
      " 'worst texture' 'worst perimeter' 'worst area' 'worst concavity'\n",
      " 'worst concave points' 'worst fractal dimension']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import RFECV\n",
    "\n",
    "rfe_cv = RFECV(LogisticRegression(max_iter=2000), cv=10)\n",
    "rfe_cv.fit(X_train_scaled, y_train)\n",
    "print(rfe_cv.support_.sum(), '\\n', rfe_cv.support_)\n",
    "print(cancer.feature_names[rfe_cv.support_])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       2.990331\n",
       "score_time     0.015922\n",
       "test_score     0.943609\n",
       "train_score    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rfe_pipe = make_pipeline(\n",
    "    StandardScaler(),\n",
    "    RFECV(LogisticRegression(max_iter=2000), cv=10),\n",
    "    RandomForestClassifier(n_estimators=100, random_state=42),\n",
    ")\n",
    "\n",
    "pd.DataFrame(cross_validate(rfe_pipe, X_train, y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Slow because there is cross validation within cross validation \n",
    "- Not a big improvement in scores compared to all features on this toy case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Search and score\n",
    "\n",
    "- Define a **scoring function** $f(S)$ that measures the quality of the set of features $S$. \n",
    "- Now **search** for the set of features $S$ with the best score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### General idea of search and score methods \n",
    "\n",
    "- Example: Suppose you have three features: $A, B, C$\n",
    "    - Compute **score** for $S = \\{\\}$\n",
    "    - Compute **score** for $S = \\{A\\}$\n",
    "    - Compute **score** for $S= \\{B\\}$\n",
    "    - Compute **score** for $S = \\{C\\}$\n",
    "    - Compute **score** for $S = \\{A,B\\}$    \n",
    "    - Compute **score** for $S = \\{A,C\\}$\n",
    "    - Compute **score** for $S = \\{B,C\\}$\n",
    "    - Compute **score** for $S = \\{A,B,C\\}$    \n",
    "- Return $S$ with the best score.  \n",
    "- How many distinct combinations we have to try out? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Forward or backward selection \n",
    "\n",
    "- Also called wrapper methods\n",
    "- Shrink or grow feature set by removing or adding one feature at a time \n",
    "- Makes the decision based on whether adding/removing the feature improves the CV score or not"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/forward_selection.png)\n",
    "\n",
    "<!-- <img src='img/forward_selection.png' width=\"1000\" height=\"1000\" /> -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       9.790154\n",
       "score_time     0.012406\n",
       "test_score     0.933020\n",
       "train_score    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_selection import SequentialFeatureSelector\n",
    "\n",
    "pipe_forward = make_pipeline(\n",
    "    StandardScaler(),\n",
    "    SequentialFeatureSelector(LogisticRegression(max_iter=1000), direction=\"forward\"),\n",
    "    RandomForestClassifier(n_estimators=100, random_state=42),\n",
    ")\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_forward, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       13.900554\n",
       "score_time      0.012378\n",
       "test_score      0.950627\n",
       "train_score     1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_forward = make_pipeline(\n",
    "    StandardScaler(),\n",
    "    SequentialFeatureSelector(\n",
    "        LogisticRegression(max_iter=1000), direction=\"backward\", n_features_to_select=15\n",
    "    ),\n",
    "    RandomForestClassifier(n_estimators=100, random_state=42),\n",
    ")\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_forward, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other ways to search \n",
    "\n",
    "- Stochastic local search\n",
    "    - Inject randomness so that we can explore new parts of the search space\n",
    "    - Simulated annealing\n",
    "    - Genetic algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Warnings about feature selection \n",
    "\n",
    "- A feature's relevance is only defined in the context of other features.\n",
    "    - Adding/removing features can make features relevant/irrelevant.\n",
    "- If features can be predicted from other features, you cannot know which one to pick. \n",
    "- Relevance for features does not have a causal relationship. \n",
    "- Don't be overconfident. \n",
    "    - The methods we have seen probably do not discover the ground truth and how the world really works.\n",
    "    - They simply tell you which features help in predicting $y_i$ for the data you have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "---\n",
    "\n",
    "### (Optional) Problems with feature selection \n",
    "\n",
    "- The term 'relevance' is not clearly defined.\n",
    "- What all things can go wrong with feature selection?\n",
    "- Attribution: From CPSC 340. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Example: Is \"Relevance\" clearly defined?\n",
    "\n",
    "- Consider a supervised classification task of predicting whether someone has particular genetic variation (SNP)\n",
    "\n",
    "<img src='img/sex_mom_dad.png' width=\"600\" height=\"600\" />\n",
    "\n",
    "- True model: You almost have the same value as your biological mom."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- True model: You almost have the same value for SNP as your biological mom.\n",
    "    - (SNP = biological mom) with very high probability \n",
    "    - (SNP != biological mom) with very low probability \n",
    "    \n",
    "\n",
    "<img src='img/SNP.png' width=\"400\" height=\"400\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- What if \"mom\" feature is repeated?\n",
    "- Should we pick both? Should we pick one of them because it predicts the other? \n",
    "- Dependence, collinearity for linear models\n",
    "    - If a feature can be predicted from the other, don't know which one to pick. \n",
    "\n",
    "<img src='img/sex_mom_mom2_dad.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- What if we add (maternal) \"grandma\" feature?\n",
    "- Is it relevant? \n",
    "    - We can predict SNP accurately using this feature\n",
    "- Conditional independence\n",
    "    - But grandma is irrelevant given biological mom feature\n",
    "    - Relevant features may become irrelevant given other features\n",
    "\n",
    "<img src='img/sex_mom_dad_grandma.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- What if we do not know biological mom feature and we just have grandma feature\n",
    "- It becomes relevant now. \n",
    "    - Without mom feature this is the best we can do. \n",
    "- General problem (\"taco Tuesday\" problem)\n",
    "    - Features can become relevant due to missing information \n",
    "\n",
    "\n",
    "<img src='img/sex_dad_grandma.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- Are there any relevant features now?\n",
    "- They may have some common maternal ancestor.   \n",
    "- What if mom likes dad because they share SNP? \n",
    "- General problem (Confounding)\n",
    "    - Hidden features can make irrelevant features relevant.\n",
    "\n",
    "<img src='img/sex_dad.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- Now what if we have \"sibling\" feature? \n",
    "- The feature is relevant in predicting SNP but not the cause of SNP. \n",
    "- General problem (non causality)\n",
    "    - the relevant feature may not be causal \n",
    "\n",
    "<img src='img/sex_dad_sibling.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- What if you are given \"baby\" feature?\n",
    "- Now the sex feature becomes relevant. \n",
    "    - \"baby\" feature is relevant when sex == F\n",
    "    \n",
    "- General problem (context specific relevance)\n",
    "    - adding a feature can make an irrelevant feature relevant\n",
    "\n",
    "<img src='img/sex_dad_baby.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "[End of optional subsection]\n",
    "\n",
    "---\n",
    "\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Warnings about feature selection \n",
    "\n",
    "- A feature is only relevant in the context of other features.\n",
    "    - Adding/removing features can make features relevant/irrelevant.\n",
    "- Confounding factors can make irrelevant features the most relevant.\n",
    "- If features can be predicted from other other features, you cannot know which one to pick. \n",
    "- Relevance for features does not have a causal relationship. \n",
    " \n",
    "- Is feature selection completely hopeless?\n",
    "    - It is messy but we still need to do it. So we try to do our best! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### General advice on finding relevant features\n",
    "\n",
    "- Try forward selection. \n",
    "- Try other feature selection methods (e.g., `RFE`, simulated annealing, genetic algorithms)\n",
    "- Talk to domain experts; they probably have an idea why certain features are relevant.\n",
    "- Don't be overconfident. \n",
    "    - The methods we have seen probably do not discover the ground truth and how the world really works.\n",
    "    - They simply tell you which features help in predicting $y_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Relevant resources \n",
    "- [Genome-wide association study](https://en.wikipedia.org/wiki/Genome-wide_association_study)\n",
    "- [sklearn feature selection](https://scikit-learn.org/stable/modules/feature_selection.html)\n",
    "- [PyData: A Practical Guide to Dimensionality Reduction Techniques](https://www.youtube.com/watch?v=ioXKxulmwVQ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### True/False questions for class discussion \n",
    "\n",
    "1. Simple association-based feature selection approaches do not take into account the interaction between features.\n",
    "2. You can carry out feature selection using linear models by pruning the features which have very small weights (i.e., coefficients less than a threshold).\n",
    "3. Forward search is guaranteed to find the best feature set.  \n",
    "4. The order of features removed given by `rfe.ranking_` is the same as the order of original feature importances given by the model.  "
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}