Advanced Featuretools RUL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Remaining Useful Life (advanced)\n",
    "<p style=\"margin:30px\">\n",
    "    <img style=\"display:inline; margin-right:50px\" width=50% src=\"https://www.featuretools.com/wp-content/uploads/2017/12/FeatureLabs-Logo-Tangerine-800.png\" alt=\"Featuretools\" />\n",
    "    <img style=\"display:inline\" width=15% src=\"https://upload.wikimedia.org/wikipedia/commons/e/e5/NASA_logo.svg\" alt=\"NASA\" />\n",
    "</p>\n",
    "\n",
    "This notebook has a more advanced workflow than [the other notebook](Simple%20Featuretools%20RUL%20Demo.ipynb) for predicting Remaining Useful Life (RUL). If you are a new to either this dataset or Featuretools, I would recommend reading the other notebook first.\n",
    "\n",
    "## Highlights\n",
    "* Demonstrate how novel entityset structures improve predictive accuracy\n",
    "* Use TSFresh Primitives from a featuretools [addon](https://docs.featuretools.com/getting_started/install.html#add-ons)\n",
    "* Improve Mean Absolute Error by tuning hyper parameters with [BTB](https://github.com/HDI-Project/BTB)\n",
    "\n",
    "Here is a collection of mean absolute errors from both notebooks. Though we've used averages where possible (denoted by \\*), the randomness in the Random Forest Regressor and how we choose labels from the train data changes the score.\n",
    "\n",
    "|                                 | Train/Validation MAE|  Test MAE|\n",
    "|---------------------------------|---------------------|----------|\n",
    "| Median Baseline                 | 72.06*              | 50.66*   |\n",
    "| Simple Featuretools             | 40.92*              | 39.56    |\n",
    "| Advanced: Custom Primitives     | 35.90*              | 28.84    |\n",
    "| Advanced: Hyperparameter Tuning | 34.80*              | 27.85    |\n",
    "\n",
    "\n",
    "# Step 1: Load Data\n",
    "We load in the train data using the same function we used in the previous notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import composeml as cp\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import featuretools as ft\n",
    "import utils\n",
    "import os\n",
    "\n",
    "from tqdm import tqdm\n",
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data with:\n",
      "61249 Recordings\n",
      "249 Engines\n",
      "21 Sensor Measurements\n",
      "3 Operational Settings\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>engine_no</th>\n",
       "      <th>time_in_cycles</th>\n",
       "      <th>operational_setting_1</th>\n",
       "      <th>operational_setting_2</th>\n",
       "      <th>operational_setting_3</th>\n",
       "      <th>sensor_measurement_1</th>\n",
       "      <th>sensor_measurement_2</th>\n",
       "      <th>sensor_measurement_3</th>\n",
       "      <th>sensor_measurement_4</th>\n",
       "      <th>sensor_measurement_5</th>\n",
       "      <th>...</th>\n",
       "      <th>sensor_measurement_14</th>\n",
       "      <th>sensor_measurement_15</th>\n",
       "      <th>sensor_measurement_16</th>\n",
       "      <th>sensor_measurement_17</th>\n",
       "      <th>sensor_measurement_18</th>\n",
       "      <th>sensor_measurement_19</th>\n",
       "      <th>sensor_measurement_20</th>\n",
       "      <th>sensor_measurement_21</th>\n",
       "      <th>index</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>42.0049</td>\n",
       "      <td>0.8400</td>\n",
       "      <td>100.0</td>\n",
       "      <td>445.00</td>\n",
       "      <td>549.68</td>\n",
       "      <td>1343.43</td>\n",
       "      <td>1112.93</td>\n",
       "      <td>3.91</td>\n",
       "      <td>...</td>\n",
       "      <td>8074.83</td>\n",
       "      <td>9.3335</td>\n",
       "      <td>0.02</td>\n",
       "      <td>330</td>\n",
       "      <td>2212</td>\n",
       "      <td>100.00</td>\n",
       "      <td>10.62</td>\n",
       "      <td>6.3670</td>\n",
       "      <td>0</td>\n",
       "      <td>2000-01-01 00:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>20.0020</td>\n",
       "      <td>0.7002</td>\n",
       "      <td>100.0</td>\n",
       "      <td>491.19</td>\n",
       "      <td>606.07</td>\n",
       "      <td>1477.61</td>\n",
       "      <td>1237.50</td>\n",
       "      <td>9.35</td>\n",
       "      <td>...</td>\n",
       "      <td>8046.13</td>\n",
       "      <td>9.1913</td>\n",
       "      <td>0.02</td>\n",
       "      <td>361</td>\n",
       "      <td>2324</td>\n",
       "      <td>100.00</td>\n",
       "      <td>24.37</td>\n",
       "      <td>14.6552</td>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-01 00:10:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>42.0038</td>\n",
       "      <td>0.8409</td>\n",
       "      <td>100.0</td>\n",
       "      <td>445.00</td>\n",
       "      <td>548.95</td>\n",
       "      <td>1343.12</td>\n",
       "      <td>1117.05</td>\n",
       "      <td>3.91</td>\n",
       "      <td>...</td>\n",
       "      <td>8066.62</td>\n",
       "      <td>9.4007</td>\n",
       "      <td>0.02</td>\n",
       "      <td>329</td>\n",
       "      <td>2212</td>\n",
       "      <td>100.00</td>\n",
       "      <td>10.48</td>\n",
       "      <td>6.4213</td>\n",
       "      <td>2</td>\n",
       "      <td>2000-01-01 00:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>42.0000</td>\n",
       "      <td>0.8400</td>\n",
       "      <td>100.0</td>\n",
       "      <td>445.00</td>\n",
       "      <td>548.70</td>\n",
       "      <td>1341.24</td>\n",
       "      <td>1118.03</td>\n",
       "      <td>3.91</td>\n",
       "      <td>...</td>\n",
       "      <td>8076.05</td>\n",
       "      <td>9.3369</td>\n",
       "      <td>0.02</td>\n",
       "      <td>328</td>\n",
       "      <td>2212</td>\n",
       "      <td>100.00</td>\n",
       "      <td>10.54</td>\n",
       "      <td>6.4176</td>\n",
       "      <td>3</td>\n",
       "      <td>2000-01-01 00:30:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>25.0063</td>\n",
       "      <td>0.6207</td>\n",
       "      <td>60.0</td>\n",
       "      <td>462.54</td>\n",
       "      <td>536.10</td>\n",
       "      <td>1255.23</td>\n",
       "      <td>1033.59</td>\n",
       "      <td>7.05</td>\n",
       "      <td>...</td>\n",
       "      <td>7865.80</td>\n",
       "      <td>10.8366</td>\n",
       "      <td>0.02</td>\n",
       "      <td>305</td>\n",
       "      <td>1915</td>\n",
       "      <td>84.93</td>\n",
       "      <td>14.03</td>\n",
       "      <td>8.6754</td>\n",
       "      <td>4</td>\n",
       "      <td>2000-01-01 00:40:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       engine_no  time_in_cycles  operational_setting_1  \\\n",
       "index                                                     \n",
       "0              1               1                42.0049   \n",
       "1              1               2                20.0020   \n",
       "2              1               3                42.0038   \n",
       "3              1               4                42.0000   \n",
       "4              1               5                25.0063   \n",
       "\n",
       "       operational_setting_2  operational_setting_3  sensor_measurement_1  \\\n",
       "index                                                                       \n",
       "0                     0.8400                  100.0                445.00   \n",
       "1                     0.7002                  100.0                491.19   \n",
       "2                     0.8409                  100.0                445.00   \n",
       "3                     0.8400                  100.0                445.00   \n",
       "4                     0.6207                   60.0                462.54   \n",
       "\n",
       "       sensor_measurement_2  sensor_measurement_3  sensor_measurement_4  \\\n",
       "index                                                                     \n",
       "0                    549.68               1343.43               1112.93   \n",
       "1                    606.07               1477.61               1237.50   \n",
       "2                    548.95               1343.12               1117.05   \n",
       "3                    548.70               1341.24               1118.03   \n",
       "4                    536.10               1255.23               1033.59   \n",
       "\n",
       "       sensor_measurement_5  ...  sensor_measurement_14  \\\n",
       "index                        ...                          \n",
       "0                      3.91  ...                8074.83   \n",
       "1                      9.35  ...                8046.13   \n",
       "2                      3.91  ...                8066.62   \n",
       "3                      3.91  ...                8076.05   \n",
       "4                      7.05  ...                7865.80   \n",
       "\n",
       "       sensor_measurement_15  sensor_measurement_16  sensor_measurement_17  \\\n",
       "index                                                                        \n",
       "0                     9.3335                   0.02                    330   \n",
       "1                     9.1913                   0.02                    361   \n",
       "2                     9.4007                   0.02                    329   \n",
       "3                     9.3369                   0.02                    328   \n",
       "4                    10.8366                   0.02                    305   \n",
       "\n",
       "       sensor_measurement_18  sensor_measurement_19  sensor_measurement_20  \\\n",
       "index                                                                        \n",
       "0                       2212                 100.00                  10.62   \n",
       "1                       2324                 100.00                  24.37   \n",
       "2                       2212                 100.00                  10.48   \n",
       "3                       2212                 100.00                  10.54   \n",
       "4                       1915                  84.93                  14.03   \n",
       "\n",
       "       sensor_measurement_21  index                time  \n",
       "index                                                    \n",
       "0                     6.3670      0 2000-01-01 00:00:00  \n",
       "1                    14.6552      1 2000-01-01 00:10:00  \n",
       "2                     6.4213      2 2000-01-01 00:20:00  \n",
       "3                     6.4176      3 2000-01-01 00:30:00  \n",
       "4                     8.6754      4 2000-01-01 00:40:00  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path = 'data/train_FD004.txt'\n",
    "data = utils.load_data(data_path)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also make cutoff times by using [Compose](https://compose.featurelabs.com) for generating labels on engines that reach at least 100 cycles. For each engine, we generate 10 labels that are spaced 10 cycles apart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed: 00:01 | Remaining: 00:00 | Progress: 100%|██████████| engine_no: 2490/2490 \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>engine_no</th>\n",
       "      <th>time</th>\n",
       "      <th>remaining_useful_life</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-01 16:40:00</td>\n",
       "      <td>220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-01 18:20:00</td>\n",
       "      <td>210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-01 20:00:00</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-01 21:40:00</td>\n",
       "      <td>190</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-01 23:20:00</td>\n",
       "      <td>180</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    engine_no                time  remaining_useful_life\n",
       "id                                                      \n",
       "0           1 2000-01-01 16:40:00                    220\n",
       "1           1 2000-01-01 18:20:00                    210\n",
       "2           1 2000-01-01 20:00:00                    200\n",
       "3           1 2000-01-01 21:40:00                    190\n",
       "4           1 2000-01-01 23:20:00                    180"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def remaining_useful_life(df):\n",
    "    return len(df) - 1\n",
    "\n",
    "lm = cp.LabelMaker(\n",
    "    target_entity='engine_no',\n",
    "    time_index='time',\n",
    "    labeling_function=remaining_useful_life,\n",
    ")\n",
    "\n",
    "label_times = lm.search(\n",
    "    data.sort_values('time'),\n",
    "    num_examples_per_instance=10,\n",
    "    minimum_data=100,\n",
    "    gap=10,\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "label_times.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to make 5 sets of cutoff times to use for cross validation by random sampling the labels times we created previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>engine_no</th>\n",
       "      <th>time</th>\n",
       "      <th>remaining_useful_life</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-01 18:20:00</td>\n",
       "      <td>210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-02 07:40:00</td>\n",
       "      <td>130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2</td>\n",
       "      <td>2000-01-03 22:10:00</td>\n",
       "      <td>198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2</td>\n",
       "      <td>2000-01-04 04:50:00</td>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>3</td>\n",
       "      <td>2000-01-06 11:40:00</td>\n",
       "      <td>136</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    engine_no                time  remaining_useful_life\n",
       "id                                                      \n",
       "1           1 2000-01-01 18:20:00                    210\n",
       "9           1 2000-01-02 07:40:00                    130\n",
       "10          2 2000-01-03 22:10:00                    198\n",
       "14          2 2000-01-04 04:50:00                    158\n",
       "27          3 2000-01-06 11:40:00                    136"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splits = 5\n",
    "cutoff_time_list = []\n",
    "\n",
    "for i in range(splits):\n",
    "    sample = label_times.sample(n=249, random_state=i)\n",
    "    sample.sort_index(inplace=True)\n",
    "    cutoff_time_list.append(sample)\n",
    "\n",
    "cutoff_time_list[0].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to do something fancy for our entityset. The values for `operational_setting` 1-3 are continuous but create an implicit relation between different engines. If two engines have a similar `operational_setting`, it could indicate that we should expect the sensor measurements to mean similar things. We make clusters of those settings using [KMeans](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) from scikit-learn and make a new entity from the clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entityset: Dataset\n",
       "  Entities:\n",
       "    recordings [Rows: 61249, Columns: 29]\n",
       "    engines [Rows: 249, Columns: 2]\n",
       "    settings_clusters [Rows: 50, Columns: 2]\n",
       "  Relationships:\n",
       "    recordings.engine_no -> engines.engine_no\n",
       "    recordings.settings_clusters -> settings_clusters.settings_clusters"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nclusters = 50\n",
    "\n",
    "def make_entityset(data, nclusters, kmeans=None):\n",
    "    X = data[[\n",
    "        'operational_setting_1',\n",
    "        'operational_setting_2',\n",
    "        'operational_setting_3',\n",
    "    ]]\n",
    "\n",
    "    if kmeans is None:\n",
    "        kmeans = KMeans(n_clusters=nclusters).fit(X)\n",
    "\n",
    "    data['settings_clusters'] = kmeans.predict(X)\n",
    "\n",
    "    es = ft.EntitySet('Dataset')\n",
    "\n",
    "    es.entity_from_dataframe(\n",
    "        dataframe=data,\n",
    "        entity_id='recordings',\n",
    "        index='index',\n",
    "        time_index='time',\n",
    "    )\n",
    "\n",
    "    es.normalize_entity(\n",
    "        base_entity_id='recordings',\n",
    "        new_entity_id='engines',\n",
    "        index='engine_no',\n",
    "    )\n",
    "\n",
    "    es.normalize_entity(\n",
    "        base_entity_id='recordings',\n",
    "        new_entity_id='settings_clusters',\n",
    "        index='settings_clusters',\n",
    "    )\n",
    "\n",
    "    return es, kmeans\n",
    "\n",
    "\n",
    "es, kmeans = make_entityset(data, nclusters)\n",
    "es"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize EntitySet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: Dataset Pages: 1 -->\n",
       "<svg width=\"556pt\" height=\"540pt\"\n",
       " viewBox=\"0.00 0.00 555.83 540.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 536)\">\n",
       "<title>Dataset</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-536 551.83,-536 551.83,4 -4,4\"/>\n",
       "<!-- recordings -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>recordings</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"167.83,-95.5 167.83,-531.5 379,-531.5 379,-95.5 167.83,-95.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"273.42\" y=\"-516.3\" font-family=\"Times,serif\" font-size=\"14.00\">recordings (61249 rows)</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"167.83,-509.5 379,-509.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-494.3\" font-family=\"Times,serif\" font-size=\"14.00\">index : index</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-480.3\" font-family=\"Times,serif\" font-size=\"14.00\">engine_no : id</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-466.3\" font-family=\"Times,serif\" font-size=\"14.00\">time_in_cycles : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-452.3\" font-family=\"Times,serif\" font-size=\"14.00\">operational_setting_1 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-438.3\" font-family=\"Times,serif\" font-size=\"14.00\">operational_setting_2 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-424.3\" font-family=\"Times,serif\" font-size=\"14.00\">operational_setting_3 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-410.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_1 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-396.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_2 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-382.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_3 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-368.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_4 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-354.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_5 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-340.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_6 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-326.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_7 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-312.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_8 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-298.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_9 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-284.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_10 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-270.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_11 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-256.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_12 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-242.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_13 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-228.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_14 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-214.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_15 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-200.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_16 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-186.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_17 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-172.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_18 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_19 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-144.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_20 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-130.3\" font-family=\"Times,serif\" font-size=\"14.00\">sensor_measurement_21 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-116.3\" font-family=\"Times,serif\" font-size=\"14.00\">time : datetime_time_index</text>\n",
       "<text text-anchor=\"start\" x=\"175.83\" y=\"-102.3\" font-family=\"Times,serif\" font-size=\"14.00\">settings_clusters : id</text>\n",
       "</g>\n",
       "<!-- engines -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>engines</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-58.5 264.83,-58.5 264.83,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"132.42\" y=\"-43.3\" font-family=\"Times,serif\" font-size=\"14.00\">engines (249 rows)</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"0,-36.5 264.83,-36.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-21.3\" font-family=\"Times,serif\" font-size=\"14.00\">engine_no : index</text>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-7.3\" font-family=\"Times,serif\" font-size=\"14.00\">first_recordings_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- recordings&#45;&gt;engines -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>recordings&#45;&gt;engines</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M216.33,-95.37C216.33,-95.37 216.33,-68.76 216.33,-68.76\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"219.83,-68.76 216.33,-58.76 212.83,-68.76 219.83,-68.76\"/>\n",
       "<text text-anchor=\"middle\" x=\"187.18\" y=\"-70.86\" font-family=\"Times,serif\" font-size=\"14.00\">engine_no</text>\n",
       "</g>\n",
       "<!-- settings_clusters -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>settings_clusters</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"283,-0.5 283,-58.5 547.83,-58.5 547.83,-0.5 283,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"415.42\" y=\"-43.3\" font-family=\"Times,serif\" font-size=\"14.00\">settings_clusters (50 rows)</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"283,-36.5 547.83,-36.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"291\" y=\"-21.3\" font-family=\"Times,serif\" font-size=\"14.00\">settings_clusters : index</text>\n",
       "<text text-anchor=\"start\" x=\"291\" y=\"-7.3\" font-family=\"Times,serif\" font-size=\"14.00\">first_recordings_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- recordings&#45;&gt;settings_clusters -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>recordings&#45;&gt;settings_clusters</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M331,-95.37C331,-95.37 331,-68.76 331,-68.76\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"334.5,-68.76 331,-58.76 327.5,-68.76 334.5,-68.76\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.73\" y=\"-70.86\" font-family=\"Times,serif\" font-size=\"14.00\">settings_clusters</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x138103e50>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: DFS and Creating a Model\n",
    "In addition to changing our `EntitySet` structure, we're also going to use the [Complexity](http://tsfresh.readthedocs.io/en/latest/api/tsfresh.feature_extraction.html#tsfresh.feature_extraction.feature_calculators.cid_ce) time series primitive from the featuretools [addon](https://docs.featuretools.com/getting_started/install.html#add-ons) of ready-to-use TSFresh Primitives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 304 features\n",
      "Elapsed: 02:31 | Progress: 100%|██████████\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LAST(recordings.sensor_measurement_21)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_4)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_9)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_10)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_2)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_11)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_8)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_18)</th>\n",
       "      <th>LAST(recordings.operational_setting_2)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_5)</th>\n",
       "      <th>...</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.CID_CE(recordings.sensor_measurement_14, normalize=False), normalize=False)</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.LAST(recordings.sensor_measurement_20), normalize=False)</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.MAX(recordings.sensor_measurement_16), normalize=False)</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.MAX(recordings.sensor_measurement_19), normalize=False)</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.LAST(recordings.operational_setting_3), normalize=False)</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.LAST(recordings.sensor_measurement_21), normalize=False)</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.MAX(recordings.sensor_measurement_4), normalize=False)</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.MAX(recordings.operational_setting_1), normalize=False)</th>\n",
       "      <th>CID_CE(recordings.settings_clusters.MAX(recordings.sensor_measurement_5), normalize=False)</th>\n",
       "      <th>remaining_useful_life</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>engine_no</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6.3072</td>\n",
       "      <td>1113.93</td>\n",
       "      <td>8304.43</td>\n",
       "      <td>1.01</td>\n",
       "      <td>548.90</td>\n",
       "      <td>41.80</td>\n",
       "      <td>2211.96</td>\n",
       "      <td>2212</td>\n",
       "      <td>0.8400</td>\n",
       "      <td>3.91</td>\n",
       "      <td>...</td>\n",
       "      <td>65.844882</td>\n",
       "      <td>139.955323</td>\n",
       "      <td>0.064807</td>\n",
       "      <td>79.742945</td>\n",
       "      <td>211.660105</td>\n",
       "      <td>84.039649</td>\n",
       "      <td>1693.873832</td>\n",
       "      <td>208.779151</td>\n",
       "      <td>51.059638</td>\n",
       "      <td>210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6.2907</td>\n",
       "      <td>1113.44</td>\n",
       "      <td>8307.98</td>\n",
       "      <td>1.02</td>\n",
       "      <td>548.92</td>\n",
       "      <td>41.71</td>\n",
       "      <td>2211.97</td>\n",
       "      <td>2212</td>\n",
       "      <td>0.8400</td>\n",
       "      <td>3.91</td>\n",
       "      <td>...</td>\n",
       "      <td>91.824908</td>\n",
       "      <td>180.779167</td>\n",
       "      <td>0.086023</td>\n",
       "      <td>106.560992</td>\n",
       "      <td>282.842712</td>\n",
       "      <td>108.513678</td>\n",
       "      <td>2220.595981</td>\n",
       "      <td>269.414339</td>\n",
       "      <td>65.551137</td>\n",
       "      <td>130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.8027</td>\n",
       "      <td>1128.23</td>\n",
       "      <td>8341.37</td>\n",
       "      <td>1.02</td>\n",
       "      <td>555.49</td>\n",
       "      <td>41.81</td>\n",
       "      <td>2222.92</td>\n",
       "      <td>2223</td>\n",
       "      <td>0.8402</td>\n",
       "      <td>5.48</td>\n",
       "      <td>...</td>\n",
       "      <td>242.126348</td>\n",
       "      <td>134.050263</td>\n",
       "      <td>0.064807</td>\n",
       "      <td>73.827621</td>\n",
       "      <td>195.959179</td>\n",
       "      <td>80.411061</td>\n",
       "      <td>1638.058020</td>\n",
       "      <td>199.766926</td>\n",
       "      <td>48.863007</td>\n",
       "      <td>198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.8949</td>\n",
       "      <td>1118.05</td>\n",
       "      <td>8346.87</td>\n",
       "      <td>1.02</td>\n",
       "      <td>555.80</td>\n",
       "      <td>41.81</td>\n",
       "      <td>2222.86</td>\n",
       "      <td>2223</td>\n",
       "      <td>0.8403</td>\n",
       "      <td>5.48</td>\n",
       "      <td>...</td>\n",
       "      <td>302.730551</td>\n",
       "      <td>154.102034</td>\n",
       "      <td>0.072111</td>\n",
       "      <td>90.420000</td>\n",
       "      <td>240.000000</td>\n",
       "      <td>92.203085</td>\n",
       "      <td>1897.506399</td>\n",
       "      <td>230.525935</td>\n",
       "      <td>56.138476</td>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.5137</td>\n",
       "      <td>1402.21</td>\n",
       "      <td>9072.08</td>\n",
       "      <td>1.30</td>\n",
       "      <td>642.13</td>\n",
       "      <td>47.27</td>\n",
       "      <td>2388.12</td>\n",
       "      <td>2388</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>14.62</td>\n",
       "      <td>...</td>\n",
       "      <td>326.066224</td>\n",
       "      <td>184.877804</td>\n",
       "      <td>0.089443</td>\n",
       "      <td>110.741431</td>\n",
       "      <td>293.938769</td>\n",
       "      <td>111.023411</td>\n",
       "      <td>2315.993077</td>\n",
       "      <td>271.827606</td>\n",
       "      <td>66.096208</td>\n",
       "      <td>136</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 305 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           LAST(recordings.sensor_measurement_21)  \\\n",
       "engine_no                                           \n",
       "1                                          6.3072   \n",
       "1                                          6.2907   \n",
       "2                                          8.8027   \n",
       "2                                          8.8949   \n",
       "3                                         23.5137   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_4)  \\\n",
       "engine_no                                          \n",
       "1                                        1113.93   \n",
       "1                                        1113.44   \n",
       "2                                        1128.23   \n",
       "2                                        1118.05   \n",
       "3                                        1402.21   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_9)  \\\n",
       "engine_no                                          \n",
       "1                                        8304.43   \n",
       "1                                        8307.98   \n",
       "2                                        8341.37   \n",
       "2                                        8346.87   \n",
       "3                                        9072.08   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_10)  \\\n",
       "engine_no                                           \n",
       "1                                            1.01   \n",
       "1                                            1.02   \n",
       "2                                            1.02   \n",
       "2                                            1.02   \n",
       "3                                            1.30   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_2)  \\\n",
       "engine_no                                          \n",
       "1                                         548.90   \n",
       "1                                         548.92   \n",
       "2                                         555.49   \n",
       "2                                         555.80   \n",
       "3                                         642.13   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_11)  \\\n",
       "engine_no                                           \n",
       "1                                           41.80   \n",
       "1                                           41.71   \n",
       "2                                           41.81   \n",
       "2                                           41.81   \n",
       "3                                           47.27   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_8)  \\\n",
       "engine_no                                          \n",
       "1                                        2211.96   \n",
       "1                                        2211.97   \n",
       "2                                        2222.92   \n",
       "2                                        2222.86   \n",
       "3                                        2388.12   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_18)  \\\n",
       "engine_no                                           \n",
       "1                                            2212   \n",
       "1                                            2212   \n",
       "2                                            2223   \n",
       "2                                            2223   \n",
       "3                                            2388   \n",
       "\n",
       "           LAST(recordings.operational_setting_2)  \\\n",
       "engine_no                                           \n",
       "1                                          0.8400   \n",
       "1                                          0.8400   \n",
       "2                                          0.8402   \n",
       "2                                          0.8403   \n",
       "3                                          0.0000   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_5)  ...  \\\n",
       "engine_no                                         ...   \n",
       "1                                           3.91  ...   \n",
       "1                                           3.91  ...   \n",
       "2                                           5.48  ...   \n",
       "2                                           5.48  ...   \n",
       "3                                          14.62  ...   \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.CID_CE(recordings.sensor_measurement_14, normalize=False), normalize=False)  \\\n",
       "engine_no                                                                                                                    \n",
       "1                                                  65.844882                                                                 \n",
       "1                                                  91.824908                                                                 \n",
       "2                                                 242.126348                                                                 \n",
       "2                                                 302.730551                                                                 \n",
       "3                                                 326.066224                                                                 \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.LAST(recordings.sensor_measurement_20), normalize=False)  \\\n",
       "engine_no                                                                                                 \n",
       "1                                                 139.955323                                              \n",
       "1                                                 180.779167                                              \n",
       "2                                                 134.050263                                              \n",
       "2                                                 154.102034                                              \n",
       "3                                                 184.877804                                              \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.MAX(recordings.sensor_measurement_16), normalize=False)  \\\n",
       "engine_no                                                                                                \n",
       "1                                                   0.064807                                             \n",
       "1                                                   0.086023                                             \n",
       "2                                                   0.064807                                             \n",
       "2                                                   0.072111                                             \n",
       "3                                                   0.089443                                             \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.MAX(recordings.sensor_measurement_19), normalize=False)  \\\n",
       "engine_no                                                                                                \n",
       "1                                                  79.742945                                             \n",
       "1                                                 106.560992                                             \n",
       "2                                                  73.827621                                             \n",
       "2                                                  90.420000                                             \n",
       "3                                                 110.741431                                             \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.LAST(recordings.operational_setting_3), normalize=False)  \\\n",
       "engine_no                                                                                                 \n",
       "1                                                 211.660105                                              \n",
       "1                                                 282.842712                                              \n",
       "2                                                 195.959179                                              \n",
       "2                                                 240.000000                                              \n",
       "3                                                 293.938769                                              \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.LAST(recordings.sensor_measurement_21), normalize=False)  \\\n",
       "engine_no                                                                                                 \n",
       "1                                                  84.039649                                              \n",
       "1                                                 108.513678                                              \n",
       "2                                                  80.411061                                              \n",
       "2                                                  92.203085                                              \n",
       "3                                                 111.023411                                              \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.MAX(recordings.sensor_measurement_4), normalize=False)  \\\n",
       "engine_no                                                                                               \n",
       "1                                                1693.873832                                            \n",
       "1                                                2220.595981                                            \n",
       "2                                                1638.058020                                            \n",
       "2                                                1897.506399                                            \n",
       "3                                                2315.993077                                            \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.MAX(recordings.operational_setting_1), normalize=False)  \\\n",
       "engine_no                                                                                                \n",
       "1                                                 208.779151                                             \n",
       "1                                                 269.414339                                             \n",
       "2                                                 199.766926                                             \n",
       "2                                                 230.525935                                             \n",
       "3                                                 271.827606                                             \n",
       "\n",
       "           CID_CE(recordings.settings_clusters.MAX(recordings.sensor_measurement_5), normalize=False)  \\\n",
       "engine_no                                                                                               \n",
       "1                                                  51.059638                                            \n",
       "1                                                  65.551137                                            \n",
       "2                                                  48.863007                                            \n",
       "2                                                  56.138476                                            \n",
       "3                                                  66.096208                                            \n",
       "\n",
       "           remaining_useful_life  \n",
       "engine_no                         \n",
       "1                            210  \n",
       "1                            130  \n",
       "2                            198  \n",
       "2                            158  \n",
       "3                            136  \n",
       "\n",
       "[5 rows x 305 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from featuretools.tsfresh import CidCe\n",
    "\n",
    "fm, features = ft.dfs(\n",
    "    entityset=es,\n",
    "    target_entity='engines',\n",
    "    agg_primitives=['last', 'max', CidCe(normalize=False)],\n",
    "    trans_primitives=[],\n",
    "    chunk_size=.26,\n",
    "    cutoff_time=cutoff_time_list[0],\n",
    "    max_depth=3,\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "fm.to_csv('advanced_fm.csv')\n",
    "fm.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We build 4 more feature matrices with the same feature set but different cutoff times. That lets us test the pipeline multiple times before using it on test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [08:39<00:00, 129.77s/it]\n"
     ]
    }
   ],
   "source": [
    "fm_list = [fm]\n",
    "\n",
    "for i in tqdm(range(1, splits)):\n",
    "    es = make_entityset(data, nclusters, kmeans=kmeans)[0]\n",
    "    fm = ft.calculate_feature_matrix(\n",
    "        entityset=es,\n",
    "        features=features,\n",
    "        chunk_size=.26,\n",
    "        cutoff_time=cutoff_time_list[i],\n",
    "    )\n",
    "    fm_list.append(fm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.feature_selection import RFE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46.2, 34.8, 36.8, 34.6, 37.2]\n",
      "Average MAE: 37.9, Std: 4.25\n",
      "\n",
      "1: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_13)) [0.090]\n",
      "2: MAX(recordings.sensor_measurement_13) [0.078]\n",
      "3: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_4)) [0.054]\n",
      "4: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_11)) [0.050]\n",
      "5: MAX(recordings.sensor_measurement_17) [0.048]\n",
      "-----\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def pipeline_for_test(fm_list, hyperparams=None, do_selection=False):\n",
    "    scores = []\n",
    "    regs = []\n",
    "    selectors = []\n",
    "\n",
    "    hyperparams = hyperparams or {\n",
    "        'n_estimators': 100,\n",
    "        'max_feats': 50,\n",
    "        'nfeats': 50,\n",
    "    }\n",
    "\n",
    "    for fm in fm_list:\n",
    "        X = fm.copy().fillna(0)\n",
    "        y = X.pop('remaining_useful_life')\n",
    "\n",
    "        n_estimators = int(hyperparams['n_estimators'])\n",
    "        max_features = int(hyperparams['max_feats'])\n",
    "        max_features = min(max_features, int(hyperparams['nfeats']))\n",
    "        reg = RandomForestRegressor(n_estimators=n_estimators, max_features=max_features)\n",
    "\n",
    "        X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "\n",
    "        if do_selection:\n",
    "            reg2 = RandomForestRegressor(n_estimators=10, n_jobs=3)\n",
    "            selector = RFE(reg2, int(hyperparams['nfeats']), step=25)\n",
    "            selector.fit(X_train, y_train)\n",
    "            X_train = selector.transform(X_train)\n",
    "            X_test = selector.transform(X_test)\n",
    "            selectors.append(selector)\n",
    "\n",
    "        reg.fit(X_train, y_train)\n",
    "        regs.append(reg)\n",
    "\n",
    "        preds = reg.predict(X_test)\n",
    "        mae = mean_absolute_error(preds, y_test)\n",
    "        scores.append(mae)\n",
    "\n",
    "    return scores, regs, selectors\n",
    "\n",
    "\n",
    "scores, regs, selectors = pipeline_for_test(fm_list)\n",
    "print([float('{:.1f}'.format(score)) for score in scores])\n",
    "\n",
    "mean, std = np.mean(scores), np.std(scores)\n",
    "info = 'Average MAE: {:.1f}, Std: {:.2f}\\n'\n",
    "print(info.format(mean, std))\n",
    "\n",
    "most_imp_feats = utils.feature_importances(fm_list[0], regs[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data with:\n",
      "41214 Recordings\n",
      "248 Engines\n",
      "21 Sensor Measurements\n",
      "3 Operational Settings\n",
      "Elapsed: 00:03 | Progress: 100%|██████████\n",
      "Mean Abs Error: 28.87\n"
     ]
    }
   ],
   "source": [
    "data_test = utils.load_data('data/test_FD004.txt')\n",
    "\n",
    "es_test, _ = make_entityset(\n",
    "    data_test,\n",
    "    nclusters,\n",
    "    kmeans=kmeans,\n",
    ")\n",
    "\n",
    "fm_test = ft.calculate_feature_matrix(\n",
    "    entityset=es_test,\n",
    "    features=features,\n",
    "    verbose=True,\n",
    "    chunk_size=.26,\n",
    ")\n",
    "\n",
    "X = fm_test.copy().fillna(0)\n",
    "\n",
    "y = pd.read_csv(\n",
    "    'data/RUL_FD004.txt',\n",
    "    sep=' ',\n",
    "    header=None,\n",
    "    names=['remaining_useful_life'],\n",
    "    index_col=False,\n",
    ")\n",
    "\n",
    "preds = regs[0].predict(X)\n",
    "mae = mean_absolute_error(preds, y)\n",
    "print('Mean Abs Error: {:.2f}'.format(mae))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Feature Selection and Scoring\n",
    "Here, we'll use [Recursive Feature Elimination](http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html). In order to set ourselves up for later optimization, we're going to write a generic `pipeline` function which takes in a set of hyperparameters and returns a score. Our pipeline will first run `RFE` and then split the remaining data for scoring by a `RandomForestRegressor`. We're going to pass in a list of hyperparameters, which we will tune later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can use that selector and regressor to score the test values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: Hyperparameter Tuning\n",
    "Because of the way we set up our pipeline, we can use a Gaussian Process to tune the hyperparameters. We will use [BTB](https://github.com/HDI-Project/BTB) from the [HDI Project](https://github.com/HDI-Project). This will search through the hyperparameters `n_estimators` and `max_feats` for RandomForest, and the number of features for RFE to find the hyperparameter set that has the best average score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/30 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[n_est, max_feats, nfeats]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  3%|▎         | 1/30 [00:11<05:34, 11.52s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0. {'n_estimators': 21, 'max_feats': 44, 'nfeats': 66} -- Average MAE: 37.3, Std: 4.73\n",
      "Raw: [41.3, 36.3, 43.9, 33.9, 31.0]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  7%|▋         | 2/30 [00:25<05:40, 12.15s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. {'n_estimators': 28, 'max_feats': 10, 'nfeats': 11} -- Average MAE: 34.1, Std: 2.92\n",
      "Raw: [38.2, 34.3, 35.0, 33.8, 29.1]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 30/30 [06:50<00:00, 13.67s/it]\n"
     ]
    }
   ],
   "source": [
    "from btb import HyperParameter, ParamTypes\n",
    "from btb.tuning import GP\n",
    "\n",
    "\n",
    "def run_btb(fm_list, n=30, best=45):\n",
    "    hyperparam_ranges = [\n",
    "        ('n_estimators', HyperParameter(ParamTypes.INT, [10, 200])),\n",
    "        ('max_feats', HyperParameter(ParamTypes.INT, [5, 50])),\n",
    "        ('nfeats', HyperParameter(ParamTypes.INT, [10, 70])),\n",
    "    ]\n",
    "\n",
    "    tuner = GP(hyperparam_ranges)\n",
    "    shape = (n, len(hyperparam_ranges))\n",
    "    tested_parameters = np.zeros(shape, dtype=object)\n",
    "    scores = []\n",
    "\n",
    "    print('[n_est, max_feats, nfeats]')\n",
    "\n",
    "    best_hyperparams = None\n",
    "    best_sel = None\n",
    "    best_reg = None\n",
    "\n",
    "    for i in tqdm(range(n)):\n",
    "        hyperparams = tuner.propose()\n",
    "\n",
    "        cvscores, regs, selectors = pipeline_for_test(\n",
    "            fm_list,\n",
    "            hyperparams=hyperparams,\n",
    "            do_selection=True,\n",
    "        )\n",
    "\n",
    "        bound = np.mean(cvscores)\n",
    "        tested_parameters[i, :] = hyperparams\n",
    "        tuner.add(hyperparams, -np.mean(cvscores))\n",
    "\n",
    "        if np.mean(cvscores) + np.std(cvscores) < best:\n",
    "            best = np.mean(cvscores)\n",
    "            best_hyperparams = hyperparams\n",
    "            best_reg = regs[0]\n",
    "            best_sel = selectors[0]\n",
    "\n",
    "            info = '{}. {} -- Average MAE: {:.1f}, Std: {:.2f}'\n",
    "            mean, std = np.mean(cvscores), np.std(cvscores)\n",
    "            print(info.format(i, best_hyperparams, mean, std))\n",
    "            print('Raw: {}'.format([float('{:.1f}'.format(s)) for s in cvscores]))\n",
    "\n",
    "    return best_hyperparams, (best_sel, best_reg)\n",
    "\n",
    "\n",
    "best_hyperparams, best_pipeline = run_btb(fm_list, n=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Abs Error on Test: 47.75\n",
      "1: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_13)) [0.196]\n",
      "2: MAX(recordings.sensor_measurement_15) [0.155]\n",
      "3: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_4)) [0.139]\n",
      "4: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_11)) [0.130]\n",
      "5: MAX(recordings.sensor_measurement_11) [0.085]\n",
      "-----\n",
      "\n"
     ]
    }
   ],
   "source": [
    "X = fm_test.copy().fillna(0)\n",
    "\n",
    "y = pd.read_csv(\n",
    "    'data/RUL_FD004.txt',\n",
    "    sep=' ',\n",
    "    header=None,\n",
    "    names=['remaining_useful_life'],\n",
    "    index_col=False,\n",
    ")\n",
    "\n",
    "preds = best_pipeline[1].predict(best_pipeline[0].transform(X))\n",
    "score = mean_absolute_error(preds, y)\n",
    "print('Mean Abs Error on Test: {:.2f}'.format(score))\n",
    "\n",
    "most_imp_feats = utils.feature_importances(\n",
    "    X.iloc[:, best_pipeline[0].support_],\n",
    "    best_pipeline[1],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix: Averaging old scores\n",
    "To make a fair comparison between the previous notebook and this one, we should average scores where possible. The work in this section is exactly the work in the previous notebook plus some code for taking the average in the validation step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 304 features\n",
      "Elapsed: 01:54 | Progress: 100%|██████████"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/4 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [07:39<00:00, 114.85s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[37.1, 32.1, 35.3, 33.7, 27.8]\n",
      "Average MAE: 33.20, Std: 3.17\n",
      "\n",
      "[65.5, 54.2, 55.6, 46.5, 52.2]\n",
      "Baseline by Median MAE: 54.82, Std: 6.18\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from featuretools.primitives import Min\n",
    "\n",
    "old_fm, features = ft.dfs(\n",
    "    entityset=es,\n",
    "    target_entity='engines',\n",
    "    agg_primitives=['last', 'max', 'min'],\n",
    "    trans_primitives=[],\n",
    "    cutoff_time=cutoff_time_list[0],\n",
    "    max_depth=3,\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "old_fm_list = [old_fm]\n",
    "\n",
    "for i in tqdm(range(1, splits)):\n",
    "    es = make_entityset(data, nclusters, kmeans=kmeans)[0]\n",
    "    old_fm = ft.calculate_feature_matrix(\n",
    "        entityset=es,\n",
    "        features=features,\n",
    "        cutoff_time=cutoff_time_list[i],\n",
    "    )\n",
    "    old_fm_list.append(fm)\n",
    "\n",
    "old_scores = []\n",
    "median_scores = []\n",
    "\n",
    "for fm in old_fm_list:\n",
    "    X = fm.copy().fillna(0)\n",
    "    y = X.pop('remaining_useful_life')\n",
    "\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "\n",
    "    reg = RandomForestRegressor(n_estimators=10)\n",
    "    reg.fit(X_train, y_train)\n",
    "    preds = reg.predict(X_test)\n",
    "\n",
    "    mae = mean_absolute_error(preds, y_test)\n",
    "    old_scores.append(mae)\n",
    "\n",
    "    medianpredict = [np.median(y_train) for _ in y_test]\n",
    "    mae = mean_absolute_error(medianpredict, y_test)\n",
    "    median_scores.append(mae)\n",
    "\n",
    "print([float('{:.1f}'.format(score)) for score in old_scores])\n",
    "mean, std = np.mean(old_scores), np.std(old_scores)\n",
    "info = 'Average MAE: {:.2f}, Std: {:.2f}\\n'\n",
    "print(info.format(mean, std))\n",
    "\n",
    "print([float('{:.1f}'.format(score)) for score in median_scores])\n",
    "mean, std = np.mean(median_scores), np.std(median_scores)\n",
    "info = 'Baseline by Median MAE: {:.2f}, Std: {:.2f}\\n'\n",
    "print(info.format(mean, std))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[47.4, 48.1, 47.4, 46.7, 46.5]\n",
      "Baseline by Median MAE: 47.18, Std: 0.58\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y = pd.read_csv(\n",
    "    'data/RUL_FD004.txt',\n",
    "    sep=' ',\n",
    "    header=None,\n",
    "    names=['remaining_useful_life'],\n",
    "    index_col=False,\n",
    ")\n",
    "\n",
    "median_scores_2 = []\n",
    "\n",
    "for ct in cutoff_time_list:\n",
    "    medianpredict2 = [np.median(ct['remaining_useful_life'].values) for _ in y.values]\n",
    "    mae = mean_absolute_error(medianpredict2, y)\n",
    "    median_scores_2.append(mae)\n",
    "\n",
    "print([float('{:.1f}'.format(score)) for score in median_scores_2])\n",
    "mean, std = np.mean(median_scores_2), np.std(median_scores_2)\n",
    "info = 'Baseline by Median MAE: {:.2f}, Std: {:.2f}\\n'\n",
    "print(info.format(mean, std))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save output files\n",
    "os.makedirs(\"output\", exist_ok=True)\n",
    "fm.to_csv('output/advanced_train_feature_matrix.csv')\n",
    "cutoff_time_list[0].to_csv('output/advanced_train_label_times.csv')\n",
    "fm_test.to_csv('output/advanced_test_feature_matrix.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "    <img src=\"https://www.featurelabs.com/wp-content/uploads/2017/12/logo.png\" alt=\"Featuretools\" />\n",
    "</p>\n",
    "\n",
    "Featuretools was created by the developers at [Feature Labs](https://www.featurelabs.com/). If building impactful data science pipelines is important to you or your business, please [get in touch](https://www.featurelabs.com/contact)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}