AWSLabs-ML-Deploy-BlueGreen.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploying a Machine Learning Model\n",
    "\n",
    "TEST 1 This lab will train and deploy a model using the Linear Learner algorithm.  You will test that model here within this notebook.\n",
    "\n",
    "You will then train and deploy a model using XGBoost as the algorithm (with the same data set).\n",
    "\n",
    "You following that deployment, you will create a new endpoint configuration which will add the new XGBoost model as a production variant without sending any traffic to it - then, once ready to perform your \"blue green\" deployment - you will update the weights of the two models witin the endpoint configuration to \"swap\" from the linear learner model to the XGBoost model.\n",
    "\n",
    "\n",
    "\n",
    "The first few steps of this lab have to do with a fucntion called \"Feature Engineering\".  While we do some very basic adjustments to our core dataset in this lab, please know that there are several other steps we could and should take to make the model more accurate if this were a real-world development.  Those extra steps have been left out of this lab for the sake of simplicity.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Load your data\n",
    "\n",
    "We are using a set of housing data from King Country Seattle for our training.  This data is in a CSV file and we need to read it into a Pandas dataframe which we will call \"df\".\n",
    "\n",
    "Once the data is loaded, we call a head method to get a look at the first few lines.\n",
    "\n",
    "We will start by loading several libraries and utilities we will use throughout the lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pandas/core/computation/expressions.py:21: UserWarning: Pandas requires version '2.8.0' or newer of 'numexpr' (version '2.7.3' currently installed).\n",
      "  from pandas.core.computation.check import NUMEXPR_INSTALLED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n"
     ]
    }
   ],
   "source": [
    "# Load Libraries and utilities needed\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import sagemaker\n",
    "import boto3\n",
    "import os\n",
    "import re\n",
    "from sklearn.model_selection import train_test_split\n",
    "import time\n",
    "import io \n",
    "import sagemaker.amazon.common as smac\n",
    "import os\n",
    "import math\n",
    "from sagemaker.amazon.amazon_estimator import image_uris\n",
    "from sklearn import metrics \n",
    "from io import StringIO \n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.amazon.amazon_estimator import image_uris\n",
    "from sagemaker.serializers import CSVSerializer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>price</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>sqft_living</th>\n",
       "      <th>sqft_lot</th>\n",
       "      <th>floors</th>\n",
       "      <th>waterfront</th>\n",
       "      <th>view</th>\n",
       "      <th>...</th>\n",
       "      <th>grade</th>\n",
       "      <th>sqft_above</th>\n",
       "      <th>sqft_basement</th>\n",
       "      <th>yr_built</th>\n",
       "      <th>yr_renovated</th>\n",
       "      <th>zipcode</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "      <th>sqft_living15</th>\n",
       "      <th>sqft_lot15</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7129300520</td>\n",
       "      <td>20141013T000000</td>\n",
       "      <td>221900.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1180</td>\n",
       "      <td>5650</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>1180</td>\n",
       "      <td>0</td>\n",
       "      <td>1955</td>\n",
       "      <td>0</td>\n",
       "      <td>98178</td>\n",
       "      <td>47.5112</td>\n",
       "      <td>-122.257</td>\n",
       "      <td>1340</td>\n",
       "      <td>5650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6414100192</td>\n",
       "      <td>20141209T000000</td>\n",
       "      <td>538000.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2.25</td>\n",
       "      <td>2570</td>\n",
       "      <td>7242</td>\n",
       "      <td>2.0</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>2170</td>\n",
       "      <td>400</td>\n",
       "      <td>1951</td>\n",
       "      <td>1991</td>\n",
       "      <td>98125</td>\n",
       "      <td>47.7210</td>\n",
       "      <td>-122.319</td>\n",
       "      <td>1690</td>\n",
       "      <td>7639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5631500400</td>\n",
       "      <td>20150225T000000</td>\n",
       "      <td>180000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>770</td>\n",
       "      <td>10000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>6</td>\n",
       "      <td>770</td>\n",
       "      <td>0</td>\n",
       "      <td>1933</td>\n",
       "      <td>0</td>\n",
       "      <td>98028</td>\n",
       "      <td>47.7379</td>\n",
       "      <td>-122.233</td>\n",
       "      <td>2720</td>\n",
       "      <td>8062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2487200875</td>\n",
       "      <td>20141209T000000</td>\n",
       "      <td>604000.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3.00</td>\n",
       "      <td>1960</td>\n",
       "      <td>5000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>1050</td>\n",
       "      <td>910</td>\n",
       "      <td>1965</td>\n",
       "      <td>0</td>\n",
       "      <td>98136</td>\n",
       "      <td>47.5208</td>\n",
       "      <td>-122.393</td>\n",
       "      <td>1360</td>\n",
       "      <td>5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1954400510</td>\n",
       "      <td>20150218T000000</td>\n",
       "      <td>510000.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2.00</td>\n",
       "      <td>1680</td>\n",
       "      <td>8080</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>8</td>\n",
       "      <td>1680</td>\n",
       "      <td>0</td>\n",
       "      <td>1987</td>\n",
       "      <td>0</td>\n",
       "      <td>98074</td>\n",
       "      <td>47.6168</td>\n",
       "      <td>-122.045</td>\n",
       "      <td>1800</td>\n",
       "      <td>7503</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           id             date     price  bedrooms  bathrooms  sqft_living  \\\n",
       "0  7129300520  20141013T000000  221900.0         3       1.00         1180   \n",
       "1  6414100192  20141209T000000  538000.0         3       2.25         2570   \n",
       "2  5631500400  20150225T000000  180000.0         2       1.00          770   \n",
       "3  2487200875  20141209T000000  604000.0         4       3.00         1960   \n",
       "4  1954400510  20150218T000000  510000.0         3       2.00         1680   \n",
       "\n",
       "   sqft_lot  floors waterfront  view  ... grade  sqft_above  sqft_basement  \\\n",
       "0      5650     1.0          N     0  ...     7        1180              0   \n",
       "1      7242     2.0          N     0  ...     7        2170            400   \n",
       "2     10000     1.0          N     0  ...     6         770              0   \n",
       "3      5000     1.0          N     0  ...     7        1050            910   \n",
       "4      8080     1.0          N     0  ...     8        1680              0   \n",
       "\n",
       "   yr_built  yr_renovated  zipcode      lat     long  sqft_living15  \\\n",
       "0      1955             0    98178  47.5112 -122.257           1340   \n",
       "1      1951          1991    98125  47.7210 -122.319           1690   \n",
       "2      1933             0    98028  47.7379 -122.233           2720   \n",
       "3      1965             0    98136  47.5208 -122.393           1360   \n",
       "4      1987             0    98074  47.6168 -122.045           1800   \n",
       "\n",
       "   sqft_lot15  \n",
       "0        5650  \n",
       "1        7639  \n",
       "2        8062  \n",
       "3        5000  \n",
       "4        7503  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the dataset into initial dataframe df\n",
    "df = pd.read_csv(\"kc_house_data_2.csv\")\n",
    "\n",
    "# Review the first 5 rows of the data\n",
    "df.head()\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see this dataset has 21 columns:\n",
    "* `id` - Unique id number\n",
    "* `date` - Date of the house sale\n",
    "* `price` - Price the house sold for\n",
    "* `bedrooms` - Number of bedrooms\n",
    "* `bathrooms` - Number of bathrooms\n",
    "* `sqft_living` - Number of square feet of the living space\n",
    "* `sqft_lot` - Number of square feet of the lot\n",
    "* `floors` - Number of floors in the house\n",
    "* `waterfront` - Whether the home is on the waterfront\n",
    "* `view` - Number of lot sides with a view\n",
    "* `condition` - Condition of the house\n",
    "* `grade` - Classification by construction quality \n",
    "* `sqft_above` - Number of square feet above ground\n",
    "* `sqft_basement` - Number of square feet below ground\n",
    "* `yr_built` - Year built\n",
    "* `yr_renovated` - Year renovated\n",
    "* `zipcode` - ZIP code\n",
    "* `lat` - Latitude\n",
    "* `long` - Longitude\n",
    "* `sqft_living15` - Number of square feet of living space in 2015 (can differ from `sqft_living` in the case of recent renovations)\n",
    "* `sqft_lot15` - Number of square feet of lot space in 2015 (can differ from `sqft_lot` in the case of recent renovations)\n",
    "\n",
    "This is an excellent dataset to work with and we could do a tremendous amount of feature engineering on it, however, this lab is focused on deploying our model.  But before we can do that, there are some basic feature engineering steps we must take.  Let's start by loading some data from our dataframe called df - to a new one called data.  Doing this will allow us to retain our original dataframe in case we would like to go back to it.\n",
    "\n",
    "We start by adding the square feet of living space to our new dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - create our working dataframe\n",
    "\n",
    "We start by adding the square feet of living space to our new dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sqft_living</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1680</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sqft_living\n",
       "0         1180\n",
       "1         2570\n",
       "2          770\n",
       "3         1960\n",
       "4         1680"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add sqft living to our new dataframe called 'data'\n",
    "data = df[['sqft_living']].copy()\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we add other features which do not need to be converted or \"engineered\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sqft_living</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>sqft_lot</th>\n",
       "      <th>floors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1180</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>5650</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2570</td>\n",
       "      <td>3</td>\n",
       "      <td>2.25</td>\n",
       "      <td>7242</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>770</td>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>10000</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1960</td>\n",
       "      <td>4</td>\n",
       "      <td>3.00</td>\n",
       "      <td>5000</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1680</td>\n",
       "      <td>3</td>\n",
       "      <td>2.00</td>\n",
       "      <td>8080</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sqft_living  bedrooms  bathrooms  sqft_lot  floors\n",
       "0         1180         3       1.00      5650     1.0\n",
       "1         2570         3       2.25      7242     2.0\n",
       "2          770         2       1.00     10000     1.0\n",
       "3         1960         4       3.00      5000     1.0\n",
       "4         1680         3       2.00      8080     1.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['bedrooms'] = df['bedrooms']\n",
    "data['bathrooms'] = df['bathrooms']\n",
    "data['sqft_lot'] = df['sqft_lot']\n",
    "data['floors'] = df['floors']\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Basic Feature Engineering\n",
    "\n",
    "### Categorical variables\n",
    "\n",
    "Let's start by including some categorical features, beginning with simple binary variables.\n",
    "\n",
    "The dataset has the `waterfront` feature, which is a binary variable. We should change the encoding from `'Y'` and `'N'` to `1` and `0`. \n",
    "\n",
    "This can be done using the `map` function ([documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.map.html)) provided by Pandas.  It expects either a function to apply to that column or a dictionary to look up the correct transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# use the map function to add waterfront information as a binary categorical variable.\n",
    "\n",
    "data['waterfront'] = df['waterfront'].map({'Y':1, 'N':0})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=21613, step=1)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df['condition'])\n",
    "df['condition'].index.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also encode many class categorical variables. Look at column `condition`, which gives a score of the quality of the house. Looking into the [data source](https://info.kingcounty.gov/assessor/esales/Glossary.aspx?type=r#b) shows that the condition can be thought of as an ordinal categorical variable, so it makes sense to encode it with the order.\n",
    "\n",
    "> Using the same map method as above, we will encode the ordinal categorical variable `condition` into the numerical range of 1 through 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sqft_living</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>sqft_lot</th>\n",
       "      <th>floors</th>\n",
       "      <th>waterfront</th>\n",
       "      <th>condition</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1180</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>5650</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2570</td>\n",
       "      <td>3</td>\n",
       "      <td>2.25</td>\n",
       "      <td>7242</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>770</td>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>10000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1960</td>\n",
       "      <td>4</td>\n",
       "      <td>3.00</td>\n",
       "      <td>5000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1680</td>\n",
       "      <td>3</td>\n",
       "      <td>2.00</td>\n",
       "      <td>8080</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sqft_living  bedrooms  bathrooms  sqft_lot  floors  waterfront  condition\n",
       "0         1180         3       1.00      5650     1.0           0          3\n",
       "1         2570         3       2.25      7242     2.0           0          3\n",
       "2          770         2       1.00     10000     1.0           0          3\n",
       "3         1960         4       3.00      5000     1.0           0          5\n",
       "4         1680         3       2.00      8080     1.0           0          3"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use the map fuction to add condition as an ordinal categorical variable.\n",
    "\n",
    "data['condition'] = df['condition'].map({'Poor':1, 'Fair':2, 'Average':3, 'Good':4, 'Very Good':5})\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use one hot encoding to convert some of our nominal categorical features to binary as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# one hot encoding\n",
    "data = pd.concat([data, pd.get_dummies(df['zipcode'])], axis=1)\n",
    "\n",
    "# Scaling of larger features\n",
    "sqft_min = data['sqft_living'].min()\n",
    "sqft_max = data['sqft_living'].max()\n",
    "data['sqft_living'] = data['sqft_living'].map(lambda x : (x-sqft_min)/(sqft_max - sqft_min))\n",
    "\n",
    "sqft_min2 = data['sqft_lot'].min()\n",
    "sqft_max2 = data['sqft_lot'].max()\n",
    "data['sqft_lot'] = data['sqft_lot'].map(lambda x : (x-sqft_min2)/(sqft_max2 - sqft_min2))\n",
    "\n",
    "cond_min = data['condition'].min()\n",
    "cond_max = data['condition'].max()\n",
    "data['condition'] = data['condition'].map(lambda x : (x-cond_min)/(cond_max - cond_min))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sqft_living</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>sqft_lot</th>\n",
       "      <th>floors</th>\n",
       "      <th>waterfront</th>\n",
       "      <th>condition</th>\n",
       "      <th>98001</th>\n",
       "      <th>98002</th>\n",
       "      <th>98003</th>\n",
       "      <th>...</th>\n",
       "      <th>98146</th>\n",
       "      <th>98148</th>\n",
       "      <th>98155</th>\n",
       "      <th>98166</th>\n",
       "      <th>98168</th>\n",
       "      <th>98177</th>\n",
       "      <th>98178</th>\n",
       "      <th>98188</th>\n",
       "      <th>98198</th>\n",
       "      <th>98199</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.067170</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.003108</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.172075</td>\n",
       "      <td>3</td>\n",
       "      <td>2.25</td>\n",
       "      <td>0.004072</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.036226</td>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.005743</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.126038</td>\n",
       "      <td>4</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.002714</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.104906</td>\n",
       "      <td>3</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.004579</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 147 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   sqft_living  bedrooms  bathrooms  sqft_lot  floors  waterfront  condition  \\\n",
       "0     0.067170         3       1.00  0.003108     1.0           0        0.5   \n",
       "1     0.172075         3       2.25  0.004072     2.0           0        0.5   \n",
       "2     0.036226         2       1.00  0.005743     1.0           0        0.5   \n",
       "3     0.126038         4       3.00  0.002714     1.0           0        1.0   \n",
       "4     0.104906         3       2.00  0.004579     1.0           0        0.5   \n",
       "\n",
       "   98001  98002  98003  ...  98146  98148  98155  98166  98168  98177  98178  \\\n",
       "0  False  False  False  ...  False  False  False  False  False  False   True   \n",
       "1  False  False  False  ...  False  False  False  False  False  False  False   \n",
       "2  False  False  False  ...  False  False  False  False  False  False  False   \n",
       "3  False  False  False  ...  False  False  False  False  False  False  False   \n",
       "4  False  False  False  ...  False  False  False  False  False  False  False   \n",
       "\n",
       "   98188  98198  98199  \n",
       "0  False  False  False  \n",
       "1  False  False  False  \n",
       "2  False  False  False  \n",
       "3  False  False  False  \n",
       "4  False  False  False  \n",
       "\n",
       "[5 rows x 147 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another method we can use to look at our data is the describe method.  This will give us some basic statistical information about our features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sqft_living</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>sqft_lot</th>\n",
       "      <th>floors</th>\n",
       "      <th>waterfront</th>\n",
       "      <th>condition</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>21613.000000</td>\n",
       "      <td>21613.000000</td>\n",
       "      <td>21613.000000</td>\n",
       "      <td>21613.000000</td>\n",
       "      <td>21613.000000</td>\n",
       "      <td>21613.000000</td>\n",
       "      <td>21613.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.135087</td>\n",
       "      <td>3.370842</td>\n",
       "      <td>2.114757</td>\n",
       "      <td>0.008836</td>\n",
       "      <td>1.494309</td>\n",
       "      <td>0.007542</td>\n",
       "      <td>0.602357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.069316</td>\n",
       "      <td>0.930062</td>\n",
       "      <td>0.770163</td>\n",
       "      <td>0.025091</td>\n",
       "      <td>0.539989</td>\n",
       "      <td>0.086517</td>\n",
       "      <td>0.162686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.085811</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.750000</td>\n",
       "      <td>0.002738</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.122264</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.250000</td>\n",
       "      <td>0.004300</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.170566</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>0.006159</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>33.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        sqft_living      bedrooms     bathrooms      sqft_lot        floors  \\\n",
       "count  21613.000000  21613.000000  21613.000000  21613.000000  21613.000000   \n",
       "mean       0.135087      3.370842      2.114757      0.008836      1.494309   \n",
       "std        0.069316      0.930062      0.770163      0.025091      0.539989   \n",
       "min        0.000000      0.000000      0.000000      0.000000      1.000000   \n",
       "25%        0.085811      3.000000      1.750000      0.002738      1.000000   \n",
       "50%        0.122264      3.000000      2.250000      0.004300      1.500000   \n",
       "75%        0.170566      4.000000      2.500000      0.006159      2.000000   \n",
       "max        1.000000     33.000000      8.000000      1.000000      3.500000   \n",
       "\n",
       "         waterfront     condition  \n",
       "count  21613.000000  21613.000000  \n",
       "mean       0.007542      0.602357  \n",
       "std        0.086517      0.162686  \n",
       "min        0.000000      0.000000  \n",
       "25%        0.000000      0.500000  \n",
       "50%        0.000000      0.500000  \n",
       "75%        0.000000      0.750000  \n",
       "max        1.000000      1.000000  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will setup our training data, split it and convert it to RecordIP format for use in training our Linear Learner model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n"
     ]
    }
   ],
   "source": [
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "# S3 bucket for saving code and model artifacts.\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "prefix = \"linear-learner\" #prefix is a sub-folder/key within the S3 bucket\n",
    "output_location = 's3://{}/{}/output'.format(bucket, prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Split training, validation, and test\n",
    "ys = np.array(df['price']).astype(\"float32\")\n",
    "xs = np.array(data).astype(\"float32\")\n",
    "\n",
    "np.random.seed(8675309)\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(xs, ys, test_size=0.2)\n",
    "val_features, test_features, val_labels, test_labels = train_test_split(test_features, test_labels, test_size=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n",
      "uploaded training data location: s3://sagemaker-us-east-1-128847997663/linear-learner/train/linear-train-data\n",
      "uploaded training data location: s3://sagemaker-us-east-1-128847997663/linear-learner/test/linear-test-data\n",
      "Training artifacts will be uploaded to: s3://sagemaker-us-east-1-128847997663/linear-learner/output\n"
     ]
    }
   ],
   "source": [
    "#Create a SageMaker session\n",
    "sagemaker_session = sagemaker.Session()\n",
    "\n",
    "#Need to convert dataset to RecordIO format for Linear Learner to understand\n",
    "\n",
    "buf = io.BytesIO()\n",
    "smac.write_numpy_to_dense_tensor(buf, train_features, train_labels)\n",
    "buf.seek(0) \n",
    "\n",
    "###Uploading training data\n",
    "#Filename for training data we are uploading to S3 \n",
    "key = 'linear-train-data'\n",
    "#Upload training data to S3\n",
    "boto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', key)).upload_fileobj(buf)\n",
    "s3_train_data = 's3://{}/{}/train/{}'.format(bucket, prefix, key)\n",
    "print('uploaded training data location: {}'.format(s3_train_data))\n",
    "\n",
    "###Uploading test data\n",
    "buf = io.BytesIO() # create an in-memory byte array (buf is a buffer I will be writing to)\n",
    "smac.write_numpy_to_dense_tensor(buf, train_features, train_labels)\n",
    "buf.seek(0)\n",
    "\n",
    "#Sub-folder for test data\n",
    "key = 'linear-test-data'\n",
    "boto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'test', key)).upload_fileobj(buf)\n",
    "s3_test_data = 's3://{}/{}/test/{}'.format(bucket, prefix, key)\n",
    "print('uploaded training data location: {}'.format(s3_test_data))\n",
    "\n",
    "###Model Artifacts\n",
    "output_location = 's3://{}/{}/output'.format(bucket, prefix)\n",
    "print('Training artifacts will be uploaded to: {}'.format(output_location))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While there are many more features we could use and feature engineering we could do, we will stop here and move forward with training our model.\n",
    "\n",
    "## Step 4 - Train Model using the Linear Learner algorithm\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the container that holds the linear learner algorithm\n",
    "container1 = image_uris.retrieve('linear-learner', boto3.Session().region_name)\n",
    "\n",
    "# Create the esitmator\n",
    "linear_model = sagemaker.estimator.Estimator(container1,\n",
    "                                       role, \n",
    "                                       instance_count = 1, \n",
    "                                       instance_type = 'ml.m4.xlarge',\n",
    "                                       output_path = output_location,\n",
    "                                       sagemaker_session = sagemaker_session)\n",
    "\n",
    "# And set the hyperparameters\n",
    "linear_model.set_hyperparameters(feature_dim = 77,\n",
    "                               predictor_type = 'regressor',\n",
    "                               mini_batch_size = 20,\n",
    "                               epochs = 5,\n",
    "                               num_models = 10,\n",
    "                               loss = 'absolute_loss')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: linear-learner-2023-12-01-14-50-57-995\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-12-01 14:50:58 Starting - Starting the training job...\n",
      "2023-12-01 14:51:24 Starting - Preparing the instances for training............\n",
      "2023-12-01 14:53:02 Downloading - Downloading input data...\n",
      "2023-12-01 14:53:37 Training - Downloading the training image......\n",
      "2023-12-01 14:54:43 Training - Training image download completed. Training in progress..\u001b[34mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:04 INFO 140563914995520] Reading default configuration from /opt/amazon/lib/python3.8/site-packages/algorithm/resources/default-input.json: {'mini_batch_size': '1000', 'epochs': '15', 'feature_dim': 'auto', 'use_bias': 'true', 'binary_classifier_model_selection_criteria': 'accuracy', 'f_beta': '1.0', 'target_recall': '0.8', 'target_precision': '0.8', 'num_models': 'auto', 'num_calibration_samples': '10000000', 'init_method': 'uniform', 'init_scale': '0.07', 'init_sigma': '0.01', 'init_bias': '0.0', 'optimizer': 'auto', 'loss': 'auto', 'margin': '1.0', 'quantile': '0.5', 'loss_insensitivity': '0.01', 'huber_delta': '1.0', 'num_classes': '1', 'accuracy_top_k': '3', 'wd': 'auto', 'l1': 'auto', 'momentum': 'auto', 'learning_rate': 'auto', 'beta_1': 'auto', 'beta_2': 'auto', 'bias_lr_mult': 'auto', 'bias_wd_mult': 'auto', 'use_lr_scheduler': 'true', 'lr_scheduler_step': 'auto', 'lr_scheduler_factor': 'auto', 'lr_scheduler_minimum_lr': 'auto', 'positive_example_weight_mult': '1.0', 'balance_multiclass_weights': 'false', 'normalize_data': 'true', 'normalize_label': 'auto', 'unbias_data': 'auto', 'unbias_label': 'auto', 'num_point_for_scaler': '10000', '_kvstore': 'auto', '_num_gpus': 'auto', '_num_kv_servers': 'auto', '_log_level': 'info', '_tuning_objective_metric': '', 'early_stopping_patience': '3', 'early_stopping_tolerance': '0.001', '_enable_profiler': 'false'}\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:04 INFO 140563914995520] Merging with provided configuration from /opt/ml/input/config/hyperparameters.json: {'epochs': '5', 'feature_dim': '77', 'loss': 'absolute_loss', 'mini_batch_size': '20', 'num_models': '10', 'predictor_type': 'regressor'}\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:05 INFO 140563914995520] Final configuration: {'mini_batch_size': '20', 'epochs': '5', 'feature_dim': '77', 'use_bias': 'true', 'binary_classifier_model_selection_criteria': 'accuracy', 'f_beta': '1.0', 'target_recall': '0.8', 'target_precision': '0.8', 'num_models': '10', 'num_calibration_samples': '10000000', 'init_method': 'uniform', 'init_scale': '0.07', 'init_sigma': '0.01', 'init_bias': '0.0', 'optimizer': 'auto', 'loss': 'absolute_loss', 'margin': '1.0', 'quantile': '0.5', 'loss_insensitivity': '0.01', 'huber_delta': '1.0', 'num_classes': '1', 'accuracy_top_k': '3', 'wd': 'auto', 'l1': 'auto', 'momentum': 'auto', 'learning_rate': 'auto', 'beta_1': 'auto', 'beta_2': 'auto', 'bias_lr_mult': 'auto', 'bias_wd_mult': 'auto', 'use_lr_scheduler': 'true', 'lr_scheduler_step': 'auto', 'lr_scheduler_factor': 'auto', 'lr_scheduler_minimum_lr': 'auto', 'positive_example_weight_mult': '1.0', 'balance_multiclass_weights': 'false', 'normalize_data': 'true', 'normalize_label': 'auto', 'unbias_data': 'auto', 'unbias_label': 'auto', 'num_point_for_scaler': '10000', '_kvstore': 'auto', '_num_gpus': 'auto', '_num_kv_servers': 'auto', '_log_level': 'info', '_tuning_objective_metric': '', 'early_stopping_patience': '3', 'early_stopping_tolerance': '0.001', '_enable_profiler': 'false', 'predictor_type': 'regressor'}\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python3.8/site-packages/mxnet/model.py:97: SyntaxWarning: \"is\" with a literal. Did you mean \"==\"?\n",
      "  if num_device is 1 and 'dist' not in kvstore:\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python3.8/site-packages/scipy/optimize/_shgo.py:495: SyntaxWarning: \"is\" with a literal. Did you mean \"==\"?\n",
      "  if cons['type'] is 'ineq':\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python3.8/site-packages/scipy/optimize/_shgo.py:743: SyntaxWarning: \"is not\" with a literal. Did you mean \"!=\"?\n",
      "  if len(self.X_min) is not 0:\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:08 WARNING 140563914995520] Loggers have already been setup.\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:08 INFO 140563914995520] Final configuration: {'mini_batch_size': '20', 'epochs': '5', 'feature_dim': '77', 'use_bias': 'true', 'binary_classifier_model_selection_criteria': 'accuracy', 'f_beta': '1.0', 'target_recall': '0.8', 'target_precision': '0.8', 'num_models': '10', 'num_calibration_samples': '10000000', 'init_method': 'uniform', 'init_scale': '0.07', 'init_sigma': '0.01', 'init_bias': '0.0', 'optimizer': 'auto', 'loss': 'absolute_loss', 'margin': '1.0', 'quantile': '0.5', 'loss_insensitivity': '0.01', 'huber_delta': '1.0', 'num_classes': '1', 'accuracy_top_k': '3', 'wd': 'auto', 'l1': 'auto', 'momentum': 'auto', 'learning_rate': 'auto', 'beta_1': 'auto', 'beta_2': 'auto', 'bias_lr_mult': 'auto', 'bias_wd_mult': 'auto', 'use_lr_scheduler': 'true', 'lr_scheduler_step': 'auto', 'lr_scheduler_factor': 'auto', 'lr_scheduler_minimum_lr': 'auto', 'positive_example_weight_mult': '1.0', 'balance_multiclass_weights': 'false', 'normalize_data': 'true', 'normalize_label': 'auto', 'unbias_data': 'auto', 'unbias_label': 'auto', 'num_point_for_scaler': '10000', '_kvstore': 'auto', '_num_gpus': 'auto', '_num_kv_servers': 'auto', '_log_level': 'info', '_tuning_objective_metric': '', 'early_stopping_patience': '3', 'early_stopping_tolerance': '0.001', '_enable_profiler': 'false', 'predictor_type': 'regressor'}\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:08 WARNING 140563914995520] Loggers have already been setup.\u001b[0m\n",
      "\u001b[34mProcess 7 is a worker.\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:08 INFO 140563914995520] Using default worker.\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:08 INFO 140563914995520] Checkpoint loading and saving are disabled.\u001b[0m\n",
      "\u001b[34m[2023-12-01 14:55:08.443] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 0, \"duration\": 17, \"num_examples\": 1, \"num_bytes\": 12720}\u001b[0m\n",
      "\u001b[34m#metrics {\"StartTime\": 1701442508.4440413, \"EndTime\": 1701442508.4440858, \"Dimensions\": {\"Algorithm\": \"Linear Learner\", \"Host\": \"algo-1\", \"Operation\": \"training\", \"Meta\": \"init_train_data_iter\"}, \"Metrics\": {\"Total Records Seen\": {\"sum\": 20.0, \"count\": 1, \"min\": 20, \"max\": 20}, \"Total Batches Seen\": {\"sum\": 1.0, \"count\": 1, \"min\": 1, \"max\": 1}, \"Max Records Seen Between Resets\": {\"sum\": 20.0, \"count\": 1, \"min\": 20, \"max\": 20}, \"Max Batches Seen Between Resets\": {\"sum\": 1.0, \"count\": 1, \"min\": 1, \"max\": 1}, \"Reset Count\": {\"sum\": 1.0, \"count\": 1, \"min\": 1, \"max\": 1}, \"Number of Records Since Last Reset\": {\"sum\": 0.0, \"count\": 1, \"min\": 0, \"max\": 0}, \"Number of Batches Since Last Reset\": {\"sum\": 0.0, \"count\": 1, \"min\": 0, \"max\": 0}}}\u001b[0m\n",
      "\u001b[34m[12/01/2023 14:55:08 ERROR 140563914995520] Customer Error: Incorrect configuration or train dataset provided. The value of feature_dim hyperparameter '77' doesn't match train data dimensionality '147'. Please set feature_dim to the number of features in train dataset.\u001b[0m\n",
      "\n",
      "2023-12-01 14:55:25 Uploading - Uploading generated training model\n",
      "2023-12-01 14:55:25 Failed - Training job failed\n"
     ]
    }
   ],
   "source": [
    "# Now we can pass in S3 training_data path variable we declared earlier and train our first model\n",
    "linear_model.fit({'train': s3_train_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Deploy Linear Learner the Model\n",
    "\n",
    "We will now deploy our initial model built using Linear Learner.  \n",
    "\n",
    "First we will set our endpoint name and model name variables, then we can deploy directly using the .deploy() method.  This will create a new endpoint configuration and endpoint that are hosted by Sagemaker.\n",
    "\n",
    "We will use the same process when we initially deploy the XGBoost model later in the lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set variable names\n",
    "endpoint_name = 'home-price-regressor-endpoint'\n",
    "linear_model_name = 'linear-regressor-model'\n",
    "\n",
    "#Deploy our initial model\n",
    "\n",
    "home_price_regressor = linear_model.deploy(initial_instance_count = 1,\n",
    "                                           instance_type = 'ml.m4.xlarge',\n",
    "                                           endpoint_name= endpoint_name,\n",
    "                                           model_name= linear_model_name\n",
    "                                          )\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We need to make sure the data is in correct format for deployed model\n",
    "from sagemaker.serializers import CSVSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "home_price_regressor.serializer = CSVSerializer()\n",
    "home_price_regressor.deserializer = JSONDeserializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6 - Interact with the Linear Learner model\n",
    "\n",
    "In a real-world setting, this is where we would be when we first deploy our model.  Since we have a functioning model deployed, we will now push our test data to it and then chart how it does via a scatter plot.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result1 = home_price_regressor.predict(test_features)\n",
    "result1 #should be a JSON\n",
    "\n",
    "#Iterate the result JSON to get an NP array of all the predictions so we can compare to Y test\n",
    "predictions = np.array([res['score'] for res in result1['predictions']])\n",
    "#predictions #should now be an numpy array\n",
    "\n",
    "#Visualize how accurate predictions are relative to y_test\n",
    "plt.scatter(test_labels, predictions)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see - it does ok.  If this were a real-world setting, we would go back and perform further feature engineering.  However, for the purpose of this lab, let's assume that this meets our business needs and thus we begin using the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The scenario:  \n",
    "\n",
    "After we have run our model for a time, we decide that a different algorithm may do a better job.  Our first step in replacing our current model is by training a new one.  This could also happen if we were retraining due to concept drift or other events where we needed an updated model but did not want to interupt our production environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7 - Train a new model\n",
    "\n",
    "Now we will train using the same information but using the XGBoost algorithm.  XGBoost requires the input to be in CSV format vs pulling direct from a dataframe as we did above.  Therefore, we will need to setup the train, test and validation data then export it to an S3 bucket in a CSV format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create out train, test and validation sets\n",
    "XX_train = pd.concat((pd.DataFrame(train_labels),pd.DataFrame(train_features)),axis = 1)\n",
    "XX_valid = pd.concat((pd.DataFrame(val_labels),pd.DataFrame(val_features)),axis = 1)\n",
    "XX_test = pd.concat((pd.DataFrame(test_labels),pd.DataFrame(test_features)),axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Reformat csv file and then create our input sets\n",
    "csv_buffer = StringIO()\n",
    "\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "prefix_out = 'output'\n",
    "output_path = 's3://{}/{}/{}'.format(bucket, prefix_out, 'houseing-xgb')\n",
    "\n",
    "XX_train.to_csv(csv_buffer,header=False, index=False)\n",
    "s3_resource = boto3.resource('s3')\n",
    "s3_resource.Object(bucket, 'train.csv').put(Body=csv_buffer.getvalue())\n",
    "\n",
    "XX_valid.to_csv(csv_buffer,header=False, index=False)\n",
    "s3_resource = boto3.resource('s3')\n",
    "s3_resource.Object(bucket, 'valid.csv').put(Body=csv_buffer.getvalue())\n",
    "\n",
    "XX_test.to_csv(csv_buffer,header=False, index=False)\n",
    "s3_resource = boto3.resource('s3')\n",
    "s3_resource.Object(bucket, 'test.csv').put(Body=csv_buffer.getvalue())\n",
    "\n",
    "s3_input_train = TrainingInput(s3_data='https://{}.s3.amazonaws.com/train.csv'.format(bucket), content_type='csv')\n",
    "s3_input_validation = TrainingInput(s3_data='https://{}.s3.amazonaws.com/valid.csv'.format(bucket),content_type='csv')\n",
    "s3_input_test = TrainingInput(s3_data='https://{}.s3.amazonaws.com/test.csv'.format(bucket),content_type='csv')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Now setup the container, estimator and hyperparameters just as we did with the first model\n",
    "container = sagemaker.image_uris.retrieve('xgboost',boto3.Session().region_name,'latest')\n",
    "\n",
    "data_channels = {'train': s3_input_train, 'validation': s3_input_validation, 'test': s3_input_test}\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(container,\n",
    "                                    role=sagemaker.get_execution_role(), \n",
    "                                    train_instance_count=1, \n",
    "                                    train_instance_type='ml.m4.xlarge',\n",
    "                                    output_path=output_path,\n",
    "                                    sagemaker_session=sess)\n",
    "\n",
    "xgb.set_hyperparameters(predictor_type='regressor',\n",
    "                        max_depth=200,\n",
    "                        num_round=100)\n",
    "\n",
    "# and finally train our new model\n",
    "xgb.fit({'train': s3_input_train, 'validation': s3_input_validation, 'test': s3_input_test})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8 - Deploy the new model\n",
    "\n",
    "We will now deploy our new model with its own endpoint and endpoint configuration.  It will start out as stand alone thus allows us to test it separately if we want.  We will not do this in this lab at this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set our variables, then again, use the .deploy() method to deploy our model\n",
    "xgb_endpoint_name = 'xgb-regressor-endpoint'\n",
    "xgb_model_name = 'xgb-regressor-model'\n",
    "\n",
    "xgb_price_regressor = xgb.deploy(initial_instance_count=1,\n",
    "                           instance_type='ml.m4.xlarge',\n",
    "                           endpoint_name = xgb_endpoint_name,\n",
    "                           model_name = xgb_model_name,\n",
    "                          )\n",
    "\n",
    "xgb_price_regressor.serializer = CSVSerializer()\n",
    "xgb_price_regressor.deserializer = JSONDeserializer()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point - we now have our original production endpoint and endpoing configuration which is hosting our linear learner model.  We also have a separate endpoint and endpoint configuration for our XGBoost model.  However, we don't want to switch over all our production traffic to this separate endpoint - so let's update the first one so we can do a Blue/Green deployment.\n",
    "\n",
    "# Step 9 - Create a new endpoint configuration\n",
    "This new endpoint configuration will include both the original model and the new model as two seperate production varients.  We will also set the initial weight to only send traffic to the orginal - thus ensureing there is no change to our production traffic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get the current endpoint configuration\n",
    "sage_client = sess.sagemaker_client\n",
    "endpoint = sage_client.describe_endpoint(EndpointName=home_price_regressor.endpoint_name)\n",
    "endpoint_config = sage_client.describe_endpoint_config(\n",
    "    EndpointConfigName=endpoint['EndpointConfigName'])\n",
    "\n",
    "# Set the current deployment weight to 1 to ensure all traffic continues to the same model as before\n",
    "current_model_config = endpoint_config['ProductionVariants'][0]\n",
    "current_model_config['InitialVariantWeight'] = 1\n",
    "current_model_config['VariantName'] = 'linear-learner'\n",
    "\n",
    "# Now setup a new variant and configuration for that variant\n",
    "Variant = 'xgboost'\n",
    "\n",
    "xgb_model_config = {'ModelName': xgb_model_name,\n",
    "                      'InitialInstanceCount': 1,\n",
    "                      'InstanceType': 'ml.m4.xlarge',\n",
    "                      'VariantName': Variant,\n",
    "                      'InitialVariantWeight': 0}\n",
    "\n",
    "# And now create the new endpoint configuration using the two production variants from above.\n",
    "sage_client.create_endpoint_config(\n",
    "    EndpointConfigName='AB-Config',\n",
    "    ProductionVariants=[current_model_config,\n",
    "                        xgb_model_config])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Now we update the original endpoint with the new endpoint configuration - this may take a few minutes\n",
    "sage_client.update_endpoint(\n",
    "    EndpointName=endpoint['EndpointConfigName'],\n",
    "    EndpointConfigName='AB-Config')\n",
    "\n",
    "result = sess.wait_for_endpoint(endpoint['EndpointConfigName'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point - everything is processing as before - with the only difference being that we have two production varients in our endpoint configuration.\n",
    "\n",
    "When it comes time to do the cutover - we perform:\n",
    "\n",
    "# Step 10 - The Blue/Green Deployment\n",
    "Simply by changing the weights within the endpoint configuration, we switch all traffic from the linear learner model to the XGBoost model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sage_client.update_endpoint_weights_and_capacities(\n",
    "    EndpointName=endpoint['EndpointConfigName'],\n",
    "    DesiredWeightsAndCapacities=[\n",
    "        {\n",
    "            'VariantName': 'linear-learner',\n",
    "            'DesiredWeight': 0\n",
    "        },\n",
    "        {\n",
    "            'VariantName': 'xgboost',\n",
    "            'DesiredWeight': 1\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "response = sess.wait_for_endpoint(endpoint['EndpointConfigName'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "You have now trained and deployed two separate models, and then performed a Blue/Green deployment to switch from one to the other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean-up\n",
    "\n",
    "If you're done with this notebook, please run the cell below to remove the hosted endpoint and avoid any charges from any stray instances being left on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker.Session().delete_endpoint(xgb_price_regressor.endpoint_name)\n",
    "sagemaker.Session().delete_endpoint(home_price_regressor.endpoint_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}