Regression_on_Boston_House_Prices_5.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyMHAoGpBB80L4aWTGoBbbBx",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Ash100/Biopython/blob/main/Regression_on_Boston_House_Prices_5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Regression of Boston House Prices\n",
        "I am **Dr. Ashfaq Ahmad**, and this notebook is created for teaching and research purposes. Refering to the people working in the field of Biology, I have tried my level best to keep it as simple as possible. For Detailed instruction and understandings, please watch a video tutorial on **https://www.youtube.com/@Bioinformaticsinsights**\n",
        "\n",
        "This notebook is based on the book **Deep Learning with Python** by Jason Brownlee."
      ],
      "metadata": {
        "id": "SMwmOssJyyrt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this project tutorial you will discover how to develop and evaluate neural network models using Keras for a regression problem.\n",
        "Regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the 'outcome' or 'response' variable, or a 'label' in machine learning parlance) and one or more independent variables (often called 'predictors', 'covariates', 'explanatory variables' or 'features'). The most common form of regression analysis is linear regression, in which one finds the line (or a more complex linear combination) that most closely fits the data according to a specific mathematical criterion."
      ],
      "metadata": {
        "id": "YLYk1ldLzKnc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##About the Dataset\n",
        "The problem that we will look at in this tutorial is the Boston house price dataset. The dataset describes properties of houses in Boston suburbs and is concerned with modeling the price of houses in those suburbs in thousands of dollars. As such, this is a regression predictive modeling\n",
        "problem. The dataset can be downloaded from Kaggle repository.\n",
        "\n",
        "There are 13 input variables that describe the properties of a given Boston suburb.\n",
        "The full list of attributes in this dataset are as follows:\n",
        "1. CRIM: per capita crime rate by town.\n",
        "2. ZN: proportion of residential land zoned for lots over 25,000 sq.ft.\n",
        "3. INDUS: proportion of non-retail business acres per town.\n",
        "4. CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).\n",
        "5. NOX: nitric oxides concentration (parts per 10 million).\n",
        "6. RM: average number of rooms per dwelling.\n",
        "7. AGE: proportion of owner-occupied units built prior to 1940.\n",
        "8. DIS: weighted distances to five Boston employment centers.\n",
        "9. RAD: index of accessibility to radial highways.\n",
        "10. TAX: full-value property-tax rate per 10,000.\n",
        "11. PTRATIO: pupil-teacher ratio by town.\n",
        "12. B: 1000(Bk  0.63)2 where Bk is the proportion of blacks by town.\n",
        "13. LSTAT: % lower status of the population.\n",
        "14. MEDV: Median value of owner-occupied homes in 1000s"
      ],
      "metadata": {
        "id": "BT3VGGV5z-ic"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Develop a Baseline Neural Network Model"
      ],
      "metadata": {
        "id": "oynvzaRs0wUA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Install Keras and Scikit\n",
        "!pip install --upgrade keras\n",
        "!pip install --upgrade scikit_learn"
      ],
      "metadata": {
        "id": "LmaeILDy1Cfs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Install as per your need\n",
        "!pip install scikeras[tensorflow-cpu]"
      ],
      "metadata": {
        "id": "Qe2HZvpG1NbB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Install as per your need\n",
        "!pip install scikeras[tensorflow]      # gpu compute platform"
      ],
      "metadata": {
        "id": "3XJmqYNR1Q1r"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "IiHfPVlRyxwW"
      },
      "outputs": [],
      "source": [
        "import numpy\n",
        "import pandas\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.wrappers.scikit_learn import KerasRegressor\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import KFold\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.pipeline import Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Load your Dataset\n",
        "We can now load our dataset from a file in the local directory. The dataset is in fact not in\n",
        "CSV format, the attributes are instead separated by\n",
        "whitespace. We can load this easily using the Pandas library. We can then split the input (X)\n",
        "and output (Y ) attributes so that they are easier to model with Keras and scikit-learn."
      ],
      "metadata": {
        "id": "ZaHuxEOt2IRq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load dataset\n",
        "dataframe = pandas.read_csv(\"/content/sample_data/housing.csv\", delim_whitespace=True, header=None)\n",
        "dataset = dataframe.values\n",
        "# split into input (X) and output (Y) variables\n",
        "X = dataset[:,0:13]\n",
        "Y = dataset[:,13]"
      ],
      "metadata": {
        "id": "wYChZMeN2VC8"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.optimizers import Adam\n",
        "from keras.losses import mean_squared_error\n",
        "from keras.activations import relu\n",
        "\n",
        "# Define base model\n",
        "def baseline_model():\n",
        "    # Create model\n",
        "    model = Sequential()\n",
        "    model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu'))\n",
        "    model.add(Dense(1, kernel_initializer='normal'))\n",
        "\n",
        "    # Compile model\n",
        "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
        "    return model"
      ],
      "metadata": {
        "id": "_QAbyYy-3H31"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The Keras wrapper object for use in scikit-learn as a regression estimator is called KerasRegressor.\n",
        "We will create an instance and pass it both the name of the function to create the neural network model as well as some parameters to pass along to the *fit()* function of the model later, suchas the number of epochs and batch size."
      ],
      "metadata": {
        "id": "TqPAxWWR3sip"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# fix random seed for reproducibility\n",
        "seed = 7\n",
        "numpy.random.seed(seed)\n",
        "# evaluate model with standardized dataset\n",
        "estimator = KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=0)\n"
      ],
      "metadata": {
        "id": "lBnBL4dy33OF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Model Evaluation\n",
        "from sklearn.model_selection import KFold, cross_val_score\n",
        "from sklearn.linear_model import LinearRegression\n",
        "\n",
        "# Define your estimator (e.g., Linear Regression)\n",
        "estimator = LinearRegression()\n",
        "\n",
        "# Define the number of folds for cross-validation\n",
        "kfold = KFold(n_splits=10, shuffle=True, random_state=seed)\n",
        "\n",
        "# Perform cross-validation\n",
        "results = cross_val_score(estimator, X, Y, cv=kfold)\n",
        "\n",
        "# Print the mean and standard deviation of the results\n",
        "print(\"Baseline: %.2f (%.2f) MSE\" % (results.mean(), results.std()))\n"
      ],
      "metadata": {
        "id": "eVs_W1g84I5d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Standardizing the dataset\n",
        "An important concern with the Boston house price dataset is that the input attributes all vary in their scales because they measure different quantities. It is almost always good practice to prepare your data before modeling it using a neural network model. Continuing on from the\n",
        "above baseline model, we can re-evaluate the same model using a standardized version of the input dataset."
      ],
      "metadata": {
        "id": "iJ_OYJz-5OP2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy\n",
        "import pandas\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.optimizers import Adam\n",
        "from keras.losses import mean_squared_error\n",
        "from keras.initializers import normal\n",
        "from keras.activations import relu\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import KFold\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# Load dataset\n",
        "dataframe = pandas.read_csv(\"/content/sample_data/housing.csv\", delim_whitespace=True, header=None)\n",
        "dataset = dataframe.values\n",
        "# split into input (X) and output (Y) variables\n",
        "X = dataset[:,0:13]\n",
        "Y = dataset[:,13]\n",
        "\n",
        "# Define base model\n",
        "def baseline_model():\n",
        "    # Create model\n",
        "    model = Sequential()\n",
        "    model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu'))\n",
        "    model.add(Dense(1, kernel_initializer='normal'))\n",
        "    # Compile model\n",
        "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
        "    return model\n",
        "\n",
        "# Fix random seed for reproducibility\n",
        "seed = 7\n",
        "numpy.random.seed(seed)\n",
        "\n",
        "# Evaluate model with standardized dataset\n",
        "estimators = []\n",
        "estimators.append(('standardize', StandardScaler()))\n",
        "estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50,\n",
        "                                          batch_size=5, verbose=0)))\n",
        "pipeline = Pipeline(estimators)\n",
        "kfold = KFold(n_splits=10, random_state=seed, shuffle=True)\n",
        "results = cross_val_score(pipeline, X, Y, cv=kfold)\n",
        "print(\"Standardized: %.2f (%.2f) MSE\" % (results.mean(), results.std()))"
      ],
      "metadata": {
        "id": "kYDvZ2aV5dxF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Tuning of Network Topology by adding extra hidden layers"
      ],
      "metadata": {
        "id": "rgzTl91G6Czp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy\n",
        "import pandas\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.optimizers import Adam\n",
        "from keras.losses import mean_squared_error\n",
        "from keras.initializers import normal\n",
        "from keras.activations import relu\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import KFold\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# Load dataset\n",
        "dataframe = pandas.read_csv(\"/content/sample_data/housing.csv\", delim_whitespace=True, header=None)\n",
        "dataset = dataframe.values\n",
        "# split into input (X) and output (Y) variables\n",
        "X = dataset[:,0:13]\n",
        "Y = dataset[:,13]\n",
        "\n",
        "# Define the model\n",
        "def larger_model():\n",
        "    # Create model\n",
        "    model = Sequential()\n",
        "    model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu'))\n",
        "    model.add(Dense(6, kernel_initializer='normal', activation='relu'))\n",
        "    model.add(Dense(1, kernel_initializer='normal'))\n",
        "\n",
        "    # Compile model\n",
        "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
        "    return model\n",
        "\n",
        "# Fix random seed for reproducibility\n",
        "seed = 7\n",
        "numpy.random.seed(seed)\n",
        "\n",
        "# Evaluate model with standardized dataset\n",
        "estimators = []\n",
        "estimators.append(('standardize', StandardScaler()))\n",
        "estimators.append(('mlp', KerasRegressor(build_fn=larger_model, epochs=50, batch_size=5,\n",
        "                                          verbose=0)))\n",
        "pipeline = Pipeline(estimators)\n",
        "kfold = KFold(n_splits=10, shuffle=True)  # Enable shuffling\n",
        "results = cross_val_score(pipeline, X, Y, cv=kfold)\n",
        "print(\"Larger: %.2f (%.2f) MSE\" % (results.mean(), results.std()))"
      ],
      "metadata": {
        "id": "rhg50Gia6zxF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Let's make the network more wider\n",
        "Here we will introduce a hidden layer with 20 neurons."
      ],
      "metadata": {
        "id": "kFnLemUvOtFr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy\n",
        "import pandas\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.wrappers.scikit_learn import KerasRegressor\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import KFold\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# load dataset\n",
        "dataframe = pandas.read_csv(\"/content/sample_data/housing.csv\", delim_whitespace=True, header=None)\n",
        "dataset = dataframe.values\n",
        "# split into input (X) and output (Y) variables\n",
        "X = dataset[:,0:13]\n",
        "Y = dataset[:,13]\n",
        "\n",
        "# define wider model\n",
        "def wider_model():\n",
        "    # create model\n",
        "    model = Sequential()\n",
        "    model.add(Dense(20, input_dim=13, kernel_initializer='normal', activation='relu'))\n",
        "    model.add(Dense(1, kernel_initializer='normal'))\n",
        "    # Compile model\n",
        "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
        "    return model\n",
        "\n",
        "# fix random seed for reproducibility\n",
        "seed = 7\n",
        "numpy.random.seed(seed)\n",
        "\n",
        "# evaluate model with standardized dataset\n",
        "estimators = []\n",
        "estimators.append(('standardize', StandardScaler()))\n",
        "estimators.append(('mlp', KerasRegressor(build_fn=wider_model, epochs=100, batch_size=5,\n",
        "                                         verbose=0)))\n",
        "pipeline = Pipeline(estimators)\n",
        "kfold = KFold(n_splits=10, shuffle=True)\n",
        "results = cross_val_score(pipeline, X, Y, cv=kfold)\n",
        "print(\"Wider: %.2f (%.2f) MSE\" % (results.mean(), results.std()))"
      ],
      "metadata": {
        "id": "X9mOJL1AO6Tj"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}