notebooks/00-get_started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Signaux Faibles - Data Science Démo\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "Never commit this notebook along with its output.\n",
    "\n",
    "Please make sure that you have implemented filter \"strip-notebook-output\" in .git/config:\n",
    "```python\n",
    "[filter \"strip-notebook-output\"]\n",
    "        clean = \"jupyter nbconvert --to=notebook --ClearOutputPreprocessor.enabled=True --stdout %f\"\n",
    "        smudge = cat\n",
    "        required\n",
    "```\n",
    "    \n",
    "Only if this first option does not work and you must stage notebooks, please run `jupyter nbconvert --clear-output --inplace 00-get_started.ipynb`\n",
    "or use Kernel > Restart Kernel and Clear all Outputs...\n",
    "</div>\n",
    "\n",
    "The purpose of this repo is to get your started using the `predictsignauxfaibles` repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "You should have created a `.env` file at the root of your local copy of the repo. The required entries are documented in `.env.example`. _Never_ commit your `.env` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure that the latest version of the `predictsignauxfaibles` package is installed.\n",
    "\n",
    "```sh\n",
    "cd predictsignauxfaibles\n",
    "git pull origin #<the branch you are interested in trying>\n",
    "pip install -e .\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set logging level to INFO\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)\n",
    "\n",
    "# Import required libraries and modules\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure that you have access to MongoDB. If you are ensure how to do this, just ask."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data\n",
    "\n",
    "The easiest way to load a dataset is via the `SFDataset` class which does all of the MongoDB-related heavy-lifting for you.\n",
    "\n",
    "There is also a `OversampledSFDataset` class available that lets your ask for a given proportion of positive observations in the resulting dataset.\n",
    "\n",
    "The package (should be) well-documented, use `help(SFDataset)` for help on how to use these objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from predictsignauxfaibles.data import SFDataset, OversampledSFDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MY_FEATURES = [\n",
    "    \"montant_part_ouvriere_past_1\",\n",
    "    \"montant_part_patronale_past_1\",\n",
    "    \"ratio_dette\",\n",
    "]\n",
    "\n",
    "# It's always a good idea to query periods, siret, and outcomes too\n",
    "FIELDS_TO_QUERY =  [\"siret\", \"siren\", \"periode\", \"outcome\", \"time_til_outcome\"] + MY_FEATURES\n",
    "\n",
    "dataset = SFDataset(\n",
    "    date_min=\"2015-01-01\",\n",
    "    date_max=\"2020-06-30\",\n",
    "    fields=FIELDS_TO_QUERY,\n",
    "    sample_size=100\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have successfully created an (empty) dataset. Use the `fetch_data` method to fill it. The data is stored as a Pandas DataFrame in the `.data` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.fetch_data()\n",
    "\n",
    "# show first 5 rows of dataset\n",
    "dataset.data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some commonly-used preprocessing tasks are implemented as SFDataset methods :\n",
    "- fill missing values with their defaults values defined in `config.py`\n",
    "- drop any remaining observation with NAs\n",
    "- remove \"strong signals\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.replace_missing_data().remove_na(ignore = [\"time_til_outcome\"]).remove_strong_signals()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also manipulate `dataset.data` yourself if you want to perform your own transformation of the data.\n",
    "\n",
    "Look into the `predictsignauxfaibles.preprocessors` module for common preprocessing functions.\n",
    "\n",
    "We also use transformation pipelines for model-specific preprocessing tasks. (see `predictsignauxfaibles.pipelines`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run a model\n",
    "\n",
    "Models are configured in a python conf file stored in `models/<model_name>/model_conf.py`. Some conf values can be changed at run time via the CLI (`python -m predictsignauxfaibles --help` for more info)\n",
    "\n",
    "Every model run produces 2 files in `model_runs/<model_id>/` (**which is never commited to Git**):\n",
    "- the model's predictions in csv format\n",
    "- some model's statistics and information in json format\n",
    "\n",
    "The environment variable `ENV` allows you to run the model in `develop` mode (by default, on a few thousands of observations) or in `prod` (using much more data but taking longer to run).\n",
    "\n",
    "```sh\n",
    "export ENV=prod # or develop to make this faster\n",
    "python -m predictsignauxfaibles\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}