robust_trip_planner.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Uncertainty aware journey planner\n",
    "\n",
    "Classical Journey planners usually come with the assumption that the means of transportation such as trains, buses or trams do not have delays. However, because it is impossible to avoid the mentioned events, we wanted to improve the actual existing solutions, by designing a product which takes into consideration the delays, thus being able to predict also the probability of completing each suggested itinerary, using historical data from SBB.\n",
    "\n",
    "This project represents a prototype for a robust uncertainty aware journey planner, and the structure of the notebook follows our decomposition into smaller tasks:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Structure:\n",
    "\n",
    "   * **I. Metadata processing**: In this part, we simply get acquinted with the data, and also process extra data that might help us in the future steps.\n",
    "   * **II. Computing the quality of a transfer**: In this section, we modelled the distributions of delays for given transfers.\n",
    "   * **III. Deterministic journey planner**: We decided to use Open Trip Planner (OTP), an open-source solution for non-uncertainty aware journey planners. \n",
    "   * **IV. Quality aware journey planner**: Most important part of the project, represents the fusion between the last two sections. We integrate the qualities of whole itineraries into the OTP solution, to construct the final product.\n",
    "   * **V. Visualizing confidence of trips**: Constructing Isochronous Maps for Zurich area, which also proves our results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imports needed throughout the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pylab as plt\n",
    "plt.rcParams['figure.figsize'] = (10,6)\n",
    "plt.rcParams['font.size'] = 18\n",
    "plt.style.use('fivethirtyeight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://10.90.38.21:4046\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v2.2.0.2.6.4.0-91</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>yarn</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>final_proj-musuroi</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        "
      ],
      "text/plain": [
       "<SparkContext master=yarn appName=final_proj-musuroi>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import getpass\n",
    "import pyspark\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "conf = pyspark.conf.SparkConf()\n",
    "conf.setMaster('yarn')\n",
    "conf.setAppName('final_proj-{0}'.format(getpass.getuser()))\n",
    "conf.set('spark.executor.memory', '4g')\n",
    "conf.set('spark.executor.instances', '6')\n",
    "conf.set('spark.executor.cores', 2)\n",
    "conf.set('spark.port.maxRetries', '100')\n",
    "sc = pyspark.SparkContext.getOrCreate(conf)\n",
    "conf = sc.getConf()\n",
    "sc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "import requests\n",
    "import time\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "import pyspark.sql.functions as fct\n",
    "\n",
    "from datetime import datetime\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual, widgets\n",
    "\n",
    "from geopy.distance import distance as geo_dist\n",
    "\n",
    "from pyspark.sql.functions import unix_timestamp, to_timestamp, hour, to_date, date_format, month\n",
    "from pyspark.sql.types import FloatType, StringType, IntegerType, DoubleType, StructType\n",
    "from pyspark.sql.types import *\n",
    "from pyspark.ml.feature import StringIndexer, VectorAssembler, VectorIndexer\n",
    "from pyspark.ml.regression import DecisionTreeRegressor\n",
    "from pyspark.sql.functions import collect_list, struct, count, lit, when, col\n",
    "from pyspark.sql.functions import udf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession(sc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.csv('/datasets/project/istdaten/*/*/*', sep=';', header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we rename the columns to English."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = 'TripDate string, TripId string, OperatorId string, OperatorAbbrv string, OperatorName string, ProductId string, LineId string, LineType string, UmlaufId string, TransportType string, AdditionalTrip boolean, FailedTrip boolean, BPUIC string, StopName string, ArrivalTimeScheduled string, ArrivalTimeActual string, ArrivalTimeActualStatus string,     DepartureTimeScheduled string, DepartureTimeActual string, DepartureTimeActualStatus string, SkipStation boolean'\n",
    "columns = list(map(lambda x: x.split()[0],columns.split(',')))\n",
    "\n",
    "for old, new in zip(df.columns, columns):\n",
    "    df = df.withColumnRenamed(old, new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## I. Metadata processing\n",
    "\n",
    "We need to process the metadata provided in order to extract station names with their coordinates. These are needed later when we query the journey planner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I. 1. Creating the metadata dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observed that the actual metadata given is not matching the live data, in the sense that the stop names are not matching in the two datasets. That is why we will use another dataset for accessing the longitude and latitude for different stop names, which overlap more with the live data. This is taken from SBB website, and will be also used for computing the routes in the OTP solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_stops_df(txt_path):\n",
    "    df_stop = pd.read_csv(txt_path, names=['1', 'StopName_Meta', 'Lat', 'Long', '2', '3'],\n",
    "                         usecols=['StopName_Meta', 'Lat', 'Long'], skiprows=[0])\n",
    "    \n",
    "    #Treating special case\n",
    "    df_stop.loc[df_stop.StopName_Meta == \"Isola Superiore\", \"Lat\"] = 45.901230\n",
    "    df_stop.loc[df_stop.StopName_Meta == \"Isola Superiore\", \"Long\"] = 8.520450\n",
    "\n",
    "    return df_stop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StopName_Meta</th>\n",
       "      <th>Lat</th>\n",
       "      <th>Long</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Anzola, chiesa</td>\n",
       "      <td>45.989901</td>\n",
       "      <td>8.345062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Altoggio</td>\n",
       "      <td>46.167251</td>\n",
       "      <td>8.345807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Antronapiana</td>\n",
       "      <td>46.060122</td>\n",
       "      <td>8.113620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Anzola</td>\n",
       "      <td>45.989870</td>\n",
       "      <td>8.345717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Baceno</td>\n",
       "      <td>46.261498</td>\n",
       "      <td>8.319253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Beura Cardezza, centro</td>\n",
       "      <td>46.079062</td>\n",
       "      <td>8.299274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Bognanco, T. Villa Elda</td>\n",
       "      <td>46.122296</td>\n",
       "      <td>8.210772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Boschetto</td>\n",
       "      <td>46.065650</td>\n",
       "      <td>8.261132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Cadarese</td>\n",
       "      <td>46.297881</td>\n",
       "      <td>8.362633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Caddo</td>\n",
       "      <td>46.134019</td>\n",
       "      <td>8.286195</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             StopName_Meta        Lat      Long\n",
       "0           Anzola, chiesa  45.989901  8.345062\n",
       "1                 Altoggio  46.167251  8.345807\n",
       "2             Antronapiana  46.060122  8.113620\n",
       "3                   Anzola  45.989870  8.345717\n",
       "4                   Baceno  46.261498  8.319253\n",
       "5   Beura Cardezza, centro  46.079062  8.299274\n",
       "6  Bognanco, T. Villa Elda  46.122296  8.210772\n",
       "7                Boschetto  46.065650  8.261132\n",
       "8                 Cadarese  46.297881  8.362633\n",
       "9                    Caddo  46.134019  8.286195"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Df_meta = create_stops_df('stops.txt')\n",
    "Df_meta = Df_meta.drop_duplicates()\n",
    "Df_meta.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here Isola Superiore is an error in the dataset which has coordinates for latitude and longitude equal to 0, thus we adapted it with coordinates from Google Maps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I. 2. Creating a dataframe to later map live data to timetables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, given the different line ids between the timetable and line data, we need a dataframe containing the ProductId, LineType and OperatorName for busses and trams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Create Dataframe to find LineId from ProductId, LineType and OperatorName. Relevant for Bus and Tram\n",
    "df_BT = df.where(col('ProductId') != 'Zug').select('ProductId','LineType','OperatorName','LineId').distinct().toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From testing we noticed that some LineId cannot be retrieved with this method. So we cannot find Busses or Trams in the live data though they are in the schedule / timetable. We then noticed some providers actually do not share their live data, such as the 'Transport Lausannois'. Here are the companies that provide data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Auto AG Rothenburg', 'Auto AG Schwyz', 'Auto AG Uri',\n",
       "       'Autobusbetrieb RBS', 'Automobil Rottal AG', 'Automobildienst SZU',\n",
       "       'Automobildienste Aare Seeland mobil', 'BLS AG (brs)',\n",
       "       'Basler Verkehrsbetriebe', 'Busbetrieb Grenchen und Umgebung',\n",
       "       'Busbetrieb Solothurn und Umgebung', 'Busland AG',\n",
       "       'PostAuto Schweiz', 'Regionale Verkehrsbetriebe Schaffhausen',\n",
       "       'Rhätische Bahn', 'Stadtbus Chur', 'Stadtbus Winterthur',\n",
       "       'Städtische Verkehrsbetriebe Bern', 'Verkehrsbetriebe Glattal',\n",
       "       'Verkehrsbetriebe Luzern', 'Verkehrsbetriebe STI AG',\n",
       "       'Verkehrsbetriebe Schaffhausen', 'Verkehrsbetriebe Zürich',\n",
       "       'Verkehrsbetriebe Zürich INFO+',\n",
       "       'Verkehrsbetriebe Zürichsee und Oberland', 'Vierwaldstättersee',\n",
       "       'Zugerland Verkehrsbetriebe'], dtype=object)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(df_BT['OperatorName'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cases where the data is missing (such as for TL), the default distribution (which is a low-bound) will be applied."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II. Computing the quality of a transfer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumptions: \n",
    "\n",
    "   * everytime when making a transfer in the same station, the traveler needs one minute for actually changing transport.\n",
    "   * even though a train departs late all the time in a specific station, the trip planner will never use the fact that it does so, so we will consider all the late departures as on-time departures. \n",
    "   * even though a train arrives early all the time in a specific station, the trip planner will never use the fact that it does so, so we will consider all early arrivals as on-time arrivals. \n",
    "   * when we will query our set of distributions with a parameter which was not encountered so far, we will return a default distribution, which is basically the distribution of all delays in the live data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main idea: \n",
    "\n",
    "The idea behind computing the quality of a specific transfer given the *expected arrival hour* in the station and the *expected departure hour* from that same station, and some *extra information* regarding the trip before the transfer and the one after the transfer:\n",
    "\n",
    "   * First, we compute the **discrete distribution of arrival delays $\\mathcal{D}_a$** in that station, given the information of the trip before the transfer.\n",
    "   * Then, we compute the **discrete distribution of negative departure delays $\\mathcal{D}_d$** in that station, given the information of the trip after the transfer.\n",
    "   * Next, we compute the probability of successfully realizing the transfer, by computing a convolution between the two given distributions. Therefore, assuming that the time of transfer is $k$ minutes, then we would simply compute:\n",
    "      \n",
    "      $\\sum\\limits_{t_a }\\Pr[\\mathcal{D}_a = t_a] \\cdot \\Pr[\\mathcal{D}_d = k-1+t_a]$,\n",
    "      \n",
    "       where we have taken into consideration the minute needed by the traveler for changing the transport. \n",
    "       \n",
    "       \n",
    "We mention that we considered that 1 minute is needed for a transfer in the same station. If the traveller needs to walk between stations, then the value can be adapted in the computation of probabilities.\n",
    "\n",
    "---\n",
    "       \n",
    "Therefore, we first need to decide what are the features which will decide the distributions of the delays. For that, we will use a **Decision Tree Regressor**, selecting several features which might be important from the data, and the target label will be the delay for each datapoint, expressed in seconds. Then, we will train the regressor on both departures and arrivals data, and will look into which are the most important features in each case, for making a good prediction of the delay time. \n",
    "\n",
    "We have to emphasize that we considered this method, because of the way that Decision Trees decide which are the most important feature, i.e. the one which have the most variance of delays between the different values for the specific feature. \n",
    "\n",
    "After constructing the Decision Tree and deciding which are the most important features, we will construct the distributions of the delays from the **actual data**, by grouping the datapoints with the same value for the decisive features, and making the distribution of delays for each group.\n",
    "\n",
    "We decided to use the actual data instead of modelling the distribution of delays using a fixed distribution family (e.g. Log-normal or Gamma distributions), because we consider that the actual data is more relevant, then considering just an estimator or to assume that it follows a distribution in a family of distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### II. 1. Constructing the Decision Tree Regressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step in constructing the Decision Tree Regressor is to construct some potential important features from the given data, and also to compute the delays for each datapoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATE_FORMAT_SCHEDULED = 'dd.MM.yyyy HH:mm' \n",
    "DATE_FORMAT_ACTUAL = 'dd.MM.yyyy HH:mm:ss' # both formats are used\n",
    "\n",
    "df_processed = df.withColumn('ArrivalTimeScheduledDate', to_timestamp(df.ArrivalTimeScheduled, DATE_FORMAT_SCHEDULED))\n",
    "df_processed = df_processed.withColumn('DepartureTimeScheduledDate', to_timestamp(df_processed.DepartureTimeScheduled, DATE_FORMAT_SCHEDULED))\n",
    "\n",
    "df_processed = df_processed.withColumn('ArrivalTimeScheduled', unix_timestamp(df_processed.ArrivalTimeScheduled, DATE_FORMAT_SCHEDULED))\n",
    "df_processed = df_processed.withColumn('ArrivalTimeActual', unix_timestamp(df_processed.ArrivalTimeActual, DATE_FORMAT_ACTUAL))\n",
    "df_processed = df_processed.withColumn('DepartureTimeScheduled', unix_timestamp(df_processed.DepartureTimeScheduled, DATE_FORMAT_SCHEDULED))\n",
    "df_processed = df_processed.withColumn('DepartureTimeActual', unix_timestamp(df_processed.DepartureTimeActual, DATE_FORMAT_ACTUAL))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look into how the data looks so far:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(TripDate='13.09.2017', TripId='80:06____:17010:000', OperatorId='80:06____', OperatorAbbrv='DB', OperatorName='DB Regio AG', ProductId='Zug', LineId='17010', LineType='RE', UmlaufId=None, TransportType='RE', AdditionalTrip='false', FailedTrip='false', BPUIC='8500090', StopName='Basel Bad Bf', ArrivalTimeScheduled=None, ArrivalTimeActual=None, ArrivalTimeActualStatus='PROGNOSE', DepartureTimeScheduled=1505274300, DepartureTimeActual=1505274300, DepartureTimeActualStatus='PROGNOSE', SkipStation='false', ArrivalTimeScheduledDate=None, DepartureTimeScheduledDate=datetime.datetime(2017, 9, 13, 5, 45))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_processed.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we also add the hour of departure and of the arrival to the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[line_id: string, product_id: string, stop_name: string, additional_trip: string, arrival_hour: string, departure_hour: string, day_of_week: string, delta_arrival: float, delta_departure: float]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_to_classify = df_processed.select(\n",
    "    df_processed.LineId.alias('line_id'), \n",
    "    df_processed.ProductId.alias('product_id'), \n",
    "    df_processed.StopName.alias('stop_name'),\n",
    "    df_processed.AdditionalTrip.alias('additional_trip'), \n",
    "    hour(df_processed.ArrivalTimeScheduledDate).alias(\"arrival_hour\").astype(StringType()),\n",
    "    hour(df_processed.DepartureTimeScheduledDate).alias(\"departure_hour\").astype(StringType()),\n",
    "    date_format(to_date(df_processed.TripDate, 'dd.MM.yyyy'), 'u').alias(\"day_of_week\"),\n",
    "    ((df_processed.ArrivalTimeActual - df_processed.ArrivalTimeScheduled)).alias(\"delta_arrival\").astype(FloatType()),\n",
    "    ((df_processed.DepartureTimeActual - df_processed.DepartureTimeScheduled)).alias(\"delta_departure\").astype(FloatType()))\n",
    "\n",
    "df_to_classify.cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(line_id='17010', product_id='Zug', stop_name='Basel Bad Bf', additional_trip='false', arrival_hour=None, departure_hour='5', day_of_week='3', delta_arrival=None, delta_departure=0.0),\n",
       " Row(line_id='17012', product_id='Zug', stop_name='Basel Bad Bf', additional_trip='false', arrival_hour=None, departure_hour='6', day_of_week='3', delta_arrival=None, delta_departure=0.0),\n",
       " Row(line_id='17013', product_id='Zug', stop_name='Basel Bad Bf', additional_trip='false', arrival_hour='6', departure_hour=None, day_of_week='3', delta_arrival=180.0, delta_departure=None),\n",
       " Row(line_id='17014', product_id='Zug', stop_name='Basel Bad Bf', additional_trip='false', arrival_hour=None, departure_hour='9', day_of_week='3', delta_arrival=None, delta_departure=0.0),\n",
       " Row(line_id='17015', product_id='Zug', stop_name='Basel Bad Bf', additional_trip='false', arrival_hour='8', departure_hour=None, day_of_week='3', delta_arrival=300.0, delta_departure=None)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_to_classify.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, for using the Decision Tree Regressor, and because each feature is in fact categorial, we must index each one of them using a *StringIndexer*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transform_dataset(dataset, departure):\n",
    "    '''\n",
    "    Function that transforms a dataset, adding for each categorial feature a column, which represents the output of the \n",
    "    StringIndexer applied to that column. \n",
    "    \n",
    "    Parameters:\n",
    "        - dataset: the dataset to be processed\n",
    "        - departure: True if the dataset is for departures, False otherwise\n",
    "    '''\n",
    "    \n",
    "    line_id_indexer = StringIndexer(inputCol=\"line_id\", outputCol=\"line_id_cat\", handleInvalid='keep') # keep nulls \n",
    "    product_id_indexer = StringIndexer(inputCol=\"product_id\", outputCol=\"product_id_cat\", handleInvalid='skip')\n",
    "    stop_name_indexer = StringIndexer(inputCol=\"stop_name\", outputCol=\"stop_name_cat\", handleInvalid='skip')\n",
    "    additional_trip_indexer = StringIndexer(inputCol=\"additional_trip\", outputCol=\"additional_trip_cat\", handleInvalid='skip')\n",
    "    day_of_week_indexer = StringIndexer(inputCol=\"day_of_week\", outputCol=\"day_of_week_cat\", handleInvalid='skip')\n",
    "    departure_hour_indexer = StringIndexer(inputCol=\"departure_hour\", outputCol=\"departure_hour_cat\", handleInvalid='skip')\n",
    "    arrival_hour_indexer = StringIndexer(inputCol=\"arrival_hour\", outputCol=\"arrival_hour_cat\", handleInvalid='skip')\n",
    "\n",
    "    indexers = [line_id_indexer, product_id_indexer, stop_name_indexer, additional_trip_indexer,day_of_week_indexer]\n",
    "    \n",
    "    if departure:\n",
    "        indexers.append(departure_hour_indexer)\n",
    "    else:\n",
    "        indexers.append(arrival_hour_indexer)\n",
    "\n",
    "    indexed = dataset\n",
    "\n",
    "    for indexer in indexers:\n",
    "        indexed = indexer.fit(indexed).transform(indexed) # add columns to dataset\n",
    "        \n",
    "    return indexed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we use the *VectorAssembler* to construct the column for features, which will be used by the Decision Tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_features_column(dataset, is_departure):\n",
    "    '''\n",
    "    Function that computes the features column for the given dataset.\n",
    "    \n",
    "    Parameters:\n",
    "        - dataset: the dataset to compute the features column for\n",
    "        - is_departure: True is dataset is used for departures, False otherwise.\n",
    "    '''\n",
    "    input_cols = ['line_id_cat', 'product_id_cat', 'stop_name_cat', 'additional_trip_cat', 'day_of_week_cat']\n",
    "    \n",
    "    if is_departure:\n",
    "        input_cols.append('departure_hour_cat') # departure dataset\n",
    "    else:\n",
    "        input_cols.append('arrival_hour_cat') # arrival dataset\n",
    "        \n",
    "    vector_assembler = VectorAssembler(inputCols = input_cols, outputCol = 'features')\n",
    "    dataset = transform_dataset(dataset, is_departure) # add categorial features\n",
    "    \n",
    "    df_features = vector_assembler.transform(dataset) # add features column\n",
    "    # Use VectorIndexer to make sure that the added features are recognized as categorical\n",
    "    \n",
    "    featureIndexer = \\\n",
    "        VectorIndexer(inputCol=\"features\", outputCol=\"indexedFeatures\", maxCategories=100000000).fit(df_features)\n",
    "    \n",
    "    df_features = featureIndexer.transform(df_features) # transform features to categorical\n",
    "    \n",
    "    if is_departure:\n",
    "        df_final = df_features.select(df_features.indexedFeatures, df_features.delta_departure.alias(\"delta\"))\n",
    "    else:\n",
    "        df_final = df_features.select(df_features.indexedFeatures, df_features.delta_arrival.alias(\"delta\"))\n",
    "    \n",
    "    return df_final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we construct our datasets to input to the Decision Tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Construct departures dataset\n",
    "df_departure_to_regress = df_to_classify.filter(\n",
    "    df_to_classify.departure_hour.isNotNull() & # filter only departures\n",
    "    df_to_classify.delta_departure.isNotNull())\n",
    "\n",
    "df_departure = compute_features_column(df_departure_to_regress, is_departure=True)\n",
    "\n",
    "# Construct arrivals dataset\n",
    "df_arrival_to_regress = df_to_classify.filter(\n",
    "    df_to_classify.arrival_hour.isNotNull() & # filter only arrivals\n",
    "    df_to_classify.delta_arrival.isNotNull())\n",
    "\n",
    "df_arrival = compute_features_column(df_arrival_to_regress, is_departure=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the generated dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(indexedFeatures=DenseVector([14483.0, 2.0, 2333.0, 0.0, 2.0, 18.0]), delta=0.0),\n",
       " Row(indexedFeatures=DenseVector([10292.0, 2.0, 2333.0, 0.0, 2.0, 11.0]), delta=0.0),\n",
       " Row(indexedFeatures=DenseVector([10275.0, 2.0, 2333.0, 0.0, 2.0, 13.0]), delta=0.0),\n",
       " Row(indexedFeatures=DenseVector([13971.0, 2.0, 2333.0, 0.0, 2.0, 12.0]), delta=0.0),\n",
       " Row(indexedFeatures=DenseVector([14059.0, 2.0, 2333.0, 0.0, 2.0, 8.0]), delta=780.0)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_departure.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(indexedFeatures=DenseVector([10223.0, 2.0, 2006.0, 0.0, 2.0, 11.0]), delta=180.0),\n",
       " Row(indexedFeatures=DenseVector([9633.0, 2.0, 2006.0, 0.0, 2.0, 4.0]), delta=300.0),\n",
       " Row(indexedFeatures=DenseVector([12909.0, 2.0, 2006.0, 0.0, 2.0, 12.0]), delta=60.0),\n",
       " Row(indexedFeatures=DenseVector([12924.0, 2.0, 2006.0, 0.0, 2.0, 8.0]), delta=300.0),\n",
       " Row(indexedFeatures=DenseVector([10172.0, 2.0, 2006.0, 0.0, 2.0, 6.0]), delta=300.0)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_arrival.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we write the function for training the Decision Tree Regressor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_regressor(dataset):\n",
    "    dt = DecisionTreeRegressor(featuresCol ='indexedFeatures', labelCol = 'delta', maxBins=100000000, maxDepth=3)\n",
    "    dt_model = dt.fit(dataset)\n",
    "    \n",
    "    return dt_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we train the decision trees for both datasets and we extract the most important features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature importances departures: (6,[0,2,5],[0.2845102900825045,0.0974031881251019,0.6180865217923937])\n",
      "Feature importances arrivals: (6,[0,2,3,5],[0.33459346799953293,0.06060240870976139,0.03545788856464659,0.569346234726059])\n"
     ]
    }
   ],
   "source": [
    "# Get most important fetrain_regressorpartures dataset\n",
    "regressor_departures = train_regressor(df_departure)\n",
    "print(\"Feature importances departures: {}\".format(regressor_departures.featureImportances))\n",
    "\n",
    "# Get most important features for departures dataset\n",
    "regressor_arrivals = train_regressor(df_arrival)\n",
    "print(\"Feature importances arrivals: {}\".format(regressor_arrivals.featureImportances))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we can see that the 3 most important features are, in both cases, the *hour*, the *line_id* and the *stop_name*. We can see that everything makes very much sense, because we have big differences of delays between normal hours and rush hours, for example, and also specific stops and routes have usually more delays than the others.\n",
    "\n",
    "Therefore, we continue by constructing the probability distributions for each possible value of the three most important features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### II. 2. Computing the probability distributions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we only consider the three most important features in the two initial datasets. We will consider the unity of time to be the minute from now on, instead of seconds: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_best_feat_departures = df_departure_to_regress.select(\n",
    "                df_departure_to_regress.departure_hour,\n",
    "                df_departure_to_regress.stop_name,\n",
    "                df_departure_to_regress.line_id,\n",
    "                (df_departure_to_regress.delta_departure / 60).astype(IntegerType()).alias(\"delta_minutes\"))\n",
    "\n",
    "df_best_feat_departures = df_best_feat_departures.filter(df_best_feat_departures.delta_minutes <= 0) \n",
    "# only keep departures which left on time or earlier, we do not want to base our recommendation on assumption\n",
    "# that a train or bus leaves with a delay.\n",
    "\n",
    "df_best_feat_arrival = df_arrival_to_regress.select(\n",
    "                df_arrival_to_regress.arrival_hour,\n",
    "                df_arrival_to_regress.stop_name,\n",
    "                df_arrival_to_regress.line_id,\n",
    "                (df_arrival_to_regress.delta_arrival / 60).astype(IntegerType()).alias(\"delta_minutes\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(departure_hour='5', stop_name='Basel Bad Bf', line_id='17010', delta_minutes=0),\n",
       " Row(departure_hour='6', stop_name='Basel Bad Bf', line_id='17012', delta_minutes=0),\n",
       " Row(departure_hour='9', stop_name='Basel Bad Bf', line_id='17014', delta_minutes=0),\n",
       " Row(departure_hour='10', stop_name='Basel Bad Bf', line_id='17016', delta_minutes=0),\n",
       " Row(departure_hour='14', stop_name='Basel Bad Bf', line_id='17024', delta_minutes=0)]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_best_feat_departures.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we want to make the distribution of delays for each possible value of the features, for both departures and arrivals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_departures_grouped_count = df_best_feat_departures.groupby( \n",
    "                df_best_feat_departures.departure_hour,\n",
    "                df_best_feat_departures.stop_name,\n",
    "                df_best_feat_departures.line_id,\n",
    "                df_best_feat_departures.delta_minutes).agg(count(lit(1)).alias(\"count_min\")) # add a count for each possible value\n",
    "        \n",
    "df_departures_distribution = df_departures_grouped_count.\\\n",
    "                                    groupby('departure_hour', 'stop_name', 'line_id').\\\n",
    "                                    agg(collect_list(struct('delta_minutes', 'count_min')).alias('counts'))\n",
    "\n",
    "# for each value of (departure_hour, stop_name, line_id), we have a list of the form [(delay_minutes, count)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(departure_hour='16', stop_name='Vevey', line_id='12257', delta_minutes=0, count_min=148),\n",
       " Row(departure_hour='16', stop_name='Pully', line_id='12262', delta_minutes=0, count_min=52),\n",
       " Row(departure_hour='22', stop_name='St-Saphorin', line_id='12285', delta_minutes=0, count_min=1)]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_departures_grouped_count.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------+--------------------+----------+-----------------+\n",
      "|departure_hour|           stop_name|   line_id|           counts|\n",
      "+--------------+--------------------+----------+-----------------+\n",
      "|             0|                 Bex|      3591|         [[0,50]]|\n",
      "|             0|         Biel/Bienne|      7845|         [[0,65]]|\n",
      "|             0|       Birkenstrasse| 85:836:83|         [[0,38]]|\n",
      "|             0| Bonstetten-Wettswil|     18594|          [[0,6]]|\n",
      "|             0|  Brügg (Bürglen UR)|85:816:403|         [[0,47]]|\n",
      "|             0|Chur, Böschenstrasse|85:766:904|          [[0,1]]|\n",
      "|             0|           Courgenay|     30486|         [[0,16]]|\n",
      "|             0|            Dietikon|     19292|        [[0,164]]|\n",
      "|             0|Dietikon, Birmens...|85:849:303|[[-1,6], [0,507]]|\n",
      "|             0|           Entlebuch|     30692|         [[0,12]]|\n",
      "+--------------+--------------------+----------+-----------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_departures_distribution.show(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_key_for_feature_values(hour, line_id, stop_name):\n",
    "    return '{}#{}#{}'.format(hour, line_id, stop_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "collected = df_departures_distribution.collect()\n",
    "\n",
    "distribution_departures = {\n",
    "    compute_key_for_feature_values(x.departure_hour, x.line_id, x.stop_name) : \n",
    "    list(sorted(x.counts, key=lambda y: y[0])) for x in collected}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do the same now for the arrivals: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_arrivals_grouped_count = df_best_feat_arrival.groupby( \n",
    "                df_best_feat_arrival.arrival_hour,\n",
    "                df_best_feat_arrival.stop_name,\n",
    "                df_best_feat_arrival.line_id,\n",
    "                df_best_feat_arrival.delta_minutes).agg(count(lit(1)).alias(\"count_min\")) # add a count for each possible value\n",
    "        \n",
    "df_arrivals_distribution = df_arrivals_grouped_count.\\\n",
    "                                    groupby('arrival_hour', 'stop_name', 'line_id').\\\n",
    "                                    agg(collect_list(struct('delta_minutes', 'count_min')).alias('counts'))\n",
    "        \n",
    "collected = df_arrivals_distribution.collect()\n",
    "\n",
    "distribution_arrivals = {\n",
    "    compute_key_for_feature_values(x.arrival_hour, x.line_id, x.stop_name) : \n",
    "    list(sorted(x.counts, key=lambda y: y[0])) for x in collected}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want to include a default distribution, for the case we have new data, which was not encountered so far. We will compute it as the distribution of all the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_default_distrib_departures = df_best_feat_departures.groupby('delta_minutes').agg(count(lit(1)).alias(\"count_min\")) # add a count for each possible value\n",
    "collected_default = df_default_distrib_departures.collect()\n",
    "default_departures = list(sorted(collected_default, key=lambda x: x[0]))\n",
    "\n",
    "df_default_distrib_arrivals = df_best_feat_arrival.groupby('delta_minutes').agg(count(lit(1)).alias(\"count_min\")) # add a count for each possible value\n",
    "collected_default = df_default_distrib_arrivals.collect()\n",
    "default_arrivals = list(sorted(collected_default, key=lambda x: x[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we add the default values to the dictionary of distributions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "distribution_departures['default'] = default_departures\n",
    "distribution_arrivals['default'] = default_arrivals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we transform the counts to probabilities, to be able to compute the final quality faster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transform_to_proba(counts_list):\n",
    "    total_sum = 0\n",
    "    final_proba = []\n",
    "    \n",
    "    for row in counts_list:\n",
    "        total_sum += row.count_min\n",
    "        \n",
    "    for row in counts_list:\n",
    "        final_proba.append((row.delta_minutes, row.count_min / total_sum))\n",
    "        \n",
    "    return final_proba"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "distribution_departures = {k : transform_to_proba(v) for k, v in distribution_departures.items()}\n",
    "distribution_arrivals = {k : transform_to_proba(v) for k, v in distribution_arrivals.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We finally write the computed dictionaries to file, to be able to load them later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILE_DISTRIBUTION_DEPARTURES = './data/distrib_departures.pic'\n",
    "FILE_DISTRIBUTION_ARRIVALS = './data/distrib_arrivals.pic'\n",
    "\n",
    "pickle.dump(distribution_departures, open(FILE_DISTRIBUTION_DEPARTURES, 'wb'))\n",
    "pickle.dump(distribution_arrivals, open(FILE_DISTRIBUTION_ARRIVALS, 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### II. 3. The exposed API for computing distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the last part is to write a function which receives the features of a specific transfer, and it returns the quality of the transfer, by performing the convolution of the corresponding distributions, using the formula:\n",
    "\n",
    "$\\sum\\limits_{t_a }\\Pr[\\mathcal{D}_a = t_a] \\cdot \\Pr[\\mathcal{D}_d = k-1+t_a]$,\n",
    "      \n",
    "where we have taken into consideration the minute needed by the traveler for changing the transport. \n",
    "       \n",
    "Here, we considered $\\mathcal{D}_a$ to be the distribution of arrivals and $\\mathcal{D}_d$ the distribution of departures.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATE_FORMAT = '%b %d %Y %H:%M:%S'\n",
    "\n",
    "FILE_DISTRIBUTION_DEPARTURES = './data/distrib_departures.pic'\n",
    "FILE_DISTRIBUTION_ARRIVALS = './data/distrib_arrivals.pic'\n",
    "\n",
    "class TransferQualityComputer:\n",
    "    def __init__(self):\n",
    "        self.distribution_departures = pickle.load(open(FILE_DISTRIBUTION_DEPARTURES, 'rb'))\n",
    "        self.distribution_arrivals = pickle.load(open(FILE_DISTRIBUTION_ARRIVALS, 'rb'))\n",
    "        \n",
    "    def compute_key_for_feature_values(self, hour, line_id, stop_name): # same as before\n",
    "        return '{}#{}#{}'.format(hour, line_id, stop_name)\n",
    "    \n",
    "    def compute_quality(self, arrival_timestamp, departure_timestamp, arrival_stop_name, departure_stop_name, \n",
    "                        arrival_line_id, departure_line_id, walktime=1):\n",
    "        # timestamp in the format: Dec 31 2017 20:40:49,01\n",
    "\n",
    "        arrival_time = datetime.strptime(arrival_timestamp[:-3], DATE_FORMAT)\n",
    "        departure_time = datetime.strptime(departure_timestamp[:-3], DATE_FORMAT)\n",
    "\n",
    "        arrival_hour = arrival_time.hour\n",
    "        departure_hour = departure_time.hour\n",
    "        delta_minutes = int((time.mktime(departure_time.timetuple()) - time.mktime(arrival_time.timetuple())) / 60)\n",
    "\n",
    "        if delta_minutes < 0:\n",
    "            return 0 # impossible to complete the transfer\n",
    "\n",
    "        departure_key = self.compute_key_for_feature_values(departure_hour, departure_line_id, departure_stop_name)\n",
    "        if departure_key in self.distribution_departures:\n",
    "            departure_dist = self.distribution_departures[departure_key]\n",
    "        else: \n",
    "            departure_dist = self.distribution_departures['default'] # default distribution\n",
    "\n",
    "        arrival_key = self.compute_key_for_feature_values(arrival_hour, arrival_line_id, arrival_stop_name)\n",
    "        if arrival_key in self.distribution_arrivals:\n",
    "            arrival_dist = self.distribution_arrivals[arrival_key]\n",
    "        else: \n",
    "            arrival_dist = self.distribution_arrivals['default'] # default distribution\n",
    "\n",
    "        total_proba = 0\n",
    "\n",
    "        for dep_delay, dep_proba in departure_dist:\n",
    "            for arr_delay, arr_proba in arrival_dist:\n",
    "\n",
    "                delta_minutes = (departure_time - arrival_time).total_seconds() // 60\n",
    "                if delta_minutes >= dep_delay + arr_delay + walktime: \n",
    "                # consider also walktime between stations, if the same station then we considered the walk time 1min\n",
    "                    total_proba += (dep_proba * arr_proba)\n",
    "                else:\n",
    "                    break\n",
    "\n",
    "        return total_proba"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7443859535306165\n"
     ]
    }
   ],
   "source": [
    "computer = TransferQualityComputer()\n",
    "\n",
    "print(computer.compute_quality('Dec 31 2017 08:40:49,01', 'Dec 31 2017 08:42:58,01', 'Dietikon, Birmensdorferstrasse', 'Dietikon, Birmensdorferstrasse', '85:849:303', '85:849:303', walktime=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III. Deterministic journey planner\n",
    "\n",
    "In our approach we use a standalone classic journey planner that works with a GTFS timetable. This planner is used to compute routes between two points in the deterministic way, without taking into consideration delays.\n",
    "\n",
    "It is implemented as a server which exposes a REST endpoint which allows GET requests to be made in order to obtain itineraries between two points.\n",
    "\n",
    "Below, we implement the requests to this planner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need an utility function that given a stop name uses the metadata to return the coordinates of the stop name. These coordinates are needed in the request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Get latitude and longitude from a stop name (string)\n",
    "def get_lat_long(name): \n",
    "    tmp = Df_meta.loc[Df_meta['StopName_Meta'] == name][['Lat', 'Long']]\n",
    "    \n",
    "    assert len(tmp) != 0, \"Problement with the location {}\".format(name)\n",
    "    \n",
    "    tmp = tmp.iloc[0]\n",
    "    return tmp['Lat'], tmp['Long']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### III. 1. The GET request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can implement the actual GET request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Make a request to the OTP server\n",
    "def return_request(fromPlace, toPlace, departure, Months, Days, Hours, AM_PM, Minutes, lat_long_from = False, lat_long_to = False):\n",
    "    \n",
    "    \n",
    "    # Handle the case where the stopnames have an added 'stop' at the beginning\n",
    "    if (fromPlace.split(' ')[0] == 'stop'):\n",
    "        fromPlace = toPlace[5:-1]\n",
    "    if (toPlace.split(' ')[0] == 'stop'):\n",
    "        toPlace = toPlace[5:-1]\n",
    "        \n",
    "    # If lattitude and longitude are not specified, find them thanks to the stop name\n",
    "    # Otherwise use their values straight away (safer from dataset mismatches)\n",
    "    if lat_long_from == False:\n",
    "        lat_from, long_from = get_lat_long(fromPlace)\n",
    "    else:\n",
    "        lat_from, long_from = lat_long_from[0], lat_long_from[1] \n",
    "\n",
    "    if lat_long_to == False:\n",
    "        lat_to, long_to = get_lat_long(toPlace)\n",
    "    else:\n",
    "        lat_to, long_to = lat_long_to[0], lat_long_to[1]\n",
    "    \n",
    "    #Compose the url for the request\n",
    "    url = 'http://10.90.38.21:8829/otp/routers/default/plan?fromPlace=stop+'\n",
    "    url += '+'.join(fromPlace.split()) +  '+%3A%3A' + str(lat_from) + '%2C' + str(long_from)\n",
    "    url += '&toPlace=stop+' +  '+'.join(toPlace.split()) +  '+%3A%3A' + str(lat_to) + '%2C' + str(long_to)\n",
    "    url += '&time={}%3A{}{}&date={}-{}-2018&mode=TRANSIT%2CWALK&maxWalkDistance=804.672&arriveBy={}&wheelchair=false&locale=en&numItineraries=3'.format(Hours, Minutes, AM_PM, Months, Days, not(departure))\n",
    "\n",
    "    #make the request and return the json\n",
    "    r = requests.get(url)\n",
    "\n",
    "    return r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### III. 2. Processing request answer\n",
    "The server responds to the GET request with a json structure encompassing a number of itineraries with additional information about each of them.\n",
    "\n",
    "We process this json in order to extract the data that is of interest to us and put it in a standard format which will be used throughout the later code when dealing with itineraries.\n",
    "\n",
    "Mainly, the data of interest for each itinerary returned by the server is mainly composed of details about the lets of the itinerary, like the transportation time (eg. BUS, RAIL, WALK), the coordinates of the stops and the arrival and departure times for each them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_json_extract_itineraries(json_data, df_BT):\n",
    "    #Instatiate the list of itineraries to be returned\n",
    "    itinerary_list = []\n",
    "    \n",
    "    #Avoid error. Usually caused by an empty json\n",
    "    if 'plan' not in json_data:\n",
    "        return []\n",
    "    \n",
    "    #Iterate over the itineraries\n",
    "    for route in json_data['plan']['itineraries']:\n",
    "        #Instatiate an itinerary\n",
    "        itinerary_ = []\n",
    "        #Iterate over the trip (legs) in an itinerary\n",
    "        for leg in route['legs']: \n",
    "            #Here handle relevant data for one trip of the itinerary   \n",
    "            mode = leg['mode']\n",
    "            from_ = leg['from']['name']\n",
    "            lat_from = leg['from']['lat']\n",
    "            lon_from = leg['from']['lon']\n",
    "            to_ = leg['to']['name']\n",
    "            lat_to = leg['to']['lat']\n",
    "            lon_to = leg['to']['lon']\n",
    "            \n",
    "            start_time = str(leg['from']['departure'])\n",
    "            departure_time = time.strftime(\"%b %d %Y %H:%M:%S,%M\", time.localtime(float(start_time[:len(start_time)-3])))\n",
    "            end_time = str(leg['endTime'])\n",
    "            arrival_time = time.strftime(\"%b %d %Y %H:%M:%S,%M\", time.localtime(float(end_time[:len(end_time)-3])))\n",
    "            duration = str(leg['duration'])\n",
    "            \n",
    "            #Not all trips have these attrubutes, so set default (meaning unknown) values if missing\n",
    "            route_id = 0\n",
    "            trip_id = 0\n",
    "            agency_name = 'unknown'\n",
    "            \n",
    "            if ('routeShortName' in leg.keys()):\n",
    "                route_id = leg['routeShortName']\n",
    "            if('tripShortName' in leg.keys()):\n",
    "                trip_id = leg['tripShortName']\n",
    "            if('agencyName' in leg.keys()):\n",
    "                agency_name = leg['agencyName']\n",
    "                \n",
    "            #For trains ('RAIL') the line_id is given by the trip_id\n",
    "            line_id = trip_id\n",
    "            \n",
    "            #For Bus and Tram there is no single indicator for line_id. \n",
    "            #We combine product_id, operatorName and routeID to uniquely identify the line_id in a pre-created dataframe\n",
    "            #In case of another mode of transport (product_id), also identify its line_id this way\n",
    "            if mode != 'RAIL' and mode!='WALK':\n",
    "                if mode == 'BUS' or mode == 'Bus':\n",
    "                    tmp = df_BT.query('ProductId.str.lower() == \"bus\" & OperatorName == @agency_name & LineType == @route_id')[:1]\n",
    "                elif mode =='TRAM' or mode == 'Tram':\n",
    "                    tmp = df_BT.query('ProductId.str.lower() == \"tram\" & OperatorName == @agency_name & LineType == @route_id')[:1]\n",
    "                else: \n",
    "                    tmp = df_BT.query('ProductId == @mode & OperatorName == @agency_name & LineType == @route_id')[:1]\n",
    "                if len(tmp) == 0:\n",
    "                    line_id = 'unknown'\n",
    "                else:\n",
    "                    line_id = tmp['LineId'].iloc[0]\n",
    "                \n",
    "           # print('Trip from {} at {} to {} at {} with {} and line_id: {}'.format(from_,departure_time,to_,arrival_time, mode, line_id))\n",
    "        \n",
    "            itinerary_.append({'product_id': mode, 'from': from_, 'lat_long_from': [lat_from, lon_from] ,'departure_time':departure_time,'to':to_, 'lat_long_to': [lat_to, lon_to], 'arrival_time':arrival_time, 'line_id': line_id})\n",
    "        itinerary_list.append(itinerary_)\n",
    "    return itinerary_list "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV. Quality aware journey planner\n",
    "\n",
    "In this section, we will perform the fusion between the last two sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IV. 1. Filtering itineraries by quality\n",
    "\n",
    "As we said previously, we are using a deterministic journey planner in order to get itineraries between two stops.\n",
    "\n",
    "On top of these itineraries we add the **confidence** of each transfer using the *TransferQualityComputer* functionality implemented above.\n",
    "\n",
    "We implement below a function that given an itinerary containing multiple legs with transfer, it computes its whole confidenc.\n",
    "\n",
    "### Assumption: all the delays are independent, so the quality of an itinerary is simply the multiplication of qualities of transfers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Compute the overall quality of an itinerary\n",
    "def comp_itinerary_quality(itinerary, quality_computer):\n",
    "    #Quality start at 1 and decays with every transfer\n",
    "    itinerary_quality = 1\n",
    "    \n",
    "    prev_leg = None\n",
    "    crt_leg = None\n",
    "    walking_time = 1\n",
    "    #iterate over the trips of an itinerary and affect the overall itinerary quality\n",
    "    for crt_leg in itinerary:\n",
    "        #Walk is a special case of trip. It should not diminish quality, but influence the next trip.\n",
    "        if crt_leg['product_id'] == 'WALK':\n",
    "            walking_time += (datetime.strptime(crt_leg['arrival_time'][:-3], DATE_FORMAT) - \\\n",
    "                            datetime.strptime(crt_leg['departure_time'][:-3], DATE_FORMAT)).total_seconds()//60\n",
    "        else:\n",
    "            if prev_leg is not None:\n",
    "                #Compute the quality of the transfer\n",
    "                transfer_quality = quality_computer.compute_quality(\n",
    "                    prev_leg['arrival_time'],\n",
    "                    crt_leg['departure_time'],\n",
    "                    prev_leg['to'],\n",
    "                    crt_leg['from'],\n",
    "                    prev_leg['line_id'],\n",
    "                    crt_leg['line_id'],\n",
    "                    walking_time\n",
    "                )\n",
    "                #Affect the itinerary\n",
    "                itinerary_quality *= transfer_quality\n",
    "                \n",
    "            prev_leg = crt_leg\n",
    "            walking_time = 1\n",
    "            \n",
    "    return itinerary_quality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another utility function that we need further down the process is to simply split a list of itineraries into itineraries that have a quality bigger than a specified threshold and itineraries with a lower quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_with_quality(itinerary_list, Quality, quality_computer):\n",
    "    itinerary_quality_ = [0,]*len(itinerary_list)\n",
    "    #print('Quality of itineraries:  ')\n",
    "    for i_ in range(len(itinerary_list)):\n",
    "        itinerary_quality_[i_] = comp_itinerary_quality(itinerary_list[i_], quality_computer)\n",
    "        #print('Itinerary number {}, quality: {}'.format(i_,itinerary_quality_[i_]))\n",
    "    itinerary_list_accepted = np.array(itinerary_list)[[it_>Quality for it_ in itinerary_quality_]].tolist()\n",
    "    itinerary_list_refused = np.array(itinerary_list)[[not(it_>Quality) for it_ in itinerary_quality_]].tolist()\n",
    "    return itinerary_list_accepted, itinerary_list_refused"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IV. 2. Explore itineraries \"around\" a too-low-quality itinerary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function explores itineraries the are slightly different from the itinerary passed as argument: At every transfer station of the original itinerary, it will explore three new itineraries that can reach the final destination. Finally it will append all these newly found itineraries to a list.\n",
    "\n",
    "For example for an itinerary with 3 trips: there will be 2 transfer stations. The function will return 6 itineraries. 3 itineraries will have as first trip the one of the original itinerary (and then different ways to reach the destination). The 3 other itineraries will have as first and second trips the one of the original itinerary (and then different ways to reach the destination).\n",
    "\n",
    "The logic behind this is that by searching three itineraries from a given station, we will have the possibility to take the first available train, or wait and take a later one. As such, this function will return itineraries that are similar to the original one except that one of the transfers is longer. The quality will thus be increased.\n",
    "\n",
    "If a transfer station of the original itinerary already disrespects the quality criterion, the search stops there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def explore_itineraries(itinerary, df_BT, quality, quality_computer):\n",
    "    itinerary_list = []\n",
    "    for j_ in range(len(itinerary)-1):\n",
    "        #leg_1 = itinerary[j_]\n",
    "        if comp_itinerary_quality(itinerary[0:j_+1], quality_computer) < quality:\n",
    "            continue\n",
    "        arr_month, arr_day, arr_hour, arr_AM_PM, arr_minute, arr_second = date_to_cells(itinerary[j_]['arrival_time'])\n",
    "        #new_semi_its = request_with_quality(fromPlace = leg_1['to'], toPlace = itinerary[-1]['to'], Months = arr_month, Days = arr_day, Hours = arr_hour, Minutes = arr_minute, Seconds = arr_second, AM_PM = arr_AM_PM, departure = True, Quality = quality, lat_long_from = leg_1['lat_long_from'], lat_long_to = itinerary[-1]['lat_long_to'] )\n",
    "        temp_json = return_request(fromPlace = itinerary[j_]['to'], toPlace = itinerary[-1]['to'], Months = arr_month, Days = arr_day, Hours = arr_hour, Minutes = arr_minute, AM_PM = arr_AM_PM, departure = True, lat_long_from = itinerary[j_]['lat_long_to'], lat_long_to = itinerary[-1]['lat_long_to'])\n",
    "        #print(temp_json)\n",
    "        new_partial_its = read_json_extract_itineraries(temp_json, df_BT)\n",
    "        new_itineraries = [np.append(itinerary[:j_+1],new_partial_its[k_]).tolist() for k_ in range(len(new_partial_its))]\n",
    "        \n",
    "        itinerary_list.extend(new_itineraries)\n",
    "    return itinerary_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transform a date in string format (as in the timetable), in separate cells for month, day, hour, ect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def date_to_cells(date):\n",
    "    dt = datetime.strptime(date[:-3], DATE_FORMAT)\n",
    "    return dt.month,\\\n",
    "          dt.day,\\\n",
    "          (dt.hour if dt.hour <= 12 else dt.hour-12),\\\n",
    "          ('AM' if dt.hour <= 12 else 'PM'), \\\n",
    "          dt.minute,\\\n",
    "          dt.second"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IV. 3. Find the best itineraries\n",
    "\n",
    "Now that we are able to find itineraries given a specified confidence(quality) threshold, we need to expose a nice API that would allow this functionality to be exploited in an easy manner, hiding the complexity of the search and uncertanity distribution behind a simple function call\n",
    "\n",
    "First we initialize the a TransferQualityComputer object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_computer = TransferQualityComputer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can implement the actual function that searches for an itinerary given a quality threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_itinerary_with_quality(fromPlace , toPlace, Months, Days, Hours, AM_PM, Minutes, departure, quality, quality_computer):\n",
    "    #Get the json of quickest itineraries from local OTP server\n",
    "    test_json = return_request(fromPlace=fromPlace , toPlace= toPlace ,Months = Months, Days= Days, Hours= Hours, AM_PM = AM_PM, Minutes = Minutes, departure = departure)\n",
    "    \n",
    "    #Read json and create itinerary list of dicts\n",
    "    itinerary_first_list = read_json_extract_itineraries(test_json, df_BT)\n",
    "    itinerary_acc, itinerary_refu = split_with_quality(itinerary_first_list, quality, quality_computer)\n",
    "    itinerary_searched = []\n",
    "    iter_=0\n",
    "\n",
    "    ## sort bad quality itineraries by arrival time\n",
    "    sorter_ids = np.argsort([itinerary_refu[i_][-1]['arrival_time'] for i_ in range(len(itinerary_refu))])\n",
    "    itinerary_refu = np.array(itinerary_refu)[sorter_ids].tolist()\n",
    "    \n",
    "    while len(itinerary_refu) != 0:\n",
    "        if len(itinerary_acc)>=3:\n",
    "            break\n",
    "        iter_+=1\n",
    "        \n",
    "        itinerary_searched_ = itinerary_refu.pop(0)\n",
    "        itinerary_searched.append(itinerary_searched_)\n",
    "        \n",
    "        itinerary_test_list_explored = explore_itineraries(itinerary_searched_, df_BT, quality, quality_computer)\n",
    "        itinerary_acc_explored, itinerary_refu_explored = split_with_quality(itinerary_test_list_explored, quality, quality_computer)\n",
    "        \n",
    "        for iti_refu in itinerary_refu_explored:\n",
    "            if not(any([(iti_refu[:-1] == iti[:-1]) for iti in itinerary_refu+itinerary_searched])):\n",
    "                itinerary_refu.append(iti_refu)\n",
    "        for iti_acc in itinerary_acc_explored:\n",
    "            if not(any([(iti_acc == iti) for iti in itinerary_acc])):\n",
    "                itinerary_acc.append(iti_acc)\n",
    "\n",
    "        ## sort by arrival time\n",
    "        sorter_ids = np.argsort([itinerary_refu[i_][-1]['arrival_time'] for i_ in range(len(itinerary_refu))])\n",
    "        itinerary_refu = np.array(itinerary_refu)[sorter_ids].tolist()\n",
    "    \n",
    "    \n",
    "    ## sort selected itineraries by arrival time\n",
    "    sorter_ids = np.argsort([itinerary_acc[i_][-1]['arrival_time'] for i_ in range(len(itinerary_acc))])\n",
    "    itinerary_acc = np.array(itinerary_acc)[sorter_ids].tolist()\n",
    "\n",
    "    \n",
    "#     for i_, itinerary_acc_ in enumerate(itinerary_acc[:3]):\n",
    "#         print('\\n Itinerary number: {}'.format(i_))\n",
    "#         for j_, leg in enumerate(itinerary_acc_):\n",
    "#             print('Leg {}: Take {} from {} at {} to {} arriving at {}'.format(j_, leg['product_id'],leg['from'], leg['departure_time'], leg['to'], leg['arrival_time']))\n",
    "    return itinerary_acc, itinerary_first_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IV. 4. Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can test it with a sample example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_ = 0.90\n",
    "fromPlace_ = \"Zürich, Sunnau\"\n",
    "toPlace_ = 'Lausanne'\n",
    "Months_ = 2\n",
    "Days_ = 4\n",
    "Hours_ = 6\n",
    "AM_PM_ = 'PM' \n",
    "Minutes_ = 20\n",
    "departure_ = True\n",
    "quality_itins, classic_itins = find_itinerary_with_quality(fromPlace_ , toPlace_, Months_, Days_, Hours_, AM_PM_, Minutes_, departure_, quality_, quality_computer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to further validate our itineraries, we will take a look at what would've been the itineraries computes in the deterministic way and our itineraries.\n",
    "\n",
    "First we implement an utility function that counts the number of trips in a transfer without taking into consideration walking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_length_no_walking(itinerary):\n",
    "    length = 0\n",
    "    for leg in itinerary:\n",
    "        if leg['product_id'] != 'WALK':\n",
    "            length += 1\n",
    "    return length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Fastest itineraries without quality constraint:\n",
      "Itinerary number: 0, quality: 0.7682491228769168, dpt: Feb 04 2018 18:30:00,30, arr: Feb 04 2018 21:16:23,16, transfers: 4\n",
      "Itinerary number: 1, quality: 0.9886327436915376, dpt: Feb 04 2018 18:46:00,46, arr: Feb 04 2018 21:40:23,40, transfers: 3\n",
      "Itinerary number: 2, quality: 0.9746956413346076, dpt: Feb 04 2018 19:00:00,00, arr: Feb 04 2018 21:45:28,45, transfers: 3\n",
      "\n",
      " Fastest itineraries with quality constraint:\n",
      "Itinerary number: 0, quality: 0.9886327436915376, dpt: Feb 04 2018 18:46:00,46, arr: Feb 04 2018 21:40:23,40, legs: 3\n",
      "Itinerary number: 1, quality: 0.9963089153889833, dpt: Feb 04 2018 18:30:00,30, arr: Feb 04 2018 21:40:23,40, legs: 3\n",
      "Itinerary number: 2, quality: 0.9746956413346076, dpt: Feb 04 2018 19:00:00,00, arr: Feb 04 2018 21:45:28,45, legs: 3\n",
      "Itinerary number: 3, quality: 0.9997160704145367, dpt: Feb 04 2018 18:30:00,30, arr: Feb 04 2018 21:45:28,45, legs: 3\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print('\\n Fastest itineraries without quality constraint:')\n",
    "itinerary_initial_quality_ = [0,]*len(classic_itins)\n",
    "for i_ in range(len(classic_itins)):\n",
    "    itinerary_initial_quality_[i_] = comp_itinerary_quality(classic_itins[i_], quality_computer)\n",
    "    print('Itinerary number: {}, quality: {}, dpt: {}, arr: {}, transfers: {}'\\\n",
    "                  .format(i_,\n",
    "                          itinerary_initial_quality_[i_],\n",
    "                          classic_itins[i_][0]['departure_time'],\n",
    "                          classic_itins[i_][-1]['arrival_time'],\n",
    "                          compute_length_no_walking(classic_itins[i_])))\n",
    "\n",
    "print('\\n Fastest itineraries with quality constraint:')\n",
    "## Print out the arrival time and quality of the three selected paths\n",
    "itinerary_selected_quality_ = [0,]*len(quality_itins)\n",
    "for i_ in range(len(quality_itins)):\n",
    "    itinerary_selected_quality_[i_] = comp_itinerary_quality(quality_itins[i_], quality_computer)\n",
    "    print('Itinerary number: {}, quality: {}, dpt: {}, arr: {}, legs: {}'\\\n",
    "                  .format(i_,\n",
    "                          itinerary_selected_quality_[i_],\n",
    "                          quality_itins[i_][0]['departure_time'],\n",
    "                          quality_itins[i_][-1]['arrival_time'],\n",
    "                          compute_length_no_walking(quality_itins[i_])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Get news from SBB - extra info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look for possible news from SBB for the given date and array of stop names. Typically news from SBB will say if a line is broken or if delays can be expected. The user will interpret this information by hisself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_info(date, stopName):\n",
    "    url = 'https://data.sbb.ch/api/records/1.0/search/?dataset=rail-traffic-information&lang=en&rows=1000&sort=validityend&facet=validitybegin&facet=validityend&refine.validitybegin={}'.format(date[0])\n",
    "    tmp = requests.get(url).json()\n",
    "    infos = []\n",
    "    for el in tmp['records']: \n",
    "        end = str(el['fields']['validityend'].split('T')[0]).split('-')\n",
    "        if((int(end[0]) == int(date[0]) and int(end[1]) == int(date[1]) and int(end[2]) < int(date[2])) or (int(end[0]) == int(date[0]) and int(end[1]) < int(date[1])) or (int(end[0]) < int(date[0]))):\n",
    "            break\n",
    "        #print(end)\n",
    "        title = el['fields']['title']\n",
    "        if('End of announcement:' in title): \n",
    "            pass\n",
    "        else:\n",
    "            if(len(title.split(':')) > 1):\n",
    "                title = str(title.split(':')[1])\n",
    "            title = title.replace(' and', '-').replace('engineering work is in progress', '').replace(',','').replace('.', '').replace('Between', '').replace('In', '').replace(' station', '').replace('Work due to a disruption','').strip()\n",
    "            #print(title.split('- '))\n",
    "            for el_title in title.split('- '): \n",
    "                for el_stop in stopName: \n",
    "                    if(el_title.strip() == el_stop.strip()): \n",
    "                        print(el_title)\n",
    "                        infos.append(el['fields']['description'])\n",
    "    return infos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## V. Visual validation of the robust journey planner\n",
    "\n",
    "One validation method that is suitable is to use an isochronous map in order to visualize how far one can hypothetically go in a fixed time interval and from a fixed station.\n",
    "\n",
    "We use this visualization method to compare our robust journey planner with a deterministic journey planner. \n",
    "\n",
    "We do so by first computing the isochronous map for the deterministic journey planner and show the quality of each trip.Then, we compute another isochronous map using our robust journey planner where we set a threshold for confidence of 95%. \n",
    "\n",
    "The main idea is to see that for stations which have a low confidence when using itineraries recommended by the deterministic journey planner, the robust journey planner recommends different itineraries but which have the expected quality.\n",
    "\n",
    "As we are interested in the area surrounding Zurich HB by a radius of 10km we compute the data for the map by querying the route planner for itineraries from Zurich HB to every other station within the 10km radius.\n",
    "\n",
    "For each of the stations, we will plot a circle centered in it with radius directly proportional with the walking time left up until the time limit. We set an average walking speed of 5km/h and using the time left, we compute the distance around the station that can be walked.\n",
    "\n",
    "For each station we also get the certainty of arriving there in % of times we would be able to actually make the journey and this value between (0,1) is mapped to a step color scale. Hence, green corresponds to the (95%, 100%) interval, blue to the (25%, 95%), and red to (0%, 25%). We chose these intervals in order to have a clear differentiation between the itineraries with really high confidence and the others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "import branca.colormap as cm\n",
    "import folium\n",
    "\n",
    "ZURICH_HB_COORDS = [47.377941, 8.540141]\n",
    "\n",
    "AVERAGE_WALKING_SPEED_PER_SECOND = 1.38889 # 5kph but in meters per second\n",
    "STEP_CM = cm.StepColormap(\n",
    "    ['red', 'blue','green'],\n",
    "    index=[0, 0.25, 0.95, 1]\n",
    ")\n",
    "STEP_CM.caption = 'Quality of trip'\n",
    "\n",
    "\n",
    "def add_circle(m, coords, quality, time_left_in_seconds, popup_data):\n",
    "    radius = time_left_in_seconds * AVERAGE_WALKING_SPEED_PER_SECOND / 10\n",
    "    folium.Circle(\n",
    "        coords,\n",
    "        radius,\n",
    "        fill=True,\n",
    "        fill_color=STEP_CM(quality),\n",
    "        fill_opacity=0.2,\n",
    "        stroke=False,\n",
    "        fill_rule='nonzero',\n",
    "        popup=\"Arrival time: {}<br\\>Q : {:.3f}<br\\>Time left: {} mins<br\\>Legs: {}\"\\\n",
    "                    .format(popup_data['arrival_time'], quality, (time_left_in_seconds//60), popup_data['nb_transfers'])\n",
    "    ).add_to(m)\n",
    "    \n",
    "\n",
    "    \n",
    "def create_map_with_quality(source_name, source_coord, stations_data):\n",
    "    m = folium.Map(source_coord, zoom_start=13, tiles='Stamen toner') \n",
    "    m.add_child(STEP_CM)\n",
    "    popup_data = {}\n",
    "    for data in stations_data:\n",
    "        popup_data['station_name'] = data[0]\n",
    "        popup_data['arrival_time'] = data[3]\n",
    "        popup_data['nb_transfers'] = compute_length_no_walking(data[5])\n",
    "        add_circle(m, data[1], data[2], data[4]*60, popup_data)\n",
    "    return m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing we do, we select the stations that are at most 10 km from Zurich HB.\n",
    "\n",
    "For this, we compute the distance from Zurich HB for every stop name in the df_meta. Then we keep those stops that have this distance less than 10km."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Coordinate of the main station of Zürich\n",
    "Lat_zu = 47.377941\n",
    "Long_ZU = 8.540141\n",
    "def dist_to_ZU(lat, long): \n",
    "    res = str(geo_dist((lat, long), (Lat_zu, Long_ZU)))\n",
    "    res = round(float(res.split()[0]),1)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "Df_meta['Dist in km'] = Df_meta.apply(lambda x: dist_to_ZU(x['Lat'], x['Long']), axis=1)\n",
    "stops_zurich = Df_meta[Df_meta['Dist in km'] < 10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have in **stops_zurich** the stops that are in a 10km radius from Zurich, we extract the names as we need them for the route query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "zurich_stations = stops_zurich['StopName_Meta'].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we take a look at the station names, we can see there are duplicates which we choose to drop as they do not come with information about additional stops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Urdorf',\n",
       " 'Urdorf',\n",
       " 'Urdorf',\n",
       " 'Birmensdorf ZH',\n",
       " 'Birmensdorf ZH',\n",
       " 'Birmensdorf ZH',\n",
       " 'Bonstetten-Wettswil',\n",
       " 'Bonstetten-Wettswil',\n",
       " 'Bonstetten-Wettswil',\n",
       " 'Urdorf Weihermatt']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zurich_stations[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "zurich_stations = list(set(zurich_stations))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V. 1. Computing the travel times to stops close to Zurich\n",
    "\n",
    "The next step in the visualization process is to compute the **arrival time** and **qualities** to the stops of interest. \n",
    "\n",
    "In order to do this, we query our route planner for routes from Zurich HB to every stop within 10km obtaining itineraries of which we are interested only in the arrival time at the final stop and the quality.\n",
    "\n",
    "There are a two parameters that will shape our visualization:\n",
    "    1. the start time we set for the trips - parameter required to make the queries\n",
    "    2. the maximum length in time of the trips - used to filter the destinations to which the travel time takes more than this value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first implement some utility functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_datetime_string(dt):\n",
    "    return datetime.strptime(dt, '%b %d %Y %H:%M:%S,%f')\n",
    "\n",
    "def compute_remaining_travel_time(itinerary, departure_datetime, max_travel_minutes):\n",
    "    '''\n",
    "    Function that computes travel minutes left from the quota specified by max_travel_minutes\n",
    "    '''\n",
    "    last_step = itinerary[-1]\n",
    "    arrival_datetime = parse_datetime_string(last_step['arrival_time'])\n",
    "    travel_time_minutes = (arrival_datetime - departure_datetime).total_seconds()/60\n",
    "    return max_travel_minutes - travel_time_minutes\n",
    "    \n",
    "\n",
    "def compute_length_no_walking(itinerary):\n",
    "    length = 0\n",
    "    for leg in itinerary:\n",
    "        if leg['product_id'] != 'WALK':\n",
    "            length += 1\n",
    "    return length\n",
    "\n",
    "def compute_itineraries_deterministic(station, source_station, departure_datetime, max_travel_minutes):\n",
    "    '''\n",
    "    Function that obtaines for the specified parameters, itineraries in the deterministic way, i.e. without taking\n",
    "    into consideration the confidence of intervals\n",
    "    '''\n",
    "    request_json = return_request(source_station,\n",
    "                                  station,\n",
    "                                  True,\n",
    "                                  departure_datetime.month,\n",
    "                                  departure_datetime.day,\n",
    "                                  departure_datetime.hour if departure_datetime.hour <= 12 else departure_datetime.hour-12,\n",
    "                                  'AM' if departure_datetime.hour <= 12 else 'PM',\n",
    "                                  departure_datetime.minute,\n",
    "                                  departure_datetime.second,)\n",
    "\n",
    "    itineraries = read_json_extract_itineraries(request_json, df_BT)\n",
    "    \n",
    "    return itineraries\n",
    "\n",
    "def compute_itineraries_given_confidence(station, source_station, departure_datetime, max_travel_minutes, quality_computer, quality=0.95):\n",
    "    '''\n",
    "    Function that, for the given parameters, obtains the itineraries that respect the confidence(quality)\n",
    "    threshold specified as a parameter\n",
    "    '''\n",
    "    itineraries = find_itinerary_with_quality(source_station,\n",
    "                                              station,\n",
    "                                             departure_datetime.month,\n",
    "                                             departure_datetime.day,\n",
    "                                             departure_datetime.hour if departure_datetime.hour <= 12 else departure_datetime.hour-12,\n",
    "                                            'AM' if departure_datetime.hour <= 12 else 'PM',\n",
    "                                             departure_datetime.minute,\n",
    "                                             departure=True,\n",
    "                                             quality=quality,\n",
    "                                             quality_computer=quality_computer\n",
    "                                             )[0]\n",
    "    return itineraries\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def get_stop_plot_data(station, source_station, departure_datetime, max_travel_minutes, quality_computer, with_confidence=True, quality=0.95):\n",
    "    '''\n",
    "    Function that computes, for each station from stations_names which is within max_travel_minutes\n",
    "    of source_station, the coords, quality and time left from max_travel_minutes after arriving there\n",
    "    '''\n",
    "    if with_confidence:\n",
    "        itineraries = compute_itineraries_given_confidence(station, source_station, departure_datetime, max_travel_minutes,\n",
    "                                                          quality_computer, quality=quality)\n",
    "    else:\n",
    "        itineraries = compute_itineraries_deterministic(station, source_station, departure_datetime, max_travel_minutes)\n",
    "    \n",
    "    if len(itineraries) == 0:\n",
    "         return None\n",
    "\n",
    "    fastest_itinerary = itineraries[0]\n",
    "    remaining_travel_minutes = compute_remaining_travel_time(fastest_itinerary, departure_datetime, max_travel_minutes)\n",
    "\n",
    "    if remaining_travel_minutes > 0:\n",
    "        quality = comp_itinerary_quality(fastest_itinerary, quality_computer)\n",
    "        plot_data = (\n",
    "            station,\n",
    "            fastest_itinerary[-1]['lat_long_to'],\n",
    "            quality,\n",
    "            fastest_itinerary[-1]['arrival_time'],\n",
    "            remaining_travel_minutes,\n",
    "            fastest_itinerary\n",
    "        )\n",
    "        return plot_data\n",
    "    else:\n",
    "        return None\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we mentioned in the explanation above, here we will compute the two maps for comparison.\n",
    "\n",
    "First we compute the isochronous map for the deterministic journey planner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters of the visualization\n",
    "ZURICH_HB_NAME = 'Zürich HB'\n",
    "\n",
    "Months_ = 4\n",
    "Days_ = 6\n",
    "Hours_ = 8\n",
    "AM_PM_ = 'AM'\n",
    "Minutes_ = 30\n",
    "\n",
    "departure_ = True\n",
    "Max_travel_time_ = 60 # in minutes\n",
    "Hours_24 = Hours_%12 if AM_PM_ == 'AM' else (Hours_%12)+12\n",
    "departure_datetime = datetime(2018, Months_, Days_, Hours_24, Minutes_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compute for each station the arrival time and confidence in the case of deterministic journey planner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "PLOT_DATA_FILENAME = 'generated_maps/zurich-830am-60mins_deterministic_itineraries.pkl'\n",
    "\n",
    "if not os.path.isfile(PLOT_DATA_FILENAME):\n",
    "    deterministic_plot_data = list(filter(lambda x: x is not None, \n",
    "                                             map(lambda station: get_stop_plot_data(station,\n",
    "                                                                                     ZURICH_HB_NAME,\n",
    "                                                                                     departure_datetime,\n",
    "                                                                                     Max_travel_time_,\n",
    "                                                                                     quality_computer,\n",
    "                                                                                     with_confidence=False),\n",
    "                                                 zurich_stations)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the plot data compute above in order to be able so reuse it a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.isfile(PLOT_DATA_FILENAME):\n",
    "    with open(PLOT_DATA_FILENAME, 'rb') as f:\n",
    "        deterministic_plot_data = pickle.load(f)\n",
    "else:\n",
    "    with open(PLOT_DATA_FILENAME, 'wb') as f:\n",
    "        pickle.dump(deterministic_plot_data, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "deterministic_map = create_map_with_quality('Zurich HB', ZURICH_HB_COORDS, deterministic_plot_data)\n",
    "deterministic_map.save('generated_maps/zurich-830am-60mins_deterministic_itineraries.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can directly see the map of [Zürich 8:30AM deterministic itineraries](https://enguerrandgrx.github.io/maps/zurich-830am-60mins_deterministic_itineraries.html) and compare with [Zürich 8:30AM itineraries with 95 of confidance](https://enguerrandgrx.github.io/maps/zurich-830am-60mins_robust_itineraries_confidence95.html) which is computed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "PLOT_DATA_FILENAME = 'generated_maps/zurich-830am-60mins_robust_itineraries_confidence95.pkl'\n",
    "\n",
    "if not os.path.isfile(PLOT_DATA_FILENAME):\n",
    "    robust_plot_data = list(filter(lambda x: x is not None, \n",
    "                                             map(lambda station: get_stop_plot_data(station,\n",
    "                                                                                     ZURICH_HB_NAME,\n",
    "                                                                                     departure_datetime,\n",
    "                                                                                     Max_travel_time_,\n",
    "                                                                                     quality_computer,\n",
    "                                                                                     with_confidence=True,\n",
    "                                                                                     quality=0.95),\n",
    "                                                 zurich_stations)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.isfile(PLOT_DATA_FILENAME):\n",
    "    with open(PLOT_DATA_FILENAME, 'rb') as f:\n",
    "        robust_plot_data = pickle.load(f)\n",
    "else:\n",
    "    with open(PLOT_DATA_FILENAME, 'wb') as f:\n",
    "        pickle.dump(robust_plot_data, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "robust_map = create_map_with_quality('Zurich HB', ZURICH_HB_COORDS, robust_plot_data)\n",
    "robust_map.save('generated_maps/zurich-830am-60mins_robust_itineraries_confidence95.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V. 2. Comparison with Bern\n",
    "\n",
    "We did not limit ourselves only to the Zurich area in none of the choices we made throughout implementing the journey planner.\n",
    "\n",
    "Hence, we compute the same two isochronous maps also for the Bern area."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "BERN_NAME = 'Bern'\n",
    "BERN_COORDS = (46.94972, 7.43944)\n",
    "def dist_to_BN(lat, long): \n",
    "    res = str(geo_dist((lat, long), BERN_COORDS))\n",
    "    res = round(float(res.split()[0]),1)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first filter the stations for the ones close to Bern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "Df_meta['km_to_bern'] = Df_meta.apply(lambda x: dist_to_BN(x['Lat'], x['Long']), axis=1)\n",
    "stops_bern = Df_meta[Df_meta['km_to_bern'] < 10]\n",
    "bern_stations = stops_bern['StopName_Meta'].tolist()\n",
    "bern_stations = list(set(bern_stations))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can compute for each station the arrival time and with which confidence, in the case of the deterministic journey planner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "PLOT_DATA_FILENAME = 'generated_maps/bern-830am-60mins-deterministic_itineraries.pkl'\n",
    "\n",
    "if not os.path.isfile(PLOT_DATA_FILENAME):\n",
    "    deterministic_plot_data_bern = list(filter(lambda x: x is not None, \n",
    "                                             map(lambda station: get_stop_plot_data(station,\n",
    "                                                                                     BERN_NAME,\n",
    "                                                                                     departure_datetime,\n",
    "                                                                                     Max_travel_time_,\n",
    "                                                                                     quality_computer,\n",
    "                                                                                     with_confidence=False),\n",
    "                                                 bern_stations)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the plot data computed above in a pickle such that we do not have to make requests to the journey planner again for these stations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.isfile(PLOT_DATA_FILENAME):\n",
    "    with open(PLOT_DATA_FILENAME, 'rb') as f:\n",
    "        deterministic_plot_data_bern = pickle.load(f)\n",
    "else:\n",
    "    with open(PLOT_DATA_FILENAME, 'wb') as f:\n",
    "        pickle.dump(deterministic_plot_data_bern, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can simply create the map."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "deterministic_map_bern = create_map_with_quality(\"Bern\", BERN_COORDS, deterministic_plot_data_bern)\n",
    "deterministic_map_bern.save('generated_maps/bern-830am-60mins-deterministic_itineraries.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can directly see the map of [Bern 8:30AM deterministic itineraries](https://enguerrandgrx.github.io/maps/bern-830am-60mins-deterministic_itineraries.html) and compare with [Bern 8:30AM itineraries with 95 of confidance](https://enguerrandgrx.github.io/maps/bern-830am-60mins-robust_itineraries_quality95.html) which is computed below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do the same steps for the robust map, the one where we set a threshold of quality for itineraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "PLOT_DATA_FILENAME = 'generated_maps/bern-830am-60mins-robust_itineraries_quality95.pkl'\n",
    "\n",
    "if not os.path.isfile(PLOT_DATA_FILENAME):\n",
    "    robust_plot_data_bern = list(filter(lambda x: x is not None, \n",
    "                                             map(lambda station: get_stop_plot_data(station,\n",
    "                                                                                     BERN_NAME,\n",
    "                                                                                     departure_datetime,\n",
    "                                                                                     Max_travel_time_,\n",
    "                                                                                     quality_computer,\n",
    "                                                                                     with_confidence=True,\n",
    "                                                                                     quality=0.95),\n",
    "                                                 bern_stations)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.isfile(PLOT_DATA_FILENAME):\n",
    "    with open(PLOT_DATA_FILENAME, 'rb') as f:\n",
    "        robust_plot_data_bern = pickle.load(f)\n",
    "else:\n",
    "    with open(PLOT_DATA_FILENAME, 'wb') as f:\n",
    "        pickle.dump(robust_plot_data_bern, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "robust_map_bern = create_map_with_quality(\"Bern\", BERN_COORDS, robust_plot_data_bern)\n",
    "robust_map_bern.save('generated_maps/bern-830am-60mins-robust_itineraries_quality95.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "\n",
    "We constructed a prototype for a *robust certainty-aware journey planner*. Even though there are multiple simplifying assumptions in the data, such as that the *delays are independent*, or that the *client cannot use the fact that a train departs late from a station*, our solution still shows that this kinds of products are certainly feasible to implement by a team of developers and data scientists. \n",
    "\n",
    "We hope that in the near future, companies like SBB will provide data for the entire year, which will make the product more robust and have better features for computing the distributions, and also that they will implement similar products, which could improve the lifes of many clients."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}