Evaluate_power_vs_trip_start_end_analysed_master.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for reading and validating data\n",
    "import emeval.input.spec_details as eisd\n",
    "import emeval.input.phone_view as eipv\n",
    "import emeval.input.eval_view as eiev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualization helpers\n",
    "import emeval.viz.phone_view as ezpv\n",
    "import emeval.viz.eval_view as ezev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Metrics helpers\n",
    "import emeval.metrics.baseline_segmentation as embs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For plots\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.collections import PatchCollection\n",
    "from matplotlib.patches import Rectangle\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For maps\n",
    "import folium\n",
    "import branca.element as bre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For easier debugging while working on modules\n",
    "import importlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "pd.options.display.float_format = '{:.6f}'.format\n",
    "import arrow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "THIRTY_MINUTES = 30 * 60\n",
    "TIME_THRESHOLD = THIRTY_MINUTES"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The spec\n",
    "\n",
    "The spec defines what experiments were done, and over which time ranges. Once the experiment is complete, most of the structure is read back from the data, but we use the spec to validate that it all worked correctly. The spec also contains the ground truth for the legs. Here, we read the spec for the trip to UC Berkeley."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASTORE_URL = \"http://cardshark.cs.berkeley.edu\"\n",
    "AUTHOR_EMAIL = \"shankari@eecs.berkeley.edu\"\n",
    "sd_la = eisd.SpecDetails(DATASTORE_URL, AUTHOR_EMAIL, \"unimodal_trip_car_bike_mtv_la\")\n",
    "sd_sj = eisd.SpecDetails(DATASTORE_URL, AUTHOR_EMAIL, \"car_scooter_brex_san_jose\")\n",
    "sd_ucb = eisd.SpecDetails(DATASTORE_URL, AUTHOR_EMAIL, \"train_bus_ebike_mtv_ucb\")\n",
    "sd_ucb_reroute = eisd.SpecDetails(DATASTORE_URL, AUTHOR_EMAIL, \"train_bus_ebike_sm_reroute_mtv_ucb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The views\n",
    "\n",
    "There are two main views for the data - the phone view and the evaluation view. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phone view\n",
    "\n",
    "In the phone view, the phone is primary, and then there is a tree that you can traverse to get the data that you want. Traversing that tree typically involves nested for loops; here's an example of loading the phone view and traversing it. You can replace the print statements with real code. When you are ready to check this in, please move the function to one of the python modules so that we can invoke it more generally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(eipv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pv_la = eipv.PhoneView(sd_la)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv_sj = eipv.PhoneView(sd_sj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv_ucb = eipv.PhoneView(sd_ucb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv_ucb_reroute = eipv.PhoneView(sd_ucb_reroute)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of detected trips versus ground truth trips\n",
    "\n",
    "Checks to see how many spurious transitions there were"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(eapv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "av_la = eapv.create_analysed_view(pv_la, \"http://localhost:8080\", \"analysis/recreated_location\", \"analysis/cleaned_trip\", \"analysis/cleaned_section\")\n",
    "av_sj = eapv.create_analysed_view(pv_sj, \"http://localhost:8080\", \"analysis/recreated_location\", \"analysis/cleaned_trip\", \"analysis/cleaned_section\")\n",
    "av_ucb = eapv.create_analysed_view(pv_ucb, \"http://localhost:8080\", \"analysis/recreated_location\", \"analysis/cleaned_trip\", \"analysis/cleaned_section\")\n",
    "av_ucb_reroute = eapv.create_analysed_view(pv_ucb_reroute, \"http://localhost:8080\", \"analysis/recreated_location\", \"analysis/cleaned_trip\", \"analysis/cleaned_section\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start and end times mismatch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curr_run = av_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][4]\n",
    "print(curr_run.keys())\n",
    "print([sr[\"data\"][\"start_fmt_time\"] for sr in av_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"sensed_trip_ranges\"]])\n",
    "print([sr[\"data\"][\"start_fmt_time\"] for sr in curr_run[\"sensed_trip_ranges\"]])\n",
    "embs.find_matching_segments(curr_run[\"evaluation_trip_ranges\"], \"trip_id\",\n",
    "                            [sr[\"data\"] for sr in curr_run[\"sensed_trip_ranges\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[1,2,3][1:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tradeoff_entries(pv):\n",
    "    tradeoff_entry_list = []\n",
    "    for phone_os, phone_map in pv.map().items():\n",
    "        print(15 * \"=*\")\n",
    "        print(phone_os, phone_map.keys())\n",
    "        for phone_label, phone_detail_map in phone_map.items():\n",
    "            print(4 * ' ', 15 * \"-*\")\n",
    "            print(4 * ' ', phone_label, phone_detail_map.keys())\n",
    "            if \"control\" in phone_detail_map[\"role\"]:\n",
    "                print(\"Ignoring %s phone %s since they are always on\" % (phone_detail_map[\"role\"], phone_label))\n",
    "                continue\n",
    "            # this spec does not have any calibration ranges, but evaluation ranges are actually cooler\n",
    "            for r in phone_detail_map[\"evaluation_ranges\"]:\n",
    "                print(8 * ' ', 30 * \"=\")\n",
    "                print(8 * ' ',r.keys())\n",
    "                print(8 * ' ',r[\"trip_id\"], r[\"eval_common_trip_id\"], r[\"eval_role\"], len(r[\"evaluation_trip_ranges\"]))\n",
    "                bcs = r[\"battery_df\"][\"battery_level_pct\"]\n",
    "                delta_battery = bcs.iloc[0] - bcs.iloc[-1]\n",
    "                print(\"Battery starts at %d, ends at %d, drain = %d\" % (bcs.iloc[0], bcs.iloc[-1], delta_battery))\n",
    "                sensed_trips = len(r[\"sensed_trip_ranges\"])\n",
    "                matching_trip_map = embs.find_matching_segments(r[\"evaluation_trip_ranges\"], \"trip_id\", \n",
    "                                                                [tr[\"data\"] for tr in r[\"sensed_trip_ranges\"]])\n",
    "                print(matching_trip_map)\n",
    "                for trip in r[\"evaluation_trip_ranges\"]:\n",
    "                    sensed_trip_range = matching_trip_map[trip[\"trip_id\"]]\n",
    "                    trip[\"matching_trip_range\"] = sensed_trip_range\n",
    "                    results = embs.get_count_start_end_ts_diff(trip, sensed_trip_range)\n",
    "                    print(\"Got results %s\" % results)\n",
    "                    tradeoff_entry = {\"phone_os\": phone_os, \"phone_label\": phone_label,\n",
    "                                      \"timeline\": pv.spec_details.curr_spec[\"id\"],\n",
    "                                      \"range_id\": r[\"trip_id\"],\n",
    "                                     \"run\": r[\"trip_run\"], \"duration\": r[\"duration\"],\n",
    "                                     \"role\": r[\"eval_role_base\"], \"battery_drain\": delta_battery,\n",
    "                                     \"trip_count\": sensed_trips,\n",
    "                                      \"trip_id\": trip[\"trip_id\"]}\n",
    "                    tradeoff_entry.update(results)\n",
    "                    tradeoff_entry_list.append(tradeoff_entry)\n",
    "    return tradeoff_entry_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# We are not going to look at battery life at the evaluation trip level; we will end with evaluation range\n",
    "# since we want to capture the overall drain for the timeline\n",
    "tradeoff_entries_list = []\n",
    "tradeoff_entries_list.extend(get_tradeoff_entries(av_la))\n",
    "tradeoff_entries_list.extend(get_tradeoff_entries(av_sj))\n",
    "tradeoff_entries_list.extend(get_tradeoff_entries(av_ucb))\n",
    "tradeoff_entries_list.extend(get_tradeoff_entries(av_ucb_reroute))\n",
    "tradeoff_df = pd.DataFrame(tradeoff_entries_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's merge the reroutes\n",
    "tradeoff_df.timeline.replace(\"train_bus_ebike_sm_reroute_mtv_ucb\", \"train_bus_ebike_mtv_ucb\", inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r2q_map = {\"power_control\": 0, \"HAMFDC\": 1, \"MAHFDC\": 2, \"HAHFDC\": 3, \"accuracy_control\": 4}\n",
    "q2r_map = {0: \"power\", 1: \"HAMFDC\", 2: \"MAHFDC\", 3: \"HAHFDC\", 4: \"accuracy\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a number so that can get the plots to come out in order\n",
    "tradeoff_df[\"quality\"] = tradeoff_df.role.apply(lambda r: r2q_map[r])\n",
    "tradeoff_df[\"count_diff\"] = tradeoff_df[[\"count\"]] - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trip count analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scatter plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ifig, ax = plt.subplots(nrows=1, ncols=1, figsize=(12,4))\n",
    "errorboxes = []\n",
    "for key, df in tradeoff_df.query(\"phone_os == 'android'\").groupby([\"role\", \"timeline\"]):\n",
    "    print(key, df)\n",
    "    tcd = df.trip_count\n",
    "    bd = df.battery_drain\n",
    "    print(\"Plotting rect with params %s, %d, %d\" % (str((tcd.min(), bd.min())),\n",
    "                                                    tcd.max() - tcd.min(),\n",
    "                                                    bd.max() - bd.min()))\n",
    "    print(tcd.min(), tcd.max(), tcd.std())\n",
    "    xerror = np.array([[tcd.min(), tcd.max()]])\n",
    "    print(xerror.shape)\n",
    "    ax.errorbar(x=tcd.mean(), y=bd.mean(), xerr=[[tcd.min()], [tcd.max()]], yerr=[[bd.min()], [bd.max()]], label=key)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timeline + trip specific variation\n",
    "\n",
    "How many sensed trips matched to each ground truth trip?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ifig, ax_array = plt.subplots(nrows=2,ncols=3,figsize=(9,6), sharex=False, sharey=True)\n",
    "timeline_list = [\"train_bus_ebike_mtv_ucb\", \"car_scooter_brex_san_jose\", \"unimodal_trip_car_bike_mtv_la\"]\n",
    "for i, tl in enumerate(timeline_list):\n",
    "    tradeoff_df.query(\"timeline == @tl & phone_os == 'android'\").boxplot(ax = ax_array[0][i], column=[\"count_diff\"], by=[\"quality\"])\n",
    "    ax_array[0][i].set_title(tl)\n",
    "    tradeoff_df.query(\"timeline == @tl & phone_os == 'ios'\").boxplot(ax = ax_array[1][i], column=[\"count_diff\"], by=[\"quality\"])\n",
    "    ax_array[1][i].set_title(\"\")\n",
    "    # tradeoff_df.query(\"timeline == @tl & phone_os == 'ios'\").boxplot(ax = ax_array[2][i], column=[\"visit_reports\"], by=[\"quality\"])\n",
    "    # ax_array[2][i].set_title(\"\")\n",
    "\n",
    "    # print(android_ax_returned.shape, ios_ax_returned.shape)\n",
    "\n",
    "for i, ax in enumerate(ax_array[0]):\n",
    "    ax.set_xticklabels([q2r_map[int(t.get_text())] for t in ax.get_xticklabels()])\n",
    "    ax.set_xlabel(\"\")\n",
    "\n",
    "for i, ax in enumerate(ax_array[1]):\n",
    "    ax.set_xticklabels([q2r_map[int(t.get_text())] for t in ax.get_xticklabels()])\n",
    "    ax.set_xlabel(\"\")\n",
    "\n",
    "# for ax in ax_array[1]:\n",
    "#     ax.set_xticklabels(q2r_ios_list[1:])\n",
    "#     ax.set_xlabel(\"\")\n",
    "\n",
    "# for ax in ax_array[2]:\n",
    "#     ax.set_xticklabels(q2r_ios_list[1:])\n",
    "#     ax.set_xlabel(\"\")\n",
    "\n",
    "ax_array[0][0].set_ylabel(\"Difference in trip counts (android)\")\n",
    "ax_array[1][0].set_ylabel(\"Difference in trip counts (ios)\")\n",
    "# ax_array[2][0].set_ylabel(\"Difference in visit reports (ios)\")\n",
    "ifig.suptitle(\"Trip count differences v/s configured quality over multiple timelines\")\n",
    "# ifig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timeline specific variation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_count_with_errors(ax_array, phone_os):\n",
    "    for i, (tl, trip_gt) in enumerate(timeline_trip_gt.items()):\n",
    "        ax_array[i].bar(0, trip_gt)\n",
    "        for q in range(1,4):\n",
    "            curr_df = tradeoff_df.query(\"timeline == @tl & phone_os == @phone_os & quality == @q\")\n",
    "            print(\"%s %s %s values = %s %s %s\" % (phone_os, tl, q2r_map[q], curr_df.trip_count.min(), curr_df.trip_count.mean(), curr_df.trip_count.max()))\n",
    "            lower_error = curr_df.trip_count.mean() - curr_df.trip_count.min()\n",
    "            upper_error = curr_df.trip_count.max() - curr_df.trip_count.mean()\n",
    "            ax_array[i].bar(x=q, height=curr_df.trip_count.mean(),\n",
    "                            yerr=[[lower_error], [upper_error]])\n",
    "            print(\"%s %s %s errors = %s %s %s\" % (phone_os, tl, q2r_map[q], lower_error, curr_df.trip_count.mean(), upper_error))\n",
    "            ax_array[i].set_title(tl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ifig, ax_array = plt.subplots(nrows=2,ncols=3,figsize=(10,5), sharex=False, sharey=True)\n",
    "timeline_trip_gt = {\"train_bus_ebike_mtv_ucb\": 3,\n",
    "                    \"car_scooter_brex_san_jose\": 2,\n",
    "                    \"unimodal_trip_car_bike_mtv_la\": 2}\n",
    "\n",
    "plot_count_with_errors(ax_array[0], \"android\")\n",
    "plot_count_with_errors(ax_array[1], \"ios\")\n",
    "\n",
    "for ax in ax_array[0]:\n",
    "    ax.set_xticks(range(0,4))\n",
    "    ax.set_xticklabels([\"truth\"] + [q2r_map[r] for r in range(1,4)])\n",
    "    ax.set_yticks(range(0,tradeoff_df.trip_count.max(),3))\n",
    "    \n",
    "for ax in ax_array[1]:\n",
    "    ax.set_xticks(range(0,4))\n",
    "    ax.set_xticklabels([\"truth\"] + [q2r_map[r] for r in range(1,4)])\n",
    "    ax.set_yticks(range(0,tradeoff_df.trip_count.max(),3))\n",
    "    \n",
    "ax_array[0,0].set_ylabel(\"nTrips (android)\")\n",
    "ax_array[1,0].set_ylabel(\"nTrips (ios)\")\n",
    "    \n",
    "ifig.tight_layout(pad=0.85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start end results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for r, df in tradeoff_df.query(\"timeline == @tl & phone_os == 'android'\").groupby(\"role\"):\n",
    "    print(r, df.trip_count.mean() , df.trip_count.min(), df.trip_count.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HAHFDC phone ran out of battery on all three runs of the `train_bus_ebike_mtv_ucb` timeline, so the trips never ended. Let's remove those so that they don't obfuscate the values from the other runs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_of_battery_phones = tradeoff_df.query(\"timeline=='train_bus_ebike_mtv_ucb' & role=='HAHFDC' & trip_id=='berkeley_to_mtv_SF_express_bus_0' & phone_os == 'android'\")\n",
    "for i in out_of_battery_phones.index:\n",
    "    tradeoff_df.loc[i,\"end_diff_mins\"] = float('nan')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tradeoff_df.query(\"timeline=='train_bus_ebike_mtv_ucb' & role=='HAHFDC' & trip_id=='berkeley_to_mtv_SF_express_bus_0' & phone_os == 'android'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ifig, ax_array = plt.subplots(nrows=1,ncols=4,figsize=(12,3), sharex=False, sharey=True)\n",
    "tradeoff_df.query(\"phone_os == 'android'\").boxplot(ax = ax_array[0], column=[\"start_diff_mins\"], by=[\"quality\"])\n",
    "ax_array[0].set_title(\"start time (android)\")\n",
    "tradeoff_df.query(\"phone_os == 'android'\").boxplot(ax = ax_array[1], column=[\"end_diff_mins\"], by=[\"quality\"])\n",
    "ax_array[1].set_title(\"end time (android)\")\n",
    "tradeoff_df.query(\"phone_os == 'ios'\").boxplot(ax = ax_array[2], column=[\"start_diff_mins\"], by=[\"quality\"])\n",
    "ax_array[2].set_title(\"start_time (ios)\")\n",
    "tradeoff_df.query(\"phone_os == 'ios'\").boxplot(ax = ax_array[3], column=[\"end_diff_mins\"], by=[\"quality\"])\n",
    "ax_array[3].set_title(\"end_time (ios)\")\n",
    "\n",
    "    # print(android_ax_returned.shape, ios_ax_returned.shape)\n",
    "\n",
    "ax_array[0].set_xticklabels([q2r_map[int(t.get_text())] for t in ax_array[0].get_xticklabels()])\n",
    "ax_array[1].set_xticklabels([q2r_map[int(t.get_text())] for t in ax_array[1].get_xticklabels()])\n",
    "ax_array[2].set_xticklabels([q2r_map[int(t.get_text())] for t in ax_array[2].get_xticklabels()])\n",
    "ax_array[3].set_xticklabels([q2r_map[int(t.get_text())] for t in ax_array[3].get_xticklabels()])\n",
    "\n",
    "for ax in ax_array:\n",
    "    ax.set_xlabel(\"\")\n",
    "\n",
    "ax_array[1].text(0.55,25,\"Excluding trips where battery ran out\")\n",
    "\n",
    "ax_array[0].set_ylabel(\"Diff (mins)\")\n",
    "# ifig.suptitle(\"Trip start end accuracy v/s configured quality\")\n",
    "ifig.suptitle(\"\")\n",
    "# ifig.tight_layout(pad=1.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timeline specific"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ifig, ax_array = plt.subplots(nrows=4,ncols=3,figsize=(10,10), sharex=False, sharey=True)\n",
    "timeline_list = [\"train_bus_ebike_mtv_ucb\", \"car_scooter_brex_san_jose\", \"unimodal_trip_car_bike_mtv_la\"]\n",
    "for i, tl in enumerate(timeline_list):\n",
    "    tradeoff_df.query(\"timeline == @tl & phone_os == 'android'\").boxplot(ax = ax_array[0][i], column=[\"start_diff_mins\"], by=[\"quality\"])\n",
    "    ax_array[0][i].set_title(tl)\n",
    "    tradeoff_df.query(\"timeline == @tl & phone_os == 'android'\").boxplot(ax = ax_array[1][i], column=[\"end_diff_mins\"], by=[\"quality\"])\n",
    "    ax_array[1][i].set_title(\"\")\n",
    "    tradeoff_df.query(\"timeline == @tl & phone_os == 'ios'\").boxplot(ax = ax_array[2][i], column=[\"start_diff_mins\"], by=[\"quality\"])\n",
    "    ax_array[2][i].set_title(\"\")\n",
    "    tradeoff_df.query(\"timeline == @tl & phone_os == 'ios'\").boxplot(ax = ax_array[3][i], column=[\"end_diff_mins\"], by=[\"quality\"])\n",
    "    ax_array[3][i].set_title(\"\")\n",
    "\n",
    "    # print(android_ax_returned.shape, ios_ax_returned.shape)\n",
    "\n",
    "for ax in ax_array[0]:\n",
    "    ax.set_xticklabels([q2r_map[int(t.get_text())] for t in ax.get_xticklabels()])\n",
    "    ax.set_xlabel(\"\")\n",
    "    \n",
    "for ax in ax_array[1]:\n",
    "    ax.set_xticklabels([q2r_map[int(t.get_text())] for t in ax.get_xticklabels()])\n",
    "    ax.set_xlabel(\"\")\n",
    "\n",
    "ax_array[1,0].text(0.55,25,\"Excluding trips where battery ran out\")\n",
    "\n",
    "for ax in ax_array[2]:\n",
    "    ax.set_xticklabels([q2r_map[int(t.get_text())] for t in ax.get_xticklabels()])\n",
    "    ax.set_xlabel(\"\")\n",
    "\n",
    "for ax in ax_array[3]:\n",
    "    ax.set_xticklabels([q2r_map[int(t.get_text())] for t in ax.get_xticklabels()])\n",
    "    ax.set_xlabel(\"\")\n",
    "\n",
    "ax_array[0][0].set_ylabel(\"Start time diff (android)\")\n",
    "ax_array[1][0].set_ylabel(\"End time diff (android)\")\n",
    "ax_array[2][0].set_ylabel(\"Start time diff (ios)\")\n",
    "ax_array[3][0].set_ylabel(\"End time diff (ios)\")\n",
    "ifig.suptitle(\"Trip start end accuracy (mins) v/s configured quality over multiple timelines\")\n",
    "\n",
    "# ifig.tight_layout(pad=2.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outlier checks\n",
    "\n",
    "We can have unexpected values for both time and count. Unfortunately, there is no overlap between the two (intersection is zero). So we will look at a random sample from both cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected_legs = \"&\".join([\"not (trip_id == 'bus trip with e-scooter access_0' & count == 2)\",\n",
    "                            \"not (trip_id == 'mtv_to_berkeley_sf_bart_0' & count == 3)\"])\n",
    "count_outliers = tradeoff_df.query(\"count > 1 & %s\" % expected_legs)\n",
    "count_outliers[[\"phone_os\", \"range_id\", \"trip_id\", \"run\", \"role\", \"count\", \"start_diff_mins\", \"end_diff_mins\"]].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tradeoff_df.query(\"count < 1 & role == 'HAHFDC'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_outliers = tradeoff_df.query(\"start_diff_mins == 30 | end_diff_mins == 30\")\n",
    "time_outliers[[\"phone_os\", \"range_id\", \"trip_id\", \"run\", \"role\", \"start_diff_mins\", \"end_diff_mins\"]].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(len(time_outliers.index.union(count_outliers.index)), len(time_outliers.index.intersection(count_outliers.index)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_outliers.sample(n=3, random_state=1)[[\"phone_os\", \"range_id\", \"trip_id\", \"run\", \"role\", \"count\", \"start_diff_mins\", \"end_diff_mins\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_outliers.sample(n=3, random_state=1)[[\"phone_os\", \"range_id\", \"trip_id\", \"run\", \"role\", \"count\", \"start_diff_mins\", \"end_diff_mins\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tradeoff_df.query(\"timeline == 'train_bus_ebike_mtv_ucb' & quality == 3 & phone_os == 'ios'\").sort_values(by=\"start_diff_mins\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fmt = lambda ts: arrow.get(ts).to(\"America/Los_Angeles\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import IPython.display as ipyd\n",
    "\n",
    "def check_outlier(eval_range, trip_idx, mismatch_key):\n",
    "    eval_trip_range = eval_range[\"evaluation_trip_ranges\"][trip_idx]\n",
    "    print(\"Trip %s, ground truth experiment for metric %s, experiment %s -> %s, trip %s -> %s\" %\n",
    "          (eval_range[\"trip_id\"], mismatch_key,\n",
    "           fmt(eval_range[\"start_ts\"]), fmt(eval_range[\"end_ts\"]),\n",
    "           fmt(eval_trip_range[\"start_ts\"]), fmt(eval_trip_range[\"end_ts\"])))\n",
    "    print([(fmt(sr[\"start_ts\"]), fmt(sr[\"end_ts\"])) for sr in eval_trip_range[\"matching_trip_range\"][\"match\"]])\n",
    "    print(\"**** For entire experiment ***\")\n",
    "    print([(fmt(sr[\"data\"][\"start_ts\"]), fmt(sr[\"data\"][\"end_ts\"])) for sr in eval_range[\"sensed_trip_ranges\"]])\n",
    "    ipyd.display(eval_range[\"transition_df\"][[\"transition\", \"fmt_time\"]])\n",
    "    if mismatch_key == \"end_ts\":\n",
    "        # print(\"Transitions after trip end\")\n",
    "        # print(eval_range[\"transition_df\"].query(\"ts > %s\" % eval_trip_range[\"end_ts\"])[[\"transition\", \"fmt_time\"]])\n",
    "        return ezpv.display_map_detail_from_df(eval_trip_range[\"location_df\"])\n",
    "    else:\n",
    "        return ezpv.display_map_detail_from_df(eval_trip_range[\"location_df\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  MAHFDC is just terrible\n",
    "\n",
    "It looks like with MAHFDC, we essentially get no trip ends on android. Let's investigate these a bit further.\n",
    "- run 0: trip never ended: trip actually ended just before next trip started `15:01:26`. And then next trip had geofence exit, but we didn't detect it because it never ended, so we didn't create a sensed range for it.\n",
    "- run 1: trip ended but after 30 mins: similar behavior; trip ended just before next trip started `15:49:39`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tradeoff_df.query(\"phone_os == 'android' & role == 'MAHFDC' & timeline == 'car_scooter_brex_san_jose'\")[[\"range_id\", \"trip_id\", \"run\", \"role\", \"count\", \"start_diff_mins\", \"end_diff_mins\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FMT_STRING = \"HH:mm:SS\"\n",
    "for t in av_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][3][\"evaluation_trip_ranges\"]:\n",
    "    print(sd_sj.fmt(t[\"start_ts\"], FMT_STRING), \"->\", sd_sj.fmt(t[\"end_ts\"], FMT_STRING))\n",
    "av_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][3][\"transition_df\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_outlier(av_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][4], 0, \"end_ts\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_outlier(av_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][4], 1, \"end_ts\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FMT_STRING = \"HH:mm:SS\"\n",
    "for t in pv_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][4][\"evaluation_trip_ranges\"]:\n",
    "    print(sd_sj.fmt(t[\"start_ts\"], FMT_STRING), \"->\", sd_sj.fmt(t[\"end_ts\"], FMT_STRING))\n",
    "pv_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][4][\"transition_df\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### HAHFDC detection errors on the second run "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 155 \t38.000000 \t1 \t44696.648613 \t5.613607 \tucb-sdb-ios-2 \tios \tHAHFDC v/s HAMFDC:HAHFDC_1 \tHAHFDC \t1 \t12.156766 \ttrain_bus_ebike_mtv_ucb \t3 \tberkeley_to_mtv_SF_express_bus_0 \t3 \t0\n",
    "# 153 \t38.000000 \t1 \t44696.648613 \t0.287331 \tucb-sdb-ios-2 \tios \tHAHFDC v/s HAMFDC:HAHFDC_1 \tHAHFDC \t1 \t14.105514 \ttrain_bus_ebike_mtv_ucb \t3 \tmtv_to_berkeley_sf_bart_0 \t3 \t0\n",
    "    \n",
    "check_outlier(av_ucb_reroute.map()[\"ios\"][\"ucb-sdb-ios-2\"][\"evaluation_ranges\"][1], 0, \"start_ts\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_outlier(av_ucb_reroute.map()[\"ios\"][\"ucb-sdb-ios-2\"][\"evaluation_ranges\"][1], 2, \"start_ts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Visit detection kicked in almost at the end of the trip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 44 \tios \tsuburb_city_driving_weekend_0 \t1 \tHAMFDC \t0 \t30.000000 \t30.000000\n",
    "check_outlier(av_la.map()[\"ios\"][\"ucb-sdb-ios-3\"][\"evaluation_ranges\"][4], 0, \"start_ts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Trip end never detected\n",
    "\n",
    "Trip ended at 14:11, experiment ended at 14:45. No stopped_moving for the last trip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 65 \tandroid \tbus trip with e-scooter access_0 \t2 \tHAMFDC \t1 \t3.632239 \t30.000000\n",
    "check_outlier(av_sj.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][2], 1, \"end_ts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Trip end detection errors on iOS\n",
    "Original experiment, explanation for the outliers on the HAHFDC and MAHFDC first runs to San Jose\n",
    "- HAHFDC: Trip end detected 1.5 hours after real end, but before next trip start\n",
    "- MAHFDC: Trip end detected 5 hours after real end, at the end of the next trip\n",
    "- MAHFDC: Clearly this was not even detected as a separate trip, so this is correct. There was a spurious trip from `17:42:22` - `17:44:22` which ended up matching this. But clearly because of the missing trip end detection, both the previous trip and this one were incorrect. You can click on the points at the Mountain View library to confirm when the trip ended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = bre.Figure()\n",
    "fig.add_subplot(1,3,1).add_child(check_outlier(av_sj.map()[\"ios\"][\"ucb-sdb-ios-2\"][\"evaluation_ranges\"][0], 0, \"end_ts\"))\n",
    "fig.add_subplot(1,3,2).add_child(check_outlier(av_sj.map()[\"ios\"][\"ucb-sdb-ios-3\"][\"evaluation_ranges\"][0], 0, \"end_ts\"))\n",
    "fig.add_subplot(1,3,3).add_child(check_outlier(av_sj.map()[\"ios\"][\"ucb-sdb-ios-3\"][\"evaluation_ranges\"][0], 1, \"start_ts\"))\n",
    "# check_outlier(pv_sj.map()[\"ios\"][\"ucb-sdb-ios-2\"][\"evaluation_ranges\"][0], 0, \"end_ts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### No geofence exit ever detected\n",
    "\n",
    "On the middle trip of the second round of data collection to the San Jose library, we got no geofence exits. The entire list of transitions is \n",
    "\n",
    "```\n",
    "transition                          fmt_time\n",
    "3     T_VISIT_ENDED  2019-08-06T11:29:20.573817-07:00\n",
    "6   T_VISIT_STARTED  2019-08-06T11:29:20.911773-07:00\n",
    "8     T_VISIT_ENDED  2019-08-06T11:35:38.250980-07:00\n",
    "9   T_VISIT_STARTED  2019-08-06T12:00:05.445936-07:00\n",
    "12     T_TRIP_ENDED  2019-08-06T12:00:07.093790-07:00\n",
    "15    T_VISIT_ENDED  2019-08-06T15:59:13.998068-07:00\n",
    "18  T_VISIT_STARTED  2019-08-06T17:12:38.808743-07:00\n",
    "21     T_TRIP_ENDED  2019-08-06T17:12:40.504285-07:00\n",
    "```\n",
    "\n",
    "We did get visit notifications, so we did track location points (albeit after a long time), and we did get the trip end notifications, but we have no sensed trips. Had to handle this in the code as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_outlier(av_sj.map()[\"ios\"][\"ucb-sdb-ios-2\"][\"evaluation_ranges\"][4], 0, \"start_ts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### No geofence exit ever detected\n",
    "\n",
    "On the middle trip of the second round of data collection to the San Jose library, we got no geofence exits.\n",
    "We did get visit notifications, so we did track location points (albeit after a long time), and we did get the trip end notifications, but we have no sensed trips. Had to handle this in the code as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 81 \tios \tbus trip with e-scooter access_0 \t1 \tHAHFDC \t0 \t30.000000 \t30.000000\n",
    "check_outlier(av_sj.map()[\"ios\"][\"ucb-sdb-ios-2\"][\"evaluation_ranges\"][4], 1, \"end_ts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7 mapped trips for one\n",
    "\n",
    "This is essentially from the time that I wandered around looking for the bikeshare bike. This raises the question of whether I should filter out the points within the polygon in this case too. Overall, I think not. The only part within the polygon that we don't guarantee is the ground truth trajectory. We still do have the ground truth of the trip/section start end, and there really is no reason why we should have had so many \"trips\" when I was walking around. I certainly didn't wait for too long while walking and this was not semantically a \"trip\" by any stretch of the imagination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 113 \tandroid \tberkeley_to_mtv_SF_express_bus_0 \t2 \tHAMFDC \t7 \t2.528077 \t3.356611\n",
    "check_outlier(av_ucb.map()[\"android\"][\"ucb-sdb-android-3\"][\"evaluation_ranges\"][2], 2, \"end_ts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trip split into two in medium accuracy *only*\n",
    "\n",
    "Actual trip ends at `14:21`. In medium accuracy, detected trips were `14:12:15 -> 14:17:33` and  `14:22:14 -> 14:24:15`. This was after we reached the destination, but there is a large gap because we basically got no points for a large part of the trip. This seems correct - it looks like iOS is just prematurely detecting the trip end in the MA case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 127 \tios \twalk_urban_university_0 \t1 \tMAHFDC \t2 \t4.002549 \t2.352913\n",
    "fig = bre.Figure()\n",
    "\n",
    "def compare_med_high_accuracy():\n",
    "    trip_idx = 1\n",
    "    mismatch_key = \"end_ts\"\n",
    "    ha_range = av_ucb.map()[\"ios\"][\"ucb-sdb-ios-2\"][\"evaluation_ranges\"][1]\n",
    "    ha_trip_range = ha_range[\"evaluation_trip_ranges\"][trip_idx]\n",
    "    eval_range = av_ucb.map()[\"ios\"][\"ucb-sdb-ios-3\"][\"evaluation_ranges\"][1]\n",
    "    eval_trip_range = eval_range[\"evaluation_trip_ranges\"][trip_idx]\n",
    "    print(\"Trip %s, ground truth experiment for metric %s, %s, trip %s, high accuracy %s\" %\n",
    "          (eval_range[\"trip_id\"], mismatch_key,\n",
    "           fmt(eval_range[mismatch_key]), fmt(eval_trip_range[mismatch_key]), fmt(ha_trip_range[mismatch_key])))\n",
    "    print(eval_trip_range[\"transition_df\"][[\"transition\", \"fmt_time\"]])\n",
    "    print(\"**** Expanded ***\")\n",
    "    print(eval_range[\"transition_df\"].query(\"%s < ts < %s\" %\n",
    "        ((eval_trip_range[\"end_ts\"] - 30*60), (eval_trip_range[\"end_ts\"] + 30*60)))[[\"transition\", \"fmt_time\"]])\n",
    "    fig = bre.Figure()\n",
    "    fig.add_subplot(1,2,1).add_child(ezpv.display_map_detail_from_df(ha_trip_range[\"location_df\"]))\n",
    "    fig.add_subplot(1,2,2).add_child(ezpv.display_map_detail_from_df(eval_trip_range[\"location_df\"]))\n",
    "    return fig\n",
    "compare_med_high_accuracy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[{'start_ts': fmt(1564089135.368705), 'end_ts': fmt(1564089453.8783798)},\n",
    "{'start_ts': fmt(1564089734.305933), 'end_ts': fmt(1564089855.8683748)}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We just didn't detect any trip ends in the middle\n",
    "\n",
    "We only detected a trip end at the Mountain View station. This is arguably more correct than the multiple trips that we get with a dwell time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 120 \tios \tmtv_to_berkeley_sf_bart_0 \t2 \tHAHFDC \t2 \t3.175024 \t1.046759\n",
    "check_outlier(av_ucb.map()[\"ios\"][\"ucb-sdb-ios-2\"][\"evaluation_ranges\"][2], 0, \"end_ts\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}