OpenPATH_timeline_dump_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ac9142cb",
   "metadata": {},
   "source": [
    "## Example of how to work with the location data in the timeline dump data\n",
    "\n",
    "This has been tested with the following configuration:\n",
    "\n",
    "```\n",
    "- branca=0.4.1\n",
    "- folium=0.11.0\n",
    "- ipython=7.17.0\n",
    "- jupyter=1.0.0\n",
    "- pandas=1.1.0\n",
    "```\n",
    "\n",
    "An easy approach to set all the dependencies is:\n",
    "- use the `setup/environment36.yml` and `setup/environment36.notebook.additions.yml` in https://github.com/e-mission/e-mission-server/, OR\n",
    "- set the `EMISSION_SERVER_HOME` environment variable and then run:\n",
    "   - `bash setup.sh`\n",
    "   - `source activate.sh`\n",
    "   - `./em-jupyter-notebook.sh`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e1f1afc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import folium\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed9c1173",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"folium %s\" % folium.__version__)\n",
    "print(\"json %s\" % json.__version__)\n",
    "print(\"pandas %s\" % pd.__version__)\n",
    "print(\"numpy %s\" % np.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d79bdd05",
   "metadata": {},
   "source": [
    "### Get the file from your phone\n",
    "\n",
    "Get a timeline dump by using the \"Download json dump\" option from the profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c240639e",
   "metadata": {},
   "outputs": [],
   "source": [
    "TEST_FILE = \"/tmp/2023-06-05.2023-06-05.timeline.iphone\"\n",
    "all_entries = json.load(open(TEST_FILE))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2300fda5",
   "metadata": {},
   "source": [
    "### Get raw and filtered locations\n",
    "\n",
    "Filtered locations are filtered based on the accuracy of the point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9948aa76",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_loc_df = pd.json_normalize([e for e in all_entries if e[\"metadata\"][\"key\"] == \"background/location\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3649ed8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_loc_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "294f9677",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_loc_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d687ed46",
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_loc_df = pd.json_normalize([e for e in all_entries if e[\"metadata\"][\"key\"] == \"background/filtered_location\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2299dcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_loc_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb2f79cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_loc_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b653eee",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(raw_loc_df), len(filtered_loc_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27e01832",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_loc_df['data.loc.coordinates']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb93df86",
   "metadata": {},
   "outputs": [],
   "source": [
    "geojson_linestring = {\"type\": \"LineString\", \"coordinates\": raw_loc_df['data.loc.coordinates'].to_list()}; geojson_linestring"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ff7acb",
   "metadata": {},
   "source": [
    "### Visualize them using folium\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d0230dd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "m = folium.Map()\n",
    "m.add_child(folium.GeoJson(geojson_linestring))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0108dca6",
   "metadata": {},
   "source": [
    "### Look at the time difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb865569",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_diff = raw_loc_df[\"data.ts\"].diff()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18d8b5ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_diff.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07172b71",
   "metadata": {},
   "source": [
    "### Narrow down to the location with the max density\n",
    "\n",
    "This timeline was generated on a phone with duty cycling turned on, so we did not capture any data while not in motion. So there are a few points with a large diff. Let's filter until we get to the the highest density occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6aba1ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_diff[ts_diff < 2000].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48d27b8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_diff[ts_diff < 50].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2d6e33a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_diff[ts_diff < 10].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4538dcf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_diff[ts_diff < 2].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3f3119f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_diff[np.logical_and(ts_diff < 1.25, ts_diff > 0.75)].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8f7af07",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}