Sprint4_Alexander_Chen_adjusted_valueerror_message.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final sprint:\n",
    "At the end of the last sprint I indicated 2 problems that happered the full workability of this worksheet. \n",
    "1) stack overflow from python / data limits from jupyter.\n",
    "2) problems with visualization.\n",
    "\n",
    "1) I belief this problem is due to the inability of python to optimize their tail recursions. this means that after a certain limit their is a stackover from the language itself.\n",
    "Combined with this I should have spread the load of the calculation over several functions instead of putting it all in one. \n",
    "2) The problem of visualization has not been clearly solved. But I belief I have found a work around. \n",
    "\n",
    "below you will find the reworked functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# importing pandas for the dataframes\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import math as math\n",
    "\n",
    "#importing sys to increase the stack limit, YOU NEED TO RUN YOUR PYTHON CODE IN ROOT (MAC) OTHERWISE YOU GET \n",
    "#'NOT ALLOWED TO RAISE MAXIMUM LIMIT' VALUEERROR MESSAGE.\n",
    "import resource\n",
    "resource.setrlimit(resource.RLIMIT_STACK, (2**29,-1))\n",
    "\n",
    "#importing the higher function reduce for a specific part of the code\n",
    "from functools import reduce\n",
    "\n",
    "\n",
    "#the proces ID to get into bigquery\n",
    "projectid = \"vast-art-167309\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "class acceleration:    \n",
    "                               \n",
    "    def bigquery_adjusted (self, vehicle):\n",
    "        \"\"\"specific function which targets the bigquery database and reformats it\"\"\"\n",
    "        car_list = []\n",
    "        #the query to bigquery\n",
    "        query_df = pd.read_gbq(\"select timestamp, value, vehicle from [headingon_params.speed_obd] where vehicle = '%s' order by timestamp asc\" %(vehicle), projectid)\n",
    "        end_row = query_df['timestamp'].count() -1\n",
    "        return self.filter_function_forloop(vehicle, end_row, query_df, car_list)\n",
    "\n",
    "    def filter_function_forloop (self,vehicle, end, dataframe, car_list):\n",
    "        \"\"\"This function filters the data which is pulled by the f(bigquery_adjusted) and filters out the blocks that will be used\n",
    "            by f(core_function_forloop) \"\"\"\n",
    "        for row in range (0, end):\n",
    "            if dataframe.iloc[row,1] == 0.0:\n",
    "                row_add = row + 1\n",
    "                if dataframe.iloc[row_add,1] != 0.0:\n",
    "                    row_2_added = row_add + 1\n",
    "                    if dataframe.iloc[row_2_added,1] != 0.0:\n",
    "                        row_3_added = row_2_added + 1\n",
    "                        if dataframe.iloc[row_3_added,1] != 0.0:\n",
    "                            self.core_function_forloop(vehicle, row, dataframe, car_list)\n",
    "                        else:\n",
    "                            pass\n",
    "                    else:\n",
    "                        pass\n",
    "                else:\n",
    "                    pass\n",
    "            else:\n",
    "                pass\n",
    "        return car_list\n",
    "            \n",
    "    def core_function_forloop(self,vehicle, row, dataframe, car_list):\n",
    "        \"\"\"this function calculates the acceleration over a 3 second window\"\"\"\n",
    "        temp_list = []\n",
    "        timestamp = dataframe.iat[row,0]\n",
    "        for i in range(1,4):\n",
    "            temp_row = row + i\n",
    "            temp_list.append(dataframe.iat[temp_row,1])\n",
    "        average_km = (reduce((lambda x, y: x+y), temp_list) / 3) \n",
    "        car_list.append({'timestamp': timestamp, 'value': average_km, 'vehicle': vehicle})\n",
    "        return\n",
    "\n",
    "###########################################################################################################################\n",
    "####This part of the class acceleration is my coded version of the Local Outlier Factor ###################################\n",
    "###########################################################################################################################\n",
    "    \n",
    "    \n",
    "    def selecting_k_distance_function(self, dataframe, k, car):\n",
    "        \"\"\"Main function under which all other function for determining the LOF factor is placed\"\"\"\n",
    "        #determines the length of the dataframe\n",
    "        row_dataframe = dataframe['timestamp'].count()\n",
    "        k_value = k + 1\n",
    "        dataframe= dataframe.round(1)\n",
    "        #counter which will be used as x coordinate\n",
    "        dataframe['counter'] = range(len(dataframe))\n",
    "        #unique id with x and y coordinates\n",
    "        dataframe['unique_id'] = list(zip(dataframe['counter'], dataframe['value']))\n",
    "        column_layout = ['unique_id']\n",
    "        column_values = []\n",
    "        observation_values = []\n",
    "        #builds the appropriate dataframe size according to the inputs given\n",
    "        for i in range(1, k_value):\n",
    "            dataframe['value_k' + str(i)] = 1000\n",
    "            column_layout.append('value_k' + str(i))\n",
    "            column_values.append('value_k' + str(i))\n",
    "            dataframe['observation_k' + str(i)] = -1\n",
    "            column_layout.append('observation_k' + str(i))\n",
    "            observation_values.append('observation_k' + str(i))\n",
    "        dataframe = dataframe[column_layout]\n",
    "        #calculating the Local reach density\n",
    "        for i in range(1, k_value):\n",
    "            self.all_columns_function(0,dataframe, i)\n",
    "        dataframe['sum for LRD'] =  dataframe[column_values].sum(axis=1)\n",
    "        dataframe['LRD'] = 1/(dataframe['sum for LRD']/k)\n",
    "        \n",
    "        observations =[]\n",
    "        # not sure to be happy or worried about this fact that i can do a for-loop and listcomprehension simultaneously\n",
    "        # reason for this choice: a list comprehension does not change data once '=' is reached, at least that is what my debugger insinuates \n",
    "        for i in dataframe[observation_values]:\n",
    "            observations.append('check_'+ i)\n",
    "            dataframe['check_'+ i] = [dataframe.loc[dataframe.loc[j,i], 'LRD'] for j in range(row_dataframe)]\n",
    "        dataframe['LOF'] = (dataframe[observations].sum(axis=1)/k)/dataframe.loc[:,'LRD']\n",
    "        \n",
    "        inliers_dataframe = dataframe[dataframe['LOF'] < 1.6]\n",
    "        outliers_dataframe = dataframe[dataframe['LOF'] > 1.5]\n",
    "        \n",
    "        inliers_dataframe['counter'], inliers_dataframe['value'] = zip(*inliers_dataframe['unique_id'])\n",
    "        inliers_dataframe.plot.scatter(x='counter', y='value', title=str(car) + \" WITHOUT outliers\")\n",
    "        \n",
    "        inliers_dataframe.hist(column='value', normed=1)\n",
    "\n",
    "    \n",
    "    def all_columns_function(self, base_row,dataframe, k):\n",
    "        \"\"\"this function goes composes all the necessary columns\"\"\"\n",
    "        row_dataframe = (dataframe['unique_id'].count() - 1) \n",
    "        if base_row > row_dataframe:\n",
    "            return\n",
    "        row_list_seq = list(range((base_row+1), (row_dataframe + 1)))\n",
    "        row_list_seq.extend(range(base_row))\n",
    "        for i in row_list_seq:\n",
    "            self.min_function(base_row,i ,row_dataframe, dataframe, k)   \n",
    "        base_row +=1\n",
    "        return self.all_columns_function(base_row,dataframe, k)       \n",
    "        \n",
    "    def min_function(self,base_row, next_row,row_dataframe,dataframe, k):\n",
    "        \"\"\"This function together with f(euclides_function) determines the lowest euclidean value and places it in the correc\n",
    "            column\"\"\"\n",
    "        if k > 1:\n",
    "            other_column_value = dataframe.loc[base_row,('value_k' + str(k-1))]\n",
    "            current_value = dataframe.loc[base_row, ('value_k' + str(k))]\n",
    "            temp_value = self.euclides_function(base_row,next_row, row_dataframe, dataframe)\n",
    "            if (temp_value < current_value and temp_value > other_column_value):\n",
    "                dataframe.loc[base_row,('value_k' + str(k))] = temp_value\n",
    "                coordinate_of_value = dataframe.iloc[next_row, 0]\n",
    "                coordinate_column = 'observation_k' + str(k)\n",
    "                dataframe.set_value(base_row,coordinate_column,coordinate_of_value[0])\n",
    "            else:\n",
    "                pass\n",
    "        else:\n",
    "            current_value = dataframe.loc[base_row, ('value_k' + str(k))]\n",
    "            temp_value = self.euclides_function(base_row,next_row, row_dataframe, dataframe)\n",
    "            if temp_value < current_value:\n",
    "                dataframe.loc[base_row,('value_k' + str(k))] = temp_value\n",
    "                coordinate_of_value = dataframe.iloc[next_row, 0]\n",
    "                coordinate_column = 'observation_k' + str(k)\n",
    "                dataframe.set_value(base_row,coordinate_column,coordinate_of_value[0])\n",
    "            else:\n",
    "                pass      \n",
    "    \n",
    "    \n",
    "      #the formula to calculate the euclidean distance between two point (as the crows fly)\n",
    "    def euclides_function (self, base_row, next_row, row_dataframe,dataframe):\n",
    "        \"\"\"calculates either the euclidean value or the straight line depending on the circumstance\"\"\"\n",
    "        tuple_base = dataframe.iloc[base_row,0]\n",
    "        tuple_next = dataframe.iloc[next_row,0]  \n",
    "        \n",
    "        if tuple_base[1] == tuple_next[1]:\n",
    "            euclide_distance = abs(tuple_base[0]-tuple_next[0])\n",
    "            return euclide_distance\n",
    "        else:\n",
    "            euclide_distance = math.sqrt(math.pow((tuple_base[0] - tuple_next[0]),2) + \n",
    "                                         math.pow((tuple_base[1] - tuple_next[1]),2))\n",
    "            return euclide_distance\n",
    "        \n",
    "        \n",
    "def everything_together (list_vehicle, k_value):\n",
    "    \"\"\"retrieving the data from the bigquery, calculating the acceleration, with the k value\"\"\"\n",
    "    for car in list_vehicle:\n",
    "        car_present = acceleration()\n",
    "        car_present.dataframeALL = pd.DataFrame(car_present.bigquery_adjusted(car))\n",
    "        car_present.dataframeALL['counter'] = range(len(car_present.dataframeALL))\n",
    "        car_present.dataframeALL.plot.scatter(x='counter', y='value', title=str(car) + \" with outliers\")\n",
    "        car_present.dataframe_validation = car_present.dataframeALL[(car_present.dataframeALL['value'] < 1)]\n",
    "        print(car_present.dataframe_validation)\n",
    "        car_present.selecting_k_distance_function(car_present.dataframeALL,k_value, car)\n",
    "    return   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### list of cars:\n",
    "The above has been individually tested and works as expected. Bellow we have composed a new list of all the names of the cars. I am using the old names since the new names seem to have some problems in their unique identification keys.\n",
    "\n",
    "also, we will use a k of 7. I have done several tests and the value of k has a significant influence on the outliers up until a point afterwhich it starts to plateau. However, as indictated in my previous sprint I have not found any clear writting on *which* k is best for the amount of observations being used. So I am going for k=7 though the program has been written in a way that you can change the k arbitrarily. Which I have done bellow for comvenience. But feel free to experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#cars_list = [ 'car2', 'car4', 'car5']\n",
    "cars_list = [ 'car1', 'car2', 'car3', 'car4', 'car5', 'car6', 'car7']\n",
    "\n",
    "#cars_list=['4T3BK3BB7AU032766', 'KMHLB8ULDU069747', 'TMBAA25J883113476', 'TSMMZC21S00134117', \n",
    "#           'WDD1760082J557012', 'WVWZZZ3CZDE025430', 'YV1MW84F1B2624570']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the calculation\n",
    "\n",
    "Bellow you find the calculation, be aware that this is all written in python dataframes. And in this case with k=3 there are at least 2\\*3 columns added to calculate the LOF for EACH car. So it will take quite some time for the whole process to finish. If you want to experiment with less intensive calculations then i suggest to change the amount of cars in the list. Also, it seems that the linux envirroment is noticeably faster, haven't look into why.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requesting query... ok.\n",
      "Job ID: job_ECmSI9F73Y6wBfqNOV9tm5A1jn1w\n",
      "Query running...\n",
      "Query done.\n",
      "Cache hit.\n",
      "\n",
      "Retrieving results...\n",
      "Got 14042 rows.\n",
      "\n",
      "Total time taken 1.27 s.\n",
      "Finished at 2017-12-02 13:30:36.\n",
      "Empty DataFrame\n",
      "Columns: [timestamp, value, vehicle, counter]\n",
      "Index: []\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Alexander_Baloche/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:89: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requesting query... ok.\n",
      "Job ID: job_1_ozfiRug843uylvTOAXc0BlBblK\n",
      "Query running...\n",
      "Query done.\n",
      "Cache hit.\n",
      "\n",
      "Retrieving results...\n",
      "  Got page: 2; 80% done. Elapsed 9.89 s.\n",
      "  Got page: 3; 100% done. Elapsed 12.79 s.\n",
      "Got 248905 rows.\n",
      "\n",
      "Total time taken 15.58 s.\n",
      "Finished at 2017-12-02 13:30:57.\n",
      "Empty DataFrame\n",
      "Columns: [timestamp, value, vehicle, counter]\n",
      "Index: []\n"
     ]
    }
   ],
   "source": [
    "everything_together(cars_list, 9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### interesting results:\n",
    "Now what is interesting is the difference between the histograms. For instance car4 of k=9 you see a classic staircase reduction. While if you look at car2 of k=9, you see that the average accelaration is much much higher (frequency tops somewhere around 12 kmh). And finally, looking at car5 of k=9 this person is almost exclusively accelarating at a much lower speed. This could indicate much more congestion driving, where you 'inch forward' most of the time. Looking at the data in this context I would think that the tires of car2 would be more 'worn' than for instance car5 which seems to go extremely cautionly. This is only looking at the data not at the state of the tires etc. \n",
    "\n",
    "The next step would be to physically measure the tire 'threading' over time and see if the parameter 'accelaration' is statistically sufficent to explain the rate of tire thread 'wearing'. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}