High Flow Values

[kengodleskidot]

When reviewing aggregations of the 30-second (raw) data to 5-minutes we have come across observed flow values that are extremely high (500+ vehicles / lane / 5-minutes). Below is an analysis of the aggregated 5-minute data for detectors in lane 1 over a one-month time period as an example:

I then reviewed the data (Raw and 5-minute) in current PeMS to see how high flow values are being handled and observed the following:

• Nearly all the stations I reviewed with high flow values were flagged as Good based on the current diagnostic tests when in fact the data being provided appeared bad
• When comparing the raw data to the 5-minute data in the current PeMS, a variety of things were happening. In the example below the 5-minute data for lane 1 used observed flow data, lane 2 used normalized flow data, lanes 3 and 4 appeared to be using imputed data, all lanes for this station returned a Good status and the % observed value of 100% gives the impression that the values are all observed which they are not:
  Observed Data
  
  5-Minute Processed Data
  
  Detector Health
  

We do not currently have a diagnostic test for mainline and HOV stations to diagnose if high flow values at the detector level should result in a Bad detector status. To help identify issues associated with excessively high flow values I will create a new diagnostic test to check the number of high flow values for mainline and HOV lanes similar to what is being done for the High Occupancy diagnostic test. We will also want to incorporate logic in the final 5-minute aggregate table that looks at the observed flow value and replaces excessively high flow values with either the imputed value or the max capacity value at the lane level.

[kengodleskidot]

@mmmiah and I will review the 'int_performance__detector_metrics_agg_five_minutes' model to see how we incorporate checking for high flow values and how to handle this scenario with imputed data.
@thehanggit will perform some statistical analysis on the flow data so we can come up with a high flow threshold to incorporate in the logic to be added to the 'int_performance__detector_metrics_agg_five_minutes' model

[jkarpen]

Marking this paused for now, work will continue after #302 is completed.

[jkarpen]

@thehanggit is going to talk to @ZhenyuZhu-Caltrans about this one, it may make more sense to move this into the project going on with the AAE team since they may want to use machine learning techniques for this.