Compare meta data between old PeMS and Modern PeMS

[mmmiah]

We need to compare meta data between old and modern pems to understand the consistency between these two platform. Lets grab one week data from INT_DIAGNOSTICS__DETECTOR_STATUS (modern PeMS ) and DETECTOR_STATUS (Old PeMS) and compare statistically and visually. Summarize the findings in this issue. It is also related #398

[mmmiah]

Analysis Topic : Spatial and Temporal Validation of Old and Moder PeMS Meta Data
Goal : Conduct the Spatial and Temporal Validation of old and Modern PeMS meta data

Data Source: Seven days daily level data was downloaded from old and modern PeMS where time windows vary from 2024-11-01 to 2024-11-07. The details are shown in below table-

Data Processing/Methodology:

Step 1: Download data from old and Modern PeMS for same time windows and count them to compare
Step 2: Match these two data from based on following conditions-
Old PeMS [Timestamp]= Modern PeMS [Timestamp]
Old PeMS [Station Id]= Modern PeMS [Station Id]
Old PeMS [Detector Id]= Modern PeMS [Detector Id]]
Old PeMS [Freeway Id]= Modern PeMS [Freeway Id]
Old PeMS [Freeway Direction]= Modern PeMS [Freeway Direction]
Old PeMS [District Id]= Modern PeMS [District Id]
Step 3: Compare the Detector Count by Sample Date, and Caltrans District
Step 4: Re-match old and Modern PeMS data set with additional condition along with all previous condition-
Old PeMS [Status]= Modern PeMS [Status]
Step 5: Compare the Good and bad Status count between these two data frame by District

Analysis and Results : It was observed that the detector counts showed no difference between the old and modern PeMS datasets for the considered time windows, indicating a perfect temporal match between the two datasets.

We also analyzed the metadata count spatially by district and found no differences between the old and modern PeMS metadata, as shown in the table below-

Finally, we compared the daily health status between the two data frames and found that 6,909 detectors had mismatched health status between the old and modern PeMS, as presented in the table below.

This inconsistency accounts for only 2.07% of the seven-day data, indicating a minor discrepancy in health status between the two data frames that needs to be addressed.

The attached is the list of 6,909 rows from the old and modern PeMS datasets. This includes a column labeled status_diff, which highlights the locations and details of these inconsistencies.

Findings :

1. It was found that old and Modern PeMS meta data have 100% matching in terms of spatial and Temporal detector count
2. However, we found that 2.07% deviation of detector status match between old and modern PeMS

Action Items : ken is working to solve the issue through #398

modern_Pems_unmatched_status_data.csv

[mmmiah]

PPT for this Analysis :
Meta Data Comparision_PPT.pptx
A ArcGIS Dashboard : https://d3giscoreage.dot.ca.gov/portal/apps/dashboards/5258ed24ffb2412f94e3dd9c5d9f671b