Data Quality Daily Row Check Mart Model #416
Conversation
…ata models that should match. This data model can be used for a data quality visualization using kibana or another other tools.
kengodleskidot
requested review from
ian-r-rose,
JamesSLogan,
thehanggit and
mmmiah
| @@ -0,0 +1,67 @@ | |||
| {{ config( | |||
| materialized="table" | |||
There was a problem hiding this comment.
This is minor, but this config could be left out since models in the marts directory are already configured (in dbt_profile.yml) to already be materialized as tables.
…ons for the detector status model
@mmmiah pointed out that we should only be comparing detector counts for station types of ML and HV only since the the imputed and performance data models are for those station types only. The mart data model has been updated to reflect these station types. The updated model reflects row counts that match, great catch @mmmiah!
We will still include the good/bad/total detector count from the detector_status model for QC usage. |
There was a problem hiding this comment.
@kengodleskidot , why the all GOOD_STATUS_COUNT is zero. Does that mean all detectors are down???
There was a problem hiding this comment.
I am guessing that it will change once it will go through night job in production. Is that the case here?
The data model created in this PR can be used to create data quality visualizations/reports. The model aggregates the daily number of detectors for 4 data models that should all match. Below is an example of how two of the models are matching but subsequent models do not along with a screenshot in PeMS for comparison purposes:
I anticipate the Good and Bad counts to update as needed after the full data refresh. This should assist with #413, #397 and #398.