Skip to content

Commit

Permalink
v2.0.7.9042
Browse files Browse the repository at this point in the history
* refactor: Text edits of flags in Report_QC.rmd
* refactor: Update Report_QC.rmd in shiny rmd folder
  • Loading branch information
leppott committed Dec 23, 2024
1 parent 175952e commit 66c88ce
Show file tree
Hide file tree
Showing 7 changed files with 54 additions and 16 deletions.
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: ContDataQC
Title: Quality Control (QC) of Continous Monitoring Data
Version: 2.0.7.9040
Version: 2.0.7.9042
Authors@R: c(
person("Erik W", "Leppo", email="[email protected]",role=c("aut","cre")),
person("Ann","Roseberry Lincoln", role="ctb"),
Expand Down
17 changes: 16 additions & 1 deletion NEWS
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,25 @@ NEWS-ContDataQC

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

#> Last Update: 2024-12-23 15:40:30.505169
#> Last Update: 2024-12-23 16:25:53.162752

# Version History

## v2.0.7.9042

2024-12-23

- refactor: Text edits of flags in Report_QC.rmd
- refactor: Update Report_QC.rmd in shiny rmd folder

## v2.0.7.9041

2024-12-23

- refactor: Update thresholds display table
- DO Spike
- Add DO Adj, DO % Sat, and Salinity

## v2.0.7.9040

2024-12-23
Expand Down
17 changes: 16 additions & 1 deletion NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,25 @@ NEWS-ContDataQC

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

#> Last Update: 2024-12-23 15:40:30.505169
#> Last Update: 2024-12-23 16:25:53.162752

# Version History

## v2.0.7.9042

2024-12-23

- refactor: Text edits of flags in Report_QC.rmd
- refactor: Update Report_QC.rmd in shiny rmd folder

## v2.0.7.9041

2024-12-23

- refactor: Update thresholds display table
- DO Spike
- Add DO Adj, DO % Sat, and Salinity

## v2.0.7.9040

2024-12-23
Expand Down
6 changes: 6 additions & 0 deletions NEWS.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ cat(paste0("Last Update: ",Sys.time()))

# Version History

## v2.0.7.9042
2024-12-23

* refactor: Text edits of flags in Report_QC.rmd
* refactor: Update Report_QC.rmd in shiny rmd folder

## v2.0.7.9041
2024-12-23

Expand Down
13 changes: 7 additions & 6 deletions inst/rmd/Report_QC.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -200,12 +200,13 @@ if(boo_DEBUG==TRUE){
# FLAGS
Each data point is checked by each of the four QC tests below.

* **Gross Range Test (Gross):** Test if data point exceeds sensor or user defined min/max. The values are user defined based on parameter being measured and measuring instrument.
* **Spike Test (Spike):** Test if data point exceeds a user defined threshold relative to the previous data point. The user defined values are based on the parameter being measured.
* **Rate of Change Test (RoC):** Test if a data point exceeds a number of standard deviations from the previous data points over a user defined time period. Default is a 25 hour time period and 3 standard deviations.
* **Flat Line Test (Flat):** Test if a data point is within a user defined threshold from previous data points over a user defined range. Default is 3 previous points for suspect and 5 points for failure. The threshold is user defined and based on the measured parameter and sensitivity of the measuring instrument.
* **Unrealistic values (Gross range):** Entries are flagged fi values are above or below
upper or lower limits.
* **Spikes:** Entries are flagged if adjacent points change by more than 'x' amount.
* **Rate of Change (RoC):** Entries are flagged if the RoC exceeds a given threshold (e.g., >= 3 standard deviations within 25 hours).
* **Flat Line (Flat):** Entries are flagged if a certain number of consecutive measurements are within a certain amount of each other (e.g., > 10 consecutive temperature measurements are within 0.01. degrees C of on another).

Flags are assigned to each data point (a single measured parameter at a unique point in time) first by each of the QC tests below. Flags are assigned numerical values;
Flags are assigned to each data point (a single measured parameter at a unique point in time) first by each of the QC tests below. Flags are assigned the following categories;

```{r flags_legend}
cat(paste("* **",ContData.env$myFlagVal.Pass,"** = Pass,",sep=""))
Expand All @@ -227,7 +228,7 @@ cat(paste("* **",ContData.env$myFlagVal.Suspect,"** = no Fail and at least one S
cat("\n")
cat(paste("* **",ContData.env$myFlagVal.Fail,"** = at least one Fail,",sep=""))
cat("\n")
cat(paste("* **",ContData.env$myFlagVal.NoData,"** = all tests were Missing Data.",sep=""))
cat(paste("* **",ContData.env$myFlagVal.NoData,"** = unable to perform QC check because no adjacent value(s) to compare to, either due to missing data or because the measurement was taken at the beginning or end of the deployment period.",sep=""))
```

The thresholds used in the tests are defined in the config file and are listed below.
Expand Down
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/global.R
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ library(shinyalert)
# Sys.setenv(PATH = paste(Sys.getenv("PATH"), "C:\\Rtools\\bin", sep = ";"))

# Version Number
version <- "2.0.7.9041"
version <- "2.0.7.9042"

#Maximum individual file size that can be uploaded is 70 MB
options(shiny.maxRequestSize = 70 * 1024^2)
Expand Down
13 changes: 7 additions & 6 deletions inst/shiny-examples/ContDataQC/rmd/Report_QC.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -200,12 +200,13 @@ if(boo_DEBUG==TRUE){
# FLAGS
Each data point is checked by each of the four QC tests below.

* **Gross Range Test (Gross):** Test if data point exceeds sensor or user defined min/max. The values are user defined based on parameter being measured and measuring instrument.
* **Spike Test (Spike):** Test if data point exceeds a user defined threshold relative to the previous data point. The user defined values are based on the parameter being measured.
* **Rate of Change Test (RoC):** Test if a data point exceeds a number of standard deviations from the previous data points over a user defined time period. Default is a 25 hour time period and 3 standard deviations.
* **Flat Line Test (Flat):** Test if a data point is within a user defined threshold from previous data points over a user defined range. Default is 3 previous points for suspect and 5 points for failure. The threshold is user defined and based on the measured parameter and sensitivity of the measuring instrument.
* **Unrealistic values (Gross range):** Entries are flagged fi values are above or below
upper or lower limits.
* **Spikes:** Entries are flagged if adjacent points change by more than 'x' amount.
* **Rate of Change (RoC):** Entries are flagged if the RoC exceeds a given threshold (e.g., >= 3 standard deviations within 25 hours).
* **Flat Line (Flat):** Entries are flagged if a certain number of consecutive measurements are within a certain amount of each other (e.g., > 10 consecutive temperature measurements are within 0.01. degrees C of on another).

Flags are assigned to each data point (a single measured parameter at a unique point in time) first by each of the QC tests below. Flags are assigned numerical values;
Flags are assigned to each data point (a single measured parameter at a unique point in time) first by each of the QC tests below. Flags are assigned the following categories;

```{r flags_legend}
cat(paste("* **",ContData.env$myFlagVal.Pass,"** = Pass,",sep=""))
Expand All @@ -227,7 +228,7 @@ cat(paste("* **",ContData.env$myFlagVal.Suspect,"** = no Fail and at least one S
cat("\n")
cat(paste("* **",ContData.env$myFlagVal.Fail,"** = at least one Fail,",sep=""))
cat("\n")
cat(paste("* **",ContData.env$myFlagVal.NoData,"** = all tests were Missing Data.",sep=""))
cat(paste("* **",ContData.env$myFlagVal.NoData,"** = unable to perform QC check because no adjacent value(s) to compare to, either due to missing data or because the measurement was taken at the beginning or end of the deployment period.",sep=""))
```

The thresholds used in the tests are defined in the config file and are listed below.
Expand Down

0 comments on commit 66c88ce

Please sign in to comment.