diff --git a/DESCRIPTION b/DESCRIPTION index a0e8314..ff8a449 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: ContDataQC Title: Quality Control (QC) of Continous Monitoring Data -Version: 2.0.7.9031 +Version: 2.0.7.9032 Authors@R: c( person("Erik W", "Leppo", email="Erik.Leppo@tetratech.com",role=c("aut","cre")), person("Ann","Roseberry Lincoln", role="ctb"), diff --git a/NEWS b/NEWS index fd55834..c4eacb3 100644 --- a/NEWS +++ b/NEWS @@ -3,12 +3,21 @@ NEWS-ContDataQC - #> Last Update: 2024-02-27 14:17:40.109261 + #> Last Update: 2024-03-01 13:34:32.498829 # Version History ## v2.0.7.9031 +2024-03-01 + +- refactor: Update Shiny app + - typos + - indents + - replace some tables with figures + +## v2.0.7.9031 + 2024-02-27 - refactor: Update Shiny app text and tables, Issue \#156 diff --git a/NEWS.md b/NEWS.md index fd55834..c4eacb3 100644 --- a/NEWS.md +++ b/NEWS.md @@ -3,12 +3,21 @@ NEWS-ContDataQC - #> Last Update: 2024-02-27 14:17:40.109261 + #> Last Update: 2024-03-01 13:34:32.498829 # Version History ## v2.0.7.9031 +2024-03-01 + +- refactor: Update Shiny app + - typos + - indents + - replace some tables with figures + +## v2.0.7.9031 + 2024-02-27 - refactor: Update Shiny app text and tables, Issue \#156 diff --git a/NEWS.rmd b/NEWS.rmd index 85c985c..bc90815 100644 --- a/NEWS.rmd +++ b/NEWS.rmd @@ -21,6 +21,14 @@ cat(paste0("Last Update: ",Sys.time())) # Version History +## v2.0.7.9031 +2024-03-01 + +* refactor: Update Shiny app + + typos + + indents + + replace some tables with figures + ## v2.0.7.9031 2024-02-27 diff --git a/inst/shiny-examples/ContDataQC/global.R b/inst/shiny-examples/ContDataQC/global.R index 3e80838..e867d96 100644 --- a/inst/shiny-examples/ContDataQC/global.R +++ b/inst/shiny-examples/ContDataQC/global.R @@ -22,7 +22,7 @@ library(shinyjs) # Sys.setenv(PATH = paste(Sys.getenv("PATH"), "C:\\Rtools\\bin", sep = ";")) # Version Number -version <- "2.0.7.9031" +version <- "2.0.7.9032" #Maximum individual file size that can be uploaded is 70 MB options(shiny.maxRequestSize = 70 * 1024^2) diff --git a/inst/shiny-examples/ContDataQC/rmd/App_1b_TestData.rmd b/inst/shiny-examples/ContDataQC/rmd/App_1b_TestData.rmd index c3805ee..e9430f9 100644 --- a/inst/shiny-examples/ContDataQC/rmd/App_1b_TestData.rmd +++ b/inst/shiny-examples/ContDataQC/rmd/App_1b_TestData.rmd @@ -44,4 +44,4 @@ You’ll see two sets of folders. Within each folder are example input and outpu * **miniDOT_concatenate** - When data are initially downloaded from miniDOT sensors, there are separate .txt files for each day (in this example, 325 individual files). Go to the **Automated Reformat – miniDOT** tab and run the **Concatenate** function to combine them all into one file. -* **miniDOT_reformat** – run the combined file through the Reformat function. The function reformats the file so that it is ready to run through the QC report function. Next, go to the Main **Functions-Import Files**s tab, import the reformatted file, run it through the ‘QC raw data’ function, and check the flagged data (edit if needed). \ No newline at end of file +* **miniDOT_reformat** – run the combined file through the Reformat function. The function reformats the file so that it is ready to run through the QC report function. Next, go to the Main **Functions-Import Files** tab, import the reformatted file, run it through the ‘QC raw data’ function, and check the flagged data (edit if needed). diff --git a/inst/shiny-examples/ContDataQC/rmd/App_1c_FAQ.rmd b/inst/shiny-examples/ContDataQC/rmd/App_1c_FAQ.rmd index c5757e1..3c7a318 100644 --- a/inst/shiny-examples/ContDataQC/rmd/App_1c_FAQ.rmd +++ b/inst/shiny-examples/ContDataQC/rmd/App_1c_FAQ.rmd @@ -23,7 +23,7 @@ if(boo_DEBUG==TRUE){ **Failure to run:** When the Shiny app fails to run, the screen will gray out and the message 'disconnected from the server' will appear. The problem typically stems from a formatting issue with the input file, in particular the Date/Time field. Check your input file, refresh the app and try again. If you still have problems, click the 'contact us' link at the bottom of this page and request assistance. Be prepared to share a copy of your input file(s) and the error message. -**Speed:** If you have a slow internet connection, you can run ContDataQC as a Shiny app on your local computer and it will likely be faster. Running it locally requires that you have R software and the ContDataQC R package installed on your computer, which can be downloaded from GitHub GitHub +**Speed:** If you have a slow internet connection, you can run ContDataQC as a Shiny app on your local computer and it will likely be faster. Running it locally requires that you have R software and the ContDataQC R package installed on your computer, which can be downloaded from GitHub Exit EPA Website @@ -42,7 +42,7 @@ if(boo_DEBUG==TRUE){ **Usability on phones:** Mobile use of this app is possible although the screen size of a phone may make it impractical. **Data summary and visualization:** The ContDataSumViz app has more summary and visualization options. -Shiny app (beta version): https://contdataqcsumviz_containerized_stg.app.cloud.gov/ +Shiny app (beta version): https://dmap-contdataqcsumviz.app.cloud.gov/ Exit EPA Website diff --git a/inst/shiny-examples/ContDataQC/rmd/App_1d_Tips.rmd b/inst/shiny-examples/ContDataQC/rmd/App_1d_Tips.rmd index dba23b7..168d38b 100644 --- a/inst/shiny-examples/ContDataQC/rmd/App_1d_Tips.rmd +++ b/inst/shiny-examples/ContDataQC/rmd/App_1d_Tips.rmd @@ -24,31 +24,46 @@ Below are tips and links to resources from the Regional Monitoring Network (RMN) ## Site visits It helps speed up the Quality Control (QC) process if you follow a checklist during site visits and document anything that might affect the quality of the data (e.g., sensor out of water or buried in sediment, beaver activity, low battery). + * **Resource:** [Site visit checklist](SiteVisitChecklist.zip) [ZIP] ## Sensor configuration Some people have had problems with air and water sensors being out of sync ( -e.g., one records at 11:00 and the other records at 11:07). If you are deploying air and water sensors at a site, make sure you configure them so that they are recording at the same time. This will make data processing faster and easier. +e.g., one records at 11:00 and the other records at 11:07). If you are deploying +air and water sensors at a site, make sure you configure them so that they are +recording at the same time. This will make data processing faster and easier. + * **Resource:** [Onset HOBO configuration tips](HOBO_ConfigLaunch_20170803.pdf) [PDF] Another issue that sometimes occurs is data overlap (where more than one file has measurements covering part of the same time periods). Make sure you clear the sensor's memory when you download data and relaunch sensors to avoid overlapping data. + * **Resource:** [Onset HOBO download and relaunch tips](HOBO_DataDownload_20170823.pdf) [PDF] ## QC workflow -There are generally two scenarios: 1) users QC their data after each download and work with one file per site at a time; or 2) users have a backlog of data that cover multiple deployment periods that they aggregate into one file before performing QC. +There are generally two scenarios: 1) users QC their data after each download +and work with one file per site at a time; or 2) users have a backlog of data +that cover multiple deployment periods that they aggregate into one file before +performing QC. + * **Resources:** Suggested workflows for each scenario: + [Single file](Workflow_QC_report_20220824.pdf) [PDF] + [Multiple files](Workflow_DataPileup_20220824.pdf) [PDF] ## Data edits -When reviewing the QC reports, it is important to be consistent in how you handle flagged data. Here is an example of an approach - +When reviewing the QC reports, it is important to be consistent in how you +handle flagged data. Here is an example of an approach - -* If you are certain a data point is erroneous, delete the measurement and flag as ‘F’. If you are not sure, flag the data as ‘S’ and do not delete. Let the people using the data decide whether to remove questionable measurements from their analysis. +* If you are certain a data point is erroneous, delete the measurement and flag +as ‘F’. If you are not sure, flag the data as ‘S’ and do not delete. Let the +people using the data decide whether to remove questionable measurements from +their analysis. + +* Document that you checked each point flagged as ‘F’ and ‘S’ by adding a note +or data qualifier to the Comment column. -* Document that you checked each point flagged as ‘F’ and ‘S’ by adding a note or data qualifier to the Comment column. * **Resource:** [List of example data qualifiers](DataQualifiers_20220210.xlsx) [XLSX] * Leave missing data cells as is (vs. deleting) @@ -56,20 +71,30 @@ When reviewing the QC reports, it is important to be consistent in how you handl ## Accuracy checks -Accuracy checks are comparisons of discrete or in situ measurements taken in the lab and/or in the field with sensor measurements from the closest date/time. The difference between the sensor and discrete measurements should be within the accuracy quoted by the manufacturer (e.g., ±0.2°C if you are using the Onset HOBO proV2 sensor). +Accuracy checks are comparisons of discrete or in situ measurements taken in the +lab and/or in the field with sensor measurements from the closest date/time. The +difference between the sensor and discrete measurements should be within the +accuracy quoted by the manufacturer (e.g., ±0.2°C if you are using the Onset +HOBO proV2 sensor). + * **Resource:** [Example accuracy check worksheet](EXAMPLE_AccuracyCheckWkst.xlsx) [XLSX] ## Visual checks of time series plots -Visual checks of time series plots are an important part of the QC process. Some issues, such as dewatering, sediment burial, ice cover and beaver activity, show fairly common patterns, and we've been compiling examples of these patterns to help people recognize potential QC issues with their data. +Visual checks of time series plots are an important part of the QC process. Some issues, such as dewatering, sediment burial, ice cover and beaver activity, show fairly common patterns, and we've been compiling examples of these patterns to +help people recognize potential QC issues with their data. + * **Resource:** [Visual checks](PlotQC_WatchList_20220824.pdf) [PDF] ## Checking sensor data against other data sources -Some partners have been downloading data from nearby weather stations and USGS gages, as well as modeled air temperature and precipitation data from sources such as Daymet (https://daymet.ornl.gov/getdata +Some partners have been downloading data from nearby weather stations and USGS +gages, as well as modeled air temperature and precipitation data from sources +such as Daymet (https://daymet.ornl.gov/getdata Exit EPA Website ), and comparing those data to sensor measurements as part of their QC process. + * **Resource:** [Daymet or USGS gage check](Daymet_Wx_Gage.zip) [ZIP] diff --git a/inst/shiny-examples/ContDataQC/rmd/App_1f_RelatedApps.rmd b/inst/shiny-examples/ContDataQC/rmd/App_1f_RelatedApps.rmd index bca931c..0b7195f 100644 --- a/inst/shiny-examples/ContDataQC/rmd/App_1f_RelatedApps.rmd +++ b/inst/shiny-examples/ContDataQC/rmd/App_1f_RelatedApps.rmd @@ -27,7 +27,7 @@ Below are links to some additional Shiny apps and/or R code that may also be use ### ContDataSumViz, for summarizing and visualizing QC’d continuous sensor data -Shiny app (beta version): https://contdataqcsumviz_containerized_stg.app.cloud.gov/ +Shiny app (beta version): https://dmap-contdataqcsumviz.app.cloud.gov/ Exit EPA Website @@ -65,7 +65,7 @@ Shiny app: -GitHub: https://github.com/leppott/LakeMonitoR/ +GitHub: https://github.com/leppott/LakeMonitoR/ Exit EPA Website diff --git a/inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd b/inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd index d8141df..ec11fa7 100644 --- a/inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd +++ b/inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd @@ -85,49 +85,12 @@ Running the HOBOware reformat function on a HOBO U20 water level logger file. **Before** -```{r} -library(readxl) -library(knitr) -library(kableExtra) -# state directories -table.dir <- "tables" -table.file <- "HOBOreformat_BEFORE_20240226.xlsx" -table.sheet <- "Sheet1" - -table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet - , na = c("NA", ""), trim_ws = TRUE, skip = 0 - , col_names = TRUE) -# Munge -## remove extra colnames -colnames(table)[2:6] <- "" - -options(knitr.kable.NA = '') -# kable(table1) -table %>% - kbl() %>% - kable_styling(full_width = F, position = "left") - -cat("\n\n") +```{r, fig.alt="HOBO U-20 file before running the reformat function"} +knitr::include_graphics("RMD_Images/HOBOreformat_BEFORE.jpg") ``` **After** -```{r} -library(readxl) -library(knitr) -library(kableExtra) -# state directories -table.dir <- "tables" -table.file <- "HOBOreformat_AFTER_20240226.xlsx" -table.sheet <- "Sheet1" - -table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet - , na = c("NA", ""), trim_ws = TRUE, skip = 0 - , col_names = TRUE) - -options(knitr.kable.NA = '') -# kable(table1) -table %>% - kbl() %>% - kable_styling(full_width = F, position = "left") +```{r, fig.alt="HOBO U-20 file after running the reformat function."} +knitr::include_graphics("RMD_Images/HOBOreformat_AFTER.jpg") ``` diff --git a/inst/shiny-examples/ContDataQC/rmd/App_2a2_miniDOT1.Rmd b/inst/shiny-examples/ContDataQC/rmd/App_2a2_miniDOT1.Rmd index 603acc4..8360a48 100644 --- a/inst/shiny-examples/ContDataQC/rmd/App_2a2_miniDOT1.Rmd +++ b/inst/shiny-examples/ContDataQC/rmd/App_2a2_miniDOT1.Rmd @@ -33,24 +33,8 @@ When data are initially downloaded from miniDOT sensors, there are separate Below is an example of what the output file looks like after running the **Concatenate** function. -```{r} -library(readxl) -library(knitr) -library(kableExtra) -# state directories -table.dir <- "tables" -table.file <- "miniDOT_After_Concatenate_20240226.xlsx" -table.sheet <- "Sheet1" - -table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet - , na = c("NA", ""), trim_ws = TRUE, skip = 0 - , col_names = TRUE) - -options(knitr.kable.NA = '') -# kable(table1) -table %>% - kbl() %>% - kable_styling(full_width = F, position = "left") +```{r, fig.alt="miniDOT DO file after running the Concatenate function."} +knitr::include_graphics("RMD_Images/miniDOT_AFTER_Concatenate.jpg") ``` ## Additional Notes diff --git a/inst/shiny-examples/ContDataQC/rmd/App_2a3_miniDOT2.Rmd b/inst/shiny-examples/ContDataQC/rmd/App_2a3_miniDOT2.Rmd index 2076142..39b87f1 100644 --- a/inst/shiny-examples/ContDataQC/rmd/App_2a3_miniDOT2.Rmd +++ b/inst/shiny-examples/ContDataQC/rmd/App_2a3_miniDOT2.Rmd @@ -52,48 +52,15 @@ Below is an example of what files look like before and after running the **Refor **Before** -```{r} -library(readxl) -library(knitr) -library(kableExtra) -# state directories -table.dir <- "tables" -table.file <- "miniDOTreformat_ BEFORE_20240226.xlsx" -table.sheet <- "Sheet1" - -table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet - , na = c("NA", ""), trim_ws = TRUE, skip = 0 - , col_names = TRUE) - -options(knitr.kable.NA = '') -# kable(table1) -table %>% - kbl() %>% - kable_styling(full_width = F, position = "left") - -cat("\n\n") + +```{r, fig.alt="miniDOT DO file before running the reformat function."} +knitr::include_graphics("RMD_Images/miniDOT_Before_Reformat.jpg") ``` **After** -```{r} -library(readxl) -library(knitr) -library(kableExtra) -# state directories -table.dir <- "tables" -table.file <- "miniDOTreformat_ AFTER_20240226.xlsx" -table.sheet <- "Sheet1" - -table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet - , na = c("NA", ""), trim_ws = TRUE, skip = 0 - , col_names = TRUE) - -options(knitr.kable.NA = '') -# kable(table1) -table %>% - kbl() %>% - kable_styling(full_width = F, position = "left") +```{r, fig.alt="miniDOT DO file after running the reformat function."} +knitr::include_graphics("RMD_Images/miniDOT_After_Reformat.jpg") ``` ## Additional Notes diff --git a/inst/shiny-examples/ContDataQC/rmd/App_3c3_QCThresh_Eval.rmd b/inst/shiny-examples/ContDataQC/rmd/App_3c3_QCThresh_Eval.rmd index eefc12c..a21be2e 100644 --- a/inst/shiny-examples/ContDataQC/rmd/App_3c3_QCThresh_Eval.rmd +++ b/inst/shiny-examples/ContDataQC/rmd/App_3c3_QCThresh_Eval.rmd @@ -22,6 +22,9 @@ if(boo_DEBUG==TRUE){ If you have one or more years of continuous data for a site, we encourage you to evaluate the performance of the QC test thresholds for each parameter at that site and customize the configuration file if needed. Make sure you consider what units you are using, since units have a large effect on thresholds. **Resources for evaluating thresholds:** + * [Pivot tables and charts](EvaluateThresholds.zip) [ZIP] for evaluating the Unrealistic values ('Gross range') and Spike tests in Excel. + * [R code statistics](TimMartin_R_ThresholdEval.zip) [ZIP] that can help inform thresholds for all four QC tests. -* [Threshold evaluation worksheet](ThresholdsCheckWorksheet_20220826.xlsx) [XLSX] that lists the default thresholds for each parameter and has columns where you enter customized thresholds for one or multiple sites. \ No newline at end of file + +* [Threshold evaluation worksheet](ThresholdsCheckWorksheet_20220826.xlsx) [XLSX] that lists the default thresholds for each parameter and has columns where you enter customized thresholds for one or multiple sites. diff --git a/inst/shiny-examples/ContDataQC/rmd/RMD_Images/HOBOreformat_AFTER.jpg b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/HOBOreformat_AFTER.jpg new file mode 100644 index 0000000..2bff662 Binary files /dev/null and b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/HOBOreformat_AFTER.jpg differ diff --git a/inst/shiny-examples/ContDataQC/rmd/RMD_Images/HOBOreformat_BEFORE.jpg b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/HOBOreformat_BEFORE.jpg new file mode 100644 index 0000000..4418e0e Binary files /dev/null and b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/HOBOreformat_BEFORE.jpg differ diff --git a/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_AFTER_Concatenate.jpg b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_AFTER_Concatenate.jpg new file mode 100644 index 0000000..9abd709 Binary files /dev/null and b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_AFTER_Concatenate.jpg differ diff --git a/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_After_Reformat.jpg b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_After_Reformat.jpg new file mode 100644 index 0000000..23b8f99 Binary files /dev/null and b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_After_Reformat.jpg differ diff --git a/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_Before_Reformat.jpg b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_Before_Reformat.jpg new file mode 100644 index 0000000..9abd709 Binary files /dev/null and b/inst/shiny-examples/ContDataQC/rmd/RMD_Images/miniDOT_Before_Reformat.jpg differ diff --git a/inst/shiny-examples/ContDataQC/rmd/tables/App_Information_Tables.xlsx b/inst/shiny-examples/ContDataQC/rmd/tables/App_Information_Tables.xlsx index da9acda..4e8aac3 100644 Binary files a/inst/shiny-examples/ContDataQC/rmd/tables/App_Information_Tables.xlsx and b/inst/shiny-examples/ContDataQC/rmd/tables/App_Information_Tables.xlsx differ diff --git a/inst/shiny-examples/ContDataQC/rmd/tables/HOBOreformat_AFTER_20240226.xlsx b/inst/shiny-examples/ContDataQC/rmd/tables/HOBOreformat_AFTER_20240226.xlsx index f2f922b..62b0137 100644 Binary files a/inst/shiny-examples/ContDataQC/rmd/tables/HOBOreformat_AFTER_20240226.xlsx and b/inst/shiny-examples/ContDataQC/rmd/tables/HOBOreformat_AFTER_20240226.xlsx differ diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1b_TestData.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1b_TestData.html index f725ece..5c5584b 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1b_TestData.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1b_TestData.html @@ -55,7 +55,7 @@

miniDOT DO Sensors

  • miniDOT_reformat – run the combined file through the Reformat function. The function reformats the file so that it is ready to run through the QC report function. Next, go to the Main -Functions-Import Filess tab, import the reformatted +Functions-Import Files tab, import the reformatted file, run it through the ‘QC raw data’ function, and check the flagged data (edit if needed).

    • diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1c_FAQ.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1c_FAQ.html index 502119e..c36ef32 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1c_FAQ.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1c_FAQ.html @@ -18,7 +18,7 @@

    Basic Information

    can run ContDataQC as a Shiny app on your local computer and it will likely be faster. Running it locally requires that you have R software and the ContDataQC R package installed on your computer, which can be -downloaded from GitHub +downloaded from     GitHub Exit EPA Website .

    Internet browsers: This Shiny app has been tested @@ -49,7 +49,7 @@

    Basic Information

    Data summary and visualization: The ContDataSumViz app has more summary and visualization options. Shiny app (beta version): -https://contdataqcsumviz_containerized_stg.app.cloud.gov/ Exit EPA +https://dmap-contdataqcsumviz.app.cloud.gov/ Exit EPA Website

    R code: https://github.com/USEPA/dmap-ContDataQCSumViz Exit EPA diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1d_Tips.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1d_Tips.html index 0cdf8cb..b8493e9 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1d_Tips.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1d_Tips.html @@ -13,7 +13,11 @@

    Site visits

    It helps speed up the Quality Control (QC) process if you follow a checklist during site visits and document anything that might affect the quality of the data (e.g., sensor out of water or buried in sediment, -beaver activity, low battery). * Resource: Site visit checklist [ZIP]

    +beaver activity, low battery).

    +

    Sensor configuration

    @@ -21,21 +25,34 @@

    Sensor configuration

    sync ( e.g., one records at 11:00 and the other records at 11:07). If you are deploying air and water sensors at a site, make sure you configure them so that they are recording at the same time. This will -make data processing faster and easier. * Resource: Onset HOBO configuration tips -[PDF]

    +make data processing faster and easier.

    +

    Another issue that sometimes occurs is data overlap (where more than one file has measurements covering part of the same time periods). Make sure you clear the sensor’s memory when you download data and relaunch -sensors to avoid overlapping data. * Resource: Onset HOBO download and relaunch -tips [PDF]

    +sensors to avoid overlapping data.

    +

    QC workflow

    There are generally two scenarios: 1) users QC their data after each download and work with one file per site at a time; or 2) users have a backlog of data that cover multiple deployment periods that they -aggregate into one file before performing QC. * -Resources: Suggested workflows for each scenario: + Single file [PDF] + Multiple files [PDF]

    +aggregate into one file before performing QC.

    +

    Data edits

    @@ -59,9 +76,11 @@

    Accuracy checks

    taken in the lab and/or in the field with sensor measurements from the closest date/time. The difference between the sensor and discrete measurements should be within the accuracy quoted by the manufacturer -(e.g., ±0.2°C if you are using the Onset HOBO proV2 sensor). * -Resource: Example accuracy check -worksheet [XLSX]

    +(e.g., ±0.2°C if you are using the Onset HOBO proV2 sensor).

    +

    Visual checks of time series plots

    @@ -69,7 +88,10 @@

    Visual checks of time series plots

    process. Some issues, such as dewatering, sediment burial, ice cover and beaver activity, show fairly common patterns, and we’ve been compiling examples of these patterns to help people recognize potential QC issues -with their data. * Resource: Visual checks [PDF]

    +with their data.

    +

    Checking sensor data against other data sources

    @@ -78,7 +100,10 @@

    Checking sensor data against other data sources

    data from sources such as Daymet (https://daymet.ornl.gov/getdata Exit EPA Website ), and comparing those data to sensor -measurements as part of their QC process. * Resource: -Daymet or USGS gage check [ZIP]

    +measurements as part of their QC process.

    +
    diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1f_RelatedApps.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1f_RelatedApps.html index 14a23bf..4e3fb35 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1f_RelatedApps.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1f_RelatedApps.html @@ -12,7 +12,7 @@

    Related Apps

    ContDataSumViz, for summarizing and visualizing QC’d continuous sensor data

    Shiny app (beta version): -https://contdataqcsumviz_containerized_stg.app.cloud.gov/ Exit EPA +https://dmap-contdataqcsumviz.app.cloud.gov/ Exit EPA Website

    R code: https://github.com/USEPA/dmap-ContDataQCSumViz Exit EPA @@ -38,8 +38,8 @@

    LakeMonitoR, for summarizing and visualizing QC’d continuous

    Shiny app: https://nalms.shinyapps.io/LakeMonitoR/ Exit EPA Website

    -

    GitHub: <a class=“menu__link” https://github.com/leppott/LakeMonitoR/” -target=“_blank”>https://github.com/leppott/LakeMonitoR/ Exit EPA +

    GitHub: +https://github.com/leppott/LakeMonitoR/ Exit EPA Website

    diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a1_HOBO.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a1_HOBO.html index 8720f0c..9bca9d5 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a1_HOBO.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a1_HOBO.html @@ -91,327 +91,8 @@

    Example

    Running the HOBOware reformat function on a HOBO U20 water level logger file.

    Before

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -Plot Title: B-009-7 Vaughn Crk - - - - - -
    -Date Time, GMT-05:00 - -Water BP, psi (LGR S/N: 10246120) - -Water Temp, °C (LGR S/N: 10246120) - -Air BP, psi (LGR S/N: 10246124) - -Water Level, feet (LGR S/N: 10246120) - -AirTemp, °C (LGR S/N: 10246124) -
    -42068 - -14.5947 - -11.916 - -14.1206 - -1.02 - -18.331 -
    -42068.020833333336 - -14.597200000000001 - -11.916 - -14.118 - -1.032 - -16.902999999999999 -
    -42068.041666666664 - -14.5928 - -12.013 - -14.112500000000001 - -1.034 - -16.713000000000001 -
    -42068.0625 - -14.5928 - -12.013 - -14.1083 - -1.044 - -16.902999999999999 -
    -42068.083333333336 - -14.5908 - -12.11 - -14.113200000000001 - -1.028 - -16.902999999999999 -
    +

    HOBO U-20 file before running the reformat function

    After

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -SiteID - -Date.Time - -Air.Temp.C - -Air.LoggerID - -Air.RowID - -Water.Temp.C - -Water.LoggerID - -Water.RowID - -Water.Level.ft -
    -B0997 - -2015-03-05 00:00:00 - -18.331 - -10246124 - -1 - -11.916 - -10246120 - -1 - -1.020 -
    -B0997 - -2015-03-05 00:30:00 - -16.903 - -10246124 - -2 - -11.916 - -10246120 - -2 - -1.032 -
    -B0997 - -2015-03-05 01:00:00 - -16.713 - -10246124 - -3 - -12.013 - -10246120 - -3 - -1.034 -
    -B0997 - -2015-03-05 01:30:00 - -16.903 - -10246124 - -4 - -12.013 - -10246120 - -4 - -1.044 -
    -B0997 - -2015-03-05 02:00:00 - -16.903 - -10246124 - -5 - -12.110 - -10246120 - -5 - -1.028 -
    +

    HOBO U-20 file after running the reformat function.

    diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a2_miniDOT1.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a2_miniDOT1.html index 849f1fe..c60bf3f 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a2_miniDOT1.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a2_miniDOT1.html @@ -34,210 +34,7 @@

    Test files

    miniDOT test data [ZIP]

    Below is an example of what the output file looks like after running the Concatenate function.

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -Unix.Timestamp - -UTC_Date_._Time - -Local.Time - -Battery - -Temperature - -Dissolved.Oxygen - -Dissolved.Oxygen.Saturation - -Q - -serialnum -
    -(Second) - -(none) - -(none) - -(Volt) - -(deg C) - -(mg/l) - -(%) - -(none) - -(none) -
    -1575496440 - -43803.912499999999 - -43803.912499999999 - -3.44 - -17.667000000000002 - -8.7409999999999997 - - -1.0049999999999999 - -7392-354869 -
    -1575498240 - -43803.933333333334 - -43803.933333333334 - -3.44 - -17.878 - -8.7059999999999995 - - -1.006 - -7392-354869 -
    -1575500040 - -43803.95416666667 - -43803.95416666667 - -3.44 - -18.155999999999999 - -8.6170000000000009 - - -1.0049999999999999 - -7392-354869 -
    -1575501840 - -43803.974999999999 - -43803.974999999999 - -3.44 - -18.442 - -8.5749999999999993 - - -1.006 - -7392-354869 -
    -1575503640 - -43803.995833333334 - -43803.995833333334 - -3.45 - -18.643999999999998 - -8.5559999999999992 - - -1.006 - -7392-354869 -
    +

    miniDOT DO file after running the Concatenate function.

    Additional Notes

    diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a3_miniDOT2.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a3_miniDOT2.html index 7eb7874..cfd9a21 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a3_miniDOT2.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a3_miniDOT2.html @@ -34,337 +34,9 @@

    Test files

    Below is an example of what files look like before and after running the Reformat function.

    Before

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -Unix.Timestamp - -UTC_Date_._Time - -Local.Time - -Battery - -Temperature - -Dissolved.Oxygen - -Dissolved.Oxygen.Saturation - -Q - -serialnum -
    -(Second) - -(none) - -(none) - -(Volt) - -(deg C) - -(mg/l) - -(%) - -(none) - -(none) -
    -1575496440 - -43803.912499999999 - -43803.912499999999 - -3.44 - -17.667000000000002 - -8.7409999999999997 - - -1.0049999999999999 - -7392-354869 -
    -1575498240 - -43803.933333333334 - -43803.933333333334 - -3.44 - -17.878 - -8.7059999999999995 - - -1.006 - -7392-354869 -
    -1575500040 - -43803.95416666667 - -43803.95416666667 - -3.44 - -18.155999999999999 - -8.6170000000000009 - - -1.0049999999999999 - -7392-354869 -
    -1575501840 - -43803.974999999999 - -43803.974999999999 - -3.44 - -18.442 - -8.5749999999999993 - - -1.006 - -7392-354869 -
    -1575503640 - -43803.995833333334 - -43803.995833333334 - -3.45 - -18.643999999999998 - -8.5559999999999992 - - -1.006 - -7392-354869 -
    +

    miniDOT DO file before running the reformat function.

    After

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -Date.Time - -Battery - -Water.Temp.C - -DO.mg.L - -Q - -SiteID -
    -2019-12-04 21:54:00 - -3.44 - -17.667 - -8.741 - -1.005 - -7392-354869 -
    -2019-12-04 22:24:00 - -3.44 - -17.878 - -8.706 - -1.006 - -7392-354869 -
    -2019-12-04 22:54:00 - -3.44 - -18.156 - -8.617 - -1.005 - -7392-354869 -
    -2019-12-04 23:24:00 - -3.44 - -18.442 - -8.575 - -1.006 - -7392-354869 -
    -2019-12-04 23:54:00 - -3.45 - -18.644 - -8.556 - -1.006 - -7392-354869 -
    +

    miniDOT DO file after running the reformat function.

    Additional Notes

    diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2b_OrganizingFiles.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2b_OrganizingFiles.html index 013c438..429488f 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2b_OrganizingFiles.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2b_OrganizingFiles.html @@ -165,7 +165,7 @@

    Folder descriptions

    If there is a nearby, representative weather station, we recommend comparing your sensor data to the weather station data as an additional QC check. Modeled daily air temperature and precipitation data are also -available from Daymet (https://daymet.ornl.gov/) and PRISM (https://prism.oregonstate.edu/). +available from Daymet and PRISM. diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_3c3_QCThresh_Eval.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_3c3_QCThresh_Eval.html index 9e0eaa9..1e7a719 100644 --- a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_3c3_QCThresh_Eval.html +++ b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_3c3_QCThresh_Eval.html @@ -8,11 +8,16 @@

    Evaluate Thresholds

    each parameter at that site and customize the configuration file if needed. Make sure you consider what units you are using, since units have a large effect on thresholds.

    -

    Resources for evaluating thresholds: * Pivot tables and charts [ZIP] for -evaluating the Unrealistic values (‘Gross range’) and Spike tests in -Excel. * R code statistics -[ZIP] that can help inform thresholds for all four QC tests. * Threshold evaluation -worksheet [XLSX] that lists the default thresholds for each -parameter and has columns where you enter customized thresholds for one -or multiple sites.

    +

    Resources for evaluating thresholds:

    +