Skip to content

Commit

Permalink
v2.0.7.9026
Browse files Browse the repository at this point in the history
- refactor: Move version number from title to about, Issue \#161
- refactor: Add file identifiers to download links, Issue \#160
  • Loading branch information
leppott committed Jan 31, 2024
1 parent 1dd6106 commit 8dc679e
Show file tree
Hide file tree
Showing 46 changed files with 715 additions and 650 deletions.
4 changes: 2 additions & 2 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: ContDataQC
Title: Quality Control (QC) of Continous Monitoring Data
Version: 2.0.7.9025
Version: 2.0.7.9026
Authors@R: c(
person("Erik W", "Leppo", email="[email protected]",role=c("aut","cre")),
person("Ann","Roseberry Lincoln", role="ctb"),
Expand Down Expand Up @@ -42,5 +42,5 @@ Remotes:
tsangyp/StreamThermal,
jasonelaw/iha
StagedInstall: no
RoxygenNote: 7.3.0
RoxygenNote: 7.3.1
Config/testthat/edition: 3
9 changes: 8 additions & 1 deletion NEWS
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,17 @@ NEWS-ContDataQC

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

#> Last Update: 2024-01-25 12:04:12.540163
#> Last Update: 2024-01-31 14:12:04.111378

# Version History

## v2.0.7.9026

2024-01-31

- refactor: Move version number from title to about, Issue \#161
- refactor: Add file identifiers to download links, Issue \#160

## v2.0.7.9025

2024-01-25
Expand Down
9 changes: 8 additions & 1 deletion NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,17 @@ NEWS-ContDataQC

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

#> Last Update: 2024-01-25 12:04:12.540163
#> Last Update: 2024-01-31 14:12:04.111378

# Version History

## v2.0.7.9026

2024-01-31

- refactor: Move version number from title to about, Issue \#161
- refactor: Add file identifiers to download links, Issue \#160

## v2.0.7.9025

2024-01-25
Expand Down
6 changes: 6 additions & 0 deletions NEWS.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ cat(paste0("Last Update: ",Sys.time()))

# Version History

## v2.0.7.9026
2024-01-31

* refactor: Move version number from title to about, Issue #161
* refactor: Add file identifiers to download links, Issue #160

## v2.0.7.9025
2024-01-25

Expand Down
78 changes: 58 additions & 20 deletions data-raw/ProcessRMD_Gibbs_ShinyHTML.R
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,68 @@
# 2022-09-16
#~~~~~~~~~~~~~~

# AUTO, RMD 2 HTML ----
# EWL, 2024-01-31
# Render RMD to HTML and move to www

# Packages
# libary(rmarkdown)

# Files
myFiles <- list.files(path = "inst/shiny-examples/ContDataQC/rmd"
, pattern = "^App_"
, full.names = TRUE)

# Loop over files


# Render as HTML
path_shiny_www <- file.path("inst"
, "shiny-examples"
, "ContDataQC"
, "www"
, "RMD_HTML")


for (i in myFiles) {
# file name w/o extension
#i_fn <- tools::file_path_sans_ext(basename(i))
# save to HTML
rmarkdown::render(input = i
, output_dir = path_shiny_www)
}## FOR ~ i

shell.exec(normalizePath(path_shiny_www))


# MANUAL, Move Knitted HTML ----
# Knit to HTML manually
# Move those HTML files to www

path_rmd <- file.path("inst", "shiny-examples", "ContDataQC", "rmd")
path_shiny_www <- file.path("inst", "shiny-examples", "ContDataQC", "www", "RMD_HTML")

# open folder
shell.exec(normalizePath(path_rmd))

# Copy
myFile <- list.files(path = path_rmd
, pattern = "^App_.+\\.html$"
, full.names = TRUE)
file.copy(myFile
, file.path(path_shiny_www, basename(myFile))
, overwrite = TRUE)
# Delete
unlink(myFile)




# OLD code ----
#
# Packages
# libary(rmarkdown)

# # Files
# path_rmd <- "data-raw/rmd"
# myFiles <- list.files(path = path_rmd
Expand All @@ -33,23 +91,3 @@
# }## FOR ~ i
#
# shell.exec(normalizePath(path_shiny_www))

#______________________________
# Knit to HTML manually

path_rmd <- file.path("inst", "shiny-examples", "ContDataQC", "rmd")
path_shiny_www <- file.path("inst", "shiny-examples", "ContDataQC", "www", "RMD_HTML")

# open folder
shell.exec(normalizePath(path_rmd))

# Copy
myFile <- list.files(path = path_rmd
, pattern = "^App_.+\\.html$"
, full.names = TRUE)
file.copy(myFile
, file.path(path_shiny_www, basename(myFile))
, overwrite = TRUE)
# Delete
unlink(myFile)

1 change: 1 addition & 0 deletions inst/shiny-examples/ContDataQC/external/tab_1_About.R
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ function() {
tabsetPanel(type = "tabs"

, tabPanel("Overview"
, p(paste0("Version ", version, "."))
, includeHTML("www/RMD_HTML/App_1a_Overview.html")
)## tabPanel ~ END

Expand Down
3 changes: 3 additions & 0 deletions inst/shiny-examples/ContDataQC/global.R
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ library(shinyjs)
# #access Window's zipping abilities
# Sys.setenv(PATH = paste(Sys.getenv("PATH"), "C:\\Rtools\\bin", sep = ";"))

# Version Number
version <- "2.0.7.9026"

#Maximum individual file size that can be uploaded is 70 MB
options(shiny.maxRequestSize = 70 * 1024^2)

Expand Down
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/rmd/App_1a_Overview.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ The ContDataQC R package and Shiny app were developed with the following objecti

The figure below lays out the general workflow for using the ContDataQC R tools. Items in bold blue text show functions that can be performed with ContDataQC.

Want to try it yourself? Click [here](Custom_QC_Config_ECO66G20_test1.R) to
Want to try it yourself? Click [here](Custom_QC_Config_ECO66G20_test1.R) [R] to
download test files.

![ContDataQC Shiny App workflow](RMD_Images/1.1_Fig1_About_20230828.jpg){width=95%}
Expand Down
4 changes: 2 additions & 2 deletions inst/shiny-examples/ContDataQC/rmd/App_1b_TestData.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ if(boo_DEBUG==TRUE){
There are two sets of test files, one from Onset HOBO U20 water level loggers (non-vented pressure transducers) and one from a miniDOT DO sensor. We are happy to add more test data to this website. If you have example files that you would like to contribute, please send them to Jen Stamp ([Jen.Stamp\@tetratech.com](mailto:[email protected]){.email}).

# Onset HOBO
The Onset HOBO U20 files can be downloaded [here](TestData_HOBOU20_20220822.zip).
The Onset HOBO U20 files can be downloaded [here](TestData_HOBOU20_20220822.zip) [ZIP].
The test data consist of two comma-separated values (CSV) files. The files are from the same site but cover two different time periods.

* **Test**
Expand All @@ -36,7 +36,7 @@ The test data consist of two comma-separated values (CSV) files. The files are f

# miniDOT DO Sensors

The miniDOT DO test files can be downloaded [here](TestData_miniDOT.zip).
The miniDOT DO test files can be downloaded [here](TestData_miniDOT.zip) [ZIP].
They are from one site and one deployment period.
You’ll see two sets of folders. Within each folder are example input and output files.

Expand Down
18 changes: 9 additions & 9 deletions inst/shiny-examples/ContDataQC/rmd/App_1d_Tips.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -40,21 +40,21 @@ people using the data decide whether to remove questionable measurements from
their analysis.

* Document that you checked each point flagged as ‘F’ and ‘S’ by adding a data
qualifier to the Comment column. Click [here](DataQualifiers_20220210.xlsx) to
qualifier to the Comment column. Click [here](DataQualifiers_20220210.xlsx) [XLSX] to
download the list of data qualifiers currently being used by RMN partners.

* Leave missing data cells blank.

## Workflow

Click [here](Workflow_QC_report_20220824.pdf) for recommendations on a workflow
Click [here](Workflow_QC_report_20220824.pdf) [PDF] for recommendations on a workflow
for reviewing the QC reports.

If you have a backlog of data files from multiple deployments and are wondering
how to most efficiently QC the files, [here](Workflow_DataPileup_20220824.pdf)
how to most efficiently QC the files, [here](Workflow_DataPileup_20220824.pdf) [PDF]
is a recommended workflow.

Click [here](SiteVisitChecklist.zip) for a site visit checklist to help ensure
Click [here](SiteVisitChecklist.zip) [ZIP] for a site visit checklist to help ensure
that you do not forget to document information that is important for the data QC
process.

Expand All @@ -64,12 +64,12 @@ Some people have had problems with air and water sensors being out of sync (
e.g., one records at 11:00 and the other records at 11:07). If you are deploying
air and water sensors at a site, make sure they are recording at the same time.
This will make data processing faster and easier. If you are using Onset HOBO
sensors, click [here](HOBO_ConfigLaunch_20170803.pdf) for tips on configuring
sensors, click [here](HOBO_ConfigLaunch_20170803.pdf) [PDF] for tips on configuring
them in a way that ensures they will record at the same time.

Another issue that sometimes occurs is data overlap (where more than one file
has measurements covering part of the same time periods). If you are using Onset
HOBO sensors, click [here](HOBO_DataDownload_20170823.pdf) for procedures to
HOBO sensors, click [here](HOBO_DataDownload_20170823.pdf) [PDF] for procedures to
follow during data download to clear sensor memory and avoid overlapping data.

## Accuracy checks
Expand All @@ -79,20 +79,20 @@ lab and/or in the field with sensor measurements from the closest date/time. The
sensor measurement should be within the accuracy quoted by the manufacturer (
e.g., ±0.2°C if you are using the Onset HOBO proV2 sensor). Accuracy checks are
used to validate your data, and in some cases, correct for sensor drift. Click
[here](EXAMPLE_AccuracyCheckWkst.xlsx) to download an example of an accuracy
[here](EXAMPLE_AccuracyCheckWkst.xlsx) [XLSX] to download an example of an accuracy
check worksheet being kept by an RMN partner.

## Visual checks of time series plots

Visual checks of time series plots are an important part of the QC process.
Click [here](PlotQC_WatchList_20220824.pdf) to see a compilation of patterns to
Click [here](PlotQC_WatchList_20220824.pdf) [PDF] to see a compilation of patterns to
watch for when interpreting the plots. They include ice cover, beaver activity,
leaf packs, dewatering, and dead batteries. We will continue to add slides to
this as we receive materials.

## Checking sensor data against other data sources

This is optional but encouraged if time permits. Click [here](Daymet_Wx_Gage.zip) for instructional
This is optional but encouraged if time permits. Click [here](Daymet_Wx_Gage.zip) [ZIP] for instructional
materials from David Smith (Kentucky Division of Water) on how he has been
downloading data from nearby weather stations and USGS gages, as well as modeled
air temperature and precipitation data from Daymet (
Expand Down
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/rmd/App_1e_Advanced.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ and generate outputs for each input file that matches the criteria.
Users with intermediate to advanced-level R skills can customize the settings
(e.g., add new parameters, change units, change file naming and formatting
requirements, change the format of one of the QC reports from Word to html or
PDF) by editing a plain text [configuration file](Config_default.zip). Users
PDF) by editing a plain text [configuration file](Config_default.zip) [ZIP]. Users
have more flexibility using the Shiny app versus R package when it comes to
setting directories and file naming schemes because they can browse to the
desired input file(s).
Expand Down
10 changes: 5 additions & 5 deletions inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@ The HOBOware reformat function is for data files from Onset HOBO sensors. It con

HOBO sensor files must first be opened with HOBOware software, then processed and exported as CSV files. If you follow these instructions, you will be able to run them through the reformat function without manipulation -

* [HOBOware default settings](HOBOware_DefaultSettings_20220824.pdf)
* [HOBOware default settings](HOBOware_DefaultSettings_20220824.pdf) [PDF]

* [Temperature (proV2)](HOBOware_Temperature_20220825.pdf)
* [Temperature (proV2)](HOBOware_Temperature_20220825.pdf) [PDF]

* [Water level (U20 series)](HOBOware_WaterLevel_U20_20220824.pdf)
* [Water level (U20 series)](HOBOware_WaterLevel_U20_20220824.pdf) [PDF]

* [Dissolved oxygen (U26)](HOBOware_DO_20220824.pdf)
* [Dissolved oxygen (U26)](HOBOware_DO_20220824.pdf) [PDF]

### Instructions

Expand All @@ -47,7 +47,7 @@ HOBO sensor files must first be opened with HOBOware software, then processed an

* Unzip/extract the file(s) or else the files won’t be seen as an option to upload to the QC report function (Main Functions - QC raw data). After file is unzipped, a new folder called ‘HOBO’ will appear. The original zipped file can be deleted if desired.

Time-saver tip: some internet browsers, like Google Chrome and Mozilla Firefox, allow users to change settings so that users can select which folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up in Chrome. With Mozilla Firefox, go to Settings > General > Files and Applications > Check box for “always ask where to save files.”
Time-saver tip: some internet browsers, like Google Chrome and Mozilla Firefox, allow users to change settings so that users can select which folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) [PDF] for instructions on how to set this up in Chrome. With Mozilla Firefox, go to Settings > General > Files and Applications > Check box for “always ask where to save files.”

Your files are now ready to be run through the QC function!
Below are examples of what HOBO U20 CSV files look like before and after they are run through the reformat function. The reformat function does the following -
Expand Down
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/rmd/App_2a2_miniDOT1.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ function; or 2) the **PME java app** that comes with the logger.

## Additional Notes

* Want to try it yourself? Click [here](TestData_miniDOT.zip) to
* Want to try it yourself? Click [here](TestData_miniDOT.zip) [ZIP] to
download miniDOT DO test files.

* Why are the DO saturation values all NA? This occurs when no inputs are
Expand Down
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/rmd/App_2a3_miniDOT2.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ ready to run through the QC report function.

## Additional Notes

- Want to try it yourself? Click [here](TestData_miniDOT.zip) to
- Want to try it yourself? Click [here](TestData_miniDOT.zip) [ZIP] to
download miniDOT DO test files.

* Why are the DO saturation values all NA? This occurs when no inputs are
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Input files need to be formatted a specific way in order for the Shiny app and R

* There are two required fields: SiteID; and “Date Time” (with Date and Time combined into one field).

* If using the default [configuration file](Config_default.zip), column names must match the capitalization, spelling and symbology shown in the table below (for example, R will not recognize ‘sensor depth ft’; it needs to be ‘Sensor Depth ft’)
* If using the default [configuration file](Config_default.zip) [ZIP], column names must match the capitalization, spelling and symbology shown in the table below (for example, R will not recognize ‘sensor depth ft’; it needs to be ‘Sensor Depth ft’)

* Either periods or spaces in the column headings are acceptable. For example, in the table below, air temperature is shown as ‘’Air Temp C’ but it can also be entered as ‘Air.Temp.C’.

Expand Down
4 changes: 2 additions & 2 deletions inst/shiny-examples/ContDataQC/rmd/App_2b_OrganizingFiles.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ we recommend setting up folders as shown below as a starting point. Folders and
files can be customized as you gain more experience. The files should be backed
up regularly and accessible to other members of your data management team.

Click [here](SiteX_20230828.zip) to download a zipped file with a
Click [here](SiteX_20230828.zip) [ZIP] to download a zipped file with a
ready-made set of (empty) folders. Put the file in the desired location, unzip
it and then replace ‘SiteX’ with the appropriate site name.

Expand Down Expand Up @@ -63,4 +63,4 @@ table2a %>%
kable_styling(full_width = F, position = "left")
```

Click [here](EXAMPLE_AccuracyCheckWkst.xlsx) to download example Accuracy Check worksheet.
Click [here](EXAMPLE_AccuracyCheckWkst.xlsx) [XLSX] to download example Accuracy Check worksheet.
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/rmd/App_2c_NamingFiles.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Depth can include decimals (for example, 0.1M is ok) and does not need to
include leading zeros (either 01M or 1M are acceptable). Units can be upper or
lower case (M or m), and abbreviated (M) or spelled out (meters). You can change
the double-dash separator if desired by customizing the
[configuration file](Config_default.zip) (for example, you could change it to a
[configuration file](Config_default.zip) [ZIP] (for example, you could change it to a
tilde ~).

If you are using the Shiny app and have more than one type of sensor deployed at
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ if(boo_DEBUG==TRUE){

Discrete (in situ) measurements can be entered for any parameter. This needs to be done manually after the input file has been formatted (for example, after a HOBO file has been run through the Reformat function). Just add a new column into the input file and use the following naming scheme for the column heading: parameter name and units, with ‘Discrete’ at the beginning (for example, ‘Discrete Air Temp C’). The discrete data point(s) will be included in the time series plots in the Word or html QC reports that are generated when the user runs the QC function (below is an example).

Discrete measurements are important for validating your data, and in some cases, can be used to correct for sensor drift. Accuracy checks are comparisons of discrete or in situ measurements taken in the lab and/or in the field with sensor measurements from the closest date/time. The sensor measurement should be within the accuracy quoted by the manufacturer (e.g., ±0.2°C if you are using the Onset HOBO proV2 sensor). Click [here](EXAMPLE_AccuracyCheckWkst.xlsx) to download an example of an accuracy check worksheet being kept by an RMN partner.
Discrete measurements are important for validating your data, and in some cases, can be used to correct for sensor drift. Accuracy checks are comparisons of discrete or in situ measurements taken in the lab and/or in the field with sensor measurements from the closest date/time. The sensor measurement should be within the accuracy quoted by the manufacturer (e.g., ±0.2°C if you are using the Onset HOBO proV2 sensor). Click [here](EXAMPLE_AccuracyCheckWkst.xlsx) [XLSX] to download an example of an accuracy check worksheet being kept by an RMN partner.

![Example QC plot with discrete data](RMD_Images/2.3_Discrete_Fig1_20230828.jpg)
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/rmd/App_3a1_About.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ if(boo_DEBUG==TRUE){

6. **The user can run the aggregated file through the Summary Statistics function** to generate CSV files with an assortment of summary statistics (e.g., min, median, max, quartiles) and a PDF with time series plots for each parameter.
After running each function, users must unzip/extract the files or else the files won’t be seen as an option to upload for the next step. After files are unzipped, new folders will automatically appear and the original zipped files can be deleted if desired.
Time-saver tip: some internet browsers, like Google Chrome and Mozilla Firefox, allow users to change settings so that users can select which folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up in Chrome. With Mozilla Firefox, go to Settings > General > Files and Applications > Check box for “always ask where to save files.”
Time-saver tip: some internet browsers, like Google Chrome and Mozilla Firefox, allow users to change settings so that users can select which folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) [PDF] for instructions on how to set this up in Chrome. With Mozilla Firefox, go to Settings > General > Files and Applications > Check box for “always ask where to save files.”

Some users may prefer to use the R code instead of the Shiny app.
For more information, see the ‘About - Advanced’ tab
Expand Down
Loading

0 comments on commit 8dc679e

Please sign in to comment.