Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Edits to status Rmd page #735

Merged
merged 4 commits into from
Oct 25, 2024
+16 −16
Merged

Edits to status Rmd page #735

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dfb0e26
edits to status page
ehinman Oct 25, 2024
8fb83b1
more edits to status page
ehinman Oct 25, 2024
f0f4f11
small correction of version
ehinman Oct 25, 2024
ef5927a
incorporate feedback
ehinman Oct 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions vignettes/Status.Rmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,32 +17,32 @@ editor_options:

This page will be updated frequently with information about the status of dataRetrieval services.

Contact [email protected] with additional questions!
Contact [email protected] with additional questions.

# Overview

As of March 11, 2024, NWIS **discrete water quality** services are "frozen": any public data retrieval will not include any new data. Learn more about the upcoming change and where to find the new samples data in our [blog](https://waterdata.usgs.gov/blog/changes-to-sample-data/).
On March 11, 2024, NWIS **discrete water quality** services were "frozen": any public data retrieval using `readNWISqw()` did not include any new data. As of dataRetrieval v2.7.17, `readNWISqw()` has been retired and replaced by `readWQPqw()`. Learn more about the change and where to find the new samples data in our [blog](https://waterdata.usgs.gov/blog/changes-to-sample-data/).

If you have additional questions about the NWIS qw data service, email [email protected].


### WQP: USGS Data
## Locating USGS data using the Water Quality Portal

**New USGS data**: temporarily not accessible on the main Water Quality Portal (WQP) page (www.waterqualitydata.us). Data are still being collected, but are not available on this webpage. This limited availability is expected to last a few months.
New USGS data (post March 11, 2024) **are temporarily not accessible** on the **main** Water Quality Portal (WQP) page (www.waterqualitydata.us). Data are still being collected, but are not available on this webpage. This limited availability is expected to last a few months.

**New USGS data** _are_ accessible in a pre-release (*beta*) version of the [WQP web page](https://www.waterqualitydata.us/beta/) and new [wqx3 web services](https://waterqualitydata.us/wqx3/). Data are available in the "WQX version 3.0 format" (WQX = [Water Quality Exchange](https://exchangenetwork.net/data-exchange/wqx/)) for these new "Data Profiles" (how the data is formatted by the WQP):
However, new USGS data **are accessible** in a pre-release (*beta*) version of the [WQP web page](https://www.waterqualitydata.us/beta/) and new [wqx3 web services](https://waterqualitydata.us/wqx3/). Data are available in the "WQX version 3.0 format" (WQX = [Water Quality Exchange](https://exchangenetwork.net/data-exchange/wqx/)) for these new "Data Profiles" (how the data is formatted by the WQP):

* Monitoring Location
* Results - Narrow
* Results - Full Physical Chemical
* Results - Basic Physical Chemical
* Sampling Activity

Guidance on how to use the new web page and web services are available in the [User Guide](https://www.waterqualitydata.us/beta/portal_userguide/) and [Web Services Guide](https://www.waterqualitydata.us/beta/webservices_documentation/). Additional profiles will continue to be added to this beta release in the coming weeks.
Guidance on how to use the new web page and web services are available in the [User Guide](https://www.waterqualitydata.us/beta/portal_userguide/) and [Web Services Guide](https://www.waterqualitydata.us/beta/webservices_documentation/). Additional profiles will continue to be added over time.

During the beta period users may encounter bugs or identify issues with the implementation of the WQX 3.0 format: we welcome (and encourage!) your feedback to help improve these offerings, just send an email to [email protected].
**Disclaimer:** During the beta period, users may encounter bugs or identify issues with the implementation of the WQX 3.0 format: we welcome (and encourage!) your feedback to help improve these offerings, just send an email to [email protected].

The current WQP data profiles (available on the main Water Quality Portal web pages and from the current web services, https://www.waterqualitydata.us) deliver data in "WQX version 2.0" format. These will remain available for a period of time after the rollout of version 3.0. Eventually they will be retired, but there is not yet an estimated time line.
The current WQP data profiles (available on the main Water Quality Portal web pages and from the current web services, https://www.waterqualitydata.us) deliver data in "WQX version 2.0" (what we're referring to as the "legacy") format. These will remain available for a period of time after the rollout of version 3.0. Eventually they will be retired, but there is not yet an estimated time line.

# What to expect: dataRetrieval specific

Expand Down Expand Up @@ -71,9 +71,9 @@ knitr::kable(df)

## readWQPqw

This function is generally advertised as a user-friendly function since it only works with a known list of sites, parameter codes or characterisitic names, and start/end dates.
The `readWQPqw()` function is generally advertised as a user-friendly function since it only works with a known list of sites, parameter codes or characterisitic names, and start/end dates.

As of `dataRetrieval` 2.7.15.1, this function will use the WQX 3.0 "narrow" dataProfile. There is an argument "legacy" that allows the user to switch back to the WQX 2.0 profile. Keep in mind the 2.0 profiles will eventually be retired. For any more flexibility, users will need to use the "readWQPdata" function.
As of `dataRetrieval` 2.7.17, this function will use the default WQX version 2 dataProfile, specified by the `legacy = TRUE` argument. Setting `legacy = FALSE` will return the WQX 3.0 "narrow" dataProfile. Keep in mind the 2.0 profiles will eventually be retired. For any more flexibility, users will need to use the `readWQPdata()` function.

An example of a WQX 3.0 return:

Expand All @@ -97,11 +97,11 @@ attr(rawPcode_legacy, "url")

## readWQPdata

The "readWQPdata" function is the most flexible function to get WQP data. Currently there are 4 options that use the new WQX 3.0 profiles, and 11 legacy options.
The `readWQPdata()` function is the most flexible function to get WQP data. Currently there are 11 legacy options and 5 options that use the new WQX 3.0 profiles. Note that `readWQPdata()` does not leverage a `legacy` argument to specify which profile version the user would like returned, but instead relies on the user's specification of `service` and `dataProfile` arguments.

### WQX 3.0

There are now 3 "services" available: ResultWQX, StationWQX and ActivityWQX. The "ResultWQX" service has multiple available "dataProfiles".
There are currently three WQX 3.0 "services" available: ResultWQX, StationWQX and ActivityWQX. The "ResultWQX" service has multiple available "dataProfiles".

| Service | dataProfile |
| -------------- | ---------- |
Expand Down Expand Up @@ -147,7 +147,7 @@ attr(data_sites, "url")

There are 8 services available from the legacy WQP. The Station and Result legacy services can still be accessed, but users should move to StationWQX, ResultWQX, and ActivityWQX. As other former services become available in WQX 3.0, we will update these documents.

| Service | dataProfile | WQX 3.0 option? |
| Service | dataProfile | WQX 3.0 service "analog" |
| -------------- | ---------- | ---------- |
| Station | | StationWQX |
| Result | resultPhysChem | ResultWQX |
Expand Down Expand Up @@ -231,9 +231,9 @@ dl_data <- readWQPdata(
## whatNWISdata


NWIS services are "frozen": the returned data availability will also be frozen ONLY for "qw" data_type_cd results. All other data types should not be affected.
NWIS discrete water quality services are "frozen": the returned data availability will also be frozen ONLY for "qw" data_type_cd results. All other data types should not be affected.

When the NWIS services are decommissioned (possible September 2024): there will no longer be any "qw" information provided in the output of `whatNWISdata`. Discrete water-quality availability will be available via WQP services. More information will be provided as we learn more.
When the NWIS services are decommissioned (likely in 2025): there will no longer be any "qw" information provided in the output of `whatNWISdata`. Discrete water-quality availability will be available via WQP services. More information will be provided as we learn more.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think this is incorrect. I'm wondering if whatNWISdata still returns older qw data, or if the service HAS been decommissioned.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just looking at this. It's still showing qw data, but I'll try to get clarification on what will happen when the NWIS qw service actually gets decommissioned

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qwdata isn't slated for decommission until end of Q1, so it should still be working. Another tact for this warning is to just say that the 'qw' information has not been updated since March 2024 and will soon become deprecated: we recommend users obtain water quality data availability from the WQP services instead. Or something

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but will there be any qw info from the site inventory after Q1, or will there always be the stale info?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if that info is being pulled from an rdb then yes, that info will no longer be available. IoW will not support a public endpoint for data in the qwdata DB indefinitely, at some point it will be decommissioned.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That info is coming from the site service (which is waterservices, not waterdata) - but it is RDB. So something like:

https://waterservices.usgs.gov/nwis/site/?seriesCatalogOutput=true&sites=385032115220501,391133115264801&format=rdb

I'm guessing (and hoping!) that the qw data just goes away because it would be much more confusing to be wrong/outdated. But I'm still not clear if we know that anwer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the data should stop flowing when qwdata is decommissioned.


Here's an example of what will change:

Expand All @@ -245,7 +245,7 @@ nrow(what_NWIS[what_NWIS$data_type_cd == "qw",])
[1] 381
```

So for site "05114000", there are 381 NWIS qw parameters that have been measured. Starting mid-March 2024, the data availability for those 381 parameters will be frozen...even if new data is collected. Eventually those 381 rows of data will not be returned, only 26 rows of data will be returned (407-381).
So for site "05114000", there are 381 NWIS qw parameters that have been measured. Since mid-March 2024, the data availability for those 381 parameters are frozen...even if new data are collected. Eventually those 381 rows of data will not be returned, only 26 rows of data will be returned (407-381).

New services/functions are being developed to replace the lost functionality so check back here for updated information.

Expand Down
Loading