first pass QC of underway data

[ecrockford]

Establish some simple filters to pass over underway data for rough QC prior to uploading to the API.

Output NA for skipped data.
When looking at max/min values use some sort of rolling average or rolling median?
If 1 parameter is bad, do we mask out all parameters? Do we group TSG together and any others separate?

Possible filters:

• minimum salinity value
• minimum flow meter value (Armstrong)
• maximum fluorometer reading
• Temperature.... extreme max and min?

[joefutrelle]

Locate column header for flow meter for different cruises.

[sbeaulieu]

Column header for salinity: SBE45S. Column header for flow meter: FLOW. A challenge with flow meter is that the average value (in ml/s) differs between cruises (e.g., ~50, ~80, ~130 depending on cruise). I saw zero values in several, and NAN values in some. Let's discuss with Taylor re: thresholds for minimum salinity and flow meter values.

[ecrockford]

Yes thanks for pointing that out Stace. A while ago Joe and I discussed this. We thought maybe a running average with significant deviation from that average could work? Something along those lines. I've heard in passing from the ship's techs that the flow meter isn't super great so we should take values with a grain of salt. Basically, looking for deviations from consistency.

[sbeaulieu]

Noting that in today's Zoom we discussed providing a quality flag column for each of these: salinity, flow meter, and fluorometer. We also discussed providing a comments column that would be auto-populated to alert the end user that a flag(s) was applied.

[joefutrelle]

Endeavor data does not use the column headers specified here, so the API will need to skip adding quality flags for that data.

Once column names are regularized, this will not require per-cruise or per-vessel configurations (although regularization will).

[joefutrelle]

Related to #30

[sbeaulieu]

develop QA/QC checks using Armstrong data and evaluate with Taylor

[joefutrelle]

From the original description:

Establish some simple filters to pass over underway data for rough QC prior to uploading to the API.

If these tools are not part of the REST API then we should consider tracking this work elsewhere. I'm leaving the issue open for now but addressing it may not touch this codebase.