Observation Data QC improvements

rinex-cli/config/sv_manual_gap.json

@@ -0,0 +1,7 @@
+{
+    "classification": "Sv",
+    "gap_tolerance": {
+        "centuries": 0,
+        "nanoseconds": 7200000000000
+    }
+}

rinex-cli/doc/qc.md

@@ -1,10 +1,10 @@
 Quality Check (QC)
 ==================
 
-RINEX quality check is a special mode of this tool, activated with the `--qc` option.
+RINEX quality check is a special mode, activated with `--qc`.
 
 QC is first developed for Observation files analysis, but this tool  
-will accept other RINEX files, for which it will compute basic statistical anlysis. 
+will accept other RINEX files, for which it will compute basic statistical analysis. 
 
 QC is a well known procedure in data preprocessing. 
 There are a few differences people who are very familiar with `teqc` must
@@ -17,7 +17,7 @@ to advanced "teqc" users.  Among them:
 
 * Quick GNSS filters (`-G`, `-R`, ...) still exist
 * In augmented mode, `-no_orbit X` is feasible if you know how to operate the [preprocessor](doc/preprocessing.md)
-* Similar position anlysis and reporting
+* Similar position analysis and reporting
 * Similar signals analysis and reporting
 
 ## Differences with `teqc`
@@ -31,13 +31,14 @@ Although we support as many Navigation files to be specified, when the _augmente
 This allows for example Glonass and other NAV context to be correctly defined.
 
 1. `--fp [FILE]` for the Observation file
-2. `--nav [FILE1] [FILE2]..` : pass NAV contexts
+2. `--nav [FILE1] --nav [FILE2]..` : define NAV context 
+3. `--sp3 [FILE1] --sp3 [FILE2]..` : pass SP3 data
 
 We will generate products the
 [workspace](https://github.com/gwbres/rinex/tree/rinex-cli/workspace)
 folder, that includes QC reports.
 
-Unlike teqc, we do not support BINEX nor SP3 input data/files as of today.
+Unlike teqc, we do not support BINEX input data/files as of today.
 
 Unlike teqc we do not limit ourselves to the analysis of
 GPS and Glonass constellations.
@@ -57,10 +58,10 @@ For example, this would apply to
 ## QC specific command line options
 
 * `--qc-only`: ensures the tool will only perform the QCs,   
-other graphs or record analysis is turned off. This is the most efficient  
-QC mode.  
+other graphs or record analysis is turned off.   
+This command line option makes the quality checks and reporting faster.
 
-* `--qc-config`: pass a configuration file for QC reporting and calculations management (see down below) 
+* `--qc-cfg`: QC analysis and reporting fine tuning (see down below).
 
 ## Basic QC (No NAV)
 
@@ -82,83 +83,85 @@ Run this configuration for the most basic QC:
 
 ```bash
 rinex-cli \
-    -P gps,glo \
+    -P GPS,GLO \
     --qc-only \
-    --qc-conf rinex-cli/config/basic.json \
+    --qc-cfg rinex-cli/config/gnss_snr30db.json \
     --fp test_resources/CRNX/V3/ESBC00DNK_R_20201770000_01D_30S_MO.crx.gz
 ```
 
-`--qc-conf` is independent to `--qc` activation.   
 If you activate QC without any configuration, the default configuration is used.
 
-basic.json specifies that we want to report on a constellation basis.  
-If you compare this report to the previous one
+## QC Configuration file
+
+The QC configuration file allows precise control of how analysis are performed.  
+We provide example configurations in `rinex-cli/config`.  
+
+All parameters are optional, in a configuration file description. That means,
+the default value is used if one parameter is omitted. For example, the classification
+method is "mandatory" to a QC analysis. If not defined, the GNSS classification method is used.
+
+
+### QC Classification method
+
+- "GNSS": the provided context is splitted into all constellations that remain
+after pre processing. You can one sub report per constellation.
+It is most convenient when one or two constellations are left out, for example with `-P GLO,GPS`.  
+The report is designed in alphabetical order.
+
+- "Sv" : split by Satellite vehicle. It is most convenient when a few vehicles are isolated,
+for example either by retaining a single constellation : `-P GPS`, or by retaining
+specific vehicles: `-P G08,G15,C19,C31,R08`. 
 
-1. you get one table per constellation. We only retained GPS and Glonass in this example,   
-therefore the report is made of two tables.
-2. All statistical analysis are made on constellations separately and independently    
-3. A "25%" statistical window is specified. "window" accepts either a Duration  
-or a percentage of the file. Here we use the latter, and 25% means we will have 4 statistical  
-analysis performed over the course of 6 hours, because this file is 24h long.  
-4. You also have less information in this basic configuration, because most calculations are turned off
+- "Physics": split by Observable. 
 
-Try this configuration now:
+### Sv classification case
+
+The Sv classification case is interesting, because the Sampling/Gap anlalysis emphasizes
+when isolated vehicles go out of sight.
+
+For example, compare the report generated by this "default" run : 
 
 ```bash
-rinex-cli \
-    -F gps,glo \
-    --qc-only \
-    --qc-conf rinex-cli/config/basic_manual_gap.json \
-    --fp test_resources/CRNX/V3/ESBC00DNK_R_20201770000_01D_30S_MO.crx.gz
+./target/release/rinex-cli \
+    --fp test_resources/CRNX/V3/MOJN00DNK_R_20201770000_01D_30S_MO.crx.gz \
+    -P G08,G15,G16,R23,C19,C09 \
+    --qc
 ```
 
-A "10%" statistical slot is specified, you get more statistical analysis because the time slot   
-now spans approximately 2 hours. 
-Also "manual_gap" is specified and set to "10 seconds". That means that a duration  
-of 10 seconds is now considered as an abnormal gap, in the data gap analysis.  
-In default configuration, there is no manual gap. That means an abnormal gap  
-is any abnormal duration above the dominant epoch interval (sample rate).
+To this one :
+
+```bash
+./target/release/rinex-cli \
+    --fp test_resources/CRNX/V3/MOJN00DNK_R_20201770000_01D_30S_MO.crx.gz \
+    -P G08,G15,G16,R23,C19,C09 \
+    --qc --qc-cfg rinex-cli/config/sv_manual_gap.json
+```
 
-When no configuration files are given, the default configuration is used
+### SNR parametrization
 
-```json
 TODO
-```
 
-## Advanced configurations
+### CS analysis 
 
-Now let's move on to more "advanced" configuration, in which basically all   
-calculations are active and customized
+TODO
 
-```bash
-rinex-cli \
-    -P gps,glo \
-    --qc-separate \
-    --qc-conf rinex-cli/config/basic_manual_gap.json \
-    --fp test_resources/CRNX/V3/ESBC00DNK_R_20201770000_01D_30S_MO.crx.gz
-```
+## QC with Navigation Data
 
-## Basic QC (with NAV)
+Providing Navigation Data is the only way for complete QCs.  
+With such context, we can now use the "elevation mask" in the configuration file:
 
-Let's go back to our basic demo and provide Navigation context:
+Compare these two runs. In the second one, Sv loss of sight is larger because
+the go out of sight more rapidly, due to the stringent elevation criteria :
 
 ```bash
 rinex-cli \
     -P gps,glo \
-    --qc-separate \
-    --qc-conf rinex-cli/config/basic.json \
+    -P G08,G15,G16,R23,C19,C09 \
+    --qc --qc-cfg rinex-cli/config/sv_manual_gap_ev35.json
     --fp test_resources/CRNX/V3/ESBC00DNK_R_20201770000_01D_30S_MO.crx.gz \
     --nav test_resources/OBS/V3/ESBC00DNK_R_20201770000_01D_MN.rnx.gz
 ```
-
-Navigation context is fully taken into account in advanced calculations
 
-```bash
-rinex-cli \
-    -F gps,glo \
-    --qc-separate \
-    --qc-conf rinex-cli/config/advanced_study.json \
-    --fp test_resources/CRNX/V3/ESBC00DNK_R_20201770000_01D_30S_MO.crx.gz \
-    --nav test_resources/OBS/V3/ESBC00DNK_R_20201770000_01D_MN.rnx.gz
-``
+## QC on other RINEX
 
+TODO

rinex-cli/src/cli.rs

@@ -298,6 +298,7 @@ Refer to README"))
             if let Ok(content) = std::fs::read_to_string(path) {
                 let opts = serde_json::from_str(&content);
                 if let Ok(opts) = opts {
+                    info!("qc parameter file \"{}\"", path);
                     opts
                 } else {
                     error!("failed to parse parameter file \"{}\"", path);

rinex-cli/src/main.rs

@@ -270,18 +270,18 @@ pub fn main() -> Result<(), rinex::Error> {
      * Code Multipath analysis
      */
     if cli.multipath() {
-        let data = ctx
-            .primary_data()
-            .observation_phase_align_origin()
-            .observation_phase_carrier_cycles()
-            .mp();
-        plot::plot_gnss_dcb(
-            &mut plot_ctx,
-            "Code Multipath Biases",
-            "Meters of delay",
-            &data,
-        );
-        info!("--mp analysis");
+        //let data = ctx
+        //    .primary_data()
+        //    .observation_phase_align_origin()
+        //    .observation_phase_carrier_cycles()
+        //    .mp();
+        //plot::plot_gnss_dcb(
+        //    &mut plot_ctx,
+        //    "Code Multipath Biases",
+        //    "Meters of delay",
+        //    &data,
+        //);
+        warn!("--mp analysis not available yet");
     }
     /*
      * [GF] recombination visualization requested
@@ -471,8 +471,6 @@ pub fn main() -> Result<(), rinex::Error> {
      */
     if qc {
         info!("entering qc mode");
-        let mut qc_opts = cli.qc_config();
-
         /*
          * QC Config / versus command line
          * let the possibility to define some parameters

rinex/Cargo.toml

@@ -15,11 +15,11 @@ rust-version = "1.64"
 [features]
 default = [] # no features by default
 sbas = ["geo", "wkt"]
-obs = ["statrs"]
+obs = []
 meteo = []
 nav = []
 processing = []
-qc = ["horrorshow", "processing", "sp3"] # rinex Quality Check (mainly OBS RINEX) 
+qc = ["horrorshow", "processing", "sp3", "statrs"] # rinex Quality Check (mainly OBS RINEX) 
 
 [package.metadata.docs.rs]
 all-features = true

rinex/src/algorithm/averaging.rs

@@ -1,58 +1,43 @@
-use crate::prelude::*;
+use hifitime::{Duration, Epoch};
 
-pub enum AverageType {
-    MovingAverage,
-    Exponential,
-    Cummulative,
+fn moving_average<T: std::default::Default>(
+    data: Vec<(Epoch, T)>,
+    window: Duration,
+) -> Vec<(Epoch, T)> {
+    let mut acc = T::default();
+    let mut prev_epoch: Option<Epoch> = None;
+    let mut ret: Vec<(Epoch, T)> = Vec::new();
+    for (epoch, value) in data {}
+    ret
 }
 
-pub struct Averager {
-    buffer: Vec<f64>,
-    next_epoch: Option<Epoch>,
-    window: Duration,
-    avgtype: AverageType,
+#[derive(Debug, Clone, Copy)]
+pub enum Averager {
+    MovingAverage(Duration),
 }
 
-impl Averager {
-    pub fn new(window: Duration) -> Self {
-        Self {
-            buffer: Vec::new(),
-            next_epoch: None,
-            window,
-        }
+impl Default for Averager {
+    fn default() -> Self {
+        Self::MovingAverage(Duration::from_seconds(600.0_f64))
     }
-    pub fn average(&mut self, data: (f64, Epoch)) -> Option<f64> {
-        match self.avgtype {
-            AverageType::MovingAverage => self.moving_average(data),
-            AverageType::Exponential => self.moving_average(data),
-            AverageType::Cummulative => self.moving_average(data),
-        }
+}
+
+impl Averager {
+    pub fn mov(window: Duration) -> Self {
+        Self::MovingAverage(window)
     }
-    pub fn moving_average(&mut self, data: (f64, Epoch)) -> Option<f64> {
-        self.buffer.push(data.0);
-        if self.next_epoch.is_none() {
-            self.next_epoch = Some(data.1 + self.window);
-        }
-        if let Some(next_epoch) = self.next_epoch {
-            if data.1 >= next_epoch {
-                self.next_epoch = Some(data.1 + self.window);
-                let mut avg = 0.0_f64;
-                for b in &self.buffer {
-                    avg += b;
-                }
-                let ret = avg / self.buffer.len() as f64;
-                self.buffer.clear();
-                return Some(ret);
-            }
+    pub fn eval<T: std::default::Default>(&self, input: Vec<(Epoch, T)>) -> Vec<(Epoch, T)> {
+        match self {
+            Self::MovingAverage(dt) => moving_average(input, *dt),
         }
-        None
     }
 }
 
 #[cfg(test)]
 mod test {
+    use super::*;
     #[test]
     fn test_moving_average() {
-
+        let mov = Averager::mov(Duration::from_seconds(10.0_f64));
     }
 }