readthedocs updated and minor change to get_watsontx CLI script (#197)

* watson msg keys added * Entry points fix (#196) * CLI scripts moved to modules with defined entry points * Basic unittesting added for CLI scripts * CLI unit tests moved to module scripts * PR tests updated with new structuring * qc added to autodocs * All station data fetching added * Link updated * CLI functions updated * CLI examples updated * Watson tx de-bug * Payload info removed from requirements * Duplicate file de-bug * GIOS/Watson station identifier * Naming if statement changed Co-authored-by: Mads Christian Lund <[email protected]> * sub-headers added * Better file renaming solution * filename solution with re --------- Co-authored-by: Mads Christian Lund <[email protected]>
GEUS-Glaciology-and-Climate · Nov 2, 2023 · 97586e3 · 97586e3
1 parent 7261c27
commit 97586e3
Show file tree

Hide file tree

Showing 9 changed files with 77 additions and 82 deletions.
diff --git a/.github/workflows/joss-pdf.yml b/.github/workflows/joss-pdf.yml
@@ -1,5 +1,7 @@
-on: [push]
-
+on:
+  workflow_dispatch:
+#on: [push]
+
 jobs:
   paper:
     runs-on: ubuntu-latest

diff --git a/docs/guide_developer.rst b/docs/guide_developer.rst
@@ -4,29 +4,26 @@ Developer guide
 
 Contributions, bug reports and fixes, documentation improvements, enhancements and ideas are welcome. A good starting place to look at is:
 
-1. PROMICE-AWS-data-issues_, where we report suspicious or incorrect data
-2. pypromice's GitHub Issues_, for an overview of known bugs, developments and ideas
-
-.. _PROMICE-AWS-data-issues: https://github.com/GEUS-Glaciology-and-Climate/PROMICE-AWS-data-issues
-.. _Issues: https://github.com/GEUS-Glaciology-and-Climate/pypromice/issues
+1. `PROMICE-AWS-data-issues <https://github.com/GEUS-Glaciology-and-Climate/PROMICE-AWS-data-issues>_`, where we report suspicious or incorrect data
+2. pypromice's `GitHub Issues <https://github.com/GEUS-Glaciology-and-Climate/pypromice/issues>`_, for an overview of known bugs, developments and ideas
 
 
 Data reports
 ============
 
 Automatic weather station (AWS) data from the Greenland Ice Sheet are often imperfect due to the complexity and conditions involved in installing and maintaining the AWS. 
 
-If you are using our AWS data and something seems suspicious or erroneous, you can check the PROMICE-AWS-data-issues_ space to see if has previously been flagged and/or fixed. If not, then please follow the conventions stated in the repository and open an issue.
+If you are using our AWS data and something seems suspicious or erroneous, you can check the `PROMICE-AWS-data-issues <https://github.com/GEUS-Glaciology-and-Climate/PROMICE-AWS-data-issues>`_ space to see if has previously been flagged and/or fixed. If not, then please follow the conventions stated in the repository and open an issue.
 
 .. note::
 
-	Data visualisations best demonstrate data problems and are greatly appreciated in solving data issues. If you are unsure, see examples of our closed issues in PROMICE-AWS-data-issues_ 
+	Data visualisations best demonstrate data problems and are greatly appreciated in solving data issues. If you are unsure, see examples of our closed issues in `PROMICE-AWS-data-issues <https://github.com/GEUS-Glaciology-and-Climate/PROMICE-AWS-data-issues>`_ 
 
 
 Bug reports and enhancement requests
 ====================================
 
-Bug reports are essential to improving the stability and usability of pypromice. These should be raised on pypromice's GitHub Issues_. A complete and reproducible report is essential for bugs to be resolved easily, therefore bug reports must:
+Bug reports are essential to improving the stability and usability of pypromice. These should be raised on pypromice's `GitHub Issues <https://github.com/GEUS-Glaciology-and-Climate/pypromice/issues>`_. A complete and reproducible report is essential for bugs to be resolved easily, therefore bug reports must:
 
 1. Include a concise and self-contained Python snippet reproducing the problem. For example:
 
@@ -47,9 +44,7 @@ Bug reports are essential to improving the stability and usability of pypromice.
 
 .. note:: 
 
-	Before submitting an issue, please make sure that your installation is correct and working from either the pip installation or the main_ branch of the pypromice repository.
-
-.. _main: https://github.com/GEUS-Glaciology-and-Climate/pypromice/tree/main
+	Before submitting an issue, please make sure that your installation is correct and working from either the pip installation or the `main <https://github.com/GEUS-Glaciology-and-Climate/pypromice/tree/main>`_ branch of the pypromice repository.
 
 
 Contributing to pypromice
@@ -61,9 +56,7 @@ You can work directly with pypromice's development if you have a contribution, s
 Forking 
 -------
 
-In order to contribute, you will need your own fork of the pypromice GitHub repository to work on the code. Go to the repo_ and choose the ``Fork`` option. This now creates a copy in your own GitHub space, which is connected to the upstream pypromice repository.
-
-.. _repo: https://github.com/GEUS-Glaciology-and-Climate/pypromice
+In order to contribute, you will need your own fork of the pypromice GitHub repository to work on the code. Go to the `repo <https://github.com/GEUS-Glaciology-and-Climate/pypromice>`_ and choose the ``Fork`` option. This now creates a copy in your own GitHub space, which is connected to the upstream pypromice repository.
 
 
 Creating a development branch
@@ -85,15 +78,13 @@ To contribute your changes to pypromice, you need to make a pull request from yo
 
 .. code:: console
 
-	$ git fetch upstream
+	$ git fetch
 	$ git merge upstream/main
 
-And then open a pull request as documented here_. Make sure to include the following in your pull request description:
+And then open a pull request as documented `here <https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork>`_. Make sure to include the following in your pull request description:
 
 1. The aim of your changes
 2. Details of what these changes are
 3. Any limitations or further development needed
 
 Your pull request will be reviewed and, if valid and suitable, will be accepted. Following this, you will be listed as a contributor to pypromice!
-
-.. _here: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork
diff --git a/docs/guide_user.rst b/docs/guide_user.rst
@@ -9,9 +9,7 @@ Two components are needed to perform Level 0 to Level 3 processing:
 - A Level 0 dataset file (.txt), or a collection of Level 0 dataset files
 - A station config file (.toml)
 
-Two test station datasets and config files are available with pypromice as an example of the Level 0 to Level 3 processing. These can be found on the Github repo here_, in the ``src/pypromice/test/`` directory in the cloned repo.
-
-.. _here: https://github.com/GEUS-Glaciology-and-Climate/pypromice/tree/joss-doc-edits/src/pypromice/test
+Two test station datasets and config files are available with pypromice as an example of the Level 0 to Level 3 processing. These can be found on the Github repo `here <https://github.com/GEUS-Glaciology-and-Climate/pypromice/tree/main/src/pypromice/test>`_, in the ``src/pypromice/test/`` directory in the cloned repo.
 
 
 These can be processed from Level 0 to a Level 3 data product as an ``AWS`` object in pypromice.  
@@ -66,15 +64,15 @@ The Level 0 to Level 3 processing can also be executed from a CLI using the ``ge
 
 .. code:: console
 
-    $ getL3 -c src/pypromice/test/test_config1.toml -i src/pypromice/test -o src/pypromice/test
+    $ get_l3 -c src/pypromice/test/test_config1.toml -i src/pypromice/test -o src/pypromice/test
 
 
 Loading our data
 ================
 
 Import from Dataverse (no downloads!)
 -------------------------------------
-The automated weather station (AWS) datasets from the PROMICE and GC-Net monitoring programmes are openly available on our Dataverse_. These can be imported directly with pypromice, with no downloading required.
+The automated weather station (AWS) datasets from the PROMICE and GC-Net monitoring programmes are openly available on the `GEUS Dataverse <https://dataverse.geus.dk/dataverse/AWS>`_. These can be imported directly with pypromice, with no downloading required.
 
 .. code:: python
 
@@ -89,27 +87,25 @@ All available AWS datasets are retrieved by station name. Use ``aws_names()`` to
 
 	n = pget.aws_names()
 	print(n)
-
-.. _Dataverse: https://dataverse.geus.dk/dataverse/AWS
-
+	
 
 Download with pypromice
 -----------------------
 AWS data can be downloaded to file with pypromice. Open up a CLI and use the ``getData`` command.
 
 .. code:: console
 
-	$ getData -n KPC_U_hour.csv
+	$ get_promice_data -n KPC_U_hour.csv
 
 Files are downloaded to the current directory as a CSV formatted file. Use the ``-h`` help flag to explore further input variables.
 
 .. code:: console
 
-	$ getData -h
+	$ get_promice_data -h
 
 .. note::
 
-	Currently, this functionality within pypromice is only for our hourly AWS data. For daily and monthly AWS data, please download these from the Dataverse_.
+	Currently, this functionality within pypromice is only for our hourly AWS data. For daily and monthly AWS data, please download these from the `GEUS Dataverse <https://dataverse.geus.dk/dataverse/AWS>`_.
 
 
 Load from NetCDF file
@@ -162,17 +158,9 @@ Once loaded, variables from an AWS dataset can be simply plotted with using pand
 
 .. note::
 
-	Variable names are provided in the dataset metadata, or can be found on in our variables look-up table here_. For more complex plotting, please see either the xarray_ or pandas_ plotting documentation.
-
-.. _here: https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/src/pypromice/process/variables.csv
-.. _xarray: https://docs.xarray.dev/en/stable/user-guide/plotting.html
-.. _pandas: https://pandas.pydata.org/docs/user_guide/10min.html#plotting
+	Variable names are provided in the dataset metadata, or can be found on in our `variables look-up table <https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/src/pypromice/process/variables.csv>`_. For more complex plotting, please see either the `xarray <https://docs.xarray.dev/en/stable/user-guide/plotting.html>`_ or `pandas <https://pandas.pydata.org/docs/user_guide/10min.html#plotting>`_ plotting documentation.
 
 
 .. warning::
 
-	Plotting with either xarray or pandas requires the matplotlib_ package. This is not supplied as a dependency with pypromice, so please install matplotlib separately if you wish to do so.
-
-.. _matplotlib: https://matplotlib.org/
-
-.. _matplotlib: https://matplotlib.org/
+	Plotting with either xarray or pandas requires `matplotlib <https://matplotlib.org/>`_. This is not supplied as a dependency with pypromice, so please install matplotlib separately if you wish to do so.
diff --git a/docs/index.rst b/docs/index.rst
@@ -1,22 +1,18 @@
 pypromice
 =========
 
-pypromice_ is designed for processing and handling PROMICE_ and GC-Net_ automated weather station (AWS) data. The PROMICE (Programme for Monitoring of the Greenland Ice Sheet) weather station network monitors glacier mass balance in the melt zone of the Greenland Ice Sheet, providing ground truth data to calibrate mass budget models. GC-Net (Greenland Climate Network) weather stations measure snowfall and surface properties in the accumulation zone, providing valuable knowledge on the Greenland Ice Sheet's mass gain and climatology.
+`pypromice <https://github.com/GEUS-Glaciology-and-Climate/pypromice>`_ is designed for processing and handling `PROMICE <https://promice.dk>`_ and `GC-Net <http://cires1.colorado.edu/steffen/gcnet/>`_ automated weather station (AWS) data. The PROMICE (Programme for Monitoring of the Greenland Ice Sheet) weather station network monitors glacier mass balance in the melt zone of the Greenland Ice Sheet, providing ground truth data to calibrate mass budget models. GC-Net (Greenland Climate Network) weather stations measure snowfall and surface properties in the accumulation zone, providing valuable knowledge on the Greenland Ice Sheet's mass gain and climatology.
 
-The PROMICE and GC-Net monitoring networks have unique AWS configurations and provide specialized data, therefore a toolbox is needed to handle and process their data. pypromice is the go-to toolbox for handling and processing climate and glacier datasets from the PROMICE and GC-Net monitoring networks. New releases of pypromice are uploaded alongside PROMICE AWS data releases to our Dataverse_ for transparency purposes and to encourage collaboration on improving our data.
+The PROMICE and GC-Net monitoring networks have unique AWS configurations and provide specialized data, therefore a toolbox is needed to handle and process their data. pypromice is the go-to toolbox for handling and processing climate and glacier datasets from the PROMICE and GC-Net monitoring networks. New releases of pypromice are uploaded alongside PROMICE AWS data releases to the `GEUS Dataverse <https://dataverse.geus.dk/dataverse/AWS>`_ for transparency purposes and to encourage collaboration on improving our data.
 
 If you intend to use PROMICE and GC-Net AWS data and/or pypromice in your work, please cite these publications below, along with any other applicable PROMICE publications where possible:
 
-Fausto, R.S., van As, D., Mankoff, K.D., Vandecrux, B., Citterio, M., Ahlstrøm, A.P., Andersen, S.B., Colgan, W., Karlsson, N.B., Kjeldsen, K.K., Korsgaard, N.J., Larsen, S.H., Nielsen, S., Pedersen, A.Ø., Shields, C.L., Solgaard, A.M., and Box, J.E. (2021) Programme for Monitoring of the Greenland Ice Sheet (PROMICE) automatic weather station data, Earth Syst. Sci. Data, 13, 3819–3845, https://doi.org/10.5194/essd-13-3819-2021
+Fausto, R.S., van As, D., Mankoff, K.D., Vandecrux, B., Citterio, M., Ahlstrøm, A.P., Andersen, S.B., Colgan, W., Karlsson, N.B., Kjeldsen, K.K., Korsgaard, N.J., Larsen, S.H., Nielsen, S., Pedersen, A.Ø., Shields, C.L., Solgaard, A.M., and Box, J.E. (2021) Programme for Monitoring of the Greenland Ice Sheet (PROMICE) automatic weather station data, Earth Syst. Sci. Data, 13, 3819–3845, `https://doi.org/10.5194/essd-13-3819-2021 <https://doi.org/10.5194/essd-13-3819-2021>`_
 
-How, P., Wright, P.J., Mankoff, K., Vandecrux, B., Fausto, R.S. and Ahlstrøm, A.P. (2023) pypromice: A Python package for processing automated weather station data, Journal of Open Source Software, 8(86), 5298, https://doi.org/10.21105/joss.05298
+How, P., Wright, P.J., Mankoff, K., Vandecrux, B., Fausto, R.S. and Ahlstrøm, A.P. (2023) pypromice: A Python package for processing automated weather station data, Journal of Open Source Software, 8(86), 5298, `https://doi.org/10.21105/joss.05298 <https://doi.org/10.21105/joss.05298>`_
 
-How, P., Lund, M.C., Nielsen, R.B., Ahlstrøm, A.P., Fausto, R.S., Larsen, S.H., Mankoff, K.D., Vandecrux, B., Wright, P.J. (2023) pypromice, GEUS Dataverse, https://doi.org/10.22008/FK2/3TSBF0
+How, P., Lund, M.C., Nielsen, R.B., Ahlstrøm, A.P., Fausto, R.S., Larsen, S.H., Mankoff, K.D., Vandecrux, B., Wright, P.J. (2023) pypromice, GEUS Dataverse, `https://doi.org/10.22008/FK2/3TSBF0 <https://doi.org/10.22008/FK2/3TSBF0>`_
 
-.. _pypromice: https://github.com/GEUS-Glaciology-and-Climate/pypromice
-.. _PROMICE: https://promice.dk
-.. _GC-Net: http://cires1.colorado.edu/steffen/gcnet/
-.. _Dataverse: https://dataverse.geus.dk/dataverse/PROMICE
 
 .. toctree::
    :maxdepth: 2

diff --git a/docs/install.rst b/docs/install.rst
@@ -17,7 +17,7 @@ For the most up-to-date version, pypromice can be installed directly from the re
 Developer install
 *****************
 
-pypromice can be ran in an environment with the pypromice package cloned from GitHub_. 
+pypromice can be ran in an environment with the pypromice package cloned from the `GitHub repo <https://github.com/GEUS-Glaciology-and-Climate/pypromice>`_. 
 
 .. code:: console
 
@@ -27,14 +27,11 @@ pypromice can be ran in an environment with the pypromice package cloned from Gi
 	$ cd pypromice/
 	$ pip install .
 
-pypromice is also provided with a conda environment configuration environment.yml_ for a more straightforward set-up, if needed:
+pypromice is also provided with a `conda environment configuration file <https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/environment.yml>`_ for a more straightforward set-up, if needed:
 
 .. code:: console
 
 	$ conda env create --file environment.yml -n pypromice
-	
-.. _GitHub: https://github.com/GEUS-Glaciology-and-Climate/pypromice
-.. _environment.yml: https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/environment.yml
 
 The package has inbuilt unit tests, which can be run to test the package installation:
 
@@ -52,15 +49,12 @@ Additional dependencies
 
 Additional packages are required if you wish to use pypromice's post-processing functionality. 
 
-eccodes_ is the official package for BUFR encoding and decoding, which pypromice uses for post-process formatting. Try firstly to install with conda-forge like so:
+`eccodes <https://confluence.ecmwf.int/display/ECC/ecCodes+installation>`_ is the official package for BUFR encoding and decoding, which pypromice uses for post-process formatting. Try firstly to install with conda-forge like so:
 
 .. code:: console
 
 	$ conda install -c conda-forge eccodes
 
 .. note::
 
-	If the environment cannot resolve the eccodes installation then follow the steps documented here_ to download eccodes and then install eccodes' python bindings using pip: ``pip3 install eccodes-python``
-
-.. _eccodes: https://confluence.ecmwf.int/display/ECC/ecCodes+installation
-.. _here: https://gist.github.com/MHBalsmeier/a01ad4e07ecf467c90fad2ac7719844a
+	If the environment cannot resolve the eccodes installation then follow the steps documented `here <https://gist.github.com/MHBalsmeier/a01ad4e07ecf467c90fad2ac7719844a>`_ to download eccodes and then install eccodes' python bindings using pip: ``pip3 install eccodes-python``
diff --git a/docs/modules.rst b/docs/modules.rst
@@ -1,6 +1,9 @@
 process
 =======
 
+process.aws
+-----------
+
 .. automodule:: process.aws
    :members:
    :undoc-members:
@@ -33,22 +36,42 @@ process.L2toL3
 postprocess
 ===========
 
+postprocess.csv2bufr
+--------------------
+
 .. automodule:: postprocess.csv2bufr
    :members:  
    :undoc-members:
    :show-inheritance:
 
+qc
+==
+
+qc.persistence
+--------------
+
+.. automodule:: qc.persistence
+   :members:  
+   :undoc-members:
+   :show-inheritance:
+
 get 
 ===
 
-.. automodule:: get
+get.get
+-------
+
+.. automodule:: get.get
    :members:
    :undoc-members:
    :show-inheritance:
 
 tx
 ==
 
+tx.tx
+-----
+
 .. automodule:: tx.tx
    :members:
    :undoc-members:

diff --git a/docs/technical_process.rst b/docs/technical_process.rst
@@ -20,10 +20,7 @@ Payload handling
 Payload decoder
 ===============
 
-``PayloadFormat`` handles the message types and decoding templates. These can be imported from file, with two default CSV files provided with pypromice - payload_formatter.csv_ and payload_type.csv_.
-
-.. _payload_formatter.csv: https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/src/pypromice/tx/payload_formats.csv
-.. _payload_type.csv: https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/src/pypromice/tx/payload_types.csv
+``PayloadFormat`` handles the message types and decoding templates. These can be imported from file, with two default CSV files provided with pypromice - `payload_formatter.csv <https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/src/pypromice/tx/payload_formats.csv>`_ and `payload_type.csv <https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/src/pypromice/tx/payload_types.csv>`_.
 
 
 Payload processing
@@ -37,7 +34,7 @@ The following function can be executed from a CLI to fetch ``L0`` transmission m
 
 .. code:: console
 	
-	$ getL0tx -a accounts.ini -p credentials.ini -c tx/config 
+	$ get_l0tx -a accounts.ini -p credentials.ini -c tx/config 
 	-u last_aws_uid.ini -o tx
 
 .. note::
@@ -54,13 +51,13 @@ To process from L0>>L3, the following function can be executed in a CLI.
 
 .. code:: console
 	
-	$ getL3 -c config/KPC_L.toml -i . -o ../../aws-l3/tx"
+	$ get_l3 -c config/KPC_L.toml -i . -o ../../aws-l3/tx"
 
 And in parallel through all configuration .toml files ``$imei_list``
 
 .. code:: console
 
-	$ parallel --bar "getL3 -c ./{} -i . -o ../../aws-l3/tx" ::: $(ls $imei_list)
+	$ parallel --bar "get_l3 -c ./{} -i . -o ../../aws-l3/tx" ::: $(ls $imei_list)
 
 
 Station configuration
@@ -109,6 +106,4 @@ The TOML config file has the following expectations and behaviors:
 
 .. note::
 
-	Be aware the column names should follow those defined in the variables look-up table found here_. Any column names provided that are not in this look-up table will be passed through the processing untouched.
-
-.. _here: https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/src/pypromice/process/variables.csv
+	Be aware the column names should follow those defined in pypromice's `variables look-up table <https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/main/src/pypromice/process/variables.csv>`_. Any column names provided that are not in this look-up table will be passed through the processing untouched.