refactor: always try syncing from both grace and grace-fo missions (#71)

feat: added AW13 models using IJ05-R2 ice history

feat: allow input ascii harmonic files to have additional columns

update version number

docs: update environment file

fix: index using granules

doc/environment.yml

@@ -10,6 +10,10 @@ dependencies:
   - geos
   - graphviz
   - h5py
+  - ipywidgets>=7.6,<8.0
+  - ipympl
+  - jupyterlab=3
+  - jupyterlab_widgets
   - lxml
   - markupsafe==2.0.1
   - matplotlib
@@ -28,6 +32,7 @@ dependencies:
   - sphinx-markdown-tables
   - sphinx_rtd_theme
   - texlive-core
+  - tk
   - pip:
      - git+https://github.com/tsutterley/read-GRACE-geocenter.git
      - git+https://github.com/tsutterley/geoid-toolkit.git

doc/source/getting_started/NASA-Earthdata.rst

@@ -16,8 +16,12 @@ NASA Earthdata uses `OAuth2 <https://wiki.earthdata.nasa.gov/pages/viewpage.acti
 
 PO.DAAC
 #######
-The `Physical Oceanography Distributed Active Archive Center (PO.DAAC) <https://podaac.jpl.nasa.gov/>`_ provides data and related information pertaining to the physical processes and conditions of the global oceans, including measurements of ocean winds, temperature, topography, salinity, circulation and currents, and sea ice.
-If any problems contact JPL PO.DAAC support at `[email protected] <mailto:[email protected]>`_ or the NASA EOSDIS support team `[email protected] <mailto:[email protected]>`_.
+The `Physical Oceanography Distributed Active Archive Center (PO.DAAC) <https://podaac.jpl.nasa.gov/>`_
+provides data and related information pertaining to the physical processes and conditions of the global oceans,
+including measurements of ocean winds, temperature, topography, salinity, circulation and currents, and sea ice.
+PO.DAAC hosts
+If any problems contact JPL PO.DAAC support at `[email protected] <mailto:[email protected]>`_
+or the NASA EOSDIS support team `[email protected] <mailto:[email protected]>`_.
 
 WebDAV
 ------
@@ -51,6 +55,25 @@ Or set environmental variables for your NASA Earthdata and JPL WebDAV credential
     export EARTHDATA_PASSWORD=<password>
     export PODAAC_PASSWORD=<webdav>
 
+NASA Common Metadata Repository
+###############################
+
+The NASA Common Metadata Repository (CMR) is a catalog of all data
+and service metadata records contained as part of NASA's Earth
+Observing System Data and Information System (EOSDIS).
+Querying the CMR system is a way of quickly performing a search
+through the NASA Earthdata archive.
+Basic queries for the granule names, PO.DAAC URLs and modification times
+of GRACE/GRACE-FO data are available through the ``cmr`` routine in the
+``utilities`` module.
+For AWS instances in ``us-west-2``, CMR queries can access urls for S3 endpoints.
+
+.. code-block:: python
+
+   ids,urls,mtimes = gravity_toolkit.utilities.cmr(mission='grace-fo',
+      center='JPL', release='RL06', level='L2', product='GSM',
+      solution='BA01', provider='POCLOUD', endpoint='s3', verbose=False)
+
 Other Data Access Examples
 ##########################
 - `Curl and Wget <https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+cURL+And+Wget>`_

doc/source/user_guide/calc_mascon.rst

@@ -58,6 +58,7 @@ Command Line Options
     * ``'ICE6G'``: `ICE-6G GIA Models <https://doi.org/10.1002/2014JB011176>`_
     * ``'Wu10'``: `Wu (2010) GIA Correction <https://doi.org/10.1038/ngeo938>`_
     * ``'AW13-ICE6G'``: `Geruo A ICE-6G GIA Models <https://doi.org/10.1093/gji/ggs030>`_
+    * ``'AW13-IJ05'``: `Geruo A IJ05-R2 GIA Models <https://doi.org/10.1093/gji/ggs030>`_
     * ``'Caron'``: `Caron JPL GIA Assimilation <https://doi.org/10.1002/2017GL076644>`_
     * ``'ICE6G-D'``: `ICE-6G Version-D GIA Models <https://doi.org/10.1002/2016JB013844>`_
     * ``'ascii'``: reformatted GIA in ascii format

doc/source/user_guide/gfz_isdc_grace_ftp.md

@@ -3,6 +3,8 @@ gfz_isdc_grace_ftp.py
 
 - Syncs GRACE/GRACE-FO and auxiliary data from the [GFZ Information System and Data Center (ISDC)](http://isdc.gfz-potsdam.de/grace-isdc/)
 - Syncs CSR/GFZ/JPL files for RL04/RL05/RL06 GAA/GAB/GAC/GAD/GSM (GAA and GAB are GFZ/JPL only)
+- Gets the latest technical note (TN) files
+- Gets the monthly GRACE/GRACE-FO newsletters
 - Creates an index file for each data product
 
 #### Calling Sequence
@@ -13,9 +15,6 @@ python gfz_isdc_grace_ftp.py --directory <path_to_grace_directory> --release RL0
 
 #### Command Line Options
 - `-D X`, `--directory X`: Working Data Directory
-- `-m X`, `--mission X`: Mission to sync between GRACE and GRACE-FO
-   * `'grace'`
-   * `'grace-fo'`
 - `-c X`, `--center X`: GRACE/GRACE-FO Processing Center (CSR,GFZ,JPL)
 - `-r X`, `--release X`: GRACE/GRACE-FO Data Releases to sync (RL05,RL06)
 - `-v X`, `--version X`: GRACE/GRACE-FO Level-2 Data Version to sync (0,1)

doc/source/user_guide/grace_spatial_maps.rst

@@ -58,6 +58,7 @@ Command Line Options
     * ``'ICE6G'``: `ICE-6G GIA Models <https://doi.org/10.1002/2014JB011176>`_
     * ``'Wu10'``: `Wu (2010) GIA Correction <https://doi.org/10.1038/ngeo938>`_
     * ``'AW13-ICE6G'``: `Geruo A ICE-6G GIA Models <https://doi.org/10.1093/gji/ggs030>`_
+    * ``'AW13-IJ05'``: `Geruo A IJ05-R2 GIA Models <https://doi.org/10.1093/gji/ggs030>`_
     * ``'Caron'``: `Caron JPL GIA Assimilation <https://doi.org/10.1002/2017GL076644>`_
     * ``'ICE6G-D'``: `ICE-6G Version-D GIA Models <https://doi.org/10.1002/2016JB013844>`_
     * ``'ascii'``: reformatted GIA in ascii format

doc/source/user_guide/mascon_reconstruct.rst

@@ -49,6 +49,7 @@ Command Line Options
     * ``'ICE6G'``: `ICE-6G GIA Models <https://doi.org/10.1002/2014JB011176>`_
     * ``'Wu10'``: `Wu (2010) GIA Correction <https://doi.org/10.1038/ngeo938>`_
     * ``'AW13-ICE6G'``: `Geruo A ICE-6G GIA Models <https://doi.org/10.1093/gji/ggs030>`_
+    * ``'AW13-IJ05'``: `Geruo A IJ05-R2 GIA Models <https://doi.org/10.1093/gji/ggs030>`_
     * ``'Caron'``: `Caron JPL GIA Assimilation <https://doi.org/10.1002/2017GL076644>`_
     * ``'ICE6G-D'``: `ICE-6G Version-D GIA Models <https://doi.org/10.1002/2016JB013844>`_
     * ``'ascii'``: reformatted GIA in ascii format

doc/source/user_guide/podaac_cumulus.md

@@ -2,7 +2,7 @@ podaac_cumulus.py
 =================
 
 - Syncs GRACE/GRACE-FO data from [NASA JPL PO.DAAC Cumulus AWS S3 bucket](https://podaac.jpl.nasa.gov/cloud-datasets/about)
-- S3 Cumulus syncs are only available in AWS instances in us-west-2
+- S3 Cumulus syncs are only available in AWS instances in `us-west-2`
 - Creates an index file for each data product
 
 #### Calling Sequence
@@ -16,9 +16,6 @@ python podaac_cumulus.py --user <username> --directory <path_to_grace_directory>
 - `-W X`, `--password X`: Password for NASA Earthdata Login
 - `-N X`, `--netrc X`: Path to .netrc file for authentication
 - `-D X`, `--directory X`: Working Data Directory
-- `-m X`, `--mission X`: Mission to sync between GRACE and GRACE-FO
-   * `'grace'`
-   * `'grace-fo'`
 - `-c X`, `--center X`: GRACE/GRACE-FO Processing Center (CSR,GFZ,JPL)
 - `-r X`, `--release X`: GRACE/GRACE-FO Data Releases to sync (RL06)
 - `-v X`, `--version X`: GRACE/GRACE-FO Level-2 Data Version to sync (0,1)

doc/source/user_guide/podaac_grace_sync.md

@@ -18,9 +18,6 @@ python podaac_grace_sync.py --user <username> --directory <path_to_grace_directo
 - `-W X`, `--webdav X`: WebDAV Password for JPL PO.DAAC Drive Login
 - `-N X`, `--netrc X`: Path to .netrc file for authentication
 - `-D X`, `--directory X`: Working Data Directory
-- `-m X`, `--mission X`: Mission to sync between GRACE and GRACE-FO
-   * `'grace'`
-   * `'grace-fo'`
 - `-c X`, `--center X`: GRACE/GRACE-FO Processing Center (CSR,GFZ,JPL)
 - `-r X`, `--release X`: GRACE/GRACE-FO Data Releases to sync (RL06)
 - `-v X`, `--version X`: GRACE/GRACE-FO Level-2 Data Version to sync (0,1)

doc/source/user_guide/scale_grace_maps.rst

@@ -60,6 +60,7 @@ Command Line Options
     * ``'ICE6G'``: `ICE-6G GIA Models <https://doi.org/10.1002/2014JB011176>`_
     * ``'Wu10'``: `Wu (2010) GIA Correction <https://doi.org/10.1038/ngeo938>`_
     * ``'AW13-ICE6G'``: `Geruo A ICE-6G GIA Models <https://doi.org/10.1093/gji/ggs030>`_
+    * ``'AW13-IJ05'``: `Geruo A IJ05-R2 GIA Models <https://doi.org/10.1093/gji/ggs030>`_
     * ``'Caron'``: `Caron JPL GIA Assimilation <https://doi.org/10.1002/2017GL076644>`_
     * ``'ICE6G-D'``: `ICE-6G Version-D GIA Models <https://doi.org/10.1002/2016JB013844>`_
     * ``'ascii'``: reformatted GIA in ascii format

gravity_toolkit/harmonics.py

@@ -31,6 +31,7 @@
         include utf-8 encoding in reads to be windows compliant
         added GIA model reader and drift functions
         include filename attribute when modifying harmonic objects
+        allow input ascii files to have additional columns
     Updated 12/2021: logging case_insensitive_filename output for debugging
     Updated 11/2021: kwargs to index, netCDF4 and HDF5 read functions
     Updated 10/2021: using python logging for handling verbose output
@@ -211,10 +212,7 @@ def from_ascii(self, filename, **kwargs):
         self.mmax = 0
         #-- for each line in the file
         for line in file_contents:
-            if kwargs['date']:
-                l1,m1,clm1,slm1,time = rx.findall(line)
-            else:
-                l1,m1,clm1,slm1 = rx.findall(line)
+            l1,m1,clm1,slm1,*aux = rx.findall(line)
             #-- convert line degree and order to integers
             l1,m1 = np.array([l1,m1],dtype=np.int64)
             self.lmax = np.copy(l1) if (l1 > self.lmax) else self.lmax
@@ -227,17 +225,14 @@ def from_ascii(self, filename, **kwargs):
         self.slm = np.zeros((self.lmax+1,self.mmax+1))
         #-- if the ascii file contains date variables
         if kwargs['date']:
-            self.time = np.float64(time)
+            self.time = np.float64(aux[0])
             self.month = np.int64(calendar_to_grace(self.time))
             #-- adjust months to fix special cases if necessary
             self.month = adjust_months(self.month)
         #-- extract harmonics and convert to matrix
         #-- for each line in the file
         for line in file_contents:
-            if kwargs['date']:
-                l1,m1,clm1,slm1,time = rx.findall(line)
-            else:
-                l1,m1,clm1,slm1 = rx.findall(line)
+            l1,m1,clm1,slm1,*aux = rx.findall(line)
             #-- convert line degree and order to integers
             ll,mm = np.array([l1,m1],dtype=np.int64)
             #-- convert fortran exponentials if applicable
@@ -548,6 +543,7 @@ def from_GIA(self, filename, **kwargs):
             - ``'ICE6G'``: ICE-6G GIA Models
             - ``'Wu10'``: Wu (2010) GIA Correction
             - ``'AW13-ICE6G'``: Geruo A ICE-6G GIA Models
+            - ``'AW13-IJ05'``: Geruo A IJ05-R2 GIA Models
             - ``'Caron'``: Caron JPL GIA Assimilation
             - ``'ICE6G-D'``: ICE-6G Version-D GIA Models
             - ``'ascii'``: reformatted GIA in ascii format

gravity_toolkit/read_GIA_model.py

@@ -18,6 +18,7 @@
         ICE6G: ICE-6G GIA Models
         Wu10: Wu (2010) GIA Correction
         AW13-ICE6G: Geruo A ICE-6G GIA Models
+        AW13-IJ05: Geruo A IJ05-R2 GIA Models
         Caron: Caron JPL GIA Assimilation
         ICE6G-D: ICE-6G Version-D GIA Models
         ascii: reformatted GIA in ascii format
@@ -100,6 +101,7 @@
         check if GIA data file is present in file-system
         include utf-8 encoding in reads to be windows compliant
         use the maximum degree within the GIA model as the default LMAX
+        added AW13 models using IJ05-R2 ice history
     Updated 05/2021: define int/float precision to prevent deprecation warning
     Updated 04/2021: use regular expressions to find ICE6G-D header positions
     Updated 08/2020: flake8 compatible regular expression strings
@@ -153,6 +155,7 @@ def read_GIA_model(input_file, GIA=None, MMAX=None, DATAFORM=None, **kwargs):
             - ``'ICE6G'``: ICE-6G GIA Models [Peltier2015]_
             - ``'Wu10'``: Wu (2010) GIA Correction [Wu2010]_
             - ``'AW13-ICE6G'``: Geruo A ICE-6G GIA Models [A2013]_
+            - ``'AW13-IJ05'``: Geruo A IJ05-R2 GIA Models [A2013]_
             - ``'Caron'``: Caron JPL GIA Assimilation [Caron2018]_
             - ``'ICE6G-D'``: ICE-6G Version-D GIA Models [Peltier2018]_
             - ``'ascii'``: reformatted GIA in ascii format
@@ -312,6 +315,15 @@ def read_GIA_model(input_file, GIA=None, MMAX=None, DATAFORM=None, **kwargs):
         file_pattern = r'stokes\.(ice6g)[\.\_](.*?)(\.txt)?$'
         #-- default degree of truncation
         LMAX = 100 if not kwargs['LMAX'] else kwargs['LMAX']
+    elif (GIA == 'AW13-IJ05'):
+        #-- AW13-IJ05: Geruo A IJ05-R2 GIA Models
+        prefix = 'AW13_IJ05'
+        gia_Ylms['citation'] = 'A_et_al._(2013)'
+        gia_Ylms['reference'] = 'https://doi.org/10.1093/gji/ggs030'
+        #-- regular expressions file pattern
+        file_pattern = r'stokes\.(R2)_(.*?)(\_ANT)?$'
+        #-- default degree of truncation
+        LMAX = 256 if not kwargs['LMAX'] else kwargs['LMAX']
     else:
         #-- return empty GIA harmonics to degree and order
         LMAX = 60 if not kwargs['LMAX'] else kwargs['LMAX']
@@ -341,7 +353,7 @@ def read_GIA_model(input_file, GIA=None, MMAX=None, DATAFORM=None, **kwargs):
     gia_Ylms['l'],gia_Ylms['m'] = (np.arange(LMAX+1),np.arange(LMAX+1))
 
     #-- Reading GIA files (ICE-6G and Wu have more complex formats)
-    if GIA in ('IJ05-R2', 'W12a', 'SM09', 'AW13-ICE6G'):
+    if GIA in ('IJ05-R2', 'W12a', 'SM09', 'AW13-ICE6G', 'AW13-IJ05'):
         #-- AW13, IJ05, W12a, SM09
         #-- AW13 notes: file headers
         #-- IJ05 notes: need to scale by 1e-11 for geodesy-normalization
@@ -603,6 +615,11 @@ def read_GIA_model(input_file, GIA=None, MMAX=None, DATAFORM=None, **kwargs):
         #-- extract the ice history and case flags
         hist,case,sf=re.findall(file_pattern,os.path.basename(input_file)).pop()
         gia_Ylms['title'] = '{0}_{1}_{2}'.format(prefix,hist,case)
+    elif (GIA == 'AW13-IJ05'):
+        #-- AW13-IJ05: Geruo A IJ05-R2 GIA Models
+        #-- adding file specific earth parameters
+        vrs,param,aux=re.findall(file_pattern,os.path.basename(input_file)).pop()
+        gia_Ylms['title'] = '{0}_{1}_{2}'.format(prefix,vrs,param)
 
     #-- output harmonics to ascii, netCDF4 or HDF5 file
     if DATAFORM in ('ascii', 'netCDF4', 'HDF5'):

gravity_toolkit/tools.py

@@ -467,13 +467,14 @@ def select_corrections(self, **kwargs):
         # ICE6G: ICE-6G GIA Models
         # Wu10: Wu (2010) GIA Correction
         # AW13-ICE6G: Geruo A ICE-6G GIA Models
+        # AW13-IJ05: Geruo A IJ05-R2 GIA Models
         # Caron: Caron JPL GIA Assimilation
         # ICE6G-D: ICE-6G Version-D GIA Models
         # ascii: GIA reformatted to ascii
         # netCDF4: GIA reformatted to netCDF4
         # HDF5: GIA reformatted to HDF5
         gia_list = ['[None]','IJ05-R2','W12a','SM09','ICE6G',
-            'Wu10','AW13-ICE6G','Caron','ICE6G-D',
+            'Wu10','AW13-ICE6G','AW13-IJ05','Caron','ICE6G-D',
             'ascii','netCDF4','HDF5']
         self.GIA = ipywidgets.Dropdown(
             options=gia_list,

scripts/calc_mascon.py

@@ -40,6 +40,7 @@
         ICE6G: ICE-6G GIA Models
         Wu10: Wu (2010) GIA Correction
         AW13-ICE6G: Geruo A ICE-6G GIA Models
+        AW13-IJ05: Geruo A IJ05-R2 GIA Models
         Caron: Caron JPL GIA Assimilation
         ICE6G-D: ICE-6G Version-D GIA Models
         ascii: reformatted GIA in ascii format
@@ -806,6 +807,7 @@ def main():
     models['ICE6G'] = 'ICE-6G GIA Models'
     models['Wu10'] = 'Wu (2010) GIA Correction'
     models['AW13-ICE6G'] = 'Geruo A ICE-6G GIA Models'
+    models['AW13-IJ05'] = 'Geruo A IJ05-R2 GIA Models'
     models['Caron'] = 'Caron JPL GIA Assimilation'
     models['ICE6G-D'] = 'ICE-6G Version-D GIA Models'
     models['ascii'] = 'reformatted GIA in ascii format'