Cached models (#97)

* Adding flag to bypass the server-side cached models. * Describe magnetic model handling and caching * Bump version to 0.11.3 --------- Co-authored-by: Martin Paces <[email protected]>
ESA-VirES · Sep 7, 2023 · fbd5285 · fbd5285
1 parent 0e74835
commit fbd5285
Show file tree

Hide file tree

Showing 9 changed files with 159 additions and 2 deletions.
diff --git a/docs/available_parameters.rst b/docs/available_parameters.rst
@@ -1,3 +1,5 @@
+.. _Swarm parameters:
+
 Available parameters for Swarm
 ==============================
 
@@ -208,6 +210,8 @@ SW_OPER_VOBS_1M_2\_:SecularVariation ``SiteCode,B_SV,sigma_SV``
 
 ----
 
+.. _Swarm models:
+
 ``models``
 ----------
 

diff --git a/docs/capabilities.rst b/docs/capabilities.rst
@@ -22,7 +22,7 @@ VirES provides more than just *access* to data. Some operations can be peformed
       | :py:meth:`viresclient.SwarmRequest.get_orbit_number`
       | :py:meth:`viresclient.SwarmRequest.get_times_for_orbits`
     **Geomagnetic model evaluation**
-      | Forwards evaluation of magnetic field models when a magnetic dataset is selected (e.g. ``MAGx_LR``)
+      | Forwards evaluation of magnetic field models when a magnetic dataset is selected (e.g. ``MAGx_LR``). For more detail, see :ref:`Geomagnetic model handling`.
       | :py:meth:`viresclient.SwarmRequest.available_models`
       | :py:meth:`viresclient.SwarmRequest.get_model_info`
       | `models` option in :py:meth:`viresclient.SwarmRequest.set_products`

diff --git a/docs/geomagnetic_models.rst b/docs/geomagnetic_models.rst
@@ -0,0 +1,112 @@
+Geomagnetic model handling
+==========================
+
+Model evaluation
+----------------
+
+The geomagnetic models provided by VirES are all based on spherical harmonics, though they differ in their parameterisation and time-dependence. They provide predictions of the different geomagnetic field sources (e.g. core, lithosphere, ionosphere, magnetosphere) and are generally valid near to Earth's surface (i.e. at ground level and in LEO). To select appropriate models for your use case, you should refer to the scientific publications related to these models.
+
+In VirES, we provide model evaluations calculated at the same sample points as the data products. This means that the spherical harmonic expansion is made at the time and location of each datapoint (e.g. in the case of ``MAGx_LR``, at every second). The main purpose is to provide the data-model residuals, or magnetic perturbations, useful for studying the external magnetic field, typically ionospheric in origin.
+
+*Pending:* In a future VirES release, some interpolation will be used to provide the magnetic model predictions along the 50Hz ``MAGx_HR`` products (to improve speed). The predictions at the 1Hz (``MAGx_LR``) locations will be used and a cubic interpolation performed to provide the predictions at the 50Hz locations.
+
+The magnetic data products provide the magnetic field vector as the parameter ``B_NEC`` in the NEC (North, East, Centre) frame, as well as the magnetic field intensity/magnitude (scalar), ``F``. When also requesting models from VirES (by supplying the ``models`` kwarg to :py:meth:`viresclient.SwarmRequest.set_products`), the corresponding model predictions will be returned in parameters named ``B_NEC_<model-name>`` or ``F_<model-name>``. Alternatively, the data-model residual alone, named ``B_NEC_res_<model-name>`` or ``F_res_<model-name>`` can be returned directly by also supplying the kwarg ``residuals=True``. Models should be provided as a list, like ``models=["CHAOS", "IGRF"]``.
+
+Available models
+----------------
+
+See :ref:`Available parameters for Swarm / models<Swarm models>` for the list of available models.
+
+You can use :py:meth:`viresclient.SwarmRequest.available_models` and :py:meth:`viresclient.SwarmRequest.get_model_info` to query the details of models.
+
+Composed and custom models
+--------------------------
+
+When providing ``models`` to :py:meth:`viresclient.SwarmRequest.set_products`, they can be customised:
+
+| *Rename:*
+| ``models=["Model='CHAOS-Core'"]``
+| This will provide the ``CHAOS-Core`` model renamed to ``Model``, so that the returned parameters will include ``B_NEC_Model`` instead of ``B_NEC_CHAOS-Core``.
+
+| *Compose (combine):*
+| ``models=["Model='CHAOS-Core' + 'CHAOS-Static'"]``
+| This sums together the contribution from ``CHAOS-Core`` and ``CHAOS-Static`` into a custom model called ``Model``.
+
+| *Customise:*
+| ``models=["Model='CHAOS-Core'(max_degree=20) + 'CHAOS-Static'(min_degree=21,max_degree=80)"]``
+| This limits the spherical harmonic degree used in the model calculation.
+
+Note that single and double quotes are interchangeable, and must be used sometimes in order to enclose a model name and thus distinguish usage of a hyphen (``-``) in the model name from an arithmetic minus.
+
+For more examples, see https://notebooks.vires.services/notebooks/02b__viresclient-available-data#manipulation-of-models
+
+You can query information about your selected models using :py:meth:`viresclient.SwarmRequest.get_model_info`:
+
+.. code-block:: python
+
+  from viresclient import SwarmRequest
+
+  request = SwarmRequest()
+
+  request.get_model_info(
+      models=["Model='CHAOS-Core'(max_degree=20) + 'CHAOS-Static'(min_degree=21,max_degree=80)"]
+  )
+
+.. code-block::
+
+  {'Model': {'expression': "'CHAOS-Core'(max_degree=20,min_degree=1) + 'CHAOS-Static'(max_degree=80,min_degree=21)",
+    'validity': {'start': '1997-02-07T05:23:17.067838Z',
+     'end': '2024-03-01T02:57:24.851521Z'},
+    'sources': ['CHAOS-7_static.shc',
+     'SW_OPER_MCO_SHA_2X_19970101T000000_20230807T235959_0715',
+     'SW_OPER_MCO_SHA_2X_20230808T000000_20240229T235959_0715']}}
+
+Model caching
+-------------
+
+To speed up usage of commonly used expensive models, the server stores and uses a cache of some of the model values (so that they do not always need to be evaluated from scratch). This should happen transparently so you generally do not need to worry about it, but it may be helpful to understand when the cache might *not* be used, causing data requests to take longer.
+
+.. note::
+
+  The caching mechanism can be bypassed (forcing direct evaluation of models) by supplying ``ignore_cached_models=True`` in :py:meth:`viresclient.SwarmRequest.set_products`
+
+Cached models (these are chosen as they are both expensive and commonly used)::
+
+    CHAOS-Static
+    MIO_SHA_2C-Primary
+    MIO_SHA_2C-Secondary
+    MLI_SHA_2C
+
+The predictions for these models are cached only at the positions and times defined by the following products (i.e. low resolution magnetic products)::
+
+    SW_OPER_MAGx_LR_1B (x=A,B,C)
+    GRACE_x_MAG (x=A,B)
+    GFx_OPER_FGM_ACAL_CORR (x=1,2)
+    GO_MAG_ACAL_CORR
+    GO_MAG_ACAL_CORR_ML
+    CS_OPER_MAG
+
+The logic describing when the cache is used is as follows:
+
+.. image:: https://mermaid.ink/img/pako:eNqFkk1v2zAMhv8KoVMLND20PRlDiyYxugKxW6wekCAeBkZiEgGyFEhUt6Dpf5_8kSw7zQdDfvnyISnzQ0inSGRibdwvuUXPUE1rC-l5vChSyIB0dq030SNrZx8uhyCMRveHN0ar0KsDjM_lSQzsmgMsenF8UemGjLaUWO_kScFqDxLllh6-rPz9pD21ZRi1DWCQKTA0bfFjuXHHXVA4wPxcKd2pSv-eL78H6tk_emWxzN_RxATtkUkerC0B5jfD183fLpWjANYx0G-UbPbQIMvtWcd0vblOBjsqHp9-zr7BzjsVJUMgQ5JJXR6ZXZMvvCV_9KQBbv-JtoivJ0RLf83L6XP5lEH5UsEir-C5eJ3lRV5W-TSl3w3pt8tZahY9aMvkd850P6gFxKDtBvrehluY3y0ncaUlhF034v9yxJVoyDeoVdqNj5ZRizRGQ7XI0lHRGqPhWtT2M1njTqX7zZVm50W2RhPoSmBk97a3UmTsIx1NU40bj83JRV1S0S9ht4uffwB28dYw?type=png
+    :alt: Flowchart showing the cache usage logic
+
+*Custom* configured models, e.g. ``CHAOS-Static(max_degree=80)``, are not cached and must be evaluated directly.
+
+*Composed models*, i.e. ``Model = Model1 + Model2``, will use the cache for sub-models where available. For example, choosing ``CHAOS-Core + CHAOS-Static`` will make use of the cache for ``CHAOS-Static`` (an expensive model), but will directly evaluate ``CHAOS-Core`` (a cheap model), and combine the result. The same is true for *alias* models such as ``CHAOS`` (which equates to ``CHAOS-Core + CHAOS-Static + CHAOS-MMA``).
+
+When the source products or model are updated, the cache needs to be re-generated accordingly. This means means there is some delay before the cache is available again (while the changes are still being processed). In cases where the cache has been obsoleted, the system falls back to evaluating the model directly. In short, the caching mechanism prefers model consistency over performance.
+
+Model values through HAPI
+-------------------------
+
+What is HAPI? See https://notebooks.vires.services/notebooks/02h1_hapi
+
+When accessing magnetic datasets, there are additional HAPI parameters available::
+
+    B_NEC_Model
+    F_Model
+    B_NEC_res_Model
+    F_res_Model
+
+These give, respectively, vector and scalar magnetic model values and data-model residuals using the full CHAOS model (core + lithosphere + magnetosphere). These are provided through the cache as described above.
diff --git a/docs/index.rst b/docs/index.rst
@@ -25,6 +25,7 @@ Introduction
   capabilities
   available_parameters
   available_parameters_aeolus
+  geomagnetic_models
   notebook_intro
   release_notes
 

diff --git a/docs/release_notes.rst b/docs/release_notes.rst
@@ -4,6 +4,12 @@ Release notes
 Change log
 ----------
 
+Changes from 0.11.2 to 0.11.3
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Added ``ignore_cached_models=True`` in :py:meth:`viresclient.SwarmRequest.set_products`
+- Added description of model handling in docs
+
 Changes from 0.11.1 to 0.11.2
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/src/viresclient/__init__.py b/src/viresclient/__init__.py
@@ -35,4 +35,4 @@
 from ._config import ClientConfig, set_token
 from ._data_handling import ReturnedData, ReturnedDataFile
 
-__version__ = "0.11.2"
+__version__ = "0.11.3"
diff --git a/src/viresclient/_client_swarm.py b/src/viresclient/_client_swarm.py
@@ -263,6 +263,7 @@ class SwarmWPSInputs(WPSInputs):
         "sampling_step",
         "response_type",
         "custom_shc",
+        "ignore_cached_models",
     ]
 
     def __init__(
@@ -276,6 +277,7 @@ def __init__(
         sampling_step=None,
         response_type=None,
         custom_shc=None,
+        ignore_cached_models=False,
     ):
         # Set up default values
         # Obligatory - these must be replaced before the request is made
@@ -290,6 +292,7 @@ def __init__(
         self.filters = None if filters is None else filters
         self.sampling_step = None if sampling_step is None else sampling_step
         self.custom_shc = None if custom_shc is None else custom_shc
+        self.ignore_cached_models = ignore_cached_models
 
     @property
     def collection_ids(self):
@@ -345,6 +348,17 @@ def model_expression(self, model_expression):
         else:
             raise TypeError("model_expression must be a string")
 
+    @property
+    def ignore_cached_models(self):
+        return self._ignore_cached_models
+
+    @ignore_cached_models.setter
+    def ignore_cached_models(self, value):
+        if isinstance(value, bool):
+            self._ignore_cached_models = value
+        else:
+            raise TypeError
+
     @property
     def begin_time(self):
         return self._begin_time
@@ -1532,6 +1546,7 @@ def set_products(
         auxiliaries=None,
         residuals=False,
         sampling_step=None,
+        ignore_cached_models=False,
     ):
         """Set the combination of products to retrieve.
 
@@ -1545,6 +1560,7 @@ def set_products(
             auxiliaries (list(str)): from .available_auxiliaries()
             residuals (bool): True if only returning measurement-model residual
             sampling_step (str): ISO_8601 duration, e.g. 10 seconds: PT10S, 1 minute: PT1M
+            ignore_cached_models (bool): True if cached models should be ignored and calculated on-the-fly
 
         """
         if self._collection_list is None:
@@ -1643,6 +1659,8 @@ def set_products(
         self._request_inputs.variables = list(variables)
         self._request_inputs.sampling_step = sampling_step
         self._request_inputs.custom_shc = custom_shc
+        self._request_inputs.ignore_cached_models = ignore_cached_models
+
         return self
 
     def set_range_filter(self, parameter, minimum=None, maximum=None, negate=False):

diff --git a/src/viresclient/_wps/templates/vires_fetch_filtered_data.xml b/src/viresclient/_wps/templates/vires_fetch_filtered_data.xml
@@ -15,6 +15,14 @@
         <wps:LiteralData>{{ model_expression|cdata }}</wps:LiteralData>
       </wps:Data>
     </wps:Input>
+    {% if ignore_cached_models -%}
+    <wps:Input>
+      <ows:Identifier>ignore_cached_models</ows:Identifier>
+      <wps:Data>
+        <wps:LiteralData>true</wps:LiteralData>
+      </wps:Data>
+    </wps:Input>
+    {% endif -%}
     {% endif -%}
     {% if custom_shc -%}
     <wps:Input>

diff --git a/src/viresclient/_wps/templates/vires_fetch_filtered_data_async.xml b/src/viresclient/_wps/templates/vires_fetch_filtered_data_async.xml
@@ -15,6 +15,14 @@
         <wps:LiteralData>{{ model_expression|cdata }}</wps:LiteralData>
       </wps:Data>
     </wps:Input>
+    {% if ignore_cached_models -%}
+    <wps:Input>
+      <ows:Identifier>ignore_cached_models</ows:Identifier>
+      <wps:Data>
+        <wps:LiteralData>true</wps:LiteralData>
+      </wps:Data>
+    </wps:Input>
+    {% endif -%}
     {% endif -%}
     {% if custom_shc -%}
     <wps:Input>