Skip to content

Commit

Permalink
Update documentation and config comments
Browse files Browse the repository at this point in the history
These are based on suggested changes from @eclare108213.
  • Loading branch information
xylar committed Dec 26, 2024
1 parent e12dedc commit 7507fec
Show file tree
Hide file tree
Showing 7 changed files with 167 additions and 30 deletions.
60 changes: 59 additions & 1 deletion docs/tutorials/dev_add_task.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ analysis task that is as close as possible to the new analysis, and to copy
that existing task as a template for the new task. That is the strategy we
will demonstrate here.

.. note::

The changes will not be broadly available until the next MPAS-Analysis
release and to the broader E3SM community after the next
`E3SM-Unified <https://github.com/E3SM-Project/e3sm-unified>`_ release.

To provide a real example, we will show how we copy and modify an analysis
task used to compute the anomaly in ocean heat content
(:py:class:`~mpas_analysis.ocean.ClimatologyMapOHCAnomaly`) to instead compute
Expand All @@ -22,6 +28,22 @@ notebooks include hard-coded paths and are otherwise not easily applied to new
simulations without considerable effort. This is the motivation for adapting
the code to MPAS-Analysis.


.. note::

If one just wishes to add a new field that already exists in MPAS-Ocean or
MPAS-Seaice output, only a few of the steps below are necessary:

1. Follow step 1 to set up an ```mpas_dev``` environment.
2. Copy an existing `ocean <https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/mpas_analysis/ocean>`_
or `sea_ice <https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/mpas_analysis/sea_ice>`_
python module to a new name and edit it as needed for the new fields.
3. Follow step 6 to add config options
4. Follow step 7 to add the task to the list of tasks known to
MPAS-Analysis
5. Follow step 8 to update Analysis Tasks in the user's guide and
``api.rst`` in the developer's guide to include the new analysis task.

1. Getting started
------------------

Expand All @@ -31,6 +53,14 @@ cloning it onto the machine(s) where you will do your development, making
a worktree for the feature you will develop, creating a conda environment for
testing your new MPAS-Analysis development, and running MPAS-Analysis.

.. note::

Make sure you follow the tutorial for developers, not for users, since the
tutorial for users installs the latest release of MPAS-Analysis, which you
cannot modify. Similarly, changes must be tested in your own development
environment (often called ``mpas_dev``) rather than the in a shared
environment like `E3SM-Unified <https://github.com/E3SM-Project/e3sm-unified>`_.

Then, please follow the :ref:`tutorial_understand_a_task`. This will give
you a tour of the :py:class:`~mpas_analysis.ocean.ClimatologyMapOHCAnomaly`
analysis task that we will use as a starting point for developing a new task.
Expand Down Expand Up @@ -528,6 +558,9 @@ I'll create or recreate my ``mpas_dev`` environment as in
conda activate mpas_dev
python -m pip install --no-deps --no-build-isolation -e .
This last command installs the ``mpas_analysis`` package into the conda
environment.

4.1 ``ClimatologyMapBSF`` class
-------------------------------

Expand Down Expand Up @@ -1092,7 +1125,32 @@ A quick way to check if the task has been added correctly is to run:
You should see the new task in the list of tasks.


8. The full code for posterity
8. Adding documentation
-----------------------

You need to add the task to the documentation. The easiest way to do this
is to copy an existing task's documentation (the more similar, the better) in
the `tasks <https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/docs/users_guide/tasks>`_
directory and then modify it.

You also need to add the tasks class and public methods to the
`api.rst <https://github.com/MPAS-Dev/MPAS-Analysis/blob/develop/docs/developers_guide/api.rst>`_
in the developer's guide. Again, the easiest approach is to copy the section
for a similar task and modify as needed.

With the ``mpas_dev`` environment activated, you can run:

.. code-block:: bash
cd docs
make clean html
to build the docs locally in the ``_build/html`` subdirectory. When generating
documentation on HPC machines, you will want to copy the html output to the
public web space to view it, or if the web portal is being cranky, scp it to
your local machine.

9. The full code for posterity
------------------------------

Since the ``ClimatologyMapBSF`` analysis task is not in MPAS-Analysis yet and
Expand Down
33 changes: 33 additions & 0 deletions docs/tutorials/dev_getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ development, and creating a conda environment that includes the
``mpas_analysis`` package and all of its dependencies, installed in a mode
appropriate for development.

.. warning::

Beware of hyphens and underscores. The conda package name and some
environment names use a hyphen (``mpas-analysis``). The python package and
the command name use an underscore (``mpas_analysis``).

1. Getting started on GitHub
----------------------------

Expand Down Expand Up @@ -267,6 +273,16 @@ In this mode, any edits you make to the code in the worktree will be available
in the conda environment. If you run ``mpas_analysis`` on the command line,
it will know about the changes.

This command only needs to be done once after the ``mpas_dev`` environment is
built if you are not using worktrees.

.. note::

If you do use worktrees, rerun the ``python -m pip install ...`` command
each time you switch to developing a new branch, since otherwise the
version of ``mpas_analysis`` in the ``mpas_dev`` environment will be the
one you were developing previously.

.. _tutorial_dev_get_started_activ_env:

4.4 Activating the environment
Expand Down Expand Up @@ -327,6 +343,13 @@ need to follow steps 2-6 of the :ref:`tutorial_getting_started` tutorial.
7. Running MPAS-Analysis on an E3SM supported machine
-----------------------------------------------------

.. warning::

Run ``mpas_analysis`` on a compute node, not on an HPC login nodes (front
ends), because it uses too many resources to be safely run on a login node.
When using a compute node interactively, activate the ``mpas_dev``
environment, even if it was activated on the login node. Be sure to

7.1 Configuring MPAS-Analysis
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -415,6 +438,16 @@ following:
# serial but with a command)
mapParallelExec = None
If you find that new jobs are being lanuched for ncremap tasks, set:

.. code-block:: ini
[execute]
...
# "None" if ncremap should perform remapping without a command, or "srun"
# possibly with some flags if it should be run with that command
ncremapParallelExec = None
If you are running into trouble with MPAS-Analysis, such as running out of
memory, you may want to explore other config options from this section.

Expand Down
63 changes: 39 additions & 24 deletions docs/tutorials/getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,45 +5,51 @@ User: Getting Started

This tutorial walks a new user through the basics of using MPAS-Analysis.

.. warning::

Beware of hyphens and underscores. The conda package name and some
environment names use a hyphen (``mpas-analysis``). The python package and
the command name use an underscore (``mpas_analysis``).

1 Setting up a Conda Environment
---------------------------------

MPAS-Analysis relies on several packages that are only available as conda
packages from the ``conda-forge`` channel. The first step for running
MPAS-Analysis is to create a conda environment with all the needed packages.

1.1 Installing Miniconda
~~~~~~~~~~~~~~~~~~~~~~~~
1.1 Installing Miniforge3
~~~~~~~~~~~~~~~~~~~~~~~~~

If you have not yet installed Anaconda or Miniconda, you will need to begin
there. The concept behind Anaconda is that just about everything you would
need for a typical python workflow is included. The concept behind Miniconda
is that you create different environments for different purposes. This allows
for greater flexibility and tends to lead to fewer conflicts between
incompatible packages, particularly when using a channel other than the
``defaults`` supplied by Anaconda. Since we will use the ``conda-forge``
channel, the Miniconda approach is strongly recommended.
If you have not yet installed Anaconda, Miniconda or Miniforge, you will need
to begin there. The concept behind Anaconda is that just about everything you
would need for a typical python workflow is included. The concept behind
Miniconda and Miniforg3 is that you create different environments for different
purposes. This allows for greater flexibility and tends to lead to fewer
conflicts between incompatible packages, particularly when using a channel
other than the ``defaults`` supplied by Anaconda. Since we will use the
``conda-forge`` channel, the Miniforge approach is strongly recommended.

First download the `Miniconda3 installer`_ for your operating system, then run
First download the `Miniforge3 installer`_ for your operating system, then run
it:

.. code-block:: bash
$ /bin/bash Miniconda3-latest-Linux-x86_64.sh
$ /bin/bash Miniforge3-Linux-x86_64.sh
.. note::

MPAS-Analysis and many of the packages it depends on support OSX and Linux
but not Windows.
but not Windows. Typically, HPC machines need the Linux-x86_64 version.

In this tutorial, we assume that Miniconda is installed in the default location,
``~/miniconda3``. If you choose to install it somewhere else, just make sure
to make the appropriate substitution whenever you see a reference to this path
below.
In this tutorial, we assume that Miniforge3 is installed in the default
location, ``~/miniforge3``. If you choose to install it somewhere else, just
make sure to make the appropriate substitution whenever you see a reference to
this path below.

You will see prompt like this::

Do you wish the installer to initialize Miniconda3
Do you wish the installer to initialize Miniforge3
by running conda init? [yes|no]

You may wish to skip the step (answer "no") if you are working on a system
Expand All @@ -53,13 +59,13 @@ shells, this is:

.. code-block:: bash
$ source ~/miniconda3/etc/profile.d/conda.sh
$ source ~/miniforge3/etc/profile.d/conda.sh
If you use ``csh``, ``tcsh`` or related shells, this becomes:

.. code-block:: csh
> source ~/miniconda3/etc/profile.d/conda.csh
> source ~/miniforge3/etc/profile.d/conda.csh
1.2 Creating a conda environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -87,17 +93,17 @@ environment, you will need to run either for ``bash``:

.. code-block:: bash
$ source ~/miniconda3/etc/profile.d/conda.sh
$ source ~/miniforge3/etc/profile.d/conda.sh
$ conda activate mpas-analysis
or for ``csh``:

.. code-block:: csh
> source ~/miniconda3/etc/profile.d/conda.csh
> source ~/miniforge3/etc/profile.d/conda.csh
> conda activate mpas-analysis
You can skip the ``source`` command if you chose to initialize Miniconda3 so it
You can skip the ``source`` command if you chose to initialize Miniforge3 so it
loads automatically.

2 Downloading observations
Expand Down Expand Up @@ -489,6 +495,10 @@ Then, running MPAS-Analysis is as simple as:
$ mpas_analysis myrun.cfg
.. note::
Adding ``--purge`` and/or ``--verbose`` flags to the ``mpas_analysis``
command can be helpful, see the troubleshooting section below.

Typical output is the analysis is running correctly looks something like:

.. code-block:: none
Expand Down Expand Up @@ -580,6 +590,11 @@ troubleshooting these errors. Please search the documentation, Google the error
online, or get in touch with the MPAS-Analysis developer team (by
`posting an issue`_ on GitHub) if you are experiencing an error.


.. note::
When all else fails, manually remove the entire MPAS-Analysis data output
directories and the public html pages.

7.1 Purging old Analysis
~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -622,7 +637,7 @@ determine the cause of the error. If not, please include them if you are
`posting an issue`_ on GitHub.


.. _`Miniconda3 installer`: https://docs.conda.io/en/latest/miniconda.html
.. _`Miniforge3 installer`: https://github.com/conda-forge/miniforge?tab=readme-ov-file#miniforge3
.. _`conda-forge channel`: https://conda-forge.org/
.. _`config file`: https://github.com/MPAS-Dev/MPAS-Analysis/tree/main/configs
.. _`Ultra-low-res ocean and sea-ice dataset`: https://web.lcrc.anl.gov/public/e3sm/diagnostics/test_output/20200305.A_WCYCL1850.ne4_oQU480.anvil/20200305.A_WCYCL1850.ne4_oQU480.anvil.ocean_seaice.tar.gz
Expand Down
7 changes: 7 additions & 0 deletions docs/users_guide/config/climatology.rst
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,13 @@ specify a different year to use for computing anomalies::

anomalyRefYear = 249

.. note::

Simulations that branch from another run should use the start year of the
branch, not the start year of the simulation that was branched from as
``anomalyRefYear``, since MPAS-Analysis will not be able to find data from
the original run.

.. _config_remapping:

Remapping Options
Expand Down
12 changes: 10 additions & 2 deletions example.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ base_path = /path/to/diagnostics
# directory containing model results
baseDirectory = /dir/for/model/output

# Note: an absolute path can be supplied for any of these subdirectories.
# NOTE: an absolute path can be supplied for any of these subdirectories.
# A relative path is assumed to be relative to baseDirectory.
# In this example, results are assumed to be in <baseDirecory>/run

Expand Down Expand Up @@ -166,7 +166,7 @@ htmlSubdirectory = html
# mpas_analysis config.analysis --generate \
# only_ocean,no_timeSeries,timeSeriesSST
#
# Note: if an appropriate reference year isn't available for computing
# NOTE: if an appropriate reference year isn't available for computing
# anomalies, include 'no_anomaly' in the generate list
generate = ['all_publicObs']

Expand All @@ -178,6 +178,10 @@ generate = ['all_publicObs']
# the year from which to compute anomalies if not the start year of the
# simulation. This might be useful if a long spin-up cycle is performed and
# only the anomaly over a later span of years is of interest.
# NOTE: Simulations that branch from another run should use the start year of
# the branch, not the start year of the simulation that was branched
# from, since MPAS-Analysis will not be able to find data from the
# original run.
# anomalyRefYear = 249

# the first year over which to average climatalogies
Expand All @@ -193,6 +197,10 @@ endYear = 20
# the year from which to compute anomalies if not the start year of the
# simulation. This might be useful if a long spin-up cycle is performed and
# only the anomaly over a later span of years is of interest.
# NOTE: Simulations that branch from another run should use the start year of
# the branch, not the start year of the simulation that was branched
# from, since MPAS-Analysis will not be able to find data from the
# original run.
# anomalyRefYear = 249

# start and end years for timeseries analysis. Use endYear = end to indicate
Expand Down
12 changes: 10 additions & 2 deletions example_e3sm.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ mainRunName = runName
# directory containing model results
baseDirectory = /dir/for/model/output

# Note: an absolute path can be supplied for any of these subdirectories.
# NOTE: an absolute path can be supplied for any of these subdirectories.
# A relative path is assumed to be relative to baseDirectory.
# In this example, results are assumed to be in <baseDirecory>/run

Expand Down Expand Up @@ -118,7 +118,7 @@ htmlSubdirectory = html
# mpas_analysis config.analysis --generate \
# only_ocean,no_timeSeries,timeSeriesSST
#
# Note: if an appropriate reference year isn't available for computing
# NOTE: if an appropriate reference year isn't available for computing
# anomalies, include 'no_anomaly' in the generate list
generate = ['all_publicObs']

Expand All @@ -129,6 +129,10 @@ generate = ['all_publicObs']
# the year from which to compute anomalies if not the start year of the
# simulation. This might be useful if a long spin-up cycle is performed and
# only the anomaly over a later span of years is of interest.
# NOTE: Simulations that branch from another run should use the start year of
# the branch, not the start year of the simulation that was branched
# from, since MPAS-Analysis will not be able to find data from the
# original run.
# anomalyRefYear = 249

# the first year over which to average climatalogies
Expand All @@ -144,6 +148,10 @@ endYear = 20
# the year from which to compute anomalies if not the start year of the
# simulation. This might be useful if a long spin-up cycle is performed and
# only the anomaly over a later span of years is of interest.
# NOTE: Simulations that branch from another run should use the start year of
# the branch, not the start year of the simulation that was branched
# from, since MPAS-Analysis will not be able to find data from the
# original run.
# anomalyRefYear = 249

# start and end years for timeseries analysis. Use endYear = end to indicate
Expand Down
Loading

0 comments on commit 7507fec

Please sign in to comment.