From 7507fec15a552dbc3c3e2b232301bfc3a3578f1c Mon Sep 17 00:00:00 2001 From: Xylar Asay-Davis Date: Thu, 26 Dec 2024 13:34:14 +0100 Subject: [PATCH] Update documentation and config comments These are based on suggested changes from @eclare108213. --- docs/tutorials/dev_add_task.rst | 60 ++++++++++++++++++++++- docs/tutorials/dev_getting_started.rst | 33 +++++++++++++ docs/tutorials/getting_started.rst | 63 +++++++++++++++---------- docs/users_guide/config/climatology.rst | 7 +++ example.cfg | 12 ++++- example_e3sm.cfg | 12 ++++- mpas_analysis/default.cfg | 10 +++- 7 files changed, 167 insertions(+), 30 deletions(-) diff --git a/docs/tutorials/dev_add_task.rst b/docs/tutorials/dev_add_task.rst index 31135282f..245b8ff1e 100644 --- a/docs/tutorials/dev_add_task.rst +++ b/docs/tutorials/dev_add_task.rst @@ -9,6 +9,12 @@ analysis task that is as close as possible to the new analysis, and to copy that existing task as a template for the new task. That is the strategy we will demonstrate here. +.. note:: + + The changes will not be broadly available until the next MPAS-Analysis + release and to the broader E3SM community after the next + `E3SM-Unified `_ release. + To provide a real example, we will show how we copy and modify an analysis task used to compute the anomaly in ocean heat content (:py:class:`~mpas_analysis.ocean.ClimatologyMapOHCAnomaly`) to instead compute @@ -22,6 +28,22 @@ notebooks include hard-coded paths and are otherwise not easily applied to new simulations without considerable effort. This is the motivation for adapting the code to MPAS-Analysis. + +.. note:: + + If one just wishes to add a new field that already exists in MPAS-Ocean or + MPAS-Seaice output, only a few of the steps below are necessary: + + 1. Follow step 1 to set up an ```mpas_dev``` environment. + 2. Copy an existing `ocean `_ + or `sea_ice `_ + python module to a new name and edit it as needed for the new fields. + 3. Follow step 6 to add config options + 4. Follow step 7 to add the task to the list of tasks known to + MPAS-Analysis + 5. Follow step 8 to update Analysis Tasks in the user's guide and + ``api.rst`` in the developer's guide to include the new analysis task. + 1. Getting started ------------------ @@ -31,6 +53,14 @@ cloning it onto the machine(s) where you will do your development, making a worktree for the feature you will develop, creating a conda environment for testing your new MPAS-Analysis development, and running MPAS-Analysis. +.. note:: + + Make sure you follow the tutorial for developers, not for users, since the + tutorial for users installs the latest release of MPAS-Analysis, which you + cannot modify. Similarly, changes must be tested in your own development + environment (often called ``mpas_dev``) rather than the in a shared + environment like `E3SM-Unified `_. + Then, please follow the :ref:`tutorial_understand_a_task`. This will give you a tour of the :py:class:`~mpas_analysis.ocean.ClimatologyMapOHCAnomaly` analysis task that we will use as a starting point for developing a new task. @@ -528,6 +558,9 @@ I'll create or recreate my ``mpas_dev`` environment as in conda activate mpas_dev python -m pip install --no-deps --no-build-isolation -e . +This last command installs the ``mpas_analysis`` package into the conda +environment. + 4.1 ``ClimatologyMapBSF`` class ------------------------------- @@ -1092,7 +1125,32 @@ A quick way to check if the task has been added correctly is to run: You should see the new task in the list of tasks. -8. The full code for posterity +8. Adding documentation +----------------------- + +You need to add the task to the documentation. The easiest way to do this +is to copy an existing task's documentation (the more similar, the better) in +the `tasks `_ +directory and then modify it. + +You also need to add the tasks class and public methods to the +`api.rst `_ +in the developer's guide. Again, the easiest approach is to copy the section +for a similar task and modify as needed. + +With the ``mpas_dev`` environment activated, you can run: + +.. code-block:: bash + + cd docs + make clean html + +to build the docs locally in the ``_build/html`` subdirectory. When generating +documentation on HPC machines, you will want to copy the html output to the +public web space to view it, or if the web portal is being cranky, scp it to +your local machine. + +9. The full code for posterity ------------------------------ Since the ``ClimatologyMapBSF`` analysis task is not in MPAS-Analysis yet and diff --git a/docs/tutorials/dev_getting_started.rst b/docs/tutorials/dev_getting_started.rst index f79ce9a7d..4a624c988 100644 --- a/docs/tutorials/dev_getting_started.rst +++ b/docs/tutorials/dev_getting_started.rst @@ -10,6 +10,12 @@ development, and creating a conda environment that includes the ``mpas_analysis`` package and all of its dependencies, installed in a mode appropriate for development. +.. warning:: + + Beware of hyphens and underscores. The conda package name and some + environment names use a hyphen (``mpas-analysis``). The python package and + the command name use an underscore (``mpas_analysis``). + 1. Getting started on GitHub ---------------------------- @@ -267,6 +273,16 @@ In this mode, any edits you make to the code in the worktree will be available in the conda environment. If you run ``mpas_analysis`` on the command line, it will know about the changes. +This command only needs to be done once after the ``mpas_dev`` environment is +built if you are not using worktrees. + +.. note:: + + If you do use worktrees, rerun the ``python -m pip install ...`` command + each time you switch to developing a new branch, since otherwise the + version of ``mpas_analysis`` in the ``mpas_dev`` environment will be the + one you were developing previously. + .. _tutorial_dev_get_started_activ_env: 4.4 Activating the environment @@ -327,6 +343,13 @@ need to follow steps 2-6 of the :ref:`tutorial_getting_started` tutorial. 7. Running MPAS-Analysis on an E3SM supported machine ----------------------------------------------------- +.. warning:: + + Run ``mpas_analysis`` on a compute node, not on an HPC login nodes (front + ends), because it uses too many resources to be safely run on a login node. + When using a compute node interactively, activate the ``mpas_dev`` + environment, even if it was activated on the login node. Be sure to + 7.1 Configuring MPAS-Analysis ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -415,6 +438,16 @@ following: # serial but with a command) mapParallelExec = None +If you find that new jobs are being lanuched for ncremap tasks, set: + +.. code-block:: ini + [execute] + + ... + # "None" if ncremap should perform remapping without a command, or "srun" + # possibly with some flags if it should be run with that command + ncremapParallelExec = None + If you are running into trouble with MPAS-Analysis, such as running out of memory, you may want to explore other config options from this section. diff --git a/docs/tutorials/getting_started.rst b/docs/tutorials/getting_started.rst index 009feaab6..2e8ee8d9d 100644 --- a/docs/tutorials/getting_started.rst +++ b/docs/tutorials/getting_started.rst @@ -5,6 +5,12 @@ User: Getting Started This tutorial walks a new user through the basics of using MPAS-Analysis. +.. warning:: + + Beware of hyphens and underscores. The conda package name and some + environment names use a hyphen (``mpas-analysis``). The python package and + the command name use an underscore (``mpas_analysis``). + 1 Setting up a Conda Environment --------------------------------- @@ -12,38 +18,38 @@ MPAS-Analysis relies on several packages that are only available as conda packages from the ``conda-forge`` channel. The first step for running MPAS-Analysis is to create a conda environment with all the needed packages. -1.1 Installing Miniconda -~~~~~~~~~~~~~~~~~~~~~~~~ +1.1 Installing Miniforge3 +~~~~~~~~~~~~~~~~~~~~~~~~~ -If you have not yet installed Anaconda or Miniconda, you will need to begin -there. The concept behind Anaconda is that just about everything you would -need for a typical python workflow is included. The concept behind Miniconda -is that you create different environments for different purposes. This allows -for greater flexibility and tends to lead to fewer conflicts between -incompatible packages, particularly when using a channel other than the -``defaults`` supplied by Anaconda. Since we will use the ``conda-forge`` -channel, the Miniconda approach is strongly recommended. +If you have not yet installed Anaconda, Miniconda or Miniforge, you will need +to begin there. The concept behind Anaconda is that just about everything you +would need for a typical python workflow is included. The concept behind +Miniconda and Miniforg3 is that you create different environments for different +purposes. This allows for greater flexibility and tends to lead to fewer +conflicts between incompatible packages, particularly when using a channel +other than the ``defaults`` supplied by Anaconda. Since we will use the +``conda-forge`` channel, the Miniforge approach is strongly recommended. -First download the `Miniconda3 installer`_ for your operating system, then run +First download the `Miniforge3 installer`_ for your operating system, then run it: .. code-block:: bash - $ /bin/bash Miniconda3-latest-Linux-x86_64.sh + $ /bin/bash Miniforge3-Linux-x86_64.sh .. note:: MPAS-Analysis and many of the packages it depends on support OSX and Linux - but not Windows. + but not Windows. Typically, HPC machines need the Linux-x86_64 version. -In this tutorial, we assume that Miniconda is installed in the default location, -``~/miniconda3``. If you choose to install it somewhere else, just make sure -to make the appropriate substitution whenever you see a reference to this path -below. +In this tutorial, we assume that Miniforge3 is installed in the default +location, ``~/miniforge3``. If you choose to install it somewhere else, just +make sure to make the appropriate substitution whenever you see a reference to +this path below. You will see prompt like this:: - Do you wish the installer to initialize Miniconda3 + Do you wish the installer to initialize Miniforge3 by running conda init? [yes|no] You may wish to skip the step (answer "no") if you are working on a system @@ -53,13 +59,13 @@ shells, this is: .. code-block:: bash - $ source ~/miniconda3/etc/profile.d/conda.sh + $ source ~/miniforge3/etc/profile.d/conda.sh If you use ``csh``, ``tcsh`` or related shells, this becomes: .. code-block:: csh - > source ~/miniconda3/etc/profile.d/conda.csh + > source ~/miniforge3/etc/profile.d/conda.csh 1.2 Creating a conda environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -87,17 +93,17 @@ environment, you will need to run either for ``bash``: .. code-block:: bash - $ source ~/miniconda3/etc/profile.d/conda.sh + $ source ~/miniforge3/etc/profile.d/conda.sh $ conda activate mpas-analysis or for ``csh``: .. code-block:: csh - > source ~/miniconda3/etc/profile.d/conda.csh + > source ~/miniforge3/etc/profile.d/conda.csh > conda activate mpas-analysis -You can skip the ``source`` command if you chose to initialize Miniconda3 so it +You can skip the ``source`` command if you chose to initialize Miniforge3 so it loads automatically. 2 Downloading observations @@ -489,6 +495,10 @@ Then, running MPAS-Analysis is as simple as: $ mpas_analysis myrun.cfg +.. note:: + Adding ``--purge`` and/or ``--verbose`` flags to the ``mpas_analysis`` + command can be helpful, see the troubleshooting section below. + Typical output is the analysis is running correctly looks something like: .. code-block:: none @@ -580,6 +590,11 @@ troubleshooting these errors. Please search the documentation, Google the error online, or get in touch with the MPAS-Analysis developer team (by `posting an issue`_ on GitHub) if you are experiencing an error. + +.. note:: + When all else fails, manually remove the entire MPAS-Analysis data output + directories and the public html pages. + 7.1 Purging old Analysis ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -622,7 +637,7 @@ determine the cause of the error. If not, please include them if you are `posting an issue`_ on GitHub. -.. _`Miniconda3 installer`: https://docs.conda.io/en/latest/miniconda.html +.. _`Miniforge3 installer`: https://github.com/conda-forge/miniforge?tab=readme-ov-file#miniforge3 .. _`conda-forge channel`: https://conda-forge.org/ .. _`config file`: https://github.com/MPAS-Dev/MPAS-Analysis/tree/main/configs .. _`Ultra-low-res ocean and sea-ice dataset`: https://web.lcrc.anl.gov/public/e3sm/diagnostics/test_output/20200305.A_WCYCL1850.ne4_oQU480.anvil/20200305.A_WCYCL1850.ne4_oQU480.anvil.ocean_seaice.tar.gz diff --git a/docs/users_guide/config/climatology.rst b/docs/users_guide/config/climatology.rst index 83036aed3..ed3f1140f 100644 --- a/docs/users_guide/config/climatology.rst +++ b/docs/users_guide/config/climatology.rst @@ -79,6 +79,13 @@ specify a different year to use for computing anomalies:: anomalyRefYear = 249 +.. note:: + + Simulations that branch from another run should use the start year of the + branch, not the start year of the simulation that was branched from as + ``anomalyRefYear``, since MPAS-Analysis will not be able to find data from + the original run. + .. _config_remapping: Remapping Options diff --git a/example.cfg b/example.cfg index 6f358fdb7..4d5a6767d 100644 --- a/example.cfg +++ b/example.cfg @@ -100,7 +100,7 @@ base_path = /path/to/diagnostics # directory containing model results baseDirectory = /dir/for/model/output -# Note: an absolute path can be supplied for any of these subdirectories. +# NOTE: an absolute path can be supplied for any of these subdirectories. # A relative path is assumed to be relative to baseDirectory. # In this example, results are assumed to be in /run @@ -166,7 +166,7 @@ htmlSubdirectory = html # mpas_analysis config.analysis --generate \ # only_ocean,no_timeSeries,timeSeriesSST # -# Note: if an appropriate reference year isn't available for computing +# NOTE: if an appropriate reference year isn't available for computing # anomalies, include 'no_anomaly' in the generate list generate = ['all_publicObs'] @@ -178,6 +178,10 @@ generate = ['all_publicObs'] # the year from which to compute anomalies if not the start year of the # simulation. This might be useful if a long spin-up cycle is performed and # only the anomaly over a later span of years is of interest. +# NOTE: Simulations that branch from another run should use the start year of +# the branch, not the start year of the simulation that was branched +# from, since MPAS-Analysis will not be able to find data from the +# original run. # anomalyRefYear = 249 # the first year over which to average climatalogies @@ -193,6 +197,10 @@ endYear = 20 # the year from which to compute anomalies if not the start year of the # simulation. This might be useful if a long spin-up cycle is performed and # only the anomaly over a later span of years is of interest. +# NOTE: Simulations that branch from another run should use the start year of +# the branch, not the start year of the simulation that was branched +# from, since MPAS-Analysis will not be able to find data from the +# original run. # anomalyRefYear = 249 # start and end years for timeseries analysis. Use endYear = end to indicate diff --git a/example_e3sm.cfg b/example_e3sm.cfg index 8ef4de423..0e7621f6b 100644 --- a/example_e3sm.cfg +++ b/example_e3sm.cfg @@ -55,7 +55,7 @@ mainRunName = runName # directory containing model results baseDirectory = /dir/for/model/output -# Note: an absolute path can be supplied for any of these subdirectories. +# NOTE: an absolute path can be supplied for any of these subdirectories. # A relative path is assumed to be relative to baseDirectory. # In this example, results are assumed to be in /run @@ -118,7 +118,7 @@ htmlSubdirectory = html # mpas_analysis config.analysis --generate \ # only_ocean,no_timeSeries,timeSeriesSST # -# Note: if an appropriate reference year isn't available for computing +# NOTE: if an appropriate reference year isn't available for computing # anomalies, include 'no_anomaly' in the generate list generate = ['all_publicObs'] @@ -129,6 +129,10 @@ generate = ['all_publicObs'] # the year from which to compute anomalies if not the start year of the # simulation. This might be useful if a long spin-up cycle is performed and # only the anomaly over a later span of years is of interest. +# NOTE: Simulations that branch from another run should use the start year of +# the branch, not the start year of the simulation that was branched +# from, since MPAS-Analysis will not be able to find data from the +# original run. # anomalyRefYear = 249 # the first year over which to average climatalogies @@ -144,6 +148,10 @@ endYear = 20 # the year from which to compute anomalies if not the start year of the # simulation. This might be useful if a long spin-up cycle is performed and # only the anomaly over a later span of years is of interest. +# NOTE: Simulations that branch from another run should use the start year of +# the branch, not the start year of the simulation that was branched +# from, since MPAS-Analysis will not be able to find data from the +# original run. # anomalyRefYear = 249 # start and end years for timeseries analysis. Use endYear = end to indicate diff --git a/mpas_analysis/default.cfg b/mpas_analysis/default.cfg index 21db49d24..92ff9526a 100755 --- a/mpas_analysis/default.cfg +++ b/mpas_analysis/default.cfg @@ -117,7 +117,7 @@ regionMaskSubdirectory = mpas_analysis/region_masks # directory containing model results baseDirectory = /dir/for/model/output -# Note: an absolute path can be supplied for any of these subdirectories. +# NOTE: an absolute path can be supplied for any of these subdirectories. # A relative path is assumed to be relative to baseDirectory. # By default, results are assumed to be directly in baseDirectory, # i.e. /./ @@ -213,6 +213,10 @@ generate = ['all_publicObs', 'no_icebergs', 'no_climatologyMapAntarcticMelt', # the year from which to compute anomalies if not the start year of the # simulation. This might be useful if a long spin-up cycle is performed and # only the anomaly over a later span of years is of interest. +# NOTE: Simulations that branch from another run should use the start year of +# the branch, not the start year of the simulation that was branched +# from, since MPAS-Analysis will not be able to find data from the +# original run. # anomalyRefYear = 249 # the first year over which to average climatalogies @@ -293,6 +297,10 @@ subprocessCount = 1 # the year from which to compute anomalies if not the start year of the # simulation. This might be useful if a long spin-up cycle is performed and # only the anomaly over a later span of years is of interest. +# NOTE: Simulations that branch from another run should use the start year of +# the branch, not the start year of the simulation that was branched +# from, since MPAS-Analysis will not be able to find data from the +# original run. # anomalyRefYear = 249 # start and end years for timeseries analysis. Out-of-bounds values will lead