environment changes not working as expected on standalone setups

[seb-wahl]

This is something that has bugged me for a long time:
For coupled setups I can put e.g. (the comment below might not be valid anymore but was one or two years ago when I put it there):

oifs:
        # Ok here's something that's not intuitive and certainly needs improvement in the next version
        # compiletime and runtime env changes have to be placed within one of the components (does not
        # matter which one) and then they are valid for all components.
        compiletime_environment_changes:
          iolibraries: geomar_libs
          choose_computer.name:
             levante:
                #compiler_mpi: intel2022_impi2021
                compiler_mpi: intel2022_openmpi
                iolibraries: system_libs
             blogin:
                compiler_mpi: intel2019_impi2019_nemo4
             glogin:
                compiler_mpi: intel2019_impi2019
             juwels:
                compiler_mpi: intel2020_psmpi2020
             nesh:
             ...

And I have the compiler_mpi: settings for this coupled setup as I need them. However this doesn't work if I want to do environment_changes in a standalone setup. If I put the following in the nemo.yaml

choose_version:
  GYRE_PISCES:
    choose_computer.name:
      blogin:
         environment_changes:
           compiler_mpi: intel2019_impi2019_nemo4
           iolibraries: geomar_libs
           add_module_actions:
             - "load gcc/9.3.0"
    requires:
    - xios-trunk
    - nemobasemodel-4.2.x

the add_module_actions: are performed correctly, but the settings for compiler_mpi, and iolibraries: are ignored.
It seems that esm_master does treat standalone setups differently than coupled setups, where you have a config/setups/setup_name/setup_name.yaml with subsections for each model component.
And please ignore my nasty comments in nemo.yaml I put there a long time ago :-)

My workaround for now is to manually change the machine.yaml before starting a standalone test run.

[mandresm]

I stumbdde across a similar issue, when working on #906. The solution was to put the environment_changes outside of the choose block and then have the switch inside (same was as you have for your compiltime_environment_changes):
https://github.com/esm-tools/esm_tools/pull/906/files#diff-67938ac13364b10a3f2b7bbb8127d0d752d1908763519e5fb89a8d52b1897331R467-R473

I think you can probably leave the environment_changes nested inside the version and still get the functionality to work:

choose_version:
  GYRE_PISCES:
    environment_changes:
      choose_computer.name:
        blogin:
           compiler_mpi: intel2019_impi2019_nemo4
           iolibraries: geomar_libs
           add_module_actions:
             - "load gcc/9.3.0"

[mandresm]

by the way, we are aware that this is far from perfect, and confusing, but there are higher priorities in the pipeline right now. I'll add this issue to the cleanup project, hopefully we can tackle the environment before version 7.

[seb-wahl]

I tried that before and it doesn't make a difference. I think the problem is that things such as add_module_actions: or choose_computer.name: work, while changing a variable such as compiler_mpi: does not work / does not have an effect at this stage as a choose_compiler_mpi: does exist in e.g. blogin.yaml and choices have already been made when (in our case) the settings in nemo.yaml are evaluated. Does this explanation make sense and if yes: Can this be fixed?

[mandresm]

I'll need to debug it as I was having the same problem with blogin but I found a solution. Can you provide a branch name or commit hash to reproduce it?

[seb-wahl]

You can use release. As you can't do esm_master get-nemo-GYRE_PISCES I put a tar file of the code on /scratch/usr/shkifmsw/tmp/nemo-GYRE_PISCES.tar.gz. To reproduce, do

ssh into blogin
tar zxf /scratch/usr/shkifmsw/tmp/nemo-GYRE_PISCES.tar.gz
esm_master comp-nemo-GYRE_PISCES/nemo

You can do Ctrl+C shortly after and then check the newly generated script comp-nemo-GYRE_PISCES_script.sh if it has the correct settings.
Let me know if you need more info

[mandresm]

I got the following:
cp: cannot open ‘/scratch/usr/shkifmsw/tmp/nemo-GYRE_PISCES.tar.gz’ for reading: Permission denied

[seb-wahl]

You were too quick. Now you have access.

[mandresm]

Haha thanks :)

[mandresm]

I can confirm that I can reproduce the problem and I have a provisional solution for it.

The *environment_changes dictionaries can be indented into choose_ blocks as in your version switch. That works well for add_module_actions and add_export_vars because they are list expansion or dictionary merges, and the composition order of the information does not matter (if the last environment information expansion is the one that takes place in your switch it does not matter because we just want to add).

For variables (non lists/dictionaries) that can trigger choose blocks defined in the <computer>.yaml are treated different: esm_environment checks first if these variables have been define in some *environment_changes command so that it knows how to solve the choose_ blocks within <computer>.yaml before doing anything else. This works, but only if the *environments_changes commands are specified in the base indentation of a section (e.g. nemo.environment_changes), and not within a choose_ block.

A tested provisional solution to your problem would be:

hlrn_compiler_mpi: intel2019_impi2019
hlrn_iolibraries: geomar_libs
environment_changes:
  choose_computer.name:
    glogin:
      compiler_mpi: ${nemo.hlrn_compiler_mpi}
      iolibraries: ${nemo.hlrn_iolibraries}
    blogin:
      compiler_mpi: ${nemo.hlrn_compiler_mpi}
      iolibraries: ${nemo.hlrn_iolibraries}

choose_version:
  GYRE_PISCES:
      hlrn_compiler_mpi: intel2019_impi2019_nemo4
      hlrn_iolibraries: geomar_libs

This works because the dependency of the variables in environment_changes triggers an early resolution of the chooses involving this variable. You could go one step further and remove the choose_computer.name from the environment_changes, and make the environment_changes be platform-agnostic, and then do all the selection of the variables with multiple indented chooses, similar to what you proposed on the second snipped of your first post.

So the general recommendation for now: try to keep the *environment_changes dictionaries indented only within component sections.

Potential solutions for the future would be:

1. the parser checks that there are no 3rd or higher order of indentation for *environment_changes variables, and throws and error if there are (faster solution)
2. rework the esm_environment to be able to handle environment-choose-resolving variables that are withing other chooses
3. refactor completely the esm_environment and design a better more-robust syntax with all the knowledge we have now (that would be my preferred option, but I don't see that happening any time soon).

[seb-wahl]

Thanks a lot for your detailed answer and efforts @mandresm This explains all the mysterious behavior I experienced with ESM-Tools and the environment_changes in the past. I'll build a clean solution based on your provisional solution for our yaml files.