compiling on GAEA failure

[jiandewang]

I got error in compiling UFS on GAEA (using the head of UFS as of today)
Lmod has detected the following error: These module(s) or extension(s) exist
but cannot be loaded as requested: "cmake/3.20.1"
Try: "module spider cmake/3.20.1" to see how to load the module(s).

I had module purge before I launch the job. Is there any step that I missed ?

error log: /lustre/f2/scratch/Jiande.Wang/FV3_RT/rt_42430/compile_001

[jkbk2004]

@jiandewang let me check. @natalie-perlin FYI: just in case hpc stack issue on gaea

[zach1221]

Could be a permission related issue. @natalie-perlin are you able to confirm the stack modules on Gaea are available for all to use?

[jiandewang]

I am still getting that error, can someone help me on this as I need to do testing on GAEA

[natalie-perlin]

I got error in compiling UFS on GAEA (using the head of UFS as of today) Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "cmake/3.20.1" Try: "module spider cmake/3.20.1" to see how to load the module(s).

I had module purge before I launch the job. Is there any step that I missed ?

error log: /lustre/f2/scratch/Jiande.Wang/FV3_RT/rt_42430/compile_001

@jiandewang @zach1221 @jkbk2004

The Lmod is likely not initialized properly. Doing module purge is not the right thing to do on Gaea, so this could be the cause. Login to Gaea is closed now, apparently due to maintenance.
What are the exact steps you take, and at what point the purge is done?

[jkbk2004]

@natalie-perlin Gaea c4 was not available and in service pretty much last week. It was back on-line from weekend, I guess. We need to make sure current hpc stack is functional.

[natalie-perlin]

@jkbk2004 @jiandewang -
Current installation of the hpc-stack (standard location) on Gaea has been verified in several tests.
See also a comment by @zach1221 on successful tests: #1683 (comment)

Errors such as
These module(s) or extension(s) exist but cannot be loaded as requested: "cmake/3.20.1" Try: "module spider cmake/3.20.1" to see how to load the module(s).
indicate improper Lmod initialization

[jkbk2004]

Maintenance is on-going. we will test on Gaea after maintenance is done.

[jiandewang]

@natalie-perlin
what I did in the past several years is
(1) switch to bash (my default shell is tcsh)
(2) module purge
(3) run rt.sh from ufs-weather-model
this worked fine for years until about two month ago when I started to encounter Lmod problem

now GAEA is back so I just had a try and still got the same issue
If you see /lustre/f2/scratch/ncep/Jiande.Wang/FV3_RT/rt_27677/compile_001
the "out" file shows
Initializing lua module environment Lmod 8.7.12, loading modules (wait...)
the "err" file shows
Lmod has detected the following error: These module(s) or extension(s) exist
but cannot be loaded as requested: "cmake/3.20.1"

apparently Lmod is not initialized here, so my question is: is there any module that I need to pre-load before run the rt.sh ?
rt.sh is supposed to do module purge and then load all needed module for the compiling and run jobs, but seems it is not doing as I expected.

[jkbk2004]

I am using bash. do you want me to check steps like module purge and load and build again. I mean to make sure about the lmod issue?

[jiandewang]

I am using bash. do you want me to check steps like module purge and load and build again. I mean to make sure about the lmod issue?

that will be great. BTW, is your default shell bash or something else ? Mine is tcsh by default

[jkbk2004]

Yes, my default is bash. Can you take a look : /lustre/f2/pdata/ncep/Jong.Kim/4jiande ?

[jkbk2004]

Oh! sounds like I can reproduce same error: Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "cmake/3.20.1"

[natalie-perlin]

@natalie-perlin what I did in the past several years is (1) switch to bash (my default shell is tcsh) (2) module purge (3) run rt.sh from ufs-weather-model this worked fine for years until about two month ago when I started to encounter Lmod problem

now GAEA is back so I just had a try and still got the same issue If you see /lustre/f2/scratch/ncep/Jiande.Wang/FV3_RT/rt_27677/compile_001 the "out" file shows Initializing lua module environment Lmod 8.7.12, loading modules (wait...) the "err" file shows Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "cmake/3.20.1"

apparently Lmod is not initialized here, so my question is: is there any module that I need to pre-load before run the rt.sh ? rt.sh is supposed to do module purge and then load all needed module for the compiling and run jobs, but seems it is not doing as I expected.

1. Please use bash shell on Gaea!

2. You could not simply do module purge on Gaea. A default module management on Gaea is not Lmod, but different module environment initialized using modules/2.3.11.4. Lmod is initialized plus all the standard/default modules are loaded when
   the script /lustre/f2/dev/role.epic/contrib/Lmod_init.sh is sourced, which is done in ./tests/module-setup.sh :

elif [[ $MACHINE_ID = gaea* ]] ; then
    # We are on GAEA.
    if ( ! eval module help > /dev/null 2>&1 ) ; then
        # We cannot simply load the module command.  The GAEA
        # /etc/profile modifies a number of module-related variables
        # before loading the module command.  Without those variables,
        # the module command fails.  Hence we actually have to source
        # /etc/profile here.
        source /etc/profile
    fi
    source /lustre/f2/dev/role.epic/contrib/Lmod_init.sh

The script ./tests/module-setup.sh itself is called from a line 131 in ./tests/rt.sh

3. Make sure you are using the new version of the ufs-weather-model

[jiandewang]

I think it is the default shell that is causing the trouble. I have asked Bin.Li, Denise, and now Jong for a test, none of them had trouble and all of their default shell is bash. But mine is tcsh. Previously I had to switch to bash manually, then do module purge, then run rt.sh, that worked fine. But this method stopped to work 2 months ago.
If you see /lustre/f2/scratch/ncep/Jiande.Wang/FV3_RT/rt_27677/compile_001/out
Initializing lua module environment Lmod 8.7.12, loading modules (wait...), it stopped there, which means Lmod is not really being loaded

[JessicaMeixner-NOAA]

i know there were problems before using tcsh on gaea a long time ago and the solution in the end was "swtich default shell to bash", this was admittedly years ago now.

[jiandewang]

i know there were problems before using tcsh on gaea a long time ago and the solution in the end was "swtich default shell to bash", this was admittedly years ago now.

just had a conversion with Bin Li who made that switch before, SA will make that switch for your account on all R&D machines in one shot, they can't do switch on one particular machine, this is something I am trying to avoid.

[jkbk2004]

If I use module purge, I get same issue. they recommend to use module rm and module unload. With these, I have no problem to repeatedly load modulefiles.

[natalie-perlin]

@jiandewang @jkbk2004 @JessicaMeixner-NOAA -

If you happen to run module purge, please close that shell and open new bash on Gaea, so you could see the default modules are loaded (~26 modules; modules/3.2.11.4 being the first, DefApps being the last). After that you could proceed with Lmod initialization, loading modules, etc.

A simple test of the ufs-weather-model being compiled on Gaea, without running RTs, using the build.sh script approach:

bash
git clone --recursive https://github.com/ufs-community/ufs-weather-model
cd ufs-weather-model/
source /lustre/f2/dev/role.epic/contrib/Lmod_init.sh
module use modulefiles
module load ufs_gaea.intel
export CMAKE_FLAGS="-DAPP=S2S -export CMAKE_FLAGS="-DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16"
./build.sh 2>&1 | tee build.log

Please see below some other piece of information that might add some insight on how and why the modules are set on Gaea in this way, and the way the solution was found for both login nodes and compute nodes.

The ways things are done when you run /lustre/f2/dev/role.epic/contrib/Lmod_init.sh: 1) First, the list of currently loaded modules is stored in a variable array; 2) modules are purged; 3) User's profile shell from the installed Lmod library is initialized; 3) default system module environment is loaded, modules/3.2.11.4; 4) the remaining modules determined in step 1 are loaded in a hierarchical order; 5) environment variable MODULESHOME is set to point to its path corresponding to Lmod.

@jiandewang - please test it again with tcsh. I've made a small adjustment to a Lmod_init.sh script in step 3: instead of user's default shell sourced from Lmod library (tcsh in your case), a BASH shell from the Lmod library is always sourced. (It may or may not be a solution)

There is another reason why we cannot explicitly specify (i.e., hard-code) modules to be loaded in step 4, and need to use a flexible way of getting the list of modules in step 1. The list of default modules differs on login nodes and on compute nodes, so there is no one-fits-all list.

[jiandewang]

@natalie-perlin thanks for the detailed instruction and explanation, it works now for my account by following the steps you told us.

[natalie-perlin]

@jiandewang @jkbk2004 -
Could this issue be marked as "Answered"? I do not seem to have permissions for that.

[jkbk2004]

@natalie-perlin Once we recover baseline on Gaea, we will close this. @zach1221 FYI: we will try Monday, I guess.

[jswhit]

@natalie-perlin the problem remains that the ufs_gaea.intel modules will not load on the c4 compute nodes (although they will on the front ends). That means the runtime libraries aren't available and the model will fail. I believe this is due to the fact that only PrgEnv-intel/6.0.10 is available on the c4 compute nodes (PrgEnv-intel/6.0.5 is not, which is what is used for compilation). I have filed a gaea help ticket on this but so far no resolution.

[natalie-perlin]

@jswhit - thank you for testing!! Yes, there is a difference in PrgEnv modules on C4. But the main issue I see is that the module intel/2021.3.0 that was used to build the hpc-stack is not available either C4 ( or C5), for that matter... What is available on all three systems is intel/2022.0.2.

The first solution that comes to mind is to rebuild the software stack with intel/2022.0.2. The PrgEnv-intel module is also loaded during the build, and thus the differences in PrgEnv-intel versions on C3 and C4/C5 would remain - but it may or may not be a dealbreaker.

[natalie-perlin]

#1724 (comment)

[natalie-perlin]

An update on software stacks built for Gaea:
#1755 (comment)

An updated stack had been prepared upgraded C3 and C4 partition on Gaea
/lustre/f2/dev/role.epic/contrib/hpc-stack/intel-classic-2022.0.2/

The ufs_gaea.intel.lua module loads the stack as following:

prepend_path("MODULEPATH","/lustre/f2/dev/role.epic/contrib/hpc-stack/intel-classic-2022.0.2/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))

load(pathJoin("intel-classic", os.getenv("intel_classic_ver") or "2022.0.2"))
load(pathJoin("cray-mpich", os.getenv("cray_mpich_ver") or "7.7.20"))
load(pathJoin("hpc-intel-classic", os.getenv("hpc_intel_classic_ver") or "2022.0.2"))
load(pathJoin("hpc-cray-mpich", os.getenv("hpc_cray_mpich_ver") or "7.7.20"))
load(pathJoin("libpng", os.getenv("libpng_ver") or "1.6.37"))

A subset of regression tests (from # ATM tests line untill the end of the list in rt.conf) has successfully completed.

[natalie-perlin]

@jkbk2004 @jiandewang -
could this discussion/question be labeled as "answered" now?

[jiandewang]

@natalie-perlin sure you can labled as answered

[natalie-perlin]

@jkbk2004 - could somebody who has permissions mark this Q. as "answered"?

[jiandewang]

answered