Compiling MONC

[leifdenby]

@stevenleeds and @ralphburton I was thinking we should work out these issues around compiling MONC. I feel like the instructions for how to compile should come with the source code itself (not be available on a wiki or file download behind a login somewhere). I was surprised to learn that MONC can be compiled with the makefile in the source tree root as well as with fcm, the MONC documentation document from Todd doesn't mention the makefile.

My questions I'd like to work out are:

1. Is the makefile approach to compiling MONC only for use on ARCHER?

2. Is it possible to couple MONC to CASIM with the makefile approach? We need CASIM for EUREC4A modelling (right?) so if it's not possible with the makefile then this needs resolving somehow

3. Do want to continue with both the makefile and fcm means of compiling MONC?

4. Are you happy with me adding a file somewhere describing the compilation process? I would like to write down both which prerequisite packages are required (i.e. which things to module load on ARCHER/ARC4) and what commands to actually run to do the compilation.

[ralphburton]

Leif, Steef
I'll make a fuller reply to this but just some updates: n.b. numbers (I,II etc) here don't refer to your questions above!
(I) The Cray team are still updating the Cray libraries and completion date (the issue has been re-classified as "in-depth") is for 4th October.
(II) However I can compile the "old" MONC 0.9.0 on Archer now using GNU.
Compilation is very straightforward, (1) load some modules, (2) after doing make in the casim directory, you set an environment variable in the main make in the monc directory to point to casim, then do make GNU. Whole thing takes a few minutes. I've tested on bomex_casim.mcf (tested in the sense that it runs and results look sensible - nothing more rigorous)
(III) I can compile (with some caveats....) the Leeds branch using the above method. However I'm not 100% happy with it yet. Will write further about this.

I'll set out the Archer GNU compilation instructions in a file, this can be added to eventually, e.g. instructions for compiling with Cray, instructions for Leeds branch, etc.

The makefiles appear very generic and don't have any Archer-specific elements as far as I can see. The modules that you load will be machine-dependent, of course.

Cheers
Ralph

[sjboeing]

Thanks Ralph and Leif: there seems to be an explicit reference to compiling with CASIM in the makefile, which makes me think it should be possible to do this. Has anybody tried this already? From what I remember with MPIC, the cray libraries are typically just a bit faster than the GNU ones.

[ralphburton]

Steef, yes see above - you compile casim then point to the casim base directory from within the monc makefile.
It works for me and the bomex_casim testcase works (as far as I can see) compiling with GNU.

Yes same with WRF - Cray is faster.

[leifdenby]

Great, thanks both. Ok I think that means that

1. No, the makefile should be useable besides on ARCHER. I wonder if anyone has tried this, on say ARC4

2. It is possible to build with CASIM. @ralphburton did you follow in specific instructions for this or did you make it up yourself? 😄 It would be great to type up these instructions I think.

Does that mean that you both feel like we could stop using FCM and use the Makefile instead? Or do you think we should try maintaining both? Currently the automated tests here on github use fcm, but I would prefer to use the makefile instead to be honest as I find makefiles simpler.

[ralphburton]

Hi all
(1) Not sure - I could try it out on Jasmin perhaps.
(2) I found that by compiling as-is, the casim modules were not built in. There do not appear to be any instructions for this aspect that I could see.....But in the makefile there is a reference to CASIM_ROOT, and once that is specified it knows where the casim modules are, and compiles with them.

I have made a short readme (should be atached?) - to be added to over time - that describes the steps on Archer.
INSTALL_ARCHER.READ.ME.TXT

I would personally prefer makefiles too, as you say, simpler! Plus, I have never used fcm. It doesn't appear to be installed on Archer (either by default or as a module)

[MarkUoLeeds]

People who use the UM on Archer have access the "shared software":

export UMDIR=/work/n02/n02/hum
export PATH=${PATH}:$UMDIR/software/bin

older versions do this also
. $UMDIR/vn$VN/$TARGET_MC/scripts/.umsetvars_$VN

/work/n02/n02/hum/software/bin/fcm

FCM 2017.10.0 (/fs2/y07/y07/umshared/software/fcm-2017.10.0)

[MarkUoLeeds]

I looked back at some MONC work I did and saw that I had run MONC on Archer in April:
/work/n02/n02/mricha/monc_sims/RCE_nocasim
but had a problem writing the final NetCDF file (never followed that up as I moved back to investigating ARC4 for Andy Ross and Craig Poku).

My build is here:
~/MONC/r7765_monc_tr
you see it came from MOSRS trunk and I used:

module load cray-netcdf-hdf5parallel
module load fftw
fcm  make -F ./fcm-make - f ./fcm-make/monc-cray-cray.cfg 

Sorry if you needed that earlier.

[leifdenby]

I looked back at some MONC work I did and saw that I had run MONC on Archer in April:

Thanks @MarkUoLeeds. So in summary: we know how to compile on ARCHER now, but the run doesn't create usable output? Is that reasonable interpretation?

[ralphburton]

Quick one - Cray team on Archer are still updating the necessary libraries needed to compile/run MONC with Cray compiler (this is classed as an "in-depth query"). Something must have changed since April. (Note: it will actually compile with Cray , but fails at run-time).
I have compiled with GNU and it seems to work and produces usable output for the bomex_casim case, 398x398 gridpoints. I'll share the directory on Archer next week (w/c/ 12th Oct)

[ralphburton]

Meeting with Mark and Chris, 27/11/20

Had a very useful general chat about MONC and Archer.

Mark doesn't have access to n02 yet but expects fcm to be available soon on n02 since the UM requires fcm. Chris noted problems with makefile dependencies on Leeds branch of Monc.

Went through MONC compilation and execution (on screen): appears to compile; but crashes with errors - error messages don't shed much light and only 1 similar reference found via a google search.

Errors at run-time are:

WARNING: libu: memcpy AMD cpuid detection failed
Mark notes that “WARINING” is a string from within the monc.exe:
mricha@eslogin003:~/MONC/r7765_monc_tr> strings !$ | grep -i cpuid
strings build/bin/monc_driver.exe | grep -i cpuid
CPUID.(EAX=0x1,ECX=0x0): %08x %08x %08x %08x
WARNING: libu: memcpy Intel cpuid detection failed
WARNING: libu: memcpy AMD cpuid detection failed

plus many of the following:
MPIR_Init_thread(547):
MPID_Init(203).......: channel initialization failed
MPID_Init(584).......: PMI2 init failed: 1

Tried the following in the meeting:
changing moncs_per_io_server=15
including all relevant modules at head of slurm script

• still not working. Same errors as above.

Went through GNU compilation - throws up GNU error already encountered (related to Archer2 GNU being later than V4). This issue has been dealt with and Chris has placed a workaround (see model_core/src/components/registry.F90 etc), in
https://github.com/Leeds-MONC/monc/pull/28/files