Model branches for Levante

[mandresm]

A list of branches from different models and components with the optimized flags for Levante and ESM-Tools compilations supported.

Please, feel free to add to this list with a comment.

Supported ESM-Tools compilations

This are the models/coupled setups supported in Levante to the date (esm_master install/get/comp- followed by the elements on this list):

• awicm3-frontiers
• awicm3-frontiers-xios
• awicm3-v3.0
• fesom-1.4
• awicm-CMIP6 (missing mesh distributions in the pool)
• awiesm-2.1

Model and component branches updated with the correct compiling flags

FESOM-1.4

• master
• fesom_recom_1.4_master

FESOM-2

• awicm-3-frontiers_parallel-restart
• awicm3-frontiers-freeze-candidate-5
• 2.1-awiesm-2.1 (not yet merged CMake: migrates changes for Levante to AWIESM 2.1 branch FESOM/fesom2#345)

OpenIFS

• In theory all branches (compiler flags set through ESM-Tools)

ECHAM

• 6.3.04p1 (tag)
• master
• 6.3.05p2-awiesm-2.1

OASIS3MCT

• 2.8
• awicm-3-frontiers (compiler flags set through ESM-Tools)
• 2.8mct-awiesm-2.1

XIOS

• master

RNFMAP

• awicm-3-frontiers (compiler flags set through ESM-Tools)

IN CONSTRUCTION...

[pgierz]

See also #630. This has all been consolidated into 6.3.05p2-awiesm-2.1 for ECHAM, 2.1-awiesm-2.1 for FESOM, and 2.8mct-awiesm-2.1 for OASIS.

[mandresm]

I've updated the list above with that info

[a270105]

I did some tests with awiesm-2.1-recom (with 33 fesom tracers) on levante:
runtime.pdf
For the last one the configuration for mistral was used and for the first two I tried different configurations for levante. With the first one (1024 proc) the model runs at a similar speed as on mistral (864 proc).

[a270105]

runtime.pdf
Sorry that the runtime column was in the second page in the last file...

[mandresm]

Thanks for the info

[mandresm]

The content of the PDF in a table:

Number of nodes	Echamnproca	Echam nprocb	fesom	Runtime per month	Notes
4	16	16	256	22 min
7	24	24	288	19 min	Not filling up full nodes
8	32	16	512	11 min

[a270105]

One update:

Number of nodes	Echamnproca	Echam nprocb	fesom	Runtime per month	Runtime of fesom	Node hour per year
4	16	16	256	22 min	15 min	18.4
8	32	16	512	11 min	7 min	18.9
12	32	32	512	9 min	7 min	21.6

[mandresm]

Thanks for the update @a270105, this is for AWIESM-REcoM right? Are those numbers what you would expect from Mistral for similar experiments?

[xshi-awi]

Here some numbers about the model speed (with isotope).

Number of nodes	Echam nproca	Echam nprocb	fesom	Runtime per year	Node hour per year
6	32	16	256	40 min	4
8	32	16	512	37 min	4.93
8	48	16	256	40 min	5.33
9	48	16	384	24.1 min (best)	3.615 (best)
10	48	16	512	28 min	4.67
13	48	24	512	26.23 min	5.68
16	48	32	512	Segmentation fault	Segmentation fault

[mandresm]

Thanks!

[pgierz]

Who is @a270105 by the way?
If you are running benchmarks, please let me know so this can be coordinated. I will open a separate discussion thread for that, as here we are only listing supported branches, i.e. you have a running model with the current defaults. Wall time is in my view a separate topic.

[a270105]

I am Ying. With Miguel's help last week I did modifications in cmake files and did some tests with awiesm-2.1-recom on levante. Miguel told me to report the runtime here.

[mandresm]

yeah, I asked both Ying and Xiaoxu to report here. I thought it would be nice to have it all together and we can update the 1st post with the supported branches

[JanStreffing]

Not sure if on interest, but here are the AWI-CM3 benchmarks on Levante (TCO95L91-CORE2) (no recom or wiso of course)

Total Nodes	OIFS MPI	OIFS OMP	FESOM	XIOS	Runoff	Runtime per year	NH/y
5	128	2	128	128	128	106 min	8
16	256	4	768	128	128	20 min	5.3
32	384	8	768	128	128	13 min	6.7

For TCO319L137-DART I had some very slow FESOM2 performance when I tried a month ago. I'd be interested if anyone has run a larger setup recently.

[mandresm]

A comment on OMP: I was on a talk by ATOS about Levante this week, and they mentioned that due to the NUMA domains of the new EPYC 7003 MILAN Zen3 CPUs, 2 is not a recommended number of threads for OpenMP, but instead 4 or 8 (2 sockets per node, each CPU with 64 cores, each node with 8 NUMA domains, each comprising 8 cores). I understood that, for some reason, they are favouring 4 threads instead of 8. They will upload the video of the presentation, then we can have a look and see if that's the case. So for larger simulations, if 8 runs slow, I'd try 4 and increasing the number of MPI threads, maybe that helps?

[xshi-awi]

LGM simulation (without wiso) can get a maximum speed of 16.47 min/year, here is a summary:

Number of nodes	Echam nproca	Echam nprocb	fesom	Runtime per year	Node hour per year
7	32	16	288	22.8 min	2.66
9	32	24	288	20.63 min	3.0945
7	32	16	384	24.73 min	2.885
8	32	16	512	35 min	4.67
9	32	24	384	18.8 min	2.82
9	48	16	384	16.47 min (best)	2.47 (best)
10	32	24	512	18.8 min	3.13

[chrisdane]

awicm-1.0-recom test runs with different fesom core mesh distributions (from /work/ab0995/a270152/testing_pool/AWICM/FESOM1/MESHES/core/dist):

computer	fesom1 nproc	average realtime hours per model year (min-max)	average node hours per model year	nnodes	number of runs
mistral	288	3.37 (3.23-3.94)	110	24,36	77
levante	128	>5h (exceeded 5h time limit)	NA	6	1
levante	256	3.64	25	7	1
levante	288	3.14	22	7	1
levante	384	2.10	17	8	1
levante	512	2.05	18	9	1

[chrisdane]

yes @mandresm, I was wondering if we should test further nproc for fesom1?

[mandresm]

Tido has copied our mesh distributions to the pool (/pool/data/AWICM/FESOM1/MESHES/core/dist/), and there you can find a 512p partition (4 nodes), you can test that one.

[mandresm]

and let me know if you'd like me to generate more partitions. You can also generate them yourself following the README.rst that we included here: /pool/data/AWICM/FESOM1/MESHES/core/dist/README.rst

[chrisdane]

Added 512; seems to converge

[pgierz]

As a side info: scaling tests of AWI-ESM showed that going with more than 9 total nodes does not seem to increase performance.

[chrisdane]

1 year of awicm-1.0-recom restarted from a mistral simulation; physics look good; levante runs are runtime-ordered:

id	computer	fesom1 nproc	echam6 nproca	echam6 nprocb	average runtime per model year in hours (% wrt to ref mean)	average node hours per model year	nnodes	number of runs
ref	mistral	288	24	24	3.37 (min=3.23, max=3.94)	110	24,36	77
1	levante	512	48	16	2.28 (68%)	22.82	10	1
2	levante	512	32	24	2.32 (69%)	23.19	10	1
3	levante	512	24	32	2.48 (74%)	24.81	10	1
4	levante	512	16	32	2.49 (74%)	19.89	8	1
5	levante	384	48	16	2.92 (87%)	26.26	9	1
6	levante	384	24	32	2.96 (88%)	26.68	9	1
7	levante	384	32	16	3.02 (90%)	21.18	7	1
8	levante	384	16	32	3.05 (91%)	21.32	7	1
9	levante	512	32	16	3.15 (93%)	25.22	8	1
10	levante	384	32	24	4.45 (132%)	40.05	9	1

To restart from a mistral run, oasis.lresume must be false in the runscript. lresume of all other model components was true. Thanks a lot @mandresm who figured that out!