Enable ufs-weather-model on Gaea-C6

[RatkoVasic-NOAA]

Description

Gaea C6 nodes are now available. We should enable ufs-weather-model

Solution

Enable ufs-weather-model for Gaea-C6

Alternatives

None

Related to

Depends on spack-stack installation, which is ready on C6.

[GeorgeVandenberghe-NOAA]

I have built and run it with my own software stack on Gaea C6. Kicked off, can no longer run by policy but this is a datapoint

[jkbk2004]

@BrianCurtis-NOAA Sounds a version is available at /ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0

[BrianCurtis-NOAA]

@BrianCurtis-NOAA Sounds a version is available at /ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0

Thanks! I'll add that path, do you know if there will be a modulefiles associated with that c6 stack?

[RatkoVasic-NOAA]

@BrianCurtis-NOAA the only current problem is that we don't have disk space on C6 machine (f6 filesystem) for staged data (regression tests...). So, yes, you can test /ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0 installation, but don't know where to stage RT data.

[RatkoVasic-NOAA]

We have ready spack-stack 1.6.0 on Gaea-C6:

/ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core
/ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/gsi-addon/install/modulefiles/Core
/ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core
/ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/fms-2024.01/install/modulefiles/Core

Note changes comparing to Gaea-C5:

C5:
stack-intel/2023.1.0
stack-cray-mpich/8.1.25
C6:
stack-intel/2023.2.0
stack-cray-mpich/8.1.29

[AnnaSmoot-NOAA]

I'm trying to build SRW app on Gaea C6. I've made the changes to my build_gaea_intel.lua file as shown in the previous post for spack, intel, and cray-mpich, but I still get an error for intel. I see there is also:

/ncrc/proj/epic/spack-stack/latest-ue-intel/stack-intel/2023.1.0.lua

Should I be using this path? Or do you know of some other way to fix stack-intel/2023.2.0 being unknown?

I also have some other modules that are unknown:
"jasper/2.0.32" "netcdf-c/4.9.2" "wgrib2/2.0.8" "libpng/1.6.37" "cmake/3.23.1"

Does anyone have recommendations for these?

[RatkoVasic-NOAA]

@AnnaSmoot-NOAA can you please share with me your moduolefile directory, and which modulefile you are using? I can take a look.

[AnnaSmoot-NOAA]

@RatkoVasic-NOAA Thank you for your help! I'm using this .lua at this path:

/gpfs/f6/drsa-fire2/proj-shared/Anna.Smoot/srw_app_develop/ufs-srweather-app/modulefiles/build_gaea_intel.lua

[RatkoVasic-NOAA]

I have no access to /gpfs/f6/drsa-fire2/proj-shared/
could you please copy your modulefile/ directory to /gpfs/f6/drsa-fire2/world-shared/
Thanks.

[AnnaSmoot-NOAA]

@RatkoVasic-NOAA That makes sense. It's here now:

/gpfs/f6/drsa-fire2/world-shared/Anna.Smoot/build_gaea_intel.lua

[RatkoVasic-NOAA]

@AnnaSmoot-NOAA I needed whole modulefile/ directory (or just srw_common.lua file).

[RatkoVasic-NOAA]

@AnnaSmoot-NOAA , there are two changes you need to do in build_gaea_intel.lua:

< stack_intel_ver=os.getenv("stack_intel_ver") or "2023.2.0"
---
> stack_intel_ver=os.getenv("stack_intel_ver") or "2023.1.0"
12c12
< stack_mpich_ver=os.getenv("stack_mpich_ver") or "8.1.29"
---
> stack_mpich_ver=os.getenv("stack_mpich_ver") or "8.1.25"

Try it now, and if it doesn't work, send me your srw_common.lua file.

[AnnaSmoot-NOAA]

@RatkoVasic-NOAA, I updated the build_gaea_intel.lua file and the errors changed a little:
The following module(s) are unknown: "PrgEnv-intel/8.3.3" "intel-classic/2023.1.0" "craype/2.7.20"

The updated build_gaea_intel and srw_common luas are at: /gpfs/f6/drsa-fire2/world-shared/Anna.Smoot/

[RatkoVasic-NOAA]

@AnnaSmoot-NOAA
you can take corrected build_gaea_intel.lua form here:
/ncrc/home2/Ratko.Vasic/Anna/modulefiles/build_gaea_intel.lua
That one is using old unified-env environment (should work).

If you want to use new environment (upp-addon-env), then take both files:
/ncrc/home2/Ratko.Vasic/Anna/test/modulefiles/build_gaea_intel.lua /ncrc/home2/Ratko.Vasic/Anna/test/modulefiles/srw_common.lua

[AnnaSmoot-NOAA]

@RatkoVasic-NOAA Thanks so much! I used the new environment and successfully built the SRW app.

[junwang-noaa]

@jkbk2004 May I ask if you have a time line when the UFS weather model will be ported to C6?

[jkbk2004]

@jkbk2004 May I ask if you have a time line when the UFS weather model will be ported to C6?

@junwang-noaa Internal C6 on-boarding session continues toward end of September. I think it's a reasonable to set a target migration around second week of October.

[GeorgeVandenberghe-NOAA]

I can build it with my offline stack but have encountered evidence of subtle library issues that prevent running it. I would need a testcase to validate the built result. THe issue I am running into is on Hercules with PIO in MOM6 causing file write failure aborts, reported in a separate thread and since it's my offline stack, not relevant to create a GFS issue about it. It runs fine on hercules when built with the admin stack.

…

On Thu, Sep 19, 2024 at 2:19 PM JONG KIM ***@***.***> wrote: @jkbk2004 <https://github.com/jkbk2004> May I ask if you have a time line when the UFS weather model will be ported to C6? @junwang-noaa <https://github.com/junwang-noaa> Internal C6 on-boarding session continues toward end of September. I think it's a reasonable to set a target migration around second week of October. — Reply to this email directly, view it on GitHub <#2407 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FU53DOD2QZ4SWAA36LZXLMQHAVCNFSM6AAAAABNAJMWOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRRGEZDMMJUGE> . You are receiving this because you commented.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[RatkoVasic-NOAA]

@junwang-noaa , what is missing is common disk space for C6 (for ICs and baselines RT data). We don't have yet space on /f6 file system (like we had on /f5). @ulmononian is working with admins to provide us with that space. As soon as we get it, we can enable ufs-WM on that machine. Libraries are up-to-date here.

[sanAkel]

Hi @RatkoVasic-NOAA is there any update on ⬆️?

[sanAkel]

Can anyone suggest if ⬇️ can be ignored:

No supported cpu target is set, CRAY_CPU_TARGET=x86-64 will be used.
Load a valid targeting module or set CRAY_CPU_TARGET

I used these modules:

you can take corrected build_gaea_intel.lua form here:
/ncrc/home2/Ratko.Vasic/Anna/modulefiles/build_gaea_intel.lua

and built:

export CMAKE_FLAGS="-DAPP=NG-GODAS"
./build.sh

My log file is at: /ncrc/home1/Santha.Akella/build.log

Thanks for any suggestions.

[RatkoVasic-NOAA]

Hi @RatkoVasic-NOAA is there any update on ⬆️?

We still have no access to C6 storage and we don't have CPU time. Once we get that, we'll start porting to this new machine.

[jkbk2004]

@sanAkel We can follow up in #2448.

[RatkoVasic-NOAA]

@sanAkel

Gaea6>ll -d /ncrc/home1/Santha.Akella/
drwx------ 5 Santha.Akella ncep 4096 Oct 04 2024 00:41:45 /ncrc/home1/Santha.Akella/

[sanAkel]

@sanAkel

Gaea6>ll -d /ncrc/home1/Santha.Akella/
drwx------ 5 Santha.Akella ncep 4096 Oct 04 2024 00:41:45 /ncrc/home1/Santha.Akella/

Sorry about that! Please try again. 🙏

Santha.Akella@gaea61:/ncrc/home1> pwd
/ncrc/home1
Santha.Akella@gaea61:/ncrc/home1>
Santha.Akella@gaea61:/ncrc/home1> ls -al | grep 'Santha'
drwxr-xr-x 5 Santha.Akella ncep 4096 Oct 3 20:41 Santha.Akella

[GeorgeVandenberghe-NOAA]

Are there any RDHPCS C6 projects that require ufs-weather-model or it supporting applications, to run. If so are these funded and directed projects stalled indefinitely until and unless disk space for the port and regression tests is ever provided.?

…

On Thu, Oct 3, 2024 at 8:45 PM RatkoVasic-NOAA ***@***.***> wrote: Hi @RatkoVasic-NOAA <https://github.com/RatkoVasic-NOAA> is there any update on ⬆️ <#2407 (comment)> ? We still have no access to C6 storage and we don't have CPU time. Once we get that, we'll start porting to this new machine. — Reply to this email directly, view it on GitHub <#2407 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FQ6DUSYOTDE3IEALSLZZXQJ3AVCNFSM6AAAAABNAJMWOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJSGU3TCOJSGQ> . You are receiving this because you commented.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[sanAkel]

Are there any RDHPCS C6 projects that require ufs-weather-model or it supporting applications, to run.

If so are these funded and directed
projects stalled indefinitely until and unless disk space for the port and
regression tests is ever provided.?

Yes to both!

[GeorgeVandenberghe-NOAA]

Understood. Why does senior management tolerate this?

…

On Fri, Oct 4, 2024 at 10:24 AM Santha Akella ***@***.***> wrote: Are there any RDHPCS C6 projects that require ufs-weather-model or it supporting applications, to run. If so are these funded and directed projects stalled indefinitely until and unless disk space for the port and regression tests is ever provided.? Yes to both! — Reply to this email directly, view it on GitHub <#2407 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FRLF2GI2ZPTQMCE2RDZZ2QJ3AVCNFSM6AAAAABNAJMWOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJTHAZTONBVGE> . You are receiving this because you commented.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)