Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HRv4 hangs on orion and hercules #2486

Open
RuiyuSun opened this issue Oct 30, 2024 · 73 comments
Open

HRv4 hangs on orion and hercules #2486

RuiyuSun opened this issue Oct 30, 2024 · 73 comments
Labels
bug Something isn't working

Comments

@RuiyuSun
Copy link
Contributor

RuiyuSun commented Oct 30, 2024 •
edited
Loading

George V. noticed that The HRv4 does not work on Hercules or Orion. It hangs sometime after WW3 starts. No relevant message in the log files about the hanging.

To Reproduce: Run an HRv4 experiment on Hercules or Orion

Additional context

Output
@RuiyuSun RuiyuSun added the bug Something isn't working label Oct 30, 2024
@GeorgeVandenberghe-NOAA
Copy link
Collaborator

This happens at high ATM resolution C1152.

@RuiyuSun
Copy link
Contributor Author

RuiyuSun commented Nov 4, 2024

I made a HRv4 test run on orion as well. As reported previously, it hung at the beginning of the run.

The log file is at /work2/noaa/stmp/rsun/ROTDIRS/HRv4

HOMEgfs=/work/noaa/global/rsun/git/global-workflow.hr.v4 (source)
EXPDIR=/work/noaa/global/rsun/para_gfs/HRv4
COMROOT=/work2/noaa/stmp/rsun/ROTDIRS
RUNDIRS=/work2/noaa/stmp/rsun/RUNDIRS

@LarissaReames-NOAA
Copy link
Collaborator

@RuiyuSun Denise reports that the privacy settings on your directories are preventing her from accessing them. Could you check on that and report back when it's fixed so others can look at your forecast?

@RuiyuSun
Copy link
Contributor Author

RuiyuSun commented Nov 5, 2024

@DeniseWorthen I made the changes. Please try again.

@JessicaMeixner-NOAA
Copy link
Collaborator

I've made a few test runs on my end and here are some observations:

  • This also fails at C768 S2SW
  • This fails at C1152 S2S (so I do not think this is wave-grid related).

Consistently all runs I have made, also the same as @RuiyuSun runs stall out here:

    0:  fcst_initialize total time:    200.367168849800
    0:  fv3_cap: field bundles in fcstComp export state, FBCount=            8
    0:  af allco wrtComp,write_groups=           4
 9216: NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to    32768.
 9216:  &MPP_IO_NML
 9216:  HEADER_BUFFER_VAL       =       16384,
 9216:  GLOBAL_FIELD_ON_ROOT_PE = T,
 9216:  IO_CLOCKS_ON    = F,
 9216:  SHUFFLE =           0,
 9216:  DEFLATE_LEVEL   =          -1,
 9216:  CF_COMPLIANCE   = F
 9216:  /
 9216: NOTE from PE     0: MPP_IO_SET_STACK_SIZE: stack size set to     131072.
 9216: NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 16000000.
 9216:  num_files=           2
 9216:  num_file=           1 filename_base= atm output_file= netcdf_parallel
 9216:  num_file=           2 filename_base= sfc output_file= netcdf_parallel
 9216:  grid_id=            1  output_grid= gaussian_grid
 9216:  imo=        4608 jmo=        2304
 9216:  ideflate=           1
 9216:  quantize_mode=quantize_bitround quantize_nsd=           5
 9216:  zstandard_level=           0
    0:  af wrtState reconcile, FBcount=           8
    0:  af get wrtfb=output_atm_bilinear rc=           0

With high resolution runs (C768 & C1152) for various machines we've had to use different number of write grid tasks. I've tried a few and all are stalling though. This is using ESMF managed threading, so one thing to try might be moving away from that?

To run a high res test case:

git clone --recursive https://github.com/NOAA-EMC/global-workflow
cd global-workflow/sorc
./build_all.sh
./link_workflow.sh
cd ../../
mkdir testdir 
cd testdir 
source ../global-workflow/workflow/gw_setup.sh 
HPC_ACCOUNT=marine-cpu pslot=C1152t02 RUNTESTS=`pwd` ../global-workflow/workflow/create_experiment.py --yaml ../global-workflow/ci/cases/hires/C1152_S2SW.yaml

Change C1152 to C768 to run that resolution and also change your HPC_ACCOUNT, pslot, as desired. Lastly, if you want to turn off waves, you change that in C1152_S2SW.yaml. If you want to change resources, look in global-workflow/parm/config/gfs/config.ufs in the C768/C1152 section.

If you want to run S2S only, change the app in global-workflow/ci/cases/hires/C1152_S2SW.yaml

My latest run log files can be found at:
/work2/noaa/marine/jmeixner/wavesforhr5/test01/C1152t0*/COMROOT/C1152t0*/logs/2019120300/gfs_fcst_seg0.log
(several runs are in progress, but they've all been running for over an hour an all hung on the same spot, despite changing write grid tasks).

@JessicaMeixner-NOAA
Copy link
Collaborator

@GeorgeVandenberghe-NOAA suggested trying 2 write groups with 240 tasks in them. I meant to try that but tried 2 write groups with 360 tasks per group unintentionally, but I did turn on all PET files as @LarissaReames-NOAA thought that might have helpful info.

The rundirectory is here: /work2/noaa/marine/jmeixner/wavesforhr5/test01/STMP/RUNDIRS/C1152t06/gfs.2019120300/gfsfcst.2019120300/fcst.272800

The log file is here: /work2/noaa/marine/jmeixner/wavesforhr5/test01/C1152t06/COMROOT/C1152t06/logs/2019120300/gfs_fcst_seg0.log

The PET logs to me also point to write group issues. Any help with this would be greatly appreciated.

Tagging @aerorahul for awareness.

@JacobCarley-NOAA
Copy link

Thanks to everyone for the work on this. Has anyone tried this configuration with the write component off? That might help isolate where there problem is (hopefully) and then we can direct this accordingly for further debugging.

@JessicaMeixner-NOAA
Copy link
Collaborator

I have not tried this without the write component.

@DusanJovic-NOAA
Copy link
Collaborator

@JessicaMeixner-NOAA and others, I grabbed the run directory from the last experiment you ran (/work2/noaa/marine/jmeixner/wavesforhr5/test01/STMP/RUNDIRS/C1152t06/gfs.2019120300/gfsfcst.2019120300/fcst.272800), changed it to run just ATM component and converted it to run with traditional threading. It is currently running in /work2/noaa/stmp/djovic/stmp/fcst.272800, and it passed the initialization phase and finished writing 000 and 003 hour outputs successfully. I submitted the job with just 30 min wall-clock time limit, so it will fail soon.

I suggest you try running full coupled version with traditional threading if it's easy to reconfigure it.

@jiandewang
Copy link
Collaborator

some good news:
I tried HR4 tag, the only thing I changed is WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS from 20 to 10 and model is running Note my run is S2S. See log file at
/work/noaa/marine/Jiande.Wang/HERCULES/HR4/work/HR4-20191203/COMROOT/2019120300/HR4-20191203/logs/2019120300/gfsfcst_seg0.log

@jiandewang
Copy link
Collaborator

my 48hr run finished

@JessicaMeixner-NOAA
Copy link
Collaborator

@DusanJovic-NOAA I tried running without ESMF threading - but am struggling to get it set-up correctly and go through. @aerorahul is it expected that turning off esmf managed threading in the workflow should work?

I'm also trying on hercules to replicated @jiandewang's success but with S2SW.

@jiandewang
Copy link
Collaborator

I also lanched one S2SW but it's still in pending status

@JessicaMeixner-NOAA
Copy link
Collaborator

WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS=10 with S2S did not work on orion: /work2/noaa/marine/jmeixner/wavesforhr5/test01/C1152t03/COMROOT/C1152t03/logs/2019120300/gfs_fcst_seg0.log

@jiandewang
Copy link
Collaborator

mine is on hercules

@jiandewang
Copy link
Collaborator

@JessicaMeixner-NOAA my gut feeling is the issue is related to the memory/node, hercules has more than orion. Maybe you can try 5 on orion

@aerorahul
Copy link
Contributor

@DusanJovic-NOAA I tried running without ESMF threading - but am struggling to get it set-up correctly and go through. @aerorahul is it expected that turning off esmf managed threading in the workflow should work?

I'm also trying on hercules to replicated @jiandewang's success but with S2SW.

Traditional threading is not yet supported in the global-workflow as an option. We have the toggle for it, but it requires a different set of ufs_configure files and I think we are waiting for that kind of work to be in the ufs-weather-model repo.

@DusanJovic-NOAA
To run w/ traditional threading, what else did you update in the test case borrowed from @JessicaMeixner-NOAA?

@DusanJovic-NOAA
Copy link
Collaborator

DusanJovic-NOAA commented Nov 8, 2024 •
edited
Loading

I only changed ufs.configure:

  1. remove all components except ATM
  2. change globalResourceControl: from true to false
  3. change ATM_petlist_bounds: to be 0 3023 - this numbers are lower and upper bounds of MPI ranks (0 based) used by the ATM model, in this case 24166 + 2360, where 24 and 16 are layout values from input.nml and 2360 are write comp values from model_configure
  4. change ATM_omp_num_threads: from 4 to 1

And, I added job_card by copying one of the job_card from regression test run and changed:

  1. export OMP_NUM_THREADS=4 - where 4 is a number of OMP threads
  2. srun --label -n 3024 --cpus-per-task=4 ./ufs_model.x - here 3024 is a number of MPI ranks, 4 is a number of threads
  3. #SBATCH --nodes=152
    #SBATCH --ntasks-per-node=80

80 is then number of cores on hercules compute nodes
152 is the minimal number of nodes such that 152*80 >= 3024

@aerorahul
Copy link
Contributor

I only changed ufs.configure:

  1. remove all components except ATM
  2. change globalResourceControl: from true to false
  3. change ATM_petlist_bounds: to be 0 3023 - this numbers are lowe and upper bounds of MPI ranks used by the ATM model, in this case 24_16_6 + 2_360, where 24 and 16 are layout values from input.nml and 2_360 are write comp values from model_configure

And, I added job_card by copying one of the job_card from regression test run and changed:

  1. export OMP_NUM_THREADS=4 - where 4 is a number of OMP threads
  2. srun --label -n 3024 --cpus-per-task=4 ./ufs_model.x - here 3024 is a number of MPI ranks, 4 is a number of threads
  3. #SBATCH --nodes=152
    #SBATCH --ntasks-per-node=80

80 is then number of cores on hercules compute nodes 152 is the minimal number of nodes such that 152*80 >= 3024

Ok. Yes. That makes sense for the atm-only.
Does your ufs.configure have a line for

ATM_omp_num_threads:            @[atm_omp_num_threads]

@[atm_omp_num_threads] would have been 4. Did you remove it? Or does it not matter since globalResourceControl is set to false?

The original value for ATM_petlist_bounds must have been 0 755 that you changed to 0 3023, I am assuming.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

GeorgeVandenberghe-NOAA commented Nov 8, 2024 via email

@DusanJovic-NOAA
Copy link
Collaborator

I just fixed my comment about ATM_omp_num_threads:. I set it to 1 from 4, I'm not sure if it's ignored when globalResourceControl is set to false

The original value for ATM_petlist_bounds was something like 12 thousand or something like that, that included MPI ranks times 4 threads.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

GeorgeVandenberghe-NOAA commented Nov 8, 2024 via email

@aerorahul
Copy link
Contributor

@JessicaMeixner-NOAA
I think the global-workflow is coded to use the correct ufs_configure template and set the appropriate values for PETLIST_BOUNDS and OMP_NUM_THREADS in the ufs_configure file.
The default in the global-workflow is to use ESMF_THREADING = YES. I am pretty sure one could use traditional threading as well, but is an unconfirmed fact as there was still work being done to confirm traditional threading will work on WCOSS2 with the slignshot updates and whatnot. Details on that are fuzzy to me at the moment.

BLUF, you/someone from the applications team could try traditional threading and we could gain some insight on performance at those resolutions. Thanks~

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

GeorgeVandenberghe-NOAA commented Nov 8, 2024 via email

@aerorahul
Copy link
Contributor

Ok. @GeorgeVandenberghe-NOAA. Where do we employ traditional threading C768 and up? If so, we can set a flag in the global-workflow for those resolutions to use traditional threading. It should be easy enough to set that up.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

GeorgeVandenberghe-NOAA commented Nov 8, 2024 via email

@JessicaMeixner-NOAA
Copy link
Collaborator

Unfortunately I was unable to replicate @jiandewang hercules success for HR4 tag with the top of develop. Moreover, 10 write tasks per group was not a lucky number for orion either.

@JessicaMeixner-NOAA
Copy link
Collaborator

Unfortunately I was unable to replicate @jiandewang hercules success for HR4 tag with the top of develop. Moreover, 10 write tasks per group was not a lucky number for orion either.

Note this was with added waves - so this might have also failed for @jiandewang if he has used waves.

@jiandewang
Copy link
Collaborator

summary for more tests I did on HERCULES:
(1) S2S, fv3 layout=8x16, write task per group=10, runs fine, further repeated 3 cases, all fine
(2) same as (1) but layout=24x16, hang
(3) repeat (1) and (2) but S2SW, all hang

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Nov 16, 2024 •
edited
Loading

I can confirm that with my DATM-S2SW test and using the points list in Dusan's run directory, I see a initialization time of ~27 minutes

20111001       0       0 WW3 InitializeRealize time:  1631.157

/work2/noaa/stmp/dworthen/stmp/dworthen/ww3pio/datm.hr4

My previous test had used the points list I got from Jiande, which contains only 611 points vs Dusan's 4264.

I had actually looked into this long-point finding time a bit, because George had mentioned it during the scalability meetings. It looks to me that every DE is searching the global domain for whether the global point list is contained w/in the grid. But each DE has a copy of the same ntri triangle list, as well as the global point list. So why does every DE need to do the search? And, for that matter, only the nappnt processor outputs the points, which are all retrieved from global arrays (either global input fields or va). So maybe only nappnt should do the search. I could be wrong, I haven't spent a lot of time on it.

EDIT: Also note that because of the 'negative longitude' problem, WW3 actually only ends up finding ~40% of the available points anyway:

Point output requested for  1624 points

@JessicaMeixner-NOAA
Copy link
Collaborator

@DeniseWorthen - We have two issues with the long point initialization. 1. some mis-match between grid and points that is not properly taken care of (NOAA-EMC/WW3#1273) and 2. The fact that the search algorithm is not the fastest and even slower if it has to go through every one b/c of the mismatch issue. (NOAA-EMC/WW3#1179). Once issue 1 is solved if there's still an issue my plan is to pre-process this part. This is an issue and is a top priority for me to get fixed ASAP. While the slow initialization needs to be avoided for this debugging work, I believe it's a separate, unrelated issue to the hanging we're seeing on orion/hercules.

Also not to muddy the waters here, but one of the things I'm trying to work on is to run a different grid for the wave model in the g-w and get everything set-up to do some testing - I'm running into issues with the other grid where things are segfaulting on the write grid component. This is a guess - but my guess is we found magic combinations of things that worked and now that we've slightly changed things those combinations no longer work and likely are related to the hanging issues but cannot prove that. I'm trying various combinations of the write grid component tasks to see if I can't find something that works. I am doing this today because I hadn't been able to make a successful run post cactus maintenance (didn't get enough chance to say anything with certainty or rule out user error) but wanted to get things in before the maintenance on dogwood in case wcoss2 runs also became an issue for c1152 post maintenance.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Nov 16, 2024 •
edited
Loading

@JessicaMeixner-NOAA I agree the "hang" is almost certainly due to the point output search.

For the point output search, it is the is_in_ungrid call you reference in your issue (https://github.com/NOAA-EMC/WW3/blob/7705171721e825d58e1e867e552e328fc812bfdd/model/src/w3triamd.F90#L1604) is the one which may need only to be called by the nappnt processor in w3init? Each DE has a copy of the ntri array that is being searched.

    IF ( FLOUT(2) ) CALL W3IOPP ( NPT, XPT, YPT, PNAMES, IMOD )
#ifdef W3_PDLIB
    CALL DEALLOCATE_PDLIB_GLOBAL(IMOD)
#endif

EDIT---oops, just re-read your post. You don't think the hang is related to the the point search. Hmmm....I guess it depends on how long people have waited for before declaring the job "hung"?

@JessicaMeixner-NOAA
Copy link
Collaborator

JessicaMeixner-NOAA commented Nov 16, 2024 •
edited
Loading

@DeniseWorthen - Let me rephrase a little. I agree that the hangs you are seeing are due to the wave initialization - which is in urgent need of a solution. However, we can get around the wave model initialization hang issue by reducing the number of points and ensuring the points we're looking for are from 0 to 360. If you do that, I still think we're going to get model hangs on orion/hercules (for example, we still have hangs on orion with S2S, we did find a combo that worked for S2S on hercules, but I think we'll still have that hang if we add waves back in but with a different point list like I had in my cases with different sets of points). That's why I want to make sure we update the ww3_shel.inp so that we are not having the known wave initialization issue cause problems.

@JessicaMeixner-NOAA
Copy link
Collaborator

Also - udpate on my WCOSS2 runs. Reducing the total number of write grid components seems to have helped. I'll post more details on Monday.

@JessicaMeixner-NOAA
Copy link
Collaborator

Also - udpate on my WCOSS2 runs. Reducing the total number of write grid components seems to have helped. I'll post more details on Monday.

On WCOSS2 running with a different wave grid, I got a segfault (full log file is on dogwood /lfs/h2/emc/couple/noscrub/jessica.meixner/WaveUglo15km/Test03/COMROOT/Test03/logs/2020021300/gfs_fcst_seg0.log):

zeroing coupling accumulated fields at kdt=           12
nid001107.dogwood.wcoss2.ncep.noaa.gov 0: PASS: fcstRUN phase 2, n_atmsteps =               11 time is         0.792372
nid001652.dogwood.wcoss2.ncep.noaa.gov 9216:   d3d_on= F
nid001652.dogwood.wcoss2.ncep.noaa.gov: rank 9280 died from signal 9
nid001553.dogwood.wcoss2.ncep.noaa.gov 2056: forrtl: error (78): process killed (SIGTERM)

Rank 9280 is a write grid component task as
ATM_petlist_bounds: 0 10175
ATM_omp_num_threads: 4
layout = 24,16

For this run I had:
write_groups: 4
write_tasks_per_group: 60

Changing this to:
write_groups: 2
write_tasks_per_group: 120

The successful log file is here: /lfs/h2/emc/couple/noscrub/jessica.meixner/WaveUglo15km/Test04/COMROOT/Test04/logs/2020021300/gfs_fcst_seg0.log

I suspect the issues we see on WCOSS2 are similar to what we've seen on hercules/orion but manifesting in segfaults versus hanging, but I could be wrong.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Nov 18, 2024 •
edited
Loading

In the job cards I got from Rahul's sandboxes, the nodes are specified as either

#SBATCH --nodes=63-63

for C768 and

#SBATCH --nodes=82-82

for C1152.

I'm not familiar w/ this notation. What does 82-82 mean?

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

@DeniseWorthen
Copy link
Collaborator

@GeorgeVandenberghe-NOAA Since you seem to know, what does specifying the nodes like this mean?

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

@JacobCarley-NOAA
Copy link

In the job cards I got from Rahul's sandboxes, the nodes are specified as either

#SBATCH --nodes=63-63

for C768 and

#SBATCH --nodes=82-82

for C1152.

I'm not familiar w/ this notation. What does 82-82 mean?

Hi @DeniseWorthen. Here's the relevant snippet from the slurm documentation on sbatch:

-N, --nodes=[-maxnodes]|<size_string>
Request that a minimum of minnodes nodes be allocated to this job. A maximum node count may also be specified with maxnodes. If only one number is specified, this is used as both the minimum and maximum node count. Node count can be also specified as size_string. The size_string specification identifies what nodes values should be used. Multiple values may be specified using a comma separated list or with a step function by suffix containing a colon and number values with a "-" separator. For example, "--nodes=1-15:4" is equivalent to "--nodes=1,5,9,13". The partition's node limits supersede those of the job. If a job's node limits are outside of the range permitted for its associated partition, the job will be left in a PENDING state. This permits possible execution at a later time, when the partition limit is changed. If a job node limit exceeds the number of nodes configured in the partition, the job will be rejected. Note that the environment variable SLURM_JOB_NUM_NODES will be set to the count of nodes actually allocated to the job. See the ENVIRONMENT VARIABLES section for more information. If -N is not specified, the default behavior is to allocate enough nodes to satisfy the requested resources as expressed by per-job specification options, e.g. -n, -c and --gpus. The job will be allocated as many nodes as possible within the range specified and without delaying the initiation of the job. The node count specification may include a numeric value followed by a suffix of "k" (multiplies numeric value by 1,024) or "m" (multiplies numeric value by 1,048,576).
NOTE: This option cannot be used in with arbitrary distribution.

So, I'm pretty sure it's just specifying the minimum and maximum number of nodes the job can run with. In this case they are the same.

@DeniseWorthen
Copy link
Collaborator

@JacobCarley-NOAA Thanks. That makes sense.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Nov 20, 2024 •
edited
Loading

I copied Rahul's C768 run directory (also created my own fix subdir) and compiled both top-develop and the HR4 tag in debug mode using

./compile.sh hercules "-DAPP=S2SW -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8_ugwpv1 -DPDLIB=ON -DDEBUG=ON" s2sw.dev.db intel  NO NO 2>&1 | tee s2sw.dev.db.log

When I try c768s2sw_gfsfcst.sh, both dev and the tag give me a seg fault (they don't even start):

 159: [hercules-01-36:826937:0:826937] Caught signal 8 (Floating point exception: floating-point invalid operation)
6082: ==== backtrace (tid: 630246) ====
6082:  0 0x000000000005f14c ucs_callbackq_cleanup()  ???:0
6082:  1 0x000000000005f40a ucs_callbackq_cleanup()  ???:0
6082:  2 0x0000000000054d90 __GI___sigaction()  :0
6082:  3 0x0000000000048f52 ucp_proto_perf_envelope_make()  ???:0
6082:  4 0x0000000000054bbc ucp_proto_select_elem_trace()  ???:0
6082:  5 0x0000000000056261 ucp_proto_select_lookup_slow()  ???:0
6082:  6 0x0000000000056725 ucp_proto_select_short_init()  ???:0
6082:  7 0x000000000004bc1c ucp_worker_add_rkey_config()  ???:0
6082:  8 0x00000000000648ff ucp_proto_rndv_ctrl_init()  ???:0
6082:  9 0x0000000000064aff ucp_proto_rndv_rts_init()  ???:0
6082: 10 0x0000000000054a42 ucp_proto_select_elem_trace()  ???:0
6082: 11 0x0000000000056261 ucp_proto_select_lookup_slow()  ???:0
6082: 12 0x0000000000056725 ucp_proto_select_short_init()  ???:0
6082: 13 0x000000000004b789 ucp_worker_get_ep_config()  ???:0
6082: 14 0x00000000000a159c ucp_wireup_init_lanes()  ???:0
6082: 15 0x00000000000339ce ucp_ep_create_to_worker_addr()  ???:0
6082: 16 0x0000000000034b33 ucp_ep_create()  ???:0
6082: 17 0x00000000000078bb mlx_av_insert()  mlx_av.c:0
6082: 18 0x00000000006595fb fi_av_insert()  /p/pdsd/scratch/Uploads/IMPI/other/software/libfabric/linux/v1.9.0/include/rdma/fi_domain.h:414
6082: 19 0x00000000006595fb insert_addr_table_roots_only()  /build/impi/_buildspace/release/../../src/mpid/ch4/netmod/ofi/ofi_init.c:448
6082: 20 0x00000000006595fb MPIDI_OFI_mpi_init_hook()  /build/impi/_buildspace/release/../../src/mpid/ch4/netmod/ofi/ofi_init.c:1604
6082: 21 0x00000000002296f4 MPID_Init()  /build/impi/_buildspace/release/../../src/mpid/ch4/src/ch4_init.c:1544
6082: 22 0x00000000004ce935 MPIR_Init_thread()  /build/impi/_buildspace/release/../../src/mpi/init/initthread.c:175
6082: 23 0x00000000004ce935 PMPI_Init_thread()  /build/impi/_buildspace/release/../../src/mpi/init/initthread.c:318
6082: 24 0x000000000117376d ESMCI::VMK::init()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/VM/src/ESMCI_VMKernel.C:423
6082: 25 0x00000000012f9e3f ESMCI::VM::initialize()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/VM/src/ESMCI_VM.C:3200
6082: 26 0x00000000009da3c5 c_esmc_vminitialize_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/VM/interface/ESMCI_VM_F.C:1186
6082: 27 0x0000000000cc6810 esmf_vmmod_mp_esmf_vminitialize_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/VM/interface/ESMF_VM.F90:9321
6082: 28 0x0000000000b1bc47 esmf_initmod_mp_esmf_frameworkinternalinit_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Superstructure/ESMFMod/src/ESMF_Init.F90:711
6082: 29 0x0000000000b2140e esmf_initmod_mp_esmf_initialize_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Superstructure/ESMFMod/src/ESMF_Init.F90:401
6082: 30 0x0000000000431e9c MAIN__()  /work/noaa/nems/dworthen/ufs-weather-model/driver/UFS.F90:97
6082: 31 0x0000000000431abd main()  ???:0
6082: 32 0x000000000003feb0 __libc_start_call_main()  ???:0
6082: 33 0x000000000003ff60 __libc_start_main_alias_2()  :0
6082: 34 0x00000000004319d5 _start()  ???:0
6082: =================================

Run directory (hercules): /work2/noaa/stmp/dworthen/c768s2sw.2

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

For hercules my current snapshot is on/work2/noaa/noaatest/gwv/herc/hr4j/the rundir is ./dc   the source dir is ./sorc and
to build, I cd to ./sorc/ufs_model.fd, load the compilers and set
export PREFIX=/work/noaa/noaatest/gwv/herc/simstacks/simstack.1008/netcdf140.492.460.mapl241.fms2301.crtm240
 export NETP=/work/noaa/noaatest/gwv/herc/simstacks/simstack.1008/netcdf140.492.460.mapl241.fms2301.crtm240
 export CMAKE_PREFIX_PATH=/work/noaa/noaatest/gwv/herc/simstacks/simstack.1008/netcdf140.492.460.mapl241.fms2301.crtm240
 export ESMFMKFILE=/work/noaa/noaatest/gwv/herc/simstacks/simstack.1008/netcdf140.492.460.mapl241.fms2301.crtm240/ESMF_8_5_0/lib/esmf.mk
With this done, the following script
rm -rf build
mkdir build
cd build
export CMAKE_PREFIX_PATH=$NETP/fms.2024.01:$NETP
 cmake .. -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8_ugwpv1,FV3_GFS_v17_coupled_p8_ugwpv1,FV3_global_nest_v1 -DPDLIB=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Release -DMOM6SOLO=ON
make -j 8 VERBOSE=1
builds it.  I am sick and tired of broken stacks and just gave up and built my own :-( However I do think this would work with the current Hercules spack-stack.. haven't tried it.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Nov 20, 2024 •
edited
Loading

I checked again that my configuration was a copy of Rahul's c768 run directory. I used the compile in debug mode and it fails immediately w/ the error I posted above. That run directory is /work2/noaa/stmp/dworthen/c768s2sw

I then used Dusan's instructions posted earlier for using traditional threading. He did it by removing all other components except ATM, so I made a similar adjustment w/ all components included. Using the same executable, it ran for 25 minutes of calendar time. That run directory is /work2/noaa/stmp/dworthen/c768s2sw.2. I used here job_card so check the out and err files.

I haven't yet tried the 2nd case w/ a non-debug compile. I did confirm that the first case hangs w/ a release compile.

Also, I made the WW3 points list only 240 long in both cases. (See ww3_shel.nml, which is being used in my tests since it is easy then to point to a different point list.)

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

Okay on Hercules
24x32 ATM decomposition, two threads per task ESMF RESOURCE CONTROL FALSE , 4 I/O groups, 160 MPI ranks per group. 240 OCN tasks, 120 ICE tasks 1400 WAVE tasks. 32 tasks per node

On Orion 24x24 ATM decomposition 2 groups of 240 I/O tasks 240 OCN 120 ICE 998 WAVE tasks. 2 threads per task. 16 tasks per node

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

The hangs I reported earlier seem to happen at higher decompositions and resource usages. Running that down.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

The problem what we can't quickly find WHERE in the various component(s) we are getting stuck remains an issue.

@DeniseWorthen
Copy link
Collaborator

Based on my testing, the issue seems to be fundamentally one w/ using ESMF managed threading. I've been doing all my testing in /work2/noaa/stmp/dworthen/hangtests, with sub-dirs there for ESMF-managed threading (ESMFT) and traditional threading (TRADT).

I can run the test case w/ traditional threading w/ the G-W executable (from Rahu's sandbox), with my own compile and with my own debug compile.

I cannot run with ESMF managed threading either w/ the G-W executable, my own compile or my own debug compile. I've tried w/ and w/o waves. In all cases, I either get a hang or, with debug, I get the error I posted about regarding floating point exception.

@JacobCarley-NOAA I think at this point it is not a WAV issue, assuming you reduce the points list to something small. I think others are better suited to debugging it. That will allow me to return my focus to the grid-imprint issue (#2466), which I know is also very high priority.

@BrianCurtis-NOAA
Copy link
Collaborator

I wonder if there was a build option missed that is causing managed threading to not work correctly?

@BrianCurtis-NOAA
Copy link
Collaborator

What I mean is with the ESMF library built in those stacks.

@JessicaMeixner-NOAA
Copy link
Collaborator

@JacobCarley-NOAA - as a near-term work around I plan to request a feature in the global-workflow to add traditional threading to enable orion/hercules in the near term, unless you'd prefer a different path forward?

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

@JacobCarley-NOAA
Copy link

@DeniseWorthen Thanks so much for your efforts. Please proceed to return to the grid imprint issue (#2466).

@JessicaMeixner-NOAA I think the ability to run with traditional threading (no managed threading) was added to GW earlier this year (see GW Issue 2277). However, I'm not sure if it's working. If it's not, I'd recommend proceeding with opening a new issue for this feature. Since something might already exist, hopefully it's not too much of a lift to get it going. This will hopefully get you working in the short-ish term.

Now, there's still something going on that we need understand. @GeorgeVandenberghe-NOAA Would you be able to continue digging into this issue?

@JessicaMeixner-NOAA
Copy link
Collaborator

JessicaMeixner-NOAA commented Nov 22, 2024 •
edited
Loading

@JacobCarley-NOAA a comment from @aerorahul earlier in this thread:

Traditional threading is not yet supported in the global-workflow as an option. We have the toggle for it, but it requires a different set of ufs_configure files and I think we are waiting for that kind of work to be in the ufs-weather-model repo.

I'll open a g-w issue (update: g-w issue: NOAA-EMC/global-workflow#3122)

@JessicaMeixner-NOAA JessicaMeixner-NOAA mentioned this issue Nov 22, 2024
Open
@GeorgeVandenberghe-NOAA
Copy link
Collaborator

@JacobCarley-NOAA
Copy link

Thanks @GeorgeVandenberghe-NOAA! Just send me a quick note offline (email is fine) when you need a component expert to jump in and I'll be happy to coordinate accordingly.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

It looks like the hangs are related to the total number of WAVE tasks but are also related to total resource usage.

I have verified that a 16x16 decomposition (ATM) with traditional threads (two per rank) and 1400 wave ranks does not hang on either Orion or Hercules but a 24x32 decomposition with 1400 wave ranks does. 998 rank runs do get through with a 24x32 decomposition. So it looks like total job resources is a contributing issue. It isn't just a hard barrier that we can't run 1400 wave tasks on orion or hercules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@JacobCarley-NOAA @aerorahul @jiandewang @JessicaMeixner-NOAA @DeniseWorthen @RuiyuSun @DusanJovic-NOAA @LarissaReames-NOAA @GeorgeVandenberghe-NOAA @BrianCurtis-NOAA