AWICM3-v3.1 does not (re)start on Levante since 8 December

[mbutzin]

After 994 simulated years without (almost) any complains AWICM3-v3.1 has stopped running on 8 December. Since then restarts fail with the following error messages (I also the attach a complete compute log file):

2640 ::: Executing the step: finalize_config (step [13/20] of the job: prepare)
2641 ===============================================================================
2642 Traceback (most recent call last):
2643 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1850, in recursive_get
2644 result = config_to_search[this_config]
2645 KeyError: 'pool_dir'
2646
2647 During handling of the above exception, another exception occurred:
2648
2649 Traceback (most recent call last):
2650 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1956, in actually_find_variable
2651 var_result = recursive_get(full_config, config_elements)
2652 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1857, in recursive_get
2653 return recursive_get(result, my_config_elements)
2654 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1852, in recursive_get
2655 raise ValueError(
2656 ValueError: ("Exactly None! Couldn't find an answer for:", [])
2657
2658 During handling of the above exception, another exception occurred:
2659
2660 Traceback (most recent call last):
2661 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1850, in recursive_get
2662 result = config_to_search[this_config]
2663 KeyError: 'pool_dir'
2664
2665 During handling of the above exception, another exception occurred:
2666
2667 Traceback (most recent call last):
2668 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1964, in actually_find_variable
2669 var_result = recursive_get(full_config, config_elements)
2670 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1857, in recursive_get
2671 return recursive_get(result, my_config_elements)
2672 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1852, in recursive_get
2673 raise ValueError(
2674 ValueError: ("Exactly None! Couldn't find an answer for:", [])
2675
2676 During handling of the above exception, another exception occurred:
2677
2678 Traceback (most recent call last):
2679 File "/home/a/a270108/.local/bin/esm_runscripts", line 33, in
2680 sys.exit(load_entry_point('esm-tools', 'console_scripts', 'esm_runscripts')())
2681 File "/home/a/a270108/esm_tools/src/esm_runscripts/cli.py", line 274, in main
2682 Setup = SimulationSetup(command_line_config)
2683 File "/home/a/a270108/esm_tools/src/esm_runscripts/sim_objects.py", line 42, in init
2684 self.config = prepare.run_job(self.config)
2685 File "/home/a/a270108/esm_tools/src/esm_runscripts/prepare.py", line 17, in run_job
2686 helpers.evaluate(config, "prepare", "prepare_recipe")
2687 File "/home/a/a270108/esm_tools/src/esm_runscripts/helpers.py", line 69, in evaluate
2688 config = esm_plugin_manager.work_through_recipe(
2689 File "/home/a/a270108/esm_tools/src/esm_plugin_manager/esm_plugin_manager.py", line 141, in work_through_recipe
2690 config = getattr(submodule, workitem)(config)
2691 File "/home/a/a270108/esm_tools/src/esm_runscripts/prepare.py", line 780, in finalize_config
2692 config.finalize()
2693 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 3045, in finalize
2694 self.run_recursive_functions(self)
2695 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 3052, in run_recursive_functions
2696 recursive_run_function(
2697 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1812, in recursive_run_function
2698 right[key] = recursive_run_function(
2699 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1812, in recursive_run_function
2700 right[key] = recursive_run_function(
2701 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1780, in recursive_run_function
2702 right = func(tree + [None], right, *args, **kwargs)
2703 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1881, in find_variable
2704 var_result, var_attrs = actually_find_variable(tree, var, full_config)
2705 File "/home/a/a270108/esm_tools/src/esm_parser/esm_parser.py", line 1967, in actually_find_variable
2706 raise EsmParserError(
2707 esm_parser.esm_parser.EsmParserError: Sorry, a variable was not resolved: general.pool_dir not found

Any idea would be helpful ...

pin14_spinup_v31_awicm3_compute_29940101-29941231_2837844.log

[JanStreffing]

First glance: fesom clock does not match the restart file date.

127:  fesom oasis_enddef: COMPLETED
  0:  Survived cpl_oasis3mct_define
  0:  FESOM ---->     cpl_oasis3mct_define_unstr nsend, nrecv:           7          13
  0:  raw restart time    28855200.0000000      does not match current clock time   31533600.0000000     
  0: 1
srun: error: l10632: task 0: Exited with exit code 1
srun: launch/slurm: _step_signal: Terminating StepId=2837844.0
  0: slurmstepd: error: *** STEP 2837844.0 ON l10632 CANCELLED AT 2022-12-08T14:45:39 ***
  0: slurmstepd: error:  mpi/pmix_v3: _errhandler: l10632 [0]: pmixp_client_v2.c:211: Error handler invoked: status = -25, source = [slurm.pmix.2837844.0:0]
srun: Job step aborted: Waiting up to 302 seconds for job step to finish.
srun: error: l30191: task 768: Terminated

[ackerlar]

By renaming the compute nodes in configs/machines/all_machines.yaml

from:

levante:
        login_nodes: 'levante*'
        compute_nodes: '^l[A-Za-z0-9]+\.atos\.local$'

into:

levante:
        login_nodes: 'levante*'
        compute_nodes: '^l[A-Za-z0-9]+\.lvt\.dkrz$'

[mbutzin]

Could the other issue be related to the renamed levante nodes, that was solved here: #873 ?
Yes, this is/was the case - thank you, @ackerlar.
Just a small correction: According to #872 (@vkolatschek) the second line should read
'compute_nodes: '^l[A-Za-z0-9]+.lvt.dkrz.de$'

[mandresm]

@ackerlar The conversation on #873 is somewhat cryptic ... How was it fixed?

Alternatively release branch already has this fix implemented.

[mandresm]

Hey @mbutzin, can you confirm that the levante fix solved the problem and you have a running simulation now?

[mbutzin]

Yes, this is correct.