Missing variables in jsbach restart files in AWI-ESM-2.1

[jannitzbon]

Hi, I am trying to set up simulations with AWI-ESM-2.1-wiso as a restart from simulations I ran in March 2022. I am running into the problem that some variables cannot be found in the restart files.

  0:  Reading tslclim from unit  92
  0:  IO_read_stream: failed to read from rerun file: burned_YC_acid_ag_wood
  0:  IO_read_stream: failed to read from rerun file: burned_YC_water_ag_green
  0:  IO_read_stream: failed to read from rerun file: burned_YC_water_ag_wood
  0:  IO_read_stream: failed to read from rerun file: burned_YC_ethanol_ag_green
  0:  IO_read_stream: failed to read from rerun file: burned_YC_ethanol_ag_wood
  0:  IO_read_stream: failed to read from rerun file: box_burned_frac_diag_avg
  0:  IO_read_stream: failed to read from rerun file: dist_nitrogen_2_atmos
  0:  IO_read_stream: failed to read from rerun file: dist_nitrogen_2_sminn
  0:  IO_read_stream: failed to read from rerun file: burned_YC_acid_ag_green
  0:  IO_read_stream: failed to read from rerun file: frac_1hr_wood
  0:  IO_read_stream: failed to read from rerun file: frac_10hr_wood
  0:  IO_read_stream: failed to read from rerun file: frac_100hr_wood
  0:  IO_read_stream: failed to read from rerun file: frac_1000hr_wood
  0:  IO_read_stream: failed to read from rerun file: NI_acc
  0:  IO_read_stream: failed to read from rerun file: burned_YC_nonsoluble_ag_green
  0:  IO_read_stream: failed to read from rerun file: burned_YC_nonsoluble_ag_wood
  0:  IO_read_stream: failed to read from rerun file: burned_Cpool_litter_wood_ag
  0:  IO_read_stream: failed to read from rerun file: burned_Cpool_litter_green_ag
  0:  IO_read_stream: failed to read from rerun file: burned_Cpool_green
  0:  IO_read_stream: failed to read from rerun file: postfire_mortality
  0:  IO_read_stream: failed to read from rerun file: Box_flux_harvest_2atmos
  0: 
  0: ================================================================================
  0: 
  0: FATAL ERROR in IO_read_stream: variable expected in rerun file 'restart_awiesm_albedo_test_LGM_veg.nc' is not present: Box_flux_harvest_2atmos
  0:  FINISH called from PE: 0

As far as I can tell, I am using the same version of ECHAM/JSBACH and I also compared the respective namelists. To me it looks like my current setup is assuming a different configuration of JSBACH which requires further variables. However, I have no clue where this difference compared to my old runs could stem from.

Maybe someone has another idea where to look for or how to fix this?

Best
Jan

[pgierz]

Hi @jannitzbon,

It looks like you are trying to restart a run with dynamic vegetation from one without? Could that be the case? As far as I remember, the _veg is only used when dynamic vegetation is turned on in JSBACH, but I would need to look into the handbook to be sure.

Some info might be helpful:

• Location of the experiment?
• Which run and configuration you branched off from?

@christian-stepanek might also have some JSBACH hints.

[jannitzbon]

Hi @pgierz ,
thanks for your reply. Sorry for not providing those additional details initially.

• The current experiment is located here: /albedo/work/user/jnitzbon/experiments/awiesm-2.1-wiso/awiesm_albedo_test_LGM/

• I branched it off from: /home/jnitzbon/remote/ollie/work/ollie/jnitzbon/experiments/awiesm2.1-wiso/awiesm-2.1_wiso0_pf0_LGM

In the meantime, I found the cause for the problem. Indeed, both simulations had dynamic vegetation switched on. However, they differed for the land use change (set by lcc_forcing_type), and the nitrogen cycle (set by with_nitrogen). If I deactivate these, the model runs. Not sure why this did not cause problems in the past...

From the default namelists provided with esm_tools, it seems that the default is to activate nitrogen and use the 'transitions' model for land use change. At least the latter seems a bit strange for paleo simulations, and it might make sense to have it deactivated by default. For PI, that might be different though. In the end, it is anyway the responsibility of the user I guess...

Thanks
Jan

[jannitzbon]

Hi again @pgierz @seb-wahl
I have a follow-up question on this one. JSBACH is crashing at a similar point, but I it seems to be due to a different cause.

I managed to branch off PI simulations with AWI-ESM-2.1 on albedo. However, I am now facing problems with branching off from Mid-Holocene (MH) and Last-Interglacial (LIG) simulations done some time back by Xiaoxu. The only difference to the PI simulations are different orbital forcings, GHG levels, and a different initial vegetation cover (different jsbach.nc file). In all cases, dynamic vegetation is enabled and land use change disabled.

The following error is showing up in the log file for the MH simulation:

  0:  Reading tslclim from unit  92
  0:  IO_read_stream: failed to read from rerun file: burned_YC_acid_ag_wood
  ... (some more warning for other variables) ...
  0:  IO_read_stream: failed to read from rerun file: IR_ROS
  0: double free or corruption (!prev)
srun: error: prod-125: task 0: Aborted (core dumped)
srun: launch/slurm: _step_signal: Terminating StepId=5290318.0
  0: slurmstepd: error: *** STEP 5290318.0 ON prod-125 CANCELLED AT 2023-05-31T12:16:52 ***

However, I don't think it is related to the preceding warnings regarding the missing variables in the rerun files, since these are also showing up in the log file of the PI simulations, which runs nevertheless. I looked for double free or corruption (!prev) and it seems to be related to memory allocation using the C function malloc().

For the LIG simulations, the code crashes at the same point, but the error is slightly different, but still related to memory allocation / freeing:

  0: free(): invalid next size (normal)

I am a bit lost here, since I have no idea where to start troubleshooting. Any hint or advice would be much appreciated.
The corresponding runscripts are located here: /home/jnitzbon/scripts/runscripts/awiesm_albedo/
And the experiment folders are here: /albedo/work/user/jnitzbon/experiments/awiesm-2.1-wiso/

Thanks,Jan