a2o_flux file missing

[YuchenSun33]

Dear all,
I often encounter an error while running experiments using the AWIESM model. Specifically, after the completion of a model year, the 'a2o_flux' file is not generated, which prevents the model from proceeding further. Additionally, an error code '1385423270818802975' appears in the log file. This error frequently occurs in the second model year. Does anyone know how to solve this issue?
The context of this error message in the log file is like this:

1784 0: couple_init: Port ID returned is: 16
1785 0: - with port status 14
1786 0: - with port data type 4
1787 0: - with decomposition paral(1) 3
1788 0: - paral(2) 4
1789 0: - paral(3) 0
1790 0: 1385423270818802975
1791 0: --------------------------------------------------------------------------
1792 0: MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
1793 0: with errorcode 0.
1794 0:
1795 0: NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
1796 0: You may or may not see output from other processes, depending on
1797 0: exactly when Open MPI kills them.
1798 0: --------------------------------------------------------------------------

You can find this log file at /work/ba0989/a270107/AWI-ESM2.5/EXP/PI_fesom288/run_32020101-32021231/log/PI_fesom288_awiesm_compute_8438919.log on Levante

Best regards,
Yuchen

[pgierz]

Ah, the famous "large oasis number" bug. If you looked into the oasis source code, you would find:

if (iam == master_task) then

      inquire(file=trim(rstfile),exist=exists)
      if (.not.exists) then
         write(nulprt,*) subname,' ERROR: file missing ',trim(rstfile)
         WRITE(nulprt,*) subname,' model :',compid,' proc :',mpi_rank_local
         CALL oasis_flush(nulprt)
         IF (labort) THEN
             WRITE(nulprt,*) subname,' abort by model :',compid,' proc :',mpi_rank_local
             CALL oasis_flush(nulprt)
             write(*,*) '1385423270818802975'; call oasis_abort_noarg()
         ENDIF
      else

It seems that a restart file is missing. Could you give some more details about your model run? Maybe one of the files is taking too long to write to disk and isn't in place yet for the second run.

[YuchenSun33]

Hi Paul,
Thank you very much for your reply.
This is a PI simulation with esm_tools at feat/pism_on_albedo branch and installed through esm_master install-awiesm-2.1-paleodyn-2.5 on Levante.
As you mentioned, one of the oasis restart files, a2o_flux, is missing after the first model year. Instead, a file named a2o_flux_recv appeared in the work directory and then moved to the restart/oasis3mct.
I had a solution from Xiaoxu regarding this problem. She suggested I can copy an a2o_flux file from another experiment. I tried her suggestion by linking another a2o_flux file from another PI experiment to my restart/oasis3mct, and the experiment can continue.
Based on my understanding, this a2o_flux is generated by esm_tools with cdo commands. I am thinking maybe we can do that manually in the work directory. For example, /work/ba0989/a270107/AWI-ESM2.5/EXP/PI_fesom288/run_32010101-32011231/work is the work directory for the first model year of my experiment which didn't generate a2o_flux, and I can find the files needed for generating the a2o_flux (maybe not all of them I am not sure about this).

PS, I started an LGM simulation with the same esm_tools and had this problem again. The experiment path is at: /work/ba0989/a270107/AWI-ESM2.5/EXP/LGM_bootstrap

[pgierz]

Hi @YuchenSun33,

No problem, we are here to help after all 😀

She suggested I can copy an a2o_flux file from another experiment.

I would not do this, as it could potentially introduce a jump in your experiment. There is no guarantee that a2o_flux produced by another run is close enough to what your simulation was doing. Our models aren't actually (mathematically) chaotic, but you might be accidentally introducing a condition that tips your climate into a different state.

Instead, we should try to find out where in the cdo chain something is going wrong, so we can correct that. There should be a log file for the tidy portion of the experiment, maybe we can get some hints there. Can you have a look and see if you find anything?

[YuchenSun33]

Thank you for your suggestion.
I noticed in the tidy log for the first model year that there is some information regarding Preparing oasis restart files from initial run... in the first part named ::: Executing the step: tidy_coupler (step [1/8] of the job: tidy). I put the output regarding the a2o_flux down here. This log file is at:
/work/ba0989/a270107/AWI-ESM2.5/EXP/PI_fesom288/log/PI_fesom288_awiesm_tidy_32010101-32011231.log

  50 Preparing oasis restart files from initial run...
  51 a2o_flux ['taux_atm', 'tauy_atm', 'taux_ica', 'tauy_ica', 'prec_atm', 'snow_atm', '     evap_atm', 'subl_atm', 'heat_atm', 'heat_ica', 'heat_swa', 'hydr_atm'] ['echam', 'e     cham', 'echam', 'echam', 'echam', 'echam', 'echam', 'echam', 'echam', 'echam', 'ech     am', 'echam'] ['echam6_ndd6', 'echam6_ndd6', 'echam6_ndd6', 'echam6_ndd6', 'echam6_     ndd6', 'echam6_ndd6', 'echam6_ndd6', 'echam6_ndd6', 'echam6_ndd6', 'echam6_ndd6', '     echam6_ndd6', 'echam6_ndd6']
  52 taux_atm-echam
  53 []
  54 tauy_atm-echam
  55 []
  56 taux_ica-echam
  57 []
  58 tauy_ica-echam
  59 []
  60 prec_atm-echam
  61 []
  62 snow_atm-echam
  63 []
  64 evap_atm-echam
  65 []
  66 subl_atm-echam
  67 []
  68 heat_atm-echam
  69 []
  70 heat_ica-echam
  71 []
  72 heat_swa-echam
  73 []
  74 hydr_atm-echam
  75 []
  76 cdo -O merge  a2o_flux
  77 rm onlyonetimestep.nc

Obviously, there are too many [] in the log file. Could this be related to the fact that I changed the name of the echam executable file?

[YuchenSun33]

I did a test by changing the echam executable file name back to echam, and this time, a2o_flux can be generated correctly. So maybe this is a bug in esm_tools. Is it possible to fix this?

[YuchenSun33]

I just figured out that if you insert the following three lines in the echam part of the yaml file, this problem can be avoided.

    executable_ori: echam6_ndd12_bootstrap
    bin_sources:
            echam_bin: "${bin_dir}/${executable_ori}"

By doing so, your echam executable file, e.g. echam6_ndd12_bootstrap, will be renamed to echam in the work directory.