Get ERI test to pass

[mnlevy1981]

We should be able to configure a hybrid start to look like an exact restart and pass the COMPARE_base_hybrid phase of an ERI test. This would leave the PET test as the only remaining expected failure

[mnlevy1981]

The runs are actually bit-for-bit, but the failure is coming from something funny in the comparison script. TestStatus.log is showing

FAIL
WARNING
Expected to compare 6 hist files, but only compared 5. It's possible
that the hist_file_extension entry in config_archive.xml is not correct
for some of your components.

I think the 5 files it compares are all identical

    ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.native.0003-01-19.nc.base matched ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.native.0003-01-19.nc.hybrid
    ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.rho2.0003-01-19.nc.base matched ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.rho2.0003-01-19.nc.hybrid
    ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.static.nc.base matched ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.static.nc.hybrid
    ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.z.0003-01-19.nc.base matched ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.z.0003-01-19.nc.hybrid
    ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.cpl.hi.0003-01-20-00000.nc.base matched ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.cpl.hi.0003-01-20-00000.nc.hybrid

but there are three files that are not compared:

    File 'ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.ic.0001-01-01.nc.base' had no compare counterpart in '/glade/derecho/scratch/mlevy/ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a/run' with suffix 'hybrid'
    File 'ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.h.ocean_geometry.nc.hybrid' had no original counterpart in '/glade/derecho/scratch/mlevy/ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a/run' with suffix 'base'
    File 'ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a.mom6.ic.0003-01-01.nc.hybrid' had no original counterpart in '/glade/derecho/scratch/mlevy/ERI.TL319_t232.G_JRA.derecho_intel.mom-debug.20241031_211915_d0nu8a/run' with suffix 'base'

I see two differences when comparing to an ERS test:

1. The mom6.ic files have the same date stamp, even through the restart should be 0001-01-07:

ERS.TL319_t232_wg37.GW_JRA.derecho_intel.20241031_211915_d0nu8a.mom6.ic.0001-01-01.nc.base
ERS.TL319_t232_wg37.GW_JRA.derecho_intel.20241031_211915_d0nu8a.mom6.ic.0001-01-01.nc.base.cprnc.out
ERS.TL319_t232_wg37.GW_JRA.derecho_intel.20241031_211915_d0nu8a.mom6.ic.0001-01-01.nc.rest

2. There are mom6.h.ocean_geometry files for both base and rest

ERS.TL319_t232_wg37.GW_JRA.derecho_intel.20241031_211915_d0nu8a.mom6.h.ocean_geometry.nc.base
ERS.TL319_t232_wg37.GW_JRA.derecho_intel.20241031_211915_d0nu8a.mom6.h.ocean_geometry.nc.base.cprnc.out
ERS.TL319_t232_wg37.GW_JRA.derecho_intel.20241031_211915_d0nu8a.mom6.h.ocean_geometry.nc.rest

I'll add this issue to the agenda for the next CSEG meeting, it sure feels like a problem with the test suite rather than anything on the MOM6 side.

[billsacks]

I don't know if this is helpful, but for CISM (and maybe other components), we needed to introduce this in CIME to skip trying to compare some non-standard files that are written out at initialization:

https://github.com/ESMCI/cime/blob/16ff8d7754925f05cdd86e86791481a3d54a2f6d/CIME/hist_utils.py#L326

https://github.com/ESMCI/cime/blob/16ff8d7754925f05cdd86e86791481a3d54a2f6d/CIME/hist_utils.py#L334