gnu tests failed in ctsm5.4.021 for the mizu test list

[ekluzek]

I happened to run the mizu testlist on Derecho with ctsm5.4.021 and v3.0.1 of mizuRoute and the following tests failed at runtime:

ERI_PS.f19_f19_rHDMAlk_mg17.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default     ( RUN)
ERS_PS.f19_f19_mg17.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default     ( RUN)
ERS_PS.f19_f19_rHDMAlk_mg17.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default     ( RUN)
ERS_PS.nldas2_nldas2_rHDMA_mnldas2.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default      ( RUN)
ERS_PS.nldas2_nldas2_rUSGS_mnldas2.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default      ( RUN)
PET_Mmpi-serial_P1x25.5x5_amazon_r05.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default    ( RUN)
PET_P256x8.nldas2_nldas2_rHDMA_mnldas2.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default  ( RUN)
PFS.f19_f19_rHDMA_mg17.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default  ( RUN)
SMS.f09_f09_rHDMAlk_mg17.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default        ( RUN)
SMS.f09_f09_rMERIT_mg17.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default ( RUN)
SMS.f19_f19_rMERIT_mg17.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default ( RUN)
SMS_D.5x5_amazon_r05.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default    ( RUN)
SMS_D_Mmpi-serial.5x5_amazon_r05.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default        ( RUN)
SMS_Mmpi-serial_D_P1x25.5x5_amazon_r05.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default  ( RUN)
SMS_P768x4.nldas2_nldas2_rMERIT_mnldas2.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default ( RUN)

The intel tests all worked (with one exception). So there must be something that gnu is telling us here.

[ekluzek]

Here's details on one test:

cesm.log:

dec2436.hsn.de.hpc.ucar.edu 0:  ROF: PIO root=           1
dec2436.hsn.de.hpc.ucar.edu 0:  /glade/work/erik/ctsm_worktrees/mizuRoute/share/src/shr_file_mod.F90         913 This routine is depricated - use shr_log_setLogUnit instead         -14
dec2436.hsn.de.hpc.ucar.edu: rank 0 died from signal 15

rof.log:

  inst_name = ROF
 mizuRoute tracers =            2 LIQ:ICE
(route_ini) mizuRoute run at   86400.0000000    [sec] step   1 per a coupling
 define run:
    run type              = initial
    coupling_frequency    =            1 [day]

ln.log finishes initialization, atm.log is happily advancing.

[ekluzek]

Looking at this with @nmizukami.
This is just the problem with using the older ESMF (8.6.0) for gnu while intel is using 8.8.1 which works fine. This is the problem in #452 which can be fixed either with a newer ESMF or with updated MESH files.