Threaded test that failed hanging on Derecho with intel compiler

[ekluzek]

Running ctsm5.4.021 with mizuRoute v3.0.1 one test in the mizu testlist failed.

SMS_P64x18.f19_f19_rMERIT_mg17.I2000Clm60SpMizGs.derecho_intel.mizuroute-default ( RUN)

This might just be something about the PE layout for this case It looks like it's actually dying in DATM. riof.log is empty, lnd.log is midway in the initialization phase.

atm.log ends with:

(atm_comp_nuopc):(InitializeAdvertise) datm datamode = CLMNCEP
 ATM: PIO numiotasks=           9
 ATM: PIO stride=           7
 ATM: PIO rearranger=           1
 ATM: PIO root=           1
(dshr_mesh_init) (dshr_mod:dshr_mesh_init) obtained ATM mesh and mask from /glade/campaign/cesm/cesmdata/inputdata/share/meshes/fv1.9x2.5_141008_ESMFmesh.nc
(shr_stream_init_from_xml)  getting calendar for stream 1

cesm.log talks about a fatal error for SIGTERM and process killed.

cesm.log has this in it:

dec1654.hsn.de.hpc.ucar.edu 63: libpioc.so         0000145997411633  PIOc_openfile         Unknown  Unknown
dec1654.hsn.de.hpc.ucar.edu 63: libpiof.so         000014599781851C  piolib_mod_mp_pio     Unknown  Unknown
dec1654.hsn.de.hpc.ucar.edu 63: cesm.exe           000000000119F2FF  dshr_stream_mod_m        1666  dshr_stream_mod.F90
dec1654.hsn.de.hpc.ucar.edu 63: cesm.exe           00000000011A6D60  dshr_stream_mod_m         453  dshr_stream_mod.F90
dec1654.hsn.de.hpc.ucar.edu 63: cesm.exe           0000000001193BB8  dshr_strdata_mod_         235  dshr_strdata_mod.F90
dec1654.hsn.de.hpc.ucar.edu 63: cesm.exe           0000000000573283  atm_comp_nuopc_mp         462  atm_comp_nuopc.F90

[ekluzek]

This is likely a bad PE layout. The standard test that works is:

SMS.f19_f19_rMERIT_mg17.I2000Clm60SpMizGs.derecho_intel.mizuroute-default

which has

./pelayout 
Comp  NTASKS  NTHRDS  ROOTPE PSTRIDE
CPL :   1536/     1;    128      1
ATM :    128/     1;      0      1
LND :   1536/     1;    128      1
ICE :   1536/     1;    128      1
OCN :   1536/     1;    128      1
ROF :   1536/     1;    128      1
GLC :   1536/     1;    128      1
WAV :   1536/     1;    128      1
ESP :      1/     1;      0      1
ESMF_AWARE_THREADING is False
ROOTPE is with respect to 128.0 tasks per node

This case is using 64 MPI tasks and 18 threads, so a total of 1152. But, probably 64 tasks is too few to get everything in memory. So how about something like 160 MPI tasks, and 16 threads?