time_ave.x not working on Milan nodes

[rtodling]

Somehow time_ave.x in GEOSadas-5.41.3 does not work on the Milan nodes ... the call freezes and and hangs. Not clear why. I am working to try to debug it ... but if anyone else has ran into the issue and has a solution please let me know.

[rtodling]

I just added Ben to this since I believe the issue in time_ave.x is one of reading the RC file associated w/ the quadratics ... if I don't ask for quadratics (i.e., don't specify a -rc option at command line) the code works.

Just for reference is anyone wants (and can) contribute to this ... I am using the files under:

/gpfsm/dnb05/projects/p139/rtodling/archive/541/Milan/x0050tb/diag/Y2023/M08/MM/

and two command lines: one that does not work - which is the original official command line used in our scripts

mpirun -np 144 /discover/nobackup/projects/gmao/advda/rtodling/4OPS/g540x/GEOSadas/install-SLES15/bin/time_ave.x -rc /discover/nobackup/projects/gmao/advda/rtodling/4OPS/g540x/GEOSadas/install-SLES15/etc/time_ave.rc -ops -tag x0050tb.inst3_3d_asm_Np.monthly -d x0050tb.inst3_3d_asm_Np.diurnal -hdf x0050tb.inst3_3d_asm_Np.20230801*

and another command line that works fine:

mpirun -np 144 /discover/nobackup/projects/gmao/advda/rtodling/4OPS/g540x/GEOSadas/install-SLES15/bin/time_ave.x -ops -tag x0050tb.inst3_3d_asm_Np.monthly -d x0050tb.inst3_3d_asm_Np.diurnal -hdf x0050tb.inst3_3d_asm_Np.20230801*

in which the rc option is removed ... notice I have a full month of data in the MM directory ... the command lines above use on the files corresponding to the first day of the month; otherwise it takes a year to run the job - just kidding :-) but it too long for debugging.

[rtodling]

I know what is happening ... but don't know why:

The tableEnd output from the calls:

call ESMF_ConfigNextLine ( config,tableEnd=tend ,rc=rc )

(of which there are two in the program in question) - do not return tend=.TRUE. when the table ends ... if I check on the RC code than I seem to be able to close the table since RC is non-zero when the table ends. @bena-nasa did we go thru this before w/ GSI use of this function?

[mathomp4]

@rtodling This was fixed in #86 (aka v2.1.1). The issue is that recent ESMF became stricter when processing config files like time_ave.rc. And if you look at that file in your tag and go to the ALIASES section:

ALIASES:    'u'  ,  'U'       '
            'u'  ,  'uwnd'    '
            'u'  ,  'UWND'    '
            'u'  ,  'ugrd'    '
            'u'  ,  'UGRD'    '
            'u'  ,  'ugrdprs' '
            'u'  ,  'UGRDPRS' '

Notice those end characters are single-quotes, not commas. If you convert them all in your tag to commas, we believe it should work as ESMF can now correctly see the end of the table.