Update derecho shared batch job submission (CICE-Consortium#1091)

Derecho shared node jobs intermittently abort with error message
"start failed on dec2436: No reply from shepherd after 108s"
due to PBS/MPI launch conflicts. Derecho qstat output was also recently changed to return output for completed jobs which prevented the job checking scripts from identifying jobs that have completed.

Update derecho shared batch job submission to both increase the number of shared node jobs and control the number of jobs per shared node by submitting the shared jobs on more cores than needed. In the end, an upgrade to PBS seemed to fix the shared node aborts, so this change was commented out in the PR. Derecho will continue to be closely watched.

Fix potential bug in setting ICE_MACHINE_QSTAT if the string has spaces in it.

Update job checking logic to avoid PBS output that shows completed jobs, added -v " historical ". This is far from ideal and not particularly future proof, but PBS qstat has become a mess.

Update create fails to identify test suite jobs that failed to run then generate a script to resubmit them.

Author: apcraig

cice.setup

@@ -1244,7 +1244,7 @@ EOF0
 
   if ($?ICE_MACHINE_QSTAT) then
     cat >! ${tsdir}/poll_queue.env << EOF0
-setenv ICE_MACHINE_QSTAT ${ICE_MACHINE_QSTAT}
+setenv ICE_MACHINE_QSTAT "${ICE_MACHINE_QSTAT}"
 EOF0
   endif
 

configuration/scripts/cice.batch.csh

@@ -35,22 +35,31 @@ EOFB
 
 else if (${ICE_MACHINE} =~ derecho*) then
 set memstr = ""
-if (${ncores} <= 8 && ${runlength} <= 1 && ${batchmem} <= 20) then
+set mycorespernode = ${corespernode}
+# trying to avoid shared node launch errors
+#if (${ncores} <= 24 && ${runlength} <= 1 && ${batchmem} <= 20) then
+if (${ncores} <= 16 && ${runlength} <= 1 && ${batchmem} <= 20) then
   set queue = "develop"
+#  # set develop cores to 16 or 32 to limit the number of jobs per shared node
+#  if (${mycorespernode} < 32) then
+#     @ corenum = (${mycorespernode} / 16 + 1) * 16
+#     set mycorespernode = ${corenum}
+#  endif
   set memstr = ":mem=${batchmem}GB"
 endif
 cat >> ${jobfile} << EOFB
 #PBS -q ${queue}
 #PBS -l job_priority=regular
 #PBS -N ${ICE_CASENAME}
 #PBS -A ${acct}
-#PBS -l select=${nnodes}:ncpus=${corespernode}:mpiprocs=${taskpernodelimit}:ompthreads=${nthrds}${memstr}
+#PBS -l select=${nnodes}:ncpus=${mycorespernode}:mpiprocs=${taskpernodelimit}:ompthreads=${nthrds}${memstr}
 #PBS -l walltime=${batchtime}
 #PBS -j oe
 #PBS -W umask=022
 #PBS -o ${ICE_CASEDIR}
 ###PBS -M [email protected]
 ###PBS -m be
+
 EOFB
 
 else if (${ICE_MACHINE} =~ gadi*) then

configuration/scripts/tests/baseline.script

@@ -136,7 +136,8 @@ if (${ICE_BFBCOMP} != ${ICE_SPVAL}) then
     set cnt = 0
     if (${job} =~ [0-9]*) then
       while ($qstatjob)
-        set qstatus = `${ICE_MACHINE_QSTAT} $job | grep $job | wc -l`
+        # historical avoids completed jobs on PBS (-x) and extra $job avoids superfluous header lines
+        set qstatus = `${ICE_MACHINE_QSTAT} $job | grep -iv " historical " | grep $job | wc -l`
 #        ${ICE_MACHINE_QSTAT} $job
 #        echo $job $qstatus
         if ($qstatus == 0) then

configuration/scripts/tests/create_fails.csh

@@ -3,9 +3,16 @@
 echo " "
 set tmpfile = create_fails.tmp
 set outfile = fails.ts
+set runfile = rerun.csh
+
+set delim = `pwd | rev | cut -d . -f 1 | rev`
 
 ./results.csh >& /dev/null
-cat results.log | grep ' run\| test' | grep -v "#" | grep -v PASS | cut -f 2 -d " " | sort -u >! $tmpfile
+
+#fails, both "run" and "test" that failed
+# treat decomp special
+cat results.log | grep ' run\| test' | grep -v "#" | grep -v PASS | cut -f 2 -d " " | grep -v _decomp_ | sort -u >! $tmpfile
+cat results.log | grep ' run\| test' | grep -v "#" | grep -v PASS | cut -f 2 -d " " | grep _decomp_  | rev | cut -d _ -f 2- | rev | sort -u >> $tmpfile
 
 echo "# Test  Grid  PEs  Sets" >! $outfile
 foreach line ( "`cat $tmpfile`" )
@@ -16,9 +23,25 @@ foreach line ( "`cat $tmpfile`" )
   set opts = `echo $line | cut -d "_" -f 6- | sed 's/_/,/g'`
   echo "$test  $grid  $pes  $opts" >> $outfile
 end
+rm $tmpfile
 
+#rerun, only "run" that failed
+# treat decomp special
+cat results.log | grep ' run' | grep -v "#" | grep -v PASS | cut -f 2 -d " " | grep -v _decomp_ | sort -u >! $tmpfile
+cat results.log | grep ' run' | grep -v "#" | grep -v PASS | cut -f 2 -d " " | grep _decomp_  | rev | cut -d _ -f 2- | rev | sort -u >> $tmpfile
+
+echo "#/bin/csh" >! $runfile
+foreach line ( "`cat $tmpfile`" )
+  #echo $line
+  echo "cd ${line}.${delim}; ./*.submit; cd ../; sleep 5" >> $runfile
+end
+chmod +x $runfile
 rm $tmpfile
+
 echo "$0 done"
+echo " "
+echo "Failed runs can be resubmitted by running $runfile"
+echo " "
 echo "Failed tests can be rerun with the test suite file...... $outfile"
 echo "To run a new test suite, copy $outfile to the top directory and do something like"
 echo "  ./cice.setup --suite $outfile ..."

configuration/scripts/tests/poll_queue.csh

@@ -13,7 +13,8 @@ foreach line ("`cat suite.jobs`")
   set qstatjob = 1
   if (${job} =~ [0-9]*) then
     while ($qstatjob)
-      set qstatus = `${ICE_MACHINE_QSTAT} $job | grep $job | wc -l`
+      # historical avoids completed jobs on PBS (-x) and extra $job avoids superfluous header	lines
+      set qstatus = `${ICE_MACHINE_QSTAT} $job | grep -iv " historical " | grep $job | wc -l`
 #      echo $job $qstatus
       if ($qstatus == 0) then
         echo "Job $job completed"

doc/source/user_guide/ug_testing.rst

@@ -448,11 +448,13 @@ which means by default the test suite builds and submits the jobs.  By defining
 By leveraging the **cice.setup** command line arguments ``--setup-only``, ``--setup-build``, and ``--setup-build-run`` as well as the environment variables SUITE_BUILD, SUITE_RUN, and SUITE_SUBMIT, users can run **cice.setup** and **suite.submit** in various combinations to quickly setup, setup and build, submit, resubmit, run interactively, or rebuild and resubmit full testsuites quickly and easily.  See :ref:`examplesuites` for an example.
 
 The script **create_fails.csh** will process the output from results.csh and generate a new 
-test suite file, **fails.ts**, from the failed tests.  
-**fails.ts** can then be edited and passed into ``cice.setup --suite fails.ts ...`` to rerun 
-subsets of failed tests to more efficiently move thru the development, testing, and 
-validation process.  However, a full test suite should be run on the final development
-version of the code.
+test suite file, **fails.ts**, from the failed tests.   It will also generate a script called
+**rerun.csh** for runs that failed to complete.  **rerun.csh** can be executed from the testsuite directory and
+runs that failed to complete will be resubmitted.
+**fails.ts** can be passed into ``cice.setup --suite fails.ts ...`` to setup a new test
+suite based on the failed tests to more efficiently move thru the development, testing, and 
+validation process.  However, ultimately, once all code changes are complete, a full test suite 
+should be run on the final development version of the code.
 
 To report the test results, as is required for Pull Requests to be accepted into 
 the main the CICE Consortium code see :ref:`testreporting`.