Update the qstat job completion checks

[apcraig]

PR checklist

• Short (1 sentence) summary of your PR:
  Update the qstat job completion checks
• Developer(s):
  apcraig
• Suggest PR reviewers from list in the column to the right.
• Please copy the PR test results link or provide a summary of testing completed below.
  Full Icepack test suite run on derecho intel, bit-for-bit and all tests passed.
• How much do the PR code changes differ from the unmodified code?
  • bit for bit
  • different at roundoff level
  • more substantial
• Does this PR create or have dependencies on CICE or any other models?
  • Yes
  • No
• Does this PR add any new test cases?
  • Yes
  • No
• Is the documentation being updated? ("Documentation" includes information on the wiki or in the .rst files from doc/source/, which are used to create the online technical docs at https://readthedocs.org/projects/cice-consortium-cice/.)
  • Yes
  • No, does the documentation need to be updated at a later time?
    • Yes
    • No
• Please document the changes in detail, including why the changes are made. This will become part of the PR commit log.

Update the qstat job completion checks to use

set qstatus = ${ICE_MACHINE_QSTAT} $job | grep $job | wc -l

Some recent changes made the return status of "${ICE_MACHINE_QSTAT} $job" no longer robust as a way to check if a job was in the queue or not.

[dabail10]

Does this fix the hangs?

[apcraig]

Does this fix the hangs?

TBD, it helps with the testing costs, but there is still an issue with jobs hanging for me. The cice8yr case I did a couple weeks was just a standard run, no test scripts. That is not fixed with this change.