Fix 06z gfs_sfcanl jobs failing due to a bug in CCPP by DavidHuber-NOAA · Pull Request #4389 · NOAA-EMC/global-workflow

DavidHuber-NOAA · 2025-12-29T14:00:13Z

Description

This fixes a bug in CCPP that prevented the 06Z gfs_sfcanl job from running global_cycle on the 03Z IAU time. The code change now allows the input hour to be any integer between 0 and 23.
Resolves #4364
Refs #4408 (partially resolves but a full investigation is needed)

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this change expected to change outputs YES: the CCPP code change itself is non-answer-changing, but it requires a model hash update
- GFS
- GEFS
- SFS
- GCAFS
Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES
- UFS-utils Update ccpp-physics to enable non 6-hourly global cycle updates ufs-community/UFS_UTILS#1122 (depends on Allow ihr to be non-multiples of 6 ufs-community/ccpp-physics#336)
- UFS-weather-model Allow ihr to be non-mulitples of 6 in ccpp-physics ufs-community/ufs-weather-model#3036 (comment) (depends on Allow ihr to be non-multiples of 6 ufs-community/ccpp-physics#336)

How has this been tested?

C96_atm3DVar_extended test on WCOSS2
UFS_Utils regression tests on Ursa (no change to baseline)
UFS regression tests
Full suite of GW tests on all platforms (when the UFS model hash is ready)

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have documented my code, including function, input, and output descriptions
My changes generate no new warnings
New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added

aerorahul · 2026-01-06T20:43:55Z

ush/parsing_ufs_configure.sh

+        local MOM6_OUTPUT_DIR="${MOM6_OUTPUT_DIR:-./MOM6_OUTPUT}"
+        local MOM6_RESTART_DIR="${MOM6_RESTART_DIR:-./MOM6_RESTART}"


Suggested change

local MOM6_OUTPUT_DIR="${MOM6_OUTPUT_DIR:-./MOM6_OUTPUT}"

local MOM6_RESTART_DIR="${MOM6_RESTART_DIR:-./MOM6_RESTART}"

local MOM6_OUTPUT_DIR="./MOM6_OUTPUT"

local MOM6_RESTART_DIR="./MOM6_RESTART"

This isn't an option. The workflow will always write to this space. The other instances where this variable is set in this manner is ush/parsing_namelists_FV3.sh

I agree, I don't think this should be configurable. This is assumed other places.

I also see that ufs-community/ufs-weather-model@6f22f57...f8b0802 we have a MOM6_OUTPUT_FH defined, which is defined in forecast_predet.

I'm assuming those are consistent definitions?

aerorahul

just one suggestion, looks good.

JessicaMeixner-NOAA

We should run a S2S test of some sort to make sure ocean output looks okay given the changes. I didn't follow exactly when those changes went in. @jiandewang or @dpsarmie might know more about that the MOM6_OUTPUT_FH to make sure that is as expected here now.

JessicaMeixner-NOAA · 2026-01-06T20:59:54Z

ush/parsing_ufs_configure.sh

+        local MOM6_OUTPUT_DIR="${MOM6_OUTPUT_DIR:-./MOM6_OUTPUT}"
+        local MOM6_RESTART_DIR="${MOM6_RESTART_DIR:-./MOM6_RESTART}"


I agree, I don't think this should be configurable. This is assumed other places.

I also see that ufs-community/ufs-weather-model@6f22f57...f8b0802 we have a MOM6_OUTPUT_FH defined, which is defined in forecast_predet.

I'm assuming those are consistent definitions?

dpsarmie · 2026-01-06T21:45:03Z

FHOUT_OCN_GFS / FHOUT_OCN is the variable that controls the MOM6 output frequency in GW and MOM6_OUTPUT_FH is just used to store an array of output times (in GW), correct? If so, then there might be an issue.

JessicaMeixner-NOAA · 2026-01-06T21:48:02Z

FHOUT_OCN_GFS / FHOUT_OCN is the variable that controls the MOM6 output frequency in GW and MOM6_OUTPUT_FH is just used to store an array of output times (in GW), correct? If so, then there might be an issue.

Yes. And looking a bit further, this variable should be "FHOUT_OCN" and not MOM6_OUTPUT_FH.

ClaraDraper-NOAA

I haven't tested it, but can confirm that this PR brings the needed update to CCPP/physics into UFS_UTILS to fix the gfs_sfcanl issue. It also updates the CCPP/physics used by the model to the same hash.

Note that updating the ufs_model and ufs_utils brings in many additional changes in addition to the gfs_sfcnl issue.

DavidHuber-NOAA · 2026-01-08T14:22:38Z

All tests passed on Ursa. I will run a develop C96C48mx500 case to verify the MOM6 output, then launch CI on all platforms.

DavidHuber-NOAA · 2026-01-08T14:55:35Z

I verified the 6-hour MOM6 forecast for the C96C48mx025_S2SW_gfs_cyc case was identical to last night's nightly run of the same case and also verified that MOM6 output was written every 6 hours as expected:

>cmp /scratch3/NCEPDEV/stmp/David.Huber/rt_4389/COMROOT/C96C48mx500_S2SW_cyc_gfs_4389/gfs.20211220/18/model/ocean/history/gfs.t18z.6hr_avg.f120.nc /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/nightly_6ada5183_010726/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_6ada5183-6840/gfs.20211220/18/model/ocean/history/gfs.t18z.6hr_avg.f120.nc
> echo $?
0

Proceeding with CI testing.

DavidHuber-NOAA · 2026-01-16T17:32:34Z

The gaea c6 failure was caused by a build stalling on a known-bad head node. I will relaunch tests there now that CI is running there again smoothly.

The SFS test failure on Ursa is known and that test has since been removed from the CI matrix.

The Hera failure is the only concerning issue. /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/HERA/BUILDS/GITLAB/pr_cases_4389_f6105b9e_7163/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_f6105b9e-7163/logs/2021032500/gfs_arch_tar_gfswave.log reports a failure on line 1231:

FileNotFoundError: FATAL ERROR: Required file, directory, or glob gfs.20210325/00/products/wave/gridded/global.0p50/gfs.t00z.*.*.f017.grib2 not found!

The global.0p50 directory does not exist. Instead, it is global.0p16. This is likely a change in grids that came in with the latest UFS model upgrade, which includes a WW3 update. There is likely a namelist variable change required.

DavidHuber-NOAA · 2026-01-16T17:33:43Z

@JessicaMeixner-NOAA @dpsarmie is there a required WW3 update needed with this update? It appears that the wave grid changed unexpectedly from 0p50 to 0p16.

DavidHuber-NOAA · 2026-01-16T17:48:37Z

@JessicaMeixner-NOAA @dpsarmie actually, this does not seem to be related to the model update. Last night's nightly also ran WW3 at 0p16 and archived correctly. I will continue investigating.

DavidHuber-NOAA · 2026-01-16T18:05:01Z

I see now what happened. The gfs_arch_tar_gfswave job launched before the gfs_wavepostsbs_f015-f017 job completed. This reflects another archiving dependency that is missing (#4408). I will add this dependency as well as the gfs_waveawipsgridded dependency (also documented in #4408) and relaunch tests on Hera, C6, Ursa, and WCOSS2.

DavidHuber-NOAA · 2026-01-16T18:32:10Z

I am still wrong. The wavepostsbs metatask is already a dependency of the archiving jobs. Instead, we have a silent failure. This is in /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/HERA/BUILDS/GITLAB/pr_cases_4389_f6105b9e_7163/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_f6105b9e-7163/logs/2021032500/gfs_wavepostsbs_f015-f017.log during an MPMD section of the job at line 11403:

3: + wave_grib2_sbs.sh[88]/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/HERA/BUILDS/GITLAB/pr_cases_4389_f6105b9e_7163/global-workflow/exec/gfs_ww3_grib.x
3: + wave_grib2_sbs.sh[89]export err=126
3: + wave_grib2_sbs.sh[89]err=126
3: + wave_grib2_sbs.sh[90][[ 126 -ne 0 ]]
3: + wave_grib2_sbs.sh[91]echo 'FATAL ERROR: gfs_ww3_grib.x returned non-zero status: 126; exiting!'
3: FATAL ERROR: gfs_ww3_grib.x returned non-zero status: 126; exiting!
3: + wave_grib2_sbs.sh[92]exit 126

Going to the end of the MPMD job at line 9891:

+ run_mpmd.sh[66]IFS=
+ run_mpmd.sh[66]read -r line
+ run_mpmd.sh[71]unset_strict
+ preamble.sh[45]set +eu
+ preamble.sh[46]set +o pipefail
+ run_mpmd.sh[73]srun -l --export=ALL --hint=nomultithread --multi-prog --output=mpmd.%j.%t.out -n 7 /scratch3/NCEPDEV/stmp/role.glopara/RUNDIRS/C48mx500_3DVarAOWCDA_f6105b9e-7163/gfs.2021032500/wavepostsbs_f017.82108/mpmd_cmdfile
+ run_mpmd.sh[74]err=0
+ run_mpmd.sh[75]set_strict
+ preamble.sh[35][[ YES == \Y\E\S ]]
+ preamble.sh[37]set -eu
+ preamble.sh[39]set -o pipefail
+ run_mpmd.sh[101][[ 0 -eq 0 ]]

It appears that run_mpmd.sh is not correctly catching failures of individual jobs.

DavidHuber-NOAA · 2026-01-20T13:40:50Z

All tests passed on all platforms. Requesting final approvals to merge.

JessicaMeixner-NOAA · 2026-01-20T13:47:17Z

dev/workflow/rocoto/gfs_tasks.py

+            dep_dict = {'type': 'task', 'name': f'{self.run}_fbwind'}
+            deps.append(rocoto.add_dependency(dep_dict))
+
+            if self.options['do_wave']:


This is okay to add but we don't actually archive any of the awips files for waves to my knowledge, so I don't think this is actually needed.

Good to know! I'll open an issue to remove this tarball/job.

Opened issues #4454 to retire AWIPS waves archiving.

DavidHuber-NOAA added 2 commits December 29, 2025 13:43

Update ufs_utils to allow global_cycle to run on 03z.

99dbd9a

Added my fork to UFS_Utils

d5e7e49

DavidHuber-NOAA added the GFS Change This PR, if merged, will change results for the GFS. label Dec 29, 2025

DavidHuber-NOAA changed the title ~~Fix/gfs sfcanl~~ Fix 06z gfs_sfcanl jobs failing due to a bug in CCPP Dec 29, 2025

Update UFS to incorporate the same CCPP as UFS_Utils

f9a2e4e

RussTreadon-NOAA mentioned this pull request Jan 5, 2026

Include fixes to properly handle GSI ncdiag files in anlstat job #4401

Merged

26 tasks

DavidHuber-NOAA added 2 commits January 6, 2026 20:25

Merge remote-tracking branch 'emc/develop' into fix/gfs_sfcanl

2a0ca7a

Update the WM and UFS_Utils to include sfcsub.F fix

d9c5e40

DavidHuber-NOAA requested a review from ClaraDraper-NOAA January 6, 2026 20:30

DavidHuber-NOAA marked this pull request as ready for review January 6, 2026 20:30

DavidHuber-NOAA requested a review from aerorahul as a code owner January 6, 2026 20:30

DavidHuber-NOAA requested review from CatherineThomas-NOAA, JessicaMeixner-NOAA and RuiyuSun January 6, 2026 20:31

Add MOM6 namelist variables

97d1c4b

DavidHuber-NOAA requested a review from junwang-noaa as a code owner January 6, 2026 20:32

aerorahul reviewed Jan 6, 2026

View reviewed changes

aerorahul previously approved these changes Jan 6, 2026

View reviewed changes

JessicaMeixner-NOAA reviewed Jan 6, 2026

View reviewed changes

ClaraDraper-NOAA previously approved these changes Jan 6, 2026

View reviewed changes

Make mom6 dirs non-configurable

446b134

DavidHuber-NOAA dismissed stale reviews from ClaraDraper-NOAA and aerorahul via 446b134 January 7, 2026 17:37

Merge remote-tracking branch 'emc/develop' into fix/gfs_sfcanl

0f38670

ClaraDraper-NOAA mentioned this pull request Jan 8, 2026

Identify impact of global_cycle time shift ClaraDraper-NOAA/task_tracker#47

Closed

Merge branch 'develop' into fix/gfs_sfcanl

94c179d

emcbot removed the CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress label Jan 15, 2026

DavidHuber-NOAA added 2 commits January 16, 2026 19:32

Merge remote-tracking branch 'emc/develop' into fix/gfs_sfcanl

d22ddd2

Add awips dependencies to arch_tars

7d7fe51

DavidHuber-NOAA dismissed stale reviews from aerorahul and JessicaMeixner-NOAA via 7d7fe51 January 16, 2026 20:04

Fix dep name

117d387

DavidHuber-NOAA removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed CI-Ursa-Failed **Bot use only** CI testing on Ursa for this PR has failed labels Jan 16, 2026

Merge branch 'develop' into fix/gfs_sfcanl

dbe1e28

JessicaMeixner-NOAA approved these changes Jan 20, 2026

View reviewed changes

DavidHuber-NOAA mentioned this pull request Jan 20, 2026

Retire wave AWIPS product archiving #4454

Closed

4 tasks

		local MOM6_OUTPUT_DIR="${MOM6_OUTPUT_DIR:-./MOM6_OUTPUT}"
		local MOM6_RESTART_DIR="${MOM6_RESTART_DIR:-./MOM6_RESTART}"

Conversation

DavidHuber-NOAA commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

aerorahul Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

JessicaMeixner-NOAA Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

JessicaMeixner-NOAA Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

dpsarmie commented Jan 6, 2026

Uh oh!

JessicaMeixner-NOAA commented Jan 6, 2026

Uh oh!

ClaraDraper-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA commented Jan 8, 2026

Uh oh!

DavidHuber-NOAA commented Jan 8, 2026

Uh oh!

DavidHuber-NOAA commented Jan 16, 2026

Uh oh!

DavidHuber-NOAA commented Jan 16, 2026

Uh oh!

DavidHuber-NOAA commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA commented Jan 16, 2026

Uh oh!

DavidHuber-NOAA commented Jan 16, 2026

Uh oh!

DavidHuber-NOAA commented Jan 20, 2026

Uh oh!

JessicaMeixner-NOAA Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

DavidHuber-NOAA commented Dec 29, 2025 •

edited

Loading

DavidHuber-NOAA commented Jan 16, 2026 •

edited

Loading