Skip to content

Commit 232b6d8

Browse files
ChristopherHill-NOAAchristopher hillchristopher hillgithub-actions[bot]
authored
Add compression to SBN atmospheric product generation (NOAA-EMC#4649)
# Description NCO requires that the total volume of post-processing products to be transmitted through the SBN <ins>not</ins> exceed current operational levels. The generation of these products in retrospective runs of GFSv17 have marginally exceeded operational levels, and the files must be reduced in size (in lieu of the outright removal of some). This PR simply adds a compression/packing attribute to the generation of WMO-headed atmospheric products within the script [exgfs_atmos_awips_20km_1p0deg.sh](https://github.com/ChristopherHill-NOAA/global-workflow/blob/035d50609e23554a20a0c7324e488ee628c1ab1f/dev/scripts/exgfs_atmos_awips_20km_1p0deg.sh#L67), accomplished by revising the `-set_grib_type` option of WGRIB2 from "same" to "complex2". This PR is intended to resolve NOAA-EMC#4614. # Type of change - [x] Bug fix (fixes something broken) - [ ] New feature (adds functionality) - [ ] Maintenance (code refactor, clean-up, new CI test, etc.) # Change characteristics <!-- Choose YES or NO from each of the following and delete the other --> - Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? YES (If YES, please indicate to which system(s)) - [x] GFS - [ ] GEFS - [ ] SFS - [ ] GCAFS - Is this a breaking change (a change in existing functionality)? NO - Does this change require a documentation update? NO - Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.) - [ ] EMC verif-global <!-- NOAA-EMC/EMC_verif-global#1234 --> - [ ] GDAS <!-- NOAA-EMC/GDASApp#1234 --> - [ ] GFS-utils <!-- NOAA-EMC/gfs-utils#1234 --> - [ ] GSI <!-- NOAA-EMC/GSI#1234 --> - [ ] GSI-monitor <!-- NOAA-EMC/GSI-Monitor#1234 --> - [ ] GSI-utils <!-- NOAA-EMC/GSI-Utils#1234 --> - [ ] UFS-utils <!-- ufs-community/UFS_UTILS#1234 --> - [ ] UFS-weather-model <!-- ufs-community/ufs-weather-model#1234 --> - [ ] wxflow <!-- NOAA-EMC/wxflow#1234 --> # How has this been tested? The GFS workflow was cloned and built on WCOSS. A CI test only confirmed nominal functionality with the code change, as the products resulting from the low resolution run were too small for the code change to produce the necessary effect. Offline tests executing a segment of `exgfs_atmos_awips_20km_1p0deg.sh` with files from `products/atmos/grib2/0p25` as input resulted in a reduction to the size of the products. It is recommended that the changes from this PR be tested in a single cycle, high-resolution test of GFSv17, to confirm its intended effect to reduce SBN product volume to the extent required by NCO. # Checklist - [ ] Any dependent changes have been merged and published - [ ] My code follows the style guidelines of this project - [ ] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have documented my code, including function, input, and output descriptions - [ ] My changes generate no new warnings - [ ] New and existing tests pass with my changes - [ ] This change is covered by an existing CI test or a new one has been added - [ ] Any new scripts have been added to the .github/CODEOWNERS file with owners - [ ] I have made corresponding changes to the system documentation if necessary --------- Co-authored-by: christopher hill <christopher.m.hill@clogin08.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: christopher hill <christopher.m.hill@clogin04.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent 9c88dc0 commit 232b6d8

1 file changed

Lines changed: 16 additions & 0 deletions

File tree

dev/scripts/exgfs_atmos_awips_20km_1p0deg.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,14 @@ for GRID in conus ak prico pac 003; do
184184
if [[ ${err} -ne 0 ]]; then
185185
err_exit "Failed to generate the awips Grib2 file!"
186186
fi
187+
echo "Complex2 compression/packing for grib2.awpgfs${fcsthr}.${GRID}"
188+
cpreq "grib2.awpgfs${fcsthr}.${GRID}" "tmp_grib2_${fcsthr}_${GRID}"
189+
${WGRIB2} "tmp_grib2_${fcsthr}_${GRID}" -set_grib_type complex2 \
190+
-grib_out "grib2.awpgfs${fcsthr}.${GRID}"
191+
export err=$?
192+
if [[ ${err} -ne 0 ]]; then
193+
err_exit "Failed to compress and repack the awips Grib2 file!"
194+
fi
187195

188196
##############################
189197
# Post Files to ${COMOUT_ATMOS_WMO}
@@ -214,6 +222,14 @@ for GRID in conus ak prico pac 003; do
214222
if [[ ${err} -ne 0 ]]; then
215223
err_exit "Failed to write the AWIPS grib2 file"
216224
fi
225+
echo "Complex2 compression/packing for grib2.awpgfs_20km_${GRID}_f${fcsthr}"
226+
cpreq "grib2.awpgfs_20km_${GRID}_f${fcsthr}" "tmp_grib2_${GRID}_f${fcsthr}"
227+
${WGRIB2} "tmp_grib2_${GRID}_f${fcsthr}" -set_grib_type complex2 \
228+
-grib_out "grib2.awpgfs_20km_${GRID}_f${fcsthr}"
229+
export err=$?
230+
if [[ ${err} -ne 0 ]]; then
231+
err_exit "Failed to compress and repack the AWIPS grib2 file!"
232+
fi
217233

218234
##############################
219235
# Post Files to ${COMOUT_ATMOS_WMO}

0 commit comments

Comments
 (0)