Allow reading and processing large grib2 files in chgres_cube, such as grib2 RRFSv1 output on native levels#1034
Conversation
… or greater to create index files for large files.
…ity#1027) Previously, the tests ran in serial and some parts were run from the command line. This speeds up the processing and places all the test functions in the batch queue. The functionality was not changed. Fixes ufs-community#1016.
|
Will look at this after #1028 is merged. |
modulefiles/build.hera.intel.lua
Outdated
| zlib_ver=os.getenv("zlib_ver") or "1.2.13" | ||
| load(pathJoin("zlib", zlib_ver)) | ||
| --zlib_ver=os.getenv("zlib_ver") or "1.2.13" | ||
| --load(pathJoin("zlib", zlib_ver)) |
There was a problem hiding this comment.
Was zlib causing a problem or is it not used?
There was a problem hiding this comment.
zlib was not used, apparently, and I would put it back into the module.
There was a problem hiding this comment.
I removed the module load of zlib for the intelllvm build. Everything built fine. Apparently, zlib is automatically loaded from some other module. I only tried this test on Hera.
|
@natalie-perlin - the CI checks are failing because they use the previous version of g2. Can you update the spack.yaml file to use the newer g2? Updating the ./ci/spack.yaml to have similar versions of packages as the modulefiles. In particular, these are updated: There is no nco listed (nco/5.0.6) in the spack.yaml, but it is present in the modulefiles. If the build fails, I may add nco as well. |
|
@natalie-perlin - not sure why your CI checks are breaking. As a test, I updated the |
|
@GeorgeGayno-NOAA - |
|
@natalie-perlin - what is the status of this PR? There have been no updates for a month. |
@GeorgeGayno-NOAA - will be able to get back to it again. Some higher priorities with the UFS weather-model work had to be completed first... |
|
@natalie-perlin - what is the status of this PR? |
DESCRIPTION OF CHANGES:
Allow reading and processing large grib2 files by
chgres_cube, which otherwise produce errors when reading large grib2 files, such as RRFSv1 model output on native or pressure levels for the 3-km full-NA grid. Files retrieved from https://registry.opendata.aws/noaa-rrfs/, AWS S3 buckethttps://noaa-rrfs-pds.s3.amazonaws.com/index.html#rrfs_a/
naming conventions
rrfs.t{HH}z.natlev.f{FHR}.grib2andrrfs.t{HH}z.prslev.f{FHR}.grib2TESTS CONDUCTED:
Run a test on Hera of reading rrfs.t00z.natlev.f000.grib2 file from a RRFSv1 control run for a date 2024110600, for the small test domain over California. Test runscript is as follows, placed under ./ush directory:
run_chgres_cube.sh.txt, with the following input file: fort.41.txt. Log files attached below in the Testing
A chgres_cube executable that reads a large grib2 file was successful, for the binary built with either Intel or GNU compilers, using updated spack-stack locations as in the PR #1028. The tests reported exit code=0.
Surface files were prepared by the UFS-SRW grid and orography tasks that do not yet employ fractional vegetation and soil data, and this was indicated in the end of the file, i.e.: i
Log files:
log.cube.8002181.intel.txt
log.cube.8002908.gnu.txt
Compile branch on all Tier 1 machines using Intel (Orion, Jet, Hera, Hercules, Gaea C5/C6 , but not WCOSS2).
Build Log files:
log.build.hera.intel_large_grib2.txt
log.build.orion.intel_large_grib2.txt
log.build.hercules.intel_large_grib2.log
log.build.jet.intel_large_grib2.txt
log.build.gaeac5.intel_large_grib2.txt
log.build.gaeac6.intel_large_grib2.txt
Compile branch on Hera using GNU.
log.build.hera.gnu_large_grib2.txt
Compile branch on NOAA Cloud Service providers: AWS, Azure, GCP.
log.build.noaacloud.AWS.intel_large_grib2.txt
log.build.noaacloud.Azure.intel_large_grib2.txt
log.build.noaacloud.GCP.intel_large_grib2.txt
Compile branch in 'Debug' mode on WCOSS2.
Compile with Doxygen on any machine with no errors.
Run unit tests locally on any Tier 1 machine.
Run relevant consistency tests locally on all Tier 1 machines.
Optional test.
DEPENDENCIES:
Add any links to pending PRs that are required prior to merging this PR. For example:
ufs-community/UFS_UTILS/pull/1028
DOCUMENTATION:
All new and updated source code must be documented with Doxygen.
If this PR is contributing new capabilities that need to be documented, please also include updates to the RST files in the docs/source directory as supporting material.
ISSUE:
Fixes #850, #660 and ufs-community/ufs-srweather-app#1191
CONTRIBUTORS (optional):
@LarissaReames-NOAA