Make NetCDF the Default Format for CRTM LUTs (v3.2.0)

[BenjaminTJohnson]

Description

This PR updates the CRTM to use NetCDF as the default format for all Look-Up Table (LUT) interfaces (Aerosol, Cloud, Spectral, Transmittance, and Emissivity), replacing the legacy Binary default. This aligns the codebase with modern data standards while maintaining robust backward compatibility for legacy binary datasets.

Key Changes

1. Default Format Switch:

   • Updated CRTM_LifeCycle.f90 to set netCDF as the default format for all coefficient types.
   • Updated default filenames in CRTM_LifeCycle.f90 to point to their .nc4 / .nc equivalents.
   • Refactored low-level IO modules (e.g., SpcCoeff_IO.f90, TauCoeff_IO.f90, AerosolCoeff_IO.f90) to prioritize NetCDF when format flags are unspecified.

2. Robust Backward Compatibility (Smart Detection):

   • Implemented Smart Format Detection within CRTM_Init. The system now automatically checks for the existence of SpcCoeff/TauCoeff files. If the default (NetCDF) file is missing but a legacy Binary file exists, it seamlessly falls back to Binary format. This ensures that existing user applications and regression tests relying on binary data continue to work without modification.
   • Improved Default_File_Path logic to properly utilize the File_Path argument as a fallback for NetCDF loading when NC_File_Path is not provided.

3. Test Suite Updates:

   • Updated test_spc_io.f90 to explicitly request netCDF=.FALSE. to preserve the validity of binary IO unit tests.
   • Added test_becoeff_io_nc.f90 to increase coverage for NetCDF BeCoeff IO.
   • Streamlined test/CMakeLists.txt configuration.
   • Verified that all 193 CTests pass, covering both new NetCDF workflows and legacy Binary regressions.

4. New Utilities:

   • Added TauCoeff_Inspect tool for inspecting Transmittance Coefficient files.

Testing

• ctest -j4: 100% Passed (193/193 tests).
• Verified successful softlinking of NetCDF assets in the test environment.

[BenjaminTJohnson]

This requires A LOT of testing and integration checks in FV3-JEDI / UFO workflows.

[chengdang]

This could be adopted for all LUTs, to get ride of format specification.

[chengdang]

Thanks for this effort!
All tests passed and it looks good to me.

Meanwhile, I wonder if the merge of this PR into dev will perhaps break UFO, could initiate a collaborative PR merge with https://github.com/JCSDA-internal/ufo/pull/3825, in which the coefficient format argument is added in UFO. But, with the smart format detection in CRTM_Lifecycle, we probably won't need these any more.

In any case, ShihWei could probably help test this NetCDF feature in UFO. weiwilliam

[byoung-joo]

I am glad to see that this effort is happening!