Update Intel compiler in CISM Wrapper

[Katetc]

Jim recently made updates to Derecho modules in a ccs_config tag here:
https://github.com/ESMCI/ccs_config_cesm/releases/tag/ccs_config_cesm1.0.75

This has been causing problems with CAM, CLM and MOSART due to some issues with NaN's and short-circuit "if" statement evaluations in Debug mode. I need to run some tests with CISM-Wrapper to see if we are going to have to make any sweeping changes to support these new modules.

Note: at this point, I haven't heard anything from the CESM testing group about issues with CISM. However, it's very possible that there are no "debug" CISM tests in the alpha suite. It would be very nice if I didn't have any problems, though.

[Katetc]

Part of this work will be to also update CISM standalone build scripts, and try and keep CISM and CESM in sync with libraries, modules, compilers and flags. Here's a relevant email conversation:

From Bill:

I was wondering if you could look at the standalone CISM build and think about what changes we'd need to be consistent with the CESM build – at least for flags that could conceivably lead to local sums that depend on the processor count. (As distinct from global sums across tasks, which seem to be handled just fine by the shr_reprosum code.)
Following up on this: In my latest Antarctic test, I'm getting what appears to be another semi-random 1-bit flip. I'd like to try Bill Sacks's suggestion of turning off the fused-mult-add option. Would it be sufficient to add "-no-fma" to CMAKE_C_FLAGS, CMAKE_CXX_FLAGS, and CMAKE_Fortran_FLAGS in derecho-intel-cmake?

From Kate:

I just wanted to get back to you on this. I found the flags used in a CESM test and compared them to those specified in our current Derecho-intel build. Maybe you've already been through this but here's what I see:

Cism standalone:

CMAKE_Fortran_FLAGS:STRING="-qno-opt-dynamic-align -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model source -qopt-report -O2 -debug minimal "

CESM:

CMAKE_Fortran_FLAGS:STRING="-O2 -debug minimal -no-fma -qno-opt-dynamic-align -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model source -qopt-report -march=core-avx2 [followed by a lot of library and module paths, plus some CMake flags for libraries, etc]"

So, the only differences here are the "-no-fma" and "-march=core-avx2". I think you know about -no-fma and we could add it to the stand alone build script easily. I'm not sure about "-march=core-avx2". Based on a little googling, it seems to be a CPU-specific optimization flag that is largely out of style and not used by anybody except, I guess, CESM. We can stick it in the build script to see if it does anything for you.

The answer to your second email is "yes" and I would suggest making sure that both the ".csh" and the bash scripts are updated together.

The question about flags that might affect local sums between builds is a bit confusing to me. We can have the same flags for both builds. But I think a bigger issue would be the modules/libraries and compiler versions that are used. One of the many nice things about the CIME case control system is that CIME is updated with the new compilers and appropriate library updates as they become available. Traditionally Jim runs an ensemble consistency test with the updates to make sure that the changes don't change the climate of the model (they often do change answers). I don't currently have a process in place for updating the cism standalone build scripts. I probably should have a note that each time I update CIME in the CISM-wrapper, I double check the stand alone build so those can be updated together. However, I don't update CIME in CISM-wrapper as often as it is updated in CESM, so we could still very well be behind the wheel on that one. I think it's just the consequence of maintaining two separate build systems.

Once we are passing the ERP test (exact restart with changing processor counts) we will be tested on this in the CISM-wrapper suite, in the CTSM suite and we can add a test for this in the CESM alpha tag suite. That way, if CESM updates libraries or flags that impact this feature, the test will fail and we'll know right away.
Also, for the libraries between builds, in CISM-wrapper (with CIME) this is the currently loaded software stack:
Currently Loaded Modules:

1. cesmdev/1.0 (H,S) 7) intel/2024.2.1 13) hdf5-mpi/1.12.3

2. ncarenv/24.12 (S) 8) mkl/2024.2.2 14) netcdf-mpi/4.9.3

3. conda/latest 9) kokkos/4.2.01 15) parallel-netcdf/1.14.0

4. nco/5.3.1 10) ncarcompilers/1.0.0 16) parallelio/2.6.6

5. craype/2.7.31 11) libfabric/1.15.2.0 17) esmf/8.8.1

6. cmake/3.26.6 12) cray-mpich/8.1.29

For CISM standalone it looks like:

module purge
module load ncarenv/23.09
module load intel/2023.2.1
module load ncarcompilers/1.0.0
module load cray-mpich/8.1.27
module load mkl/2023.2.0
module load netcdf/4.9.2
module load cmake/3.26.3

So, yes, Cism standalone is behind the latest here. We should probably update these. I would prefer to do it seperate from the repro-sums PR if possible. Either before or after.