The latest MPAS-Model cannot build using `make -j8`

[guoqing-noaa]

We are trying to update rrfs-workflow to use the latest MPAS-Model v8.3.1-2.16 (NOAA-EMC/rrfs-workflow#1356).

But the lastest code failed to build on Ursa and Gaea.

The error messages are something as follows:

mpif90 -D_MPI -DCORE_ATMOSPHERE -DMPAS_NAMELIST_SUFFIX=atmosphere -DMPAS_EXE_NAME=atmosphere_model -DSINGLE_PRECISION -DMPAS_NATIVE_TIMERS -DMPAS_GIT_VERSION=unknown -DMPAS_BUILD_TARGET=ifort_icx -DMPAS_SMIOL_SUPPORT -DMPAS_USE_MPI_F08 -O3 -convert big_endian -free -align array64byte -c mpas_smoke_config.F90 -I/autofs/ncrc-svm1_proj/epic/spack-stack/c6/spack-stack-1.9.3/envs/ue-oneapi-2024.2.1/install/oneapi/2024.2.1/parallel-netcdf-1.12.3-r72anpe/include -I/tmp/gge/rrfs-workflow/sorc/MPAS-Model/src/external/SMIOL -I/autofs/ncrc-svm1_proj/epic/spack-stack/c6/spack-stack-1.9.3/envs/ue-oneapi-2024.2.1/install/oneapi/2024.2.1/parallel-netcdf-1.12.3-r72anpe/include -I/tmp/gge/rrfs-workflow/sorc/MPAS-Model/src/core_atmosphere/physics/physics_noahmp/drivers/mpas -I/tmp/gge/rrfs-workflow/sorc/MPAS-Model/src/core_atmosphere/physics/physics_noahmp/utility -I/tmp/gge/rrfs-workflow/sorc/MPAS-Model/src/core_atmosphere/physics/physics_noahmp/src -I.. -I../../../../framework -I../../../../external/esmf_time_f90
../../mpas_atmphys_constants.F(10): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [MPAS_KIND_TYPES] use mpas_kind_types
-----^ ifort: remark #10448: Intel(R) Fortran Compiler Classic (ifort) is now deprecated and will be discontinued late 2024. Intel recommends that customers transition now to using the LLVM-based Intel(R) Fortran Compiler (ifx) for continued Windows* and Linux* support, new language support, new language features, and optimizations. Use '-diag-disable=10448' to disable this message. ../../mpas_atmphys_constants.F(11): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [MPAS_CONSTANTS] use mpas_constants, only : pii, cp, gravity, R_d => rgas, rvord
-----^ ../../mpas_atmphys_constants.F(37): error #6683: A kind type parameter must be a compile-time constant. [RKIND]
real(kind=RKIND),parameter:: c0 = 0.00000 -----------^
../../mpas_atmphys_constants.F(38): error #6683: A kind type parameter must be a compile-time constant. [RKIND] real(kind=RKIND),parameter:: c1 = 1.00000 -----------^ ../../mpas_atmphys_constants.F(40): error #6683: A kind type parameter must be a compile-time constant. [RKIND] real(kind=RKIND),parameter:: P0 = 100000. !reference pressure [Pa]
-----------^ ../../mpas_atmphys_constants.F(41): error #6683: A kind type parameter must be a compile-time constant. [RKIND]
real(kind=RKIND),parameter:: t00 = 273.15 !reference temperarure [K]
-----------^
../../mpas_atmphys_constants.F(42): error #6683: A kind type parameter must be a compile-time constant. [RKIND]
real(kind=RKIND),parameter:: R_v = 461.6 !gas constant for water vapor [J/kg/K]
-----------^
../../mpas_atmphys_constants.F(43): error #6683: A kind type parameter must be a compile-time constant. [RKIND]
......
......

At the same time, make -j1 and CMake (build.sh) can build the model succesfully.

After debugging, I found the make -j8 build failure came from the situation that both core_mynnsfc and core_physics_noahmp didn't specify their dependencies on core_physics_init. In the parallel build race condition, core_mynnsfc and core_physics_noahmp may start immediately while core_physics_init is still compiling init objects and hence the build fails.

Not sure why this bug is not detected by the github CI tests. I checked the github action build scripts and did see the -j8 tests there.