Closed
Conversation
…t will also run on cascadelake, broadwell)
…-parens to limit re-ordering)
… the dependency versions
Collaborator
Author
|
Next attempt - adding back in all the flags (except for # This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
#
# Build with:
# - UM7 from dev-access-esm1.6 branch
# - MOM5 from development branch
# - CICE4 from ESM1.5 and change to CICE5 once CICE5 SPR has been updated.
spack:
specs:
- access-esm1p6@latest
packages:
# mom5:
# require:
# - '@git.ad1e41e2f7687a09e82addf5edbcca18fdd43ed5=access-esm1.6'
# - fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares
# -traceback"
# - cflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"
# - cxxflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"
cice4:
require:
# '/qljzx7x'
# - '@git.b51e3529bd78fd852760e348983e2334341f861d=access-esm1.5'
- '@git.access-esm1.6-2025.04.000=access-esm1.5'
- 'fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares
-traceback"'
- 'cflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'cxxflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
um7:
require:
# '/rjo4gre'
- '@git.1ea43190add8627fb317906d257f278763b55125=access-esm1.6'
- 'fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares
-traceback -flto"'
- 'cflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'cxxflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'ldflags="-flto -fuse-ld=lld"'
# Intermediate-level dependencies
gcom4:
require:
# '/4ahbikn'
- '@git.0f17657a3b21cda9c0f3c403d99667e436062e38=access-esm1.5'
- 'fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares
-traceback -flto"'
- cflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"
- cxxflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"
- 'ldflags="-flto -fuse-ld=lld"'
oasis3-mct:
require:
# '/6quqwyb'
- '@git.7036f26ece68c26083fec2fe96e3cb1faed7559d=access-esm1.5'
- 'fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares
-traceback"'
- 'cflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'cxxflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
access-fms:
require:
# '/finlfac'
- '@git.mom5-2025.05.000'
- 'fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares
-traceback -flto"'
- 'cflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'cxxflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'ldflags="-flto -fuse-ld=lld"'
access-generic-tracers:
require:
# '/z2t66eo'
- '@git.dev-2025.05.001'
- 'fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares
-traceback"'
- 'cflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'cxxflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'ldflags="-flto -fuse-ld=lld"'
access-mocsy:
require:
# '/meqj55h'
- '@git.2017.12.0'
- 'fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares
-traceback -flto"'
- 'cflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -flto"'
- 'cxxflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens"'
- 'ldflags="-flto -fuse-ld=lld"'
# System-level packages
openmpi:
require:
- '@4.1.5'
netcdf-c:
require:
- '@4.7.4'
netcdf-fortran:
require:
- '@4.5.2'
hdf5:
require:
- '@1.10.11'
gcc-runtime:
require:
- '%gcc'
glibc:
require:
- '%gcc'
# Preferences for all packages
all:
require:
- '%[email protected]'
- target=x86_64_v4
view: true
concretizer:
unify: false
modules:
default:
tcl:
include:
- access-esm1p6
- cice4
- um7
- mom5
projections:
access-esm1p6: '{name}/latest'
cice4: '{name}/access-esm1.6-2025.04.000-{hash:7}'
um7: '{name}/ad219516fcbb5386739e3e07ab7fef6a54634034-{hash:7}'
mom5: '{name}/oneapi-floatingpoint-fix-{hash:7}'
config:
install_tree:
root: $spack/../restricted/ukmo/release
source_cache: $spack/../restricted/ukmo/source_cache
build_stage:
- $TMPDIR/restricted/spack-stage
develop:
mom5:
spec: [email protected]=access-esm1.6 fflags="-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens -assume nan_compares -assume ieee_compares -traceback -no-ftz -no-daz -flto" ldflags="-flto -fuse-ld=lld"
path: /home/593/ms2335/codes/oneapi_compilation_dir/ACCESS-NRI-MOM5concretizer output [esm1p6-oneapi-mom5-crash] [~/codes/oneapi_compilation_dir @gadi06] spack concretize -f --fresh
==> Starting concretization
==> Warning: using "[email protected]" which is a deprecated version
==> Environment concretized in 156.00 seconds
==> Concretized access-esm1p6@latest
- dzyjcfw access-esm1p6@latest%[email protected] build_system=bundle cice=4 um=access-esm1.6 arch=linux-rocky8-x86_64_v4
- r2r4jig ^[email protected]=access-esm1.5%[email protected] cflags='-march=cascadelake -fprotect-parens -g3 -mtune=sapphirerapids' cxxflags='-march=cascadelake -mtune=sapphirerapids -fprotect-parens -g3' fflags='-assume ieee_compares -traceback -fprotect-parens -assume nan_compares -march=cascadelake -mtune=sapphirerapids -g3' build_system=makefile arch=linux-rocky8-x86_64_v4
- wbwjwms ^[email protected]%[email protected]~guile build_system=generic arch=linux-rocky8-x86_64_v4
- pbde7wd ^[email protected]%[email protected]~doc+pic+shared build_system=autotools patches=b050dbd arch=linux-rocky8-x86_64_v4
- aihbmoj ^[email protected]=access-esm1.5%[email protected] cflags='-march=cascadelake -fprotect-parens -g3 -mtune=sapphirerapids' cxxflags='-fprotect-parens -g3 -mtune=sapphirerapids -march=cascadelake' fflags='-march=cascadelake -mtune=sapphirerapids -assume ieee_compares -traceback -fprotect-parens -assume nan_compares -g3' ~deterministic~optimisation_report build_system=makefile arch=linux-rocky8-x86_64_v4
[e] 2b2g2oa ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64_v4
[e] rmko4qy ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
- ue3n6v5 ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
- qukvuoc ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
- szt3af3 ^[email protected]=access-esm1.6%[email protected] fflags='-flto -mtune=sapphirerapids -no-daz -march=cascadelake -no-ftz -fprotect-parens -assume ieee_compares -g3 -assume nan_compares -traceback' ldflags='-fuse-ld=lld -flto' ~deterministic~ipo build_system=cmake build_type=RelWithDebInfo dev_path=/home/593/ms2335/codes/oneapi_compilation_dir/ACCESS-NRI-MOM5 generator=make arch=linux-rocky8-x86_64_v4
- 3dfnrqa ^[email protected]=2025.05.000%[email protected] cflags='-march=cascadelake -mtune=sapphirerapids -g3 -fprotect-parens' cxxflags='-fprotect-parens -march=cascadelake -g3 -mtune=sapphirerapids' fflags='-march=cascadelake -assume ieee_compares -flto -g3 -mtune=sapphirerapids -traceback -assume nan_compares -fprotect-parens' ldflags='-fuse-ld=lld -flto' ~gfs_phys+internal_file_nml~ipo~large_file~pic~shared build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- x6wkgoi ^[email protected]=2025.05.001%[email protected] cflags='-fprotect-parens -march=cascadelake -mtune=sapphirerapids -g3' cxxflags='-fprotect-parens -mtune=sapphirerapids -g3 -march=cascadelake' fflags='-march=cascadelake -traceback -assume nan_compares -fprotect-parens -g3 -mtune=sapphirerapids -assume ieee_compares' ldflags='-fuse-ld=lld -flto' ~ipo+use_access_fms build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- im7nk62 ^[email protected]=2017.12.0%[email protected] cflags='-flto -march=cascadelake -fprotect-parens -mtune=sapphirerapids -g3' cxxflags='-march=cascadelake -mtune=sapphirerapids -g3 -fprotect-parens' fflags='-assume ieee_compares -fprotect-parens -march=cascadelake -traceback -mtune=sapphirerapids -g3 -flto -assume nan_compares' ldflags='-fuse-ld=lld -flto' build_system=makefile arch=linux-rocky8-x86_64_v4
[e] dltod3k ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky8-x86_64_v4
- 6pglprf ^[email protected]%[email protected]~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools arch=linux-rocky8-x86_64_v4
- dcf7t2v ^[email protected]%[email protected]~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- na347ve ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
- 3nn6pr5 ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64_v4
- u3tmteg ^[email protected]=access-esm1.6%[email protected] cflags='-g3 -mtune=sapphirerapids -fprotect-parens -march=cascadelake' cxxflags='-g3 -march=cascadelake -fprotect-parens -mtune=sapphirerapids' fflags='-assume ieee_compares -g3 -fprotect-parens -mtune=sapphirerapids -march=cascadelake -flto -traceback -assume nan_compares' ldflags='-fuse-ld=lld -flto' build_system=generic optim=high arch=linux-rocky8-x86_64_v4
- 2tzqhij ^[email protected]%[email protected] build_system=makefile arch=linux-rocky8-x86_64_v4
- qipwamx ^[email protected]%[email protected] build_system=generic site=none arch=linux-rocky8-x86_64_v4
- q4l5ejj ^[email protected]=access-esm1.5%[email protected] cflags='-fprotect-parens -march=cascadelake -mtune=sapphirerapids -g3' cxxflags='-march=cascadelake -g3 -mtune=sapphirerapids -fprotect-parens' fflags='-fprotect-parens -assume ieee_compares -flto -traceback -march=cascadelake -mtune=sapphirerapids -assume nan_compares -g3' ldflags='-fuse-ld=lld -flto' +mpi build_system=generic arch=linux-rocky8-x86_64_v4 |
Collaborator
Author
|
The [esm1p6-oneapi-mom5-crash] [~/codes/oneapi_compilation_dir/v4_with_mtune_spack_install @gadi06] spack concretize -f --fresh
==> Starting concretization
==> Warning: using "[email protected]" which is a deprecated version
==> Environment concretized in 120.52 seconds
==> Concretized access-esm1p6@latest
- c4wvtow access-esm1p6@latest%[email protected] build_system=bundle cice=4 um=access-esm1.6 arch=linux-rocky8-x86_64_v4
- igu6bkl ^[email protected]=access-esm1.5%[email protected] cflags='-march=cascadelake -fprotect-parens -g3 -mtune=sapphirerapids' cxxflags='-g3 -mtune=sapphirerapids -march=cascadelake -fprotect-parens' fflags='-march=cascadelake -mtune=sapphirerapids -g3 -fprotect-parens -assume ieee_compares -traceback -assume nan_compares' build_system=makefile arch=linux-rocky8-x86_64_v4
[+] wbwjwms ^[email protected]%[email protected]~guile build_system=generic arch=linux-rocky8-x86_64_v4
[+] pbde7wd ^[email protected]%[email protected]~doc+pic+shared build_system=autotools patches=b050dbd arch=linux-rocky8-x86_64_v4
- 2w6pucb ^[email protected]=access-esm1.5%[email protected] cflags='-g3 -fprotect-parens -mtune=sapphirerapids -march=cascadelake' cxxflags='-mtune=sapphirerapids -g3 -fprotect-parens -march=cascadelake' fflags='-assume nan_compares -fprotect-parens -assume ieee_compares -traceback -g3 -march=cascadelake -mtune=sapphirerapids' ~deterministic~optimisation_report build_system=makefile arch=linux-rocky8-x86_64_v4
[e] 2b2g2oa ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64_v4
[e] rmko4qy ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
[+] ue3n6v5 ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
[+] qukvuoc ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
- 3gr2pbt ^[email protected]=access-esm1.6%[email protected] fflags='-no-ftz -assume ieee_compares -march=cascadelake -mtune=sapphirerapids -no-daz -traceback -g3 -fprotect-parens -flto -assume nan_compares' ldflags='-flto -fuse-ld=lld' ~deterministic~ipo build_system=cmake build_type=RelWithDebInfo dev_path=/home/593/ms2335/codes/oneapi_compilation_dir/ACCESS-NRI-MOM5 generator=make arch=linux-rocky8-x86_64_v4
- gixlo3e ^[email protected]=2025.05.000%[email protected] cflags='-march=cascadelake -mtune=sapphirerapids -fprotect-parens -g3' cxxflags='-fprotect-parens -mtune=sapphirerapids -march=cascadelake -g3' fflags='-flto -traceback -assume nan_compares -march=cascadelake -g3 -assume ieee_compares -fprotect-parens -mtune=sapphirerapids' ldflags='-fuse-ld=lld -flto' ~gfs_phys+internal_file_nml~ipo~large_file~pic~shared build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- dywpep3 ^[email protected]=2025.05.001%[email protected] cflags='-march=cascadelake -mtune=sapphirerapids -fprotect-parens -g3' cxxflags='-march=cascadelake -fprotect-parens -mtune=sapphirerapids -g3' fflags='-fprotect-parens -march=cascadelake -mtune=sapphirerapids -g3 -assume ieee_compares -assume nan_compares -traceback' ldflags='-fuse-ld=lld -flto' ~ipo+use_access_fms build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- f3qbo3y ^[email protected]=2017.12.0%[email protected] cflags='-mtune=sapphirerapids -flto -march=cascadelake -g3 -fprotect-parens' cxxflags='-mtune=sapphirerapids -fprotect-parens -g3 -march=cascadelake' fflags='-assume nan_compares -assume ieee_compares -traceback -mtune=sapphirerapids -fprotect-parens -g3 -march=cascadelake -flto' ldflags='-flto -fuse-ld=lld' build_system=makefile arch=linux-rocky8-x86_64_v4
[e] dltod3k ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky8-x86_64_v4
[+] 6pglprf ^[email protected]%[email protected]~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools arch=linux-rocky8-x86_64_v4
[+] dcf7t2v ^[email protected]%[email protected]~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
[+] na347ve ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
[+] 3nn6pr5 ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64_v4
- ptau5my ^[email protected]=access-esm1.6%[email protected] cflags='-mtune=sapphirerapids -g3 -march=cascadelake -fprotect-parens' cxxflags='-fprotect-parens -g3 -march=cascadelake -mtune=sapphirerapids' fflags='-mtune=sapphirerapids -assume ieee_compares -traceback -fprotect-parens -g3 -assume nan_compares -flto=thin -march=cascadelake' ldflags='-fuse-ld=lld -flto=thin' build_system=generic optim=high arch=linux-rocky8-x86_64_v4
[+] 2tzqhij ^[email protected]%[email protected] build_system=makefile arch=linux-rocky8-x86_64_v4
[+] qipwamx ^[email protected]%[email protected] build_system=generic site=none arch=linux-rocky8-x86_64_v4
- 35vgioh ^[email protected]=access-esm1.5%[email protected] cflags='-g3 -fprotect-parens -march=cascadelake -mtune=sapphirerapids' cxxflags='-fprotect-parens -march=cascadelake -mtune=sapphirerapids -g3' fflags='-flto -traceback -fprotect-parens -mtune=sapphirerapids -g3 -assume nan_compares -march=cascadelake -assume ieee_compares' ldflags='-fuse-ld=lld -flto' +mpi build_system=generic arch=linux-rocky8-x86_64_v4
==> Updating view at /g/data/tm70/ms2335/spack/0.22/environments/esm1p6-oneapi-mom5-crash/.spack-env/view |
Collaborator
Author
|
Still did not build. Removing (Might have to go back to |
Collaborator
Author
|
Concretizer output: [esm1p6-oneapi-mom5-crash] [~/codes/oneapi_compilation_dir/v4_with_mtune_spack_install @gadi09] spack concretize -f --fresh
==> Starting concretization
==> Warning: using "[email protected]" which is a deprecated version
==> Environment concretized in 116.87 seconds
==> Concretized access-esm1p6@latest
- 4fik744 access-esm1p6@latest%[email protected] build_system=bundle cice=4 um=access-esm1.6 arch=linux-rocky8-x86_64_v4
- jxisy5h ^[email protected]=access-esm1.5%[email protected] cflags='-fprotect-parens -march=cascadelake -mtune=sapphirerapids -g3' cxxflags='-g3 -mtune=sapphirerapids -march=cascadelake -fprotect-parens' fflags='-march=cascadelake -assume nan_compares -traceback -fprotect-parens -g3 -assume ieee_compares -mtune=sapphirerapids' build_system=makefile arch=linux-rocky8-x86_64_v4
[+] wbwjwms ^[email protected]%[email protected]~guile build_system=generic arch=linux-rocky8-x86_64_v4
[+] pbde7wd ^[email protected]%[email protected]~doc+pic+shared build_system=autotools patches=b050dbd arch=linux-rocky8-x86_64_v4
- 2ievlm4 ^[email protected]=access-esm1.5%[email protected] cflags='-fprotect-parens -mtune=sapphirerapids -g3 -march=cascadelake' cxxflags='-fprotect-parens -g3 -march=cascadelake -mtune=sapphirerapids' fflags='-mtune=sapphirerapids -traceback -assume nan_compares -march=cascadelake -fprotect-parens -g3 -assume ieee_compares' ~deterministic~optimisation_report build_system=makefile arch=linux-rocky8-x86_64_v4
[e] 2b2g2oa ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64_v4
[e] rmko4qy ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
[+] ue3n6v5 ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
[+] qukvuoc ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
- h6b3moi ^[email protected]=access-esm1.6%[email protected] fflags='-flto -march=cascadelake -assume nan_compares -fprotect-parens -traceback -g3 -no-ftz -mtune=sapphirerapids -no-daz -assume ieee_compares' ldflags='-flto -fuse-ld=lld' ~deterministic~ipo build_system=cmake build_type=RelWithDebInfo dev_path=/home/593/ms2335/codes/oneapi_compilation_dir/ACCESS-NRI-MOM5 generator=make arch=linux-rocky8-x86_64_v4
- bhl42ry ^[email protected]=2025.05.000%[email protected] cflags='-mtune=sapphirerapids -g3 -march=cascadelake -fprotect-parens' cxxflags='-mtune=sapphirerapids -march=cascadelake -g3 -fprotect-parens' fflags='-fprotect-parens -assume ieee_compares -traceback -flto -assume nan_compares -march=cascadelake -mtune=sapphirerapids -g3' ldflags='-fuse-ld=lld -flto' ~gfs_phys+internal_file_nml~ipo~large_file~pic~shared build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- jhspdad ^[email protected]=2025.05.001%[email protected] cflags='-fprotect-parens -g3 -mtune=sapphirerapids -march=cascadelake' cxxflags='-fprotect-parens -g3 -mtune=sapphirerapids -march=cascadelake' fflags='-mtune=sapphirerapids -g3 -traceback -assume nan_compares -march=cascadelake -assume ieee_compares -fprotect-parens' ldflags='-flto -fuse-ld=lld' ~ipo+use_access_fms build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- 3cqxsww ^[email protected]=2017.12.0%[email protected] cflags='-fprotect-parens -march=cascadelake -mtune=sapphirerapids -g3 -flto' cxxflags='-fprotect-parens -g3 -march=cascadelake -mtune=sapphirerapids' fflags='-fprotect-parens -march=cascadelake -g3 -assume nan_compares -mtune=sapphirerapids -assume ieee_compares -flto -traceback' ldflags='-fuse-ld=lld -flto' build_system=makefile arch=linux-rocky8-x86_64_v4
[e] dltod3k ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky8-x86_64_v4
[+] 6pglprf ^[email protected]%[email protected]~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools arch=linux-rocky8-x86_64_v4
[+] dcf7t2v ^[email protected]%[email protected]~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
[+] na347ve ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
[+] 3nn6pr5 ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64_v4
- acbsc56 ^[email protected]=access-esm1.6%[email protected] cflags='-fprotect-parens -march=cascadelake -mtune=sapphirerapids -g3' cxxflags='-march=cascadelake -mtune=sapphirerapids -fprotect-parens -g3' fflags='-assume ieee_compares -fprotect-parens -traceback -g3 -march=cascadelake -assume nan_compares -mtune=sapphirerapids' build_system=generic optim=high arch=linux-rocky8-x86_64_v4
[+] 2tzqhij ^[email protected]%[email protected] build_system=makefile arch=linux-rocky8-x86_64_v4
[+] qipwamx ^[email protected]%[email protected] build_system=generic site=none arch=linux-rocky8-x86_64_v4
- zra3baa ^[email protected]=access-esm1.5%[email protected] cflags='-g3 -fprotect-parens -march=cascadelake -mtune=sapphirerapids' cxxflags='-fprotect-parens -march=cascadelake -g3 -mtune=sapphirerapids' fflags='-flto -traceback -mtune=sapphirerapids -assume nan_compares -march=cascadelake -fprotect-parens -g3 -assume ieee_compares' ldflags='-flto -fuse-ld=lld' +mpi build_system=generic arch=linux-rocky8-x86_64_v4 |
Collaborator
Author
|
This failed still with Removing Concretizer output |
Collaborator
Author
|
Build succeeded - submitted payu job. |
Collaborator
Author
|
Final patch was generated with this command: [~/codes/oneapi_compilation_dir/ACCESS-NRI-MOM5 @gadi04] git diff -w *.F90 > mom5_oneapi_floating_invalid_running_with_payu_June13_nowhitespace_changes_with_minusw_flag.patchThe actual patch: diff --git a/src/mom5/ocean_core/ocean_sbc.F90 b/src/mom5/ocean_core/ocean_sbc.F90
index 4d40cec..07e2efc 100644
--- a/src/mom5/ocean_core/ocean_sbc.F90
+++ b/src/mom5/ocean_core/ocean_sbc.F90
@@ -3506,6 +3506,7 @@ subroutine get_ocean_sbc(Time, Ice_ocean_boundary, Thickness, Dens, Ext_mode, T_
! and it is for these schemes we need to reach into halos with smf_bgrid.
Velocity%smf_cgrid(:,:,:) = 0.0
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc,iec
if(Grd%umask(i,j,1) + Grd%umask(i,j-1,1) > 0.0) then
Velocity%smf_cgrid(i,j,1) = &
diff --git a/src/mom5/ocean_core/ocean_thickness.F90 b/src/mom5/ocean_core/ocean_thickness.F90
index 6b0df96..0a47de9 100644
--- a/src/mom5/ocean_core/ocean_thickness.F90
+++ b/src/mom5/ocean_core/ocean_thickness.F90
@@ -1358,6 +1358,7 @@ subroutine thickness_initialize(Grid, Thickness)
endif
do j=jsd,jed
+!DIR$ NOVECTOR
do i=isd,ied
if(Grid%kmt(i,j) > 1 .and. Thickness%pbot0(i,j) /= 0.0) then
Thickness%pbot0r(i,j) = 1.0/Thickness%pbot0(i,j)
@@ -1519,6 +1520,7 @@ subroutine ocean_thickness_init_adjust(Grid, Time, Dens, Ext_mode, Thickness)
enddo
enddo
do j=jsd,jed
+!DIR$ NOVECTOR
do i=isd,ied
if(Grid%kmt(i,j) > 1 .and. Thickness%pbot0(i,j) /= 0.0) then
Thickness%pbot0r(i,j) = 1.0/Thickness%pbot0(i,j)
@@ -1746,6 +1748,7 @@ subroutine ocean_thickness_init_adjust(Grid, Time, Dens, Ext_mode, Thickness)
enddo
enddo
do j=jsd,jed
+!DIR$ NOVECTOR
do i=isd,ied
if(Grid%kmt(i,j) > 1 .and. Thickness%pbot0(i,j) /= 0.0) then
Thickness%pbot0r(i,j) = 1.0/Thickness%pbot0(i,j)
@@ -2253,6 +2256,7 @@ subroutine thickness_restart (Time, Grid, Ext_mode, Thickness, introduce_blobs)
! inverse pbot0 values
Thickness%pbot0r = 0.0
do j=jsd,jed
+!DIR$ NOVECTOR
do i=isd,ied
if(Grid%kmt(i,j) > 1 .and. Thickness%pbot0(i,j) /= 0.0) then
Thickness%pbot0r(i,j) = 1.0/Thickness%pbot0(i,j)
diff --git a/src/mom5/ocean_core/ocean_topog.F90 b/src/mom5/ocean_core/ocean_topog.F90
index 2f90da9..297da38 100644
--- a/src/mom5/ocean_core/ocean_topog.F90
+++ b/src/mom5/ocean_core/ocean_topog.F90
@@ -357,8 +357,10 @@ subroutine ocean_topog_init (Domain, Grid, grid_file, vert_coordinate_type)
! set inverse depth on t-cells
Grid%htr = 0.0
do j=jsd,jed
+!DIR$ NOVECTOR
do i=isd,ied
if(Grid%kmt(i,j) > 0 .and. Grid%ht(i,j) /= 0.0) then
+!!! Interesting -> checking for both kmt and ht > 0.0, and without NOVECTOR) still crashed with division by zero.
Grid%htr(i,j) = 1.0/Grid%ht(i,j)
endif
enddo
diff --git a/src/mom5/ocean_diag/ocean_tracer_diag.F90 b/src/mom5/ocean_diag/ocean_tracer_diag.F90
index a9846d9..37b3ed7 100644
--- a/src/mom5/ocean_diag/ocean_tracer_diag.F90
+++ b/src/mom5/ocean_diag/ocean_tracer_diag.F90
@@ -1114,6 +1114,7 @@ subroutine calc_mixed_layer_depth(Thickness, salinity, theta, rho, pressure, &
do k=2,nk
km1 = k-1
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc,iec
kb=Grd%kmt(i,j)
if (kb == 0) then
diff --git a/src/mom5/ocean_param/lateral/ocean_submesoscale.F90 b/src/mom5/ocean_param/lateral/ocean_submesoscale.F90
index 90aad1e..5df758d 100644
--- a/src/mom5/ocean_param/lateral/ocean_submesoscale.F90
+++ b/src/mom5/ocean_param/lateral/ocean_submesoscale.F90
@@ -1426,10 +1426,24 @@ subroutine compute_bldepth(Time, Thickness, Dens, T_prog, surf_blthick)
enddo
enddo
enddo
- where(kblt>=min_kblt)
- buoy_freq_ave = sqrt(abs(buoy_freq_ave)/wrk1_2d)
- front_length_inv = min(front_length_const_inv, coriolis_param/(epsln+wrk1_2d*buoy_freq_ave))
- endwhere
+ do j=jsd,jed
+!DIR$ NOVECTOR
+ do i=isd,ied
+ if(kblt(i,j) >= min_kblt) then
+ buoy_freq_ave(i,j) = sqrt(abs(buoy_freq_ave(i,j))/wrk1_2d(i,j))
+ !MS (11 Jun, 2025): The first argument used to be front_length_const_inv but I have changed to front_length_inv to silence a compiler error
+ !front_length_inv is initialized to front_length_const_inv at the top of this block
+ front_length_inv(i,j) = min(front_length_inv(i,j), coriolis_param(i,j)/(epsln+wrk1_2d(i,j)*buoy_freq_ave(i,j)))
+ endif
+ enddo
+ enddo
+
+! MS: (11 Jun, 2025): The following code was commented out in the original code, and replaced with the double for loop code above.
+! !DIR$ NOVECTOR
+! where(kblt>=min_kblt)
+! buoy_freq_ave = sqrt(abs(buoy_freq_ave)/wrk1_2d)
+! front_length_inv = min(front_length_const_inv, coriolis_param/(epsln+wrk1_2d*buoy_freq_ave))
+! endwhere
endif
diff --git a/src/mom5/ocean_param/neutral/ocean_nphysicsC.F90 b/src/mom5/ocean_param/neutral/ocean_nphysicsC.F90
index d0adf3c..63e20a9 100644
--- a/src/mom5/ocean_param/neutral/ocean_nphysicsC.F90
+++ b/src/mom5/ocean_param/neutral/ocean_nphysicsC.F90
@@ -3243,6 +3243,7 @@ subroutine fx_flux_ndiffuse(Time, Thickness, T_prog, k)
kpkr = min(k+kr,nk)
do ip=0,1
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc-1,iec
drhodz = drhodzh(i+ip,j,k,kr)
@@ -3380,6 +3381,7 @@ subroutine fy_flux_ndiffuse(Time, Thickness, T_prog, k)
kpkr = min(k+kr,nk)
do jq=0,1
do j=jsc-1,jec
+!DIR$ NOVECTOR
do i=isc,iec
drhodz = drhodzh(i,j+jq,k,kr)
diff --git a/src/mom5/ocean_param/neutral/ocean_nphysics_util.F90 b/src/mom5/ocean_param/neutral/ocean_nphysics_util.F90
index 4b381d8..5e86a52 100644
--- a/src/mom5/ocean_param/neutral/ocean_nphysics_util.F90
+++ b/src/mom5/ocean_param/neutral/ocean_nphysics_util.F90
@@ -2406,6 +2406,7 @@ subroutine compute_eady_rate(Time, Thickness, T_prog, Dens, eady_termx, eady_ter
enddo
do k=1,nk
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc,iec
if(Thickness%depth_zt(i,j,k) <= eady_mld(i,j)) then
eady_rate(i,j,k) = Grd%tmask(i,j,k)*wrk2_2d(i,j)/(wrk1_2d(i,j)+epsln)
@@ -3134,6 +3135,7 @@ subroutine transport_on_rho_gm (Time, Dens, tx_trans_gm, ty_trans_gm)
do j = jsc,jec
if (rho_maxj(j) < Dens%potrho_ref(k_rho)) cycle
if (rho_minj(j) > Dens%potrho_ref(k_rho)) cycle
+!DIR$ NOVECTOR
do i = isc,iec
if ( Dens%potrho_ref(k_rho) > Dens%potrho(i,j,k) ) then
if (Dens%potrho_ref(k_rho) <= Dens%potrho(i,j,k+1)) then
diff --git a/src/mom5/ocean_param/sources/ocean_shortwave_csiro.F90 b/src/mom5/ocean_param/sources/ocean_shortwave_csiro.F90
index a960d2c..91eff84 100644
--- a/src/mom5/ocean_param/sources/ocean_shortwave_csiro.F90
+++ b/src/mom5/ocean_param/sources/ocean_shortwave_csiro.F90
@@ -560,12 +560,11 @@ subroutine sw_pen (z_sw, sw_fk)
! If the coefficients change, then the floor/ceiling needs
! to be reevaluated. shortwave fraction set to zero for
! depths greater than zmax_pen.
+!DIR$ NOVECTOR
do i=isc,iec
-
if ( z_sw(i,j) > zmax_pen .or. Grd%tmask(i,j,kb(i)) == 0) then
sw_fk(i,j) = 0.0
else
-
sw_fk(i,j) = F_vis * exp( -z_sw(i,j)/ssw_atten_depth(i,j) )
endif
diff --git a/src/mom5/ocean_param/vertical/ocean_vert_kpp_mom4p1.F90 b/src/mom5/ocean_param/vertical/ocean_vert_kpp_mom4p1.F90
index d83cb95..508b422 100644
--- a/src/mom5/ocean_param/vertical/ocean_vert_kpp_mom4p1.F90
+++ b/src/mom5/ocean_param/vertical/ocean_vert_kpp_mom4p1.F90
@@ -829,6 +829,7 @@ ierr = check_nml_error(io_status,'ocean_vert_kpp_mom4p1_nml')
do i=0,nni+1
zehat = deltaz*(i) + zmin
+!DIR$ NOVECTOR
do j=0,nnj+1
usta = deltau*(j) + umin
zeta = zehat/(usta**3+epsln)
@@ -1297,13 +1298,13 @@ subroutine vert_mix_kpp_mom4p1 (aidif, Time, Thickness, Velocity, T_prog, T_diag
blmc(i-1,j,k,2) +&
blmc(i+1,j,k,2) +&
blmc(i,j-1,k,2) +&
- blmc(i,j+1,k,2)) / active_cells
+ blmc(i,j+1,k,2)) * 0.125 ! (instead of dividing by active_cells==8.0, multiplying by the exact 1/8)
wrk2(i,j,k) = &
(4.0*blmc(i,j,k,3) +&
blmc(i-1,j,k,3) +&
blmc(i+1,j,k,3) +&
blmc(i,j-1,k,3) +&
- blmc(i,j+1,k,3)) / active_cells
+ blmc(i,j+1,k,3)) * 0.125 ! (instead of dividing by active_cells==8.0, multiplying by the exact 1/8)
else
wrk1(i,j,k) = blmc(i,j,k,2)
wrk2(i,j,k) = blmc(i,j,k,3)
@@ -1700,6 +1701,7 @@ subroutine bldepth(Thickness, Velocity, sw_frac_zt, do_wave)
call wscale (iwscale_use_hbl_eq_zt, Thickness%depth_zt(:,:,kl), Velocity, do_wave)
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc,iec
if((kbl(i,j) == Grd%kmt(i,j))) then
@@ -1918,6 +1920,7 @@ subroutine bldepth(Thickness, Velocity, sw_frac_zt, do_wave)
!-----------------------------------------------------------------------
if(limit_with_hekman) then
do j=jsc,jec
+!DIR$ NOVECTOR
do i = isc,iec
if (bfsfc(i,j) > 0.0) then
hekman = cekman * Velocity%ustar(i,j) / (abs(Grd%f(i,j))+epsln)
@@ -1989,6 +1992,7 @@ subroutine bldepth(Thickness, Velocity, sw_frac_zt, do_wave)
!-----------------------------------------------------------------------
do j=jsc,jec
+!DIR$ NOVECTOR
do i = isc,iec
if (kbl(i,j)==0 .or. kbl(i,j)==1) cycle
@@ -2075,6 +2079,7 @@ subroutine wscale(iwscale_use_hbl_eq_zt, zt_kl, Velocity, do_wave)
if (iwscale_use_hbl_eq_zt == 1) then
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc,iec
zehat = von_karman * sigma(i,j) * zt_kl(i,j) * bfsfc(i,j)
@@ -2115,6 +2120,7 @@ subroutine wscale(iwscale_use_hbl_eq_zt, zt_kl, Velocity, do_wave)
else
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc,iec
zehat = von_karman * sigma(i,j) * hbl(i,j) * bfsfc(i,j)
@@ -2407,6 +2413,7 @@ tau = Time%tau
do ki=1,nk
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc,iec
!-----------------------------------------------------------------------
@@ -2657,6 +2664,7 @@ subroutine blmix_kpp(Thickness, Velocity, diff_cbt, visc_cbu, do_wave)
!-----------------------------------------------------------------------
do j=jsc,jec
+!DIR$ NOVECTOR
do i= isc,iec
if (kbl(i,j)== 0 .or. kbl(i,j)==1) cycle
sig = Thickness%depth_zt(i,j,kbl(i,j)-1) / (hbl(i,j)+epsln)
@@ -2671,6 +2679,7 @@ subroutine blmix_kpp(Thickness, Velocity, diff_cbt, visc_cbu, do_wave)
call wscale(iwscale_use_hbl_eq_zt, zt_kl_dummy, Velocity, do_wave)
do j=jsc,jec
+!DIR$ NOVECTOR
do i = isc,iec
if (kbl(i,j)==0 .or. kbl(i,j)==1) cycle
sig = Thickness%depth_zt(i,j,kbl(i,j)-1) / (hbl(i,j)+epsln)
@@ -2724,6 +2733,7 @@ real, dimension(isd:,jsd:,:), intent(in) :: visc_cbu
do ki=1,nk-1
do j=jsc,jec
+!DIR$ NOVECTOR
do i = isc,iec
if(ki == (kbl(i,j) - 1) ) then
diff --git a/src/mom5/ocean_param/vertical/ocean_vert_tidal.F90 b/src/mom5/ocean_param/vertical/ocean_vert_tidal.F90
index 49d71e7..d7644d9 100644
--- a/src/mom5/ocean_param/vertical/ocean_vert_tidal.F90
+++ b/src/mom5/ocean_param/vertical/ocean_vert_tidal.F90
@@ -812,6 +812,7 @@ ierr = check_nml_error(io_status,'ocean_vert_tidal_nml')
! mask out the deep ocean regions for the drag scheme.
if(drag_mask_deep .and. .not. use_legacy_methods) then
do j=jsc,jec
+!DIR$ NOVECTOR
do i=isc,iec
if(Grd%tmask(i,j,1) == 1.0) then
temporary = wrk1_2d(i,j)/(epsln+Grd%ht(i,j))
|
Collaborator
Author
|
Huh - it's not over. Got a floating invalid forrtl: error (65): floating invalid
Image PC Routine Line Source
libpthread-2.28.s 0000155379103990 Unknown Unknown Unknown
mom5_access_cm 0000000000CD6390 compute_buoyfreq 4828 ocean_density.F90
mom5_access_cm 0000000000CC868F ocean_density_ini 1174 ocean_density.F90
mom5_access_cm 0000000000C1CC36 ocean_model_init 1287 ocean_model.F90
mom5_access_cm 0000000000BF1CC5 main 371 ocean_solo.F90
mom5_access_cm 00000000004A218D Unknown Unknown Unknown
libc-2.28.so 0000155378B517E5 __libc_start_main Unknown Unknown
mom5_access_cm 00000000004A20AE Unknown Unknown Unknownwith the following code: do i=isd,ied
wrk4(i,j,k) = wrk3(i,j,k-1)*(Thickness%depth_zwt(i,j,k)-Thickness%depth_zt(i,j,k)) &
+wrk3(i,j,k) *(Thickness%depth_zt(i,j,k) -Thickness%depth_zwt(i,j,k-1))
wrk4(i,j,k) = wrk4(i,j,k)/(epsln+Thickness%dzt(i,j,k))
Dens%drhodz_zt(i,j,k) = wrk4(i,j,k)
enddo
|
Collaborator
Author
|
Previous job ran for 55 seconds before crashing with the floating invalid error :( Might need a different strategy to prevent these invalid errors - ignoring is an option since the vectorizer is causing the issue, science results should not(?) be impacted. Another option could be to initialise with "HUGE" numbers - so the divides produce ~0, rather than blowing up (since the default initialiser is 0). |
manodeep
changed the title
Collaborator
Author
|
Next crash forrtl: error (65): floating invalid
Image PC Routine Line Source
libpthread-2.28.s 000014D0A57FD990 Unknown Unknown Unknown
mom5_access_cm 0000000000CD8440 compute_buoyfreq 4890 ocean_density.F90
mom5_access_cm 0000000000CC86CF ocean_density_ini 1174 ocean_density.F90
mom5_access_cm 0000000000C1CC76 ocean_model_init 1287 ocean_model.F90
mom5_access_cm 0000000000BF1D05 main 371 ocean_solo.F90
mom5_access_cm 00000000004A21CD Unknown Unknown Unknown
libc-2.28.so 000014D0A524B7E5 __libc_start_main Unknown Unknown
mom5_access_cm 00000000004A20EE Unknown Unknown Unknownwhich has the code do i=isd,ied
wrk3(i,j,k) = wrk1(i,j,k-1)*(Thickness%depth_zwt(i,j,k)-Thickness%depth_zt(i,j,k)) &
+wrk1(i,j,k) *(Thickness%depth_zt(i,j,k) -Thickness%depth_zwt(i,j,k-1))
wrk4(i,j,k) = wrk2(i,j,k-1)*(Thickness%depth_zwt(i,j,k)-Thickness%depth_zt(i,j,k)) &
+wrk2(i,j,k) *(Thickness%depth_zt(i,j,k) -Thickness%depth_zwt(i,j,k-1))
wrk3(i,j,k) = wrk3(i,j,k)/(epsln+Thickness%dzt(i,j,k))
wrk4(i,j,k) = wrk4(i,j,k)/(epsln+Thickness%dzt(i,j,k))
Dens%dTdz_zt(i,j,k) = wrk3(i,j,k)
Dens%dSdz_zt(i,j,k) = wrk4(i,j,k)
enddo
|
Member
I think this is worth trying, as it would preserve vectorization and avoid the very long patch. |
Collaborator
|
55 seconds - how many model timestep is this? |
Collaborator
Author
|
@JhanSrbinovsky That was a crash from invalid floating point values - so the number of timesteps doesn't really matter. I am trialling the fix now - let's see |
Collaborator
Author
|
Next error: forrtl: error (65): floating invalid
Image PC Routine Line Source
libpthread-2.28.s 0000148F5ADB1990 Unknown Unknown Unknown
mom5_access_cm 0000000000EE5851 get_ocean_sbc 3414 ocean_sbc.F90
mom5_access_cm 0000000000C39F97 update_ocean_mode 1600 ocean_model.F90
mom5_access_cm 0000000000C03A8C main 450 ocean_solo.F90
mom5_access_cm 00000000004A214D Unknown Unknown Unknown
libc-2.28.so 0000148F5A7FF7E5 __libc_start_main Unknown Unknown
mom5_access_cm 00000000004A206E Unknown Unknown Unknownwhich points to the code: do i=isd,ied
rhosfc_inv(i,j) = Grd%tmask(i,j,1)/(epsln+Dens%rho(i,j,1,tau))
alphasfc(i,j) = -rhosfc_inv(i,j)*Dens%drhodT(i,j,1)
betasfc(i,j) = rhosfc_inv(i,j)*Dens%drhodS(i,j,1)
potrhosfc_inv = Grd%tmask(i,j,1)/(epsln+Dens%potrho(i,j,1))
alphasfc2(i,j) = -potrhosfc_inv*Dens%dpotrhodT(i,j,1)
betasfc2(i,j) = potrhosfc_inv*Dens%dpotrhodS(i,j,1)
enddoPresumably, these new ones are showing up because vectorization is profitable for |
Member
Collaborator
Author
|
@micaeljtoliveira Unfortunately |
Member
Oh well, was worth a shot. |
Collaborator
Author
|
Based on @micaeljtoliveira and my discussions, tried to ignore floating point warnings with Updated spack.yaml was: develop:
mom5:
spec: [email protected]=access-esm1.6 fflags="-g3 -march=cascadelake -mtune=sapp\
hirerapids -fprotect-parens -assume nan_compares -assume ieee_compares -traceback -no-ftz -no-daz -flto -fpe3" ldflags\
="-flto -fuse-ld=lld"
path: /home/593/ms2335/codes/oneapi_compilation_dir/ACCESS-NRI-MOM5
New crash (with forrtl: error (65): floating invalid
Image PC Routine Line Source
libpthread-2.28.s 0000150193DDF990 Unknown Unknown Unknown
mom5_access_cm 0000000000EF32C1 get_ocean_sbc 3899 ocean_sbc.F90
mom5_access_cm 0000000000C39F97 update_ocean_mode 1600 ocean_model.F90
mom5_access_cm 0000000000C03A8C main 450 ocean_solo.F90
mom5_access_cm 00000000004A214D Unknown Unknown Unknown
libc-2.28.so 000015019382D7E5 __libc_start_main Unknown Unknown
mom5_access_cm 00000000004A206E Unknown Unknown Unknownwith corresponding code: do j=jsc,jec
do i=isc,iec
tmp_runoff = max(0.0,runoff(i,j))
tmp_calving= max(0.0,calving(i,j))
T_prog(index_temp)%triver(i,j) = Grd%tmask(i,j,1) &
*(tmp_runoff *T_prog(index_temp)%trunoff(i,j) &
+tmp_calving*T_prog(index_temp)%tcalving(i,j)) &
/(epsln + tmp_runoff + tmp_calving)
T_prog(index_salt)%triver(i,j) = Grd%tmask(i,j,1) &
*(tmp_runoff *T_prog(index_salt)%trunoff(i,j) &
+tmp_calving*T_prog(index_salt)%tcalving(i,j)) &
/(epsln + tmp_runoff + tmp_calving)
enddo
enddo
|
Collaborator
Author
|
Hmm - looking through the logs - I can't see the Concretizer output - contains [esm1p6-oneapi-mom5-crash] [~/codes/oneapi_compilation_dir/v4_with_mtune_spack_install @gadi07] spack concretize -f --
fresh
==> Starting concretization
==> Warning: using "[email protected]" which is a deprecated version
==> Environment concretized in 132.18 seconds
==> Concretized access-esm1p6@latest
- csdy6qd access-esm1p6@latest%[email protected] build_system=bundle cice=4 um=access-esm1.6 arch=linux-rocky8-x86_64_v4
- bn3pt64 ^[email protected]=access-esm1.5%[email protected] cflags='-march=cascadelake -fprotect-parens -mtune=sapphirerapids -g3' cxxflags='-mtune=sapphirerapids -fprotect-parens -march=cascadelake -g3' fflags='-fprotect-parens -g3 -march=cascadelake -assume nan_compares -mtune=sapphirerapids -traceback -assume ieee_compares' build_system=makefile arch=linux-rocky8-x86_64_v4
[+] wbwjwms ^[email protected]%[email protected]~guile build_system=generic arch=linux-rocky8-x86_64_v4
[+] pbde7wd ^[email protected]%[email protected]~doc+pic+shared build_system=autotools patches=b050dbd arch=linux-rocky8-x86_64_v4
- mgsn6mv ^[email protected]=access-esm1.5%[email protected] cflags='-march=cascadelake -mtune=sapphirerapids -g3 -fprotect-parens' cxxflags='-mtune=sapphirerapids -fprotect-parens -g3 -march=cascadelake' fflags='-march=cascadelake -g3 -traceback -mtune=sapphirerapids -assume nan_compares -fprotect-parens -assume ieee_compares' ~deterministic~optimisation_report build_system=makefile arch=linux-rocky8-x86_64_v4
[e] 2b2g2oa ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64_v4
[e] rmko4qy ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
[+] ue3n6v5 ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
[+] qukvuoc ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
- qipgndr ^[email protected]=access-esm1.6%[email protected] fflags='-fprotect-parens -no-ftz -mtune=sapphirerapids -g3 -fpe3 -flto -march=cascadelake -traceback -assume nan_compares -no-daz -assume ieee_compares' ldflags='-flto -fuse-ld=lld' ~deterministic~ipo build_system=cmake build_type=RelWithDebInfo dev_path=/home/593/ms2335/codes/oneapi_compilation_dir/ACCESS-NRI-MOM5 generator=make arch=linux-rocky8-x86_64_v4
- prk3wji ^[email protected]=2025.05.000%[email protected] cflags='-g3 -fprotect-parens -mtune=sapphirerapids -march=cascadelake' cxxflags='-fprotect-parens -march=cascadelake -g3 -mtune=sapphirerapids' fflags='-g3 -assume ieee_compares -march=cascadelake -mtune=sapphirerapids -traceback -assume nan_compares -flto -fprotect-parens' ldflags='-flto -fuse-ld=lld' ~gfs_phys+internal_file_nml~ipo~large_file~pic~shared build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- nlhya4a ^[email protected]=2025.05.001%[email protected] cflags='-fprotect-parens -mtune=sapphirerapids -g3 -march=cascadelake' cxxflags='-mtune=sapphirerapids -g3 -march=cascadelake -fprotect-parens' fflags='-traceback -g3 -assume ieee_compares -mtune=sapphirerapids -fprotect-parens -march=cascadelake -assume nan_compares' ldflags='-flto -fuse-ld=lld' ~ipo+use_access_fms build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- f4fdtu5 ^[email protected]=2017.12.0%[email protected] cflags='-g3 -flto -march=cascadelake -mtune=sapphirerapids -fprotect-parens' cxxflags='-g3 -fprotect-parens -march=cascadelake -mtune=sapphirerapids' fflags='-fprotect-parens -assume nan_compares -flto -assume ieee_compares -traceback -march=cascadelake -mtune=sapphirerapids -g3' ldflags='-flto -fuse-ld=lld' build_system=makefile arch=linux-rocky8-x86_64_v4
[e] dltod3k ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky8-x86_64_v4
[+] 6pglprf ^[email protected]%[email protected]~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools arch=linux-rocky8-x86_64_v4
[+] dcf7t2v ^[email protected]%[email protected]~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
[+] na347ve ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
[+] 3nn6pr5 ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64_v4
- uwg5amk ^[email protected]=access-esm1.6%[email protected] cflags='-mtune=sapphirerapids -g3 -fprotect-parens -march=cascadelake' cxxflags='-march=cascadelake -g3 -mtune=sapphirerapids -fprotect-parens' fflags='-assume ieee_compares -march=cascadelake -g3 -traceback -fprotect-parens -assume nan_compares -mtune=sapphirerapids' build_system=generic optim=high arch=linux-rocky8-x86_64_v4
[+] 2tzqhij ^[email protected]%[email protected] build_system=makefile arch=linux-rocky8-x86_64_v4
[+] qipwamx ^[email protected]%[email protected] build_system=generic site=none arch=linux-rocky8-x86_64_v4
- 22d6mxr ^[email protected]=access-esm1.5%[email protected] cflags='-fprotect-parens -g3 -march=cascadelake -mtune=sapphirerapids' cxxflags='-g3 -mtune=sapphirerapids -march=cascadelake -fprotect-parens' fflags='-assume ieee_compares -fprotect-parens -g3 -mtune=sapphirerapids -march=cascadelake -traceback -assume nan_compares' +mpi build_system=generic arch=linux-rocky8-x86_64_v4
==> Updating view at /g/data/tm70/ms2335/spack/0.22/environments/esm1p6-oneapi-mom5-crash/.spack-env/view
[esm1p6-oneapi-mom5-crash] [~/codes/oneapi_compilation_dir/v4_with_mtune_spack_install @gadi07]
|
Collaborator
Author
|
The new mom5 exe, built with |
Collaborator
Author
|
The previous spack build (that crashed) must not have honoured the |
Collaborator
Author
|
The run (with Hmmmm - we might have to confirm that the 1-year results from oneAPI are statistically identical to the ifort runs. And then set off a longer run to test the climate results. |
Collaborator
Author
|
Turns out that the division bug is only present in the I ran [esm1p6-oneapi-mom5-crash] [~/codes/oneapi_compilation_dir/v4_with_mtune_spack_install @gadi02] spack concretize -f --
fresh
==> Starting concretization
==> Warning: using "[email protected]" which is a deprecated version
==> Environment concretized in 98.39 seconds
==> Concretized access-esm1p6@latest
- xtszmnt access-esm1p6@latest%[email protected] build_system=bundle cice=4 um=access-esm1.6 arch=linux-rocky8-x86_64_v4
- dmr2jaj ^[email protected]=access-esm1.5%[email protected] cflags='-g3 -mtune=sapphirerapids -march=cascadelake -fprotect-parens' cxxflags='-mtune=sapphirerapids -march=cascadelake -fprotect-parens -g3' fflags='-assume ieee_compares -g3 -fprotect-parens -flto -march=cascadelake -mtune=sapphirerapids -assume nan_compares -traceback' ldflags='-fuse-ld=lld -flto' build_system=makefile arch=linux-rocky8-x86_64_v4
- wbwjwms ^[email protected]%[email protected]~guile build_system=generic arch=linux-rocky8-x86_64_v4
- pbde7wd ^[email protected]%[email protected]~doc+pic+shared build_system=autotools patches=b050dbd arch=linux-rocky8-x86_64_v4
- ossiqrl ^[email protected]=access-esm1.5%[email protected] cflags='-g3 -mtune=sapphirerapids -fprotect-parens -march=cascadelake' cxxflags='-g3 -fprotect-parens -march=cascadelake -mtune=sapphirerapids' fflags='-traceback -mtune=sapphirerapids -assume ieee_compares -march=cascadelake -fprotect-parens -flto -g3 -assume nan_compares' ~deterministic~optimisation_report build_system=makefile arch=linux-rocky8-x86_64_v4
[e] 2b2g2oa ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64_v4
[e] rmko4qy ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
- ue3n6v5 ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
- qukvuoc ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
- gg636mh ^[email protected]=access-esm1.6%[email protected] cflags='-mtune=sapphirerapids -march=cascadelake -g3 -fprotect-parens' cxxflags='-mtune=sapphirerapids -fprotect-parens -march=cascadelake -g3' fflags='-fpe3 -flto -assume ieee_compares -march=cascadelake -mtune=sapphirerapids -traceback -g3 -assume nan_compares -fprotect-parens' ldflags='-fuse-ld=lld -flto' ~deterministic~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64_v4
- 2asqblt ^[email protected]=2025.05.000%[email protected] cflags='-g3 -fprotect-parens -march=cascadelake -mtune=sapphirerapids' cxxflags='-march=cascadelake -fprotect-parens -g3 -mtune=sapphirerapids' fflags='-assume ieee_compares -fprotect-parens -g3 -assume nan_compares -mtune=sapphirerapids -traceback -flto -march=cascadelake' ldflags='-flto -fuse-ld=lld' ~gfs_phys+internal_file_nml~ipo~large_file~pic~shared build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- 6v4xdni ^[email protected]=2025.05.001%[email protected] cflags='-fprotect-parens -mtune=sapphirerapids -g3 -march=cascadelake' cxxflags='-march=cascadelake -fprotect-parens -g3 -mtune=sapphirerapids' fflags='-assume ieee_compares -march=cascadelake -flto -mtune=sapphirerapids -g3 -assume nan_compares -traceback -fprotect-parens' ldflags='-fuse-ld=lld -flto' ~ipo+use_access_fms build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- rhudlpi ^[email protected]=2017.12.0%[email protected] cflags='-fprotect-parens -g3 -mtune=sapphirerapids -flto -march=cascadelake' cxxflags='-g3 -mtune=sapphirerapids -march=cascadelake -fprotect-parens' fflags='-fprotect-parens -traceback -mtune=sapphirerapids -assume nan_compares -flto -g3 -march=cascadelake -assume ieee_compares' ldflags='-flto -fuse-ld=lld' build_system=makefile arch=linux-rocky8-x86_64_v4
[e] dltod3k ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky8-x86_64_v4
- 6pglprf ^[email protected]%[email protected]~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools arch=linux-rocky8-x86_64_v4
- dcf7t2v ^[email protected]%[email protected]~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- na347ve ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
- 3nn6pr5 ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64_v4
- y6wyivt ^[email protected]=access-esm1.6%[email protected] cflags='-march=cascadelake -g3 -mtune=sapphirerapids -fprotect-parens' cxxflags='-g3 -fprotect-parens -march=cascadelake -mtune=sapphirerapids' fflags='-assume nan_compares -mtune=sapphirerapids -flto -traceback -assume ieee_compares -g3 -fprotect-parens -march=cascadelake' ldflags='-flto -fuse-ld=lld' build_system=generic optim=high arch=linux-rocky8-x86_64_v4
- 2tzqhij ^[email protected]%[email protected] build_system=makefile arch=linux-rocky8-x86_64_v4
- qipwamx ^[email protected]%[email protected] build_system=generic site=none arch=linux-rocky8-x86_64_v4
- prsuwgg ^[email protected]=access-esm1.5%[email protected] cflags='-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens' cxxflags='-g3 -mtune=sapphirerapids -march=cascadelake -fprotect-parens' fflags='-assume nan_compares -traceback -assume ieee_compares -fprotect-parens -march=cascadelake -g3 -mtune=sapphirerapids -flto' ldflags='-flto -fuse-ld=lld' +mpi build_system=generic arch=linux-rocky8-x86_64_v4 |
Collaborator
Author
|
The previous build crashed with a compile failure in - - '@git.7036f26ece68c26083fec2fe96e3cb1faed7559d=access-esm1.5'
+ - '@git.99d22674df8f304eb29c216e50d39ad4c0f24c27=access-esm1.5'The concretizer output from is [esm1p6-oneapi-mom5-crash] [~/codes/oneapi_compilation_dir/v4_with_mtune_spack_install @gadi08] spack concretize -f --
fresh
==> Starting concretization
==> Warning: using "[email protected]" which is a deprecated version
==> Environment concretized in 116.10 seconds
==> Concretized access-esm1p6@latest
- tjecqak access-esm1p6@latest%[email protected] build_system=bundle cice=4 um=access-esm1.6 arch=linux-rocky8-x86_64_v4
- 246762i ^[email protected]=access-esm1.5%[email protected] cflags='-fprotect-parens -mtune=sapphirerapids -march=cascadelake -g3' cxxflags='-mtune=sapphirerapids -march=cascadelake -fprotect-parens -g3' fflags='-flto -traceback -assume nan_compares -assume ieee_compares -g3 -mtune=sapphirerapids -fprotect-parens -march=cascadelake' ldflags='-flto -fuse-ld=lld' build_system=makefile arch=linux-rocky8-x86_64_v4
[+] wbwjwms ^[email protected]%[email protected]~guile build_system=generic arch=linux-rocky8-x86_64_v4
[+] pbde7wd ^[email protected]%[email protected]~doc+pic+shared build_system=autotools patches=b050dbd arch=linux-rocky8-x86_64_v4
- y2tg7b5 ^[email protected]=access-esm1.5%[email protected] cflags='-march=cascadelake -g3 -fprotect-parens -mtune=sapphirerapids' cxxflags='-march=cascadelake -g3 -mtune=sapphirerapids -fprotect-parens' fflags='-g3 -flto -traceback -mtune=sapphirerapids -march=cascadelake -fprotect-parens -assume ieee_compares -assume nan_compares' ~deterministic~optimisation_report build_system=makefile arch=linux-rocky8-x86_64_v4
[e] 2b2g2oa ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64_v4
[e] rmko4qy ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
[+] ue3n6v5 ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
[+] qukvuoc ^[email protected]%[email protected] build_system=generic arch=linux-rocky8-x86_64_v4
- 65rarsm ^[email protected]=access-esm1.6%[email protected] cflags='-mtune=sapphirerapids -g3 -fprotect-parens -march=cascadelake' cxxflags='-fprotect-parens -g3 -mtune=sapphirerapids -march=cascadelake' fflags='-g3 -fpe3 -mtune=sapphirerapids -march=cascadelake -traceback -flto -assume nan_compares -fprotect-parens -assume ieee_compares' ldflags='-fuse-ld=lld -flto' ~deterministic~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64_v4
- qit2psp ^[email protected]=2025.05.000%[email protected] cflags='-march=cascadelake -mtune=sapphirerapids -g3 -fprotect-parens' cxxflags='-mtune=sapphirerapids -fprotect-parens -g3 -march=cascadelake' fflags='-g3 -mtune=sapphirerapids -march=cascadelake -flto -assume ieee_compares -traceback -fprotect-parens -assume nan_compares' ldflags='-flto -fuse-ld=lld' ~gfs_phys+internal_file_nml~ipo~large_file~pic~shared build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- 5c3aotn ^[email protected]=2025.05.001%[email protected] cflags='-march=cascadelake -fprotect-parens -g3 -mtune=sapphirerapids' cxxflags='-march=cascadelake -g3 -mtune=sapphirerapids -fprotect-parens' fflags='-flto -mtune=sapphirerapids -assume ieee_compares -g3 -march=cascadelake -traceback -fprotect-parens -assume nan_compares' ldflags='-flto -fuse-ld=lld' ~ipo+use_access_fms build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
- iaxs5hu ^[email protected]=2017.12.0%[email protected] cflags='-fprotect-parens -mtune=sapphirerapids -march=cascadelake -flto -g3' cxxflags='-fprotect-parens -g3 -march=cascadelake -mtune=sapphirerapids' fflags='-assume nan_compares -g3 -traceback -assume ieee_compares -march=cascadelake -fprotect-parens -mtune=sapphirerapids -flto' ldflags='-flto -fuse-ld=lld' build_system=makefile arch=linux-rocky8-x86_64_v4
[e] dltod3k ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky8-x86_64_v4
[+] 6pglprf ^[email protected]%[email protected]~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools arch=linux-rocky8-x86_64_v4
[+] dcf7t2v ^[email protected]%[email protected]~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
[+] na347ve ^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-x86_64_v4
[+] 3nn6pr5 ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64_v4
- f2n4ekz ^[email protected]=access-esm1.6%[email protected] cflags='-g3 -march=cascadelake -mtune=sapphirerapids -fprotect-parens' cxxflags='-mtune=sapphirerapids -march=cascadelake -g3 -fprotect-parens' fflags='-fprotect-parens -traceback -march=cascadelake -assume ieee_compares -g3 -flto -assume nan_compares -mtune=sapphirerapids' ldflags='-flto -fuse-ld=lld' build_system=generic optim=high arch=linux-rocky8-x86_64_v4
[+] 2tzqhij ^[email protected]%[email protected] build_system=makefile arch=linux-rocky8-x86_64_v4
[+] qipwamx ^[email protected]%[email protected] build_system=generic site=none arch=linux-rocky8-x86_64_v4
- ry7drwd ^[email protected]=access-esm1.5%[email protected] cflags='-mtune=sapphirerapids -fprotect-parens -g3 -march=cascadelake' cxxflags='-march=cascadelake -mtune=sapphirerapids -g3 -fprotect-parens' fflags='-g3 -traceback -assume ieee_compares -assume nan_compares -march=cascadelake -fprotect-parens -mtune=sapphirerapids -flto' ldflags='-fuse-ld=lld -flto' +mpi build_system=generic arch=linux-rocky8-x86_64_v4
==> Updating view at /g/data/tm70/ms2335/spack/0.22/environments/esm1p6-oneapi-mom5-crash/.spack-env/view |
This was referenced Jun 19, 2025
Collaborator
Author
|
Now that I know the MOM5 crashes were from a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge
Debugging why PI crashes with oneAPII have had a full successful run with oneAPI - see here.
Now changing the PR to be instead focussed on the optimisation for the oneAPI compiled exeHeh - apparently we will be ping-ponging between debugging and optimising.
🚀 The latest prerelease
access-esm1p6/pr88-23at a9dc871 is here: #88 (comment) 🚀