Building and running OM3 with modified MOM6 source code

[jbisits]

Hi OM3 team,

I am looking to build OM3 with modified MOM6 source code I have on gadi. The modified MOM6 source code has a new module added, MOM_numerical_mixing.F90, within the MOM6/src/diagnostics directory. I have been able to build an executable from this MOM6 source code with mom6-ninja-nci and run ANU-TUB with it.

The instructions on how to setup spack and the development environment were thorough and I think I now have everything in place. My development environment spack.yaml has the following appended to it

 develop:
    access-mom6:
        spec: access-mom6@2025.07.000
        path: /g/data/e14/jb2381/MOM6-examples/src/MOM6-nm # path to the modified source code

After running spack concretize -f --reuse-deps, spack install returns the following error:

==> Error: ProcessError: Command exited with status 2:
    'make' '-j16'

6 errors found in build log:
     957    /apps/cmake/3.31.6/bin/cmake -E cmake_copy_f90_mod mod/phillips_initialization.mod CMakeFiles/mom6lib.dir/phillips_initialization.mod.stamp IntelLLVM
     958    /apps/cmake/3.31.6/bin/cmake -E touch CMakeFiles/mom6lib.dir/src/user/Phillips_initialization.F90.o.provides.build
     959    /apps/cmake/3.31.6/bin/cmake -E cmake_copy_f90_mod mod/lock_exchange_initialization.mod CMakeFiles/mom6lib.dir/lock_exchange_initialization.mod.stamp IntelLLVM
     960    /apps/cmake/3.31.6/bin/cmake -E touch CMakeFiles/mom6lib.dir/src/user/lock_exchange_initialization.F90.o.provides.build
     961    /apps/cmake/3.31.6/bin/cmake -E cmake_copy_f90_mod mod/mom_zanna_bolton.mod CMakeFiles/mom6lib.dir/mom_zanna_bolton.mod.stamp IntelLLVM
     962    /apps/cmake/3.31.6/bin/cmake -E touch CMakeFiles/mom6lib.dir/src/parameterizations/lateral/MOM_Zanna_Bolton.F90.o.provides.build
  >> 963    /g/data/e14/jb2381/MOM6-examples/src/MOM6-nm/src/diagnostics/MOM_diagnostics.F90(36): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.  
             [MOM_NUMERICAL_MIXING]
     964    use MOM_numerical_mixing,  only : numerical_mixing
     965    ----^
  >> 966    /g/data/e14/jb2381/MOM6-examples/src/MOM6-nm/src/diagnostics/MOM_diagnostics.F90(36): error #6580: Name in only-list does not exist or is not accessible.   [NUMERICA
            L_MIXING]
     967    use MOM_numerical_mixing,  only : numerical_mixing
     968    ----------------------------------^
  >> 969    /g/data/e14/jb2381/MOM6-examples/src/MOM6-nm/src/diagnostics/MOM_diagnostics.F90(1726): error #6406: Conflicting attributes or multiple declaration of name.   [NUMER
            ICAL_MIXING]
     970          call numerical_mixing(G, GV, Tr, h, diag_pre_dyn, dt_trans, Idt, uhtr, vhtr, scale_constant, x_upwind, y_upwind, nm)
     971    -----------^
     972    compilation aborted for /g/data/e14/jb2381/MOM6-examples/src/MOM6-nm/src/diagnostics/MOM_diagnostics.F90 (code 1)
  >> 973    make[2]: *** [CMakeFiles/mom6lib.dir/build.make:978: CMakeFiles/mom6lib.dir/src/diagnostics/MOM_diagnostics.F90.o] Error 1
     974    make[2]: *** Waiting for unfinished jobs....
     975    make[2]: Leaving directory '/scratch/e14/jb2381/tmp/spack-stage/spack-stage-access-mom6-2025.07.000-tv22kxhbvmstyal7ubocwvkousqtxlxl/spack-build-tv22kxh'
  >> 976    make[1]: *** [CMakeFiles/Makefile2:90: CMakeFiles/mom6lib.dir/all] Error 2
     977    make[1]: Leaving directory '/scratch/e14/jb2381/tmp/spack-stage/spack-stage-access-mom6-2025.07.000-tv22kxhbvmstyal7ubocwvkousqtxlxl/spack-build-tv22kxh'
  >> 978    make: *** [Makefile:139: all] Error 2

See build log for details:
  /scratch/e14/jb2381/tmp/spack-stage/spack-stage-access-mom6-2025.07.000-tv22kxhbvmstyal7ubocwvkousqtxlxl/spack-build-out.txt

which indicates it cannot find the new (compiled) module. Is there anything else I need to add to the build information/process, e.g. something to INCLUDE paths, when there is something new to be compiled and found? Any help or info would be great, thanks!

[minghangli-uni]

Hi @jbisits it seems like your new module was not compiled. Did you add this module to the CMakeLists.txt?

[jbisits]

Hi @jbisits it seems like your new module was not compiled. Did you add this module to the CMakeLists.txt?

I did not! I was not sure if that had to be done. I will try this now and let you know how it goes. Thanks!

[jbisits]

Thanks for that @minghangli-uni! After adding the module to CMakeLists.txt I get a different error:

  >> 481    /g/data/e14/jb2381/MOM6-examples/src/MOM6-nm/config_src/drivers/nuopc_cap/mom_cap_methods.F90(556): error #6404: This name does not have a type, and must have an e
            xplicit type.   [SET_MISSING_STKS_TO_ZERO]
     482              if(set_missing_stks_to_zero) then
     483    -------------^
  >> 484    /g/data/e14/jb2381/MOM6-examples/src/MOM6-nm/config_src/drivers/nuopc_cap/mom_cap_methods.F90(556): error #6341: A logical data type is required in this context.  
             [SET_MISSING_STKS_TO_ZERO]
     485              if(set_missing_stks_to_zero) then
     486    -------------^
     487    compilation aborted for /g/data/e14/jb2381/MOM6-examples/src/MOM6-nm/config_src/drivers/nuopc_cap/mom_cap_methods.F90 (code 1)
  >> 488    make[2]: *** [CMakeFiles/mom6lib.dir/build.make:4163: CMakeFiles/mom6lib.dir/config_src/drivers/nuopc_cap/mom_cap_methods.F90.o] Error 1
     489    make[2]: Leaving directory '/scratch/e14/jb2381/tmp/spack-stage/spack-stage-access-mom6-2025.07.000-tv22kxhbvmstyal7ubocwvkousqtxlxl/spack-build-tv22kxh'
  >> 490    make[1]: *** [CMakeFiles/Makefile2:90: CMakeFiles/mom6lib.dir/all] Error 2
     491    make[1]: Leaving directory '/scratch/e14/jb2381/tmp/spack-stage/spack-stage-access-mom6-2025.07.000-tv22kxhbvmstyal7ubocwvkousqtxlxl/spack-build-tv22kxh'
  >> 492    make: *** [Makefile:139: all] Error 2

do you know what this might be related to? Checking mom_cap_methods.F90 there is only one instance of set_missing_stks_to_zero so it does not appear to have a data type as the error indicates. Could this be something that was missed when I merged ACCESS-NRI/MOM6@2025.07 into my branch?

[minghangli-uni]

I can’t find set_missing_stks_to_zero anywhere in access-mom6, including some random older commits. It doesn’t appear to be part of the upstream code yet (I could be wrong). Or did you introduce this flag? if yes you’ll need to declare it explicitly with the correct data type.

[jbisits]

Ah ok. I did not introduce it so I am not sure where it came from though it can only have been introduced when I forked MOM6 or at some other merge commit along the way. Will look into it.

[jbisits]

Here is where the variable is defined in the upstream repo:

https://github.com/mom-ocean/MOM6/blob/d68ff1250e3fc8c75bf3bafc60b935fe5f7417a1/config_src/drivers/nuopc_cap/mom_cap_methods.F90#L76-L84

but it is not defined in the ACCESS-MOM6:

https://github.com/ACCESS-NRI/MOM6/blob/992deead7d959997856333717eea147efdc947b4/config_src/drivers/nuopc_cap/mom_cap_methods.F90#L84-L90

So when I made merge commit between my fork, which was based on upstream, and ACCESS-NRI/MOM6, something appears to have been mixed up.

[minghangli-uni]

Thanks @jbisits yeah this flag was added in upstream MOM6 in commit mom-ocean/MOM6@3970dbd which hasnt been merged to access-mom6 yet.

The clean fix at the moment is to rebase your work onto access-mom6 then rebuild it should work

[jbisits]

@chrisb13 this may be related to #155 and #156. Once I have rebased everything I will let you know.

[jbisits]

After rebasing to ACCESS-NRI/MOM6@2025.07 everything built, thanks!

One last question. To use the ACCESS-OM3 build I now have in release-MC_25km_jra_ryf (or any other config) do I edit the config.yaml to point to the new executable? So change

exe: access-om3-MOM6-CICE6

to

exe: my_exectuable

Thanks again for the help!

[minghangli-uni]

After rebasing to ACCESS-NRI/MOM6@2025.07 everything built, thanks!

Awesome!

One last question. To use the ACCESS-OM3 build I now have in release-MC_25km_jra_ryf (or any other config) do I edit the config.yaml to point to the new executable?

You dont need to hardcode the executable path. The ACCESS approach is to let spack generate modules and load them by adding a modules section - one example is shown below,

  view: true
  concretizer:
    unify: true
  modules:
    default:
      tcl:
        include:
        - access-om3
        - access3
        projections:
          access-om3: '{name}/2025.08.001-{hash:7}'
          access3: '{name}/2025.08.1-{hash:7}'
  develop:
...

Then

spack concretize -f --reuse-deps
spack install

NB: Everytime there is a change in the spack.yaml file, you'd have to concretize it

After installation, spack will create,

…/spack/0.22/release/modules/linux-rocky8-x86_64_v4/
    ├── access-om3/
    └── access3/

Each contains a versioned modulefile for the build.

In your payu config.yaml, replace the existing module setup with:

modules:
    use:
        - /path/to/spack/0.22/release/modules/linux-rocky8-x86_64_v4
    load:
        - access-om3/2025.08.001-oet4b2g
        - access3/2025.08.1-s7y62ay

Then now payu automatically loads your newly built binaries without editing paths manually.

[jbisits]

Ok, great! Glad I asked. It may be that I did not set up the development environment correctly because I could not find anything that looked like module files. Initially, after running develop, the development environment spack.yaml did not contain these lines:

  modules:
    default:
      tcl:
        include:
        - access-om3
        - access3
        projections:
          access-om3: '{name}/2025.08.001-{hash:7}'
          access3: '{name}/2025.08.1-{hash:7}'

I have now added them. Looking at the documentation these lines appear to be there after spack develop--should they have been added to mine in the setup process? I don't recall seeing them there, or editing them. No problem either way just trying to navigate my way through this process!

[minghangli-uni]

Sorry for the confusion, the modules section isn’t created by spack develop. It used to be present in our spack.yaml, but we recently removed it as part of the new infrastructure update. The modules are now injected at build time instead of being stored in the environment file. For test builds, I manually added the modules section back when building the code under Gadi account.

@CodeGat is there a cleaner or more standard way we should handle this?

[jbisits]

No worries, thanks for letting me know! I am getting closer -- now my error occurs after payu setup:

[jb2381@gadi-login-07 25km_jra_ryf_nm]$ payu setup
laboratory path:  /scratch/e14/jb2381/access-om3
binary path:  /scratch/e14/jb2381/access-om3/bin
input path:  /scratch/e14/jb2381/access-om3/input
work path:  /scratch/e14/jb2381/access-om3/work
archive path:  /scratch/e14/jb2381/access-om3/archive
Updated metadata. Experiment UUID: 4193b806-c195-4ddf-960f-752e438f6368
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up access-om3
Checking exe, input and restart manifests
Manifest path: stored hash != calculated hash
  work/access-om3-MOM6-CICE6: md5: 6caef5a9389009d920c5ff90f5b13867 != a0e923a448ee31a5aa46cb1e1fff832f
/g/data/vk83/apps/base_conda/envs/payu-1.2.0/lib/python3.10/site-packages/payu/metadata.py:189: MetadataWarning: No pre-existing archive found. Generating a new uuid
Loading access-om3/2025.08.001-dqecnaz
  Loading requirement: access3/2025.08.000-ibd3kli
Run cannot reproduce: manifest manifests/exe.yaml is not correct

The only things I have modified from the default experiment setup before payu setup are the diag_table and the config_yaml. The changes in the config.yaml are as suggested above (except the identifiers are different):

 16 #Model software version
 17 modules:
 18     use:
 19        - /g/data/e14/jb2381/spack/0.22/release/modules/linux-rocky8-x86_64_v4 # my build with numerical mixing
 20     load:
 21         - access-om3/2025.08.001-dqecnaz
 22         - access3/2025.08.000-ibd3kli::q

And the manifest/exe.yaml is:

 1 format: yamanifest
  2 version: 1.0
  3 ---
  4 work/access-om3-MOM6-CICE6:
  5   fullpath: /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/access3-2025.03.1-3joxy64tlxmkt66pxhpw6gydxfnd7fvl/bin/access-om3-MOM6-CICE6
  6   hashes:
  7     binhash: 37359bf2f545a1dabd018143cbcde32a
  8     md5: 6caef5a9389009d920c5ff90f5b13867

Is the manifest/exe.yaml not correct because of the different hashes as indicated in the error?