claireyung:dev-MC_4km_jra_ryf+regionalpanan+isf. PR #2

[chrisb13]

A new PR for the alpha release of the regional panan with ice-shelves. This PR allows us to add @claireyung's commits on top of the latest dev-MC_25km_jra_ryf.

ADDED BY DOUGIE: This PR now includes the same commits as #814 but rebased onto an updated dev-MC_4km_jra_ryf+regionalpanan+isf branch

We plan to do a squash merge.

Related:

• claireyung:dev-MC_4km_jra_ryf+regionalpanan+isf+draft #814
• Cleaning up panan PRs and coordinating changes for alpha release of regionalpanan (MC_4km_jra_ryf) configurations #880

[chrisb13]

@dougiesquire, I think I've now done what we discussed today. This is now for the ice-shelf version. Can you have a go at resolving the conflicts please?

If needed, I imagine we can ask for Claire's help resolving any trickier bits.

[dougiesquire]

As for the non-isf configuration, I've generated repro checksums using Claire's original branch and pushed them to her branch for reference.

I'll use these to make sure/document that I don't changes answers while doing the conflict resolution.

[dougiesquire]

The 880-dev-MC_4km_jra_ryf+regionalpanan+isf_temp branch Chris set up for this PR still included the final merge commit from Claire's original branch so I've remove that (no answer-changing changes)

[dougiesquire]

!test repro commit

[github-actions]

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21696243030 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

• 880-dev-MC_4km_jra_ryf+regionalpanan+isf_temp (checksums created using commit 7ce6a41), against
• dev-MC_4km_jra_ryf+regionalpanan+isf (checksums in commit d60cb09)

Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-om3-configs/7ce6a41f20da484079d0910e3599c70567ffce2b, and the test results at https://github.com/ACCESS-NRI/access-om3-configs/runs/62567050147.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/21696243030/artifacts/5384079720.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-om3-configs/tree/d60cb09f95af27f0512ef6c6575ca8047be37b48/testing/checksum

Test summary:
🔥 test_repro_historical

[dougiesquire]

Some of Claire's inputs are in a project that tm70_ci does not have access to, so I'll run the repro test locally and commit the new checksums.

Note, we'll update this config shortly to use the copies of Claire's files in /g/data/vk83 so this is just for bookkeeping.

[dougiesquire]

The checksums I just committed match those I created using Claire's original branch.

UPDATE: this has now been squashed into 8c49aa5, along with the answer-changing changes that to the base branch that were reverted for testing.

[dougiesquire]

Here's the difference between Claire's original branch and this one.

Here's the difference between the dev-MC_4km_jra_ryf+regionalpanan branch and this one.

[dougiesquire]

@chrisb13, this is ready for review. Could you please take a look? I'll do a merge commit for this one, since it required additional changes requested by Claire

I'll update input locations, add MPI flags, update SW pen scheme etc in new PRs.

[chrisb13]

@chrisb13, this is ready for review.

Thanks @dougiesquire, sorry for my delay. Is it now in the state that you'd like me to review? I ask as it seems there's been two force pushes after your comment so just double checking.

[dougiesquire]

@chrisb13 yup - ready for review

[helenmacdonald]

@chrisb13 and I are reviewing now - looks good, thanks!
A couple of questions:

1. Looks like DT_THERM was removed from MOM_input. This means that DT_THERM will be set to DT which is 150 - which would slow the model down a bit. Is this by design @claireyung
2. the non ice shelf version is running off om3 2025.08.001 whereas the isf version is using prerelease pr142-36 is there a plan to use a released version for the isf one?

To note:

1. We need to update file paths in config.yaml to point to files under the prerelease folder
2. There are some diagnostics added in the ice-shelf configuration that could be worth adding to the non ice-shelf configuration
3. Non ice-shelf config uses automask table and ice-shelf explicitly defines the layout. We could consider doing the explicit layout for the non ice shelf config

[claireyung]

Hi @helenmacdonald

Yes, in my experience the thermodynamic timestep needs to be the same as DT or not much bigger in the ice shelf config, otherwise it crashes fairly often (more details here claireyung/mom6-panAn-iceshelf-tools#49 ). However, note that DT=150 is only necessary for the first month with the unstable initialisation of ice shelves. After the first month, I recommend swapping to DT=400 and DT_THERM = 400 and coupling timestep 400 which speed things up a bit and is pretty stable. This is in the running instructions - https://access-om3-configs.access-hive.org.au/pr-previews/573/configurations/pan-Antartic/run_panan_isf/ (maybe the link to the running instructions can be made clearer somehow? It is not the only change that is needed after the first month!)

With the executable - to run, the ice shelf version requires changes to MOM6 that aren't yet in the main ACCESS-NRI MOM6 branch. Hopefully they make it there at some point, but some of the changes are a bit hacky at the moment and might need some improvement. Some comments here ACCESS-NRI/MOM6#38

[helenmacdonald]

Thanks for your comprehensive answers @claireyung!

With the executable - to run, the ice shelf version requires changes to MOM6 that aren't yet in the main ACCESS-NRI MOM6 branch. Hopefully they make it there at some point, but some of the changes are a bit hacky at the moment and might need some improvement.

Good to know, thanks. As an aside, my understanding is that these changes still need some cleaning up before being merged and eventually ending up in a released executable. Do we have a plan for this? Is it you, or someone at ACCESS-NRI responsible for this? Similarly, has someone taken on the responsibity for pushing some of these back to MOM central (not saying you have to, just wanting to make sure it is clear who should be doing it!)?

@chrisb13 - I think we are ready to merge? We will need to remember to update the executable at a later date.

[chrisb13]

Thanks @claireyung and @helenmacdonald

note that DT=150 is only necessary for the first month with the unstable initialisation of ice shelves. After the first month, I recommend swapping to DT=400 and DT_THERM = 400 and coupling timestep 400 which speed things up a bit and is pretty stable

My memory from chats with @dougiesquire is that our plan is to release the panan+noisf from a cold start and this config from a warm start (given it's expense and the challneges of initilisation). Is my memory correct @dougiesquire?

If that's the case, then I think we should move over all the defaults to DT=400 etc.

It is not the only change that is needed after the first month!

Ok, so looking at the docs, I see changing the restart and this:

Run - ensure DT is 400 (ish), input.nml has input_filename = 'n' commented out, CLOCK time in nuopc.runconfig is months

Is there anything else @claireyung?

maybe the link to the running instructions can be made clearer somehow
When we do an alpha release we can include in the announcement that highlights the doc's if useful? (We don't typically do a "formal" announcement for alphas but we could do a forum post / put something in a github release note).

As an aside, my understanding is that these changes still need some cleaning up before being merged and eventually ending up in a released executable. Do we have a plan for this? Is it you, or someone at ACCESS-NRI responsible for this? Similarly, has someone taken on the responsibity for pushing some of these back to MOM central

I think @dougiesquire is aware of this. We've been chatting about which changes may eventually go upstream too (Angus is likely putting a MOM-ocean PR together -- so there is likely an upcoming opportunity fyi @claireyung). Technically, I think we can do an alpha release off a pre-release but I leave to @dougiesquire and @anton-seaice as to whether they'd prefer it based off an actual build release..? I note that it's not currently on @dougiesquire alpha to do list.

@chrisb13 - I think we are ready to merge? We will need to remember to update the executable at a later date.

Would be helpful to get clarity on the "to note" points we raised here first. Or at least add them to the mega-list.

[helenmacdonald]

Would be helpful to get clarity on the "to note" points we raised here first. Or at least add them to the mega-list.

I have added them to the mega-list

[dougiesquire]

My memory from chats with @dougiesquire is that our plan is to release the panan+noisf from a cold start and this config from a warm start (given it's expense and the challneges of initilisation). Is my memory correct @dougiesquire?

Nope. @claireyung's original branch did start from a restart so that is what I originally squashed/rebased to make sure that I could reproduce her answers. But, @claireyung requested on Zulip that the configuration be modified to start cold using her instructions here. I made this change in this commit.

1. We need to update file paths in config.yaml to point to files under the prerelease folder

Yes, that will be done in a follow-up PR, along with a few other changes. E.g. see the follow up PR for the non-isf configuration here: #1118

2. There are some diagnostics added in the ice-shelf configuration that could be worth adding to the non ice-shelf configuration

Other that changing a few variables to use their CMIP names, I just copied what Claire had in her diag_table as she knows the configuration best. In a general sense, different configurations will call for different diagnostics (e.g. the ice shelf diagnostics are obviously only relevant in the ice shelf configuration). If you think there's a need to change the diagnostics, this should be done in another PR with an associated issue that describes the reason for the change.

3. Non ice-shelf config uses automask table and ice-shelf explicitly defines the layout. We could consider doing the explicit layout for the non ice shelf config

Maybe. But not urgent, and again not in this PR. Please feel free to open an issue.

Good to know, thanks. As an aside, my understanding is that these changes still need some cleaning up before being merged and eventually ending up in a released executable. Do we have a plan for this? Is it you, or someone at ACCESS-NRI responsible for this? Similarly, has someone taken on the responsibity for pushing some of these back to MOM central (not saying you have to, just wanting to make sure it is clear who should be doing it!)?

Yes, I am aware of this as @chrisb13 says and it's on my todo list to get these changes into one of our release branches and eventually upstream. But, as @claireyung mentions, there's a little bit of work to do before they are ready. Alpha releases can use prerelease builds so I don't think this needs to hold us up.

[dougiesquire]

Hmmm, actually after the cold start and parameter updates requested by @claireyung, this now fails after a few hours with:

WARNING from PE    76: MOM_diabatic_aux.F90, applyBoundaryFluxesInOut(): Mass created. x,y,dh=      -4.938E+01     -8.196E+01      6.006E-12

WARNING from PE    76: MOM_diabatic_aux.F90, applyBoundaryFluxesInOut(): Mass created. x,y,dh=      -4.929E+01     -8.196E+01      4.487E-08

[Lots more of ^these]

FATAL from PE   491:  Could not find target coordinate   1036.52521775451      in get_polynomial_coordinate. This is caused by an inconsistent interpolant (perhaps not monotonically increasing):   1036.52521775451        1036.31691947091                          NaN

Image              PC                Routine            Line        Source
access-om3-MOM6-C  00000000024E4472  mpp_error_basic            80  mpp_util_mpi.inc
access-om3-MOM6-C  0000000001409A58  build_and_interpo         453  regrid_interp.F90
access-om3-MOM6-C  0000000001404CC5  build_rho_column          140  coord_rho.F90
access-om3-MOM6-C  00000000013F4976  diag_remap_update         361  MOM_diag_remap.F90
access-om3-MOM6-C  00000000013E758D  diag_update_remap        3632  MOM_diag_mediator.F90
access-om3-MOM6-C  00000000018816A6  diabatic_ale             1605  MOM_diabatic_driver.F90
access-om3-MOM6-C  000000000187412B  diabatic                  413  MOM_diabatic_driver.F90
access-om3-MOM6-C  00000000017C84E1  step_mom_thermo          1674  MOM.F90
access-om3-MOM6-C  00000000017A894F  step_mom                 1007  MOM.F90
access-om3-MOM6-C  00000000012C182C  update_ocean_mode         642  mom_ocean_model_nuopc.F90
access-om3-MOM6-C  00000000011EDD87  modeladvance             1979  mom_cap.F90
libesmf.so         00001479DD9C8417  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         00001479DD9C82B5  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         00001479DDB767FB  esmf_attachmethod     Unknown  Unknown
libesmf.so         00001479DE2DC9EE  nuopc_modelbase_m     Unknown  Unknown
libesmf.so         00001479DD7205A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         00001479DD71FFEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         00001479DDA7D584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         00001479DD720C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         00001479DDC7F2B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         00001479DDEFC409  esmf_gridcompmod_     Unknown  Unknown
libesmf.so         00001479DE2966A5  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00001479DE2982C8  nuopc_driver_mp_e     Unknown  Unknown
libesmf.so         00001479DD9C8417  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         00001479DD9C82B5  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         00001479DDB767FB  esmf_attachmethod     Unknown  Unknown
libesmf.so         00001479DE2951EB  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00001479DD7205A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         00001479DD71FFEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         00001479DDA7D584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         00001479DD720C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         00001479DDC7F2B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         00001479DDEFC409  esmf_gridcompmod_     Unknown  Unknown
libesmf.so         00001479DE2966A5  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00001479DE2982C8  nuopc_driver_mp_e     Unknown  Unknown
libesmf.so         00001479DD9C8417  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         00001479DD9C82B5  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         00001479DDB767FB  esmf_attachmethod     Unknown  Unknown
libesmf.so         00001479DE2951EB  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00001479DD7205A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         00001479DD71FFEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         00001479DDA7D584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         00001479DD720C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         00001479DDC7F2B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         00001479DDEFC409  esmf_gridcompmod_     Unknown  Unknown
access-om3-MOM6-C  00000000011B3794  esmapp                    141  esmApp.F90
access-om3-MOM6-C  0000000000A6874D  Unknown               Unknown  Unknown
libc-2.28.so       00001479D9935865  __libc_start_main     Unknown  Unknown
access-om3-MOM6-C  0000000000A6866E  Unknown               Unknown  Unknown
--------------------------------------------------------------------------

I didn't notice this previously because it does run long enough to update the checksums.

Turning off the diagnostics on rho2 coords gets us past this error, but then it fails a bit later with:

WARNING from PE   491: MOM_tracer_diabatic.F90, applyTracerBoundaryFluxesInOut(): Tracer created. x,y,dh=      -8.062E+01     -7.271E+01            NaN

FATAL from PE   491: NaN in input field of reproducing_EFP_sum(_2d).

Image              PC                Routine            Line        Source
access-om3-MOM6-C  00000000024E4472  mpp_error_basic            80  mpp_util_mpi.inc
access-om3-MOM6-C  00000000013AD342  reproducing_efp_s         210  MOM_coms.F90
access-om3-MOM6-C  00000000013AE1A1  reproducing_sum_2         308  MOM_coms.F90
access-om3-MOM6-C  0000000001491EEC  global_volume_mea         378  MOM_spatial_means.F90
access-om3-MOM6-C  0000000001EE83C5  calculate_diagnos         462  MOM_diagnostics.F90
access-om3-MOM6-C  00000000017AD3A4  step_mom                 1052  MOM.F90
access-om3-MOM6-C  00000000012C182C  update_ocean_mode         642  mom_ocean_model_nuopc.F90
access-om3-MOM6-C  00000000011EDD87  modeladvance             1979  mom_cap.F90
libesmf.so         000014F220CAF417  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         000014F220CAF2B5  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         000014F220E5D7FB  esmf_attachmethod     Unknown  Unknown
libesmf.so         000014F2215C39EE  nuopc_modelbase_m     Unknown  Unknown
libesmf.so         000014F220A075A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         000014F220A06FEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         000014F220D64584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         000014F220A07C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         000014F220F662B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         000014F2211E3409  esmf_gridcompmod_     Unknown  Unknown
libesmf.so         000014F22157D6A5  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         000014F22157F2C8  nuopc_driver_mp_e     Unknown  Unknown
libesmf.so         000014F220CAF417  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         000014F220CAF2B5  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         000014F220E5D7FB  esmf_attachmethod     Unknown  Unknown
libesmf.so         000014F22157C1EB  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         000014F220A075A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         000014F220A06FEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         000014F220D64584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         000014F220A07C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         000014F220F662B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         000014F2211E3409  esmf_gridcompmod_     Unknown  Unknown
libesmf.so         000014F22157D6A5  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         000014F22157F2C8  nuopc_driver_mp_e     Unknown  Unknown
libesmf.so         000014F220CAF417  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         000014F220CAF2B5  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         000014F220E5D7FB  esmf_attachmethod     Unknown  Unknown
libesmf.so         000014F22157C1EB  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         000014F220A075A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         000014F220A06FEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         000014F220D64584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         000014F220A07C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         000014F220F662B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         000014F2211E3409  esmf_gridcompmod_     Unknown  Unknown
access-om3-MOM6-C  00000000011B3794  esmapp                    141  esmApp.F90
access-om3-MOM6-C  0000000000A6874D  Unknown               Unknown  Unknown
libc-2.28.so       000014F21CC1C865  __libc_start_main     Unknown  Unknown
access-om3-MOM6-C  0000000000A6866E  Unknown               Unknown  Unknown
--------------------------------------------------------------------------

[dougiesquire]

> FATAL from PE   491: NaN in input field of reproducing_EFP_sum(_2d).

This error is introduced after updating to access-om3/pr142-36 (from access-om3/pr142-26). @claireyung have you see this before with these configs?

[dougiesquire]

The error occurs when trying to write global temperature diagnostics (thetaogm and tosga are configured). If I turn these off, I get:

WARNING from PE   491: MOM_tracer_diabatic.F90, applyTracerBoundaryFluxesInOut(): Tracer created. x,y,dh=      -8.062E+01     -7.271E+01            NaN

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 67 in communicator MPI COMMUNICATOR 3 CREATE FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
libpthread-2.28.s  00001499F0B09990  Unknown               Unknown  Unknown
libuct_ib.so.0.0.  00001499D217C701  Unknown               Unknown  Unknown
libucp.so.0.0.0    00001499EC3EF3CA  ucp_worker_progre     Unknown  Unknown
libopen-pal.so.40  00001499EC877923  opal_progress         Unknown  Unknown
libopen-pal.so.40  00001499EC877AD5  ompi_sync_wait_mt     Unknown  Unknown
libmpi.so.40.30.7  00001499F1AE1C79  ompi_request_defa     Unknown  Unknown
libmpi.so.40.30.7  00001499F1A958F2  MPI_Wait              Unknown  Unknown
libesmf.so         00001499F48BA01D  _ZN5ESMCI3VMK8com     Unknown  Unknown
libesmf.so         00001499F4407888  _ZN5ESMCI3XXE4exe     Unknown  Unknown
libesmf.so         00001499F4406FE2  _ZN5ESMCI3XXE4exe     Unknown  Unknown
libesmf.so         00001499F4406FE2  _ZN5ESMCI3XXE4exe     Unknown  Unknown
libesmf.so         00001499F43A1DB2  _ZN5ESMCI11ArrayB     Unknown  Unknown
libesmf.so         00001499F43A97BA  c_esmc_arraybundl     Unknown  Unknown
libesmf.so         00001499F493B56A  esmf_arraybundlem     Unknown  Unknown
libesmf.so         00001499F4B23319  esmf_fieldbundlem     Unknown  Unknown
libesmf.so         00001499F5087824  nuopc_connector_m     Unknown  Unknown
libesmf.so         00001499F45465A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         00001499F4545FEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         00001499F48A3584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         00001499F4546C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         00001499F4AA52B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         00001499F4ACA6E9  esmf_cplcompmod_m     Unknown  Unknown
libesmf.so         00001499F50BCEDC  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00001499F50BDFFE  nuopc_driver_mp_e     Unknown  Unknown
libesmf.so         00001499F47EE417  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         00001499F47EE2B5  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         00001499F499C7FB  esmf_attachmethod     Unknown  Unknown
libesmf.so         00001499F50BB1EB  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00001499F45465A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         00001499F4545FEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         00001499F48A3584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         00001499F4546C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         00001499F4AA52B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         00001499F4D22409  esmf_gridcompmod_     Unknown  Unknown
libesmf.so         00001499F50BC6A5  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00001499F50BE2C8  nuopc_driver_mp_e     Unknown  Unknown
libesmf.so         00001499F47EE417  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         00001499F47EE2B5  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         00001499F499C7FB  esmf_attachmethod     Unknown  Unknown
libesmf.so         00001499F50BB1EB  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00001499F45465A8  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         00001499F4545FEC  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         00001499F48A3584  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         00001499F4546C80  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         00001499F4AA52B8  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         00001499F4D22409  esmf_gridcompmod_     Unknown  Unknown
access-om3-MOM6-C  00000000011B3794  esmapp                    141  esmApp.F90
access-om3-MOM6-C  0000000000A6874D  Unknown               Unknown  Unknown
libc-2.28.so       00001499F075B865  __libc_start_main     Unknown  Unknown
access-om3-MOM6-C  0000000000A6866E  Unknown               Unknown  Unknown

which is probably just the same NaN issue manifesting in a different place.

[claireyung]

Hi @dougiesquire yeah I get these...
Usually making DT/DT_THERM smaller helps. But if DT is already 150, making it smaller means it probably won't finish in the normalsr walltime limit for this number of cores.

You could try ICE_SHELF_USTAR_FROM_VEL_BUGFIX = False. For some reason, since I made this True it seems a lot more sensitive than it was before (in my current run I get very big error logs with all those warnings, and sometimes it eventually crashes and I bring the timestep down but sometimes it just keeps running).

Alternatively I believe GFDL people recommend RESCALE_STRONG_DRAG = True and BT_STRONG_DRAG = True NOAA-GFDL/MOM6#971 (comment) which may be relevant. (I haven't tested it yet in this config)

[dougiesquire]

Thanks @claireyung for the suggestions. That issue looks very relevant.

Alternatively I believe GFDL people recommend RESCALE_STRONG_DRAG = True and BT_STRONG_DRAG = True NOAA-GFDL/MOM6#971 (comment) which may be relevant. (I haven't tested it yet in this config)

What extent of testing would justify adding these? If this configuration runs with those changes is that sufficient to include them? Are there some outputs/diagnostics we should be looking at (sorry, I know nothing about ice shelves)?

[claireyung]

I think if it runs, I'd be happy :)
Normally I'd look at maps of melt rate and a T-S plot to make sure the thermodynamics is right. If you have some output I'd be happy to make a evaluation notebook of that data with some plots and share it :)

[dougiesquire]

Thanks heaps @claireyung.

Setting ICE_SHELF_USTAR_FROM_VEL_BUGFIX = False doesn't help the issue unfortunately.

Alternatively I believe GFDL people recommend RESCALE_STRONG_DRAG = True and BT_STRONG_DRAG = True NOAA-GFDL/MOM6#971 (comment) which may be relevant. (I haven't tested it yet in this config)

For testing this, I've set up a new branch in the ACCESS-NRI MOM6 fork that is based on the latest NOAA-GFDL/MOM6:dev/gfdl and includes our ACCESS changes and your ice shelf changes. The branch is called dev/gfdl+access+isf and it is currently being deployed in an ACCESS-OM3 prerelease using this PR.

Just letting you know as that branch and prerelease could potentially replace ACCESS-NRI/MOM6#14 and ACCESS-NRI/ACCESS-OM3#142 if they suit your needs.

[dougiesquire]

I finally managed to get an ACCESS-OM3 prerelease deployed using the dev/gfdl+access+isf branch. Unfortunately, even with BT_STRONG_DRAG = True and RESCALE_STRONG_DRAG = True, this still crashes within the first few hours with the same error:

FATAL from PE   801: NaN in input field of reproducing_EFP_sum(_2d).

Image              PC                Routine            Line        Source
access-om3-MOM6-C  000000000256E2B2  mpp_error_basic            80  mpp_util_mpi.inc
access-om3-MOM6-C  000000000179E652  reproducing_efp_s         211  MOM_coms.F90
access-om3-MOM6-C  000000000179F4B1  reproducing_sum_2         309  MOM_coms.F90
access-om3-MOM6-C  000000000172060C  global_volume_mea         378  MOM_spatial_means.F90
access-om3-MOM6-C  00000000016E34C9  calculate_diagnos         466  MOM_diagnostics.F90
access-om3-MOM6-C  00000000012AAFA3  step_mom                 1045  MOM.F90
access-om3-MOM6-C  000000000126B2AC  update_ocean_mode         639  mom_ocean_model_nuopc.F90
access-om3-MOM6-C  000000000121B127  modeladvance             1990  mom_cap.F90
libesmf.so         0000151A97F35363  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         0000151A97F34FFE  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         0000151A98110BEB  esmf_attachmethod     Unknown  Unknown
libesmf.so         0000151A9889D964  nuopc_modelbase_m     Unknown  Unknown
libesmf.so         0000151A97C34E0E  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         0000151A97C34C37  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         0000151A98003B5F  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         0000151A97C35ACD  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         0000151A98218C75  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         0000151A98495659  esmf_gridcompmod_     Unknown  Unknown
libesmf.so         0000151A98856A35  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         0000151A98858658  nuopc_driver_mp_e     Unknown  Unknown
libesmf.so         0000151A97F35363  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         0000151A97F34FFE  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         0000151A98110BEB  esmf_attachmethod     Unknown  Unknown
libesmf.so         0000151A98855311  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         0000151A97C34E0E  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         0000151A97C34C37  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         0000151A98003B5F  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         0000151A97C35ACD  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         0000151A98218C75  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         0000151A98495659  esmf_gridcompmod_     Unknown  Unknown
libesmf.so         0000151A98856A35  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         0000151A98858658  nuopc_driver_mp_e     Unknown  Unknown
libesmf.so         0000151A97F35363  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         0000151A97F34FFE  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         0000151A98110BEB  esmf_attachmethod     Unknown  Unknown
libesmf.so         0000151A98855311  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         0000151A97C34E0E  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         0000151A97C34C37  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         0000151A98003B5F  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         0000151A97C35ACD  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         0000151A98218C75  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         0000151A98495659  esmf_gridcompmod_     Unknown  Unknown
access-om3-MOM6-C  00000000011E0994  esmapp                    141  esmApp.F90
access-om3-MOM6-C  0000000000A9360D  Unknown               Unknown  Unknown
libc-2.28.so       0000151A93E24865  __libc_start_main     Unknown  Unknown
access-om3-MOM6-C  0000000000A9352E  Unknown               Unknown  Unknown
--------------------------------------------------------------------------

[claireyung]

Bummer. I'm sorry this has been so painful @dougiesquire

It's weird because it worked fine with the very similar IAF config. My records of the first month in that IAF config say DT=150 worked. Some options are: try DT=120 (I've done this last year with an RYF when it crashed with dt=150, which i'm pretty sure was the same layout, and it just finished within 10 hours), or we could try dropping the timesteps even smaller and run the first month in two segments (which is annoying for diagnostics, but at least may not crash)

Did I muck up something with files - I feel like I saw a notification where you said one of the hashes had changed, but I can't find it now. (Possibly I was hallucinating)

Also just clarifying, do you mean it crashes after a few hours of model time or real time?

[dougiesquire]

Bummer. I'm sorry this has been so painful @dougiesquire

No stress at all :)

Some options are: try DT=120 (I've done this last year with an RYF when it crashed with dt=150, which i'm pretty sure was the same layout, and it just finished within 10 hours), or we could try dropping the timesteps even smaller and run the first month in two segments (which is annoying for diagnostics, but at least may not crash)

I've tried reducing the time step (to both 120s and 90s) without success unfortunately.

Also just clarifying, do you mean it crashes after a few hours of model time or real time?

Model time. Using the dev/gfdl+access+isf branch (access-om3/pr189-6) with BT_STRONG_DRAG = True and RESCALE_STRONG_DRAG = True, the model crashes on the 87th time step (i.e. ~3.5 hours).

Did I muck up something with files - I feel like I saw a notification where you said one of the hashes had changed, but I can't find it now. (Possibly I was hallucinating)

I'm still using your original files in this config (i.e. not the ones on vk83). But comparing to the runlog commit you provided for your IAF run (thanks!), that is one of the main differences. Possibly by using your original branch, I'm using an out-of-date input? I'll try updating that now.

[dougiesquire]

The crash still occurs using the input files on vk83 (answers don't change at all). Using access-om3/pr142-36, the model still crashes on the 225th time step (i.e. ~9.5 hours).

There's now very little difference between this configuration and the IAF one @claireyung successfully ran. The main remaining differences are:

• RYF vs IAF
• This configuration uses /g/data/vk83/prerelease/configurations/inputs/access-om3/panan.4km/2026.01.08/OBC37S_forcing_access_yr2_4km.nc for OBC forcing, whereas the IAF config uses /g/data/x77/cy8964/mom6/input/input-8km/ryf_gregorian_1980-1984_forcing_access_yr2_8km_fill.nc.

I've checked that the additional MOM parameter changes that we're brought in by the rebase are not responsible for the crash.

[claireyung]

Hmm. That is quite early to crash. Thanks for the comparison.

As for the boundary forcing file, I just copied the RYF version to make the gregorian IAF one and changed the dates (interpolating for Feb 29 if a leap year). So shouldn't have caused the difference, and it looks like the errors were coming from ice-shelfy latitudes not the northern boundary?

Just confirming, by checking it's not parameter changes brought in the rebase, you mean that the changes to USE_RIVER_HEAT_CONTENT , USE_CALVING_HEAT_CONTENT, MAX_TR_DIFFUSION_CFL and ENTHALPY_FROM_COUPLER, VELOCITY_TOLERANCE, LOTW_BBL_ANSWER_DATE are definitely not the problem?

[dougiesquire]

Just confirming, by checking it's not parameter changes brought in the rebase, you mean that the changes to USE_RIVER_HEAT_CONTENT , USE_CALVING_HEAT_CONTENT, MAX_TR_DIFFUSION_CFL and ENTHALPY_FROM_COUPLER, VELOCITY_TOLERANCE, LOTW_BBL_ANSWER_DATE are definitely not the problem?

Yup. With access-om3/pr142-36 if I revert all these parameters (and CORIOLIS_SCHEME) to the values you have in the IAF config it fails on the 215th time step with:

MOM_forcing_type, forcing_SinglePointPrint: Called from applyBoundaryFluxesInOut (grounding)
MOM_forcing_type, forcing_SinglePointPrint: lon,lat =      -4.646E+01     -8.196E+01
MOM_forcing_type, forcing_SinglePointPrint: ustar =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: tau_mag =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: buoy is not associated.
MOM_forcing_type, forcing_SinglePointPrint: sw =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: sw_vis_dir =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: sw_vis_dif =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: sw_nir_dir =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: sw_nir_dif =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: lw =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: latent =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: latent_evap_diag =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: latent_fprec_diag =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: latent_frunoff_diag =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: latent_frunoff_glc_diag =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: sens =      -3.176E+01
MOM_forcing_type, forcing_SinglePointPrint: evap =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: lprec =       8.568E-05
MOM_forcing_type, forcing_SinglePointPrint: fprec =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: vprec =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: seaice_melt =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: seaice_melt_heat =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: p_surf =       9.180E+06
MOM_forcing_type, forcing_SinglePointPrint: salt_flux =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: BBL_tidal_dis =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: ustar_tidal =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: lrunoff =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: lrunoff_glc =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: frunoff =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: frunoff_glc =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_lrunoff =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_lrunoff_glc =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_frunoff =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_frunoff_glc =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_lprec =      -7.111E-01
MOM_forcing_type, forcing_SinglePointPrint: heat_content_fprec =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_vprec =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_cond =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_massout =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_evap is not associated.
MOM_forcing_type, forcing_SinglePointPrint: heat_content_massout =       0.000E+00
MOM_forcing_type, forcing_SinglePointPrint: heat_content_massin =      -7.056E-01

WARNING from PE    77: MOM_diabatic_aux.F90, applyBoundaryFluxesInOut(): Mass created. x,y,dh=      -4.646E+01     -8.196E+01      1.430E-11


WARNING from PE   491: MOM_tracer_diabatic.F90, applyTracerBoundaryFluxesInOut(): Tracer created. x,y,dh=      -8.062E+01     -7.271E+01            NaN


FATAL from PE   491: NaN in input field of reproducing_EFP_sum(_2d).

[dougiesquire]

I was able to run @claireyung's IAF config without issue (for one day). But, if I update only the datm.streams.xml and drof.streams.xml files to use JRA v1.6 RYF forcing, the model crashes with NaN in input field of reproducing_EFP_sum(_2d) after 262 time steps.

With JRA v1.4 RYF forcing I can run for at least 1 day. I'm trying the full month at the moment.