Skip to content

Conversation

@blimlim
Copy link
Collaborator

@blimlim blimlim commented Feb 5, 2026

1. Summary:

This PR updates to the latest 2026.01.000 deployment. This uses the ACCESS-NRI fork of oasis-mct, and includes a CICE update which fixes a small problem in the flux diagnostics.

Upgrading to Oasis3-mct v5.2 requires changes to the coupler's namcouple file. These have been cherry-picked from #298.

2. Issues Addressed:

3. Dependencies (e.g. on payu, or model)

This change requires changes to (note pull request(s) where relevant):

  • workflow manager (payu):
  • model deployment (ACCESS-ESM1.6):
  • model component or library dependency:
  • input workflow:

4. Ad-hoc Testing

What ad-hoc testing was done? How are you convinced this change is correct (plots are good)?

5. CI Testing

  • !test repro has been run

6. Reproducibility

Is this reproducible with the previous commit? (If not, why not?)

  • Yes
  • No - !test repro commit has been run.

7. Performance

Has the model performance (say, throughput of model-years/wall-day) changed?

  • Yes
  • No
  • N/A (if selected, please add a brief explanation why performance testing is not necessary for this PR)

If yes, provide the numbers from your testing. Is the performance better or worse?

8. Manifests

Have you changed the executable, the input files and/or the restart files?

  • Yes
  • No

If yes, have you updated the manifests?

  • Yes
  • No

To update the manifests, run payu setup (in a cloned copy of your feature branch) with reproducibility tests turned off:

manifest:
  reproduce:
    exe: false
    input: false
    restart: false
runlog:
  enable: false

Then commit the newly created manifest files (under manifests/) only to the branch for this PR.

9. Documentation

Is the documentation updated?

  • Yes
  • N/A

10. Merge Strategy

  • Merge commit
  • Rebase and merge
  • Squash

@blimlim blimlim force-pushed the 402-access-oasis-mct branch from 79f2068 to 9621440 Compare February 5, 2026 04:14
@blimlim
Copy link
Collaborator Author

blimlim commented Feb 5, 2026

!test repro

@github-actions
Copy link

github-actions bot commented Feb 5, 2026

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

  • 402-access-oasis-mct (checksums created using commit c1c5107), against
  • dev-preindustrial+concentrations (checksums in commit 13cc7d2)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/c1c510795530efe3d4c2b1691dde14880274ec57, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62574521211.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21698578591/artifacts/5384947750.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/13cc7d229b0d4bda193879b8b30cde3441d61bec/testing/checksum

Test summary:
test_repro_historical
test_repro_determinism

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 5, 2026

!test repro

@github-actions
Copy link

github-actions bot commented Feb 5, 2026

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21698778734 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-access-oasis-mct (checksums created using commit 16ac67b), against
  • dev-preindustrial+concentrations (checksums in commit 13cc7d2)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/16ac67bbe615f15c8ebcf68b59fba608e761d406, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62575017309.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21698778734/artifacts/5385008288.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/13cc7d229b0d4bda193879b8b30cde3441d61bec/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 5, 2026

@aidanheerdegen, @manodeep, @MartinDix

I've cherry picked the changes to the namcouple for the new version of oasis from @manodeep's branch final-opt-devPI. These conflicted with @MartinDix's earlier changes to reduce the amount of oasis logging in #314. The configuration runs ok, however does produce the extra output described in #314:

debug.root.03:

(oasis_advance_avdiag) CHECK* diags, fld= icethk03 :masked:area weighted
  minimum value   =    0.000000000000000     icethk03
     min val at   = (   192,    37)    13152 icethk03
  maximum value   =    1.453360489455059     icethk03
     max val at   = (    84,     9)     2964 icethk03
  unweighted mean =   0.5661523759557128E-01 icethk03
  weighted mean   =   0.1548162290514272E-01 icethk03
  unweighted sum  =    3950.724509894155     icethk03
  weighted sum    =    5594102371375.671     icethk03

I've tried to reapply these logging changes in 16ac67b, however it's resulting in a crash from oasis:

work/atmosphere/nout.000000

 (mod_oasis_namcouple:inipar_alloc)2 Read line: 
 192 145 360 300 um1t cice LAG=+1800 SEQ=+1
 (mod_oasis_namcouple:inipar_alloc)3 Read line: P 0 P 0
 (mod_oasis_namcouple:inipar_alloc)4 Read line_aux: #
 (mod_oasis_namcouple:inipar_alloc)5 Read line: INSTANT
 (mod_oasis_namcouple:inipar_alloc)5 Read line: INT=0
 (mod_oasis_namcouple:inipar_alloc)1 Read line: 
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
 (mod_oasis_namcouple:inipar_alloc)parsing 1 Read line, clvari in 5 position: 
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST10
 (mod_oasis_namcouple:inipar_alloc)field jf :            2  lg_state(jf) :  F
 (mod_oasis_namcouple:inipar_alloc)field jf :            2  endflag :  F
 (mod_oasis_namcouple:inipar_alloc)6 Read line : INT=0
 (mod_oasis_namcouple:inipar_alloc)7 Read line: # -- 19:20:21:22:23
  
 (mod_oasis_namcouple:namcouple_abort) calling ABORT
  **** ABORT from (mod_oasis_namcouple:inipar_alloc) line number         1176
  **** (mod_oasis_namcouple:inipar_alloc) : 
  You want a transformation which is not available !
  **** (mod_oasis_namcouple:inipar_alloc) : 
  Only local transformations are available for
  **** (mod_oasis_namcouple:inipar_alloc) : 
  fields exchanged directly or output fields

I haven't figured out what's going on yet

@aidanheerdegen
Copy link
Member

It wasn't clear to me that @MartinDix's changes could be naively applied, as it looked like a significant change in format. @manodeep would know more as he made the changes.

@manodeep
Copy link
Collaborator

manodeep commented Feb 6, 2026

I am taking a look now

@blimlim Are you happy for me to push to this branch if I have any code/namelist updates?

@manodeep
Copy link
Collaborator

manodeep commented Feb 6, 2026

The namcouple is using the multi-field method exposed - I tested that and there wasn't any obvious performance gain relative to the usual way of specifying individual fields. Other than that, nothing looks out of the ordinary - Martin's changes were to remove the CHECKIN/CHECKOUT and reduce the corresponding 4 ops to 2 ops (or that's how I interpreted it)

The biggest change to the namcouple file for oasis3-mct v5.2, was having to specify the grid dimensions (on the second line) for each field exchanged.

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 8, 2026

@blimlim Are you happy for me to push to this branch if I have any code/namelist updates?

Yeah definitely happy for you to push to this branch! Had you worked out a final version of the namcouple file to use?

I'm just testing out if I can get @MartinDix's earlier changes to still work.

@manodeep
Copy link
Collaborator

manodeep commented Feb 9, 2026

!test repro

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21807989712 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-access-oasis-mct (checksums created using commit 103f471), against
  • dev-preindustrial+concentrations (checksums in commit 13cc7d2)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62914780990.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21807989712/artifacts/5425509307.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/13cc7d229b0d4bda193879b8b30cde3441d61bec/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

@manodeep
Copy link
Collaborator

manodeep commented Feb 9, 2026

Huh! Okay - then I am going to overwrite with the working version from here

@manodeep
Copy link
Collaborator

manodeep commented Feb 9, 2026

!test repro

@manodeep
Copy link
Collaborator

manodeep commented Feb 9, 2026

The previous namcouple still had a couple of places with the "4" and CHECKIN/CHECKOUT - maybe that was causing the bit repro test issue. The file now contains the raw contents from the previously working namcouple file (independent fields though, rather than multi-fields)

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

  • 402-access-oasis-mct (checksums created using commit 2ceb8f6), against
  • dev-preindustrial+concentrations (checksums in commit 13cc7d2)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/2ceb8f672eb17a67e46f3e921816d8431852fa08, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62915533993.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21808231047/artifacts/5425599145.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/13cc7d229b0d4bda193879b8b30cde3441d61bec/testing/checksum

Test summary:
test_repro_historical
test_repro_determinism

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

2 simultaneous month sanity check confirms no issues after restarting:

> nccmp -dgf base/archive/output001/atmosphere/netCDF/aiihca.pa-267602_mon.nc final-build-test/archive/output001/atmosphere/netCDF/aiihca.pa-267602_mon.nc 
>
>nccmp -dgf base/archive/output001/atmosphere/netCDF/aiihca.pe-267602_dai.nc final-build-test/archive/output001/atmosphere/netCDF/aiihca.pe-267602_dai.nc 
>

The extra diagnostic information mentioned here is also no longer included in the logs, as desired!

Copy link
Member

@aidanheerdegen aidanheerdegen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've not looked at the namcouple file, not my area of expertise, but assume @blimlim and @manodeep have that covered.

I have checked the component spack hashes against the model release DB and they match (should make that an automatic check ....).

@blimlim blimlim merged commit c7e3cd1 into dev-preindustrial+concentrations Feb 9, 2026
12 checks passed
@blimlim blimlim deleted the 402-access-oasis-mct branch February 9, 2026 05:08
@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-preindustrial+emissions

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Automatic Git cherry-picking of commit(s) c7e3cd1 into dev-preindustrial+emissions was successful.

The new pull request can be reviewed and approved here.

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-flat10

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-1pctCO2-bgc

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Automatic Git cherry-picking of commit(s) c7e3cd1 into dev-flat10 was successful.

The new pull request can be reviewed and approved here.

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-1pctCO2-rad

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Automatic Git cherry-picking of commit(s) c7e3cd1 into dev-1pctCO2-bgc was successful.

The new pull request can be reviewed and approved here.

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-1pctCO2

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Automatic Git cherry-picking of commit(s) c7e3cd1 into dev-1pctCO2-rad was successful.

The new pull request can be reviewed and approved here.

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-4xCO2+concentrations

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Automatic Git cherry-picking of commit(s) c7e3cd1 into dev-1pctCO2 was successful.

The new pull request can be reviewed and approved here.

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Automatic Git cherry-picking of commit(s) c7e3cd1 into dev-4xCO2+concentrations was successful.

The new pull request can be reviewed and approved here.

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-4xCO2+concentrations

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-4xCO2+concentrations

Automatic cherry-pick failed. It seems there are previous unfinished attempts to cherry-pick commits from this PR to the following branch(es):

If the current cherry-pick attempt is for a different set of commits, make sure that the previous attempts are fully merged and that the corresponding branches have been deleted.

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into dev-historical+concentrations

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Automatic Git cherry-picking of commit(s) c7e3cd1 into dev-historical+concentrations was successful.

The new pull request can be reviewed and approved here.

@blimlim
Copy link
Collaborator Author

blimlim commented Feb 9, 2026

!cherry-pick c7e3cd1 into test-historical+emissions

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

Automatic Git cherry-picking of commit(s) c7e3cd1 into test-historical+emissions was successful.

The new pull request can be reviewed and approved here.

@blimlim blimlim mentioned this pull request Feb 9, 2026
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants