Remove j and block Kmkhz 9c

[MetBenjaminWent]

PR Summary

Sci/Tech Reviewer: @christophermaynard
Code Reviewer: @harry-shepherd

Remove the j loop which is adversely affecting performance in the boundary layer, and improve blocking loops.

Performance data can be found in the umbrella issue, #106
closes #217
Initial KGO issues were fixed with adding denom to the private list L2546.

Taking over from #221.

Introducing further blocking loops have introduced new KGO changes, which appear to be fast debug O2 optimisation driven.
I've also noted that with differing loop ranges over threads, reducing nowaits is required, which opens the option of removing the barrier to aid further PSyclone work in the future (The barriers will remain for now).
With the above KGO occurrences, these have likely factored into the cause of KGO changes at 1T, which remain unaffected for full-debug, still indicating optimisation as the likely root cause.
As the KGO updates are holding (for the all of the tests involved in the OMP dev group) for 1, 2 or 4T, I'm reasonably confident that it's not a newly introduced race condition or similar.

Code Quality Checklist

• I have performed a self-review of my own code
• My code follows the project's style guidelines
• Comments have been included that aid understanding and enhance the readability of the code
• My changes generate no new warnings
• All automated checks in the CI pipeline have completed successfully

Testing

• I have tested this change locally, using the LFRic Apps rose-stem suite
• If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
• I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
• Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

trac.log

Test Suite Results - lfric_apps - Kmkhz-9c-by-hand-2/run7

Suite Information

Item	Value
Suite Name	Kmkhz-9c-by-hand-2/run7
Suite User	benjamin.went
Workflow Start	2026-03-05T11:07:23
Groups Run	ex1a_omp_developer

Dependency	Reference	Main Like
casim	MetOffice/casim@2026.03.1	True
jules	MetOffice/jules@2026.03.1	True
lfric_apps	MetBenjaminWent/lfric_apps@Kmkhz-9c-by-hand-2	False
lfric_core	MetOffice/lfric_core@2026.03.1	True
moci	MetOffice/moci@2026.03.1	True
SimSys_Scripts	MetOffice/SimSys_Scripts@2026.03.1	True
socrates	MetOffice/socrates@2026.03.1	True
socrates-spectral	MetOffice/socrates-spectral@2026.03.1	True
ukca	MetOffice/ukca@2026.03.1	True

Task Information

✅ succeeded tasks - 51

Test Suite Results - lfric_apps - Kmkhz-9c-by-hand-2/run6

Suite Information

Item	Value
Suite Name	Kmkhz-9c-by-hand-2/run6
Suite User	benjamin.went
Workflow Start	2026-03-05T11:05:13
Groups Run	all

Dependency	Reference	Main Like
casim	MetOffice/casim@2026.03.1	True
jules	MetOffice/jules@2026.03.1	True
lfric_apps	MetBenjaminWent/lfric_apps@Kmkhz-9c-by-hand-2	False
lfric_core	MetOffice/lfric_core@2026.03.1	True
moci	MetOffice/moci@2026.03.1	True
SimSys_Scripts	MetOffice/SimSys_Scripts@2026.03.1	True
socrates	MetOffice/socrates@2026.03.1	True
socrates-spectral	MetOffice/socrates-spectral@2026.03.1	True
ukca	MetOffice/ukca@2026.03.1	True

Task Information

❌ failed tasks - 2

Task	State
run_gungho_model_baroclinic-alt1-C24s_MG_azspice_gnu_fast-debug-64bit	failed
run_gungho_model_robert-moist-lam-BiP100x8-10x10_azspice_gnu_fast-debug-64bit	failed

• Both are due to timeouts on HPC, not related to changes

✅ succeeded tasks - 1503

Security Considerations

• I have reviewed my changes for potential security issues
• Sensitive data is properly handled (if applicable)
• Authentication and authorisation are properly implemented (if applicable)

Performance Impact

• Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

• Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

• Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

• If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

• I understand this area of code and the changes being added
• The proposed changes correspond to the pull request description
• Documentation is sufficient (do documentation papers need updating)
• Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

• All dependencies have been resolved
• Related Issues have been properly linked and addressed
• CLA compliance has been confirmed
• Code quality standards have been met
• Tests are adequate and have passed
• Documentation is complete and accurate
• Security considerations have been addressed
• Performance impact is acceptable

[MetBenjaminWent]

kmkhz_9c_after.txt
kmkhz_9c_before.txt

Listing files before and after.

[MetBenjaminWent]

From #221:

Running into some KGO issues at fast debug.
Fixing the denom value, the only new LHS variable in the history that was not in the OMP sections, means that fast-debug should be okay.

Below testing at full-debug indicates that the KGO is good, whilst 1 and 4 are technically added, 2T is consistent with trunk, and they hold between runs, which with the denom LHS above, they were not.

The most recent changes here increase the number of ii blocking loops present with the dynamic schedule.
I'll see if an update holds, otherwise my current leading theory is that, like lsppn, the blocking dynamic loops at fast debug are running into some trouble and being over-optimised. Using the UM flags resolved it.

Test Suite Results - lfric_apps - kmkhz_9c_pysclone/run20

Suite Information

Item	Value
Suite Name	kmkhz_9c_pysclone/run20
Suite User	benjamin.went
Workflow Start	2026-02-10T12:53:47
Groups Run	ex1a_omp_C48_cce_full

Dependency	Reference	Main Like
casim	MetOffice/casim@2025.12.1	True
jules	MetOffice/jules@69aaf4d	True
lfric_apps	MetBenjaminWent/lfric_apps@kmkhz_9c_pysclone	False
lfric_core	MetOffice/lfric_core@bbb3d8a	True
moci	MetOffice/moci@2025.12.1	True
SimSys_Scripts	MetOffice/SimSys_Scripts@2025.12.1	True
socrates	MetOffice/socrates@2025.12.1	True
socrates-spectral	MetOffice/socrates-spectral@2025.12.1	True
ukca	MetOffice/ukca@2025.12.1	True

Task Information

✅ succeeded tasks - 12

[MetBenjaminWent]

From #221:
Given full-debug does not change and holds between runs, likely it might be an optimisation occurence from CCE?

Updating the CCE KGOs show it holds.

Further testing, do they hold as seg-size changes?

Test Suite Results - lfric_apps - kmkhz_9c_pysclone/run21

Suite Information

Item	Value
Suite Name	kmkhz_9c_pysclone/run21
Suite User	benjamin.went
Workflow Start	2026-02-11T14:48:18
Groups Run	ex1a_omp_developer

Dependency	Reference	Main Like
casim	MetOffice/casim@2025.12.1	True
jules	MetOffice/jules@69aaf4d	True
lfric_apps	MetBenjaminWent/lfric_apps@kmkhz_9c_pysclone	False
lfric_core	MetOffice/lfric_core@bbb3d8a	True
moci	MetOffice/moci@2025.12.1	True
SimSys_Scripts	MetOffice/SimSys_Scripts@2025.12.1	True
socrates	MetOffice/socrates@2025.12.1	True
socrates-spectral	MetOffice/socrates-spectral@2025.12.1	True
ukca	MetOffice/ukca@2025.12.1	True

Task Information

✅ succeeded tasks - 51

[MetBenjaminWent]

From #221

Barriers have been left. They should be removable (after removing some of the nowaits), but I'll look at this further in the PSyclone version of this ticket.

Current leading thought is that KGO changes are Optimisation changes, but they are more widespread.

Likely the intial KGO change was caused by the seg size loop intros, and as the bounds change, but the no waits didn't, the threaded runs changed, then stabalised. Changing the segment size after updating the KGOs does not cause them to change.

Changing further syncronisation points may have allowed the compiler to change how it has optimised, generting further KGO shifts.

Full debug otherwise remains consistent as a reference point, where previously with a genuine bug, they did not. This reinforces that the KGO change is optmisation driven.

[Adrian-Lock]

All looks fine to me