Branch of fesom2.6 including recom and tracer parallelisation by a270105 · Pull Request #681 · FESOM/fesom2

a270105 · 2025-01-29T15:56:11Z

Tracer parallelisation can be switched on by __usetp in fesom2/src/CMakeLists.txt and onlye used when recom is on.

Define new variables to track tracer changes due to advection and diffusion. We want to save for now diffusion and advection contribution to the tracer changes. Horizontal and vertical diffusion includes Redi parametrization (if it is set .true.).

Fill __ciso directive to ensure that carbon isotope code works. Medusa interface is added.

… io_restart.F90

…sation

a270105 · 2025-01-29T16:00:11Z

By compilation I found an issue regrading new compilations. I need to manually delete the executable file fesom.x in fesom2.build/bin and then compile the model. Otherwise, after a successful compilation I will still find the old fesom.x in fesom2/bin.

…atbox

JanStreffing

Summary of review:
There were a few places that need changing, some that may not need to change but would require a closer look that everything is indeed ok. There are a number of style improvements to be made. All comment that start with ! kh date should start without the name and date. Same for !YY, !O:G !OG and !.OG. Git blame can tell you this info.

In addition to the standalone FESOM2 CI tests, this branch needs to be tested by me as part of AWI-CM3, by @ackerlar as part of AWI-ESM2, by @sebastianbeyer or @suvarchal as part of IFS-FESOM, by @mbutzin with active tracers. I'm assuming here, that you have already tested this branch with recom and without _usetp

JanStreffing · 2025-02-03T18:39:02Z

CMakeLists.txt

 set(USE_ICEPACK OFF CACHE BOOL "compile fesom with the Iceapck modules for sea ice column physics.")
 set(OPENMP_REPRODUCIBLE OFF CACHE BOOL "serialize OpenMP loops that are critical for reproducible results")
-set(RECOM_COUPLED OFF CACHE BOOL "compile fesom including biogeochemistry, REcoM3")
+set(RECOM_COUPLED ON CACHE BOOL "compile fesom including biogeochemistry, REcoM3")


This should be turned back for the main branch.

JanStreffing · 2025-02-03T18:39:48Z

src/MOD_TRACER.F90

 integer                                       :: AB_order=2
 integer                                       :: ID
+!___________________________________________________________________________
+! TODO: Make it as a part of namelist.tra


Should be done before more?

need to clarify with Özgür

src/cpl_driver.F90

src/fesom_module.F90

src/gen_forcing_couple.F90

src/io_restart.F90

src/oce_ale_tracer.F90

JanStreffing · 2025-02-03T19:06:39Z

In addition to the changes of the review, the merge conflicts should be solved. Most of them look like simple double additions of new features that both have to be kept. e.g. parallel tracers and icebergs.

ackerlar · 2025-02-05T09:33:50Z

Is there any template fesom-recom yaml that I can use for testing?

ackerlar · 2025-02-05T09:46:03Z

I get this error during compilation:

/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/associate_mesh_ass.h(57): warning #5117: Bad # preprocessor line
#if defined(__async_icebergs)
-^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/associate_mesh_ass.h(59): warning #5117: Bad # preprocessor line
#endif
-^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(363): error #6784: The number of actual arguments cannot be greater than the number of dummy arguments.   [OASIS_INIT_COMP]
    CALL oasis_init_comp(comp_id, comp_name, ierror, num_program_groups = num_fesom_groups)
---------^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(961): error #6404: This name does not have a type, and must have an explicit type.   [MPI_COMM_FESOM_SAME_RANK_IN_GROUPS]
      call MPI_Bcast(action, 1, MPI_LOGICAL, 0, MPI_COMM_FESOM_SAME_RANK_IN_GROUPS, MPIerr)
------------------------------------------------^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(961): error #6404: This name does not have a type, and must have an explicit type.   [MPIERR]
      call MPI_Bcast(action, 1, MPI_LOGICAL, 0, MPI_COMM_FESOM_SAME_RANK_IN_GROUPS, MPIerr)
------------------------------------------------------------------------------------^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(976): error #6404: This name does not have a type, and must have an explicit type.   [MYDIM_NOD2D]
          call MPI_Bcast(data_array, myDim_nod2d, MPI_DOUBLE_PRECISION, 0, MPI_COMM_FESOM_SAME_RANK_IN_GROUPS, MPIerr)
-------------------------------------^
compilation aborted for /home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90 (code 1)
make[2]: *** [src/CMakeFiles/fesom.dir/build.make:205: src/CMakeFiles/fesom.dir/cpl_driver.F90.o] Error 1

a270105 · 2025-02-05T12:22:28Z

I am still testing the code. So far I can run ocean-only without __usetp in CMakeLists and compile the model with __usetp. But I have not yet tested the coupled setup.

a270105 · 2025-02-05T12:52:10Z

I get this error during compilation:

/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/associate_mesh_ass.h(57): warning #5117: Bad # preprocessor line
#if defined(__async_icebergs)
-^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/associate_mesh_ass.h(59): warning #5117: Bad # preprocessor line
#endif
-^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(363): error #6784: The number of actual arguments cannot be greater than the number of dummy arguments.   [OASIS_INIT_COMP]
    CALL oasis_init_comp(comp_id, comp_name, ierror, num_program_groups = num_fesom_groups)
---------^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(961): error #6404: This name does not have a type, and must have an explicit type.   [MPI_COMM_FESOM_SAME_RANK_IN_GROUPS]
      call MPI_Bcast(action, 1, MPI_LOGICAL, 0, MPI_COMM_FESOM_SAME_RANK_IN_GROUPS, MPIerr)
------------------------------------------------^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(961): error #6404: This name does not have a type, and must have an explicit type.   [MPIERR]
      call MPI_Bcast(action, 1, MPI_LOGICAL, 0, MPI_COMM_FESOM_SAME_RANK_IN_GROUPS, MPIerr)
------------------------------------------------------------------------------------^
/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(976): error #6404: This name does not have a type, and must have an explicit type.   [MYDIM_NOD2D]
          call MPI_Bcast(data_array, myDim_nod2d, MPI_DOUBLE_PRECISION, 0, MPI_COMM_FESOM_SAME_RANK_IN_GROUPS, MPIerr)
-------------------------------------^
compilation aborted for /home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90 (code 1)
make[2]: *** [src/CMakeFiles/fesom.dir/build.make:205: src/CMakeFiles/fesom.dir/cpl_driver.F90.o] Error 1

I just compiled it with FESOM_COUPLED ON and didn't get any error. Did you compile the model with esm-tools?

ackerlar · 2025-02-05T13:12:09Z

@a270105 yes, I used esm_tools: esm_master get-awiesm-2.6, changed the FESOM branch to fesom2.6_recom_tp and then esm_master comp-awiesm-2.6

which oasis version are you using?

ackerlar · 2025-02-05T19:21:05Z

I got at least rid of this error

/home/a/a270124/model_codes/testing_and_debugging/awiesm-2.6/fesom-2.6/src/cpl_driver.F90(363): error #6784: The number of actual arguments cannot be greater than the number of dummy arguments.   [OASIS_INIT_COMP]
    CALL oasis_init_comp(comp_id, comp_name, ierror, num_program_groups = num_fesom_groups)

when switching to oasis branch feat/multi-group-support. Before I used 2.8mct-awiesm-2.1. However, the other errors remain

ackerlar · 2025-02-05T19:54:59Z

src/fesom_module.F90

-        call cpl_oasis3mct_init(f%partit,f%partit%MPI_COMM_FESOM)
+!        call cpl_oasis3mct_init(f%partit,f%partit%MPI_COMM_FESOM)
+! kh 02.12.21
+#if defined(__recom) && defined(__usetp)


"localCommunicator" is neither defined in fesom_module nor in MOD_PARTIT. Should this really be "localCommunicator" or "f%partit%MPI_COMM_FESOM"?

I can't really answer this. These lines already exited in fesom2.5-recom and another earlier version where Özgür and I merged their complex recom version and my simpler paleo version. But I can't find them in my paleo version where Kai first time implemented tracer parallelisation. There this part of code is still in fvom_main.f90 and only contains:
#if defined (__oasis)

! kh 21.03.22 pass num_fesom_groups to coupler
call cpl_oasis3mct_init(MPI_COMM_FESOM, num_fesom_groups)
#endif
t1 = MPI_Wtime()

call par_init

......

So I think it should be for the case with tracer parallelisation:
call cpl_oasis3mct_init(f%partit, f%partit%MPI_COMM_FESOM, num_fesom_groups)

ackerlar · 2025-02-05T20:23:10Z

FESOM compiled for me with some minor changes in MPI calls. I also changed

-        call cpl_oasis3mct_init(f%partit, f%partit%localCommunicator, num_fesom_groups)
+        call cpl_oasis3mct_init(f%partit, f%partit%MPI_COMM_FESOM, num_fesom_groups)

in fesom_module as localCommunicator is not defined outside of cpl_driver. Please check whether this makes sense.

However, the model crashes as FESOM is missing some entries in namelist.config. Is there a recom specific namelist.config?

a270105 · 2025-02-05T20:58:09Z

@a270105 yes, I used esm_tools: esm_master get-awiesm-2.6, changed the FESOM branch to fesom2.6_recom_tp and then esm_master comp-awiesm-2.6

which oasis version are you using?

a270105 · 2025-02-05T21:22:27Z

FESOM compiled for me with some minor changes in MPI calls. I also changed
-        call cpl_oasis3mct_init(f%partit, f%partit%localCommunicator, num_fesom_groups)
+        call cpl_oasis3mct_init(f%partit, f%partit%MPI_COMM_FESOM, num_fesom_groups)
in fesom_module as localCommunicator is not defined outside of cpl_driver. Please check whether this makes sense.

However, the model crashes as FESOM is missing some entries in namelist.config. Is there a recom specific namelist.config?

Do you compile with __usetp? If not, there is no additional entry needed in namelist.config.

…nto fesom2.6_recom_tp

ackerlar · 2025-02-06T08:09:37Z

Do you compile with __usetp? If not, there is no additional entry needed in namelist.config.

Yes, I set RECOM_COUPLED=ON which gives

if(${RECOM_COUPLED})
#   target_compile_definitions(${PROJECT_NAME} PRIVATE __recom USE_PRECISION=2 __usetp)
#   target_compile_definitions(${PROJECT_NAME} PRIVATE __recom USE_PRECISION=2)
   target_compile_definitions(${PROJECT_NAME} PRIVATE __recom USE_PRECISION=2 __3Zoo2Det __coccos __usetp)
endif()

in src/CMakeLists.txt and __usetp should be used if I understand correctly.

…recom

…nto fesom2.6_recom_tp updating local branch

…lelisation, sinking fluxes needed for MEDUSA added in io_meandata.F90

JanStreffing · 2026-02-18T09:00:04Z

@suvarchal, Can you give an estimate until when you can provide an alternative RECOM parallelization method? If not, I suggest we go ahead with this merge.

JanStreffing · 2026-03-03T09:45:13Z

@suvarchal agrees to merge this. However the branch is now in a state where some of the tests are failing.
Can you have a look @a270105? In particular:

 Error termination. Backtrace:
At line 430 of file /__w/fesom2/fesom2/src/gen_model_setup.F90
Fortran runtime error: End of file

As soon as the tests are running we can merge. Afterward we can work on getting this to run inside AWI-ESM3.

a270105 · 2026-03-03T10:56:09Z

I think that we do not need the subroutine read_namelist_run_config any more, since the only parameter that is needed for tracer parallelisation is num_fesom_groups and is read in gen_model_setup.F90 (L165):
#if defined(__recom) && defined(__usetp) ! number of groups for multi FESOM group loop parallelization integer :: num_fesom_groups=1 namelist /run_config/ num_fesom_groups #endif

JanStreffing · 2026-03-03T10:57:35Z

I think that we do not need the subroutine read_namelist_run_config any more, since the only parameter that is needed for tracer parallelisation is num_fesom_groups and is read in gen_model_setup.F90 (L165): #if defined(__recom) && defined(__usetp) ! number of groups for multi FESOM group loop parallelization integer :: num_fesom_groups=1 namelist /run_config/ num_fesom_groups #endif

Feel free to remove. I think what I tried broke it even more.

a270105 · 2026-03-03T10:59:33Z

I just commented the line in fesom_module.F90 to call this subrountine and compiled the code.
Should all tests be passed for this branch? At least I need different namelist.config, namelist.tra and namelist.recom to run my tests.

JanStreffing · 2026-03-03T13:21:32Z

These tests all need to work, yes. @suvarchal, can you have a look with @a270105 ?

suvarchal · 2026-03-04T11:30:40Z

These tests all need to work, yes. @suvarchal, can you have a look with @a270105 ?

give me a day, i will fix these tests, we can hopefully merge then.

ogurses and others added 8 commits October 15, 2024 15:08

add recom_sms diag

3e84a88

feat(recom): add ciso and medusa components

434f0e6

Fill __ciso directive to ensure that carbon isotope code works. Medusa interface is added.

input to bc_surface from medusa added and Sed_input deleted

0864237

changes in src for tracer parallelisation

a3dfd9f

change defined(__recom)||defined(__usetp) to only defined(__usetp) in…

911fd3c

… io_restart.F90

corrections to compile with tracer parallelisation on

358082d

cleanup and small corrections of flags for recom and tracer paralleli…

0d3834a

…sation

corrections in array allocation for Diags and initialization for use_…

74a5e59

…atbox

JanStreffing previously requested changes Feb 3, 2025

View reviewed changes

JanStreffing assigned a270105 Feb 3, 2025

JanStreffing added the enhancement New feature or request label Feb 3, 2025

ackerlar reviewed Feb 5, 2025

View reviewed changes

make MPI calls consistent fix?? cpl_oasis3mct_init call in fesom_module

e8117d8

a270105 closed this Feb 5, 2025

a270105 reopened this Feb 5, 2025

a270105 added 2 commits February 6, 2025 00:04

localcommunicator removed in fesom_module.F90

604a88a

Merge branch 'fesom2.6_recom_tp' of https://github.com/FESOM/fesom2 i…

cb11456

…nto fesom2.6_recom_tp

Ying Ye and others added 11 commits January 4, 2026 21:33

DIC_PI false by default

53c45ed

Merge branch 'main' into fesom2.6_recom_tp

760ebc7

updating namelists and grouping iron chesmitry paramters in namelist.…

aa514b1

…recom

Merge branch 'main' into fesom2.6_recom_tp

a8cf488

Merge branch 'fesom2.6_recom_tp' of https://github.com/FESOM/fesom2 i…

15dd785

…nto fesom2.6_recom_tp updating local branch

bugfix for sedimentary input from MEDUSA in gen_surface_forcing.F90

3737859

fix of line breaks in recom namelist

7e41192

delete namelists not needed for 2.7

7b119ff

updating namelists

d7654e5

changes by Kai for benthic and sinking fluxes when using tracer paral…

fc84f04

…lelisation, sinking fluxes needed for MEDUSA added in io_meandata.F90

modification for biological fractionation of carbon

f2951ae

JanStreffing modified the milestones: FESOM 2.7.1, FESOM 2.8 Feb 18, 2026

Merge branch 'main' into fesom2.6_recom_tp

405b830

Merge branch 'main' into fesom2.6_recom_tp

b30693e

JanStreffing and others added 2 commits March 3, 2026 10:52

add partit arg to read_namelist_run_config call

cd362a4

changes wrt. C isotopes

eb707e3

Ying Ye and others added 2 commits March 3, 2026 13:17

fix to read num_fesom_groups

303f877

Merge branch 'main' into fesom2.6_recom_tp

3cd920d

suvarchal and others added 3 commits March 3, 2026 14:36

fix preprocessor oasis and tracer parallel block

fe45593

Merge branch 'main' into fesom2.6_recom_tp

51426e8

get rid of warnings of using comma before io list

8f089c6

add switch for variable grazing efficiency

a8f42a5

Conversation

a270105 commented Jan 29, 2025

Uh oh!

a270105 commented Jan 29, 2025

Uh oh!

JanStreffing left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JanStreffing Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

JanStreffing Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

a270105 Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

suvarchal Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JanStreffing commented Feb 3, 2025

Uh oh!

ackerlar commented Feb 5, 2025

Uh oh!

ackerlar commented Feb 5, 2025

Uh oh!

a270105 commented Feb 5, 2025

Uh oh!

a270105 commented Feb 5, 2025

Uh oh!

ackerlar commented Feb 5, 2025

Uh oh!

ackerlar commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ackerlar Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

a270105 Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

ackerlar commented Feb 5, 2025

Uh oh!

a270105 commented Feb 5, 2025

Uh oh!

a270105 commented Feb 5, 2025

Uh oh!

ackerlar commented Feb 6, 2025

Uh oh!

JanStreffing commented Feb 18, 2026

Uh oh!

JanStreffing commented Mar 3, 2026

Uh oh!

a270105 commented Mar 3, 2026

Uh oh!

JanStreffing commented Mar 3, 2026

Uh oh!

a270105 commented Mar 3, 2026

Uh oh!

JanStreffing commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

suvarchal commented Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

JanStreffing left a comment •

edited

Loading

ackerlar commented Feb 5, 2025 •

edited

Loading

JanStreffing commented Mar 3, 2026 •

edited

Loading