Add tracing instrumentation to nuopc driver

[DusanJovic-NOAA]

This PR adds calls to ufs tracing routines that will create a trace file which can then be visualized, which is found to be useful in identifying various performance issues.

See ufs-weather-model issue ufs-community/ufs-weather-model#2883

Added calls are not used by default, unless build option (-DUFS_TRACING=ON) is specified when building the ufs-weather-model.

No changes in answers are expected.

[sanAkel]

@DusanJovic-NOAA is the module or a stub for the: ufs_trace_mod added?

If not:

• How can those who clone/build this repo build with -DUFS_TRACING=ON outside of the UFS, like say, someone at NCAR?
• Maybe put in a dummy stub in the external packages?

[DusanJovic-NOAA]

@DusanJovic-NOAA is the module or a stub for the: ufs_trace_mod added?

No.

If not:

• How can those who clone/build this repo build with -DUFS_TRACING=ON outside of the UFS, like say, someone at NCAR?

This tracing is specific to UFS, so when this repo is built as a subcomponent of ufs-weather-model users have an option to set -DUFS_TRACING=ON and tracing will be enabled.

• Maybe put in a dummy stub in the external packages?

What would that dummy stub do? As long as UFS_TRACING is not defined, the tracing code will not be compiled.

[sanAkel]

@DusanJovic-NOAA Passing your response to others with whom we co-share this coupler (e.g, NCAR) and see what they have to say.

Cc: @alperaltuntas

[DeniseWorthen]

One example of the ways stubs are used in the component caps is in CICE. There is a file called cice_wrapper_mod.F90 which contains UFS and CESM relevant options which do nothing (just return) depending on use-case. This allows the code itself to be free of the ifdefs

https://github.com/CICE-Consortium/CICE/blob/b01032506d71d86ca51cef8c3c28c1ab87532833/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90#L41-L43

https://github.com/CICE-Consortium/CICE/blob/b01032506d71d86ca51cef8c3c28c1ab87532833/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90#L524-L525

[DusanJovic-NOAA]

One example of the ways stubs are used in the component caps is in CICE. There is a file called cice_wrapper_mod.F90 which contains UFS and CESM relevant options which do nothing (just return) depending on use-case. This allows the code itself to be free of the ifdefs

https://github.com/CICE-Consortium/CICE/blob/b01032506d71d86ca51cef8c3c28c1ab87532833/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90#L41-L43

https://github.com/CICE-Consortium/CICE/blob/b01032506d71d86ca51cef8c3c28c1ab87532833/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90#L524-L525

I understand that the code will not have #ifdefs, but then the build system will need to differentiate whether to build and link the stub version of ufs_trace_mod or to forgo building it because the "real" ufs_trace_mod will be compiled and linked within ufs_weather_model. That means some kind of if-else or ifdef in the build system must be added. BTW, I do not see any build system in this repo, which means other 'host' models will need to be aware of this change and be modified accordingly.

[DeniseWorthen]

Isn't our build-system isolated from anyone else's because we have a MOM6 (or CICE, or CMEPS) interface w/ our own CMakeList etc?

[DusanJovic-NOAA]

Isn't our build-system isolated from anyone else's because we have a MOM6 (or CICE, or CMEPS) interface w/ our own CMakeList etc?

Yes, and since we (ufs-weather-model) control it, our build system is not a problem. But if we add the stub module, everybody else will need to update their build system to compile that new stub file. And until they update, their build will be broken. While by using #ifdefs everything will (should) just work without any modifications.

Anyway, if #idefs are a problem, I can remove them, and add the stub module. Where should it go? I guess in the same directory (i.e. config_src/drivers/nuopc_cap)

[DeniseWorthen]

Thanks Dusan. Lets see what feedback we get from NCAR side. I personally have an aversion to bunches of ifdefs, but if other users of the cap don't mind, then that is fine. I was initially responding to the comment about whether this could be a stub.

Regarding breaking the build of other systems, in both WW3 and CICE we have existing wrapper routines which could be utilized. Unfortunately not in CMEPS.

[sanAkel]

I'm not a fan of if def (compiler directives) either!

[jiandewang]

from NCAR:
Yes, I agree that there should be a stub module to eliminate ifdefs. Other than that, there should be an "only" clause for the "use ufs_trace_mod" statement, and the usage of module variables (mtype) should be discouraged, unless there is a good reason I am unaware of.

[DusanJovic-NOAA]

Would it be acceptable to replace the stub module with the wrapper module (like we did in CMEPS, for example) and update all current call statements to call the corresponding wrapper routine. That wrapper module will still have few ifdefs, but they will be located in that single module. The problem with having this stub module defined in this repo is that there could be only one of those, and if for some reason any other component defines it, there is a linker error due to multiple definitions.

[jiandewang]

@sanAkel can you review this PR ? I think the associated UWM PR is pending on this submodule PR

[sanAkel]

Thanks for adding the ufs_trace_wrapper module. Please see requested change(s).

[sanAkel]

@sanAkel can you review this PR ? I think the associated UWM PR is pending on this submodule PR

@jiandewang ✔️

[jiandewang]

This PR adds calls to ufs tracing routines that will create a trace file which can then be visualized, which is found to be useful in identifying various performance issues.

See ufs-weather-model issue ufs-community/ufs-weather-model#2883

Added calls are not used by default, unless build option (-DUFS_TRACING=ON) is specified when building the ufs-weather-model.

No changes in answers are expected.

@DusanJovic-NOAA can you modify the above as you are not using "DUSE" any more

[DusanJovic-NOAA]

This PR adds calls to ufs tracing routines that will create a trace file which can then be visualized, which is found to be useful in identifying various performance issues.
See ufs-weather-model issue ufs-community/ufs-weather-model#2883
Added calls are not used by default, unless build option (-DUFS_TRACING=ON) is specified when building the ufs-weather-model.
No changes in answers are expected.

@DusanJovic-NOAA can you modify the above as you are not using "DUSE" any more

UFS_TRACING preprocessor macro is still used in mom_ufs_trace_wrapper.F90

[gspetro-NOAA]

Testing on WM PR 2884 has completed successfully. This PR can be merged.

[jiandewang]

MOM6 merged, hash # e5a7888