Merge GEOMAR stuff finally into release

CHANGES.md

@@ -0,0 +1,33 @@
+# Changes
+
+## esm_tests: add NAMELIST_PATH and RUNSCRIPT_PATH substitutions for machine-agnostic comparison
+
+**File:** `src/esm_tests/cli.py`
+
+**Problem:** The esm_tools installation path relative to `$HOME` differs between users and machines (e.g. `~/esm_tools` vs `~/Codes/esm_tools`). After `HOME_DIR` substitution, `config_sources` entries like `esm_namelist_dir` still differed between truth files and the current run.
+
+**Fix:** Added `NAMELIST_PATH` and `RUNSCRIPT_PATH` entries to `rm_user_info` (sourced from `esm_tools.get_namelist_filepath()` and `esm_tools.get_runscript_filepath()`) placed **before** `HOME_DIR` in the dict so they are substituted first. Truth files in `last_tested/` must be regenerated on each machine with `esm_tests -s` to adopt the new placeholders.
+
+## esm_tests: fix comp script generation for models with nested pushd paths
+
+**File:** `src/esm_tests/tests.py`
+
+**Root cause:** In check-mode compilation, `comp_test()` creates dummy empty directories to trick `esm_master` into thinking source code is present, allowing it to generate `comp-*_script.sh` files without actually compiling. The folder-collection logic filtered out any `pushd` path containing `/`:
+
+```python
+and "/" not in found_format[0]
+```
+
+For `foci-mops_lmu`, NEMO's compile step uses a nested path:
+
+```
+pushd nemo-ORCA05_LIM2_FOCI_MOPS_OASISMCT4/CONFIG/ORCA05_LIM2_FOCI_MOPS_OASISMCT4
+```
+
+Because this path contains `/`, the dummy directory was never created. When `esm_master` then ran without `-c`, it detected the missing folder and aborted — before generating any `comp-*_script.sh` files.
+
+**Fix:**
+- Removed the `"/" not in found_format[0]` condition so nested paths are collected.
+- Changed `os.mkdir` to `os.makedirs(..., exist_ok=True)` so nested directories are created recursively.
+
+Other models (oasis, echam, xios) were unaffected because their `pushd` paths are all single-level (no `/`). NEMO's FOCI-MOPS configuration requires the nested `CONFIG/` subdirectory.

CONTEXT.md

@@ -1,6 +1,14 @@
 # CLAUDE.md
 
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+This file provides guidance to coding agents when working with code in this repository.
+
+## Context files
+
+Add these files to memory:
+
+- CONTEXT.md: This file, with the context for coding agents abour the tool. Expand it with relevante information about ESM-Tools is discovered during a claude-code session. Make sure it is still brief and useful for general purpose context of the tools.
+- CHANGES.md: summary of the changes carried out in the currend development. To be updated while changes are applied. Changes will be removed at the end of the development, so there might not be a CHANGES.md file. In that case create it.
+- Other md files in the root directory might describe a feature or contain a plan. Do a fast search for those files and read them for context.
 
 ## Project Overview
 
@@ -69,7 +77,7 @@ user_error("ErrorType", "message", exit_code=1, hints=["hint1"])
 
 ### ESM-Tests (`src/esm_tests/`)
 
-Testing framework for compilation scripts and simulation runs. Reference data lives in a git submodule (`src/esm_tests/resources/` → `esm_tests_info` repo) with `runscripts/` and `last_tested/` directories.
+Testing framework for compilation scripts and simulation runs. Reference data lives in a git submodule (`src/esm_tests/resources/` → `esm_tests_info` repo) with `runscripts/` and `last_tested/` directories. Machine/user-specific paths are replaced with placeholders (`<TEST_DIR>`, `<NAMELIST_PATH>`, `<RUNSCRIPT_PATH>`, `<HOME_DIR>`, `<USER_ACCOUNT>`) via `rm_user_info` in `cli.py`; order matters — specific paths before `HOME_DIR`.
 
 ## Docstring Conventions
 

configs/components/echam/echam.datasets.yaml

@@ -154,3 +154,45 @@ echam:
         r0008:
             greenhouse_hist_name: "greenhouse_ssp245"
             MAC-SP_hist_name: "MAC-SP_ssp245"
+
+    choose_use_moz:
+        true:
+            epp_dir: ${input_dir}/HAMMOZ_${echam.resolution}/epp_T63L95
+            photolysis_dir: ${input_dir}/HAMMOZ/photolysis
+            ap_index_dir: ${input_dir}/HAMMOZ/ap_index
+
+            add_input_sources:
+                moz_effxs: "${input_dir}/HAMMOZ/moz_effxs.nc"
+                moz_rsf: "${input_dir}/HAMMOZ/moz_rsf_gt200nm.nc"
+                moz_temp: "${input_dir}/HAMMOZ/moz_temp_prs_GT200nm_JPL10_c130206.nc"
+                moz_xs: "${input_dir}/HAMMOZ/moz_xs_short.nc"
+                moz_uvalbedo: "${input_dir}/HAMMOZ_${echam.resolution}/moz_uvalbedo_${echam.resolution}.nc"
+                moz_xtsoil: "${input_dir}/HAMMOZ_${echam.resolution}/soilpHfrac_${echam.resolution}.nc"
+                moz_surface_properties: "${input_dir}/HAMMOZ_${echam.resolution}/xtsurf_v2_${echam.resolution}.nc"
+
+            add_forcing_sources:
+                # pi control forcing
+                moz_lbc_pictl: "${input_dir}/HAMMOZ_${echam.resolution}/hammoz_lbc_piControl_CMIP6_CCMI2_${echam.resolution}.nc"
+                moz_sad_sulf_pictl: "${input_dir}/HAMMOZ_${echam.resolution}/moz_sad_sulf_piControl_c080220_${echam.resolution}${echam.levels}.nc"
+                # historical and scenario forcing
+                moz_lbc: 
+                    "${input_dir}/HAMMOZ_${echam.resolution}/hammoz_lbc_17500116-21011216_CMIP6_CCMI2_ssp370_${echam.resolution}.nc":
+                        to: 2014
+                    "${input_dir}/HAMMOZ_${echam.resolution}/hammoz_lbc_17500116-21011216_CMIP6_CCMI2_${echam.scenario}_${echam.resolution}.nc":
+                        from: 2015
+                moz_sad_sulf: "${input_dir}/HAMMOZ_${echam.resolution}/moz_sad_sulf_1849-2100_c080220_${echam.resolution}${echam.levels}.nc"
+                moz_epp: 
+                    "${epp_dir}/ubc_noy_epp_@YEAR@_${echam.resolution}${echam.levels}_20toplevs.nc":
+                        from: 1849
+                        to: 2100
+                moz_photolysis: 
+                    "${photolysis_dir}/etfphot_@YEAR@.nc":
+                        from: 1849
+                        to: 2100
+                moz_ap_index: 
+                    "${ap_index_dir}/apindex_1850.nc":
+                        to: 1849
+                    "${ap_index_dir}/apindex_@YEAR@.nc":
+                        from: 1850
+                        to: 2100
+

configs/components/echam/echam.yaml

@@ -9,7 +9,10 @@ echam:
             available_versions:
             - 6.3.05p2
             - 6.3.05p2-foci
+            - 6.3.05p2-moz
+            - 6.3.05p2-moz_mct2
             - 6.3.05p2-foci_autotools
+            - 6.3.05p2-foci_lmu
             - 6.3.05p2-foci_oasismct4
             - 6.3.02p4
             - 6.3.04p1-esm_interface
@@ -73,6 +76,36 @@ echam:
               6.3.05p2-foci:
                 branch: esm-tools
                 git-repository: https://git.geomar.de/foci/src/echam.git
+              6.3.05p2-moz:
+                branch: oasis-mct5
+                git-repository: https://git.geomar.de/foci/src/echam_hammoz.git
+                clean_command: rm -rf src/echam/bin; rm -rf bin; make clean; make distclean
+                comp_command: mkdir -p src/.deps yaxt/src/.deps yaxt/tests/.deps; 
+                  ./config/createMakefiles.pl; autoreconf -i --force; unset FCFLAGS; unset FFLAGS; 
+                  ./configure --prefix=$(pwd)/../ --with-fortran=intel --with-coupler=oasis3-mct-nemo --enable-cdi-pio OASIS3MCTROOT=$(pwd)/../oasis/INSTALL_OASIS.ESMTOOLS/ SUPPORTROOT=$(pwd)/../ SUPPORT_LIB='-L$(SUPPORTROOT)/lib -lsupport' SRCDIRS="support cdi" ;
+                  make -j `nproc --all`; make install -j `nproc --all`;
+                install_bins: ''
+              6.3.05p2-moz_mct2:
+                branch: master 
+                git-repository: https://git.geomar.de/foci/src/echam_hammoz.git
+                clean_command: rm -rf src/echam/bin; rm -rf bin; make clean; make distclean
+                comp_command: mkdir -p src/.deps yaxt/src/.deps yaxt/tests/.deps; 
+                  ./config/createMakefiles.pl; autoreconf -i --force; unset FCFLAGS; unset FFLAGS; 
+                  ./configure --prefix=$(pwd)/../ --with-fortran=intel --with-coupler=oasis3-mct-nemo --enable-cdi-pio OASIS3MCTROOT=$(pwd)/../ SUPPORTROOT=$(pwd)/../ SUPPORT_LIB='-L$(SUPPORTROOT)/lib -lsupport' SRCDIRS="support cdi" ;
+                  make -j `nproc --all`; make install -j `nproc --all`;
+                install_bins: ''
+              6.3.05p2-foci_lmu:
+                git-repository: https://gitlab.dkrz.de/lmu-lus/mpiesm/hbp_mpiesm-landveg.git 
+                branch: feature/couple_to_nemo3.6_mops
+                clean_command: cd src/echam; rm -rf src/echam/bin; rm -rf bin; make clean; make distclean
+                comp_command: cd src/echam; 
+                  git submodule init; git submodule update cdi || echo "No update required"; 
+                  mkdir -p src/.deps yaxt/src/.deps yaxt/tests/.deps; 
+                  ./config/createMakefiles.pl; autoreconf -i --force;
+                  unset FCFLAGS; unset FFLAGS; 
+                  ./configure --prefix=$(pwd)/../../../ --with-fortran=intel --with-coupler=oasis3-mct-nemo --enable-cdi-pio OASIS3MCTROOT=$(pwd)/../../../oasis/INSTALL_OASIS.ESMTOOLS/ SUPPORTROOT=$(pwd)/../ SUPPORT_LIB='-L$(SUPPORTROOT)/lib -lsupport' SRCDIRS="support cdi" ;
+                  make -j `nproc --all`; make install -j `nproc --all`;
+                install_bins: ''
               6.3.05p2-foci_autotools:
                 clean_command: rm -rf src/echam/bin; rm -rf bin; make clean; make distclean
                 comp_command: mkdir -p src/.deps yaxt/src/.deps yaxt/tests/.deps;
@@ -128,7 +161,7 @@ echam:
     ocean_resolution: 'GR15'
     disturbance: "1.000001"
     lresume: false
-
+    use_moz: false
 
     lgfw: False
     lupdate_orography: False
@@ -270,12 +303,33 @@ echam:
                     dataset: r0008
                     nprocrad: "remove_from_namelist"
                     lrad_async: "remove_from_namelist"
+            6.3.05p2-foci_autotools:
+                    repo_tag: not_set
+                    dataset: r0008
+                    nprocrad: "remove_from_namelist"
+                    lrad_async: "remove_from_namelist"
+            6.3.05p2-moz:
+                    repo_tag: not_set
+                    dataset: r0008
+                    nprocrad: "remove_from_namelist"
+                    lrad_async: "remove_from_namelist"
+                    use_moz: true
+            6.3.05p2-moz_mct2:
+                    repo_tag: not_set
+                    dataset: r0008
+                    nprocrad: "remove_from_namelist"
+                    lrad_async: "remove_from_namelist"
+                    use_moz: true
             6.3.05p2-foci_oasismct4:
                     repo_tag: not_set
                     dataset: r0008
                     nprocrad: "remove_from_namelist"
                     lrad_async: "remove_from_namelist"
-
+            6.3.05p2-foci_lmu:
+                    repo_tag: not_set
+                    dataset: r0008
+                    nprocrad: "remove_from_namelist"
+                    lrad_async: "remove_from_namelist"
             6.3.05p2-concurrent_radiation-paleodyn:
                     concurrent_radiation_code: true
                     repo_tag: not_applicable
@@ -333,6 +387,8 @@ echam:
                                             lrad_async: "${lrad_async}"
                     add_streams:
                         - rad
+                    add_restartstreams:
+                        - rad
             "*":
                     from_concurrent_radiation_code_choose: <NO_CONCURRENT_RADIATION_CODE>
 
@@ -497,7 +553,7 @@ echam:
     choose_lresume:
             False:
                     restart_in_modifications:
-                            "[[streams-->STREAM]]":
+                            "[[restartstreams-->STREAM]]":
                                     - "vdate <--set_global_attr-- ${start_date!syear!smonth!sday}"
                                       # - fdate "<--set_dim--" ${year_before_date}
                                       # - ndate "<--set_dim--" ${steps_in_year_before}
@@ -574,6 +630,9 @@ echam:
                         lco2_emis: true # read carbon emissions; need carbon_emissions.nc in work
                         lco2_2perc: true # limit maximum co2 growth rate to 2% wrt previous time step
             add_choose_scenario: # other loop order (1:add_forcing_files 2:add_choose_scenario) does not work
+                cmip6hist:
+                    add_forcing_files:
+                        carbon_emissions: carbon_emissions_hist
                 HIST:
                     add_forcing_files:
                         carbon_emissions: carbon_emissions_hist
@@ -626,6 +685,9 @@ echam:
                                     add_streams:
                                             - wiso
                                             - accw_wiso
+                                    add_restartstreams:
+                                            - wiso
+                                            - accw_wiso
 
                                     # The following lines take care of the renaming of the
                                     # ozon files, and their copying
@@ -642,6 +704,45 @@ echam:
                                     nwiso: 0
                                     lwiso_rerun: False
 
+    choose_use_moz:
+        true:
+            add_input_files:
+                moz_effxs: moz_effxs 
+                moz_rsf: moz_rsf 
+                moz_temp: moz_temp
+                moz_xs: moz_xs 
+                moz_uvalbedo: moz_uvalbedo
+                moz_xtsoil: moz_xtsoil
+                moz_surface_properties: moz_surface_properties 
+
+            add_choose_scenario_type:
+                'PI-CTRL':
+                    add_forcing_files:
+                        moz_lbc: moz_lbc_pictl
+                        moz_sad_sulf: moz_sad_sulf_pictl
+                    add_forcing_in_work:
+                        moz_lbc: "moz_lbc.${echam.resolution}.nc"
+                        moz_sad_sulf: "moz_sad_sulf.${echam.resolution}.nc"
+                'cmip6':
+                    add_forcing_files:
+                        moz_lbc: moz_lbc
+                        moz_sad_sulf: moz_sad_sulf
+                        moz_epp: moz_epp
+                        moz_photolysis: moz_photolysis
+                        moz_ap_index: moz_ap_index
+                    add_forcing_in_work:
+                        moz_epp: "moz_epp.${echam.resolution}${echam.levels}.nc"
+                        moz_photolysis: etfphot_@YEAR@.nc
+                        moz_ap_index: apindex_@YEAR@.nc
+                        moz_lbc: "moz_lbc.${echam.resolution}.nc"
+                        moz_sad_sulf: "moz_sad_sulf.${echam.resolution}.nc"
+
+            add_input_in_work:
+                moz_temp: "moz_temp_prs_gt200nm.nc"
+                moz_uvalbedo: "moz_uvalbedo.${echam.resolution}.nc"
+                moz_xtsoil: "xtsoil.nc"
+                moz_surface_properties: "surface_properties.nc"
+
     bin_files:
             "echam_bin": "echam_bin"
     bin_sources:
@@ -751,6 +852,21 @@ echam:
             - spim
             - ism
 
+    restartstreams:
+            - echam
+            - accw
+            - co2
+            - g3bid
+            - g3bim
+            - g3bday
+            - g3b1hi
+            - glday
+            - aclcim
+            - sp6h
+            - glim
+            - spim
+            - ism
+
     streamsnc:
             - aclcim
             - g3b1hi
@@ -807,23 +923,23 @@ echam:
             #"[[streamsnc-->STREAM]]_nc": ${general.expid}_${start_date!syear!smonth}_STREAM.nc
 
     restart_in_files:
-            "[[streams-->STREAM]]": STREAM
+            "[[restartstreams-->STREAM]]": STREAM
 
     restart_in_sources:
-            "[[streams-->STREAM]]": restart_${parent_expid}_STREAM_${parent_date!syear!smonth!sday}.nc
+            "[[restartstreams-->STREAM]]": restart_${parent_expid}_STREAM_${parent_date!syear!smonth!sday}.nc
 
     restart_in_in_work:
-            "[[streams-->STREAM]]": restart_${general.expid}_STREAM.nc
+            "[[restartstreams-->STREAM]]": restart_${general.expid}_STREAM.nc
 
 
     restart_out_files:
-            "[[streams-->STREAM]]": STREAM
+            "[[restartstreams-->STREAM]]": STREAM
 
     restart_out_sources:
-            "[[streams-->STREAM]]": restart_${general.expid}_STREAM_${other_date!syear!smonth!sday}.nc
+            "[[restartstreams-->STREAM]]": restart_${general.expid}_STREAM_${other_date!syear!smonth!sday}.nc
 
     restart_out_in_work:
-            "[[streams-->STREAM]]": restart_${general.expid}_${other_date!syear!smonth!sday!shour!sminute!ssecond}_STREAM.nc
+            "[[restartstreams-->STREAM]]": restart_${general.expid}_${other_date!syear!smonth!sday!shour!sminute!ssecond}_STREAM.nc
 
     log_files:
             atmout: atmout