Skip to content

Commit aba4bb5

Browse files
authored
Merge pull request #1456 from esm-tools/geomar_prep_release
Merge GEOMAR stuff finally into release
2 parents acafe78 + 98545bf commit aba4bb5

149 files changed

Lines changed: 19276 additions & 1538 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CHANGES.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Changes
2+
3+
## esm_tests: add NAMELIST_PATH and RUNSCRIPT_PATH substitutions for machine-agnostic comparison
4+
5+
**File:** `src/esm_tests/cli.py`
6+
7+
**Problem:** The esm_tools installation path relative to `$HOME` differs between users and machines (e.g. `~/esm_tools` vs `~/Codes/esm_tools`). After `HOME_DIR` substitution, `config_sources` entries like `esm_namelist_dir` still differed between truth files and the current run.
8+
9+
**Fix:** Added `NAMELIST_PATH` and `RUNSCRIPT_PATH` entries to `rm_user_info` (sourced from `esm_tools.get_namelist_filepath()` and `esm_tools.get_runscript_filepath()`) placed **before** `HOME_DIR` in the dict so they are substituted first. Truth files in `last_tested/` must be regenerated on each machine with `esm_tests -s` to adopt the new placeholders.
10+
11+
## esm_tests: fix comp script generation for models with nested pushd paths
12+
13+
**File:** `src/esm_tests/tests.py`
14+
15+
**Root cause:** In check-mode compilation, `comp_test()` creates dummy empty directories to trick `esm_master` into thinking source code is present, allowing it to generate `comp-*_script.sh` files without actually compiling. The folder-collection logic filtered out any `pushd` path containing `/`:
16+
17+
```python
18+
and "/" not in found_format[0]
19+
```
20+
21+
For `foci-mops_lmu`, NEMO's compile step uses a nested path:
22+
23+
```
24+
pushd nemo-ORCA05_LIM2_FOCI_MOPS_OASISMCT4/CONFIG/ORCA05_LIM2_FOCI_MOPS_OASISMCT4
25+
```
26+
27+
Because this path contains `/`, the dummy directory was never created. When `esm_master` then ran without `-c`, it detected the missing folder and aborted — before generating any `comp-*_script.sh` files.
28+
29+
**Fix:**
30+
- Removed the `"/" not in found_format[0]` condition so nested paths are collected.
31+
- Changed `os.mkdir` to `os.makedirs(..., exist_ok=True)` so nested directories are created recursively.
32+
33+
Other models (oasis, echam, xios) were unaffected because their `pushd` paths are all single-level (no `/`). NEMO's FOCI-MOPS configuration requires the nested `CONFIG/` subdirectory.

CONTEXT.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,14 @@
11
# CLAUDE.md
22

3-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
3+
This file provides guidance to coding agents when working with code in this repository.
4+
5+
## Context files
6+
7+
Add these files to memory:
8+
9+
- CONTEXT.md: This file, with the context for coding agents abour the tool. Expand it with relevante information about ESM-Tools is discovered during a claude-code session. Make sure it is still brief and useful for general purpose context of the tools.
10+
- CHANGES.md: summary of the changes carried out in the currend development. To be updated while changes are applied. Changes will be removed at the end of the development, so there might not be a CHANGES.md file. In that case create it.
11+
- Other md files in the root directory might describe a feature or contain a plan. Do a fast search for those files and read them for context.
412

513
## Project Overview
614

@@ -69,7 +77,7 @@ user_error("ErrorType", "message", exit_code=1, hints=["hint1"])
6977

7078
### ESM-Tests (`src/esm_tests/`)
7179

72-
Testing framework for compilation scripts and simulation runs. Reference data lives in a git submodule (`src/esm_tests/resources/``esm_tests_info` repo) with `runscripts/` and `last_tested/` directories.
80+
Testing framework for compilation scripts and simulation runs. Reference data lives in a git submodule (`src/esm_tests/resources/``esm_tests_info` repo) with `runscripts/` and `last_tested/` directories. Machine/user-specific paths are replaced with placeholders (`<TEST_DIR>`, `<NAMELIST_PATH>`, `<RUNSCRIPT_PATH>`, `<HOME_DIR>`, `<USER_ACCOUNT>`) via `rm_user_info` in `cli.py`; order matters — specific paths before `HOME_DIR`.
7381

7482
## Docstring Conventions
7583

configs/components/echam/echam.datasets.yaml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,3 +154,45 @@ echam:
154154
r0008:
155155
greenhouse_hist_name: "greenhouse_ssp245"
156156
MAC-SP_hist_name: "MAC-SP_ssp245"
157+
158+
choose_use_moz:
159+
true:
160+
epp_dir: ${input_dir}/HAMMOZ_${echam.resolution}/epp_T63L95
161+
photolysis_dir: ${input_dir}/HAMMOZ/photolysis
162+
ap_index_dir: ${input_dir}/HAMMOZ/ap_index
163+
164+
add_input_sources:
165+
moz_effxs: "${input_dir}/HAMMOZ/moz_effxs.nc"
166+
moz_rsf: "${input_dir}/HAMMOZ/moz_rsf_gt200nm.nc"
167+
moz_temp: "${input_dir}/HAMMOZ/moz_temp_prs_GT200nm_JPL10_c130206.nc"
168+
moz_xs: "${input_dir}/HAMMOZ/moz_xs_short.nc"
169+
moz_uvalbedo: "${input_dir}/HAMMOZ_${echam.resolution}/moz_uvalbedo_${echam.resolution}.nc"
170+
moz_xtsoil: "${input_dir}/HAMMOZ_${echam.resolution}/soilpHfrac_${echam.resolution}.nc"
171+
moz_surface_properties: "${input_dir}/HAMMOZ_${echam.resolution}/xtsurf_v2_${echam.resolution}.nc"
172+
173+
add_forcing_sources:
174+
# pi control forcing
175+
moz_lbc_pictl: "${input_dir}/HAMMOZ_${echam.resolution}/hammoz_lbc_piControl_CMIP6_CCMI2_${echam.resolution}.nc"
176+
moz_sad_sulf_pictl: "${input_dir}/HAMMOZ_${echam.resolution}/moz_sad_sulf_piControl_c080220_${echam.resolution}${echam.levels}.nc"
177+
# historical and scenario forcing
178+
moz_lbc:
179+
"${input_dir}/HAMMOZ_${echam.resolution}/hammoz_lbc_17500116-21011216_CMIP6_CCMI2_ssp370_${echam.resolution}.nc":
180+
to: 2014
181+
"${input_dir}/HAMMOZ_${echam.resolution}/hammoz_lbc_17500116-21011216_CMIP6_CCMI2_${echam.scenario}_${echam.resolution}.nc":
182+
from: 2015
183+
moz_sad_sulf: "${input_dir}/HAMMOZ_${echam.resolution}/moz_sad_sulf_1849-2100_c080220_${echam.resolution}${echam.levels}.nc"
184+
moz_epp:
185+
"${epp_dir}/ubc_noy_epp_@YEAR@_${echam.resolution}${echam.levels}_20toplevs.nc":
186+
from: 1849
187+
to: 2100
188+
moz_photolysis:
189+
"${photolysis_dir}/etfphot_@YEAR@.nc":
190+
from: 1849
191+
to: 2100
192+
moz_ap_index:
193+
"${ap_index_dir}/apindex_1850.nc":
194+
to: 1849
195+
"${ap_index_dir}/apindex_@YEAR@.nc":
196+
from: 1850
197+
to: 2100
198+

configs/components/echam/echam.yaml

Lines changed: 125 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,10 @@ echam:
99
available_versions:
1010
- 6.3.05p2
1111
- 6.3.05p2-foci
12+
- 6.3.05p2-moz
13+
- 6.3.05p2-moz_mct2
1214
- 6.3.05p2-foci_autotools
15+
- 6.3.05p2-foci_lmu
1316
- 6.3.05p2-foci_oasismct4
1417
- 6.3.02p4
1518
- 6.3.04p1-esm_interface
@@ -73,6 +76,36 @@ echam:
7376
6.3.05p2-foci:
7477
branch: esm-tools
7578
git-repository: https://git.geomar.de/foci/src/echam.git
79+
6.3.05p2-moz:
80+
branch: oasis-mct5
81+
git-repository: https://git.geomar.de/foci/src/echam_hammoz.git
82+
clean_command: rm -rf src/echam/bin; rm -rf bin; make clean; make distclean
83+
comp_command: mkdir -p src/.deps yaxt/src/.deps yaxt/tests/.deps;
84+
./config/createMakefiles.pl; autoreconf -i --force; unset FCFLAGS; unset FFLAGS;
85+
./configure --prefix=$(pwd)/../ --with-fortran=intel --with-coupler=oasis3-mct-nemo --enable-cdi-pio OASIS3MCTROOT=$(pwd)/../oasis/INSTALL_OASIS.ESMTOOLS/ SUPPORTROOT=$(pwd)/../ SUPPORT_LIB='-L$(SUPPORTROOT)/lib -lsupport' SRCDIRS="support cdi" ;
86+
make -j `nproc --all`; make install -j `nproc --all`;
87+
install_bins: ''
88+
6.3.05p2-moz_mct2:
89+
branch: master
90+
git-repository: https://git.geomar.de/foci/src/echam_hammoz.git
91+
clean_command: rm -rf src/echam/bin; rm -rf bin; make clean; make distclean
92+
comp_command: mkdir -p src/.deps yaxt/src/.deps yaxt/tests/.deps;
93+
./config/createMakefiles.pl; autoreconf -i --force; unset FCFLAGS; unset FFLAGS;
94+
./configure --prefix=$(pwd)/../ --with-fortran=intel --with-coupler=oasis3-mct-nemo --enable-cdi-pio OASIS3MCTROOT=$(pwd)/../ SUPPORTROOT=$(pwd)/../ SUPPORT_LIB='-L$(SUPPORTROOT)/lib -lsupport' SRCDIRS="support cdi" ;
95+
make -j `nproc --all`; make install -j `nproc --all`;
96+
install_bins: ''
97+
6.3.05p2-foci_lmu:
98+
git-repository: https://gitlab.dkrz.de/lmu-lus/mpiesm/hbp_mpiesm-landveg.git
99+
branch: feature/couple_to_nemo3.6_mops
100+
clean_command: cd src/echam; rm -rf src/echam/bin; rm -rf bin; make clean; make distclean
101+
comp_command: cd src/echam;
102+
git submodule init; git submodule update cdi || echo "No update required";
103+
mkdir -p src/.deps yaxt/src/.deps yaxt/tests/.deps;
104+
./config/createMakefiles.pl; autoreconf -i --force;
105+
unset FCFLAGS; unset FFLAGS;
106+
./configure --prefix=$(pwd)/../../../ --with-fortran=intel --with-coupler=oasis3-mct-nemo --enable-cdi-pio OASIS3MCTROOT=$(pwd)/../../../oasis/INSTALL_OASIS.ESMTOOLS/ SUPPORTROOT=$(pwd)/../ SUPPORT_LIB='-L$(SUPPORTROOT)/lib -lsupport' SRCDIRS="support cdi" ;
107+
make -j `nproc --all`; make install -j `nproc --all`;
108+
install_bins: ''
76109
6.3.05p2-foci_autotools:
77110
clean_command: rm -rf src/echam/bin; rm -rf bin; make clean; make distclean
78111
comp_command: mkdir -p src/.deps yaxt/src/.deps yaxt/tests/.deps;
@@ -128,7 +161,7 @@ echam:
128161
ocean_resolution: 'GR15'
129162
disturbance: "1.000001"
130163
lresume: false
131-
164+
use_moz: false
132165

133166
lgfw: False
134167
lupdate_orography: False
@@ -270,12 +303,33 @@ echam:
270303
dataset: r0008
271304
nprocrad: "remove_from_namelist"
272305
lrad_async: "remove_from_namelist"
306+
6.3.05p2-foci_autotools:
307+
repo_tag: not_set
308+
dataset: r0008
309+
nprocrad: "remove_from_namelist"
310+
lrad_async: "remove_from_namelist"
311+
6.3.05p2-moz:
312+
repo_tag: not_set
313+
dataset: r0008
314+
nprocrad: "remove_from_namelist"
315+
lrad_async: "remove_from_namelist"
316+
use_moz: true
317+
6.3.05p2-moz_mct2:
318+
repo_tag: not_set
319+
dataset: r0008
320+
nprocrad: "remove_from_namelist"
321+
lrad_async: "remove_from_namelist"
322+
use_moz: true
273323
6.3.05p2-foci_oasismct4:
274324
repo_tag: not_set
275325
dataset: r0008
276326
nprocrad: "remove_from_namelist"
277327
lrad_async: "remove_from_namelist"
278-
328+
6.3.05p2-foci_lmu:
329+
repo_tag: not_set
330+
dataset: r0008
331+
nprocrad: "remove_from_namelist"
332+
lrad_async: "remove_from_namelist"
279333
6.3.05p2-concurrent_radiation-paleodyn:
280334
concurrent_radiation_code: true
281335
repo_tag: not_applicable
@@ -333,6 +387,8 @@ echam:
333387
lrad_async: "${lrad_async}"
334388
add_streams:
335389
- rad
390+
add_restartstreams:
391+
- rad
336392
"*":
337393
from_concurrent_radiation_code_choose: <NO_CONCURRENT_RADIATION_CODE>
338394

@@ -497,7 +553,7 @@ echam:
497553
choose_lresume:
498554
False:
499555
restart_in_modifications:
500-
"[[streams-->STREAM]]":
556+
"[[restartstreams-->STREAM]]":
501557
- "vdate <--set_global_attr-- ${start_date!syear!smonth!sday}"
502558
# - fdate "<--set_dim--" ${year_before_date}
503559
# - ndate "<--set_dim--" ${steps_in_year_before}
@@ -574,6 +630,9 @@ echam:
574630
lco2_emis: true # read carbon emissions; need carbon_emissions.nc in work
575631
lco2_2perc: true # limit maximum co2 growth rate to 2% wrt previous time step
576632
add_choose_scenario: # other loop order (1:add_forcing_files 2:add_choose_scenario) does not work
633+
cmip6hist:
634+
add_forcing_files:
635+
carbon_emissions: carbon_emissions_hist
577636
HIST:
578637
add_forcing_files:
579638
carbon_emissions: carbon_emissions_hist
@@ -626,6 +685,9 @@ echam:
626685
add_streams:
627686
- wiso
628687
- accw_wiso
688+
add_restartstreams:
689+
- wiso
690+
- accw_wiso
629691

630692
# The following lines take care of the renaming of the
631693
# ozon files, and their copying
@@ -642,6 +704,45 @@ echam:
642704
nwiso: 0
643705
lwiso_rerun: False
644706

707+
choose_use_moz:
708+
true:
709+
add_input_files:
710+
moz_effxs: moz_effxs
711+
moz_rsf: moz_rsf
712+
moz_temp: moz_temp
713+
moz_xs: moz_xs
714+
moz_uvalbedo: moz_uvalbedo
715+
moz_xtsoil: moz_xtsoil
716+
moz_surface_properties: moz_surface_properties
717+
718+
add_choose_scenario_type:
719+
'PI-CTRL':
720+
add_forcing_files:
721+
moz_lbc: moz_lbc_pictl
722+
moz_sad_sulf: moz_sad_sulf_pictl
723+
add_forcing_in_work:
724+
moz_lbc: "moz_lbc.${echam.resolution}.nc"
725+
moz_sad_sulf: "moz_sad_sulf.${echam.resolution}.nc"
726+
'cmip6':
727+
add_forcing_files:
728+
moz_lbc: moz_lbc
729+
moz_sad_sulf: moz_sad_sulf
730+
moz_epp: moz_epp
731+
moz_photolysis: moz_photolysis
732+
moz_ap_index: moz_ap_index
733+
add_forcing_in_work:
734+
moz_epp: "moz_epp.${echam.resolution}${echam.levels}.nc"
735+
moz_photolysis: etfphot_@YEAR@.nc
736+
moz_ap_index: apindex_@YEAR@.nc
737+
moz_lbc: "moz_lbc.${echam.resolution}.nc"
738+
moz_sad_sulf: "moz_sad_sulf.${echam.resolution}.nc"
739+
740+
add_input_in_work:
741+
moz_temp: "moz_temp_prs_gt200nm.nc"
742+
moz_uvalbedo: "moz_uvalbedo.${echam.resolution}.nc"
743+
moz_xtsoil: "xtsoil.nc"
744+
moz_surface_properties: "surface_properties.nc"
745+
645746
bin_files:
646747
"echam_bin": "echam_bin"
647748
bin_sources:
@@ -751,6 +852,21 @@ echam:
751852
- spim
752853
- ism
753854

855+
restartstreams:
856+
- echam
857+
- accw
858+
- co2
859+
- g3bid
860+
- g3bim
861+
- g3bday
862+
- g3b1hi
863+
- glday
864+
- aclcim
865+
- sp6h
866+
- glim
867+
- spim
868+
- ism
869+
754870
streamsnc:
755871
- aclcim
756872
- g3b1hi
@@ -807,23 +923,23 @@ echam:
807923
#"[[streamsnc-->STREAM]]_nc": ${general.expid}_${start_date!syear!smonth}_STREAM.nc
808924

809925
restart_in_files:
810-
"[[streams-->STREAM]]": STREAM
926+
"[[restartstreams-->STREAM]]": STREAM
811927

812928
restart_in_sources:
813-
"[[streams-->STREAM]]": restart_${parent_expid}_STREAM_${parent_date!syear!smonth!sday}.nc
929+
"[[restartstreams-->STREAM]]": restart_${parent_expid}_STREAM_${parent_date!syear!smonth!sday}.nc
814930

815931
restart_in_in_work:
816-
"[[streams-->STREAM]]": restart_${general.expid}_STREAM.nc
932+
"[[restartstreams-->STREAM]]": restart_${general.expid}_STREAM.nc
817933

818934

819935
restart_out_files:
820-
"[[streams-->STREAM]]": STREAM
936+
"[[restartstreams-->STREAM]]": STREAM
821937

822938
restart_out_sources:
823-
"[[streams-->STREAM]]": restart_${general.expid}_STREAM_${other_date!syear!smonth!sday}.nc
939+
"[[restartstreams-->STREAM]]": restart_${general.expid}_STREAM_${other_date!syear!smonth!sday}.nc
824940

825941
restart_out_in_work:
826-
"[[streams-->STREAM]]": restart_${general.expid}_${other_date!syear!smonth!sday!shour!sminute!ssecond}_STREAM.nc
942+
"[[restartstreams-->STREAM]]": restart_${general.expid}_${other_date!syear!smonth!sday!shour!sminute!ssecond}_STREAM.nc
827943

828944
log_files:
829945
atmout: atmout

0 commit comments

Comments
 (0)