AWI-CM3 Release Evaluation Tool

A comprehensive climate model evaluation framework for analyzing AWI-CM3 (AWI Climate Model version 3) simulations. This tool provides automated diagnostics, comparisons with observations, and publication-quality visualizations for model spinup, preindustrial control, and historical experiments.

Overview

The Release Evaluation Tool (reval) is designed to streamline the assessment of AWI-CM3 model performance through:

Parallel execution of 20+ diagnostic scripts via SLURM
Multi-component analysis (atmosphere, ocean, sea ice, land vegetation)
Observation comparisons against ERA5, CERES, MODIS, GPCP, HadISST, and more
Publication-ready plots with consistent formatting and metadata

Originally developed for Streffing et al. (2022, GMD) and extended for Moon et al. (2025, ESD).

Quick Start

Prerequisites

Conda environment with required packages (see environment.yaml)
Access to AWI-CM3 model output data
SLURM cluster for parallel job submission

Setup

# Clone repository
cd /work/ab0246/a270092/software/release_evaluation_tool2

# Activate environment
source ~/loadconda.sh
conda activate reval

# Submit all enabled scripts with a config file
python reval.py -c configs/AWI-CM3-v3.3.py

Monitor Progress

# Check job status
squeue -u $USER

# Monitor logs
tail -f logs/<script_name>.log

# View completion status
cat logs/status.csv

Directory Structure

release_evaluation_tool2/
├── README.md              # This file
├── reval.py               # Main submission script
├── environment.yaml       # Conda dependencies
│
├── configs/               # Experiment-specific configurations
│   ├── AWI-CM3-v3.3.py
│   ├── CORE3_tuning.py
│   ├── HR_tuning.py
│   ├── lpjg_spinup_200y_16c.py
│   └── ...
│
├── scripts/               # Analysis scripts (part##_name.py)
│   ├── part1_mesh_plot.py
│   ├── part2_rad_balance.py
│   ├── part8_t2m_vs_era5.py
│   ├── part23_ice_cavity_velocities.py
│   └── ...
│
├── bg_routines/           # Helper functions (do not modify)
│   ├── config_loader.py
│   ├── update_status.py
│   ├── sub_fesom_mesh.py
│   ├── sub_fesom_data.py
│   └── ...
│
├── logs/                  # SLURM output logs
│   ├── status.csv         # Completion tracking
│   └── part##_*.log       # Individual script logs
│
├── output/                # Generated plots (organized by model_version)
│   └── <model_version>/
│
└── tmp/                   # Temporary SLURM scripts

Configuration System

Configuration is passed via the -c flag when running reval.py. Users only need to create or edit files in the configs/ directory. The config loader in bg_routines/ should not be modified.

python reval.py -c configs/AWI-CM3-v3.3.py

Experiment Configs (`configs/*.py`)

Define simulation-specific parameters:

# Model identification
model_version = 'AWI-ESM3.4_rc3_HR'
oasis_oifs_grid_name = 'A320'

# Spinup experiment
spinup_path  = '/path/to/spinup/outdata/'
spinup_start = 1354
spinup_end   = 1359

# Preindustrial control
pi_ctrl_path  = '/path/to/pi-control/outdata/'
pi_ctrl_start = 1400
pi_ctrl_end   = 1500

# Historical experiment
historic_path  = '/path/to/historical/outdata/'
historic_start = 1850
historic_end   = 2014

# Mesh configuration
mesh_name = 'DARS2'      # FESOM2 mesh
grid_name = 'TCo319'     # OpenIFS grid
meshpath  = '/path/to/mesh/'

# Analysis settings
reanalysis = 'ERA5'
dpi = 300

Key Variables:

model_version: Identifier for output directory
*_path: Base paths to simulation outdata
*_start, *_end: Year ranges for each experiment
historic_last25y_start/end: Period for climatology (auto-calculated)
mesh_name, grid_name: Model component identifiers
out_path: Auto-generated output directory

Analysis Scripts

Naming Convention

Scripts follow part##_description.py format and are executed in natural sort order.

Script Structure

Each script follows this template:

import sys, os
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
from bg_routines.config_loader import *

SCRIPT_NAME = os.path.basename(__file__)
update_status(SCRIPT_NAME, " Started")

try:
    # Analysis code here
    # - Load data from *_path/component/*.nc
    # - Process using CDO, xarray, pyfesom2
    # - Generate plots to out_path
    
    update_status(SCRIPT_NAME, " Completed")
except Exception as e:
    update_status(SCRIPT_NAME, f" Failed: {e}")
    raise

Workflow

1. Configure Experiment

Create or choose a config file in configs/ with your simulation paths and settings.

2. Enable Scripts

Edit reval.py to enable/disable specific analyses:

SCRIPTS.update({
    "part1_mesh_plot.py":    True,
    "part8_t2m_vs_era5.py":  True,
    "part15_enso.py":        False,  # Skip this one
    ...
})

3. Submit Jobs

python reval.py -c configs/AWI-CM3-v3.3.py

This creates individual SLURM scripts (slurm_part##_*.sh) and submits them to the compute partition.

4. Monitor Execution

Job queue: squeue -u $USER
Logs: tail -f logs/part##_*.log
Status: cat logs/status.csv
Output: ls output/<model_version>/

5. Review Results

All plots are saved to output/<model_version>/ with descriptive filenames:

t2m_vs_ERA5.png
ice_cavity_Ross_panels.png
HIST_Nino34_enso_box_index.png
etc.

SLURM Configuration

Default settings in reval.py:

#SBATCH --time=00:20:00       # 20 minutes per script
#SBATCH --mem=256G            # 256 GB memory
#SBATCH --ntasks=128          # 128 cores
#SBATCH --partition=compute
#SBATCH -A ab0246             # Account

Adjust these for specific scripts if needed (e.g., longer time for ocean sections).

Data Requirements

Model Output Structure

Expected data organization:

<experiment_path>/
├── fesom/
│   ├── temp.fesom.YYYY.nc
│   ├── salt.fesom.YYYY.nc
│   ├── u.fesom.YYYY.nc
│   └── ...
├── oifs/
│   ├── atm_remapped_1d_2t_1d_YYYY-YYYY.nc
│   ├── atm_remapped_1m_lcc_1m_YYYY-YYYY.nc
│   └── ...
└── lpj_guess/ (if using LPJ-GUESS)
    ├── run1/
    │   ├── lai.out
    │   ├── cmass.out
    │   └── ...
    └── run2/
        └── ...

Observation Data

Stored in observation_path (default: /work/ab0246/a270092/obs/):

ERA5 reanalysis
CERES radiation products
MODIS cloud cover
GPCP precipitation
HadISST sea surface temperature
PHC3 ocean climatology

Troubleshooting

Common Issues

Script shows "Started" but not "Completed"

Check log file: cat logs/part##_*.log
Common causes: missing data files, insufficient memory, timeout

"No such file or directory" errors

Verify paths in config match actual data location
Check year ranges - data must exist for all years in *_start to *_end
Ensure FESOM/OpenIFS output files follow expected naming conventions

LPJ-GUESS scripts fail

Confirm LPJ-GUESS data exists for requested years
Check that data follows outdata/lpj_guess/run*/lai.out structure
Note: LPJ-GUESS years may differ from ESM years (e.g., 6734-9999 vs 1354-1359)

Memory errors

Increase --mem in SLURM settings for data-heavy scripts
Consider processing fewer years or coarser temporal resolution

Plot quality issues

Adjust dpi in config (default: 300)
Check colormap limits in individual scripts
Verify projection settings for polar plots

Key Dependencies

Python 3.12+
PyFESOM2: FESOM2 mesh and data handling
xarray: NetCDF data structures
CDO (Python bindings): Climate Data Operators
matplotlib, cartopy: Plotting and map projections
cmocean: Oceanographic colormaps
dask: Parallel computation
scipy, scikit-learn: Scientific computing
tqdm: Progress bars

See environment.yaml for complete list.

Output Examples

Generated plots include:

Global maps: Temperature, precipitation, radiation biases
Timeseries: Sea ice extent, ENSO indices
Cross-sections: Ocean temperature/salinity transects
Hovmöller diagrams: Spatiotemporal evolution
Multi-panel figures: Comprehensive cavity circulation analysis
Statistical plots: Gregory plots, power spectra, EOFs

All plots include:

Model version and year range in title
Colorbars with units
Coastlines and grid references
High resolution (300 DPI) for publication

Citation

If you use this tool, please cite:

Streffing, J., Sidorenko, D., Semmler, T., Zampieri, L., Scholz, P., Andrés-Martínez, M., Koldunov, N., Rackow, T., Kjellsson, J., Goessling, H., Athanase, M., Wang, Q., Hegewald, J., Sein, D. V., Mu, L., Hinrichs, C., Kluft, L., Danilov, S., Jungclaus, J., and Jung, T.: AWI-CM3 model output prepared for GMD paper "AWI-CM3 coupled climate model: Description and evaluation", https://doi.org/10.5194/gmd-15-6399-2022, 2022.

Support

For questions or issues:

Check logs in logs/ directory
Review data paths in experiment configs under configs/
Consult script-specific comments in scripts/
Contact: Jan Streffing (jan.streffing@awi.de)

Version History

2025-02-04: Refactored for parallel script execution (v2)
2024-04-03: Added significance metrics for EGUSphere paper
2021-12-10: Initial Jupyter notebook version for GMD paper

Name		Name	Last commit message	Last commit date
Latest commit History 61 Commits
bg_routines		bg_routines
configs		configs
preprocessing_examples		preprocessing_examples
scripts		scripts
.gitignore		.gitignore
.gitmodules		.gitmodules
LICENSE		LICENSE
README.md		README.md
__init__.py		__init__.py
environment.yaml		environment.yaml
reval.py		reval.py

Folders and files

Latest commit

History

Repository files navigation

AWI-CM3 Release Evaluation Tool

Overview

Quick Start

Prerequisites

Setup

Monitor Progress

Directory Structure

Configuration System

Experiment Configs (configs/*.py)

Analysis Scripts

Naming Convention

Script Structure

Categories

Workflow

1. Configure Experiment

2. Enable Scripts

3. Submit Jobs

4. Monitor Execution

5. Review Results

SLURM Configuration

Data Requirements

Model Output Structure

Observation Data

Troubleshooting

Common Issues

Key Dependencies

Output Examples

Citation

Support

Version History

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Experiment Configs (`configs/*.py`)

Packages