NorESM-FATES crashes with water balance error in historic simulation with land use

[JessicaNeedham]

Brief summary of bug

NorESM-FATES crashes with water balance error in historic simulation with land use and rangeland clearing.

General bug information

CTSM version you are using:
ctsm5.3.045_noresm_v15-1-g918e2fab1

FATES
sci.1.85.1_api.40.0.0_nor_sci2_api1-4-g101a06c1 - branched from FATES noresm branch commit 3e71719 and changed rangeland clearing logic from default 4 to option 1 here: https://github.com/NorESMhub/fates/blob/3e71719091aa5ab34a09c64eb2f68e0094768af8/biogeochem/FatesLandUseChangeMod.F90#L185

Does this bug cause significantly incorrect results in the model's science? [Yes / No]

Configurations affected:
Not sure if this is caused by the land use clearing logic change or if this also happens with the default option of 4.

Details of bug

Triggered this error:

CTSM/src/biogeophys/BalanceCheckMod.F90

Line 705 in 918e2fa

call endrun(subgrid_index=indexc, subgrid_level=subgrid_level_column, msg=errmsg(sourcefile, __LINE__))

cesm.log output:

515:  WARNING:  column-level water balance error  nstep=        55739  local indexc=
 515:            1  global indexc=         1913  errh2o=   5.643917675279003E-003
 515:  CTSM is stopping because errh2o >   1.000000000000000E-005  mm
 515:  nstep                     =        55739
 515:  errh2o_col                =   5.643917675279003E-003
 515:  forc_rain                 =   0.000000000000000E+000
 515:  forc_snow                 =   0.000000000000000E+000
 515:  endwb_col                 =    2066.77913737819
 515:  begwb_col                 =    2066.78187897203
 515:  qflx_evap_tot             =   8.385511523719974E-003
 515:  qflx_sfc_irrig            =   0.000000000000000E+000
 515:  qflx_surf                 =   0.000000000000000E+000
 515:  qflx_qrgwl                =   0.000000000000000E+000
 515:  qflx_drain                =   0.000000000000000E+000
 515:  qflx_ice_runoff           =   0.000000000000000E+000
 515:  qflx_snwcp_discarded_ice  =   0.000000000000000E+000
 515:  qflx_snwcp_discarded_liq  =   0.000000000000000E+000
 515:  deltawb                   =  -2.741593848440971E-003
 515:  deltawb/dtime             =  -1.523107693578317E-006
 515:  qflx_drain_perched         =   0.000000000000000E+000
 515:  qflx_flood                 =   0.000000000000000E+000
 515:  qflx_glcice_dyn_water_flux =   0.000000000000000E+000
 515:  CTSM is stopping
 515: iam = 387: local  column   index = 1
 515: iam = 387: global column   index = 1913
 515: iam = 387: global landunit index = 725
 515: iam = 387: global gridcell index = 388
 515: iam = 387: gridcell longitude    =   27.1875000
 515: iam = 387: gridcell latitude     =    2.5021834
 515: iam = 387: column   type         = 1
 515: iam = 387: landunit type         = 1
 515:  ENDRUN:
 515:  ERROR in BalanceCheckMod.F90 at line 705

Important details of your setup / configuration so we can reproduce the bug

Case is here:

/cluster/work/users/jessica/fates-cal-runs/noresm-fates_ne16_1850_transient_rangeland_clearing.2025-11-14

fates_paramfile = '/cluster/shared/noresm/inputdata/lnd/clm2/paramdata/fates_params_sci.1.85.1_api.40.0.0_14pft_nor_sci1_api1_c251031.nc'

COMPSET='HIST_DATM%CRUJRA2024_CLM60%FATES_SICE_SOCN_SROF_SGLC_SWAV_SESP'
RES="ne16pg3_tn14"
MACH='betzy'

./xmlchange RUN_STARTDATE=1851-01-01
./xmlchange CLM_ACCELERATED_SPINUP=off
./xmlchange DATM_YR_START=1901
./xmlchange DATM_YR_END=1920
./xmlchange DATM_YR_ALIGN=1851
./xmlchange DATM_PRESAERO=hist
./xmlchange CLM_CO2_TYPE=diagnostic
./xmlchange DATM_CO2_TSERIES=20tr
./xmlchange CCSM_BGC=CO2A

user_nl_clm changes:

do_transient_lakes=.false.                                                                                                                                   
do_transient_urban=.false.                                                                                                                                   
finidat='/cluster/work/users/jessica/fates-cal-runs/noresm-fates_ne16_1850_steadystate_spinup.2025-11-07/run/noresm-fates_ne16_1850_steadystate_spinup.2025-\
11-07.clm2.r.0201-01-01-00000.nc'                                                                                                                            
paramfile='/cluster/shared/noresm/inputdata/lnd/clm2/paramdata/ctsm60_params.5.3.045_noresm_v14_c251031.nc'                                                  
use_fates_sp=.false.                                                                                                                                         
use_fates_nocomp=.true.                                                                                                                                      
use_fates_fixed_biogeog=.true.                                                                                                                               
fates_stomatal_model='medlyn2011'                                                                                                                            
fates_radiation_model = 'twostream'                                                                                                                          
fates_leafresp_model = 'ryan1991'                                                                                                                            
use_fates_luh=.true.                                                                                                                                         
use_fates_lupft=.true.                                                                                                                                       
fates_harvest_mode='luhdata_area'                                                                                                                            
use_fates_potentialveg=.false.                                                                                                                               
fluh_timeseries='/cluster/shared/noresm/inputdata/LU_data_CMIP7/LUH2_timeseries_to_surfdata_ne16np4_251106_cdf5.nc'                                          
flandusepftdat='/cluster/shared/noresm/inputdata/LU_data_CMIP7/fates_landuse_pft_map_to_surfdata_ne16np4_251106_cdf5.nc'                                     
fates_spitfire_mode=4

[JessicaNeedham]

The same error is triggered with the default land use settings.

Case:
/cluster/work/users/jessica/fates-cal-runs/noresm-fates_ne16_1850_transient_test.2025-11-15

FATES branch noresm sci.1.85.1_api.40.0.0_nor_sci2_api1 commit 3e717190.

[rosiealice]

Do you have a script that make the case that fails?

[JessicaNeedham]

Yes, @rosiealice the runscript is here:
https://github.com/JessicaNeedham/NorESM-FATES_runscripts/blob/master/create_fates_ne16_1850_transient.sh

The spinup was the PPE scramble branch which had these endrun statements commented out. The cases that crash are using the restart from that spinup, but on the current noresm tag where these are now uncommented.

[JessicaNeedham]

@mvdebolskiy - just met with @rosiealice and @maritsandstad and we thought we should comment these lines back out until we have fixed this to prevent causing problems in any coupled runs people are starting this week. It is line 705 and 790 in BalanceCheckMod. Let me know if you want me to make a PR.

CTSM/src/biogeophys/BalanceCheckMod.F90

Line 705 in 918e2fa

call endrun(subgrid_index=indexc, subgrid_level=subgrid_level_column, msg=errmsg(sourcefile, __LINE__))

CTSM/src/biogeophys/BalanceCheckMod.F90

Line 790 in 918e2fa

call endrun(subgrid_index=indexg, subgrid_level=subgrid_level_gridcell, msg=errmsg(sourcefile, __LINE__))

[mvdebolskiy]

I really do not like the idea of commenting out legit endruns. But if you want, sure, do this but someone else has to approve such a PR.

Also, just rerun the spinup and see if you hit this endrun.

[maritsandstad]

It is a problem if we crash the whole model over and over because of some weirdness in a single gridbox though, but I agree we should try to find the root cause and fix it...

[JessicaNeedham]

I re-ran the spinup for 75 years using the new tags, then switched to a transient run with land use and triggered the endrun again.

Case is here:
/cluster/work/users/jessica/fates-cal-runs/ne16_1850_transient.2025-11-19

Spinup is here:
/cluster/work/users/jessica/fates-cal-runs/noresm-fates_ne16_1850_steadystate_spinup.2025-11-18