Working toward running mizuRoute in fully coupled simulations

[ekluzek]

A project needed for water isotopes and our plan to deprecate MOSART and RTM in CESM is to enable running mizuRoute coupled to the MOM ocean model. Beyond enabling it we need to make sure the results for fully coupled simulations are reasonable and don't cause problems in coupled simulations compared to using MOSART.

We have some notes on this in the wiki. But, starting this discussion to work out things that need to be done for this to happen. I'll point to some of the notes in a reply to this main message. Also we'll have a reply that lays out a checklist of things that need to be done.

[ekluzek]

Previous meeting notes:

Meeting with Dave

https://github.com/ESCOMP/mizuRoute/wiki/Meeting-notes:--ctsm-mizuRoute#october-15th-2025-meeting-with-dave

Antarctic network:

https://github.com/ESCOMP/mizuRoute/wiki/Meeting-notes:--ctsm-mizuRoute#aug-21st-2024

Notes from meeting with Mike:

https://github.com/ESCOMP/mizuRoute/wiki/Meeting-notes:--ctsm-mizuRoute#aug-2nd-2023-with-sean-and-mike-levy

[ekluzek]

Task list of things to develop:

• cesm branch/fork for running coupled:

  • Custom sandbox
  • CESM branch -- Erik make a branch that can be shared
  • CTSM branch

• Start with the MOSART grid for mizuRoute for I compsets compare to MOSART

  • Replicate what MOSART does in mizuRoute: f09_t232.I2000Clm60SpMizGs.derecho_intel vs f09_t232.I2000Clm60Sp.derecho_intel
  • A difference is mizuRoute doesn't have HRU's over ocean. We think this might be OK, because MOSART probably only sends to ocean over land.

• Test Existing Coupled compsets:
  We thought there might be issues with the PE layout, but it looks like there won't be.

  • Fully coupled (starting in 1850): (B-case)
    • MOSART (use alias): BHISTC_LTso
    • mizuRoute (use long name): HISTC_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MIZUROUTE_DGLC%NOEVOLVE_WW3
    • grid: ne30pg3_t232
    • run both for 5 days
  • Fully coupled (starting in 1850) - same as the above except for 5months:
  • Coupled with CAM/DOCN: (F-case - subset of B-case)
    • MOSART: FHISTC_LTso
    • mizuRoute: 2000_CAM70_CLM50%SP_CICE%PRES_DOCN%DOM_MIZUROUTE_SGLC_SWAV
    • grid: ne30pg3_ne30pg3_mt232, ne16pg3_ne16pg3_mt232, ne3pg2_ne3pg3_mt232
  • Slab Ocean Model: (E-case)
    • MOSART: ETEST
    • mizuRoute:2000_CAM70_CLM50%SP_CICE_DOCN%SOM_MIZUROUTE_SGLC_SWAV
    • grid: f19_f19_mt232, ne30pg3_ne30pg3_mt232, ne30pg3_t232

• Try some custom setups:

  • CAM with MOM setup (CICE prescribed, stub wave): 2000_CAM70_CLM60%SP_CICE%PRES_MOM6_MIZUROUTE_DGLC%NOEVOLVE_SWAV

• Develop some custom setups: (we don't have these now, but good to have)

  • DATM for CLM and MOM same forcing
  • DATM for CLM and MOM different forcing from different tasks for each

[nmizukami]

Erik created a CESM branch to test mizuRoute in fully coupled model - github.com:ekluzek/test_mizuroute

git clone --origin ekluzek -b test_mizuroute [email protected]:ekluzek/CESM.git cesm_mizuRoute

Next step: try I compset to make sure CTSM-mizuRoute/mosart coupling works

[ekluzek]

The initial version of the branch was pointing to a hash that didn't exist. But, it's been updated and is now working.

[ekluzek]

Some issues that apply here:

#526
#469
#183
#153

[nmizukami]

Status as of Nov 7 2025:

Fully coupled
MOSART (use alias): BHISTC_LTso
mizuRoute (use long name): HISTC_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MIZUROUTE_DGLC%NOEVOLVE_WW3
grid: ne30pg3_t232

case created and bulit for both MOSART and mizuRoute, but not run yet.

[nmizukami]

Failed HISTC_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MIZUROUTE_DGLC%NOEVOLVE_WW3
grid: ne30pg3_t232 (mizuRoute cesm test)

Comparison of med.log file between BHISTC_LTso (mosart cesm test) and mizuRoute cesm test.

mosart cesm test run directory: glade/derecho/scratch/mizukami/SMS_D.ne30pg3_t232.BHISTC_LTso.derecho_intel.20251106_141101_firstd/run

grep -i -w rof med.log.3578963.desched1.251110-095930

 (internalstate init) rof model= mosart
  |from to -med  atm  lnd  ocn  ice  rof  wav  glc1 
  |rof         -    -    T    T    T    -    -    -
  |from to -med  atm  lnd  ocn  ice  rof  wav  glc1 
  |rof         -    -    T    T    T    -    -    -
(med.F90:DataInitialize) initializing FBs for rof
 mapping lnd->rof Flrl_rofsur via consf with lfrac normalization
 mapping lnd->rof Flrl_rofi via consf with lfrac normalization
 mapping lnd->rof Flrl_rofgwl via consf with lfrac normalization
 mapping lnd->rof Flrl_rofsub via consf with lfrac normalization
 mapping lnd->rof Flrl_irrig via consf with lfrac normalization
 mapping rof->lnd Flrr_volr via consf with one normalization
 mapping rof->lnd Flrr_volrmch via consf with one normalization
 mapping rof->lnd Flrr_flood via consf with one normalization
 mapping rof->ocn Flrr_flood via consd with one normalization
 mapping rof->ocn Forr_rofl via rof2ocn_liq with none normalization
 mapping rof->ocn Forr_rofi via rof2ocn_ice with none normalization
 mapping rof->ocn Forr_rofl_glc via rof2ocn_liq with none normalization
 mapping rof->ocn Forr_rofi_glc via rof2ocn_ice with none normalization
 mapping glc1->rof Fgrg_rofi via consd with gfrac normalization
(merge_to_lnd): Flrr_volr(lnd) = Flrr_volr(rof)
(merge_to_lnd): Flrr_volrmch(lnd) = Flrr_volrmch(rof)
(merge_to_lnd): Flrr_flood(lnd) = Flrr_flood(rof)
(merge_to_ocn): Foxx_rofl(ocn) = Forr_rofl:Flrr_flood(rof)
(merge_to_ocn): Foxx_rofi(ocn) = Forr_rofi(rof)
(merge_to_ocn): Forr_rofl_glc(ocn) = Forr_rofl_glc(rof)
(merge_to_ocn): Forr_rofi_glc(ocn) = Forr_rofi_glc(rof)
(merge_to_rof): Flrl_rofsur(rof) = lfrac*Flrl_rofsur(lnd)
(merge_to_rof): Flrl_rofi(rof) = lfrac*Flrl_rofi(lnd)
(merge_to_rof): Flrl_rofgwl(rof) = lfrac*Flrl_rofgwl(lnd)
(merge_to_rof): Flrl_rofsub(rof) = lfrac*Flrl_rofsub(lnd)
(merge_to_rof): Flrl_irrig(rof) = lfrac*Flrl_irrig(lnd)
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consf for lnd to rof srcMask =    -987987 dstMask =    -987987
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consf for rof to lnd srcMask =    -987987 dstMask =    -987987
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consd for rof to ocn srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH rof2ocn_liq via input file /glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/r05/map_r05_to_tx2_3_nnsm_e250r250_230914.nc for rof to ocn srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH rof2ocn_ice via input file /glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/r05/map_r05_to_tx2_3_nnsm_e1000r1000_230415.nc for rof to ocn srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consd for glc1 to rof srcMask =    -987987 dstMask =    -987987
 (med_map_mod: RouteHandles_init) created field_NormOne for lnd->rof with mapping consf
 (med_map_mod: RouteHandles_init) created field_NormOne for rof->lnd with mapping consf
 (med_map_mod: RouteHandles_init) created field_NormOne for rof->ocn with mapping consd
 (med_map_mod: RouteHandles_init) created field_NormOne for rof->ocn with mapping rof2ocn_ice
 (med_map_mod: RouteHandles_init) created field_NormOne for rof->ocn with mapping rof2ocn_liq
 (med_map_mod: RouteHandles_init) created field_NormOne for glc1->rof with mapping consd
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_irrig       1
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofgwl       2
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofi       3
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofsub       4
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofsur       5
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consd  gfrac  Fgrg_rofi       1
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_irrig       1
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofgwl       2
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofi       3
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofsub       4
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofsur       5
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consd for lnd to rof srcMask =    -987987 dstMask =    -987987
(med.F90:DataInitialize) rof T T
global nx,ny,ntile sizes for rof:     720     360       0

mizuRoute run directory: /glade/derecho/scratch/mizukami/SMS_D.ne30pg3_t232.HISTC_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MIZUROUTE_DGLC%NOEVOLVE_WW3.derecho_intel.20251106_141131_yedazb/run

grep -i -r -w rof med.log.3579934.desched1.251110-124915

 (internalstate init) rof model= mizuroute
  |from to -med  atm  lnd  ocn  ice  rof  wav  glc1 
  |rof         -    -    T    T    T    -    -    -
  |from to -med  atm  lnd  ocn  ice  rof  wav  glc1 
  |rof         -    -    T    T    T    -    -    -
(med.F90:DataInitialize) initializing FBs for rof
 mapping lnd->rof Flrl_rofsur via consf with lfrac normalization
 mapping lnd->rof Flrl_rofi via consf with lfrac normalization
 mapping lnd->rof Flrl_rofgwl via consf with lfrac normalization
 mapping lnd->rof Flrl_rofsub via consf with lfrac normalization
 mapping lnd->rof Flrl_irrig via consf with lfrac normalization
 mapping rof->lnd Flrr_volr via consf with one normalization
 mapping rof->lnd Flrr_volrmch via consf with one normalization
 mapping rof->lnd Flrr_flood via consf with one normalization
 mapping rof->ocn Flrr_flood via consd with one normalization
 mapping rof->ocn Forr_rofl via consd with one normalization
 mapping rof->ocn Forr_rofi via consd with one normalization
(merge_to_lnd): Flrr_volr(lnd) = Flrr_volr(rof)
(merge_to_lnd): Flrr_volrmch(lnd) = Flrr_volrmch(rof)
(merge_to_lnd): Flrr_flood(lnd) = Flrr_flood(rof)
(merge_to_ocn): Foxx_rofl(ocn) = Forr_rofl:Flrr_flood(rof)
(merge_to_ocn): Foxx_rofi(ocn) = Forr_rofi(rof)
(merge_to_rof): Flrl_rofsur(rof) = lfrac*Flrl_rofsur(lnd)
(merge_to_rof): Flrl_rofi(rof) = lfrac*Flrl_rofi(lnd)
(merge_to_rof): Flrl_rofgwl(rof) = lfrac*Flrl_rofgwl(lnd)
(merge_to_rof): Flrl_rofsub(rof) = lfrac*Flrl_rofsub(lnd)
(merge_to_rof): Flrl_irrig(rof) = lfrac*Flrl_irrig(lnd)
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consf for lnd to rof srcMask =    -987987 dstMask =    -987987
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consf for rof to lnd srcMask =    -987987 dstMask =    -987987
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consd for rof to ocn srcMask =    -987987 dstMask =          0
 (med_map_mod: RouteHandles_init) created field_NormOne for lnd->rof with mapping consf
 (med_map_mod: RouteHandles_init) created field_NormOne for rof->lnd with mapping consf
 (med_map_mod: RouteHandles_init) created field_NormOne for rof->ocn with mapping consd
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_irrig       1
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofgwl       2
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofi       3
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofsub       4
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofsur       5
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_irrig       1
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofgwl       2
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofi       3
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofsub       4
 (med_map_mod:med_packed_field_create)Packed field: destcomp,mapping,mapnorm,fldname,index:   rof  consf  lfrac  Flrl_rofsur       5
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consd for lnd to rof srcMask =    -987987 dstMask =    -987987
(med.F90:DataInitialize) rof T T
global nx,ny,ntile sizes for rof:   99427       1       0

[nmizukami]

The above errors in B-case (mizuRoute) have been resolved now

[nmizukami]

To enable area correction for import and export variable, need to change these subroutines in rof_import_export.F90.

• realize_fields: Need to compute area correction factors between area from rof input data and are from mesh. med2mod_areacor (mesh_areas/ model_areas) and mod2med_areacor (model_areas / mesh_areas)

• state_getimport: Need to correct area here by multiplying area correction factor med2mod_areacor and add input argument do_area_correction

• state_setexport: Need to correct area here by multiplying area correction factor mod2med_areacor, and add input argument do_area_correction

See MOSART rof_import_export.F90

[nmizukami]

Regarding missing discharge in Antarctica: Coupler variable Flrl_rofi is imported into rtmCTL%qsur(:,nfrz) as ice, but this is not sent to the outlet of the reach. That is why all the ice discharge are missing now. See how MOSART send this to the outlet of the river here . This is the same computation as direct to outlet. So direct_to_outlet for mizuRoute needs to be improved (or modify current method)

[nmizukami]

It seems that this is not reason for missing discharge in Antarctica. Need more digging....

[nmizukami]

Outstanding issues:

ROF-MOM mapping: Current fully coupled B-case tests used online mapping between ROF and MOM grids. Talking to @billsacks, the online mapping uses mass conserving mapping, not nearest neighbor mapping. The coupler creates mapping where both grids overlap each other, meaning the runoff discharge at outlets to the interior points (e.g., endorheic lakes) is not exported to MOM (because no overlapping interior ROF grid-boxes and ocean grid-boxes). We will need to create the mapping file with mizuRoute 0.5deg ESMF Mesh (including only land) and MOM grid as done in MOSART grid. I am guessing there is no option to switch to Nearest neighbor mapping for online mapping? Bill shared this tool , but may want to talk to Mike, who created the mapping file between MOSART grid and MOM grid that does nearest interpolation and smoothing.

ROF-Land grid mask mismatch: Where land mask in the land model grid but no land in ROF model grid, runoff exported from the land model is not passed to ROF, so water just disappears? MizuRoute currently use only land only mesh (no ocean mask), but may need to include ocean (add ocean mask) for fully coupled ESM uses).

Direct-to-outlet option: Currently mizuRoute's direct-to-outlet option is very slow, and need to implement the faster method. mizuRoute uses MPI send-receive routing for each upstream point to the outlet point (because outlet point may be handled in the different MPI task than upstream point).

[billsacks]

Regarding the ROF-MOM mapping: See ESCOMP/CMEPS#334. Briefly: there isn't currently a way to do runtime generation of an appropriate ROF-OCN mapping. Adding runtime generation of a conservative version of nearest neighbor mapping would take a moderate amount of work; extending this to also do the smoothing that's currently done would take a substantial amount of work. One or both of these are probably worthwhile long-term, but I'm not sure if they're justified in the near-term.

@nmizukami I'll be interested to hear whether it's feasible to generate offline mapping files for this, at least for now. Also, do you feel it would be reasonable in the short-term to change the CMEPS behavior so it gives an error if you try to do a ROF-OCN mapping without giving a mapping file? The current behavior is scientifically very problematic, so I think it would be best if it aborted in this case rather than letting you run with something wrong, but I'm interested to hear how this would impact you.

[nmizukami]

Hi Bill @billsacks, I will look into your tool and/or Mike's For now, I would like to keep testing at least one more thing, i.e., how mizuRoute handles zero runoff from land model (where land model has ocean, but mizuRoute has land), and pass it to the ocean. So please leave it as is for now. Once we have the nearest neighbor mapping, it would be good to have the error check and abort the run.

[ekluzek]

Will copy PE layout from: /glade/campaign/cesm/cesmdata/cseg/runs/cesm2_0/b.e30_alpha08b.B1850C_LTso.ne30_t232_wgx3.299