Negative runoff fluxes in CESM3 Development Simulations

[olyson]

In the CESM co-chairs meeting of 7/19/22, the OMWG noted that there were significant negative runoff fluxes in several of the recent CESM3 development coupled simulations, in particular, around several small islands.
We of course have had negative runoff fluxes in previous versions of the model (e.g., CESM2), due to irrigation and lakes (negative P-E).
However, these particular islands have neither irrigation nor lakes.
After a bit of analysis and discussion (@olyson @swensosc @wwieder ), we found that there are non-zero land fractions (identified by landfrac on the CLM history file) for gridcells where we don't have any surface data (pfts, lakes, glaciers, urban, crop) on the surface dataset being used. These are being modeled as wetlands which can have negative P-E and therefore negative runoff.
Several of these islands did not exist in CESM2.
For CESM3, there seems to be a bit of a mismatch between the landfrac that CLM actually uses (generated from the mesh file?) and the land/ocean mask on the surface dataset.
Question: Does this go away when we generate new surface datasets or will there always potentially be some kind of a mismatch?
@wwieder @swensosc @ekluzek @billsacks @dlawrenncar

[billsacks]

This could change somewhat with newer raw data, but this issue will potentially always exist given our current method for generating surface datasets. Basically: if some area is outside the ocean model grid then CTSM becomes the owner of that area; if none of CTSM's raw datasets claim ownership of that area, then it is set as wetland. (Though note that we're currently doing this slightly wrong - see #1716 .)

If this negative runoff is a problem we will need to either (1) come up with a different scheme for what to do when CTSM owns an area that appears to be ocean according to CTSM's raw data sets; (2) change the physics in some way to avoid generating negative runoff.

[wwieder]

Thanks for looking into this @olyson.
@billsacks I agree with your suggested solutions, which of these seem like the best option?For option_2 @swensosc suggested we could consider creating desert (bare ground) islands instead of wetland islands, which should avoid negative runoff.

[billsacks]

I'm not sure off-hand. For changing what we call these areas (wetland vs. desert, etc.), I'm worried that solving one problem will create another – like treating substantial areas as bare land that should really be water. Changing the physics to avoid negative runoff sounds like a good idea, but I think could pose some real challenges. Regardless of what we do, I feel like it's going to be hard to avoid having some situations where we generate persistently negative (P-E), and it's hard for me to envision a good solution that avoids relying on the ocean to solve the problem.

[olyson]

We asked Cecile to run a short fully coupled simulation similar to one of the CESM3-dev simulations (b.cesm3_cam058_mom_c.B1850WscMOM.f09_L58_t061.011) to see what the effects are of taking the negative runoff directly from the river mouths.
This was accomplished by setting bypass_routing_option='direct_to_outlet' and qgwl_runoff_option='negative' in the mosart namelist.
Four years of this new simulation have been completed (b.cesm3_cam058_mom_c.B1850WscMOM.f09_L58_t061.015).
Results are shown in slides 7 and 8 of this slide deck:

https://docs.google.com/presentation/d/1gEK5ASR3CNbQPkdCCPm6SIavPjX7BCYh/edit?usp=sharing&ouid=104026900087612515425&rtpof=true&sd=true

Slide 7 shows that TOTAL_DISCHARGE_TO_OCEAN_LIQ is now positive everywhere in the new simulation (015) unlike in the control (011).
Slide 8 (monthly minimum of the lrunoff field from MOM) shows that there are very few ocean grid cells that have negative runoff in 015. Note the scale for 015 is two orders of magnitude smaller than 011.

There are standard diagnostics here:

https://webext.cgd.ucar.edu/B1850/b.cesm3_cam058_mom_c.B1850WscMOM.f09_L58_t061.015/lnd/b.cesm3_cam058_mom_c.B1850WscMOM.f09_L58_t061.015.2_4-b.cesm3_cam058_mom_c.B1850WscMOM.f09_L58_t061.011.2_4/setsIndex.html

In particular, set7 is of interest. The variability in these coupled simulations is probably swamping the effects of this change, particularly since these are only 3 year averages, but I don't see anything alarming in the results for discharge of individual rivers or into the individual ocean basins.

[olyson]

There are BUDGET WARNINGs in the mosart log file. For tracer 1 which I assume is liquid water.

/glade/scratch/hannay/archive/b.cesm3_cam058_mom_c.B1850WscMOM.f09_L58_t061.015/logs/rof.log.5167742.chadmin1.ib0.cheyenne.ucar.edu.220726-082955.gz

[olyson]

Sean has fixed a couple of bugs in the MOSART code with respect to the BUDGET WARNINGs we encountered when we set bypass_routing_option='direct_to_outlet' and qgwl_runoff_option='negative' in the mosart namelist.
I ran a pair of land-only simulations, one a control (with bypass_routing_option set to the default value of 'direct_in_place') and one with 'direct_to_outlet'. There are now no warnings in the log files.
Slide 3 of the slide deck shows the effect of this on reducing the negative TOTAL_DISCHARGE_TO_OCEAN_LIQ. What remains is due to, "For example, for the caspian, those gridcells don't have an outlet on the ocean mask. And the little dots in the continental interiors I think are 'sink' points, which don't touch the ocean. So we would still have negative runoff at those locations that is too large to be compensated for by the rivers. it would get passed to the ocean at those locations just like the direct_in_place method."

https://docs.google.com/presentation/d/1gEK5ASR3CNbQPkdCCPm6SIavPjX7BCYh/edit?usp=sharing&ouid=104026900087612515425&rtpof=true&sd=true

I don't see any discernable impact on our set7 discharge to the ocean diagnostics as shown here:

https://webext.cgd.ucar.edu/I2000/ctsm51_ctsm51d100_1deg_GSWP3V1_directtooutlet_2000/lnd/ctsm51_ctsm51d100_1deg_GSWP3V1_directtooutlet_2000.6_10-ctsm51_ctsm51d100_1deg_GSWP3V1_2000.6_10/set7/set7.html

I'm going to work on quantifying the magnitude of negative runoff with respect to total runoff and the reduction we get using this method.

[ekluzek]

Adding @nmizukami to this issue as we will want to be aware of these issues in mizuRoute. We should for example make sure we implement direct_to_outlet in mizuRoute because of this issue. So direct_in_place and direct_to_outlet for bypass_routing_option should both be implemented in mizuRoute. And any fixes we need to put in place in MOSART we'll also want to get into mizuRoute.

[ekluzek]

@olyson do you have a branch of MOSART with these changes? We should get these into a PR for MOSART. You could start the PR even before we are done with it's evaluation.

[billsacks]

Thanks @olyson and @swensosc . My understanding is that we still want to move ahead with trying the replacement of wetlands with bare ground in addition to this MOSART fix; is that right? If so, I'll work on it today.

[olyson]

Yes, that is my understanding as well, thanks.

[wwieder]

Yes, that would be great, @billsacks, as Sean's mosart changes don't help with the negative runoff from wetland islands.

@olyson, I recall you saying last week that frozen runoff in the log files was different with 'direct_to_outlet' option. Is this something we should investigate?