Runoff sensitivities by hanjunkim0617 · Pull Request #781 · NOAA-GFDL/MDTF-diagnostics

hanjunkim0617 · 2025-04-09T14:11:00Z

Description
Runoff sensitivity diagnostic package

How Has This Been Tested?
I had to add the variable 'mrro' in the filedlist, so I added a few lines to "data/fieldlist_CMIP.jsonc".
The diagnostic was tested by Dani Coleman from NCAR as well.
In Ubuntu 20.04.6 LTS, it takes ~7 min. to run.

The sample output figure is attached. Please see HTML page:
MDTF_CESM2.historical.v18.r10i1p1f1_1850_2014.Zip

In addition, pre-digested observational data is attached here (inputdata/obs_data):
obs_data.Zip

Please contact hk764@cornell.edu for further questions.

Thank you!
Best regards,
Hanjun

Checklist:

The calculation of runoff sensitivities is updated. It is now using 5-year averaged anomalies to estimate runoff sensitivities, whereas the original code used the inter-annual variations.

Add exit command. Otherwise warning like below occur: INFO: <#1NYy:CESM2.historical.r10i1p1f1.runoff_sensitivities> terminated, but the subprocess did not yield a return code. This does not necessarily indicate a failure.

Updated description of html page.

src/runoff_sensitivities_config.jsonc

wrongkindofdoctor · 2025-04-09T19:40:47Z

diagnostics/runoff_sensitivities/settings.jsonc

+    "pod_env_vars" : {
+      "EXAMPLE_FAV_COLOR": "blue"
+    },


Delete unused pod_env_vars block

wrongkindofdoctor · 2025-04-09T19:41:56Z

diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

+nlon = len(lonm)
+nt = len(time)
+nyr = int(nt/12)
+syr_model = int(os.environ["FIRSTYR"])


replace with upated environment variable 'startdate'

wrongkindofdoctor · 2025-04-09T19:42:22Z

diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

+nt = len(time)
+nyr = int(nt/12)
+syr_model = int(os.environ["FIRSTYR"])
+eyr_model = int(os.environ["LASTYR"])


replace with updated environment variable 'enddate'

wrongkindofdoctor · 2025-04-09T19:43:27Z

diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

+### 4) Saving output data: ##########################
+#####################################################
+## save runoff sensitivity data in netCDF4 file
+out_path = "{WK_DIR}/model/netCDF/runoff_sensitivities_{CASENAME}.nc".format(**os.environ)


replace 'WK_DIR' with 'WORK_DIR'

@bitterbark @hanjunkim0617 was it changed to WORKING_DIR? Change it to WORK_DIR.

wrongkindofdoctor · 2025-04-09T19:47:50Z

diagnostics/runoff_sensitivities/settings.jsonc

+  "settings" : {
+    "driver" : "runoff_sensitivities_diag.py",
+    "long_name" : "diagnostic for the runoff sensitivities",
+    "realm" : ["atmos","land"],


move realm to a data block:

data": { "realm": ["atmos", "land"] }

diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

diagnostics/runoff_sensitivities/preprocessing_code_at_local_cluster.py

diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

+import time as time
+import numpy as np # python library for dealing with data array
+import cartopy.io.shapereader as shpreader # cartopy library for loading with shapefiles
+import matplotlib


diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

+## figures for P sensitivity ##
+bins=[-3.5, -3, -2.5, -2, -1.6, -1.2, -0.8, -0.4, 0, 0.4, 0.8, 1.2, 1.6, 2, 2.5, 3, 3.5];
+bins2=[-100, -3, -2.5, -2, -1.6, -1.2, -0.8, -0.4, 0, 0.4, 0.8, 1.2, 1.6, 2, 2.5, 3, 100];
+plot_colormap = [(0.4000, 0, 0, 1), (0.7316, 0, 0, 1), (0.9975, 0.0306, 0.0078, 1), (0.9845, 0.1915, 0.0484, 1), (0.9715, 0.3525, 0.0890, 1), (0.9776, 0.5636, 0.0700, 1), (0.9891, 0.7889, 0.0338, 1), (1.0000, 1.0000, 0.0263, 1), (1.0000, 1.0000, 0.4408, 1), (1.0000, 1.0000, 1.0000, 1), (1.0000, 1.0000, 1.0000, 1), (0.7325, 0.8651, 0.9386, 1), (0.5342, 0.7652, 0.8931, 1), (0.3359, 0.6652, 0.8476, 1), (0.1375, 0.5652, 0.8020, 1), (0.0477, 0.4727, 0.7506, 1), (0.0563, 0.3870, 0.6938, 1), (0.0651, 0.3013, 0.6370, 1), (0.0368, 0.1575, 0.5250, 1), (0, 0, 0.4000, 1)]


diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

+nfoc=r2_obs_pred.shape[0]; nens=r2_obs_pred.shape[1];
+hval_r2_ia_gobs=np.full(nb,False);
+hval_r2_ia_hist6=np.full(nb,False);
+hval_r2_ia_model=np.full(nb,False);


diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

hanjunkim0617 · 2025-04-10T01:05:06Z

Dear Jess or whom may it concerns. Hello! I'm new to Github. I think I got feedback from you. So, is this mean that I need to do modifications following your instruction? In addition, regarding the observational file, I uploaded it in the Pull request. Do I need to separately upload it somewhere else as well? Thank you, Best, Hanjun

…

On Wed, Apr 9, 2025 at 3:56 PM Jess ***@***.***> wrote: ***@***.**** requested changes on this pull request. ------------------------------ In src/runoff_sensitivities_config.jsonc <#781 (comment)> : > @@ -0,0 +1,99 @@ +// Configuration for MDTF-diagnostics driver script self-test. +// +// Copy this file and customize the settings as needed to run the framework on +// your own model output without repeating command-line options. Pass it to the +// framework at the end of the command line (positionally) or with the Please move this file into your POD directory if you wish to retain it for reference. Otherwise, you can remove it from the PR: git rm --cached src/runoff_sensitivities_config.jsonc git commit -m "your commit message" git push ------------------------------ In diagnostics/runoff_sensitivities/settings.jsonc <#781 (comment)> : > + "pod_env_vars" : { + "EXAMPLE_FAV_COLOR": "blue" + }, Delete unused pod_env_vars block ------------------------------ In diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py <#781 (comment)> : > +tas_path = os.environ["TAS_FILE"] +hfls_path = os.environ["HFLS_FILE"] +mrro_path = os.environ["MRRO_FILE"] + +# model grids (pr,tas,hfls,mrro must have same lon,lat grids) +lat_coor_name = os.environ["lat_coord"] +lon_coor_name = os.environ["lon_coord"] +time_coord_name = os.environ["time_coord"] +latm = nc.Dataset(pr_path).variables[lat_coor_name][:] +lonm = nc.Dataset(pr_path).variables[lon_coor_name][:] +time = nc.Dataset(pr_path).variables[time_coord_name][:] +nlat = len(latm) +nlon = len(lonm) +nt = len(time) +nyr = int(nt/12) +syr_model = int(os.environ["FIRSTYR"]) replace with upated environment variable 'startdate' ------------------------------ In diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py <#781 (comment)> : > +hfls_path = os.environ["HFLS_FILE"] +mrro_path = os.environ["MRRO_FILE"] + +# model grids (pr,tas,hfls,mrro must have same lon,lat grids) +lat_coor_name = os.environ["lat_coord"] +lon_coor_name = os.environ["lon_coord"] +time_coord_name = os.environ["time_coord"] +latm = nc.Dataset(pr_path).variables[lat_coor_name][:] +lonm = nc.Dataset(pr_path).variables[lon_coor_name][:] +time = nc.Dataset(pr_path).variables[time_coord_name][:] +nlat = len(latm) +nlon = len(lonm) +nt = len(time) +nyr = int(nt/12) +syr_model = int(os.environ["FIRSTYR"]) +eyr_model = int(os.environ["LASTYR"]) replace with updated environment variable 'enddate' ------------------------------ In diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py <#781 (comment)> : > +# display warning if the interaction term is affecting the regression models +# role of interaction term (P*T) is usually negligible, consistent to Lehner et al. 2019 +condition=np.asarray(r2_model_int<0.8) +# condition=np.asarray(np.abs(r2_model_pred_int-r2_model_pred)>0.2) +for b in range(b): + if condition[b]: + warning_message = f'Warning: Interaction term may matter for model + basin #{b+1} (r={r2_model_int[b]:.2f})' + warnings.warn(warning_message) + + + +##################################################### +### 4) Saving output data: ########################## +##################################################### +## save runoff sensitivity data in netCDF4 file +out_path = "{WK_DIR}/model/netCDF/runoff_sensitivities_{CASENAME}.nc".format(**os.environ) replace 'WK_DIR' with 'WORK_DIR' ------------------------------ In diagnostics/runoff_sensitivities/settings.jsonc <#781 (comment)> : > @@ -0,0 +1,60 @@ +// runoff sensitivities MDTF diagnostic settings file +// This should be an ascii text file in JSON format +// (https://en.wikipedia.org/wiki/JSON#Data_types_and_syntax), with the addition +// that all text to the right of a "//" is a comment and ignored (sometimes +// called "JSONC" format). +{ + "settings" : { + "driver" : "runoff_sensitivities_diag.py", + "long_name" : "diagnostic for the runoff sensitivities", + "realm" : ["atmos","land"], move realm to a data block: data": { "realm": ["atmos", "land"] } ------------------------------ In diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py <#781 (comment)> : > + +## water year average (Oct-Sep water year for every basin) +## do time average and then do basin average (save computation time a lot) +syr_sens=1945 +eyr_sens=2014 +nyr = eyr_sens - syr_sens +inds=(syr_sens-syr_model)*12 + 10 - 1 +inde=inds + (nyr)*12 +prm_wy = np.nanmean(np.reshape(prm[inds:inde,:,:],(12,nyr,nlat,nlon),order="F"), axis=0) +tasm_wy = np.nanmean(np.reshape(tasm[inds:inde,:,:],(12,nyr,nlat,nlon),order="F"), axis=0) +mrrom_wy = np.nanmean(np.reshape(mrrom[inds:inde,:,:],(12,nyr,nlat,nlon),order="F"), axis=0) +hflsm_wy = np.nanmean(np.reshape(hflsm[inds:inde,:,:],(12,nyr,nlat,nlon),order="F"), axis=0) + +## area average for specific river basins +# load basin masks from shapefile provided by GRDC +shapefile_path = "{OBS_DATA}/mrb_basins.shp".format(**os.environ) Please provide a tarball or link to a website or ftp server with required observational files to me and @bitterbark <https://github.com/bitterbark>. — Reply to this email directly, view it on GitHub <#781 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJY2WOOGG33YNJWXJUFHQQT2YV3QRAVCNFSM6AAAAAB2ZDN3X6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDONJUGU2TOOJYHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

wrongkindofdoctor · 2025-04-10T14:46:32Z

Hi, @hanjunkim0617. All you need to do is update the files with the requested changes on your local development branch on your workstation, then commit and push the updates to your remote fork. The PR will automatically update and rerun the checks.

I saw the obs data file in the initial message, and have a copy now.

diagnostics/runoff_sensitivities/preprocessing_code_at_local_cluster.py

+## import libraries
+import os
+import numpy as np
+import cartopy.io.shapereader as shpreader


diagnostics/runoff_sensitivities/runoff_sensitivities_diag.py

bitterbark · 2025-04-10T15:25:36Z

@wrongkindofdoctor I have the obs data tar ball already.

wrongkindofdoctor · 2025-04-10T16:14:48Z

@bitterbark Great! Thanks for letting me know. I'll update the review request.

bitterbark · 2025-05-15T21:07:41Z

I just started looking at this but I can't see what change Jess requested in the last message. Is it resolved?

bitterbark · 2025-05-15T21:08:27Z

I can't see the change requested by Jess in the last message. Has it been resolved?

aradhakrishnanGFDL · 2025-05-16T20:01:19Z

I can't see the change requested by Jess in the last message. Has it been resolved?

It looks like there are just a few things that need to be fixed, like removing unused imports, moving realm to the data block, etc. https://github.com/NOAA-GFDL/MDTF-diagnostics/pull/781/files/6170feed6e51d2557e02fc632593f9966339d5f1#r2036044413
It's way up this PR.

wrongkindofdoctor · 2025-05-18T14:41:04Z

@hanjunkim0617 @aradhakrishnanGFDL @bitterbark I requested that the realm entry in the settings file be moved to a data block like so:

data": { "realm": ["atmos", "land"] }

The POD runtime config file also needs to be removed from the src directory. It can be placed in the POD subdirectory for reference if desired.

Hanjun Kim and others added 12 commits October 26, 2023 12:41

runoff_sensitivities_v1

d00a44b

runoff sensitivities_v2

b7d63f9

The calculation of runoff sensitivities is updated. It is now using 5-year averaged anomalies to estimate runoff sensitivities, whereas the original code used the inter-annual variations.

runoff_sensitivities_v2

4d001ac

Update runoff_sensitivities_diag.py

b70f999

Add exit command. Otherwise warning like below occur: INFO: <#1NYy:CESM2.historical.r10i1p1f1.runoff_sensitivities> terminated, but the subprocess did not yield a return code. This does not necessarily indicate a failure.

Update runoff_sensitivities.html

74f4c04

Updated description of html page.

Update runoff_sensitivities.rst

6fdb561

Add files via upload

f5f3733

Update runoff_sensitivities.rst

9bd70d5

Update runoff_sensitivities.rst

cec07df

Update runoff_sensitivities.html

1cb0c36

Update runoff_sensitivities.html

79d3819

Update runoff_sensitivities.rst

6170fee

hanjunkim0617 marked this pull request as ready for review April 9, 2025 16:45

hanjunkim0617 requested review from aradhakrishnanGFDL, jkrasting, jtmims and wrongkindofdoctor as code owners April 9, 2025 16:45

wrongkindofdoctor self-assigned this Apr 9, 2025

wrongkindofdoctor added the diagnostic Issue pertains to a contributed diagnostic label Apr 9, 2025

wrongkindofdoctor suggested changes Apr 9, 2025

View reviewed changes

github-advanced-security bot found potential problems Apr 9, 2025

View reviewed changes

hanjunkim0617 added 3 commits April 9, 2025 21:09

Update settings.jsonc

f7cd232

Update runoff_sensitivities_diag.py

61d9258

Update preprocessing_code_at_local_cluster.py

b5a07d2

github-advanced-security bot found potential problems Apr 10, 2025

View reviewed changes

aradhakrishnanGFDL assigned bitterbark May 15, 2025

aradhakrishnanGFDL unassigned wrongkindofdoctor May 15, 2025

aradhakrishnanGFDL requested a review from bitterbark May 15, 2025 20:42

aradhakrishnanGFDL self-assigned this May 15, 2025

aradhakrishnanGFDL removed request for jkrasting and jtmims May 21, 2025 16:23

hanjunkim0617 added 2 commits July 3, 2025 14:05

Update settings.jsonc

59dea96

Merge branch 'NOAA-GFDL:main' into runoff_sensitivities

db8045c

Conversation

hanjunkim0617 commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

wrongkindofdoctor Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

wrongkindofdoctor Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

wrongkindofdoctor Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

wrongkindofdoctor Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

aradhakrishnanGFDL May 16, 2025

Choose a reason for hiding this comment

Uh oh!

wrongkindofdoctor Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check notice

Uh oh!

Check warning

Uh oh!

Uh oh!

Check warning

Uh oh!

Uh oh!

Uh oh!

hanjunkim0617 commented Apr 10, 2025 via email

Uh oh!

wrongkindofdoctor commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Check notice

Uh oh!

bitterbark commented Apr 10, 2025

Uh oh!

wrongkindofdoctor commented Apr 10, 2025

Uh oh!

bitterbark commented May 15, 2025

Uh oh!

bitterbark commented May 15, 2025

Uh oh!

aradhakrishnanGFDL commented May 16, 2025

Uh oh!

wrongkindofdoctor commented May 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hanjunkim0617 commented Apr 9, 2025 •

edited

Loading

wrongkindofdoctor commented Apr 10, 2025 •

edited

Loading