Finite-amplitude Rossby Wave Diagnostics by csyhuang · Pull Request #566 · NOAA-GFDL/MDTF-diagnostics

csyhuang · 2024-05-20T03:25:59Z

Description

This is preliminary PR for an additional POD, "Finite-amplitude Rossby Wave Diagnostics". The code could run through and produce the plots on HTML properly when lastly touched base with Jess #514 (latest commit from main in this PR: f69e745 ) , but it breaks after merging the recent commits. After discussing with my liason @yihungkuo , it seems easier on both sides for me to submit a preliminary PR to iterate (rather than fixing the issues on my own). I am therefore submitting this PR to kickstart the fix, with the understanding that the MDTF team is busy and will get back only when time allows.

Please let me know if I shall provide more information to fix this. On my side, the missing items include the (1) Detailed documentation and (2) predigest data, which do not affect the test of the code. I am still working with my collaborators to finalize them, hoping to get them ready by the end of this month.

Thanks in advance! 😄

How Has This Been Tested?

In my inputdata/ directory, I have the following files downloaded from GFDL portal:

model/GFDL-CM3_historical_r1i1p1/day/GFDL-CM3_historical_r1i1p1.ua.day.nc
model/GFDL-CM3_historical_r1i1p1/day/GFDL-CM3_historical_r1i1p1.va.day.nc
model/GFDL-CM3_historical_r1i1p1/day/GFDL-CM3_historical_r1i1p1.ta.day.nc

I run MDTF with the following command and get all the plots successfully:

./mdtf -f src/default_finite_amplitude_wave_diag.jsonc -v

Checklist:

Generated plot generated from sample data `GFDL-CM3_historical_r1i1p1.*.day.nc`

The plots look like this (if changing the hyperlink "plot" to actual PNG images) - these are generated out of a small sample so the patterns are not representative:

diagnostics/finite_amplitude_wave_diag/finite_amplitude_wave_diag_zonal_mean.py

+    dz = 1000  # TODO Variable to set earlier?
+    hmax = -SCALE_HEIGHT * np.log(gridfilled_dataset[plev_name].min() / P_GROUND)
+    kmax = int(hmax // dz) + 1
+    original_pseudoheight = convert_hPa_to_pseudoheight(original_grid[plev_name]).rename("height")


diagnostics/finite_amplitude_wave_diag/finite_amplitude_wave_diag_zonal_mean.py

+def plot_and_save_figure(seasonal_average_data, analysis_height_array, plot_dir, title_str, season,
+                         xy_mask=None, yz_mask=None):
+    if xy_mask is None:
+        xy_mask = np.zeros_like(seasonal_average_data.u_baro)


diagnostics/finite_amplitude_wave_diag/finite_amplitude_wave_diag_zonal_mean.py

+    else:
+        yland, xland = np.where(xy_mask)
+    if yz_mask is None:
+        yz_mask = np.zeros_like(seasonal_average_data.zonal_mean_u)


wrongkindofdoctor

@csyhuang. Thanks for the preliminary PR. The POD code looks okay from a technical standpoint. I just have a couple of change requests for your custom environment file and runtime configuration file that you can include with your final submission.

wrongkindofdoctor · 2024-05-29T13:12:40Z

src/default_finite_amplitude_wave_diag.jsonc

@@ -0,0 +1,128 @@
+// Configuration for MDTF-diagnostics driver script self-test.


please move this file to the diagnostics/finite_amplitude_wave_diag directory before the final submission.

@wrongkindofdoctor Hi Jess, thank you for reviewing the pull request and make changes to main accordingly! With the merged commits, ./mdtf -f src/default_finite_amplitude_wave_diag.jsonc -v no longer works. How shall I run my POD from this branch end-to-end on my machine?

(I created a new branch finite_amplitude_wave_diag_fcca20a_prelimPR from the current one, merged the new commits from main, and did the 2 changes you mentioned here. After testing them end-to-end I'll merge them to this branch.)

Thanks for your kind help!

@csyhuang Your runtime configuration file needs to be updated to match the new template format here: https://github.com/NOAA-GFDL/MDTF-diagnostics/blob/main/templates/runtime_config.jsonc.

Note that the POD_LIST is now a separate entry, WORKING_DIR is now WORK_DIR, and several of the previous settings have been removed.

Also, given that the main branch is under active development as we resolve outstanding issues with the preprocessor, your POD is not required to run with the current state of the framework. Just make sure that you have the required documentation in your RST file along with requested the changes to your test files, and the framework team will finalize testing at GFDL. We will reach out if we encounter technical issues in your POD.

@wrongkindofdoctor Hi Jess, thank you for the instructions. I have a few follow-up questions:

In runtime_config.jsonc, there are two items not present in previous commits which I don't know how to configure:

case_list

DATA_CATALOG

I looked at the instructions in README.md and tried running tools/catalog_builder/catalog_builder.py but encounter some errors (e.g. failed to import src due to missing path - shall I submit an issue ticket about these?). Is the script specific to GFDL environment?

Also, given that the main branch is under active development as we resolve outstanding issues with the preprocessor, your POD is not required to run with the current state of the framework. Just make sure that you have the required documentation in your RST file along with requested the changes to your test files, and the framework team will finalize testing at GFDL. We will reach out if we encounter technical issues in your POD.

In that case, shall I keep the script files on this branch untouched, and notify you again when I have all documentation finished (+ made the changes you mentioned)? Please let me know how to proceed. Thanks! 😄

No need. Those modules are needed for building GFDL datasets, and I haven't solved the import issue yet. I have commented out the problem lines and pushed the workaround to the main branch. Once you pull in the changes, you can use the catalog builder to generate intake catalogs for CMIP or CESM data.

wrongkindofdoctor · 2024-05-29T13:20:38Z

src/conda/env_finite_amplitude_wave_diag.yml

@@ -0,0 +1,19 @@
+name: _MDTF_finite_amplitude_wave_diag


please remove this file from the PR. The gridfill, bottleneck, and falwa packages are now included in the enve_python3_base.yml file.

This reverts commit 6985ff7.

csyhuang · 2024-08-16T21:20:29Z

src/conda/env_finite_amplitude_wave_diag.yml

+- bottleneck
+- pip=23.3.1
+- pip:
+    - git+https://github.com/csyhuang/hn2016_falwa.git


@wrongkindofdoctor Hi Jess, we figured out a solution to upgrade falwa to python 3.12 by migrating the F2PY build to meson. We tested on both Mac and Linux machine that the new installation procedures work. Do you have a chance to test if you can get our package installed via

- pip: - git+https://github.com/csyhuang/hn2016_falwa.git

We can do deployment on PyPI if cloning from GitHub is not preferred.

Thanks and we look forward to your update!

Clare

@csyhuang Excellent! I'm glad you were able to find a replacement for F2PY. Yes, please push the update to PyPI and reference the package instead of the git repo in the environment file.

@wrongkindofdoctor I've just pushed an update to the PR branch and changed the line to - falwa==2.0.0. Thanks Jess!

@csyhuang Thanks! Please move your environment file and runtime config file from /src to your POD directory; I will add falwa and the dependencies back to the python3 base env files and verify that they build in the in the CI.

@wrongkindofdoctor Sure! I've moved the files to diagnostics/finite_amplitude_wave_diag. Please keep us updated after you tested it! Thanks Jess!

@csyhuang The environments have been tested with the falwa package updates and merged into the python3_base env files.

@wrongkindofdoctor Glad that it works! Thanks Jess! We'll update you after finishing the rest of our POD!

…ectory

bitterbark · 2025-05-15T21:02:05Z

Is the only conflict in the README.md? It looks like you put in your new lines and meanwhile someone else has done theirs. If you could get the new version and add your lines so they don't conflict with the changes, it might work.

csyhuang · 2025-05-30T15:48:46Z

@bitterbark Hi Dani, sorry for the late reply as I have been doing long-distance travel in the past few weeks! ✈️ There are a few things left in this PR which still requires some work:

Data input using data catalog has not been properly configured yet, mainly because I haven't found a dataset without errors that satisfies the vertical resolution requirement of my POD (one dataset from GFDL-CM4 fulfilled the resolution requirement but its vertical coordinates were inverted by mistake - I contacted Wenhao in some previous email regarding this)
Predigest data has not been included in this POD yet, mainly because I am having trouble accessing UChicago's server - resolution is still pending
Scientific description of the POD - we'll finish that ASAP

I will try to resolve all conflicts with main branch this weekend and keep you updated. Thanks a lot!

bitterbark · 2025-06-03T20:15:42Z

Thanks for the update, Clare. Do you have the written requirements for the vertical resolution requirement? And I'm assuming you have some data that the POD does work with? If so, we can include that in the release.

…

On Fri, May 30, 2025 at 9:49 AM Clare S. Y. Huang ***@***.***> wrote: *csyhuang* left a comment (NOAA-GFDL/MDTF-diagnostics#566) <#566 (comment)> @bitterbark <https://github.com/bitterbark> Hi Dani, sorry for the late reply as I have been doing long-distance travel in the past few weeks! ✈️ There are a few things left in this PR which still requires some work: 1. Data input using data catalog has not been properly configured yet, mainly because I haven't found a dataset without errors that satisfies the vertical resolution requirement of my POD (one dataset from GFDL-CM4 fulfilled the resolution requirement but its vertical coordinates were inverted by mistake - I contacted Wenhao in some previous email regarding this) 2. Predigest data has not been included in this POD yet, mainly because I am having trouble accessing UChicago's server - resolution is still pending 3. Scientific description of the POD - we'll finish that ASAP I will try to resolve all conflicts with main branch this weekend and keep you updated. Thanks a lot! — Reply to this email directly, view it on GitHub <#566 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQG6Y5DS5NAFNJEJ6LL3BB4XLAVCNFSM6AAAAAB5HDQOFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMRSG42TCNJZGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- My working day may not be the same as your working day. Please do not feel obliged to reply to this email outside of your normal hours.

yihungkuo · 2025-10-30T20:04:18Z

@aradhakrishnanGFDL Hi Aparna, here's the PR in question. Thanks.

csyhuang · 2025-11-03T18:57:50Z

@bitterbark Dani, sorry for missing your previous message! Vertical resolution similar to ERA-5, which can resolve stratosphere up to 1hPa shall work.

@aradhakrishnanGFDL For the purpose of building the POD, may I get access to some GFDL-CM4 data that has has correct postprocessing done? Thanks!

csyhuang added 30 commits July 31, 2023 09:58

Well-formatted settings.jsonc without comments yet.

561306f

start all files

75b0f05

merged main

e9722bd

try running daily_avg

6e1d37a

daily_avg

3b25aa7

daily_avg

2fce674

daily_avg

acf7486

daily_avg

096726b

daily_avg

12d41e8

daily_avg

d45dc46

daily_avg

3d5c410

daily_avg

21ce25e

daily_avg

ccfd952

daily_avg

4897e80

add src/conda/env_finite_amplitude_wave_diag.yml

a370a58

add back ==

8741055

try other dependencies

eaa2960

try other dependencies

036b80b

add dependencies

109cd65

add var_names

8eb0780

add var_names

61a619f

add var_names

c23077c

extract 3 slices to local

0502b7e

add

f21f126

add missing line

9b7a675

add dask

4ecc91b

add missing line

c709ad1

add missing

b9c7d74

add missing

04260c4

tried on model data, found unphysical issues

957cf00

csyhuang requested review from aradhakrishnanGFDL, jtmims and wrongkindofdoctor as code owners May 20, 2024 03:26

remove redundant file

1d019fe

wrongkindofdoctor assigned wrongkindofdoctor and jtmims May 20, 2024

wrongkindofdoctor added the diagnostic Issue pertains to a contributed diagnostic label May 20, 2024

Merge branch 'main' into finite_amplitude_wave_diag_fcca20a

483b9de

github-advanced-security bot found potential problems May 20, 2024

View reviewed changes

wrongkindofdoctor suggested changes May 29, 2024

View reviewed changes

csyhuang added 7 commits May 29, 2024 10:19

merge

6985ff7

Revert "merge"

1bf6002

This reverts commit 6985ff7.

Merge remote-tracking branch 'upstream/main'

625e606

Merge remote-tracking branch 'upstream/main'

28cc5b4

Merge branch 'main' into finite_amplitude_wave_diag_fcca20a

5029c80

update src/conda/env_finite_amplitude_wave_diag.yml

6bb4a88

update env_finite_amplitude_wave_diag.yml

9c7b016

csyhuang commented Aug 16, 2024

View reviewed changes

csyhuang added 4 commits August 18, 2024 10:13

Update env file to use PyPI

d8e34f6

move environment file and runtime config file from /src to my POD dir…

6d1a94b

…ectory

Merge remote-tracking branch 'upstream/main'

31095c0

merge

9628cd8

aradhakrishnanGFDL requested a review from bitterbark May 15, 2025 20:44

aradhakrishnanGFDL unassigned jtmims and wrongkindofdoctor May 15, 2025

		@@ -0,0 +1,128 @@
		// Configuration for MDTF-diagnostics driver script self-test.

Conversation

csyhuang commented May 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Generated plot generated from sample data GFDL-CM3_historical_r1i1p1.*.day.nc

Uh oh!

Check notice

Check notice

Check notice

wrongkindofdoctor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csyhuang May 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csyhuang May 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wrongkindofdoctor May 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csyhuang Aug 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wrongkindofdoctor Aug 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bitterbark commented May 15, 2025

Uh oh!

csyhuang commented May 30, 2025

Uh oh!

bitterbark commented Jun 3, 2025 via email

Uh oh!

yihungkuo commented Oct 30, 2025

Uh oh!

csyhuang commented Nov 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

csyhuang commented May 20, 2024 •

edited

Loading

Generated plot generated from sample data `GFDL-CM3_historical_r1i1p1.*.day.nc`

csyhuang May 29, 2024 •

edited

Loading

csyhuang May 30, 2024 •

edited

Loading

wrongkindofdoctor May 30, 2024 •

edited

Loading

csyhuang Aug 16, 2024 •

edited

Loading

wrongkindofdoctor Aug 19, 2024 •

edited

Loading