Skip to content

Update 4-degree SHiELD-SOM ensemble dataset #990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
spencerkclark merged 2 commits into from
Mar 20, 2026
Merged

Update 4-degree SHiELD-SOM ensemble dataset #990

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e54decb
Add time-coarsening config for SHiELD-SOM ensemble
spencerkclark Mar 19, 2026
0c2bbc6
Merge branch 'main' into feature/recompute-4-degree-shield-som-ensemble
spencerkclark Mar 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion scripts/data_process/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ shield_som_c96_spin_up_dataset:

.PHONY: shield_som_ensemble_c96_dataset
shield_som_ensemble_c96_dataset:
./compute_dataset.sh --config configs/shield-som-ensemble-c96-$(RESOLUTION)-$(LAYERS).yaml
./compute_dataset.sh --dataset --stats --time-coarsen --config configs/shield-som-ensemble-c96-$(RESOLUTION)-$(LAYERS).yaml

.PHONY: shield_som_abrupt_co2_increase_c96_dataset
shield_som_abrupt_co2_increase_c96_dataset:
Expand Down
95 changes: 92 additions & 3 deletions scripts/data_process/configs/shield-som-ensemble-c96-4deg-8layer.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ runs:
4xCO2-ic_0003: gs://vcm-ml-raw-flexible-retention/2024-07-03-C96-SHiELD-SOM/regridded-zarrs/gaussian_grid_45_by_90/4xCO2-ic_0003
4xCO2-ic_0004: gs://vcm-ml-raw-flexible-retention/2024-07-03-C96-SHiELD-SOM/regridded-zarrs/gaussian_grid_45_by_90/4xCO2-ic_0004
4xCO2-ic_0005: gs://vcm-ml-raw-flexible-retention/2024-07-03-C96-SHiELD-SOM/regridded-zarrs/gaussian_grid_45_by_90/4xCO2-ic_0005
data_output_directory: gs://vcm-ml-intermediate/2024-07-09-vertically-resolved-4deg-c96-shield-som-ensemble-fme-dataset
data_output_directory: gs://vcm-ml-intermediate/2026-01-28-vertically-resolved-4deg-c96-shield-som-ensemble-fme-dataset
stats:
output_directory: gs://vcm-ml-intermediate/2024-07-09-vertically-resolved-4deg-c96-shield-som-ensemble-fme-dataset-stats
beaker_dataset: 2024-07-09-vertically-resolved-4deg-fme-c96-shield-som-ensemble-dataset-stats
output_directory: gs://vcm-ml-intermediate/2026-01-28-vertically-resolved-4deg-c96-shield-som-ensemble-fme-dataset-stats
beaker_dataset: 2026-01-28-vertically-resolved-4deg-fme-c96-shield-som-ensemble-dataset-stats
# These datasets already exclude the one-year divergence period for each
# ensemble member, so we can use data from all times when computing the
# stats.
Expand Down Expand Up @@ -127,3 +127,92 @@ dataset_computation:
- global_mean_co2
standard_names:
total_frozen_precip_rate: None
time_coarsen:
Copy link
Contributor

@Arcomano1234 Arcomano1234 Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: For the time coarsening this creates 2 datasets correct? One with the original time resolution and the other daily or do we already this dataset at 4deg and 6 hourly?

Copy link
Member Author

@spencerkclark spencerkclark Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, this full YAML file defines two datasets: one at the native temporal resolution and one coarsened to daily resolution. The time-coarsened dataset is derived from the native resolution dataset, so the native resolution dataset must be computed first.

This can all be handled automatically, as I took advantage of in compute-fme-dataset-ensemble-74plc, thanks to @mcgibbon's work in #891:

Details 

STEP                                          TEMPLATE                        PODNAME                                        DURATION  MESSAGE
 ✔ compute-fme-dataset-ensemble-74plc         compute-fme-dataset-ensemble
 ├─┬─✔ compute-fme-dataset-individual(0:0)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-3739697141  8m
 │ ├─✔ compute-fme-dataset-individual(1:1)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-1084221249  20m
 │ ├─✔ compute-fme-dataset-individual(2:2)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-1804246013  14m
 │ ├─✔ compute-fme-dataset-individual(3:3)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-3022590609  12m
 │ ├─✔ compute-fme-dataset-individual(4:4)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-1166380877  12m
 │ ├─✔ compute-fme-dataset-individual(5:5)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-122836337   12m
 │ ├─✔ compute-fme-dataset-individual(6:6)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-2895883749  20m
 │ ├─✔ compute-fme-dataset-individual(7:7)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-966649729   13m
 │ ├─✔ compute-fme-dataset-individual(8:8)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-3653829861  13m
 │ ├─✔ compute-fme-dataset-individual(9:9)    compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-1517009281  12m
 │ ├─✔ compute-fme-dataset-individual(10:10)  compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-3077512735  12m
 │ ├─✔ compute-fme-dataset-individual(11:11)  compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-1416641317  12m
 │ ├─✔ compute-fme-dataset-individual(12:12)  compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-2023206419  16m
 │ ├─✔ compute-fme-dataset-individual(13:13)  compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-1048689509  11m
 │ ├─✔ compute-fme-dataset-individual(14:14)  compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-2494195159  20m
 │ ├─✔ compute-fme-dataset-individual(15:15)  compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-644008053   12m
 │ └─✔ compute-fme-dataset-individual(16:16)  compute-fme-dataset-individual  compute-fme-dataset-ensemble-74plc-1597493907  20m
 ├─┬─✔ time-coarsen(0:0)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-1485623714  18m
 │ ├─✔ time-coarsen(1:1)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-1821665790  16m
 │ ├─✔ time-coarsen(2:2)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-621021970   16m
 │ ├─✔ time-coarsen(3:3)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-3544939766  19m
 │ ├─✔ time-coarsen(4:4)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-503450450   17m
 │ ├─✔ time-coarsen(5:5)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-532802790   15m
 │ ├─✔ time-coarsen(6:6)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-2860048386  18m
 │ ├─✔ time-coarsen(7:7)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-950835150   15m
 │ ├─✔ time-coarsen(8:8)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-4278124130  18m
 │ ├─✔ time-coarsen(9:9)                      time-coarsen                    compute-fme-dataset-ensemble-74plc-2254453822  18m
 │ ├─✔ time-coarsen(10:10)                    time-coarsen                    compute-fme-dataset-ensemble-74plc-2173696884  17m
 │ ├─✔ time-coarsen(11:11)                    time-coarsen                    compute-fme-dataset-ensemble-74plc-3899484094  23m
 │ ├─✔ time-coarsen(12:12)                    time-coarsen                    compute-fme-dataset-ensemble-74plc-2344054628  36m
 │ ├─✔ time-coarsen(13:13)                    time-coarsen                    compute-fme-dataset-ensemble-74plc-1270002754  32m
 │ ├─✔ time-coarsen(14:14)                    time-coarsen                    compute-fme-dataset-ensemble-74plc-654989764   32m
 │ ├─✔ time-coarsen(15:15)                    time-coarsen                    compute-fme-dataset-ensemble-74plc-4096070886  31m
 │ └─✔ time-coarsen(16:16)                    time-coarsen                    compute-fme-dataset-ensemble-74plc-2769767140  32m
 ├─┬─✔ get-stats(0:0)                         get-stats                       compute-fme-dataset-ensemble-74plc-957368618   5m
 │ ├─✔ get-stats(1:1)                         get-stats                       compute-fme-dataset-ensemble-74plc-1597225830  6m
 │ ├─✔ get-stats(2:2)                         get-stats                       compute-fme-dataset-ensemble-74plc-3313966554  5m
 │ ├─✔ get-stats(3:3)                         get-stats                       compute-fme-dataset-ensemble-74plc-3011461790  6m
 │ ├─✔ get-stats(4:4)                         get-stats                       compute-fme-dataset-ensemble-74plc-3196395034  4s
 │ ├─✔ get-stats(5:5)                         get-stats                       compute-fme-dataset-ensemble-74plc-3008347342  6m
 │ ├─✔ get-stats(6:6)                         get-stats                       compute-fme-dataset-ensemble-74plc-184258058   12m
 │ ├─✔ get-stats(7:7)                         get-stats                       compute-fme-dataset-ensemble-74plc-3106895926  11m
 │ ├─✔ get-stats(8:8)                         get-stats                       compute-fme-dataset-ensemble-74plc-524580586   10m
 │ ├─✔ get-stats(9:9)                         get-stats                       compute-fme-dataset-ensemble-74plc-1164437798  3m
 │ ├─✔ get-stats(10:10)                       get-stats                       compute-fme-dataset-ensemble-74plc-94035516    5m
 │ ├─✔ get-stats(11:11)                       get-stats                       compute-fme-dataset-ensemble-74plc-1709874022  6m
 │ ├─✔ get-stats(12:12)                       get-stats                       compute-fme-dataset-ensemble-74plc-2534945292  10m
 │ ├─✔ get-stats(13:13)                       get-stats                       compute-fme-dataset-ensemble-74plc-924009610   12m
 │ ├─✔ get-stats(14:14)                       get-stats                       compute-fme-dataset-ensemble-74plc-845880428   12m
 │ ├─✔ get-stats(15:15)                       get-stats                       compute-fme-dataset-ensemble-74plc-311478030   12m
 │ └─✔ get-stats(16:16)                       get-stats                       compute-fme-dataset-ensemble-74plc-106763308   6m
 ├───✔ combine-stats                          combine-stats                   compute-fme-dataset-ensemble-74plc-781128177   3m
 └───✔ upload-beaker-stats                    upload-beaker-stats             compute-fme-dataset-ensemble-74plc-2322081493  6s

factor: 4
data_output_directory: gs://vcm-ml-intermediate/2026-01-28-vertically-resolved-4deg-daily-c96-shield-som-ensemble-fme-dataset
stats_output_directory: gs://vcm-ml-intermediate/2026-01-28-vertically-resolved-4deg-daily-c96-shield-som-ensemble-fme-dataset-stats
snapshot_names:
- PRESsfc
- Q2m
- RH200
- RH500
- RH850
- TMP200
- TMP2m
- TMP500
- TMP850
- UGRD10m
- UGRD200
- UGRD500
- UGRD850
- VGRD10m
- VGRD200
- VGRD500
- VGRD850
- air_temperature_0
- air_temperature_1
- air_temperature_2
- air_temperature_3
- air_temperature_4
- air_temperature_5
- air_temperature_6
- air_temperature_7
- eastward_wind_0
- eastward_wind_1
- eastward_wind_2
- eastward_wind_3
- eastward_wind_4
- eastward_wind_5
- eastward_wind_6
- eastward_wind_7
- h50
- h500
- h850
- land_fraction
- northward_wind_0
- northward_wind_1
- northward_wind_2
- northward_wind_3
- northward_wind_4
- northward_wind_5
- northward_wind_6
- northward_wind_7
- ocean_fraction
- sea_ice_fraction
- soil_moisture_0
- soil_moisture_1
- soil_moisture_2
- soil_moisture_3
- specific_total_water_0
- specific_total_water_1
- specific_total_water_2
- specific_total_water_3
- specific_total_water_4
- specific_total_water_5
- specific_total_water_6
- specific_total_water_7
- surface_temperature
- total_water_path
window_names:
- DLWRFsfc
- DSWRFsfc
- DSWRFtoa
- LHTFLsfc
- PRATEsfc
- SHTFLsfc
- ULWRFsfc
- ULWRFtoa
- USWRFsfc
- USWRFtoa
- global_mean_co2
- prescribed_mixed_layer_depth
- prescribed_qflux
- tendency_of_total_water_path
- tendency_of_total_water_path_due_to_advection
- total_frozen_precipitation_rate
constant_prefixes:
- ak_
- bk_
- HGTsfc
- grid_xt
- grid_yt
Loading