Update 4-degree SHiELD-SOM ensemble dataset#990
Merged
spencerkclark merged 2 commits intomainfrom Mar 20, 2026
Merged
Conversation
spencerkclark
force-pushed
the
feature/recompute-4-degree-shield-som-ensemble
branch
from
7b9cbc5 to
e54decb
Compare
Arcomano1234
approved these changes
Mar 19, 2026
| - global_mean_co2 | ||
| standard_names: | ||
| total_frozen_precip_rate: None | ||
| time_coarsen: |
Contributor
There was a problem hiding this comment.
Question: For the time coarsening this creates 2 datasets correct? One with the original time resolution and the other daily or do we already this dataset at 4deg and 6 hourly?
Member
Author
There was a problem hiding this comment.
Correct, this full YAML file defines two datasets: one at the native temporal resolution and one coarsened to daily resolution. The time-coarsened dataset is derived from the native resolution dataset, so the native resolution dataset must be computed first.
This can all be handled automatically, as I took advantage of in compute-fme-dataset-ensemble-74plc, thanks to @mcgibbon's work in #891:
Details
STEP TEMPLATE PODNAME DURATION MESSAGE
✔ compute-fme-dataset-ensemble-74plc compute-fme-dataset-ensemble
├─┬─✔ compute-fme-dataset-individual(0:0) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-3739697141 8m
│ ├─✔ compute-fme-dataset-individual(1:1) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-1084221249 20m
│ ├─✔ compute-fme-dataset-individual(2:2) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-1804246013 14m
│ ├─✔ compute-fme-dataset-individual(3:3) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-3022590609 12m
│ ├─✔ compute-fme-dataset-individual(4:4) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-1166380877 12m
│ ├─✔ compute-fme-dataset-individual(5:5) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-122836337 12m
│ ├─✔ compute-fme-dataset-individual(6:6) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-2895883749 20m
│ ├─✔ compute-fme-dataset-individual(7:7) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-966649729 13m
│ ├─✔ compute-fme-dataset-individual(8:8) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-3653829861 13m
│ ├─✔ compute-fme-dataset-individual(9:9) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-1517009281 12m
│ ├─✔ compute-fme-dataset-individual(10:10) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-3077512735 12m
│ ├─✔ compute-fme-dataset-individual(11:11) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-1416641317 12m
│ ├─✔ compute-fme-dataset-individual(12:12) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-2023206419 16m
│ ├─✔ compute-fme-dataset-individual(13:13) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-1048689509 11m
│ ├─✔ compute-fme-dataset-individual(14:14) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-2494195159 20m
│ ├─✔ compute-fme-dataset-individual(15:15) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-644008053 12m
│ └─✔ compute-fme-dataset-individual(16:16) compute-fme-dataset-individual compute-fme-dataset-ensemble-74plc-1597493907 20m
├─┬─✔ time-coarsen(0:0) time-coarsen compute-fme-dataset-ensemble-74plc-1485623714 18m
│ ├─✔ time-coarsen(1:1) time-coarsen compute-fme-dataset-ensemble-74plc-1821665790 16m
│ ├─✔ time-coarsen(2:2) time-coarsen compute-fme-dataset-ensemble-74plc-621021970 16m
│ ├─✔ time-coarsen(3:3) time-coarsen compute-fme-dataset-ensemble-74plc-3544939766 19m
│ ├─✔ time-coarsen(4:4) time-coarsen compute-fme-dataset-ensemble-74plc-503450450 17m
│ ├─✔ time-coarsen(5:5) time-coarsen compute-fme-dataset-ensemble-74plc-532802790 15m
│ ├─✔ time-coarsen(6:6) time-coarsen compute-fme-dataset-ensemble-74plc-2860048386 18m
│ ├─✔ time-coarsen(7:7) time-coarsen compute-fme-dataset-ensemble-74plc-950835150 15m
│ ├─✔ time-coarsen(8:8) time-coarsen compute-fme-dataset-ensemble-74plc-4278124130 18m
│ ├─✔ time-coarsen(9:9) time-coarsen compute-fme-dataset-ensemble-74plc-2254453822 18m
│ ├─✔ time-coarsen(10:10) time-coarsen compute-fme-dataset-ensemble-74plc-2173696884 17m
│ ├─✔ time-coarsen(11:11) time-coarsen compute-fme-dataset-ensemble-74plc-3899484094 23m
│ ├─✔ time-coarsen(12:12) time-coarsen compute-fme-dataset-ensemble-74plc-2344054628 36m
│ ├─✔ time-coarsen(13:13) time-coarsen compute-fme-dataset-ensemble-74plc-1270002754 32m
│ ├─✔ time-coarsen(14:14) time-coarsen compute-fme-dataset-ensemble-74plc-654989764 32m
│ ├─✔ time-coarsen(15:15) time-coarsen compute-fme-dataset-ensemble-74plc-4096070886 31m
│ └─✔ time-coarsen(16:16) time-coarsen compute-fme-dataset-ensemble-74plc-2769767140 32m
├─┬─✔ get-stats(0:0) get-stats compute-fme-dataset-ensemble-74plc-957368618 5m
│ ├─✔ get-stats(1:1) get-stats compute-fme-dataset-ensemble-74plc-1597225830 6m
│ ├─✔ get-stats(2:2) get-stats compute-fme-dataset-ensemble-74plc-3313966554 5m
│ ├─✔ get-stats(3:3) get-stats compute-fme-dataset-ensemble-74plc-3011461790 6m
│ ├─✔ get-stats(4:4) get-stats compute-fme-dataset-ensemble-74plc-3196395034 4s
│ ├─✔ get-stats(5:5) get-stats compute-fme-dataset-ensemble-74plc-3008347342 6m
│ ├─✔ get-stats(6:6) get-stats compute-fme-dataset-ensemble-74plc-184258058 12m
│ ├─✔ get-stats(7:7) get-stats compute-fme-dataset-ensemble-74plc-3106895926 11m
│ ├─✔ get-stats(8:8) get-stats compute-fme-dataset-ensemble-74plc-524580586 10m
│ ├─✔ get-stats(9:9) get-stats compute-fme-dataset-ensemble-74plc-1164437798 3m
│ ├─✔ get-stats(10:10) get-stats compute-fme-dataset-ensemble-74plc-94035516 5m
│ ├─✔ get-stats(11:11) get-stats compute-fme-dataset-ensemble-74plc-1709874022 6m
│ ├─✔ get-stats(12:12) get-stats compute-fme-dataset-ensemble-74plc-2534945292 10m
│ ├─✔ get-stats(13:13) get-stats compute-fme-dataset-ensemble-74plc-924009610 12m
│ ├─✔ get-stats(14:14) get-stats compute-fme-dataset-ensemble-74plc-845880428 12m
│ ├─✔ get-stats(15:15) get-stats compute-fme-dataset-ensemble-74plc-311478030 12m
│ └─✔ get-stats(16:16) get-stats compute-fme-dataset-ensemble-74plc-106763308 6m
├───✔ combine-stats combine-stats compute-fme-dataset-ensemble-74plc-781128177 3m
└───✔ upload-beaker-stats upload-beaker-stats compute-fme-dataset-ensemble-74plc-2322081493 6s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR updates the 4-degree SHiELD-SOM ensemble processing configuration for recomputing to zarr v3, and adds a time-coarsening configuration.
Argo job:
compute-fme-dataset-ensemble-74plcSince the argo job completed successfully, I deleted the following two old datasets on GCS:
gs://vcm-ml-intermediate/2024-07-09-vertically-resolved-4deg-c96-shield-som-ensemble-fme-datasetgs://vcm-ml-intermediate/2024-07-09-vertically-resolved-4deg-c96-shield-som-ensemble-fme-dataset-statsChanges