raster-workflow: built-in multi-year / time-series pattern (per-year COGs + year=-partitioned unified hex)

[cboettig]

Feature request: built-in multi-year / time-series raster workflow (year=-partitioned unified collection)

Problem

raster-workflow handles the single-raster case well: one source → one WGS84 COG → one 122-completion h0 hex → one collection with a single COG + hex asset. But a growing class of catalog datasets are annual / time-series rasters — the same product across many years — and there is no built-in pattern for them. Today each one is built by hand-writing indexed Jobs and orchestrating the fan-out manually.

Reference build in flight now: Annual NLCD land cover, 19 years (1985–2025) at C1V2 (data-workflows #453). Others that will want the identical shape: MTBS annual burn severity, Hansen/GFC annual tree-cover loss, LCMAP, RAP annual cover, JRC Global Surface Water yearly, ESA CCI annual land cover, etc.

The target layout (already the catalog convention)

We converged on the same structure the multi-year GFW fishing-effort dataset (public-gfw/gfw-fishing-effort) uses, and it's also what the geo-agent versions dropdown needs (it resolves asset ids within one collection — geo-agent #323):

• One collection, per-year COG assets …-cog-<year> → drives the app year/versions dropdown.
• One unified hex (and hex-fractions) with year as a queryable column, so WHERE year=YYYY filters and cross-year change is a self-join on the h-cell — no separate change dataset.
• Physical layout at res-10 scale: …/hex/year=<year>/h0=<cell>/data_0.parquet (year as a hive partition so it needs zero post-hoc rewrite; GFW merges into an in-file year column, but that all-years merge is infeasible at res-10 / ~10B rows — a partition column is the scalable equivalent, same query UX).

What I'm doing by hand today (the pattern to automate)

Three indexed Jobs (manifests: data-workflows:catalog/land-cover/k8s/nlcd/):

1. COG: completions=<n_years>, one year per pod, gdalwarp -t_srs EPSG:4326 -r near -of COG → …/<name>-<year>-cog.tif.
2. hex and hex-fractions: completions = n_years × 122, index → (year=YEARS[idx/122], h0=idx%122), each pod runs cng-datasets raster --input …-<year>-cog.tif --output-parquet …/hex/year=<year>/ --h0-index <h0> ….

Proposed interface (sketch)

cng-datasets raster-workflow \
  --dataset land-cover/nlcd --bucket public-land-cover \
  --source-url 's3://…/Annual_NLCD_LndCov_{year}_CU_C1V2.tif' \
  --years 1985,1990,1994,1996,1998,2000,2005,2010,2015-2025 \
  --partition-by year \
  --value-column nlcd --hex-resampling mode --resolution 10 --parent-resolutions 9,8,7,6,5,0

• --years (list + ranges) with a {year} (and {version}) template in --source-url.
• --partition-by year → emit per-year COG assets + the year=-partitioned unified hex, and generate a STAC collection with the per-year COG asset ids + a single year-column hex asset (+ versions hint).
• Reuse the existing --hex-resampling / --nodata / resolution machinery unchanged.

Gotchas to bake in (all hit on the NLCD build — free hardening)

• Don't override PROJ_DATA/PROJ_LIB. The image ships 8 proj.db at differing layout versions; a find … | head -1 non-deterministically picks a stale one and breaks -t_srs on ~10% of pods. GDAL's default resolves /usr/local/gdal-internal/share/proj/proj.db correctly.
• Blip tolerance on the indexed Jobs: backoffLimitPerIndex + a high maxFailedIndexes (a transient Ceph RGW blip that kills a few indexes must not abort the siblings), plus rclone --retries … --low-level-retries … --retries-sleep.
• Cross-year coverage/consistency gate: per-year populated-h0 set should match a reference (all years share the CONUS extent — 6 populated h0, identical h8 set), and --partition-by year makes a natural GROUP BY year verification target. (Extends check-hex-coverage.sh to the multi-year case.)
• Version consistency: encourage building a whole series from one collection version (mixed versions break cross-year diffs).

Why file rather than patch

Per data-workflows AGENTS Hard-Boundary-2 I don't hotfix the tool. This is a clean generalization of raster-workflow; happy to keep the NLCD manifests as the working reference and review a design. Not urgent — the current build is proceeding fine by hand; the ask is to make the next annual-raster import a one-liner instead of hand-written indexed Jobs.

Refs: data-workflows #453 (reference build), geo-agent #323 (single-collection versions dropdown), public-gfw/gfw-fishing-effort (existing multi-year exemplar).

[cboettig]

Measured RAM (res-10 CONUS hex, from the NLCD build) — the 32Gi/256Gi defaults are mis-sized two ways:

Live kubectl top on the running heavy pods: 105–141 GiB actual (median ~130), never near the 256Gi request. NRP 256Gi nodes have only ~250Gi allocatable, so a 256Gi request barely places → chronic FailedScheduling. 192Gi covers the peak with margin and widens the schedulable pool a lot.

Bigger structural win — tier by h0 occupancy. In a 122-completion (or N×122 multi-year) raster hex, only the handful of h0 base cells that contain land do heavy work; the rest are empty ocean cells that finish in seconds but, under a flat request, still demand a 256Gi slot. So ~95% of pods needlessly hold huge reservations. The tool could:

• request the big memory only for populated h0 (derive the populated-h0 set from the COG's footprint up front), and give empty cells ~8–16Gi; or
• emit two Jobs (light + heavy) split on that set.
  That would collapse the scheduling bottleneck far more than any flat right-size. For the multi-year pattern, the split is per (year × populated-h0).

Suggested defaults for res-10 CONUS-scale mode/fractions: ~192Gi for populated cells (not 256), ~8–16Gi for empty. Measured on data-workflows #453.

[cboettig]

Companion design issue: #173 — memory-bounded adaptive chunking (finer H3-parent chunks + windowed COG reads + Armada) to end big-instance oversizing. Orthogonal to this one: #172 is the multi-year output structure (per-year COGs + year= hex), #173 is how each large raster hex is chunked to bound RAM. Both target the same goal — large imports as an automated mechanism, not custom handling — and compose (chunk-resolution is per year × sub-chunk).