Design a standard multi-resolution H3 hex representation (IUCN richness, iNat ranges)

[cboettig]

Some datasets are inherently multi-resolution: they contain features spanning many orders of magnitude in size, so no single native H3 resolution works.

• Too fine for a global / continent-scale feature → astronomical cell counts and OOM (e.g. a globally-distributed species range at res 8).
• Too coarse for a tiny feature → the feature collapses to a handful of cells or vanishes (e.g. a narrow-endemic range, a small wetland, at res 5).

Forcing one resolution is therefore either hopeless or lossy. We need a standard, documented way to represent and query multi-resolution hex across the catalog — today each such dataset invents its own custom handling.

Concrete cases

dataset	today	problem
IUCN richness (public-iucn)	two homes: res-8 richness/hex and res-5 hex/	not a duplicate to reconcile — a genuine multi-res need with no standard representation or documented query path
iNat ranges (public-inat/range-maps/hex)	single native res 4	coarse; under-serves small-range taxa
swap/nevada/mixres.py	bespoke multi-resolution assembler	prior art — literally hand-rolls what a standard should provide

Design questions to resolve

1. Representation. One asset with a per-row/per-feature resolution marker? Separate assets per resolution with a documented join convention? A single "mixed-resolution" hex where each feature is stored at the finest resolution it can afford, carrying parents for roll-up joins?
2. Query pattern. How does a consumer aggregate/join without knowing which resolution a given feature lives at? Must be reliable for LLM agents (the audience for these STAC descriptions).
3. cng-datasets support. What would an adaptive/per-feature resolution mode look like in the standard build (likely an enhancement on boettiger-lab/datasets)? File the enhancement once the representation is chosen.
4. STAC. h3:native_resolution is singular today; extend the convention to declare a resolution set / adaptive scheme on the asset.

Acceptance

• A written standard for multi-resolution hex: representation + query pattern + STAC declaration.
• Any required cng-datasets enhancement filed and linked.
• IUCN richness re-expressed in the standard form (retires the ad-hoc two-location split and catalog/iucn/k8s/iucn-reencode-hex.yaml).

Blocks / unblocks

These derivation jobs stay on custom COPY … PARTITION_BY (h0) code until this design (and the reducers it implies) land in the standard routines — they do real derived work, so they can't just be deleted:

• high-seas add-parents / gfw aggregate·rename·consolidate
• plant-richness / rarity-weighted-endemic-plant-richness p80-percentile hex
• gbif/2026-06/process_gbif_h3.py, overturemaps/.../derive-countries-hex.yaml
• catalog/iucn/k8s/iucn-reencode-hex.yaml (held pending the IUCN re-expression)

---

Background — completed migration work (#332 follow-up, now done)

This issue began as the #332 follow-up ("which pipeline writes iNat / IUCN so fragmentation doesn't recur"). The answer: hex fragmentation came from custom COPY … TO '<dir>/' (PARTITION_BY (h0)) jobs (multi-thread → one file per thread per partition), not cng-datasets — the standard raster/vector/repartition routines already write one data_0.parquet per h0. All of that is resolved:

• Data: catalog-wide sweep found no fragmentation and no orphans anywhere; iNat was the only dataset with real orphan cruft (hexagon/, hex/, polygon/, ~6.8 GiB) — purged.
• Manifests: 12 obsolete VARCHAR→UBIGINT reencode jobs retired (inat, wetlands×3, carbon×3, ncp, hydrobasins, overturemaps, wyoming-nlcd, wdpa) — merged in chore(#548): retire 12 obsolete reencode-hex jobs (inat, wetlands, carbon, +5) #551.

What is left is purely the multi-resolution design above — no live data defect.

Refs #332, #551.

[cboettig]

iNat orphan cleanup — done

Purged the retired custom-notebook artifacts, all verified unreferenced by STAC and superseded by the standard-built range-maps/hex/:

prefix	objects	size
hexagon/	122	1.314 GiB
hex/	27	2.215 GiB
polygon/	27	3.291 GiB

~6.8 GiB reclaimed. Survivors under public-inat/: range-maps/ (canonical hex, untouched), taxon/ + taxonomy/ (STAC assets), ranges.parquet (vector source), README.md, stac-collection.json.

Safety basis: range-maps/hex/ and hexagon/ both held 100,291 taxa with 0 taxa unique to hexagon, and range-maps/hex/ carries _cng_fid + one data_0.parquet per h0 (standard build). iNat is now fully off the custom pipeline; catalog/inat/inat-rangemaps.ipynb and catalog/inaturalist/k8s/inat-reencode-hex.yaml are obsolete and can be removed from the repo in a follow-up commit.

Remaining on this issue: the other custom *-reencode-hex.yaml writers (iucn, wetlands, carbon, ncp, overturemaps, hydrobasins, wdpa, high-seas) each need the same verify-canonical-is-standard → purge-orphans → retire pass, plus the multi-resolution design question for IUCN richness / iNat ranges.

[cboettig]

Wetlands — verify→purge→retire done (PR #551)

All three wetlands sub-datasets checked. Cleaner than iNat — no orphans, nothing to purge.

dataset	canonical hex	_cng_fid	single-file	H3 int	verdict
ramsar	ramsar/hex (native 9)	✅	83/83	✅	standard, clean
nwi	nwi-v2/hex (native 10)	✅	10/10	✅	standard, clean
glwd	glwd/hex (native 8)	— (raster, exempt)	98/98	✅	standard, clean

• The NWI reencode job read public-wetlands/nwi/hex/**, but that prefix no longer exists — canonical is nwi-v2/. No orphan to purge.
• No hex-new/ staging leftovers anywhere. Wetlands hex was already deduped in Audit + rebuild vector hex assets built before 2026-07-12: duplicate (feature, cell) rows from multipart polyfill #509 and was never in the Consolidate multi-shard h0 partitions in published hex datasets #332 fragmented set.
• The 4 obsolete reencode manifests (3 wetlands + inat) are removed in PR chore(#548): retire 12 obsolete reencode-hex jobs (inat, wetlands, carbon, +5) #551.

One thing to look at separately: glwd/hex-fractions exists on S3 but is not in the glwd STAC (assets are glwd-hex, glwd-area-hex, glwd-class-area-hex). It is probably a class-area build intermediate rather than a reencode orphan — worth confirming whether it is referenced by the glwd-class-area-consolidate build or is deletable, but out of scope for the reencode pass.

Remaining reencode datasets to run this pass on: carbon, ncp, overturemaps, hydrobasins, wdpa, high-seas.

[cboettig]

Carbon — verify→purge→retire done (folded into PR #551)

Third dataset. Like wetlands, already fully converged — no orphans, nothing to purge (public-carbon was cleaned in #330).

• Verify ✅ all 10 canonical density-carbon hex assets (irrecoverable/vulnerable/manageable × 2010/2018/2022/2023/2024) are single-file per h0, integer H3 (h0 BIGINT, h5–h9 UBIGINT), raster-derived (carbon value, _cng_fid-exempt, mean reducer).
• Purge — nothing: only hex dir names exist bucket-wide, no hex-new/ staging leftovers, no v1 irrecoverable-carbon/hex (purged in public-carbon: validate v2 time series vs published totals + ingest gaps (uncertainty/ecosystem/2010) + purge ~143 GiB stale #330).
• Retire — 3 obsolete jobs removed in PR chore(#548): retire 12 obsolete reencode-hex jobs (inat, wetlands, carbon, +5) #551: the two per-dataset reencoders + carbon-v2-sequencer.yaml. The sequencer was a completed one-off that drove the standard per-dataset hex workflows then appended the reencode; re-running it would re-fragment. The standard per-dataset workflow manifests remain as the build recipe.

Progress on this issue: iNat ✅ (purged), wetlands ✅ (clean), carbon ✅ (clean). Remaining reencode datasets: ncp, overturemaps, hydrobasins, wdpa, high-seas. So far every already-migrated dataset (wetlands, carbon) needed only retirement — the only one with real orphan cruft was iNat.

[cboettig]

Full catalog sweep — data side is clean everywhere

Swept every remaining custom-PARTITION_BY (h0) target bucket for the two failure signatures (fragmentation, -new//staging leftovers). All clean.

bucket	hex files/dirs	fragmentation	staging leftovers
public-ncp	171/171	none	none
public-overturemaps	527/527	none	none
public-hydrobasins	648/648	none	none
public-wdpa	144/144	none	none
public-iucn	3140/3140	none	none
public-gbif	244/244	none	none
public-high-seas	1608/1608	none	none
public-swap	20/20	none	none
public-wyoming	41/41	none	none
public-ca30x30 (plant-richness, rarity-weighted)	2/2 each	none	none

hex_files == hex_dirs in every bucket ⇒ single-file per h0. So no orphans to purge and no fragmentation remains catalog-wide — iNat (already fixed) was the only dataset with real orphan cruft.

Remaining custom-COPY manifests split into two classes

(a) Pure VARCHAR→UBIGINT reencode — obsolete, safe to retire (same as the inat/wetlands/carbon ones already removed in PR #551):

• catalog/ncp/k8s/ncp-reencode-hex.yaml
• catalog/hydrobasins/k8s/hydrobasins-reencode-hex.yaml
• catalog/overturemaps/k8s/overturemaps-reencode-hex.yaml
• catalog/wyoming/k8s/nlcd-2024/nlcd-2024-reencode-hex.yaml
• catalog/wdpa/k8s/wdpa-reencode-hex.yaml — doubly moot: it targets wdpa-december-2025, which was purged in Consolidate multi-shard h0 partitions in published hex datasets #332.
• catalog/iucn/k8s/iucn-reencode-hex.yaml — obsolete as a reencode, but tied to the IUCN multi-res design question below; hold with that.

(b) Real derivation jobs that happen to use PARTITION_BY (h0) — NOT obsolete reencodes (add-parents, aggregate, p80, mixres). These produce genuine derived data and need per-case judgement on whether cng-datasets supports the operation, so they are not simple deletes:

• high-seas: high-seas-cleanup-parents, eez/iho-eez-add-parents, high-seas/iho-high-seas-add-parents, vme/rfmo-vme-add-parents, meow/meow-ecoregions-add-parents, gfw/gfw-{aggregate,rename-columns,consolidate}
• plant-richness: plant-richness-p80-hex, rarity-weighted-endemic-plant-richness-p80-hex (p80 percentile derivation)
• swap/nevada/mixres.py (multi-resolution assembly), gbif/2026-06/process_gbif_h3.py, overturemaps/.../derive-countries-hex.yaml

Multi-resolution design (unblocks IUCN + swap mixres)

swap/nevada/mixres.py is literally a multi-resolution hex assembler — the same need called out for IUCN richness and iNat ranges. Standardising multi-res hex output (likely a cng-datasets enhancement) is the prerequisite for migrating class (b) cleanly.

Bottom line: the fragmentation follow-up from #332 is fully resolved on the data side. What remains under this issue is manifest hygiene (class a) plus the multi-res design (class b), neither of which is a live data defect.

[cboettig]

Status: class (a) complete

PR #551 now retires 12 dead reencode jobs (inat, wetlands ×3, carbon ×3, ncp, hydrobasins, overturemaps, wyoming-nlcd, wdpa). All verified against live data first: single-file per h0, integer H3.

Data side of this issue is fully done — catalog-wide sweep found no fragmentation and no orphans anywhere (iNat cruft purged, ~6.8 GiB).

What remains on this issue (design work, no live defect)

• Multi-resolution hex standardization — the real open question. IUCN richness (res-8 richness/hex + res-5 hex/), iNat ranges (coarse native), and swap/nevada/mixres.py all need a genuine multi-res representation; likely a cng-datasets enhancement. iucn-reencode-hex.yaml is held pending this.
• Class (b) derivation jobs — high-seas add-parents/gfw, plant-richness p80, gbif process_gbif_h3, overturemaps derive-countries. These produce real derived data via PARTITION_BY (h0); migrating them to standard routines depends on the multi-res design and on cng-datasets supporting add-parents / p80 / percentile reducers.

Recommend spinning the multi-res design into its own focused issue when ready.