Tracking: cng-datasets tooling roadmap (hex-write, ingest hardening, scale)

[cboettig]

Canonical roadmap for the remaining cng-datasets tooling/source work. This is the durable source of truth for sequencing and cross-issue dependencies — individual issues hold their own detail. Keep all work generic (this repo is the engine + workflow generators; dataset-specific pipelines live in data-workflows).

Last updated: 2026-06-24.

Recently completed (context)

Closed via #109 (#108), #110 (#87/#79/#35), #111 (#77), #113 (#104); #66/#68 verified already-fixed in #74; CI network flakiness fixed in #112.
Then #115 (#102 + #103), #116 (#43 — parquet-input path; ST_Read path was already done), #117 (#61 — parquet-input BLOB→GEOMETRY cast; ST_Read path was already done in #75).
v0.2.0 released (#118) — first tagged release.
Then #120 (#119 — clear error for unsupported geometry encodings on the parquet-input path), #121 (#107 — oversized per-feature H3 cell-array guardrail).
Then #135 (#106 — row-group byte cap workaround; #136 tracks the upstream revisit), #137 (#134 duckdb 1.5.4 bump + #130 GDB-in-zip/generic-s3 + #131 non-EPSG SRS), #138 (#122 ST_MakeValid at convert time), #139 (#133 bounded PMTiles max zoom).

Open work, grouped

A. Hex-write & H3 indexing (correctness + query perf)

• Assert all physical H3 index columns are UBIGINT after hex build (h0 exempt) #102 — assert every physical h{N≥1} column is UBIGINT after hex build (h0 exempt) — fix(#102,#103): UBIGINT hex-column assertion + intra-partition _cng_fid ordering #115
• Hex write: ORDER BY _cng_fid within each h0 partition for row-group pruning #103 — ORDER BY _cng_fid within each h0 partition for row-group pruning — fix(#102,#103): UBIGINT hex-column assertion + intra-partition _cng_fid ordering #115
• Always create _cng_fid: source fid columns are not reliably unique #43 — always create _cng_fid (monotonic row id); source fid may be a feature key, not row-unique — fix(#43): always synthesize _cng_fid on the parquet-input convert path #116
• Adaptive variable-resolution H3 polyfill for very large polygons (fine edges, coarse interior, no OOM/page-size) #98 (Plan A) — --resolution-by-area size-stratified variable-resolution polyfill + per-feature Oversized-geometry safety guardrail for the 2 GB Arrow/parquet-page limit (graceful, not a C++ assertion) #107 back-off — feat(#98): adaptive variable-resolution H3 polyfill via --resolution-by-area #123. Deferred follow-ups split out: Auto-scale hex chunk-size by feature size (cells-per-chunk budget) #124 (cells-per-chunk auto-scaling) and Recursive classify-and-descend H3 polyfill for single oversized features #125 (recursive classify-and-descend for a single feature too big even for the coarsest tier).
• Auto-scale hex chunk-size by feature size (cells-per-chunk budget) #124 — auto-scale hex chunk-size by feature size (cells-per-chunk budget); the Adaptive variable-resolution H3 polyfill for very large polygons (fine edges, coarse interior, no OOM/page-size) #98 comment's "Key lesson"
• Recursive classify-and-descend H3 polyfill for single oversized features #125 — recursive classify-and-descend polyfill for a single oversized feature (the Adaptive variable-resolution H3 polyfill for very large polygons (fine edges, coarse interior, no OOM/page-size) #98 body's long-term algorithm)

B. Geometry ingest hardening

• Hardened geometry ingest/rebuild: route through GDAL/WKB (not DuckDB COPY) + make_valid/antimeridian/CRS84 — kill the stoi landmine #106 — BLOCKED on an MRE. The stoi-crash narrative (geopandas-writer discriminator + antimeridian multipolygon) did not reproduce on small files with the pinned duckdb 1.5.3 — see the Hardened geometry ingest/rebuild: route through GDAL/WKB (not DuckDB COPY) + make_valid/antimeridian/CRS84 — kill the stoi landmine #106 comment. DuckDB stays the standard; no speculative GDAL-replacement fix. Needs the actual offending iucn-ranges-2025 geometry to re-diagnose; any fix should be DuckDB-first (clear error + convert-to-compatible-format fallback).
• Oversized-geometry safety guardrail for the 2 GB Arrow/parquet-page limit (graceful, not a C++ assertion) #107 — graceful guardrail for the 2 GB Arrow/parquet-page limit (clear error, not a C++ assertion) — fix(#107): graceful guardrail for oversized per-feature H3 cell arrays #121. Failure-mode fixing raster to h3 conversion for mobi example #2 (GDAL ogr2ogr write) is moot: no GDAL write path exists under the DuckDB-only standard.
• cng-convert-to-parquet writes BLOB instead of GEOMETRY for MULTIPOINT sources — breaks PMTiles step #61 — cng-convert-to-parquet writes BLOB instead of GEOMETRY for MULTIPOINT (missing GeoParquet metadata) → breaks PMTiles — fix(#61): cast BLOB WKB geometry to GEOMETRY on the parquet-input convert path #117
• convert_to_parquet silently passes geoarrow-native/WKT geometry through as a non-GEOMETRY column (no error, no GeoParquet metadata) #119 — convert silently passed geoarrow-native/WKT geometry through as a non-GEOMETRY column; now a clear error — fix(#119): clear error for unsupported geometry encodings on the parquet-input path #120 (found while building MREs for Hardened geometry ingest/rebuild: route through GDAL/WKB (not DuckDB COPY) + make_valid/antimeridian/CRS84 — kill the stoi landmine #106)
• cng-convert-to-parquet leaves invalid input geometries un-repaired (no ST_MakeValid) #122 — ST_MakeValid on all geometry expressions at convert time (repairs invalid input geometries at the source) — fix(#122): ST_MakeValid on all geometry expressions at convert time #138
• cng-convert-to-parquet: zipped File Geodatabase rejected + s3:// rewrite only matches hard-coded substrings #130 — zipped File Geodatabase ingest + generic s3://→https rewrite (no longer gated on hard-coded substrings) — fix(#134,#130,#131): duckdb bump, GDB-in-zip support, non-EPSG SRS crash #137
• cng-convert-to-parquet: support CSV with lat/lon columns as point geometry input #78 — support CSV with lat/lon columns as point geometry input

B2. Dependency / upstream tracking

• bump: duckdb upper-bound cap is stale — h3 community extension now ships for 1.5.4 #134 — duckdb pin bumped to ==1.5.4 (h3 community ext now ships) — fix(#134,#130,#131): duckdb bump, GDB-in-zip support, non-EPSG SRS crash #137
• Revisit row-group byte cap once upstream httpfs fix (duckdb#23408) lands #136 — revisit the fix(#106): cap row-group bytes so geometry chunks stay httpfs-readable #135 row-group byte cap once upstream duckdb#23408 lands (blocked upstream)

C. Raster

• Optional: guardrail + convenience for density (Mg/ha) rasters (mean already works; sum is the trap) #105 — guardrail + convenience for density (Mg/ha) rasters (mean works; sum is the trap)
• raster: AutoIdentifyEPSG() crashes ('Unsupported SRS') on valid rasters with no EPSG code (ESRI:102003, sphere Web-Mercator) #131 — AutoIdentifyEPSG() crash on valid EPSG-less SRS (ESRI:102003) now caught — fix(#134,#130,#131): duckdb bump, GDB-in-zip support, non-EPSG SRS crash #137

D. Scale / backends

• PMTiles step OOMs on disk for very large vector datasets (~38M features); add chunked-PMTiles backend #83 — chunked-PMTiles backend for very large vector datasets (~38M features OOM on disk)
• Hybrid workflow orchestrator: support Armada backend for hex step #39 — Armada backend for the hex step in the hybrid orchestrator

E. PMTiles publish metadata

• PMTiles: bound max zoom to justified detail; default off the unbounded --extend-zooms thrash #133 — bound PMTiles max zoom; drop unbounded --extend-zooms-if-still-dropping — fix(#133): bound PMTiles max zoom; drop --extend-zooms-if-still-dropping #139
• PMTiles assets ship with empty table:columns — populate tile-accurate fields from the footer at publish time #140 — populate tile-accurate table:columns from the PMTiles footer at publish time (tiles carry a column subset vs source parquet)
• Add geometry simplification (--simplify-tolerance) to vector convert/workflow #132 — add --simplify-tolerance geometry simplification to convert/workflow

Recommended sequence & dependencies

1. Assert all physical H3 index columns are UBIGINT after hex build (h0 exempt) #102 + Hex write: ORDER BY _cng_fid within each h0 partition for row-group pruning #103 ✅ done (fix(#102,#103): UBIGINT hex-column assertion + intra-partition _cng_fid ordering #115) — hex-write assertion + intra-partition _cng_fid ordering.
2. Always create _cng_fid: source fid columns are not reliably unique #43 ✅ done (fix(#43): always synthesize _cng_fid on the parquet-input convert path #116) — parquet-input path now always synthesizes _cng_fid.
3. cng-convert-to-parquet writes BLOB instead of GEOMETRY for MULTIPOINT sources — breaks PMTiles step #61 ✅ done (fix(#61): cast BLOB WKB geometry to GEOMETRY on the parquet-input convert path #117); convert_to_parquet silently passes geoarrow-native/WKT geometry through as a non-GEOMETRY column (no error, no GeoParquet metadata) #119 ✅ done (fix(#119): clear error for unsupported geometry encodings on the parquet-input path #120) — convert-step geometry-encoding hardening.
4. Oversized-geometry safety guardrail for the 2 GB Arrow/parquet-page limit (graceful, not a C++ assertion) #107 ✅ done (fix(#107): graceful guardrail for oversized per-feature H3 cell arrays #121). Hardened geometry ingest/rebuild: route through GDAL/WKB (not DuckDB COPY) + make_valid/antimeridian/CRS84 — kill the stoi landmine #106 is blocked on an MRE (see above) — do not implement without a reproducer.
5. Adaptive variable-resolution H3 polyfill for very large polygons (fine edges, coarse interior, no OOM/page-size) #98 ✅ Plan A done (feat(#98): adaptive variable-resolution H3 polyfill via --resolution-by-area #123): --resolution-by-area + per-feature Oversized-geometry safety guardrail for the 2 GB Arrow/parquet-page limit (graceful, not a C++ assertion) #107 back-off. Remaining Adaptive variable-resolution H3 polyfill for very large polygons (fine edges, coarse interior, no OOM/page-size) #98 scope tracked in Auto-scale hex chunk-size by feature size (cells-per-chunk budget) #124 (cells-per-chunk auto-scaling) and Recursive classify-and-descend H3 polyfill for single oversized features #125 (recursive classify-and-descend).
6. PMTiles step OOMs on disk for very large vector datasets (~38M features); add chunked-PMTiles backend #83 / Hybrid workflow orchestrator: support Armada backend for hex step #39 / Optional: guardrail + convenience for density (Mg/ha) rasters (mean already works; sum is the trap) #105 / cng-convert-to-parquet: support CSV with lat/lon columns as point geometry input #78 — larger or independent; schedule as needed.

Note: #43 and #61 were both half-landed — their ST_Read convert paths were already fixed, and the recent PRs closed the matching gap on the parquet-input path (process_parquet_input). #119 was a third instance of the same ST_Read-vs-parquet asymmetry. Worth auditing remaining convert-step features for it.

Checkboxes here track high-level status; detail and decisions go in each linked issue and its PR.

[cboettig]

#98 — --resolution-by-area (Plan A) implemented in #123.

Shipped the proven size-stratified variable-resolution H3 hex: each feature is hexed at the native resolution its planar ST_Area maps to, output carries a uniform union schema + native_res column (flat-join preserving), and the #107 oversized-feature guard now backs off per-feature. This is the recipe behind the published iucn-ranges-2025/hex product.

Two pieces of #98 are deferred to dedicated follow-ups:

• Auto-scale hex chunk-size by feature size (cells-per-chunk budget) #124 — auto-scale chunk-size by feature size (cells-per-chunk budget; the comment's "Key lesson").
• Recursive classify-and-descend H3 polyfill for single oversized features #125 — recursive classify-and-descend polyfill for a single feature too big even for the coarsest tier.

Group A is now essentially complete pending #123 merge; remaining open roadmap work is group B (#106 still blocked on an MRE, #78), C (#105), D (#83, #39), plus the two new #98 follow-ups.