PMTiles step OOMs on disk for very large vector datasets (~38M features); add chunked-PMTiles backend

[cboettig]

Problem

The default cng-datasets workflow PMTiles step is a single-pod pipeline:

ogr2ogr -f GeoJSONSeq /tmp/$DATASET.jsonl /vsicurl/s3:.../$DATASET.parquet
tippecanoe -o /tmp/$DATASET.pmtiles -l $DATASET ... /tmp/$DATASET.jsonl
rclone copy ...

This works fine for ~few-million-feature datasets but fails for very large vector
datasets. Concretely, reprocessing NWI (38,065,251 features, 76 GB merged
GeoParquet on S3) hit two failure modes:

1. Default ephemeral storage (50 Gi from the namespace LimitRange) — pod evicted:

   ephemeral local storage usage exceeds the total limit of containers 50Gi

   The intermediate GeoJSONSeq for 38M features grows past 100+ GB even with
   -select dropping bulky columns, because GeoJSON-text representation expands
   geometries 2-3x relative to compressed parquet.

2. Raised ephemeral request to 150 Gi (the 50 Gi is a LimitRange default, not a
   hard quota, so the pod scheduled). It landed on a node with only 190 GB free of
   869 GB total. The jsonl grew ~1 GB/min; the node would have hit disk-pressure
   eviction before tippecanoe finished reading.

Workaround we used (per-state + tile-join)

For NWI, source.coop already exposes per-state parquet ({ST}_Wetlands.parquet),
so we sidestepped the monolithic step:

• Indexed Job, 51 completions, parallelism 20. Each pod reads its own raw
  per-state parquet directly from S3, runs ogr2ogr -t_srs EPSG:4326 -select ...,
  then tippecanoe -l <dataset> -z14 --drop-densest-as-needed ..., uploads to
  s3://<bucket>/<dataset>-state-tiles/{ST}.pmtiles.
• Per-pod resources: 4 CPU, 16 Gi RAM, 50 Gi ephemeral — fits on any node.
• Merge job: tile-join -o $DATASET.pmtiles -f *.pmtiles concatenates all 51
  single-layer tiles into one final PMTiles. Tile-join is fast (~5 min for the
  13 GiB of per-state tiles).

Each per-state job processed 1.5K (DC) - 1.6M (AK) features in 2-15 min; the whole
51-state batch took ~71 min (AK was the longest pole). Final merge ~5 min.

Proposed enhancement: chunked-PMTiles backend

Add a chunked PMTiles backend to cng-datasets workflow, mirroring the existing
indexed-Job pattern used for hex. Suggested CLI:

cng-datasets workflow \
  --dataset nwi \
  --source-url s3://bucket/nwi.parquet \
  --bucket public-wetlands \
  --pmtiles-backend chunked \
  --pmtiles-chunk-by state_code     # or: h0 | bbox | row_groups
  --pmtiles-completions 51 \
  --pmtiles-parallelism 20

The generator would emit:

• <dataset>-state-tiles.yaml — indexed Job; each pod reads its chunk (filter on
  --pmtiles-chunk-by column), runs ogr2ogr → tippecanoe, uploads a sub-PMTiles
  to a staging prefix.
• <dataset>-pmtiles-merge.yaml — single pod, downloads staged sub-PMTiles, runs
  tile-join, uploads the final canonical PMTiles.

Design notes

• Auto-detect when chunking is needed: feature count > threshold (e.g. 5M)
  triggers chunked backend automatically; the user can override either way with
  --pmtiles-backend monolithic|chunked.
• Layer name consistency: every per-chunk tippecanoe -l <layer> MUST agree so
  tile-join produces a single-layer output (otherwise you get N layers in the
  merged PMTiles, which breaks MapLibre styling).
• Default chunking columns: state_code, country_code, h0, region — any
  string/int column with low cardinality and roughly balanced row counts. If none
  exists, fall back to row-group chunking using parquet metadata.
• Attribute pruning: when the dataset has many columns, default -select to a
  small whitelist of small string/int attrs. Bulky columns (Shape_Length,
  Shape_Area, doubles in general) should be dropped from tiles since
  vector-tile rendering doesn't use them — restyling joins back via the FID.
• Per-pod resources: 4 CPU, 16 Gi RAM, 50 Gi ephemeral is a safe default for
  chunks up to ~3M features.

Reference numbers (NWI, 38M features)

Approach	Wall-clock	Peak ephemeral / pod	Outcome
Monolithic (50 Gi default)	~80 min before evict	>50 Gi	Failed (evicted)
Monolithic (150 Gi req)	est. ~3 hr	est. 150+ Gi	Risk of node disk-pressure eviction; cancelled at 60 Gi jsonl
Per-state + tile-join	71 min (51 pods,		=20) + 5 min merge

Final PMTiles size: ~13 GiB (sum of per-state) before merge; tile-join output TBD
(probably similar or smaller after dedup).

Files (reference)

For reference, the YAMLs we used are in boettiger-lab/data-workflows:

• catalog/wetlands/k8s/nwi-v2/per-state-pmtiles.yaml
• catalog/wetlands/k8s/nwi-v2/pmtiles-merge.yaml

These would be a reasonable template for the generator.

cc @boettiger

[cboettig]

Concrete case (2026-06-20): national FEMA NFHL flood-hazard layer

Another large-vector dataset that the generated PMTiles step can't handle as-is (data-workflows #253):

• s3://public-hazard/flood-hazard.parquet — 5,631,998 MultiPolygons, 51.6 GiB (full FIRM vertex precision).
• The generated *-pmtiles.yaml does ogr2ogr … -f GeoJSONSeq /tmp/$DATASET.geojsonl /vsicurl/<public-url> then tippecanoe. For this input the intermediate GeoJSONSeq is hundreds of GB, far over the 50 GiB namespace ephemeral cap — and the /vsicurl read of 51 GiB over the public endpoint (~12 MB/s) alone is ~70 min.

So this is a second real trigger for the chunked-PMTiles backend this issue asks for. In the meantime I'm working around it by simplifying geometry first (see the companion feature request to add --simplify-tolerance to convert/workflow) — at H3 res-10 (~65 m cells) a ~10 m simplification is lossless for hex and only trims sub-display-zoom vertices for tiles, which also shrinks the PMTiles input by ~5–10×. Chunked tippecanoe + optional pre-simplification together would close this for national-scale vector.

[cboettig]

Clarification on this case: the ephemeral overflow is mitigable by redirecting the GeoJSONSeq intermediate to the shared rechunk-scratch PVC (2 Ti RWX) instead of /tmp, so the 50 GiB ephemeral cap isn't a hard barrier here — I'll wire the flood PMTiles step that way. The stronger driver for the chunked-PMTiles backend is tippecanoe time/memory at ~38M-feature scale, not disk; my 5.6M-feature (pre-simplification) case is comfortably PVC-solvable. Flagging so the issue scope stays accurate.