Dataset import: Forest Data Partnership model_2025b cocoa probability (Peru 10m + pan-tropical 100m, 2024)

[kpdavi]

Dataset import: Forest Data Partnership cocoa probability (model_2025b)

Ingest the Forest Data Partnership model_2025b cocoa per-pixel probability layer into cloud-native formats (COG + H3 hex) on NRP.

Source (Earth Engine ImageCollection — NOT a downloadable file)

Commodity	EE asset
Cocoa	projects/forestdatapartnership/assets/cocoa/model_2025b

• Band probability, values 0–1 = probability the pixel includes cocoa for the given year.
• Native 10 m, pan-tropical (−24° to 24° latitude), one image per year 2020–2024.
• Year to ingest: 2024 only.
• Product form: continuous probability (no thresholding; consumers threshold themselves).

This layer lives only inside Earth Engine — there is no public GeoTIFF/GCS mirror. Our cluster (geo-workflows) has NRP creds only and cannot read EE. So each build begins with an Earth Engine Export.image → GeoTIFF → rclone to nrp:public-commodities/raw/, run from an EE-authenticated account (Kirby). Everything after raw/ is the standard cng-datasets raster-workflow (WGS84 COG → H3 hex).

Reducer / value column

• --hex-resampling mean — probability is a per-area intensity (0–1), not an amount. sum would be meaningless. mean = area-weighted mean probability per cell.
• --value-column probability.
• Validation has no global invariant: assert every hex cell value ∈ [0, 1], and spot-check hex mean against the COG over a known producing region.

Target: bucket public-commodities (NEW bucket)

New bucket → run *-setup-bucket before the first stage-raw, and register for MinIO backup + source.coop mirror via PR (Step 7) after publish.

Phase 1 — Peru clip, native 10 m (DO FIRST)

Focus region for the project; also de-risks the EE→COG→hex path on a bounded case before the global run.

• Extent: clip to Peru, --target-extent "-81.4,-18.4,-68.6,0.1" (EPSG:4326).
• Dataset: fdap-2025b/cocoa-peru-2024.
• H3: native res 9, --parent-resolutions "8,0" (columns h9,h8,h0; carries h8, the catalog universal join key). Revised from res 10 → 9 (2026-08-07): res-10 exact_extract over a Peru-sized H3 base cell OOMed at 96Gi (~21M cells/base-cell); res 9 is ~7× lighter and still fully resolves the 10 m source (~300 native px per res-9 cell). Finer than the res-8 global layer, so the Peru focus is preserved.
• COG kept at native 10 m (source resolution unchanged; only the H3 aggregation granularity changed).
• Outside-Peru pixels are stored as NaN (no nodata tag), so hex runs with --nodata nan to exclude them.

Phase 2 — Pan-tropical, coarsened ~100 m

Global catalog layer. Full −24..24 belt at native 10 m is ~8 TB/commodity-year (infeasible); export coarsened to ~100 m (mean) → ~85 GB uncompressed, tractable.

• Extent: −24..24 lat, full longitude (crosses the antimeridian → ensure :latest seam fix).
• Dataset: fdap-2025b/cocoa-2024.
• H3: native res 8, --parent-resolutions "0" (res-8 is the catalog raster standard and the universal join key).
• Coarsen in the EE export via reduceResolution(mean); may need per-longitude-band tiled export.

Phases 1 and 2 meet at h8, so Peru (res 10) rolls up to the global (res 8) for direct comparison.

License

Catalog states CC-BY-4.0 with required attribution "Produced by Google for the Forest Data Partnership" (the 2025a layers were CC-BY-NC; confirm the exact 2025b SPDX before mirroring — it drives source.coop eligibility). Either CC-BY-4.0 or CC-BY-NC-4.0 is source.coop mirror-eligible; label correctly in STAC + license-inventory.md.

Acceptance criteria

• Phase 1: fdap-2025b/cocoa-peru-2024 COG + H3 hex (native res 9; h9,h8,h0) on nrp:public-commodities.
• Phase 2: fdap-2025b/cocoa-2024 COG + H3 hex (native res 8; h0) on nrp:public-commodities.
• Every hex cell probability ∈ [0, 1]; hex mean spot-checked against COG over a known producing region.
• STAC collection + README per dataset; scripts/verify-stac.py exits 0; license verified + labeled.
• public-commodities registered for MinIO backup + source.coop mirror (Step 7 PR); executed by geo-agent-ops.

Requested by @kpdavi. Scope: cocoa only (coffee dropped 2026-08-05).

[kpdavi]

Transfer mechanism (operational): the workflow laptop has kubectl→geo-workflows (aws + rclone-config/nrp secrets) but no local NRP rclone creds, and the cluster has no GCS creds (and must not get any). Bridge: EE export → GCS, object made public-read (or signed URL) → a geo-workflows stage-raw job curls the public https URL and rclones it to nrp:public-commodities/raw/ (standard Step-1b pattern). setup-bucket creates public-commodities on the cluster just before staging — no laptop bucket creation. Multiple EE export shards → repeat --source-url (mosaic in preprocess-cog).

[kpdavi]

Progress checkpoint (2026-08-07)

Done

• EE export (native 10 m, 35 tiles, 86 GB) → GCS kpdavis-fdap-cocoa (safe to delete).
• Mosaicked to one WGS84 COG on NRP: s3://public-commodities/raw/fdap-2025b-cocoa-peru-2024.tif (140824×204174, Float32, ~10 m, 64.7 GB). Outside-Peru = NaN (no nodata tag / no mask band).
• public-commodities bucket created (setup-bucket ran: public-read + CORS).

Hex approach (COG too big for standard per-pod localize)

• 64.7 GB COG > 50Gi ephemeral cap → localize to the shared rechunk-scratch PVC (per-pod cache subdir).
• Peru covers 4 H3 base cells → run only those. Tool --h0-index values (from s3://public-grids/hex/h0-valid.parquet): 37, 91, 96, 112. (NOT the raw h3.get_res0_cells() order.)
• Reducer mean, --value-column probability, --nodata nan (masked pixels are NaN), native res 9 / parents 8,0 (res 10 OOMed at 96Gi).

In progress

• One-cell test (h0-index 96, res 9, 64Gi) running to validate exact_extract + nodata + memory before fanning out to cells 37/91/112.

Remaining

• Validate test → run remaining 3 cells → coverage + value validation (probability ∈ [0,1]; hex mean vs COG) → STAC/README + verify-stac → PR + backup/mirror registration.
• Manifests staged in worktree catalog/commodities/k8s/cocoa-peru-2024/ (stage-mosaic, hex-test; full hex to be finalized). Not yet committed (per commit-when-asked).

[kpdavi]

Hex debugging (2026-08-13)

Two failures found + fixed on the res-9 hex over the 4 Peru base cells (37/91/96/112):

1. OOM (exit 137). The tool sizes exact_extract worker processes to the node core count (misdetects the pod CPU limit) → 256 procs → OOM at 64Gi. Fix: pin CNG_HEX_WORKERS=32, memory 96Gi. (Candidate cng-datasets bug — _cgroup_cpu_count() not honoring the cgroup quota; will file an MRE.)
2. Corrupt localized COG tile. cell 91 failed with TIFFReadEncodedTile ... tile 78665; got 0 bytes. The source COG on NRP is fine (verified by direct /vsicurl read of that tile), but a per-pod rclone localize produced a size-correct-but-truncated cache copy that the tool's size-only cache-hit check reused. Fix: a prep-cache job localizes once to /scratch/shared and verifies every tile with gdalinfo -checksum; all 4 hex pods then share that one verified copy (--local-cache-dir /scratch/shared).

Cell 37 already completed successfully (res 9, 32 workers). Re-running all 4 from the verified shared cache. Manifests: prep-cache, hex (4-cell, CHUNK_MAP 37/91/96/112) in the worktree.