STAC descriptions: surface field semantics & fix misleading metadata that cause wrong LLM answers

[cboettig]

STAC dataset descriptions: surface field semantics & fix misleading metadata that cause wrong LLM answers

From a 6-model × 34-question geo-agent accuracy sweep where we computed ground truth ourselves (Opus running authoritative duckdb-geo queries) and traced each wrong answer to its cause. A cluster of failures comes from per-dataset STAC metadata that is missing, misleading, or under-documented — fixable here in datasets. (Companion cross-cutting tool/H3-guide issue filed on boettiger-lab/mcp-data-server; analysis driven from open-llm-proxy.)

Concrete metadata fixes, each tied to a confirmed wrong answer

1. Global Fishing Watch geartype enum in STAC is INCOMPLETE and actively misleading.
The documented value list in get_stac_details omits drifting_longlines and squid_jigger — which are the two largest gears in the data (≈75% + ≈14%) of fishing effort displaced by high-seas EBSA MPAs. A model trusting the documented enum would miss the dominant categories entirely.
Fix: regenerate the enum from the actual column values, or mark it explicitly non-exhaustive with the top values listed.

2. Ramsar Criterion fields have no documented meaning — models guess and get it wrong.
For "show Ramsar sites meeting Criterion 9; explain the criterion," models gave 4 different wrong definitions. The correct definition (Criterion 9 = wetland regularly supporting ≥1% of a population of a wetland-dependent non-avian animal species) already exists in this repo at catalog/wetlands/ramsar/criteria.md — it's just not surfaced in the STAC description the agent sees.
Fix: fold the Criterion1–9 definitions into the wetlands-ramsar STAC column docs. Also strengthen the existing dedup note: despite the "duplicate hex rows" note, models still used COUNT(*) (8,347 rows vs 2,551 distinct ramsarid) — make the dedup note apply to the GeoParquet too, not just hex.

3. sponsor_type code vocabulary is inconsistent between related TPL tables.
The conservation-almanac-2024-funding table uses PRIV/TRB; the …-sites table uses PVT/TRIB. Same concept, different codes — a model filtering federal/private funding with the wrong code silently gets zero/garbage. Also amount=0 means "amount unknown," not NULL.
Fix: document both code vocabularies (and the amount=0 semantics) in the STAC column descriptions; ideally reconcile them upstream.

4. Census district hex join keys are zero-padded strings.
"LWCF in Senate District 2" returned NaN/0 for models that filtered SLDUST='2' or '02'; the correct key is SLDUST='002'. Multiple models burned turns rediscovering this.
Fix: note in the census STAC docs that district codes (SLDUST, SLDLST, CD…) are zero-padded fixed-width strings, with an example.

5. Overture country hex parquet has no name_en column.
"Vulnerable carbon in India's wetlands": the hex parquet lacks name_en, so name-based filters fail; the working key is the ISO country='IN'. Not documented.
Fix: document the available filter columns on the Overture hex asset (vs the polygon asset).

6. WDPA: large features have NULL h9.
1,297 of the biggest WDPA features (≥5,000 km²) carry only h8, with NULL h9. Joining population/richness on h9 silently drops them and corrupts global denominators (we saw this break a "% of land protected" calc).
Fix: note in the WDPA STAC docs that very large features are only present at h8 and coarser; join at h8.

7. Carbon: add an explicit unit-conversion line.
The carbon column is correctly documented as "Mg C (tonnes)," but models still mislabeled aggregated totals by ~1000× ("1,095 Gt" / "1.1 teratonnes" for India's wetlands; true ≈1,095 Mt ≈1.095 Gt).
Fix: add a one-liner: "SUM(carbon) is in tonnes; ÷1e6 = Mt C; ÷1e9 = Gt C; ÷1e15 = Pg C." Cheap, and it kills a whole class of ~1000× errors.

8. ca30x30 conserved-areas: the Gap*_acres columns are a trap.
Gap1_acres + Gap2_acres equals total Acres here (Gap3/Gap4 ≈ 0), so they are not a valid GAP-1+2-only breakdown; the correct GAP1+2 acreage uses the proportional SUM((Final_g1_p+Final_g2_p)/100 * Acres) ≈ 26.29M acres.
Fix: note the proportional method and the Gap*_acres pitfall in the dataset docs.

Offer

Happy to share the full per-question ground-truth values and the exact queries that expose each of these. Each fix above maps to at least one confirmed wrong model answer, so they're directly testable.