fix: verify-stac reports existing parquet columns as absent (nested columns, wide assets)

[eastagiletracker]

This PR proposes two fixes to verify-stac.py's check_declared_schema_matches_data, which currently reports parquet columns that exist — and hold data — as declared-column-absent. We include this PR work along with a full history of your repo at https://eastagiletracker.com/projects/322. You can sign in with your GitHub ID to claim ownership of the project.

What is wrong

1. A nested column is read as absent. The presence map came from parquet_schema(...) WHERE type IS NOT NULL. That filter drops the schema root — but the parquet schema is a tree, and a LIST / STRUCT / MAP column is a group node whose physical type is also NULL. So the filter dropped the column itself and kept its machinery leaves (element, key, value, struct fields) in its place. Every declared nested column then failed HARD, and its leaves were reported as undocumented top-level columns.

2. Any asset wider than 50 columns loses its tail. The check read the whole column list back over the MCP and diffed it in Python. The MCP query tool caps a result at 50 rows, so column 51 onwards silently vanished from the comparison and was reported absent. GROUP BY is unordered, so which columns fell off changed between runs on the same asset. This is exactly the failure check_values_match_distinct already documents and sidesteps by pushing its set-difference into DuckDB.

Both are live on the published catalog, and both are HARD, so verify-stac.py exits 1 and the advice it prints ("fix the STAC table:columns") would delete correct documentation.

Reproduction at current HEAD (86f09eb)

$ python3 scripts/verify-stac.py --no-recall https://s3-west.nrp-nautilus.io/public-iucn/taxonomy/stac-collection.json
[HARD][declared-column-absent] [iucn-taxonomy-2025] asset 'taxonomy-parquet': STAC declares column 'common_names_en' but it is absent from all 1 parquet file(s) — a stale/incorrect schema; fix the STAC table:columns.
... 8 more, one per string[] column ...
[ADVISORY][undocumented-column] [iucn-taxonomy-2025] asset 'taxonomy-parquet': parquet column 'element' is present in all 1 file(s) but not declared in table:columns
  9 hard, 3 advisory

FAIL: 9 hard finding(s) across 1 collection(s).

All nine exist and are populated — SELECT common_names_en FROM read_parquet('s3://public-iucn/taxonomy/iucn-taxonomy.parquet') WHERE common_names_en IS NOT NULL LIMIT 1 returns ['Caucasian Woodsia']. What the old query looked at is the LIST group node: SELECT name, type, converted_type FROM parquet_schema(...) WHERE name = 'common_names_en' gives type = NULL, converted_type = LIST, so type IS NOT NULL discards it.

The width cap is just as easy to see. gbif-hex-2026-06 has 61 top-level columns, and the grouped presence query returns 50 rows:

SELECT COUNT(*) FROM (SELECT DISTINCT lower(split_part(path_in_schema, ', ', 1))
                      FROM parquet_metadata('s3://public-gbif/2026-06/hex/h0=*/data_0.parquet'));
-- 61

The change

A column's name now comes from its leaf chunk's path_in_schema, whose first segment is the column that leaf belongs to (common_names_en, list, element → common_names_en; bbox, xmin → bbox). That is exact, needs no assumption about schema row order, and covers structs and maps as well as lists — it also subsumes the bbox covering-struct special case, which is why that one goes away. parquet_metadata is the column-chunk footer, so this stays a footer read, and the CTE is MATERIALIZED so each query reads it once.

Both comparisons then run in SQL and return only the discrepancies — declared columns whose file count is short, and undocumented columns present in every file — so the result is bounded by the number of problems rather than the width of the table, and is normally empty.

Deliberately unchanged: the per-file grain that #534 exists for (COUNT(DISTINCT file_name) per column, so the #520 heterogeneous hole still reads as declared-column-heterogeneous), the severities, the finding codes, and the message text. A file with no row groups now drops out of both the presence map and the file count, which is the honest reading — an empty file has no data to disagree with the STAC.

Verification

python3 -m unittest discover -s tests — 36 tests, green, both with and without duckdb installed. Against upstream HEAD the four new tests plus the six updated ones fail (10 failures); with the change all 36 pass. The new tests execute the generated SQL against a real parquet file carrying LIST, STRUCT, MAP and 61-column shapes, through a stub that truncates at 50 rows the way the MCP does — so they fail if either defect returns. One is a mutation guard: a genuinely absent declared column must still fail HARD, and does.

End to end on the live collection above: 9 hard, 3 advisory / FAIL becomes 0 hard, 2 advisory / PASS, with the two genuine advisories (geoparquet-no-geom-column, cng-fid-missing-nonspatial) retained.

Because the change touches a gate rather than a dataset, I also ran the check across 117 published collections before and after. declared-column-absent goes from 1,179 findings across 26 collections to 6 — and all 6 also fired before, so no new HARD finding appears anywhere: wdoecm-parquet OBJECTID, glwd-class-area-hex h0, regions-hex cartography / population / wikidata, and imma-hex bbox. Those look like genuine stale declarations worth a look, but they are yours to judge and are untouched here. The undocumented-column advisories go from 443 to 384: 38 nested machinery names (element, key, value, and the Overture struct fields) stop being reported, while real undeclared columns the 50-row cap had been hiding — identifiedby, recordedby, mrgid_sov5, and h4 / h6 / h7 / h9 on the GBIF hex — now surface. An undeclared bbox covering stays exempt, as before. Six assets that previously answered schema-match-check-failed are now checked, since parquet_metadata does not bind column names the way the old query did.

How this was managed

Your issues and pull requests were imported to a live board, and this fix was tracked there as its own story: the story for this work, on the board.

If you'd rather not receive contributions like this, reply no-more-prs on this pull request and we won't open any further ones on your repositories.

---

Lawrence W. Sinclair
CEO / East Agile
linkedin.com/in/lwsinclair/
eastagile.com