Happy to help test V3 geometry support — early-adopter context

[jatorre]

Hey — saw #902 / #968 / #972 land and I'm excited about the V3 geometry direction.

I'm building out a small geo-Iceberg testbed against multiple engines (https://github.com/jatorre/iceberg-geo-testbed) and DuckDB is the one I'd most like to make work end-to-end, so I figured I'd just say hi and offer to be useful while you finish things up.

If it helps, I have a hand-written V3 fixture (10 regions, geometry column with per-file bounds, static metadata.json) that's copy-pasteable into a test case. Happy to:

• run things against in-progress branches and report back
• try a PR direction you point me at
• or just stay out of the way until you're ready

Whatever's useful. No pressure either way — I know this is a moving target on your side.

[Tmonster]

Geometry support is in v1.5.3 right now. Feel free to start testing and report back. The only thing we currently do not do is write stats to the manifest files for the geometry type. Beyond that, most functionality should be available (assuming you have also loaded spatial)

[jatorre]

Thanks @Tmonster! Tested against 1.5.3 (LOAD iceberg; LOAD spatial) — here's what we found.

Working end-to-end on the schema + scan path:

• Hand-written V3 metadata.json with geometry(OGC:CRS84) column → schema parses (DESCRIBE correctly shows GEOMETRY('OGC:CRS84'))
• SELECT id, ST_AsText(geom) FROM iceberg_scan(...) returns proper POINT geometries
• Verified against three independent V3 sources: our hand-written fixture, Snowflake's managed V3 (read via direct GCS URL), and Snowflake's managed V3 (attached as a REST catalog through Snowflake Horizon)

One specific gap on the read side. When the V3 manifest does carry geometry bounds (which Snowflake writes), a query with a spatial predicate hits:

Invalid Configuration Error: Column geom lower bound deserialization failed:
Failed to deserialize blob '...' of size 16, attempting to produce value of
type 'GEOMETRY('OGC:CRS84')'

Same crash on all three sources — so this is unambiguously read-side, not a fixture issue. It looks like IcebergValue::DeserializeValue doesn't have a LogicalTypeId::GEOMETRY branch yet; the spec bound is 16 bytes (xmin/ymin for lower_bound, xmax/ymax for upper_bound, as LE doubles — Z/M would add 8 bytes each).

I know your note said the writes aren't done yet — to be clear, this is specifically the read side when another engine has written valid bounds. Happens with WHERE ST_Intersects(...) predicates, doesn't happen on full scans.

Paste-and-run repro (no clone needed — fixture is on a public GCS bucket):

LOAD iceberg; LOAD spatial;
SELECT COUNT(*) FROM iceberg_scan(
  'https://storage.googleapis.com/cartobq-iceberg-geo-testbed/v3_geometry/metadata/v1.metadata.json'
) WHERE ST_Intersects(geom, ST_MakeEnvelope(-125, 32, -115, 42));

Full write-up of the source-level area, the spec encoding, and a proposed PR shape lives at https://github.com/jatorre/iceberg-geo-testbed/blob/main/docs/duckdb-gap.md.

Happy to test a branch as soon as you have one — or to refile this as a more tightly-scoped issue if that's the preferred shape. Just let me know what's most useful.

[Tmonster]

Hi @jatorre

The branch at #1013 should solve the Paste and run repro. Feel free to test it against the rest of your testing suite.

Also, write support for geometry is also enabled, so you can go ahead and start testing that as well. The one problem is we don't write the stats to the manifest files yet

[jatorre]

Thanks @Tmonster — built #1013 locally (base v1.5-variegata, DuckDB submodule at 2a172f10f4) and re-tested against both V3 fixtures. Crash is gone; spatial predicates return correct rows.

Paste-and-run repro (the one from earlier in this thread) — no crash, returns 1000 (the entire California region in our fixture, since the envelope encloses it):

LOAD iceberg; LOAD spatial;
SELECT COUNT(*) FROM iceberg_scan(
  'https://storage.googleapis.com/cartobq-iceberg-geo-testbed/v3_geometry/metadata/v1.metadata.json'
) WHERE ST_Intersects(geom, ST_MakeEnvelope(-125, 32, -115, 42));
-- 1000

Cross-source check against Snowflake's own managed V3 table (the one where the original crash also reproduced) — same predicate, same correct result.

One nuance worth flagging in case future readers expect L3 pruning: the PR deliberately short-circuits IcebergPredicateStats::DeserializeBounds for GEOMETRY and returns empty stats (the in-source comment says as much: "DuckDB-Iceberg does not yet support deserializing avro blobs to geometry yet"). So DuckDB falls back to a full scan rather than pruning on the manifest geometry bound. EXPLAIN ANALYZE confirms:

• hand fixture: Total Files Read: 10 (out of 10)
• Snowflake-managed V3: Total Files Read: 7 (out of 7)

Just calling out for clarity — this PR is correctness-only, not yet the manifest-bound pruning. Happy to retest the bound-decoder follow-up whenever a branch is up; the encoding we worked out (packed_xy_le, 16-byte LE-double-X then LE-double-Y, per spec) is documented at https://github.com/jatorre/iceberg-geo-testbed/blob/main/docs/encoding.md if useful.

Full write-up of the verification (matrix updated, with the side-by-side V2-flat-bbox=1/10 vs V3-geometry=10/10 pruning result) is in STATUS_V3.md.

[Tmonster]

Hi @jatorre ,

The Pr is a small one just to enable reading from geometry columns with a predicate, pruning will indeed come later.
I will keep an eye on your repo, interested in adding sedona compatibility to our own testing framework.

I also just merged the PR, in about ~3 hours if you run FORCE INSTALL ICEBERG FROM CORE_NIGHTLY you should be able to run spatial predicates with no issues