fix: check_geometry_validity failed against real output - #751
Merged
Conversation
…t BLOB) Caught live during the first fully successful smoke-test run of #722's whole chain: `ST_GeomFromWKB(osm_geometry)` crashed with `_duckdb.BinderException: No function matches ... 'st_geomfromwkb(GEOMETRY)'`. docs/outputs/CONFLATED_PARQUET.md documents osm_geometry as plain WKB bytes, but the real column carries the native Parquet GEOGRAPHY logical type (see pipeline::conflate::writer's own doc comment) -- DuckDB's spatial extension auto-decodes that straight to a native GEOMETRY value on read, once loaded. ST_GeomFromWKB expects BLOB input, so calling it on an already-GEOMETRY value is a type error, not something that was ever going to work. Fix: call ST_IsValid directly, no conversion needed. Test fixtures updated to match: atp_geometry/osm_geometry now constructed as native GEOMETRY (ST_Point(...) directly), not cast to ::BLOB via ST_AsWKB -- the cast produced a column type real osm-diffs output never actually has, which is exactly why this bug shipped undetected through #743/#744's full test suites. Verified against the real completed run: validate now passes cleanly end-to-end (1,731,159 rows, all hard checks green) -- the first completely successful run of the whole #722 chain. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Caught live during the first fully successful smoke-test run of #722's whole chain:
ST_GeomFromWKB(osm_geometry)crashed withdocs/outputs/CONFLATED_PARQUET.mddocumentsosm_geometryas plain WKB bytes, but the real column carries the native Parquet GEOGRAPHY logical type -- DuckDB's spatial extension auto-decodes that straight to a nativeGEOMETRYvalue on read, once loaded.ST_GeomFromWKBexpectsBLOBinput, so calling it on an already-GEOMETRYvalue was never going to work. Fix: callST_IsValiddirectly, no conversion needed.Why this shipped undetected through #743/#744
The test fixtures constructed
atp_geometry/osm_geometryviaST_AsWKB(...)::BLOB-- a column type real osm-diffs output never actually has. Updated fixtures to use nativeGEOMETRY(ST_Point(...)directly), matching reality, which is exactly what would have caught this before it shipped.Testing
uv run pytest(78/78) anduvx ruff checkclean. Verified against the real completed run this was found on:validatenow passes cleanly end-to-end -- 1,731,159 rows, every hard check green. First fully successful run of the whole #722 chain (built frommain, containerized,--regional-extract europe/switzerland, real cgroup limits, real S3 upload).🤖 Generated with Claude Code