Skip to content

feat: drop the arrow-native ingestion path - #14

Merged
HideBa merged 4 commits into
mainfrom
develop
Aug 25, 2026
Merged

HideBa merged 4 commits into
mainfrom
develop

Conversation

@HideBa

@HideBa HideBa commented Aug 24, 2026

Copy link
Copy Markdown
Member

WKB is the only encoding CityParquet defines, so the ST_3DFromArrowNative family has nothing to read. The nested LIST/STRUCT importer, its kernel and its tests go with it.

The geometry_properties STRUCT overload of ST_3DFromWKB stays: CityParquet types geometry_properties_lod* as a STRUCT regardless of how the geometry column beside it is encoded.

BREAKING CHANGE: ST_3DFromArrowNative, ST_3DTryFromArrowNative, ST_Geom3DFromArrowNative and ST_Geom3DTryFromArrowNative are removed.

HideBa added 4 commits August 24, 2026 14:13
WKB is the only encoding CityParquet defines, so the ST_3DFromArrowNative
family has nothing to read. The nested LIST/STRUCT importer, its kernel and
its tests go with it.

The geometry_properties STRUCT overload of ST_3DFromWKB stays: CityParquet
types geometry_properties_lod* as a STRUCT regardless of how the geometry
column beside it is encoded.

BREAKING CHANGE: ST_3DFromArrowNative, ST_3DTryFromArrowNative,
ST_Geom3DFromArrowNative and ST_Geom3DTryFromArrowNative are removed.
…tation

Part C (write a package, three sidecar forms, validation off Parquet, the LoD0
bridge, all LoDs side by side) was last executed before the geometry columns
carried the Parquet GEOMETRY logical type. Re-executed end to end against
cityjson e461a32 and three_d cff32e6; every output here is the new real output.

The package is 3690150 bytes rather than 4235239. The solid-side numbers are
unchanged — 1116 parts, 1098 valid, 1915861 m3, and all 1116 still byte-identical
to the CityJSONSeq path — which is the result that matters: annotating the
footprint column left the solid round trip alone.

Two things the re-run turned up. The CRS the LoD0 column reports depends on
whether spatial is loaded: the writer stores the whole PROJJSON document in the
logical type, and spatial resolves it to EPSG:7415 for display, so the type
printed is a rendering rather than what is stored. And enable_geoparquet_conversion
no longer offers a way back to BLOB, because the promotion now follows the logical
type rather than the geo footer.

Also: this build has autoload_known_extensions off, so the cells using json and
spatial now load them explicitly instead of relying on autoload.
…BLOB

CityParquet annotates its GeoParquet-legal geometry columns with the Parquet
GEOMETRY logical type, so DuckDB promotes such a column on read and hands it to a
query as GEOMETRY. Reading a LoD0 footprint therefore meant going through
spatial's ST_AsWKB first: geometry_lod0_0::BLOB raises, the cast is unimplemented,
and enable_geoparquet_conversion does not turn the promotion off because it
follows the logical type rather than the geo footer.

ST_3DFromWKB, ST_3DTryFromWKB and ST_Geom3DFromWKB now accept the column as it
comes. Geometry::ToBinary is the bridge rather than a string_t reinterpret: the
value is physically a string but not contractually raw WKB, and going through the
documented conversion keeps that an implementation detail of core.

The argument is one ANY candidate dispatched at bind time, not a second overload.
Two overloads read better but broke ST_3DFromWKB(NULL), which has always bound:
SQLNULL casts to BLOB and to GEOMETRY at the same cost, so the binder could no
longer choose. The existing suite caught that.

Only the footprint constructor has a reachable positive case. DuckDB v1.5.4 has no
polyhedral surface in its geometry model, so ST_GeomFromWKB refuses solid bytes
outright and a solid cannot become a GEOMETRY at all. On the solid constructors the
new form matters for foreign GeoParquet columns, where MultiPolygon Z is common and
the TRY form's NULL is the useful answer; it is covered here through the null and
error contracts.

Walkthrough §22 and §23 drop the ST_AsWKB step and are re-executed against this
build; §22 keeps the bridged form alongside, measured at 0.0 difference.
three_d had no wasm recipe at all, so the only way to get a browser-loadable
artefact was the community registry's build — which lags this repository by a
good margin.

`just wasm` defaults to wasm_eh rather than wasm_mvp as the sibling
duckdb-cityjson repo does, because the consumers differ: that default serves
its wasm_mvp smoke harness, whereas here a browser is the only consumer, and a
browser instance takes eh extensions only.

vcpkg is cloned blobless rather than shallow. vcpkg resolves proj through its
versions database to a specific port tree, and a shallow clone has no history
to find it in — the install dies with "failed to unpack tree object ... vcpkg
was cloned as a shallow repository". `--filter=blob:none` keeps every tree
while fetching blobs on demand.

ST_3DTransform reads PROJ's EPSG database at runtime, which is not in the
Emscripten filesystem, so expect that one function to fail under wasm. The
other ST_3D* functions do not touch it.
@HideBa
HideBa merged commit a0936d6 into main Aug 25, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant