Skip to content

vortex: Basic Variant IO support#25

Merged
robert3005 merged 9 commits into
mainfrom
adamg/variant-io-support
Jun 17, 2026
Merged

vortex: Basic Variant IO support#25
robert3005 merged 9 commits into
mainfrom
adamg/variant-io-support

Conversation

@AdamGS

@AdamGS AdamGS commented May 27, 2026

Copy link
Copy Markdown

Adds basic support for variant in Vortex through the generic record-based writer.

Depends on:

Work that is still missing here:

  • min/max bounds for variant array

@AdamGS AdamGS force-pushed the adamg/variant-io-support branch from bee87b4 to a735a4d Compare June 4, 2026 12:58
@AdamGS AdamGS changed the title Adamg/variant io support Variant IO support Jun 4, 2026
@AdamGS AdamGS requested a review from robert3005 June 4, 2026 13:00
@AdamGS AdamGS changed the title Variant IO support vortex: Variant IO support Jun 4, 2026
@AdamGS AdamGS marked this pull request as ready for review June 4, 2026 13:00
@AdamGS AdamGS changed the title vortex: Variant IO support vortex: Basic Variant IO support Jun 4, 2026
@github-actions github-actions Bot added the DATA label Jun 4, 2026
Comment thread vortex/src/main/java/org/apache/iceberg/data/vortex/GenericVortexReader.java Outdated
@robert3005

Copy link
Copy Markdown
Member

you also have this failure

TestGenericVortex > testTypeSchema(Type) > [19] variant FAILED
    org.apache.iceberg.exceptions.RuntimeIOException: Failed to write Vortex batch
        at app//org.apache.iceberg.vortex.VortexFileAppender.flushBatch(VortexFileAppender.java:103)
        at app//org.apache.iceberg.vortex.VortexFileAppender.close(VortexFileAppender.java:129)
        at app//org.apache.iceberg.vortex.TestGenericVortex.writeAndValidate(TestGenericVortex.java:91)
        at app//org.apache.iceberg.vortex.TestGenericVortex.writeAndValidate(TestGenericVortex.java:72)
        at app//org.apache.iceberg.vortex.TestGenericVortex.writeAndValidate(TestGenericVortex.java:61)

        Caused by:
        java.io.IOException: failed to write batch
            at dev.vortex.api.VortexWriter.writeBatch(VortexWriter.java:75)
            at org.apache.iceberg.vortex.VortexFileAppender.flushBatch(VortexFileAppender.java:101)
            ... 4 more

            Caused by:
            java.lang.RuntimeException: Vortex Error: Other error: write schema mismatch: expected {id=i64, test_type=variant?, trailing_data=utf8}, got {id=i64, test_type={metadata=binary, value=binary?}?, trailing_data=utf8}
                at dev.vortex.jni.NativeWriter.writeBatch(Native Method)
                at dev.vortex.api.VortexWriter.writeBatch(VortexWriter.java:73)
                ... 5 more

@AdamGS

AdamGS commented Jun 9, 2026

Copy link
Copy Markdown
Author

I think the failure is fixed in upstream Voretx.

@AdamGS

AdamGS commented Jun 9, 2026

Copy link
Copy Markdown
Author

ok now there's some spark stuff I need to fix, will get to it later this week.

AdamGS added 8 commits June 16, 2026 12:28
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
@github-actions github-actions Bot added the SPARK label Jun 16, 2026
@robert3005 robert3005 merged commit 92aa3bd into main Jun 17, 2026
65 of 90 checks passed
@robert3005 robert3005 deleted the adamg/variant-io-support branch June 17, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants