Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions java/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,43 @@
# Version changelog

## Release v1.1.0

### Major Changes

- **License: Migrated from the Databricks License to the Apache License 2.0**

### New Features and Improvements

- **Arrow Flight Support (Experimental)**: Added support for ingesting Apache Arrow `VectorSchemaRoot` batches via Arrow Flight protocol
- **Note**: Arrow Flight is not yet supported by default from the Zerobus server side.
- New `ZerobusArrowStream` class with `ingestBatch()`, `waitForOffset()`, `flush()`, `close()`, `getUnackedBatches()` methods
- New `ArrowStreamConfigurationOptions` for configuring Arrow streams (max inflight batches, recovery, timeouts, IPC compression)
- Configurable IPC compression via `ArrowStreamConfigurationOptions.setIpcCompression()` (supports `LZ4_FRAME` and `ZSTD`)
- New `createArrowStream()` and `recreateArrowStream()` methods on `ZerobusSdk`
- Accepts `VectorSchemaRoot` directly via `ingestBatch()` (IPC serialization handled internally)
- Arrow is opt-in: add `arrow-vector` and `arrow-memory-netty` as dependencies (provided scope, `>= 15.0.0`)

### Bug Fixes

- **Classloader Isolation Compatibility**: Fixed `NoClassDefFoundError` when using the SDK inside Spring Boot. JNI class references are now cached as `GlobalRef`s during `JNI_OnLoad`, so native daemon threads no longer rely on `FindClass` through the system classloader.
- Fixed proto generation tool to skip reserved field numbers 19000-19999 for tables with more than 19000 columns

### Internal Changes

- Added `arrow-vector` 17.0.0 as provided dependency for Arrow Flight support
- Added `arrow-memory-netty` 17.0.0 as test dependency for integration tests
- Uses existing JNI Arrow Flight bindings from Rust SDK (`nativeCreateArrowStream`, `nativeIngestBatch`, etc.)

### API Changes

- Added `createArrowStream(String tableName, Schema schema, String clientId, String clientSecret)` to `ZerobusSdk`
- Added `createArrowStream(String tableName, Schema schema, String clientId, String clientSecret, ArrowStreamConfigurationOptions options)` to `ZerobusSdk`
- Added `recreateArrowStream(ZerobusArrowStream closedStream)` to `ZerobusSdk`
- Added `ZerobusArrowStream` class with methods: `ingestBatch()`, `waitForOffset()`, `flush()`, `close()`, `getUnackedBatches()`, `isClosed()`, `getTableName()`, `getOptions()`
- Added `ArrowStreamConfigurationOptions` class with fields: `maxInflightBatches`, `recovery`, `recoveryTimeoutMs`, `recoveryBackoffMs`, `recoveryRetries`, `serverLackOfAckTimeoutMs`, `flushTimeoutMs`, `connectionTimeoutMs`, `ipcCompression`
- Added `IPCCompressionType` enum with values: `NONE`, `LZ4_FRAME`, `ZSTD`
- Added optional dependency: `org.apache.arrow:arrow-vector >= 15.0.0` (provided scope)

## Release v1.0.1

### Bug Fixes
Expand Down
29 changes: 1 addition & 28 deletions java/NEXT_CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,19 @@
# NEXT CHANGELOG

## Release v1.1.0
## Release v1.2.0

### Major Changes

- **License: Migrated from the Databricks License to the Apache License 2.0**

### New Features and Improvements

- **Arrow Flight Support (Experimental)**: Added support for ingesting Apache Arrow `VectorSchemaRoot` batches via Arrow Flight protocol
- **Note**: Arrow Flight is not yet supported by default from the Zerobus server side.
- New `ZerobusArrowStream` class with `ingestBatch()`, `waitForOffset()`, `flush()`, `close()`, `getUnackedBatches()` methods
- New `ArrowStreamConfigurationOptions` for configuring Arrow streams (max inflight batches, recovery, timeouts, IPC compression)
- Configurable IPC compression via `ArrowStreamConfigurationOptions.setIpcCompression()` (supports `LZ4_FRAME` and `ZSTD`)
- New `createArrowStream()` and `recreateArrowStream()` methods on `ZerobusSdk`
- Accepts `VectorSchemaRoot` directly via `ingestBatch()` (IPC serialization handled internally)
- Arrow is opt-in: add `arrow-vector` and `arrow-memory-netty` as dependencies (provided scope, `>= 15.0.0`)

### Bug Fixes

- **Classloader Isolation Compatibility**: Fixed `NoClassDefFoundError` when using the SDK inside Spring Boot. JNI class references are now cached as `GlobalRef`s during `JNI_OnLoad`, so native daemon threads no longer rely on `FindClass` through the system classloader.
- Fixed proto generation tool to skip reserved field numbers 19000-19999 for tables with more than 19000 columns

### Documentation

### Internal Changes

- Added `arrow-vector` 17.0.0 as provided dependency for Arrow Flight support
- Added `arrow-memory-netty` 17.0.0 as test dependency for integration tests
- Uses existing JNI Arrow Flight bindings from Rust SDK (`nativeCreateArrowStream`, `nativeIngestBatch`, etc.)

### Breaking Changes

### Deprecations

### API Changes

- Added `createArrowStream(String tableName, Schema schema, String clientId, String clientSecret)` to `ZerobusSdk`
- Added `createArrowStream(String tableName, Schema schema, String clientId, String clientSecret, ArrowStreamConfigurationOptions options)` to `ZerobusSdk`
- Added `recreateArrowStream(ZerobusArrowStream closedStream)` to `ZerobusSdk`
- Added `ZerobusArrowStream` class with methods: `ingestBatch()`, `waitForOffset()`, `flush()`, `close()`, `getUnackedBatches()`, `isClosed()`, `getTableName()`, `getOptions()`
- Added `ArrowStreamConfigurationOptions` class with fields: `maxInflightBatches`, `recovery`, `recoveryTimeoutMs`, `recoveryBackoffMs`, `recoveryRetries`, `serverLackOfAckTimeoutMs`, `flushTimeoutMs`, `connectionTimeoutMs`, `ipcCompression`
- Added `IPCCompressionType` enum with values: `NONE`, `LZ4_FRAME`, `ZSTD`
- Added optional dependency: `org.apache.arrow:arrow-vector >= 15.0.0` (provided scope)

2 changes: 1 addition & 1 deletion java/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.databricks</groupId>
<artifactId>zerobus-ingest-sdk</artifactId>
<version>1.0.1</version>
<version>1.1.0</version>
<packaging>jar</packaging>
<name>Zerobus Ingest SDK for Java</name>
<description>Databricks Zerobus Ingest SDK for Java - Direct ingestion to Delta tables via native Rust SDK</description>
Expand Down
Loading