Skip to content

Commit 8069f28

Browse files
Initial commit
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
1 parent 031e1ff commit 8069f28

5 files changed

Lines changed: 42 additions & 31 deletions

File tree

‎java/CHANGELOG.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,43 @@
11
# Version changelog
22

3+
## Release v1.1.0
4+
5+
### Major Changes
6+
7+
- **License: Migrated from the Databricks License to the Apache License 2.0**
8+
9+
### New Features and Improvements
10+
11+
- **Arrow Flight Support (Experimental)**: Added support for ingesting Apache Arrow `VectorSchemaRoot` batches via Arrow Flight protocol
12+
- **Note**: Arrow Flight is not yet supported by default from the Zerobus server side.
13+
- New `ZerobusArrowStream` class with `ingestBatch()`, `waitForOffset()`, `flush()`, `close()`, `getUnackedBatches()` methods
14+
- New `ArrowStreamConfigurationOptions` for configuring Arrow streams (max inflight batches, recovery, timeouts, IPC compression)
15+
- Configurable IPC compression via `ArrowStreamConfigurationOptions.setIpcCompression()` (supports `LZ4_FRAME` and `ZSTD`)
16+
- New `createArrowStream()` and `recreateArrowStream()` methods on `ZerobusSdk`
17+
- Accepts `VectorSchemaRoot` directly via `ingestBatch()` (IPC serialization handled internally)
18+
- Arrow is opt-in: add `arrow-vector` and `arrow-memory-netty` as dependencies (provided scope, `>= 15.0.0`)
19+
20+
### Bug Fixes
21+
22+
- **Classloader Isolation Compatibility**: Fixed `NoClassDefFoundError` when using the SDK inside Spring Boot. JNI class references are now cached as `GlobalRef`s during `JNI_OnLoad`, so native daemon threads no longer rely on `FindClass` through the system classloader.
23+
- Fixed proto generation tool to skip reserved field numbers 19000-19999 for tables with more than 19000 columns
24+
25+
### Internal Changes
26+
27+
- Added `arrow-vector` 17.0.0 as provided dependency for Arrow Flight support
28+
- Added `arrow-memory-netty` 17.0.0 as test dependency for integration tests
29+
- Uses existing JNI Arrow Flight bindings from Rust SDK (`nativeCreateArrowStream`, `nativeIngestBatch`, etc.)
30+
31+
### API Changes
32+
33+
- Added `createArrowStream(String tableName, Schema schema, String clientId, String clientSecret)` to `ZerobusSdk`
34+
- Added `createArrowStream(String tableName, Schema schema, String clientId, String clientSecret, ArrowStreamConfigurationOptions options)` to `ZerobusSdk`
35+
- Added `recreateArrowStream(ZerobusArrowStream closedStream)` to `ZerobusSdk`
36+
- Added `ZerobusArrowStream` class with methods: `ingestBatch()`, `waitForOffset()`, `flush()`, `close()`, `getUnackedBatches()`, `isClosed()`, `getTableName()`, `getOptions()`
37+
- Added `ArrowStreamConfigurationOptions` class with fields: `maxInflightBatches`, `recovery`, `recoveryTimeoutMs`, `recoveryBackoffMs`, `recoveryRetries`, `serverLackOfAckTimeoutMs`, `flushTimeoutMs`, `connectionTimeoutMs`, `ipcCompression`
38+
- Added `IPCCompressionType` enum with values: `NONE`, `LZ4_FRAME`, `ZSTD`
39+
- Added optional dependency: `org.apache.arrow:arrow-vector >= 15.0.0` (provided scope)
40+
341
## Release v1.0.1
442

543
### Bug Fixes

‎java/NEXT_CHANGELOG.md‎

Lines changed: 1 addition & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,19 @@
11
# NEXT CHANGELOG
22

3-
## Release v1.1.0
3+
## Release v1.2.0
44

55
### Major Changes
66

7-
- **License: Migrated from the Databricks License to the Apache License 2.0**
8-
97
### New Features and Improvements
108

11-
- **Arrow Flight Support (Experimental)**: Added support for ingesting Apache Arrow `VectorSchemaRoot` batches via Arrow Flight protocol
12-
- **Note**: Arrow Flight is not yet supported by default from the Zerobus server side.
13-
- New `ZerobusArrowStream` class with `ingestBatch()`, `waitForOffset()`, `flush()`, `close()`, `getUnackedBatches()` methods
14-
- New `ArrowStreamConfigurationOptions` for configuring Arrow streams (max inflight batches, recovery, timeouts, IPC compression)
15-
- Configurable IPC compression via `ArrowStreamConfigurationOptions.setIpcCompression()` (supports `LZ4_FRAME` and `ZSTD`)
16-
- New `createArrowStream()` and `recreateArrowStream()` methods on `ZerobusSdk`
17-
- Accepts `VectorSchemaRoot` directly via `ingestBatch()` (IPC serialization handled internally)
18-
- Arrow is opt-in: add `arrow-vector` and `arrow-memory-netty` as dependencies (provided scope, `>= 15.0.0`)
19-
209
### Bug Fixes
2110

22-
- **Classloader Isolation Compatibility**: Fixed `NoClassDefFoundError` when using the SDK inside Spring Boot. JNI class references are now cached as `GlobalRef`s during `JNI_OnLoad`, so native daemon threads no longer rely on `FindClass` through the system classloader.
23-
- Fixed proto generation tool to skip reserved field numbers 19000-19999 for tables with more than 19000 columns
24-
2511
### Documentation
2612

2713
### Internal Changes
2814

29-
- Added `arrow-vector` 17.0.0 as provided dependency for Arrow Flight support
30-
- Added `arrow-memory-netty` 17.0.0 as test dependency for integration tests
31-
- Uses existing JNI Arrow Flight bindings from Rust SDK (`nativeCreateArrowStream`, `nativeIngestBatch`, etc.)
32-
3315
### Breaking Changes
3416

3517
### Deprecations
3618

3719
### API Changes
38-
39-
- Added `createArrowStream(String tableName, Schema schema, String clientId, String clientSecret)` to `ZerobusSdk`
40-
- Added `createArrowStream(String tableName, Schema schema, String clientId, String clientSecret, ArrowStreamConfigurationOptions options)` to `ZerobusSdk`
41-
- Added `recreateArrowStream(ZerobusArrowStream closedStream)` to `ZerobusSdk`
42-
- Added `ZerobusArrowStream` class with methods: `ingestBatch()`, `waitForOffset()`, `flush()`, `close()`, `getUnackedBatches()`, `isClosed()`, `getTableName()`, `getOptions()`
43-
- Added `ArrowStreamConfigurationOptions` class with fields: `maxInflightBatches`, `recovery`, `recoveryTimeoutMs`, `recoveryBackoffMs`, `recoveryRetries`, `serverLackOfAckTimeoutMs`, `flushTimeoutMs`, `connectionTimeoutMs`, `ipcCompression`
44-
- Added `IPCCompressionType` enum with values: `NONE`, `LZ4_FRAME`, `ZSTD`
45-
- Added optional dependency: `org.apache.arrow:arrow-vector >= 15.0.0` (provided scope)
46-

‎java/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<modelVersion>4.0.0</modelVersion>
55
<groupId>com.databricks</groupId>
66
<artifactId>zerobus-ingest-sdk</artifactId>
7-
<version>1.0.1</version>
7+
<version>1.1.0</version>
88
<packaging>jar</packaging>
99
<name>Zerobus Ingest SDK for Java</name>
1010
<description>Databricks Zerobus Ingest SDK for Java - Direct ingestion to Delta tables via native Rust SDK</description>

‎rust/Cargo.lock‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎rust/jni/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ crate-type = ["cdylib"]
1414
name = "zerobus_jni"
1515

1616
[dependencies]
17-
databricks-zerobus-ingest-sdk = { path = "../sdk", version = "1.2.0", features = ["arrow-flight"] }
17+
databricks-zerobus-ingest-sdk = { version = "1.2.0", features = ["arrow-flight"] }
1818
jni = "0.21"
1919
tokio = { version = "1.42", features = ["rt-multi-thread", "sync"] }
2020
prost = "0.13"

0 commit comments

Comments
 (0)