[All] Add CLAUDE.md files

CLAUDE.md

@@ -0,0 +1,119 @@
+# Zerobus SDK — Monorepo Guide
+
+## Architecture
+
+This is a polyglot SDK monorepo. A single Rust core (`rust/sdk/`) implements gRPC streaming, OAuth, recovery, and ingestion logic. Four wrapper SDKs expose it to other languages:
+
+| SDK        | Directory      | Binding mechanism        | Build system     |
+|------------|----------------|--------------------------|------------------|
+| Rust       | `rust/`        | Native                   | Cargo workspace  |
+| Python     | `python/`      | PyO3 / maturin           | maturin + pip    |
+| TypeScript | `typescript/`  | NAPI-RS                  | npm + napi-rs    |
+| Java       | `java/`        | JNI (`rust/jni/`)        | Maven            |
+| Go         | `go/`          | cgo + static FFI lib     | Go modules       |
+
+The data flow is: **User code → Wrapper API → FFI boundary → Rust core → gRPC → Zerobus service**.
+
+Go uses the C FFI layer (`rust/ffi/`, header auto-generated by cbindgen). Java has its own JNI crate (`rust/jni/`). Python and TypeScript bind Rust directly via PyO3 and NAPI-RS respectively.
+
+## Versioning and Breaking Changes
+
+**Strict semver.** Breaking changes are only allowed in major version bumps.
+
+- Removing or renaming a public API element is a breaking change. Mark it `#[deprecated]` (or language equivalent) in a minor release first, then remove in the next major.
+- Changing the FFI C header (`rust/ffi/zerobus.h`) in a non-additive way breaks Go and Java. Treat FFI signature changes with the same rigor as public API changes.
+- Each SDK is versioned independently. A Rust core bump does not automatically require wrapper version bumps unless the wrapper's public API changes.
+- Release tags follow `<sdk>/v<semver>` (e.g., `rust/v1.1.0`, `python/v1.0.0`).
+
+## Cross-FFI Concerns
+
+Because every non-Rust SDK crosses a foreign-function boundary, keep these in mind:
+
+### Performance
+- Every call across FFI serializes/deserializes data. Prefer batch APIs (`ingest_records`) over single-record APIs in hot paths.
+- Proto descriptors and Arrow IPC schemas are serialized as byte arrays across the boundary. These are typically one-time costs at stream creation.
+- Arrow Flight uses IPC format across FFI for near-zero-copy transfer; avoid unnecessary re-encoding.
+
+### Memory safety
+- **Rust owns the memory** for SDK and stream handles. Wrappers hold opaque pointers and must call the corresponding free/close/destroy function.
+- C-allocated error strings (`CResult.error_message`) must be freed by the caller via `zerobus_free_error_message()`.
+- Go uses `runtime.Pinner` to prevent GC from relocating memory while Rust holds a pointer. Never remove pinner calls.
+- Go's `cgo.Handle` registry keeps `HeadersProvider` callbacks alive; leaking these handles leaks Go objects.
+- Java has no finalizers on streams — users must call `close()` or use try-with-resources. Forgetting this leaks JNI resources.
+- Python and TypeScript rely on GC-triggered cleanup via PyO3/NAPI-RS reference counting.
+
+### Thread safety
+- Rust core is async (tokio). The FFI layer manages its own tokio runtime.
+- Go: safe for concurrent goroutines.
+- Python: async streams are safe; sync streams are single-threaded per instance.
+- TypeScript: async-safe via Node.js event loop.
+- Java: **not thread-safe** — external synchronization required for concurrent access.
+
+## Build and Test Commands
+
+Each SDK has its own build/test commands. Run from the SDK directory:
+
+| SDK        | Build                  | Test            | Lint               | Format              |
+|------------|------------------------|-----------------|--------------------|---------------------|
+| Rust       | `make build`           | `make test`     | `make lint`        | `make fmt`          |
+| Python     | `make build-rust`      | `make test`     | `make lint`        | `make fmt`          |
+| TypeScript | `npm run build`        | `npm test`      | `cargo clippy`     | `cargo fmt`         |
+| Java       | `mvn compile`          | `mvn test`      | `mvn spotless:check` | `mvn spotless:apply` |
+| Go         | `make build`           | `make test`     | `make lint`        | `make fmt`          |
+
+## CI/CD
+
+- `push.yml` is the main gate — uses path filtering to run only affected SDK CI jobs.
+- Cross-SDK CI (non-blocking): when `rust/**` changes, wrapper SDK tests also run against local Rust source to catch FFI breakage early.
+- Release workflows are tag-triggered (see Release Process below).
+
+## Changelog Process
+
+Every SDK has two changelog files:
+
+- **`NEXT_CHANGELOG.md`** — Accumulates entries for the upcoming release. Every PR that changes user-facing behavior must add an entry here under the appropriate section (New Features, Bug Fixes, Breaking Changes, Deprecations, API Changes, Documentation, Internal Changes).
+- **`CHANGELOG.md`** — Finalized release history. When a version-bump PR is merged, the contents of `NEXT_CHANGELOG.md` are prepended to `CHANGELOG.md` and `NEXT_CHANGELOG.md` is reset to an empty template with the next version header.
+
+**Rules:**
+- If a PR adds a feature, fixes a bug, deprecates an API, or changes behavior — it must update `NEXT_CHANGELOG.md`. No exceptions.
+- Breaking changes must be called out under `### Breaking Changes` with migration instructions.
+- Deprecations must include what to use instead and the planned removal timeline.
+
+## Documentation Requirements
+
+Every PR that changes user-facing behavior must also update:
+
+- **README** — If the change affects usage, setup, or API surface, update the SDK's `README.md`.
+- **Examples** — If the change adds a new API or modifies an existing one, add or update examples in the SDK's `examples/` directory.
+- **API docs** — Rust: doc comments. Python: docstrings + type stubs. TypeScript: JSDoc. Java: Javadoc. Go: godoc comments.
+
+## Release Process
+
+Releases are per-SDK and tag-triggered:
+
+1. **Version bump PR**: Update the version in the SDK's config file (`Cargo.toml`, `package.json`, `pom.xml`, etc.). Move `NEXT_CHANGELOG.md` contents into `CHANGELOG.md`. Reset `NEXT_CHANGELOG.md` with the next planned version header.
+2. **Merge to main**.
+3. **Tag**: Push a tag matching `<sdk>/v<semver>` (e.g., `rust/v1.2.0`, `python/v1.1.0`). This triggers the release workflow.
+4. **Release workflow**:
+   - Builds platform-specific artifacts (5 platforms: Linux x86_64/aarch64, macOS x86_64/arm64, Windows x86_64).
+   - Publishes to the SDK's package registry (crates.io, PyPI, npm, Maven Central, or git tag for Go).
+   - Creates a GitHub Release with release notes extracted from `CHANGELOG.md`.
+
+**SDK-specific release details:**
+
+| SDK        | Version source             | Registry       | Tag pattern       |
+|------------|----------------------------|----------------|-------------------|
+| Rust       | `rust/sdk/Cargo.toml`      | crates.io      | `rust/v*`         |
+| Python     | Derived from Cargo.toml via maturin | PyPI  | `python/v*`       |
+| TypeScript | `typescript/package.json`  | npm            | `typescript/v*`   |
+| Java       | `java/pom.xml`             | Maven Central  | `java/v*`         |
+| Go         | Git tag (no version file)  | pkg.go.dev     | `go/v*`           |
+| FFI        | `rust/ffi/Cargo.toml`      | GitHub Release | `ffi/v*`          |
+
+The FFI library has its own release cycle because Go and Java depend on pre-built static/dynamic libraries. An FFI release must happen before Go/Java can pick up Rust core changes.
+
+## Commit Conventions
+
+- Prefix commit messages with `[SDK]` when changes are scoped to one SDK (e.g., `[Rust]`, `[Python]`, `[All]`).
+- Signed commits required (DCO). Use `git commit -s`.
+- Present tense, imperative mood. Keep subject line under 50 chars.

go/CLAUDE.md

@@ -0,0 +1,108 @@
+# Go SDK
+
+Go wrapper around the Rust core via cgo and the C FFI library.
+
+## Structure
+
+```
+go/
+├── zerobus.go       # Public API (ZerobusSdk, ZerobusStream)
+├── ffi.go           # cgo bindings — calls C functions from zerobus.h
+├── arrow_ffi.go     # Arrow Flight cgo bindings
+├── arrow_stream.go  # Arrow stream public API
+├── ack.go           # Acknowledgment types
+├── errors.go        # ZerobusError
+├── types.go         # Configuration types
+├── build.go         # Build tags and cgo link directives
+├── build_rust.sh    # Script to build the Rust FFI library
+├── lib/             # Pre-built static libraries per platform
+│   ├── linux_x86_64/libzerobus_ffi.a
+│   ├── linux_arm64/libzerobus_ffi.a
+│   ├── darwin_x86_64/libzerobus_ffi.a
+│   ├── darwin_arm64/libzerobus_ffi.a
+│   └── windows_x86_64/libzerobus_ffi.a
+├── tests/           # Integration tests
+└── examples/        # Usage examples
+```
+
+## Build commands
+
+Run from `go/`:
+
+- `make build` — Build Rust FFI lib + Go SDK
+- `make build-rust` — Build only Rust FFI layer
+- `make build-go` — Build only Go SDK (requires pre-built FFI lib)
+- `make test` — Run tests
+- `make lint` — go vet + cargo clippy
+- `make fmt` — gofmt + cargo fmt
+
+## FFI boundary: cgo + static linking
+
+This is the most memory-management-sensitive wrapper. Key considerations:
+
+### Memory ownership
+
+- **Opaque pointers**: Go holds `unsafe.Pointer` to Rust-allocated `CZerobusSdk` / `CZerobusStream`. Rust owns the memory.
+- **Explicit free required**: `zerobus_sdk_free()`, `zerobus_stream_free()`, `zerobus_arrow_stream_free()`.
+- **Finalizers**: Both `ZerobusSdk` and `ZerobusStream` register `runtime.SetFinalizer` for GC-triggered cleanup, but explicit `Free()`/`Close()` is preferred for deterministic resource release.
+- **Error strings**: C-allocated error messages (`CResult.error_message`) must be freed via `zerobus_free_error_message()` after converting to Go string with `C.GoString()`.
+
+### Memory pinning (critical)
+
+Go's GC can relocate heap objects. When passing Go memory to C:
+- `runtime.Pinner` is used to pin Go slices/pointers before passing to Rust.
+- Every FFI function that passes Go data uses: create pinner → pin → call C → defer unpin.
+- **Never remove `runtime.Pinner` calls** — doing so causes "cgo argument has Go pointer to unpinned Go pointer" panics.
+- Requires Go 1.21+ (the `runtime.Pinner` API).
+
+### Handle registry for callbacks
+
+When using custom `HeadersProvider` (instead of default OAuth):
+- A `cgo.Handle` wraps the Go interface value and prevents GC collection.
+- Handles are stored in `streamHandleRegistry` (mutex-protected map, keyed by stream pointer).
+- **Cleanup sequence**: lock registry → delete handle → remove from map → unlock → free C stream.
+- Leaking a handle leaks the Go `HeadersProvider` object and any resources it holds.
+
+### Arrow batch cleanup
+
+- `zerobus_arrow_free_batch_array()` must be called after reading unacknowledged batches.
+- Same handle registry pattern applies for Arrow streams with custom headers.
+
+## Breaking change rules
+
+Public API is everything exported (capitalized) in the `go/` package:
+
+- Removing or renaming exported types, functions, methods, or struct fields is breaking.
+- Changing function signatures is breaking.
+- Deprecate with `// Deprecated: Use X instead.` godoc comment.
+- Go module versioning follows git tags (`go/v*`). Major version changes require a `/v2` module path suffix.
+
+## Performance notes
+
+- cgo calls have ~100-200ns overhead per invocation. Batch APIs (`IngestProtoRecords`, `IngestJsonRecords`) amortize this.
+- Record data (`[]byte` for proto, `string` for JSON) is pinned and passed by pointer — no extra copy beyond what cgo requires.
+- The static FFI library is linked at build time. No runtime loading overhead.
+
+## Thread safety
+
+Go SDK is safe for concurrent use from multiple goroutines. Internal synchronization handles concurrent `Ingest` calls. The handle registry uses a mutex.
+
+## Changelog and documentation
+
+- Every PR must update `go/NEXT_CHANGELOG.md` under the appropriate section if it changes user-facing behavior.
+- Update `go/README.md` if the change affects usage, setup, or API surface.
+- Add or update examples in `go/examples/` for new or modified APIs.
+- Add godoc comments for all new exported types, functions, and methods.
+
+## Release
+
+- Version source: git tag only (no version file in Go).
+- Tag: `go/v<semver>` → triggers `release-go.yml` → creates git tag. Go modules are resolved via git, no artifact upload needed.
+- The Go SDK links pre-built static FFI libraries from `go/lib/`. An FFI release (`ffi/v*`) must happen first if Rust FFI code changed, and the updated `.a` files must be committed to `go/lib/` before the Go release.
+- On version bump PR: move `NEXT_CHANGELOG.md` contents to `CHANGELOG.md`, reset `NEXT_CHANGELOG.md`.
+
+## Config
+
+- Go >= 1.21 required
+- CGO_ENABLED=1 required
+- Rust toolchain required for building FFI from source

java/CLAUDE.md

@@ -0,0 +1,80 @@
+# Java SDK
+
+Java wrapper around the Rust core via JNI.
+
+## Structure
+
+```
+java/
+├── src/main/java/com/databricks/zerobus/
+│   ├── ZerobusSdk.java          # Main SDK class
+│   ├── ZerobusStream.java       # Legacy generic stream (deprecated)
+│   ├── ZerobusProtoStream.java  # Proto ingestion stream
+│   ├── ZerobusJsonStream.java   # JSON ingestion stream
+│   ├── ZerobusArrowStream.java  # Arrow Flight stream (experimental)
+│   ├── NativeLoader.java        # Platform-specific JNI library loading
+│   └── ...                      # Config, exceptions, callbacks
+├── src/test/                    # JUnit 5 + Mockito tests
+├── pom.xml                      # Maven config
+└── tools/                       # Schema generation
+```
+
+The JNI native library is built from `rust/jni/`. It's packaged inside the JAR for all supported platforms.
+
+## Build commands
+
+Run from `java/`:
+
+- `mvn compile` — Compile
+- `mvn test` — Run tests
+- `mvn test -Dtest=ClassName#method` — Run specific test
+- `mvn clean package` — Build JAR (includes fat JAR)
+- `mvn spotless:check` — Check formatting (Google Java Format)
+- `mvn spotless:apply` — Auto-format
+
+## FFI boundary: JNI
+
+JNI is the most manual of all the bindings. Key considerations:
+
+- **Handle pattern**: Java holds a `long nativeHandle` field — an opaque pointer to the Rust object. All native methods pass this handle. If the handle is invalid (use-after-close), Rust panics or returns an error.
+- **No finalizers**: Unlike Go/Python/TypeScript, Java streams do **not** have GC-triggered cleanup. Users **must** call `close()` explicitly or use try-with-resources. Forgetting this leaks native memory.
+- **AutoCloseable**: SDK and stream classes implement `AutoCloseable`. Always prefer try-with-resources.
+- **Async bridge**: `CompletableFuture` bridges Rust futures to Java. Stream creation and acknowledgment waiting return futures.
+- **Thread safety**: The Java SDK is **not thread-safe**. Do not share SDK or stream instances across threads without external synchronization.
+- **Native library loading** (`NativeLoader.java`): Detects OS + arch, extracts the correct `.so`/`.dylib`/`.dll` from the JAR classpath to a temp directory, loads via `System.load()`. Marked `deleteOnExit()`.
+
+## Breaking change rules
+
+Public API is everything in `com.databricks.zerobus` with `public` visibility:
+
+- Removing or renaming public classes, methods, or fields is breaking.
+- Changing method signatures (parameter types, return types, checked exceptions) is breaking.
+- Deprecate with `@Deprecated` annotation and Javadoc explaining the replacement.
+- JNI native method signatures (`private static native ...`) are internal but must stay in sync with `rust/jni/` — changing one without the other causes `UnsatisfiedLinkError` at runtime.
+
+## Performance notes
+
+- JNI calls have non-trivial overhead (~100ns per crossing). Batch APIs reduce crossings.
+- Proto descriptors are passed as `byte[]` — copied at the boundary. This is a one-time cost at stream creation.
+- Record payloads (`byte[]` for proto, `String` for JSON) are copied across JNI. For high-throughput, prefer proto with batch ingestion.
+- Native library extraction happens once at class load time — first-use latency includes temp file I/O.
+
+## Changelog and documentation
+
+- Every PR must update `java/NEXT_CHANGELOG.md` under the appropriate section if it changes user-facing behavior.
+- Update `java/README.md` if the change affects usage, setup, or API surface.
+- Add or update examples in `java/examples/` for new or modified APIs.
+- Add Javadoc for all new public classes/methods.
+
+## Release
+
+- Version source: `java/pom.xml` (`<version>x.y.z</version>`).
+- Tag: `java/v<semver>` → triggers `release-java.yml` → builds JNI native libs for all platforms, packages into JAR → publishes to Maven Central.
+- The JAR bundles native libraries for all 5 platforms. The JNI release depends on `rust/jni/` — if Rust JNI code changed, both must be coordinated.
+- On version bump PR: update version in `pom.xml`, move `NEXT_CHANGELOG.md` contents to `CHANGELOG.md`, reset `NEXT_CHANGELOG.md`.
+
+## Config
+
+- Java 8+ compatibility (source/target 1.8)
+- Google Java Format via Spotless
+- Arrow version: 17.0.0