diff --git a/.changeset/correctness-fixes-0-8-0.md b/.changeset/correctness-fixes-0-8-0.md deleted file mode 100644 index 82f1e20..0000000 --- a/.changeset/correctness-fixes-0-8-0.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -'@kafkats/client': patch -'@kafkats/flow': patch -'@kafkats/codec-zod': patch -'@kafkats/flow-state-lmdb': patch ---- - -Correctness and reliability fixes across the producer, consumer, client protocol, and flow state stores. - -### Producer - -- `flush()` now waits for deferred drains so it no longer resolves while writes are still buffered. -- `transactionalSend` freezes its partition set up front to avoid double-calling the partitioner. -- Retry `TxnOffsetCommit` on retriable and rebalance-in-progress errors instead of failing the transaction. -- Fence the producer on `OUT_OF_ORDER_SEQUENCE_NUMBER` to surface the idempotence violation rather than silently corrupting the stream. -- Reject orphaned in-flight send promises during disconnect so callers are not left hanging. - -### Consumer - -- Always reschedule the heartbeat after non-fatal errors, preventing the consumer from dropping out of the group. -- Surface auto-commit errors to the consumer instead of swallowing them. -- Clear only the successfully committed partitions on a partial commit failure so the rest are retried. -- Fence stale fetch responses across partition replacement to avoid delivering records from a revoked assignment. - -### Client / protocol - -- Guard against varint shift overflow in the decoder. -- Close a `ConnectionPool.acquire` race that could hand out a connection mid-teardown. -- Clamp the SASL reauthentication delay to the `setTimeout` maximum. -- Don't await a response on `Produce` with `acks=0` (the broker sends none). -- Always use the v0 response header for `ApiVersions`, matching broker behavior across versions. - -### Share consumer - -- Release (rather than acknowledge) the last yielded message on shutdown so unprocessed records are redelivered. - -### Flow - -- Write to the local store before the changelog (local-first ordering) to shrink the inconsistency window on crash. -- Serialize the EOS commit triggered by a rebalance against the next message, and close the EOS rebalance offset-commit gap via an awaitable rebalance hook (shared with the consumer fix above). -- Use stream time (not wall-clock time) for window expiry. -- Implement hopping windows (`TimeWindows.advanceBy`). -- Throw on `SlidingWindows` instead of silently misbehaving (not yet implemented). -- Correct `TableGroupByNode` mapping ordering. -- Apply retention to `reduce` and session aggregators, emitting changelog tombstones for expired state. -- Surface checkpoint persistence errors instead of swallowing them. - -### flow-state-lmdb - -- Make `range()` upper bound inclusive, matching the in-memory store. -- Await `fsync` on checkpoint writes so a checkpoint is durable before it is reported complete. -- Correct `WindowStore` time-bound math. -- Use order-preserving signed-i64 encoding for time keys so range scans iterate in correct chronological order. diff --git a/.changeset/share-groups-v2.md b/.changeset/share-groups-v2.md deleted file mode 100644 index 2a05e3e..0000000 --- a/.changeset/share-groups-v2.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -'@kafkats/client': minor -'@kafkats/flow': minor -'@kafkats/codec-zod': minor -'@kafkats/flow-state-lmdb': minor ---- - -ShareConsumer: Kafka 4.2 share groups GA support (ShareFetch/ShareAcknowledge v2) - -- KIP-1206: new `acquireMode` config (`'batch_optimized'` default | `'record_limit'`) — `record_limit` strictly caps each fetch at `maxRecords`. -- KIP-1222: new `message.renew()` to extend the acquisition lock without finalizing delivery; safe to call multiple times per message. -- `ShareAcknowledgeResponse.acquisitionLockTimeoutMs` is now decoded on v2 responses. -- AckManager dedupes same-offset RENEW + finalizing-ack pairs and collapses duplicate same-offset RENEWs so the wire never carries overlapping batches. -- `@experimental` markers removed; share groups are GA in Kafka 4.2. -- Backward compatible: against Kafka 4.1 brokers the client negotiates v1 and produces byte-identical wire output to before this change. `record_limit` and `renew()` throw with a clear "requires v2 (Kafka 4.2+)" error against older brokers. diff --git a/packages/client/CHANGELOG.md b/packages/client/CHANGELOG.md index 6f071c6..07f2c28 100644 --- a/packages/client/CHANGELOG.md +++ b/packages/client/CHANGELOG.md @@ -1,5 +1,60 @@ # @kafkats/client +## 0.8.0 + +### Minor Changes + +- fe140c0: ShareConsumer: Kafka 4.2 share groups GA support (ShareFetch/ShareAcknowledge v2) + - KIP-1206: new `acquireMode` config (`'batch_optimized'` default | `'record_limit'`) — `record_limit` strictly caps each fetch at `maxRecords`. + - KIP-1222: new `message.renew()` to extend the acquisition lock without finalizing delivery; safe to call multiple times per message. + - `ShareAcknowledgeResponse.acquisitionLockTimeoutMs` is now decoded on v2 responses. + - AckManager dedupes same-offset RENEW + finalizing-ack pairs and collapses duplicate same-offset RENEWs so the wire never carries overlapping batches. + - `@experimental` markers removed; share groups are GA in Kafka 4.2. + - Backward compatible: against Kafka 4.1 brokers the client negotiates v1 and produces byte-identical wire output to before this change. `record_limit` and `renew()` throw with a clear "requires v2 (Kafka 4.2+)" error against older brokers. + +### Patch Changes + +- ca7d22f: Correctness and reliability fixes across the producer, consumer, client protocol, and flow state stores. + + ### Producer + - `flush()` now waits for deferred drains so it no longer resolves while writes are still buffered. + - `transactionalSend` freezes its partition set up front to avoid double-calling the partitioner. + - Retry `TxnOffsetCommit` on retriable and rebalance-in-progress errors instead of failing the transaction. + - Fence the producer on `OUT_OF_ORDER_SEQUENCE_NUMBER` to surface the idempotence violation rather than silently corrupting the stream. + - Reject orphaned in-flight send promises during disconnect so callers are not left hanging. + + ### Consumer + - Always reschedule the heartbeat after non-fatal errors, preventing the consumer from dropping out of the group. + - Surface auto-commit errors to the consumer instead of swallowing them. + - Clear only the successfully committed partitions on a partial commit failure so the rest are retried. + - Fence stale fetch responses across partition replacement to avoid delivering records from a revoked assignment. + + ### Client / protocol + - Guard against varint shift overflow in the decoder. + - Close a `ConnectionPool.acquire` race that could hand out a connection mid-teardown. + - Clamp the SASL reauthentication delay to the `setTimeout` maximum. + - Don't await a response on `Produce` with `acks=0` (the broker sends none). + - Always use the v0 response header for `ApiVersions`, matching broker behavior across versions. + + ### Share consumer + - Release (rather than acknowledge) the last yielded message on shutdown so unprocessed records are redelivered. + + ### Flow + - Write to the local store before the changelog (local-first ordering) to shrink the inconsistency window on crash. + - Serialize the EOS commit triggered by a rebalance against the next message, and close the EOS rebalance offset-commit gap via an awaitable rebalance hook (shared with the consumer fix above). + - Use stream time (not wall-clock time) for window expiry. + - Implement hopping windows (`TimeWindows.advanceBy`). + - Throw on `SlidingWindows` instead of silently misbehaving (not yet implemented). + - Correct `TableGroupByNode` mapping ordering. + - Apply retention to `reduce` and session aggregators, emitting changelog tombstones for expired state. + - Surface checkpoint persistence errors instead of swallowing them. + + ### flow-state-lmdb + - Make `range()` upper bound inclusive, matching the in-memory store. + - Await `fsync` on checkpoint writes so a checkpoint is durable before it is reported complete. + - Correct `WindowStore` time-bound math. + - Use order-preserving signed-i64 encoding for time keys so range scans iterate in correct chronological order. + ## 0.7.0 ### Minor Changes diff --git a/packages/client/package.json b/packages/client/package.json index 6e66b3c..0a75b8f 100644 --- a/packages/client/package.json +++ b/packages/client/package.json @@ -1,7 +1,7 @@ { "name": "@kafkats/client", "private": false, - "version": "0.7.0", + "version": "0.8.0", "description": "Pure-protocol Kafka client, consumer/producer, and EOS patterns in TypeScript.", "type": "module", "main": "dist/index.js", diff --git a/packages/codec-zod/CHANGELOG.md b/packages/codec-zod/CHANGELOG.md index bca82d1..11df433 100644 --- a/packages/codec-zod/CHANGELOG.md +++ b/packages/codec-zod/CHANGELOG.md @@ -1,5 +1,64 @@ # @kafkats/codec-zod +## 0.8.0 + +### Minor Changes + +- fe140c0: ShareConsumer: Kafka 4.2 share groups GA support (ShareFetch/ShareAcknowledge v2) + - KIP-1206: new `acquireMode` config (`'batch_optimized'` default | `'record_limit'`) — `record_limit` strictly caps each fetch at `maxRecords`. + - KIP-1222: new `message.renew()` to extend the acquisition lock without finalizing delivery; safe to call multiple times per message. + - `ShareAcknowledgeResponse.acquisitionLockTimeoutMs` is now decoded on v2 responses. + - AckManager dedupes same-offset RENEW + finalizing-ack pairs and collapses duplicate same-offset RENEWs so the wire never carries overlapping batches. + - `@experimental` markers removed; share groups are GA in Kafka 4.2. + - Backward compatible: against Kafka 4.1 brokers the client negotiates v1 and produces byte-identical wire output to before this change. `record_limit` and `renew()` throw with a clear "requires v2 (Kafka 4.2+)" error against older brokers. + +### Patch Changes + +- ca7d22f: Correctness and reliability fixes across the producer, consumer, client protocol, and flow state stores. + + ### Producer + - `flush()` now waits for deferred drains so it no longer resolves while writes are still buffered. + - `transactionalSend` freezes its partition set up front to avoid double-calling the partitioner. + - Retry `TxnOffsetCommit` on retriable and rebalance-in-progress errors instead of failing the transaction. + - Fence the producer on `OUT_OF_ORDER_SEQUENCE_NUMBER` to surface the idempotence violation rather than silently corrupting the stream. + - Reject orphaned in-flight send promises during disconnect so callers are not left hanging. + + ### Consumer + - Always reschedule the heartbeat after non-fatal errors, preventing the consumer from dropping out of the group. + - Surface auto-commit errors to the consumer instead of swallowing them. + - Clear only the successfully committed partitions on a partial commit failure so the rest are retried. + - Fence stale fetch responses across partition replacement to avoid delivering records from a revoked assignment. + + ### Client / protocol + - Guard against varint shift overflow in the decoder. + - Close a `ConnectionPool.acquire` race that could hand out a connection mid-teardown. + - Clamp the SASL reauthentication delay to the `setTimeout` maximum. + - Don't await a response on `Produce` with `acks=0` (the broker sends none). + - Always use the v0 response header for `ApiVersions`, matching broker behavior across versions. + + ### Share consumer + - Release (rather than acknowledge) the last yielded message on shutdown so unprocessed records are redelivered. + + ### Flow + - Write to the local store before the changelog (local-first ordering) to shrink the inconsistency window on crash. + - Serialize the EOS commit triggered by a rebalance against the next message, and close the EOS rebalance offset-commit gap via an awaitable rebalance hook (shared with the consumer fix above). + - Use stream time (not wall-clock time) for window expiry. + - Implement hopping windows (`TimeWindows.advanceBy`). + - Throw on `SlidingWindows` instead of silently misbehaving (not yet implemented). + - Correct `TableGroupByNode` mapping ordering. + - Apply retention to `reduce` and session aggregators, emitting changelog tombstones for expired state. + - Surface checkpoint persistence errors instead of swallowing them. + + ### flow-state-lmdb + - Make `range()` upper bound inclusive, matching the in-memory store. + - Await `fsync` on checkpoint writes so a checkpoint is durable before it is reported complete. + - Correct `WindowStore` time-bound math. + - Use order-preserving signed-i64 encoding for time keys so range scans iterate in correct chronological order. + +- Updated dependencies [ca7d22f] +- Updated dependencies [fe140c0] + - @kafkats/client@0.8.0 + ## 0.7.0 ### Minor Changes diff --git a/packages/codec-zod/package.json b/packages/codec-zod/package.json index c156977..2de42ed 100644 --- a/packages/codec-zod/package.json +++ b/packages/codec-zod/package.json @@ -1,7 +1,7 @@ { "name": "@kafkats/codec-zod", "private": false, - "version": "0.7.0", + "version": "0.8.0", "description": "Zod codec helpers for @kafkats/client.", "type": "module", "main": "dist/index.js", diff --git a/packages/flow-state-lmdb/CHANGELOG.md b/packages/flow-state-lmdb/CHANGELOG.md index 977236b..0d45165 100644 --- a/packages/flow-state-lmdb/CHANGELOG.md +++ b/packages/flow-state-lmdb/CHANGELOG.md @@ -1,5 +1,64 @@ # @kafkats/flow-state-lmdb +## 0.8.0 + +### Minor Changes + +- fe140c0: ShareConsumer: Kafka 4.2 share groups GA support (ShareFetch/ShareAcknowledge v2) + - KIP-1206: new `acquireMode` config (`'batch_optimized'` default | `'record_limit'`) — `record_limit` strictly caps each fetch at `maxRecords`. + - KIP-1222: new `message.renew()` to extend the acquisition lock without finalizing delivery; safe to call multiple times per message. + - `ShareAcknowledgeResponse.acquisitionLockTimeoutMs` is now decoded on v2 responses. + - AckManager dedupes same-offset RENEW + finalizing-ack pairs and collapses duplicate same-offset RENEWs so the wire never carries overlapping batches. + - `@experimental` markers removed; share groups are GA in Kafka 4.2. + - Backward compatible: against Kafka 4.1 brokers the client negotiates v1 and produces byte-identical wire output to before this change. `record_limit` and `renew()` throw with a clear "requires v2 (Kafka 4.2+)" error against older brokers. + +### Patch Changes + +- ca7d22f: Correctness and reliability fixes across the producer, consumer, client protocol, and flow state stores. + + ### Producer + - `flush()` now waits for deferred drains so it no longer resolves while writes are still buffered. + - `transactionalSend` freezes its partition set up front to avoid double-calling the partitioner. + - Retry `TxnOffsetCommit` on retriable and rebalance-in-progress errors instead of failing the transaction. + - Fence the producer on `OUT_OF_ORDER_SEQUENCE_NUMBER` to surface the idempotence violation rather than silently corrupting the stream. + - Reject orphaned in-flight send promises during disconnect so callers are not left hanging. + + ### Consumer + - Always reschedule the heartbeat after non-fatal errors, preventing the consumer from dropping out of the group. + - Surface auto-commit errors to the consumer instead of swallowing them. + - Clear only the successfully committed partitions on a partial commit failure so the rest are retried. + - Fence stale fetch responses across partition replacement to avoid delivering records from a revoked assignment. + + ### Client / protocol + - Guard against varint shift overflow in the decoder. + - Close a `ConnectionPool.acquire` race that could hand out a connection mid-teardown. + - Clamp the SASL reauthentication delay to the `setTimeout` maximum. + - Don't await a response on `Produce` with `acks=0` (the broker sends none). + - Always use the v0 response header for `ApiVersions`, matching broker behavior across versions. + + ### Share consumer + - Release (rather than acknowledge) the last yielded message on shutdown so unprocessed records are redelivered. + + ### Flow + - Write to the local store before the changelog (local-first ordering) to shrink the inconsistency window on crash. + - Serialize the EOS commit triggered by a rebalance against the next message, and close the EOS rebalance offset-commit gap via an awaitable rebalance hook (shared with the consumer fix above). + - Use stream time (not wall-clock time) for window expiry. + - Implement hopping windows (`TimeWindows.advanceBy`). + - Throw on `SlidingWindows` instead of silently misbehaving (not yet implemented). + - Correct `TableGroupByNode` mapping ordering. + - Apply retention to `reduce` and session aggregators, emitting changelog tombstones for expired state. + - Surface checkpoint persistence errors instead of swallowing them. + + ### flow-state-lmdb + - Make `range()` upper bound inclusive, matching the in-memory store. + - Await `fsync` on checkpoint writes so a checkpoint is durable before it is reported complete. + - Correct `WindowStore` time-bound math. + - Use order-preserving signed-i64 encoding for time keys so range scans iterate in correct chronological order. + +- Updated dependencies [ca7d22f] +- Updated dependencies [fe140c0] + - @kafkats/flow@0.8.0 + ## 0.7.0 ### Minor Changes diff --git a/packages/flow-state-lmdb/package.json b/packages/flow-state-lmdb/package.json index 29b4976..8d828a1 100644 --- a/packages/flow-state-lmdb/package.json +++ b/packages/flow-state-lmdb/package.json @@ -1,7 +1,7 @@ { "name": "@kafkats/flow-state-lmdb", "private": false, - "version": "0.7.0", + "version": "0.8.0", "description": "LMDB state store provider for @kafkats/flow", "type": "module", "main": "dist/index.js", @@ -48,7 +48,7 @@ "lmdb": "^3.2.6" }, "peerDependencies": { - "@kafkats/flow": "workspace:>=0.7.0" + "@kafkats/flow": "workspace:>=0.8.0" }, "devDependencies": { "@kafkats/flow": "workspace:^", diff --git a/packages/flow/CHANGELOG.md b/packages/flow/CHANGELOG.md index b23d1c3..a8d71a7 100644 --- a/packages/flow/CHANGELOG.md +++ b/packages/flow/CHANGELOG.md @@ -1,5 +1,64 @@ # @kafkats/flow +## 0.8.0 + +### Minor Changes + +- fe140c0: ShareConsumer: Kafka 4.2 share groups GA support (ShareFetch/ShareAcknowledge v2) + - KIP-1206: new `acquireMode` config (`'batch_optimized'` default | `'record_limit'`) — `record_limit` strictly caps each fetch at `maxRecords`. + - KIP-1222: new `message.renew()` to extend the acquisition lock without finalizing delivery; safe to call multiple times per message. + - `ShareAcknowledgeResponse.acquisitionLockTimeoutMs` is now decoded on v2 responses. + - AckManager dedupes same-offset RENEW + finalizing-ack pairs and collapses duplicate same-offset RENEWs so the wire never carries overlapping batches. + - `@experimental` markers removed; share groups are GA in Kafka 4.2. + - Backward compatible: against Kafka 4.1 brokers the client negotiates v1 and produces byte-identical wire output to before this change. `record_limit` and `renew()` throw with a clear "requires v2 (Kafka 4.2+)" error against older brokers. + +### Patch Changes + +- ca7d22f: Correctness and reliability fixes across the producer, consumer, client protocol, and flow state stores. + + ### Producer + - `flush()` now waits for deferred drains so it no longer resolves while writes are still buffered. + - `transactionalSend` freezes its partition set up front to avoid double-calling the partitioner. + - Retry `TxnOffsetCommit` on retriable and rebalance-in-progress errors instead of failing the transaction. + - Fence the producer on `OUT_OF_ORDER_SEQUENCE_NUMBER` to surface the idempotence violation rather than silently corrupting the stream. + - Reject orphaned in-flight send promises during disconnect so callers are not left hanging. + + ### Consumer + - Always reschedule the heartbeat after non-fatal errors, preventing the consumer from dropping out of the group. + - Surface auto-commit errors to the consumer instead of swallowing them. + - Clear only the successfully committed partitions on a partial commit failure so the rest are retried. + - Fence stale fetch responses across partition replacement to avoid delivering records from a revoked assignment. + + ### Client / protocol + - Guard against varint shift overflow in the decoder. + - Close a `ConnectionPool.acquire` race that could hand out a connection mid-teardown. + - Clamp the SASL reauthentication delay to the `setTimeout` maximum. + - Don't await a response on `Produce` with `acks=0` (the broker sends none). + - Always use the v0 response header for `ApiVersions`, matching broker behavior across versions. + + ### Share consumer + - Release (rather than acknowledge) the last yielded message on shutdown so unprocessed records are redelivered. + + ### Flow + - Write to the local store before the changelog (local-first ordering) to shrink the inconsistency window on crash. + - Serialize the EOS commit triggered by a rebalance against the next message, and close the EOS rebalance offset-commit gap via an awaitable rebalance hook (shared with the consumer fix above). + - Use stream time (not wall-clock time) for window expiry. + - Implement hopping windows (`TimeWindows.advanceBy`). + - Throw on `SlidingWindows` instead of silently misbehaving (not yet implemented). + - Correct `TableGroupByNode` mapping ordering. + - Apply retention to `reduce` and session aggregators, emitting changelog tombstones for expired state. + - Surface checkpoint persistence errors instead of swallowing them. + + ### flow-state-lmdb + - Make `range()` upper bound inclusive, matching the in-memory store. + - Await `fsync` on checkpoint writes so a checkpoint is durable before it is reported complete. + - Correct `WindowStore` time-bound math. + - Use order-preserving signed-i64 encoding for time keys so range scans iterate in correct chronological order. + +- Updated dependencies [ca7d22f] +- Updated dependencies [fe140c0] + - @kafkats/client@0.8.0 + ## 0.7.0 ### Minor Changes diff --git a/packages/flow/package.json b/packages/flow/package.json index 03b12ed..bdf68f6 100644 --- a/packages/flow/package.json +++ b/packages/flow/package.json @@ -1,7 +1,7 @@ { "name": "@kafkats/flow", "private": false, - "version": "0.7.0", + "version": "0.8.0", "description": "Flow-level APIs built on top of @kafkats/client.", "type": "module", "main": "dist/index.js",