Add Kafka Streams runner

[kennknowles]

Kafka Streams (https://kafka.apache.org/documentation/streams) has more and more features that could make it a viable candidate for a streaming runner. It uses DataFlow-like model.

 

Please look at the Design Document and add comments.

Imported from Jira BEAM-2466. Original Jira may contain additional context.
Reported by: klorand.

[wilsonwang371]

I am actually very interested in this topic. Generally, it is possible to have beam running on a generic faas system that with MQ support?

[je-ik]

There are several requirements that must be met. Essentially:
a) preserving per-partition order of records (i.e. records emitted in order from one distributed producer must not overtake each other when consumed)
b) producer must be able to enqueue output records for a specific consumer (e.g. assigning a key of a output record, all records with same key must then be consumed by the same instance of downstream consumer)
c) producer must be able to send record to all downstream consumers (i.e. producer must know how many consumers there - possibly - is)
d) there must be some kind of support of state commit, either at the end of bundle, during bundle commit (dataflow model), or as a flowing checkpoint barrier (flink model), there must be a way to safely store state in a distributed fault tolerant storage and be able to possibly restore the complete state from that committed state

Having these conditions met I think it should be possible (though quite hard) to implement Beam runner on top of it. Kafka definitely has all four (even without Kafka streams).

[wilsonwang371]

There are several requirements that must be met. Essentially: a) preserving per-partition order of records (i.e. records emitted in order from one distributed producer must not overtake each other when consumed) b) producer must be able to enqueue output records for a specific consumer (e.g. assigning a key of a output record, all records with same key must then be consumed by the same instance of downstream consumer) c) producer must be able to send record to all downstream consumers (i.e. producer must know how many consumers there - possibly - is) d) there must be some kind of support of state commit, either at the end of bundle, during bundle commit (dataflow model), or as a flowing checkpoint barrier (flink model), there must be a way to safely store state in a distributed fault tolerant storage and be able to possibly restore the complete state from that committed state

Having these conditions met I think it should be possible (though quite hard) to implement Beam runner on top of it. Kafka definitely has all four (even without Kafka streams).

Thank you so much for the reply.

[junaiddshaukat]

Hi, I'm interested in working on the portable Kafka Streams Runner
as part of GSoC 2026 (if it got selected).

I've been contributing to the Apache Beam TypeScript SDK — my merged
PRs include LRU cache eviction for CachingStateProvider (#37214) and
CombinePerKeyPrecombineOperator (#37466). I'm now ramping up on the
Beam portability framework and Kafka Streams to prepare for this project.

I've read through the design document and the previous attempts by
Kai Jiang and Kyle Winkelman. A few questions:

1. Should the GSoC skeleton target the portable runner API (Fn API /
   gRPC), or a classic translation-based approach?
2. What's the minimum set of transforms the skeleton should support —
   Read, ParDo, GroupByKey?
3. Is there a preference for using the Kafka Streams DSL vs the
   lower-level Processor API for the translation layer?

I'll be studying the Flink portable runner as a reference implementation.
Happy to discuss the design further.

[je-ik]

Hi @junaiddshaukat, great to hear about your interest. It's also cool that you have such a big insight into the Beam ecosystem. Maybe we can put down the design document for the GSoC together? Generally speaking, my point of view would be that:

1. the skeleton should target the FnAPI to have full support of all SDKs (it should be possible to optimize the translation for Java SDK later)
2. the skeleton should be "useful", i.e. it should be possible to run at least some basic Pipelines, that would imply we should target to implement at least:

• Read
• stateless ParDo
• GBK
• CBK
• Window
• optional: stateful ParDo, later splittable DoFn

3. this is crucial question that needs deeper analysis to make sure that the DSL aligns correctly with the Apache Beam model. My intuition here would be that the Processor API would be a better choice, because it is flexible enough to support the model and yet it should not force us to manually manage state, etc.

We would also need to define the design of watermarks, bundles (we must pay attention to Beam's guarantees and make the bundling compatible), timers (if we would support stateful ParDo).

Would you like to try to sketch a design document we could iterate on? When we create some basic doc, we can then share it on the dev@ list to get more feedback and ensure we don't hit a wall during implementation.

[junaiddshaukat]

Thank you @je-ik for the detailed response and the clear direction!
I'd love to sketch a design document and iterate on it with you.

Your points give a solid foundation to start from:

1. Fn API as the target — agreed, this ensures all SDKs work from
   day one with room for Java-specific optimizations later.
2. Processor API over DSL — makes sense given the flexibility
   needed for Beam's model. I'll analyze the alignment more deeply
   in the design doc.
3. Minimum transforms: Read, stateless ParDo, GBK, CBK, Window —
   with stateful ParDo and splittable DoFn as stretch goals.

I'll start drafting the design document this week, covering:

• High-level architecture (pipeline translation flow)
• Mapping of Beam primitives to Kafka Streams Processor API constructs
• Watermark propagation strategy
• Bundle management and Beam's guarantees
• Timer support considerations

I'll share the initial draft here for your feedback before we take it
to the dev@ list. Looking forward to collaborating!

[junaiddshaukat]

Hi @je-ik, I've put together an initial draft of the design document
for the portable Kafka Streams runner:

https://docs.google.com/document/d/1BBMURhSG4SxPcvvnKMTrmnKCr_jhXL6R4TBDBW7zsy8/edit?tab=t.0

It covers the high-level architecture, pipeline translation approach
(following the Flink portable runner pattern), transform mappings for
Read, ParDo, GBK, CBK, Window, and Flatten, along with sections on
watermark management, bundle handling, and state management.

I've included a few architecture diagrams and listed open questions at
the end — particularly around watermark derivation from Kafka consumer
positions, repartitioning strategy for GBK, and bundle-commit alignment
for exactly-once semantics.

Looking forward to your feedback so we can iterate on this before
sharing it on the dev@ list.

[je-ik]

That's awesome document, I will walk through it ASAP, during the weekend latest. Thanks a lot for this!

[junaiddshaukat]

Hi @je-ik,

I’ve revised the Design Document based on your suggestions. Could you please take a look when you have a moment and let me know your feedback?

Thank you!