How to orchestrate a Kafka streaming data processing workflow using Langflow?

[y-d-y]

Langflow is an incredibly powerful workflow engine framework, and I’ve already developed a decent understanding of its source code. However, Langflow workflows are currently triggered via API calls (or via python simple_run_flow), which means each workflow is executed only once per call.

I'm wondering if there's a way to turn a Langflow workflow into a long-running task — for example, continuously consuming data from Kafka, processing it through a Langflow workflow, and outputting the results back to Kafka — essentially having the workflow run as a streaming processor.

More specifically, here’s what I hope to achieve:

1. Add a Kafka Component to Langflow and be able to drag it onto the canvas.
2. Reuse existing data processing components in Langflow to build a streaming message processing pipeline visually.
3. Output the processed messages back to Kafka.
4. The workflow may involve multiple reads and writes to Kafka — for example:
   Kafka Component -> Filter Component -> Map Component -> Kafka Component -> Reduce Component -> Redis Component, etc., and could also include branching and conditional logic.

However, since Langflow currently only supports execution via API calls (or simple_run_flow), it's not suitable for running continuous streaming tasks.

I’d love to hear if anyone has ideas or suggestions on how to make this work.

[Grogorydm]

Hi! Have you already resolve your problem?
And I also need to integrate with Kafka inside of LF scenario only fo testing, but just looking for the worked custom component Kafka for langflow.