Awesome Kafka

A curated list of awesome things related to Apache Kafka.

Contents

• Libraries
  • Kafka
  • Kafka UI
  • Kafka Streams
  • Kafka Connect
  • REST Proxy
  • KSQL
  • Schema Registry
  • Other Awesome Kafka
  • Kafkaesque
• Resources
  • Learning
  • Blogs
  • Books

Libraries

Kafka

• topicctl - A tool for easy, declarative management of Kafka topics. Includes the ability to "apply" topic changes from YAML as well as a repl for interactive exploration of brokers, topics, consumer groups, messages, and more.
• sarama - Sarama is an MIT-licensed Go client library for Apache Kafka version 0.8 (and later).
• kadeck - Apache Kafka Monitoring, Kafka UI and data platform for Desktop or Web.
• conduktor
• mirus - Mirus is a cross data-center data replication tool for Apache Kafka.
• kareldb - A Relational Database Backed by Apache Kafka.
• kcache - An In-Memory Cache Backed by Apache Kafka.
• kafka-lag-based-assignor - Kafka partition assignor that distributes lag evenly across a consumer group.
• kafka-pixy - Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with automatic consumer group control.
• Burrow - Kafka Consumer Lag Checking.
• kafka-tools - A collection of tools for working with Apache Kafka.
• schema-registry-ui - Web tool for Avro Schema Registry.
• kafkacat - Generic command line non-JVM Apache Kafka producer and consumer.
• kafka-jackson - Kafka Serializer, Deserializer, and Serde for Jackson JSON.
• kafkahq - Kafka GUI for topics, topics data, consumers group, schema registry, connect and more.
• secor - Secor is a service implementing Kafka log persistence.
• reactive-kafka - Alpakka Kafka connector - Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
• kafka-monitor
• chaperone - A Kafka audit system.
• uReplicator - Improvement of Apache Kafka Mirrormaker.
• kafka-websocket
• kafka-unit
• cruise-control - Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a kafka cluster.
• jocko - Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native).
• hermes - Fast and reliable message broker built on top of Kafka.
• kt - Kafka command line tool.
• kafka-eagle - Used to monitor the consumer status of Kafka clusters, as well as offsets, metadata and other information.
• dockerkafka - DoctorKafka is a service for Kafka cluster auto healing and workload balancing.
• kasper - Kasper is a lightweight library for processing Kafka topics.
• Yelp kafka-utils
• kafka-spark-consumer
• kafka-streams-cep - Complex Event Processing on top of Kafka Streams.
• spring-kafka
• ksql-machine-learning-udf
• kafkabeat - Elasticsearch Beats for kafka.
• sangrenel
• Strimzi - Operator for deploying and running Apache Kafka on Kubernetes and OpenShift.
• Kafka Manager - Web-based tool for managing a Kafka cluster.
• Strimzi Kafka CLI - A CLI for Strimzi Kafka Operator.
• Kafka Cluster Kraft Mode - DockerCompose - Workable kafka cluster with kraft mode using docker-compose.
• Decaton - High throughput asynchronous task processing framework

Kafka UI

• KafkaTrail
• kafka-topics-ui
• kafka-connect-ui - Web tool for Kafka Connect.
• Kafdrop - Web UI for browsing Kafka topics and consumer groups.
• kafka-webview
• kafka-view
• cruise-control-ui
• real-time-ui-with-kafka-streams
• tsujun - Yet another Web UI for KSQL.
• kowl
• akhq

Kafka Streams

• schema-registry-ui
• kafka-streams-consumer-offsets-to-json - A Kafka Streams process to convert consumer_offsets to a JSON-readable topic.
• kafka-operator
• kafkabalancer
• mockedstreams
• winton-kafka-streams - A Python implementation of Apache Kafka Streams.

Kafka Connect

• Apache Camel Kafka Connect - 340+ Apache Camel components as Kafka Connect connectors.
• kafka-connect-client - A kafka-connect REST api client for java.
• Confluent Connector Hub
• kafka-connect-protobuf-converter - Protobuf converter plugin for Kafka Connect.
• Maxwell
• kafka-connect-transformers
• kafka-connect-mq-source
• kafka-connect-jenkins
• toketi-kafka-connect-iohub
• connectctl - Manage kafka connect connectors easily.
• kafka-connect-transform-common - Common Transforms for Kafka Connect.
• aiven-kafka-connect-transforms - A collection of Single Message Transformations (SMTs) for Kafka Connect.
• kafka-connect-file-pulse - A polyvalent, scalable and reliable, Kafka Connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka.
• kafka-connect-http
• snowflake-kafka-connector

REST Proxy

• kafka-rest - Confluent REST Proxy.
• strimzi-http-bridge - Strimzi Kafka Bridge (AMQP & HTTP).
• Zilla - An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT and the native Kafka protocol.

KSQL

Schema Registry

• apicurio-registry - Apicurio API/schema registry (includes UI).
• schema-registry - Confluent Schema registry for Kafka.
• ksql-jdbc-driver
• ballerina-schema-registry - Ballerina Confluent Schema Registry

Other Awesome Kafka

• ApacheKafka
• dharmeshkakadia/awesome-kafka
• infoslack/awesome-kafka
• awesome-kafka-playground
• ballerina-avro-kafka-serializer - Ballerina Confluent Avro Serializer/Deserializer

Kafkaesque

• Pulsar
• Flink
• Spark
• Heron
• Beam
• Arrow
• Samza
• Nats
• Chronicle-Queue
• CorfuDB
• GenStage
• faust - Python Stream Processing.
• redpanda
• Meteor - Lightweight, broker-less alternative to Kafka for message streaming.
• Bufstream - Store directly to Apache Iceberg™ tables and guarantee data quality with Bufstream, a drop-in replacement for Apache Kafka®.

Resources

Learning

• Passionate Developer: Kafka Streams DSL vs processor API
• Mastering Kafka Streams
• Spark Streaming + Kafka Integration Guide

Blogs

• CodingJunkie - Random Thoughts on Coding by Bill Bejeck.
• Understanding Kafka with Factorio

Books

• Kafka Streams in Action
• Kafka in Action
• Kafka the Definitive Guide
• Streaming Data Pipelines with Kafka