Kafka UI - Open-Source Web UI for Apache Kafka Management

•    TypeScript

---

Kafka UI is a UI for Apache Kafka to monitor and manage Apache Kafka clusters. It is a simple tool that makes your data flows observable, helps find and troubleshoot issues faster and deliver optimal performance. Its lightweight dashboard makes it easy to track key metrics of your Kafka clusters - Brokers, Topics, Partitions, Production, and Consumption.

kafka big-data web-ui streams kafka-connect apache-kafka-ui kafka-producer kafka-client kafka-streams hacktoberfest streaming-data kafka-manager kafka-cluster event-streaming cluster-management kafka-brokers

message-db - Microservice native message and event store for Postgres

•    Shell

---

A fully-featured event store and message store implemented in PostgreSQL for Pub/Sub, Event Sourcing, Messaging, and Evented Microservices applications. An event sourcing and Pub/Sub message store built on Postgres for simple cloud or local hosting. A minimalist implementation of the essential features of tools like Event Store or Kafka, with built-in support for messaging patterns like Pub/Sub, and consumer patterns like consumer groups.

postgres event-stream event-sourcing message-queue event-store event-driven pub-sub

kafka-python - Python client for Apache Kafka

•    Python

---

Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the official java client, with a sprinkling of pythonic interfaces (e.g., consumer iterators).kafka-python is best used with newer brokers (0.9+), but is backwards-compatible with older versions (to 0.8.0). Some features will only be enabled on newer brokers. For example, fully coordinated consumer groups -- i.e., dynamic partition assignment to multiple consumers in the same group -- requires use of 0.9+ kafka brokers. Supporting this feature for earlier broker releases would require writing and maintaining custom leadership election and membership / health check code (perhaps using zookeeper or consul). For older brokers, you can achieve something similar by manually assigning different partitions to each consumer instance with config management tools like chef, ansible, etc. This approach will work fine, though it does not support rebalancing on failures. See <https://kafka-python.readthedocs.io/en/master/compatibility.html> for more details.

librdkafka - The Apache Kafka C/C++ library

•    C

---

Copyright (c) 2012-2016, Magnus Edenhill.librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

kafka kafka-consumer apache-kafka high-performance librdkafka kafka-producer c-plus-plus consumer kafka-client kafka-library

kafkacat - Generic command line non-JVM Apache Kafka producer and consumer

•    C

---

kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka.In producer mode kafkacat reads messages from stdin, delimited with a configurable delimiter (-D, defaults to newline), and produces them to the provided Kafka cluster (-b), topic (-t) and partition (-p).

kafka-client kafka-library kafka

karafka - Framework used to simplify Apache Kafka based Ruby and Rails applications development.

•    Ruby

---

Framework used to simplify Apache Kafka based Ruby applications development.It allows programmers to use approach similar to standard HTTP conventions (params and params_batch) when working with asynchronous Kafka messages.

karafka-framework kafka-topic kafka kafka-client kafka-clients kafka-producer kafka-consumer apache-kafka kafka-message karafka-application sidekiq rails kafka-ruby ruby-on-rails rubygems rubygem ruby-library kafka-library

kafka-node - Node.js client for Apache Kafka 0.8 and later.

•    Javascript

---

Kafka-node is a Node.js client with Zookeeper integration for Apache Kafka 0.8.1 and later. Follow the instructions on the Kafka wiki to build Kafka 0.8 and get a test broker up and running.

kafka zookeeper consumer producer broker

kq - Kafka-based Job Queue for Python

•    Python

---

KQ (Kafka Queue) is a lightweight Python library which lets you queue and execute jobs asynchronously using Apache Kafka. It uses kafka-python under the hood. You may need to use sudo depending on your environment.

kafka job-queue async kafka-client kafka-consumer kafka-producer python3 python2 python-library python-3 python-2 asynchronous queueing jobqueue apache-kafka worker-queue producer-consumer serialization python-3-5

kafka-topics-ui - Web Tool for Kafka Topics using Kafka Rest |

•    Javascript

---

Browse Kafka topics and understand what's happening on your cluster. Find topics / view topic metadata / browse topic data (kafka messages) / view topic configuration / download data. This is a web tool for the confluentinc/kafka-rest proxy. Config: If you don't use our docker image, keep in mind that Kafka-REST-Proxy CORS support can be a bit buggy, so if you have trouble setting it up, you may need to provide CORS headers through a proxy (i.e. nginx).

schema registry kafka topics avro

kafka-stack-docker-compose - docker compose files to create a fully working kafka stack

•    Shell

---

This replicates as well as possible real deployment configurations, where you have your zookeeper servers and kafka servers actually all distinct from each other. This solves all the networking hurdles that comes with Docker and docker-compose, and is compatible cross platform. This configuration fits most development requirements.

docker-compose docker kafka-schema-registry kafka kafka-topics-ui kafka-rest-proxy zookeeper

fast-data-dev - Kafka Docker for development

•    Shell

---

Apache Kafka docker image for developers; with Landoop Lenses (landoop/kafka-lenses-dev) or Landoop's open source UI tools (landoop/fast-data-dev). Have a full fledged Kafka installation up and running in seconds and top it off with a modern streaming platform (only for kafka-lenses-dev), intuitive UIs and extra goodies. Also includes Kafka Connect, Schema Registry, Landoop Stream Reactor 25+ Connectors and more.

kafka-rest-proxy schema-registry kafka docker

Redpanda - A Kafka API compatible streaming platform for mission-critical workloads

•    C++

---

Redpanda is a streaming platform for mission critical workloads. It is Kafka API compatible so any client that works with Kafka should work out of the box with Redpanda. It does not have dependency on ZooKeeper, JVM. Redpanda is a new storage engine, optimized for streaming data, using a thread-per-core architecture focused on delivering stable tail-latencies. It transforms data inline with embedded WASM engine.

message-queue streaming kafka cpp containers realtime kubernetes queue kafka-protocol

ruby-kafka - A Ruby client library for Apache Kafka

•    Ruby

---

A Ruby client library for Apache Kafka, a distributed log and message bus. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.Although parts of this library work with Kafka 0.8 – specifically, the Producer API – it's being tested and developed against Kafka 0.9. The Consumer API is Kafka 0.9+ only.

kafka-client kafka ruby-gem kafka-library

watermill - Building event-driven applications easy way in Go.

•    Go

---

Watermill is a Go library for working efficiently with message streams. It is intended for building event driven applications, enabling event sourcing, RPC over messages, sagas and basically whatever else comes to your mind. You can use conventional pub/sub implementations like Kafka or RabbitMQ, but also HTTP or MySQL binlog if that fits your use case. Note: Watermill should run reliably in a production environment, but it is still under heavy development and the public API may change before the 1.0.0 release.

watermill event-driven event-sourcing sagas events stream-processing reactive rabbitmq kafka cqrs

pykafka - Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance

•    Python

---

PyKafka is a cluster-aware Kafka>=0.8.2 client for Python. It includes Python implementations of Kafka producers and consumers, which are optionally backed by a C extension built on librdkafka, and runs under Python 2.7+, Python 3.4+, and PyPy.PyKafka's primary goal is to provide a similar level of abstraction to the JVM Kafka client using idioms familiar to Python programmers and exposing the most Pythonic API possible.

kafka c-extension apache-kafka kafka-client kafka-library

confluent-kafka-go - Confluent's Apache Kafka Golang client

•    Go

---

confluent-kafka-go is Confluent's Golang client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-go is a lightweight wrapper around librdkafka, a finely tuned C client.

confluent golang-library golang-bindings librdkafka kafka-client consumer producer kafka-library kafka

spring-integration-kafka

•    Java

---

The Spring Integration Kafka extension project provides inbound and outbound channel adapters for Apache Kafka. Apache Kafka is a distributed publish-subscribe messaging system that is designed for high throughput (terabytes of data) and low latency (milliseconds). For more information on Kafka and its design goals, see the Kafka main page.Starting from version 2.0 version this project is a complete rewrite based on the new spring-kafka project which uses the pure java Producer and Consumer clients provided by Kafka 0.9.x.x and 0.10.x.x..

Debezium - Stream changes from your databases.

•    Java

---

Debezium is a distributed platform that turns your existing databases into event streams, so applications can see and respond immediately to each row-level change in the databases. Debezium is built on top of Apache Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, from where your application consumes them. This makes it possible for your application to easily consume all of the events correctly and completely.

change-data-capture kafka-connect apache-kafka debezium cdc database kafka kafka-producer database-migration events

kafka - Load-balancing, resuming Kafka consumer for go, backed by Zookeeper.

•    Go

---

Kafka libraries, tools and example applications built on top of the sarama package.The following tools can be useful for discovery, testing, and benchmarking. They also serve as examples of how to use Sarama.

kafka load-balancer

strimzi-kafka-operator - Apache Kafka running on Kubernetes and OpenShift

•    Java

---

Strimzi provides a way to run an Apache Kafka cluster on Kubernetes or OpenShift in various deployment configurations. See our website for more details about the project. Documentation to the current master branch as well as all releases can be found on our website.

kafka kubernetes openshift docker messaging enmasse kafka-connect kafka-streams data-streaming data-stream data-streams kubernetes-operator kubernetes-controller