librdkafka - The Apache Kafka C/C++ library

  •        54

Copyright (c) 2012-2016, Magnus Edenhill.librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.



Related Projects

confluent-kafka-go - Confluent's Apache Kafka Golang client

confluent-kafka-go is Confluent's Golang client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-go is a lightweight wrapper around librdkafka, a finely tuned C client.

pykafka - Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance

PyKafka is a cluster-aware Kafka>=0.8.2 client for Python. It includes Python implementations of Kafka producers and consumers, which are optionally backed by a C extension built on librdkafka, and runs under Python 2.7+, Python 3.4+, and PyPy.PyKafka's primary goal is to provide a similar level of abstraction to the JVM Kafka client using idioms familiar to Python programmers and exposing the most Pythonic API possible.

confluent-kafka-python - Confluent's Apache Kafka Python client

confluent-kafka-python is Confluent's Python client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-python is a lightweight wrapper around librdkafka, a finely tuned C client.

confluent-kafka-dotnet - Confluent's Apache Kafka .NET client

confluent-kafka-dotnet is Confluent's .NET client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-dotnet is a lightweight wrapper around librdkafka, a finely tuned C client.

csi-kafka - C++11 producer/consumer library for Apache Kafka v0

C++11 producer/consumer library for Apache Kafka v0.10.1.0+. Does not require zookeeper integration

kafkacat - Generic command line non-JVM Apache Kafka producer and consumer

kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka.In producer mode kafkacat reads messages from stdin, delimited with a configurable delimiter (-D, defaults to newline), and produces them to the provided Kafka cluster (-b), topic (-t) and partition (-p).


This gem is primarily used to wrap most of the [Kafka high level consumer] and [Kafka producer] API into jruby. The [Kafka Consumer Group Example] is pretty much ported to this library.This package is now distributed via but you can build it using the following instructions.

ruby-kafka - A Ruby client library for Apache Kafka

A Ruby client library for Apache Kafka, a distributed log and message bus. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.Although parts of this library work with Kafka 0.8 – specifically, the Producer API – it's being tested and developed against Kafka 0.9. The Consumer API is Kafka 0.9+ only.

php-rdkafka - Kafka client for PHP

PHP-rdkafka is a thin librdkafka binding providing a working PHP 5 / PHP 7 Kafka 0.8 / 0.9 / 0.10 client.It supports the high level and low level consumers, producer, and metadata APIs.

kafka-pixy - gRPC/REST proxy for Kafka

Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with automatic consumer group control. It is designed to hide the complexity of the Kafka client protocol and provide a stupid simple API that is trivial to implement in any language.Kafka-Pixy supports Kafka versions form 0.8.2.x to 0.10.1.x. It uses the Kafka Offset Commit/Fetch API to keep track of consumer offsets. However Group Membership API is not yet implemented, therefore it needs to talk to Zookeeper directly to manage consumer group membership.

kafka-php - Simple and high level consumer and producer client for Kafka Broker (0.7.x)

This is an alternative to the existing Kafka PHP Client which is in the incubator, the main motivation to write it was that it seemed important that the fetch requests are not loaded entirely into memory but pulled continuously from the socket as well as the fact that PHP has a different control flow and communication pattern (each request is a thread within HTTP server) so the api doesn't need to follow the scala/java object graph and can be much simpler.There is a set of example scripts under the 'script' folder. The parameters convention are shared across all the scripts but different scripts has different parameters.

rdkafka-dotnet - C# Apache Kafka client

Please update to Confluent.Kafka.rdkafka-dotnet is a C# client for Apache Kafka based on librdkafka.

kafka-python - Python client for Apache Kafka

Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the official java client, with a sprinkling of pythonic interfaces (e.g., consumer iterators).kafka-python is best used with newer brokers (0.9+), but is backwards-compatible with older versions (to 0.8.0). Some features will only be enabled on newer brokers. For example, fully coordinated consumer groups -- i.e., dynamic partition assignment to multiple consumers in the same group -- requires use of 0.9+ kafka brokers. Supporting this feature for earlier broker releases would require writing and maintaining custom leadership election and membership / health check code (perhaps using zookeeper or consul). For older brokers, you can achieve something similar by manually assigning different partitions to each consumer instance with config management tools like chef, ansible, etc. This approach will work fine, though it does not support rebalancing on failures. See <> for more details.

CSharpClient-for-Kafka -

This library was created for Kafka 0.8 with an intention to have a native library built from scratch. With Kafka protocol getting updated frequently with new features (which is expected until it reaches version 1.0), it doesn't seem beneficial to maintain a library built from scratch. The right approach (and as suggested by Confluent) for now would be to use a C# wrapper around the librdkafka C-Library, which the confluent-kafka-dotnet client is doing.So, if you are using Kafka 0.9 or higher, please move to using the confluent-kafka-dotnet client library.

uReplicator - Improvement of Apache Kafka Mirrormaker

uReplicator provides the ability to replicate across Kafka clusters in other data centers. Instead of publishing to a single Kafka cluster, you can publish data to multiple regional Kafka clusters and aggregate it all in one Kafka cluster.Kafka's current (part of 0.8.2) MirrorMaker design consumes data from a given regional Kafka cluster using a Kafka high-level consumer. With this design, rebalancing in the high level consumer (due to a addition/deletion of topics, source cluster problems, network issues and so on) affects all the topics being replicated via that Mirrormaker.

kafka-fast - fast kafka api for JVM languages implemented in clojure

The documentation contains examples in both clojure and java. From the Java APIs you can use Scala, JRuby, Groovy etc.Note that at the moment only the public producer and consumer APIs have direct Java interfaces, internal APIs like direct producer access and direct metadata access are for the moment only in clojure, albeit you can still access them from Java using the clojure.lang.RT object.

dory - Producer daemon for Apache Kafka

Dory decouples message sources from the Kafka cluster. A client is not forced to wait for an ACK after sending a message, since Dory handles the details of waiting for ACKs from Kafka and resending messages when necessary. Likewise, a client is not burdened with holding onto messages until it has a reasonable-sized batch to send to Kafka. If a client crashes immediately after sending a message to Dory, the message is safe with Dory. However, if the client assumes responsibility for interacting with Kafka, a crash will cause the loss of all batched messages, and possibly sent messages for which an ACK is pending.Dory provides uniformity of mechanism for status monitoring and data quality reporting through its web interface. Likewise, it provides a unified configuration mechanism for settings related to batching, compression, and other aspects of interaction with Kafka. This simplifies system administration, as compared to a multitude of producer mechanisms for various programming languages and applications, each with its own status monitoring, data quality reporting, and configuration mechanisms or lack thereof.