librdkafka - The Apache Kafka C/C++ library

•    C

Copyright (c) 2012-2016, Magnus Edenhill.librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

kafka-storm-starter - Code examples that show to integrate Apache Kafka 0

•    Scala

Code examples that show how to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark 1.1+ while using Apache Avro as the data serialization format. Take a look at the Kafka Streams code examples at https://github.com/confluentinc/examples.

pykafka - Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance

•    Python

PyKafka is a cluster-aware Kafka>=0.8.2 client for Python. It includes Python implementations of Kafka producers and consumers, which are optionally backed by a C extension built on librdkafka, and runs under Python 2.7+, Python 3.4+, and PyPy.PyKafka's primary goal is to provide a similar level of abstraction to the JVM Kafka client using idioms familiar to Python programmers and exposing the most Pythonic API possible.

Debezium - Stream changes from your databases.

•    Java

Debezium is a distributed platform that turns your existing databases into event streams, so applications can see and respond immediately to each row-level change in the databases. Debezium is built on top of Apache Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, from where your application consumes them. This makes it possible for your application to easily consume all of the events correctly and completely.

karafka - Framework used to simplify Apache Kafka based Ruby and Rails applications development.

•    Ruby

Framework used to simplify Apache Kafka based Ruby applications development.It allows programmers to use approach similar to standard HTTP conventions (params and params_batch) when working with asynchronous Kafka messages.

kt - Kafka command line tool

•    Go

You can download kt via the Releases section.

wirbelsturm - Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data related infrastructure

•    Shell

Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data related infrastructure. Wirbelsturm's goal is to make tasks such as "I want to deploy a multi-node Storm cluster" simple, easy, and fun.

kq - Kafka-based Job Queue for Python

•    Python

KQ (Kafka Queue) is a lightweight Python library which lets you queue and execute jobs asynchronously using Apache Kafka. It uses kafka-python under the hood. You may need to use sudo depending on your environment.

Agile_Data_Code_2 - Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

•    Jupyter

Like my work? I am Principal Consultant at Data Syndrome, a consultancy offering assistance and training with building full-stack analytics products, applications and systems. Find us on the web at datasyndrome.com. There is now a video course using code from chapter 8, Realtime Predictive Analytics with Kafka, PySpark, Spark MLlib and Spark Streaming. Check it out now at datasyndrome.com/video.

waterdrop - Ruby-Kafka based aspect oriented library for Kafka messages generating

•    Ruby

Gem used to send messages to Kafka in an easy way.message that you want to send should be either binary or stringified (to_s, to_json, etc).

kafka-connect-hdfs - Kafka Connect HDFS connector

•    Java

kafka-connect-hdfs is a Kafka Connector for copying data between Kafka and Hadoop HDFS.Documentation for this connector can be found here.

akka-stream-eventsourcing - Event sourcing for Akka Streams

•    Scala

Methods emit and respond themselves are side-effect free. They only generate Emission values that are interpreted by the EventSourcing stage.

kafka-notebook - Notes about Apache Kafka

Notes about Apache Kafka

awesome-kafka - A list about Apache Kafka

This list is for anyone wishing to learn about Apache Kafka, but do not have a starting point. You can help by sending Pull Requests to add more information.

spark-kafka-sink - A Kafka metric sink for Apache Spark

•    Scala

A Kafka metric sink for Apache Spark

cppkafka - Modern C++ Apache Kafka client library (wrapper for librdkafka)

•    C++

cppkafka allows C++ applications to consume and produce messages using the Apache Kafka protocol. The library is built on top of librdkafka, and provides a high level API that uses modern C++ features to make it easier to write code while keeping the wrapper's performance overhead to a minimum. cppkafka is a high level C++ wrapper for rdkafka, aiming to allow using rdkafka in a simple, less error prone way.

docker-kafka-zookeeper - Docker image for Kafka (0

•    Shell

Docker image for Kafka (0.10.x - 0.11.x - 1.0.x) message broker including Zookeeper

kafka-streams-scala-examples

•    Scala

This repository contains code examples for Kafka Streams API.

docker-kafka-alpine - Alpine Linux based Kafka Docker Image

•    Shell

This will create a single-node kafka broker (listening on localhost:9092), a local zookeeper instance and create the topic test-topic with 1 replication-factor and 1 partition.

storm-dynamic-spout - A framework for building spouts for Apache Storm and a Kafka based spout for dynamically skipping messages to be processed later

•    Java

A framework for building spouts for Apache Storm and a Kafka based spout for dynamically skipping messages to be processed later.