winton-kafka-streams - A Python implementation of Apache Kafka Streams

•    Python

---

Implementation of Apache Kafka's Streams API in Python. Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. Kafka has Streams API added for building stream processing applications using Apache Kafka. Applications built with Kafka's Streams API do not require any setup beyond the provision of a Kafka cluster.

kafka-streams streaming-api librdkafka

alpakka-kafka - Alpakka Kafka connector - Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka

•    Scala

---

Systems don't come alone. In the modern world of microservices and cloud deployment, new components must interact with legacy systems, making integration an important key to success. Reactive Streams give us a technology-independent tool to let these heterogeneous systems communicate without overwhelming each other. The Alpakka project is an open source initiative to implement stream-aware, reactive, integration pipelines for Java and Scala. It is built on top of Akka Streams, and has been designed from the ground up to understand streaming natively and provide a DSL for reactive and stream-oriented programming, with built-in support for backpressure. Akka Streams is a Reactive Streams and JDK 9+ java.util.concurrent.Flow-compliant implementation and therefore fully interoperable with other implementations.

kafka reactive-streams akka akka-streams reactive kafka-client kafka-connector

kafka-streams-course - Learn Kafka Streams with several examples!

•    Java

---

Learn Kafka Streams with several examples!

kafka kafka-streams learning udemy

strimzi-kafka-operator - Apache Kafka running on Kubernetes and OpenShift

•    Java

---

Strimzi provides a way to run an Apache Kafka cluster on Kubernetes or OpenShift in various deployment configurations. See our website for more details about the project. Documentation to the current master branch as well as all releases can be found on our website.

kafka kubernetes openshift docker messaging enmasse kafka-connect kafka-streams data-streaming data-stream data-streams kubernetes-operator kubernetes-controller

Debezium - Stream changes from your databases.

•    Java

---

Debezium is a distributed platform that turns your existing databases into event streams, so applications can see and respond immediately to each row-level change in the databases. Debezium is built on top of Apache Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, from where your application consumes them. This makes it possible for your application to easily consume all of the events correctly and completely.

change-data-capture kafka-connect apache-kafka debezium cdc database kafka kafka-producer database-migration events

kafka-storm-starter - Code examples that show to integrate Apache Kafka 0

•    Scala

---

Code examples that show how to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark 1.1+ while using Apache Avro as the data serialization format. Take a look at the Kafka Streams code examples at https://github.com/confluentinc/examples.

apache-kafka kafka apache-storm storm spark apache-spark integration avro apache-avro

stream-reactor - Streaming reference architecture for ETL with Kafka and Kafka-Connect

•    Scala

---

Lenses offers SQL (for data browsing and Kafka Streams), Kafka Connect connector management, cluster monitoring and more. A collection of components to build a real time ingestion pipeline.

kafka kafka-connect connector streaming cassandra hazelcast redis elasticsearch ftp influxdb coap mqtt kudu jms hbase mongodb rethinkdb documentdb cosmosdb kubernetes

faust - Python Stream Processing

•    Python

---

Faust is a stream processing library, porting the ideas from Kafka Streams to Python. It is used at Robinhood to build high performance distributed systems and real-time data pipelines that process billions of events every day.

kafka-streams kafka asyncio distributed-systems stream-processing

examples - Demo applications and code examples for Confluent Platform and Apache Kafka

•    Java

---

This repository contains code examples for Apache Kafka and the Confluent Platform.

confluent kafka kafka-streams kafka-clients example examples demo stream-processing ccloud

scalatest-embedded-kafka - A library that provides an in-memory Kafka instance to run your tests against

•    Scala

---

A library that provides an in-memory Kafka instance to run your tests against. As of version 1.1.1 the library is not dependent anymore on ScalaTest and won't transitively bring ScalaTest into your build.

kafka kafka-streams scalatest

kafka-streams-examples - Demo applications and code examples for Apache Kafka's Streams API.

•    Java

---

Kafka Streams examples

liftbridge - Lightweight, fault-tolerant message streams.

•    Go

---

Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. Use Liftbridge as a simpler and lighter alternative to systems like Kafka and Pulsar or use it to add streaming semantics to an existing NATS deployment. See the introduction post on Liftbridge and this post for more context and some of the inspiration behind it.

nats streaming messaging cloud-native grpc distributed-systems liftbridge message-queue pubsub

karafka - Framework used to simplify Apache Kafka based Ruby and Rails applications development.

•    Ruby

---

Framework used to simplify Apache Kafka based Ruby applications development.It allows programmers to use approach similar to standard HTTP conventions (params and params_batch) when working with asynchronous Kafka messages.

karafka-framework kafka-topic kafka kafka-client kafka-clients kafka-producer kafka-consumer apache-kafka kafka-message karafka-application sidekiq rails kafka-ruby ruby-on-rails rubygems rubygem ruby-library kafka-library

confluent-kafka-dotnet - Confluent's Apache Kafka .NET client

•    CSharp

---

confluent-kafka-dotnet is Confluent's .NET client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-dotnet is a lightweight wrapper around librdkafka, a finely tuned C client.

kafka nuget c-sharp confluent kafka-client kafka-clients kafka-library

confluent-kafka-python - Confluent's Apache Kafka Python client

•    Python

---

confluent-kafka-python is Confluent's Python client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-python is a lightweight wrapper around librdkafka, a finely tuned C client.

confluent kafka-client librdkafka python-client kafka-protocol kafka-library kafka

kafka-rest - Confluent REST Proxy for Kafka

•    Java

---

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster, making it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients.

confluent kafka-rest-proxy kafka schema-registry rest-api kafka-client kafka-library

ruby-kafka - A Ruby client library for Apache Kafka

•    Ruby

---

A Ruby client library for Apache Kafka, a distributed log and message bus. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.Although parts of this library work with Kafka 0.8 – specifically, the Producer API – it's being tested and developed against Kafka 0.9. The Consumer API is Kafka 0.9+ only.

kafka-client kafka ruby-gem kafka-library

kq - Kafka-based Job Queue for Python

•    Python

---

KQ (Kafka Queue) is a lightweight Python library which lets you queue and execute jobs asynchronously using Apache Kafka. It uses kafka-python under the hood. You may need to use sudo depending on your environment.

kafka job-queue async kafka-client kafka-consumer kafka-producer python3 python2 python-library python-3 python-2 asynchronous queueing jobqueue apache-kafka worker-queue producer-consumer serialization python-3-5

watermill - Building event-driven applications easy way in Go.

•    Go

---

Watermill is a Go library for working efficiently with message streams. It is intended for building event driven applications, enabling event sourcing, RPC over messages, sagas and basically whatever else comes to your mind. You can use conventional pub/sub implementations like Kafka or RabbitMQ, but also HTTP or MySQL binlog if that fits your use case. Note: Watermill should run reliably in a production environment, but it is still under heavy development and the public API may change before the 1.0.0 release.

watermill event-driven event-sourcing sagas events stream-processing reactive rabbitmq kafka cqrs

seldon-server - Machine Learning Platform and Recommendation Engine built on Kubernetes

•    Java

---

This project is not actively maintained anymore please see Seldon Core. Seldon Server is a machine learning platform that helps your data science team deploy models into production.

machine-learning deep-learning deployment kubernetes docker microservices spark kafka kafka-streams tensorflow cloud aws gcp azure seldon recommender-system recommendation-engine prediction