Kafka-Message-Server - Example application based on Apache Kafka framework to show it usage as distributed message server

•    Java

---

Apache kafka is yet another precious gem from Apache Software Foundation. Kafka was originally developed at Linkedin and later on became a member of Apache project. Apache Kafka is a distributed publish-subscribe messaging system. Kafka differs from traditional messaging system as it is designed as distributed system, persists messages on disk and supports multiple subscribers. Kafka-Message-Server is an sample application for demonstrating kafka usage as message-server. Please follow the below instructions for productive use of the sample application.

Debezium - Stream changes from your databases.

•    Java

---

Debezium is a distributed platform that turns your existing databases into event streams, so applications can see and respond immediately to each row-level change in the databases. Debezium is built on top of Apache Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, from where your application consumes them. This makes it possible for your application to easily consume all of the events correctly and completely.

change-data-capture kafka-connect apache-kafka debezium cdc database kafka kafka-producer database-migration events

Nakadi - A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

•    Java

---

Nakadi is a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues. It provides abstract event delivery via a secured RESTful API, Enable convenient development of event-driven applications and asynchronous microservices, Efficient low latency event delivery.

apis event-bus restful microservices postgresql kafka messaging message-queue

spring-integration-kafka

•    Java

---

The Spring Integration Kafka extension project provides inbound and outbound channel adapters for Apache Kafka. Apache Kafka is a distributed publish-subscribe messaging system that is designed for high throughput (terabytes of data) and low latency (milliseconds). For more information on Kafka and its design goals, see the Kafka main page.Starting from version 2.0 version this project is a complete rewrite based on the new spring-kafka project which uses the pure java Producer and Consumer clients provided by Kafka 0.9.x.x and 0.10.x.x..

strimzi-kafka-operator - Apache Kafka running on Kubernetes and OpenShift

•    Java

---

Strimzi provides a way to run an Apache Kafka cluster on Kubernetes or OpenShift in various deployment configurations. See our website for more details about the project. Documentation to the current master branch as well as all releases can be found on our website.

kafka kubernetes openshift docker messaging enmasse kafka-connect kafka-streams data-streaming data-stream data-streams kubernetes-operator kubernetes-controller

SiteWhere - The Open Platform for Internet of Things (IoT)

•    Java

---

SiteWhere is an open source platform for capturing, storing, integrating, and analyzing data from IoT devices. SiteWhere is a multi-tenant, application enablement platform for the Internet of Things (IoT) providing device management, complex event processing (CEP) and integration through a modern, scalable architecture. SiteWhere provides REST APIs for all system functionality.

internet-of-things device-management iot iot-platform mqtt iot-framework arduino raspberry-pi

seldon-server - Machine Learning Platform and Recommendation Engine built on Kubernetes

•    Java

---

This project is not actively maintained anymore please see Seldon Core. Seldon Server is a machine learning platform that helps your data science team deploy models into production.

machine-learning deep-learning deployment kubernetes docker microservices spark kafka kafka-streams tensorflow cloud aws gcp azure seldon recommender-system recommendation-engine prediction

chaperone - A Kafka audit system

•    Java

---

As Kafka audit system, Chaperone monitors the completeness and latency of data stream. The audit metrics are persisted in database for Kafka users to quantify the loss of their topics if any.Basically, Chaperone cuts timeline into 10min buckets and assigns message to corresponding bucket according to its event time. The stats of the bucket are updated accordingly, like the total message count. Periodically, the stats are sent out to a dedicated Kafka topic, say 'chaperone-audit'. ChaperoneCollector consumes those stats from this topic and persists them into database.

Azure-Functions

•    

---

Azure Functions is an event driven, compute-on-demand experience that extends the existing Azure application platform with capabilities to implement code triggered by events occurring in virtually any Azure or 3rd party service as well as on-premises systems. Azure Functions allows developers to take action by connecting to data sources or messaging solutions, thus making it easy to process and react to events. Azure Functions scale based on demand and you pay only for the resources you consume.This repository acts as a directory for folks looking for the various resources we have for Azure Functions.

azure-functions

Jafka - A fast and simple distributed publish-subscribe messaging system (mq)

•    Java

---

Jafka mq is a distributed publish-subscribe messaging system cloned from Apache Kafka.

publish-subscribe message-queue pub-sub distributed kafka-alternative kafka-clone

karafka - Framework used to simplify Apache Kafka based Ruby and Rails applications development.

•    Ruby

---

Framework used to simplify Apache Kafka based Ruby applications development.It allows programmers to use approach similar to standard HTTP conventions (params and params_batch) when working with asynchronous Kafka messages.

karafka-framework kafka-topic kafka kafka-client kafka-clients kafka-producer kafka-consumer apache-kafka kafka-message karafka-application sidekiq rails kafka-ruby ruby-on-rails rubygems rubygem ruby-library kafka-library

librdkafka - The Apache Kafka C/C++ library

•    C

---

Copyright (c) 2012-2016, Magnus Edenhill.librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

kafka kafka-consumer apache-kafka high-performance librdkafka kafka-producer c-plus-plus consumer kafka-client kafka-library

ruby-kafka - A Ruby client library for Apache Kafka

•    Ruby

---

A Ruby client library for Apache Kafka, a distributed log and message bus. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.Although parts of this library work with Kafka 0.8 – specifically, the Producer API – it's being tested and developed against Kafka 0.9. The Consumer API is Kafka 0.9+ only.

kafka-client kafka ruby-gem kafka-library

winton-kafka-streams - A Python implementation of Apache Kafka Streams

•    Python

---

Implementation of Apache Kafka's Streams API in Python. Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. Kafka has Streams API added for building stream processing applications using Apache Kafka. Applications built with Kafka's Streams API do not require any setup beyond the provision of a Kafka cluster.

kafka-streams streaming-api librdkafka

confluent-kafka-dotnet - Confluent's Apache Kafka .NET client

•    CSharp

---

confluent-kafka-dotnet is Confluent's .NET client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-dotnet is a lightweight wrapper around librdkafka, a finely tuned C client.

kafka nuget c-sharp confluent kafka-client kafka-clients kafka-library

confluent-kafka-python - Confluent's Apache Kafka Python client

•    Python

---

confluent-kafka-python is Confluent's Python client for Apache Kafka and the Confluent Platform.High performance - confluent-kafka-python is a lightweight wrapper around librdkafka, a finely tuned C client.

confluent kafka-client librdkafka python-client kafka-protocol kafka-library kafka

azure-sdk-for-php - Microsoft Azure SDK for PHP

•    PHP

---

This project provides a set of PHP client libraries that make it easy to access Microsoft Azure tables, blobs, queues, service bus (queues and topics), service runtime and service management APIs. For documentation on how to host PHP applications on Microsoft Azure, please see the Microsoft Azure PHP Developer Center.The recommended way to resolve dependencies is to install them using the Composer package manager.

azure microsoft-azure-sdk sdk

pykafka - Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance

•    Python

---

PyKafka is a cluster-aware Kafka>=0.8.2 client for Python. It includes Python implementations of Kafka producers and consumers, which are optionally backed by a C extension built on librdkafka, and runs under Python 2.7+, Python 3.4+, and PyPy.PyKafka's primary goal is to provide a similar level of abstraction to the JVM Kafka client using idioms familiar to Python programmers and exposing the most Pythonic API possible.

kafka c-extension apache-kafka kafka-client kafka-library

azure-xplat-cli - Microsoft Azure Cross Platform Command Line

•    Javascript

---

This project provides a cross-platform command line interface for developers and IT administrators to develop, deploy and manage Microsoft Azure applications.Note: The list of features may not be up-to-date. For accurate command details, type azure | azure -h | azure --help to navigate through the help system. Also, use azure config mode asm|arm to switch between service management (Version V1)and resource management (Version V2) of the Azure REST API.

azure-cli azure-api azure-xplat-cli nodejs docker node azure cli cloud-hosting deployment

kafka-python - Python client for Apache Kafka

•    Python

---

Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the official java client, with a sprinkling of pythonic interfaces (e.g., consumer iterators).kafka-python is best used with newer brokers (0.9+), but is backwards-compatible with older versions (to 0.8.0). Some features will only be enabled on newer brokers. For example, fully coordinated consumer groups -- i.e., dynamic partition assignment to multiple consumers in the same group -- requires use of 0.9+ kafka brokers. Supporting this feature for earlier broker releases would require writing and maintaining custom leadership election and membership / health check code (perhaps using zookeeper or consul). For older brokers, you can achieve something similar by manually assigning different partitions to each consumer instance with config management tools like chef, ansible, etc. This approach will work fine, though it does not support rebalancing on failures. See <https://kafka-python.readthedocs.io/en/master/compatibility.html> for more details.