Redpanda - A Kafka API compatible streaming platform for mission-critical workloads

We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.

Redpanda - A Kafka API compatible streaming platform for mission-critical workloads

Redpanda is a streaming platform for mission critical workloads. It is Kafka API compatible so any client that works with Kafka should work out of the box with Redpanda. It does not have dependency on ZooKeeper, JVM. Redpanda is a new storage engine, optimized for streaming data, using a thread-per-core architecture focused on delivering stable tail-latencies. It transforms data inline with embedded WASM engine.

Its features include:

Kafka API compatible
Up to 4x faster Kafka transactions
Automatic data and load rebalancing
Automatic hardware optimization
Unlimited partitions for extreme flexibility
Built-in HTTP support
Support for favorite Kafka tooling
Built-in Prometheus support
Native K8s operator
Automatic leadership rebalancing
TLS encryption in flight
Unlimited horizontal scaling

https://github.com/vectorizedio/redpanda
https://vectorized.io/redpanda

Tags	message-queue streaming kafka cpp containers realtime kubernetes queue kafka-protocol
Implementation	C++
License	BSL
Platform	Windows Linux MacOS

Related Projects

kq - Kafka-based Job Queue for Python

Python

KQ (Kafka Queue) is a lightweight Python library which lets you queue and execute jobs asynchronously using Apache Kafka. It uses kafka-python under the hood. You may need to use sudo depending on your environment.

kafka job-queue async kafka-client kafka-consumer kafka-producer python3 python2 python-library python-3 python-2 asynchronous queueing jobqueue apache-kafka worker-queue producer-consumer serialization python-3-5

liftbridge - Lightweight, fault-tolerant message streams.

Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. Use Liftbridge as a simpler and lighter alternative to systems like Kafka and Pulsar or use it to add streaming semantics to an existing NATS deployment. See the introduction post on Liftbridge and this post for more context and some of the inspiration behind it.

nats streaming messaging cloud-native grpc distributed-systems liftbridge message-queue pubsub

NSQ - A realtime distributed messaging platform in Go

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. It scales horizontally, without any centralized brokers. Built-in discovery simplifies the addition of nodes to the cluster.

message-queue messaging distributed publish-subscribe pubsub queue

KafkaOffsetMonitor - A little app to monitor the progress of kafka consumers and their lag wrt the queue

Scala

This is an app to monitor your kafka consumers and their position (offset) in the queue. You can see the current consumer groups, for each group the topics that they are consuming and the position of the group in each topic queue. This is useful to understand how quick you are consuming from a queue and how fast the queue is growing. It allows for debuging kafka producers and consumers or just to have an idea of what is going on in your system.

hermes - Fast and reliable message broker built on top of Kafka.

Java

Hermes is an asynchronous message broker built on top of Kafka. We provide reliable, fault tolerant REST interface for message publishing and adaptive push mechanisms for message sending.Questions? We are on gitter.

hermes kafka pubsub messaging message-queue

Plumber - A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems

Plumber is a CLI devtool for inspecting, piping, massaging and redirecting data in message systems like Kafka, RabbitMQ , GCP PubSub and many more. It helps to safely view the contents of your data streams, Decode protobuf/avro/thrift/JSON data in real-time, Route data from one place to another, Replay events into a message system on your local network and lot more.

kafka protobuf rabbitmq message-bus message-queue event-driven event-bus cli command-line

strimzi-kafka-operator - Apache Kafka running on Kubernetes and OpenShift

Java

Strimzi provides a way to run an Apache Kafka cluster on Kubernetes or OpenShift in various deployment configurations. See our website for more details about the project. Documentation to the current master branch as well as all releases can be found on our website.

kafka kubernetes openshift docker messaging enmasse kafka-connect kafka-streams data-streaming data-stream data-streams kubernetes-operator kubernetes-controller

enqueue-dev - PHP7.1+. Message queue packages for PHP, Symfony, Laravel, Yii, and Magento

Enqueue is production ready, battle-tested messaging solution for PHP. Provides a common way for programs to create, send, read messages. This is a main development repository. It provides a friendly environment for productive development and testing of all Enqueue related features&packages.

message-queue message-bus rabbitmq amqp stomp job-queue amazon-sqs redis doctrine-dbal rabbitmq-client message-passing beanstalkd gearman kafka symfony google-pubsub

Jafka - A fast and simple distributed publish-subscribe messaging system (mq)

Java

Jafka mq is a distributed publish-subscribe messaging system cloned from Apache Kafka.

publish-subscribe message-queue pub-sub distributed kafka-alternative kafka-clone

jocko - Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

Kafka/distributed commit log service in Go. docker build -t travisjeffery/jocko:latest .

kafka distributed-systems queue messaging streaming

stream-reactor - Streaming reference architecture for ETL with Kafka and Kafka-Connect

Scala

Lenses offers SQL (for data browsing and Kafka Streams), Kafka Connect connector management, cluster monitoring and more. A collection of components to build a real time ingestion pipeline.

kafka kafka-connect connector streaming cassandra hazelcast redis elasticsearch ftp influxdb coap mqtt kudu jms hbase mongodb rethinkdb documentdb cosmosdb kubernetes

MeiliES - A Rust based event store using the Redis protocol

Rust

MeiliES is an Event Sourcing database that uses the RESP (REdis Serialization Protocol) to communicate. This way it is possible to create clients by reusing the already available protocol. For example, it is possible to use the official Redis command line interface program to communicate with MeiliES. There is a release blog post if you want to know more. An event store is like a Kafka or a Rabbit MQ but it stores events on disk indefinitely. The first purpose of the server is to publish events of a stream to all subscribed clients, note that events are saved in reception order. A client can also specify from which event number (incrementing) it wants to read, therefore it is possible to recover from crashing by reading and reconstructing a state with only new events. Keep in mind that a message queue is not made for event-sourcing.

event-store redis-protocol

Nakadi - A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

Java

Nakadi is a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues. It provides abstract event delivery via a secured RESTful API, Enable convenient development of event-driven applications and asynchronous microservices, Efficient low latency event delivery.

apis event-bus restful microservices postgresql kafka messaging message-queue

RocketMQ - Distributed messaging and streaming data platform

Java

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

message-queue messaging queue pub-sub distributed

Kafdrop – Kafka Web UI

Java

Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages.

docker kubernetes kafka topic web-ui event-sourcing zookeeper pub-sub kafka-utils consumer-producer consumer-group event-streaming kafka-tools kafka-ui kafka-management

Kafka UI - Open-Source Web UI for Apache Kafka Management

TypeScript

Kafka UI is a UI for Apache Kafka to monitor and manage Apache Kafka clusters. It is a simple tool that makes your data flows observable, helps find and troubleshoot issues faster and deliver optimal performance. Its lightweight dashboard makes it easy to track key metrics of your Kafka clusters - Brokers, Topics, Partitions, Production, and Consumption.

kafka big-data web-ui streams kafka-connect apache-kafka-ui kafka-producer kafka-client kafka-streams hacktoberfest streaming-data kafka-manager kafka-cluster event-streaming cluster-management kafka-brokers

librdkafka - The Apache Kafka C/C++ library

Copyright (c) 2012-2016, Magnus Edenhill.librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

kafka kafka-consumer apache-kafka high-performance librdkafka kafka-producer c-plus-plus consumer kafka-client kafka-library

kafka-net - Native C# client for Kafka queue servers.

CSharp

kafka-client kafka-library kafka

Kafka-Message-Server - Example application based on Apache Kafka framework to show it usage as distributed message server

Java

Apache kafka is yet another precious gem from Apache Software Foundation. Kafka was originally developed at Linkedin and later on became a member of Apache project. Apache Kafka is a distributed publish-subscribe messaging system. Kafka differs from traditional messaging system as it is designed as distributed system, persists messages on disk and supports multiple subscribers. Kafka-Message-Server is an sample application for demonstrating kafka usage as message-server. Please follow the below instructions for productive use of the sample application.

koperator - Oh no! Yet another Apache Kafka operator for Kubernetes

Koperator (formerly called Banzai Cloud Kafka Operator) is an operator for managing Apache Kafka on Kubernetes that automates the provisioning, management, and autoscaling of Apache Kafka clusters deployed to K8s. We took a different approach to what's out there - we believe for a good reason - please read on to understand more about our design motivations and some of the scenarios which were driving us to create Koperator.

kubernetes kafka cruisecontrol kafka-operator