Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo

  •        176

A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC.

Its design objectives are:

  1. Fast: close to the speed of direct memory access, both produding and consuming are close to O(1) memory access.
  2. Persistent : all data in the system is persisted on disk and is crash resistant.
  3. High-throughput: even with modest hardware Luxun can support handreds of thousands of messages per second.
  4. Realtime : messages produced by producer threads should be immediately visible to consumer threads.
  5. Distributed : explicit support for partitioning messages over Luxun servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
  6. Multiple clients support : use Thrift RPC as communication infrastructure, easy to generate different clients for different platforms.
  7. Flexible consuming semantics : supports typical consume once queue, fan out queue, even supports consume by index when needed.

http://bulldog2011.github.io/luxun/
https://github.com/bulldog2011/luxun

Tags
Implementation
License
Platform

   




Related Projects

Faye - Simple pub/sub messaging for the web

  •    Javascript

Faye is a set of tools for simple publish-subscribe messaging between web clients. It ships with easy-to-use message routing servers for Node.js and Rack applications, and clients that can be used on the server and in the browser. It is based on the Bayeux protocol.

RocketMQ - Distributed messaging and streaming data platform

  •    Java

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

gnatsd - High-Performance server for NATS, the cloud native messaging system.

  •    Go

NATS Server is a simple, high performance open source messaging system for cloud native applications, IoT messaging, and microservices architectures. It implements a highly scalable and elegant publish-subscribe (pub/sub) distribution model. The performant nature of NATS make it an ideal base for building modern, reliable, scalable cloud native distributed systems.

MBassador - Light-weight, high-performance event bus implementing the publish subscribe pattern

  •    Java

MBassador is a light-weight, high-performance event bus implementing the publish subscribe pattern. It is designed for ease of use and aims to be feature rich and extensible while preserving resource efficiency and performance. The core of MBassador is built around a custom data structure that provides non-blocking reads and minimized lock contention for writes such that performance degradation of concurrent read/write access is minimal.


mist - A distributed, tag-based pub-sub service for modern web applications and container-driven cloud

  •    Go

Mist is a simple pub/sub based on the idea that messages are tagged. To subscribe, a client simply constructs a list of tags that it is interested in, and all messages that are tagged with all of those tags are sent to that client. A client can not only be a subscriber (with multiple active subscriptions), but also a publisher. Clients will receive messages for any tags they are subscribed, except message publish by themselves.

Pulsar - Distributed pub-sub Messaging System from Yahoo

  •    Java

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. It is horizontally scalable (Millions of independent topics and millions of messages published per second), Strong ordering and consistency guarantees, Low latency , REST API, Geo Replication and lot more.

golongpoll - golang long polling library

  •    Go

Golang long polling library. Makes web pub-sub easy via an HTTP long-poll server.To use, create a LongpollManager and then use it to publish events and expose an HTTP handler to subscribe to events.

Distributed Publish/Subscribe (Pub/Sub) Event System

  •    

The Web Solutions Platform (WSP) event system is a general-purpose distributed publish/subscribe event system designed for high performance and low latency. Publish/subscribe is a common communication pattern though there are many variations to solve different scenario require...

Apache Apollo - ActiveMQ's next generation of messaging

  •    Java

ActiveMQ Apollo is a faster, more reliable, easier to maintain messaging broker built from the foundations of the original ActiveMQ. It accomplishes this using a radically different threading and message dispatching architecture. Like ActiveMQ, Apollo is a multi-protocol broker and supports STOMP, AMQP, MQTT, Openwire, SSL, and WebSockets.

hemera - 🔬 Writing reliable & fault-tolerant microservices with https://nats.io

  •    Javascript

Hemera (/ˈhɛmərə/; Ancient Greek: Ἡμέρα [hɛːméra] "day") is a small wrapper around the NATS driver. NATS is a simple, fast and reliable solution for the internal communication of a distributed system. It chooses simplicity and reliability over guaranteed delivery. We want to provide a toolkit to develop micro services in an easy and powerful way. We provide a pattern matching RPC style. You don't have to worry about the transport. NATS is powerful.Hemera has not been designed for high performance on a single process. It has been designed to create lots of microservices doesn't matter where they live. It choose simplicity and reliability as primary goals. It act together with NATS as central nervous system of your distributed system. Transport independency was not considered to be a relevant factor. In addition we use pattern matching which is very powerful. The fact that Hemera needs a broker is an argument which should be taken into consideration when you compare hemera with other frameworks. The relevant difference between microservice frameworks like senecajs, molecurer is not the performance or modularity its about the complexity you need to manage. Hemera is expert in providing an interface to work with lots of services in the network, NATS is the expert to deliver the message at the right place. Hemera is still a subscriber of NATS with some magic in routing and extensions. We don't have to worry about all different aspects in a distributed system like routing, load-balancing, service-discovery, clustering, health-checks ...

NSQ - A realtime distributed messaging platform in Go

  •    Go

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. It scales horizontally, without any centralized brokers. Built-in discovery simplifies the addition of nodes to the cluster.

Apache Pulsar - Distributed pub-sub messaging platform

  •    Java

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. It has run in production at Yahoo scale for over 3 years, with millions of messages per second across millions of topics. It is Horizontally scalable, Low latency, High throughput, Multi-tenancy, Geo-replication, Transparent batching of messages, Transparent handling of partitioned topics, REST API for provisioning, admin and stats and lot more.

huststore - High-performance Distributed Storage

  •    C

huststore is a open source high performance distributed database system. It not only provides key-value storage service with extremely high performance, up to 100 thousand QPS, but also supports data structures like hash, set, sorted set, etc. Also, it can store binary data as value from a key-value pair, and thus can be used as an alternative of Redis.In addtion, huststore implements a distributed message queue by integrating a special HA module, features including message Push Stream, and message Publish-SubScribe, these features can be used as replacements of the corresponding features in rabbitmq and gearman.

PubSubJS - Dependency free publish/subscribe for JavaScript

  •    Javascript

PubSubJS is a topic-based publish/subscribe library written in JavaScript. PubSubJS has synchronisation decoupling, so topics are published asynchronously. This helps keep your program predictable as the originator of topics will not be blocked while consumers process them.

Redisson - Redis based In-Memory Data Grid for Java

  •    Java

Redisson - distributed Java objects and services (Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Executor service, Tomcat Session Manager, Scheduler service, JCache API) on top of Redis server. Rich Redis client.

private_pub - Handle pub/sub messaging through private channels in Rails using Faye.

  •    Ruby

Private Pub is a Ruby gem for use with Rails to publish and subscribe to messages through Faye. It allows you to easily provide real-time updates through an open socket without tying up a Rails process. All channels are private so users can only listen to events you subscribe them to. Watch RailsCasts Episode 316 for a demonstration of Private Pub.

emitter - High performance, distributed and low latency publish-subscribe platform.

  •    Go

Emitter is a free open source real-time messaging service that connects all devices. This publish-subscribe messaging API is built for speed and security. Emitter is a real-time communication service for connecting online devices. Infrastructure and APIs for IoT, gaming, apps and real-time web. At its core, emitter.io is a distributed, scalable and fault-tolerant publish-subscribe messaging platform based on MQTT protocol and featuring message storage.

liftbridge - Lightweight, fault-tolerant message streams.

  •    Go

Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. Use Liftbridge as a simpler and lighter alternative to systems like Kafka and Pulsar or use it to add streaming semantics to an existing NATS deployment. See the introduction post on Liftbridge and this post for more context and some of the inspiration behind it.