paxosstore - PaxosStore has been deployed in WeChat production for more than two years, providing storage services for the core businesses of WeChat backend

  •        47

PaxosStore is a distributed-database initially inspired by Google MegaStore. It's the second generation of storage system developed to support current WeChat sevice and applications. PaxosStore has been deployed in WeChat production for more than two years, providing storage services for the core businesses of WeChat backend including user account management, user relationship management (i.e., contacts), instant messaging, social networking (i.e., Moments), and online payment (i.e., WeChat Pay). Now PaxosStore is running on thousands of machines, and is able to afford billions of peak TPS.



Related Projects

rqlite - The lightweight, distributed relational database built on SQLite.

  •    Go

rqlite is a distributed relational database, which uses SQLite as its storage engine. rqlite uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all. It also gracefully handles leader elections, and tolerates failures of machines, including the leader. rqlite is available for Linux, OSX, and Microsoft Windows.rqlite gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation. With it you've got a lightweight and reliable distributed relational data store. Think etcd or Consul, but with relational data modelling also available.

ActorDB - Distributed SQL database with linear scalability

  •    Erlang

ActorDB is ideal as a server side database for apps. Think of running a large mail service, dropbox, evernote, etc. They all require server side storage for user data, but the vast majority of queries is within a specific user. With many users, the server side database can get very large. Using ActorDB you can keep a full relational database for every user and not be forced into painful scaling strategies that require you to throw away everything that makes relational databases good.

etcd - Distributed reliable key-value store for the most critical data of a distributed system

  •    Go

Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order to get stable binaries. etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.

awesome-consensus - Awesome list for Paxos and friends


A curated selection of artisanal consensus algorithms and hand-crafted distributed lock services.

Copycat - A novel implementation of the Raft consensus algorithm

  •    Java

Copycat is a fault-tolerant state machine replication framework. Built on the Raft consensus algorithm, it handles replication and persistence and enforces strict ordering of inputs and outputs, allowing developers to focus on single-threaded application logic. Its event-driven model allows for efficient client communication with replicated state machines, from simple key-value stores to wait-free locks and leader elections. You supply the state machine and Copycat takes care of the rest, making it easy to build robust, safe distributed systems.

dragonboat - A feature complete and high performance multi-group Raft library in Go.

  •    Go

Dragonboat is a high performance multi-group Raft consensus library in Go with C++11 binding support. Consensus algorithms such as Raft provides fault-tolerance by alllowing a system continue to operate as long as the majority member servers are available. For example, a Raft cluster of 5 servers can make progress even if 2 servers fail. It also appears to clients as a single node with strong data consistency always provided. All running servers can be used to initiate read requests for aggregated read throughput.

Rippled - Decentralized cryptocurrency blockchain daemon implementing the XRP Ledger in C++

  •    C++

Ripple is a network of computers which use the Ripple consensus algorithm to atomically settle and record transactions on a secure distributed database, the Ripple Consensus Ledger (RCL). Because of its distributed nature, the RCL offers transaction immutability without a central operator. The RCL contains a built-in currency exchange and its path-finding algorithm finds competitive exchange rates across order books and currency pairs.

awesome-distributed-systems - A curated list to learn about distributed systems


A (hopefully) curated list on awesome material on distributed systems, inspired by other awesome frameworks like awesome-python. Most links will tend to be readings on architecture itself rather than code itself. Read things here before you start.


  •    Go

EPaxos is an efficient, leaderless replication protocol. The name stands for Egalitarian Paxos -- EPaxos is based on the Paxos consensus algorithm. As such, it can tolerate up to F concurrent replica failures with 2F+1 total replicas. To function effectively as a replication protocol, Paxos has to rely on a stable leader replica (this optimization is known as Multi-Paxos). The leader can become a bottleneck for performance: it has to handle more messages than the other replicas, and remote clients have to contact the leader, thus experiencing higher latency. Other Paxos variants either also rely on a stable leader, or have a pre-established scheme that allows different replicas to take turns in proposing commands (such as Mencius). This latter scheme suffers from tight coupling of the performance of the system from that of every replica -- i.e., the system runs at the speed of the slowest replica.

BigchainDB - The Scalable Blockchain Database

  •    Python

BigchainDB allows developers and enterprise to deploy blockchain proof-of-concepts, platforms and applications with a scalable blockchain database, supporting a wide range of industries and use cases. It is a decentralization ecosystem: a decentralized database, at scale. It can perform 1 million writes per second throughput, store petabytes of data, and sub-second latency.

YugaByte Database - Transactional, high-performance database for building internet-scale, globally-distributed applications

  •    C++

A cloud-native database for building mission-critical applications. This repository contains the Community Edition of the YugaByte Database.YugaByte offers both SQL and NoSQL in a single, unified db. It is meant to be a system-of-record/authoritative database that applications can rely on for correctness and availability. It allows applications to easily scale up and scale down in the cloud, on-premises or across hybrid environments without creating operational complexity or increasing the risk of outages.

SummitDB - In-memory NoSQL database with ACID transactions, Raft consensus, and Redis API

  •    Go

SummitDB is an in-memory, NoSQL key/value database. It persists to disk, uses the Raft consensus algorithm, is ACID compliant, and built on a transactional and strongly-consistent model. It supports custom indexes, geospatial data, JSON documents, and user-defined JS scripting.Under the hood it utilizes Finn, Redcon, BuntDB, GJSON, and Otto.

tikv - Distributed transactional key value database powered by Rust and Raft

  •    Rust

Geo-Replication TiKV uses Raft and Placement Driver to support Geo-Replication.Horizontal scalability With Placement Driver and carefully designed Raft groups, TiKV excels in horizontal scalability and can easily scale to 100+ TBs of data.

catena - Catena is a distributed database based on a blockchain, accessible using SQL.

  •    Swift

Catena is a distributed database based on a blockchain, accessible using SQL. Catena timestamps database transactions (SQL) in a decentralized way between nodes that do not or cannot trust each other, while enforcing modification permissions ('grants') that were agreed upon earlier. A Catena blockchain contains SQL transactions that, when executed in order, lead to the agreed-upon state of the database. The transactions are automatically replicated to, validated by, and replayed on participating clients. A Catena database can be connected to by client applications using the PostgreSQL wire protocol (pq).

Atomix - Scalable, fault-tolerant distributed systems protocols and primitives for the JVM

  •    Java

Atomix is an event-driven framework for coordinating fault-tolerant distributed systems built on the Raft consensus algorithm. It provides the building blocks that solve many common distributed systems problems including group membership, leader election, distributed concurrency control, partitioning, and replication.

Cassandra - Scalable Distributed Database

  •    Java

The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. Cassandra is suitable for applications that can't afford to lose data. Data is automatically replicated to multiple nodes for fault-tolerance.

tidis - Distributed transactional NoSQL database, Redis protocol compatible using tikv as backend

  •    Go

Tidis is a Distributed NoSQL database, providing a Redis protocol API (string, list, hash, set, sorted set), written in Go. Tidis is like TiDB layer, providing protocol transform and data structure compute, powered by TiKV backend distributed storage which use Raft for data replication and 2PC for distributed transaction.

Bagri - XML/Document DB on top of distributed cache

  •    Java

Bagri is a Document Database built on top of distributed cache solution like Hazelcast or Coherence. The system allows to process semi-structured schema-less documents and perform distributed queries on them in real-time. It scales horizontally very well with use of data sharding, when all documents are distributed evenly between distributed cache partitions.

raft-rs - Raft distributed consensus algorithm implemented in Rust.

  •    Rust

When building a distributed system one principal goal is often to build in fault-tolerance. That is, if one particular node in a network goes down, or if there is a network partition, the entire cluster does not fall over. The cluster of nodes taking part in a distributed consensus protocol must come to agreement regarding values, and once that decision is reached, that choice is final. Distributed Consensus Algorithms often take the form of a replicated state machine and log. Each state machine accepts inputs from its log, and represents the value(s) to be replicated, for example, a hash table. They allow a collection of machines to work as a coherent group that can survive the failures of some of its members.

Dgraph - Fast, Transactional, Distributed Graph Database

  •    Go

Dgraph is a horizontally scalable and distributed graph database, providing ACID transactions, consistent replication and linearizable reads. It's built from ground up to perform for a rich set of queries. Being a native graph database, it tightly controls how the data is arranged on disk to optimize for query performance and throughput, reducing disk seeks and network calls in a cluster.

We have large collection of open source products. Follow the tags from Tag Cloud >>

Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.