SeaweedFS - Simple and highly scalable distributed file system

•    Go

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, SeaweedFS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

huststore - High-performance Distributed Storage

•    C

huststore is a open source high performance distributed database system. It not only provides key-value storage service with extremely high performance, up to 100 thousand QPS, but also supports data structures like hash, set, sorted set, etc. Also, it can store binary data as value from a key-value pair, and thus can be used as an alternative of Redis.In addtion, huststore implements a distributed message queue by integrating a special HA module, features including message Push Stream, and message Publish-SubScribe, these features can be used as replacements of the corresponding features in rabbitmq and gearman.

braft - An industrial-grade C++ implementation of RAFT consensus algorithm based on brpc, widely used inside Baidu to build highly-available distributed systems

•    C++

An industrial-grade C++ implementation of RAFT consensus algorithm and replicated state machine based on brpc. braft is designed and implemented for scenarios demanding for high workload and low overhead of latency, with the consideration for easy-to-understand concepts so that engineers inside Baidu can build their own distributed systems individually and correctly. Build brpc which is the main dependency of braft.

dragonboat - A feature complete and high performance multi-group Raft library in Go.

•    Go

Dragonboat is a high performance multi-group Raft consensus library in Go with C++11 binding support. Consensus algorithms such as Raft provides fault-tolerance by alllowing a system continue to operate as long as the majority member servers are available. For example, a Raft cluster of 5 servers can make progress even if 2 servers fail. It also appears to clients as a single node with strong data consistency always provided. All running servers can be used to initiate read requests for aggregated read throughput.

alluxio - Alluxio, formerly Tachyon, Unify Data at Memory Speed

•    Java

Parent POM of Alluxio project: a Memory-Speed Virtual Distributed Storage System

Sheepdog - Distributed Storage System for QEMU

•    C

Sheepdog is a distributed object storage system for volume and container services and manages the disks and nodes intelligently. Sheepdog features ease of use, simplicity of code and can scale out to thousands of nodes. The block level volume abstraction can be attached to QEMU virtual machines and Linux SCSI Target and supports advanced volume management features such as snapshot, cloning, and thin provisioning.

Ceph - Distributed Object Store

•    C++

Ceph provides seamless access to objects using native language bindings or radosgw, a REST interface that’s compatible with applications written for S3 and Swift. Ceph’s RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster. Ceph provides a POSIX-compliant network file system that aims for high performance, large data storage, and maximum compatibility with legacy applications.

Xtreemfs - Fault-tolerant distributed file system

•    C

XtreemFS is a distributed, replicated and fault-tolerant file system for federated IT infrastructures. XtreemFS replicates your file data across multiple storage servers, which can be distributed worldwide. It scales according to your needs within minutes, simply by adding new standard hardware. Starting with a single server, growing to a cluster and across datacenters.

OrangeFS - Scale-out Network File System

•    C

OrangeFS is a scale-out network file system designed for use on high-end computing (HEC) systems that provides very high-performance access to multi-server-based disk storage, in parallel. The OrangeFS server and client are user-level code, making them very easy to install and manage. OrangeFS has optimized MPI-IO support for parallel and distributed applications, and it is leveraged in production installations and used as a research platform for distributed and parallel storage.

storj - Ongoing Storj v3 development

•    Go

All of our code for Storj v3 is open source. Have a code change you think would make Storj better? Please send a pull request along! Make sure to sign our Contributor License Agreement (CLA) first. See our license section for more details. Have comments, bug reports, or suggestions? Want to propose a PR before hand-crafting it? Jump on to our Rocketchat and join the #dev channel to say hi to the developer community and to talk to the Storj core team.

moosefs - MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System

•    C

MooseFS is a Petabyte Open Source Network Distributed File System. It is easy to deploy and maintain, fault tolerant, highly performing, easily scalable, POSIX compliant. MooseFS Linux Client uses FUSE. MooseFS macOS Client uses FUSE for macOS.

LizardFS - Software Defined Storage, Distributed, Parallel, Scalable, Fault-tolerant, Geo-Redundant and Highly available file system

•    C++

LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.

BeeGFS - Parallel Cluster File System

•    C

BeeGFS (formerly FhGFS) is the leading parallel cluster file system, developed with a strong focus on performance and designed for very easy installation and management. It transparently spreads user data across multiple servers. By increasing the number of servers and disks in the system, you can simply scale performance and capacity of the file system to the level that you need, seamlessly from small clusters up to enterprise-class systems with thousands of nodes.

Pravega - Streaming as a new software defined storage primitive

•    Java

Pravega is an open source distributed storage service implementing Streams. It offers Stream as the main primitive for the foundation of reliable storage systems: a high-performance, durable, elastic, and unlimited append-only byte stream with strict ordering and consistency.

storoom - a simple distributed storage

•    DotNet

storoom is a simple distributed storage system, it uses key-value pair to store data. contains client/clientlib stornode, node to store data to physical storage system storoom, controller of stornode<s> or storoom<s> develop in vb.net

distkv - Distributed KV Storage System based on Raft and RocksDB, can also be use to store small files, like images

•    Java

Distributed KV Storage System based on Raft and RocksDB, can also be use to store small files, like images.

raft-java - Raft implementation for Java

•    Java

Raft implementation for Java

awesome-distributed-systems - Awesome list of distributed systems resources

Awesome list of distributed systems resources

sfs - The distributed object storage server used by PitchPoint Solutions to securely store billions of large and small files using minimal resources

•    Java

The most recent release of sfs is release-1.20170106133707. Oracle Java 8 and Maven 3 should be used for building.

sharkstore

•    Go

SharkStore is a distributed, persistent key-value store, whose database layer is based on the rocksdb and the replication works are based on raft. See arch.md .