orleans - Orleans - Distributed Virtual Actor Model

•    CSharp

---

Orleans is a framework that provides a straight-forward approach to building distributed high-scale computing applications, without the need to learn and apply complex concurrency or other scaling patterns.It was created by Microsoft Research implementing the Virtual Actor Model and designed for use in the cloud.

actor-model distributed-actors distributed-systems cloud-computing scalability reactive concurrency orleans

LizardFS - Software Defined Storage, Distributed, Parallel, Scalable, Fault-tolerant, Geo-Redundant and Highly available file system

•    C++

---

LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.

distributed-filesystem filesystem nas distributed-systems distributed-computing distributed-storage fault-tolerance snapshot qos erasure-coding replication replicas geo-replication hsm hierarchical-storage hadoop

awesome-distributed-systems - Awesome list of distributed systems resources

•    

---

Awesome list of distributed systems resources

awesome awesome-list distributed-systems distributed distributed-database distributed-computing distributed-storage computer-science computer-engineering

OrangeFS - Scale-out Network File System

•    C

---

OrangeFS is a scale-out network file system designed for use on high-end computing (HEC) systems that provides very high-performance access to multi-server-based disk storage, in parallel. The OrangeFS server and client are user-level code, making them very easy to install and manage. OrangeFS has optimized MPI-IO support for parallel and distributed applications, and it is leveraged in production installations and used as a research platform for distributed and parallel storage.

filesystem distributed-filesystem storage distributed-storage parallel-virtual-filesystem nas

dist-prog-book

•    CSS

---

Source repo for the book that I and my students in my course at Northeastern University, CS7680 Special Topics in Computing Systems: Programming Models for Distributed Computing, are writing on the topic of programming models for distributed systems. This is a book about the programming constructs we use to build distributed systems. These range from the small, RPC, futures, actors, to the large; systems built up of these components like MapReduce and Spark. We explore issues and concerns central to distributed systems like consistency, availability, and fault tolerance, from the lens of the programming models and frameworks that the programmer uses to build these systems.

moosefs - MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System

•    C

---

MooseFS is a Petabyte Open Source Network Distributed File System. It is easy to deploy and maintain, fault tolerant, highly performing, easily scalable, POSIX compliant. MooseFS Linux Client uses FUSE. MooseFS macOS Client uses FUSE for macOS.

dfs software-defined-storage posix filesystem file-system distributed-file-system clustering distributed-storage distributed-computing fuse big-data snapshot storage-tiering high-availability scalability storage moosefs hadoop posix-compliant

paracel - Distributed training framework with parameter server

•    C++

---

Paracel is a distributed computational framework, designed for many machine learning problems: Logistic Regression, SVD, Matrix Factorization(BFGS, sgd, als, cg), LDA, Lasso... Firstly, paracel splits both massive dataset and massive parameter space. Unlike Mapreduce-Like Systems, paracel offers a simple communication model, allowing you to work with a global and distributed key-value storage, which is called parameter server.

machine-learning distributed-computing graph c-plus-plus

Hazelcast Jet - A general purpose distributed data processing engine, built on top of Hazelcast.

•    Java

---

Hazelcast Jet is a distributed computing platform built for high-performance stream processing and fast batch processing. It embeds Hazelcast In-Memory Data Grid (IMDG) to provide a lightweight, simple-to-deploy package that includes scalable in-memory storage. Hazelcast Jet performs parallel execution to enable data-intensive applications to operate in near real-time.

in-memory data-grid big-data stream-processing data-processing real-time streams batch-processing

service-fabric - Service Fabric is a distributed systems platform for packaging, deploying, and managing stateless and stateful distributed applications and containers at large scale

•    C++

---

Service Fabric is a distributed systems platform for packaging, deploying, and managing stateless and stateful distributed applications and containers at large scale. Service Fabric runs on Windows and Linux, on any cloud, any datacenter, across geographic regions, or on your laptop. Learn about Service Fabric's Core Subsystems, mapped to this repo's folder structure.

containers orchestration microservices azure distributed-systems cloud-computing

distributed-consensus-reading-list - List of academic papers on distributed consensus

•    

---

This markdown file contains a list of academic papers (and other works) in the field of distributed consensus. Many of the papers listed below fit into more than one section. However, for simplicity, each paper is listed only in the most relevant section. Where possible, open access links for each paper have been provided. Contributions are welcome. This section lists theoretical results relating to distributed consensus.

distributed-systems networked-systems consensus-algorithm paxos atomic-broadcast-protocol zookeeper fault-tolerance distributed-computing

microdot - Microdot: An open source .NET microservices framework

•    CSharp

---

Microdot is an open source .NET framework that answers a lot of the needs for easily creating microservices. Read on for an overview of the framework, and check the wiki for more details and a tutorial for building your first service.

microservice orleans distributed-systems configuration service-discovery

Ganglia - scalable distributed monitoring system

•    C

---

Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization.

monitoring sys-admin-tools tools system-monitoring cluster distributed

storoom - a simple distributed storage

•    DotNet

---

storoom is a simple distributed storage system, it uses key-value pair to store data. contains client/clientlib stornode, node to store data to physical storage system storoom, controller of stornode<s> or storoom<s> develop in vb.net

distributed-storage distributed-systems storage

gleam - Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly

•    Go

---

Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize.Gleam is built in Go, and the user defined computation can be written in Go, Unix pipe tools, or any streaming programs.

distributed-computing map-reduce distributed-systems distributed-system distributed

tla-rust - writing correct lock-free and distributed stateful systems in Rust, assisted by TLA+

•    TLA

---

Stable stateful systems through modeling, linear types and simulation. I like to use things that wake me up at 4am as rarely as possible. Unfortunately, infrastructure vendors don't focus on reliability. Even if a company gives reliability lip service, it's unlikely that they use techniques like modeling or simulation to create a rock-solid core. Let's just build an open-source distributed store that takes correctness seriously at the local storage, sharding, and distributed transactional layers.

tla model-checking lock-free distributed

braft - An industrial-grade C++ implementation of RAFT consensus algorithm based on brpc, widely used inside Baidu to build highly-available distributed systems

•    C++

---

An industrial-grade C++ implementation of RAFT consensus algorithm and replicated state machine based on brpc. braft is designed and implemented for scenarios demanding for high workload and low overhead of latency, with the consideration for easy-to-understand concepts so that engineers inside Baidu can build their own distributed systems individually and correctly. Build brpc which is the main dependency of braft.

raft-consensus-algorithm raft raft-protocol raft-cpp raft-cxx distributed-storage distributed-consensus state-machine-replication

Atomix - Scalable, fault-tolerant distributed systems protocols and primitives for the JVM

•    Java

---

Atomix is an event-driven framework for coordinating fault-tolerant distributed systems built on the Raft consensus algorithm. It provides the building blocks that solve many common distributed systems problems including group membership, leader election, distributed concurrency control, partitioning, and replication.

raft consensus consensus-algorithm raft-protocol raft-consensus-algorithm distributed-systems leader-election distributed-lock data-structures

rqlite - The lightweight, distributed relational database built on SQLite.

•    Go

---

rqlite is a distributed relational database, which uses SQLite as its storage engine. rqlite uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all. It also gracefully handles leader elections, and tolerates failures of machines, including the leader. rqlite is available for Linux, OSX, and Microsoft Windows.rqlite gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation. With it you've got a lightweight and reliable distributed relational data store. Think etcd or Consul, but with relational data modelling also available.

sqlite distributed-systems database relational-database raft consensus distributed-database sql

SeaweedFS - Simple and highly scalable distributed file system

•    Go

---

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, SeaweedFS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

distributed-storage distributed-systems filesystem distributed-filesystem

Foundatio - Pluggable foundation blocks for building distributed apps.

•    CSharp

---

Pluggable foundation blocks for building loosely coupled distributed apps.Includes implementations in Redis, Azure, AWS and in memory (for development).

azure c-sharp lock message-bus storage redis metrics aws distributed-systems foundatio s3 jobs job-scheduler queue messaging statsd distributed-locks distributed logging abstractions