gleam - Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly

•    Go

Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize.Gleam is built in Go, and the user defined computation can be written in Go, Unix pipe tools, or any streaming programs.

protoactor-go - Proto Actor - Ultra fast distributed actors for Go, C# and Java/Kotlin

•    Go

Introducing cross platform actor support between Go and C#.Can I use this? The Go implementation is still in beta, there are users using Proto Actor for Go in production already. But be aware that the API might change over time until 1.0.

NakedTensor - Bare bone examples of machine learning in TensorFlow

•    Python

This is a bare bones example of TensorFlow, a machine learning package published by Google. You will not find a simpler introduction to it. In each example, a straight line is fit to some data. Values for the slope and y-intercept of the line that best fit the data are determined using gradient descent. If you do not know about gradient descent, check out the Wikipedia page.

elephas - Distributed Deep learning with Keras & Spark

•    Python

Schematically, elephas works as follows. Elephas brings deep learning with Keras to Spark. Elephas intends to keep the simplicity and high usability of Keras, thereby allowing for fast prototyping of distributed models, which can be run on massive data sets. For an introductory example, see the following iPython notebook.

boinc - Open-source software for volunteer computing and grid computing.

•    PHP

The University of California holds the copyright on all BOINC source code. By submitting contributions to the BOINC code, you irrevocably assign all right, title, and interest, including copyright and all copyright rights, in such contributions to The Regents of the University of California, who may then use the code for any purpose that it desires. BOINC is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

protoactor-dotnet - Proto Actor - Ultra fast distributed actors for Go, C# and Java/Kotlin

•    CSharp

Ultra-fast, distributed, cross-platform actors.This is the .NET repository for Proto Actor.

distributed - Distributed computation in Python

•    Python

A library for distributed computation. See documentation for more details.

moosefs - MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System

•    C

MooseFS is a Petabyte Open Source Network Distributed File System. It is easy to deploy and maintain, fault tolerant, highly performing, easily scalable, POSIX compliant. MooseFS Linux Client uses FUSE. MooseFS macOS Client uses FUSE for macOS.

rain - Framework for large distributed pipelines

•    Rust

Rain is an open-source distributed computational framework for processing of large-scale task-based pipelines. Rain aims to lower the entry barrier to the world of distributed computing. Our intention is to provide a light yet robust distributed framework that features an intuitive Python API, straightforward installation and deployment with insightful monitoring on top.

transient - A full stack, reactive architecture for general purpose programming

•    Haskell

It's a bit mind bending in that it's like using a higher-level list monad, but it's very, very cool. For beginning Haskellers, what would be really useful is a visualisation of what happens when you do various distributed/parallel stuff. It's almost shocking how effortlessly you can run computations across threads/nodes. The cool part is the composability in the distributed setting. You can make higher-order monadic functions that allow you to compose & reuse a long chain of distributed transactions via wormhole and teleport. Another benefit is that the transaction becomes first class and you can see exactly what's going on in one place instead of distributing the logic across actors making the code equivalent to event callbacks, as you've stated.

SmartSql - SmartSql = MyBatis

•    CSharp

SmartSql = MyBatis + Cache(Memory | Redis) + R/W Splitting +Dynamic Repository + Diagnostics ...... SmartSql draws on MyBatis's ideas, uses XML to manage SQL, and provides several filter tags to eliminate various if/else judgment branches at the code level. SmartSql will manage your SQL and filter the tags to maintain your various conditional judgments at the code level to make your code more beautiful.

LizardFS - Software Defined Storage, Distributed, Parallel, Scalable, Fault-tolerant, Geo-Redundant and Highly available file system

•    C++

LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.

JPPF - Parallelize computationally intensive tasks and execute them on a Grid

•    Java

JPPF enables applications with large processing power requirements to be run on any number of computers, in order to dramatically reduce their processing time. This is done by splitting an application into smaller parts that can be executed simultaneously on different machines.

future - :rocket: R package: future: Unified Parallel and Distributed Processing in R for Everyone

•    R

The purpose of the future package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user. In programming, a future is an abstraction for a value that may be available at some point in the future. The state of a future can either be unresolved or resolved. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is blocked until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved depends on what strategy is used to evaluate them. For instance, a future can be resolved using a sequential strategy, which means it is resolved in the current R session. Other strategies may be to resolve futures asynchronously, for instance, by evaluating expressions in parallel on the current machine or concurrently on a compute cluster.

paracel - Distributed training framework with parameter server

•    C++

Paracel is a distributed computational framework, designed for many machine learning problems: Logistic Regression, SVD, Matrix Factorization(BFGS, sgd, als, cg), LDA, Lasso... Firstly, paracel splits both massive dataset and massive parameter space. Unlike Mapreduce-Like Systems, paracel offers a simple communication model, allowing you to work with a global and distributed key-value storage, which is called parameter server.

distributed-consensus-reading-list - List of academic papers on distributed consensus

This markdown file contains a list of academic papers (and other works) in the field of distributed consensus. Many of the papers listed below fit into more than one section. However, for simplicity, each paper is listed only in the most relevant section. Where possible, open access links for each paper have been provided. Contributions are welcome. This section lists theoretical results relating to distributed consensus.

awesome-distributed-systems - Awesome list of distributed systems resources

Awesome list of distributed systems resources

sleuth - A Go library for master-less peer-to-peer autodiscovery and RPC between HTTP services

•    Go

sleuth is a Go library that provides master-less peer-to-peer autodiscovery and RPC between HTTP services that reside on the same network. It works with minimal configuration and provides a mechanism to join a local network both as a client that offers no services and as any service that speaks HTTP. Its primary use case is for microservices on the same network that make calls to one another.sleuth is dependent on libzmq, which can be installed either from source or from binaries. For more information, please refer to ØMQ: "Get the Software" or the libzmq repository.

fishnet - Distributed Stockfish analysis for lichess.org

•    Python

Install the fishnet client. fishnet is using lichess.org custom Stockfish by @ddugovic.

taskinator - A simple orchestration library for running complex processes or workflows in Ruby

•    Ruby

A simple orchestration library for running complex processes or workflows in Ruby. Processes are defined using a simple DSL, where the sequences and tasks are defined. Processes can then be queued for execution. Sequences can be synchronous or asynchronous, and the overall process can be monitored for completion or failure. The configuration and state of each process and their respective tasks is stored using Redis key/values.