s4 - Distributed Stream Computing Platform

  •        1478

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. S4 has been deployed in production systems at Yahoo to process thousands of search queries per second.




Related Projects

Akka - Build Concurrent and Scalable Applications

Akka is the platform for the next generation event-driven, scalable and fault-tolerant architectures on the JVM. It helps to write simpler correct concurrent applications using Actors, STM & Transactors. It could scale out on multi-core or multiple nodes using asynchronous message passing. For fault-tolerance it adopts the Let it crash or Embrace failure model to build applications that self-heals, systems that never stop.

ejabberd - Robust, Scalable and Extensible XMPP Server

ejabberd is a distributed, fault-tolerant technology that allows the creation of large-scale instant messaging applications. The server can reliably support thousands of simultaneous users on a single node and has been designed to provide exceptional standards of fault tolerance. As an open source technology, based on industry-standards, ejabberd can be used to build bespoke solutions very cost effectively.

Atomix - Scalable, fault-tolerant distributed systems protocols and primitives for the JVM

Atomix is an event-driven framework for coordinating fault-tolerant distributed systems built on the Raft consensus algorithm. It provides the building blocks that solve many common distributed systems problems including group membership, leader election, distributed concurrency control, partitioning, and replication.

lifeguard - Lifeguard is a highly scalable, fault tolerant distributed infrastructure and application monitoring service

Lifeguard is still alpha quality and not ready for production use.Lifeguard is a highly scalable, fault tolerant distributed infrastructure and application monitoring service. The health of your systems are determined by checks written in JavaScript that run across the monitoring cluster. The data that the checks inspect can be retrieved from various sources, such as a database, Graphite, or remote SSH commands. Failing checks notify your IT team as necessary.

ringpop-go - Scalable, fault-tolerant application-layer sharding for Go applications

Ringpop is a library that brings cooperation and coordination to distributed applications. It maintains a consistent hash ring on top of a membership protocol and provides request forwarding as a routing convenience. It can be used to shard your application in a way that's scalable and fault tolerant.to install remaining golang dependencies and install the pre-commit hook.

NSQ - A realtime distributed messaging platform in Go

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. It scales horizontally, without any centralized brokers. Built-in discovery simplifies the addition of nodes to the cluster.

Pinot - A realtime distributed OLAP datastore

Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally, so that it can scale to larger data sets and higher query rates as needed.

Elixir - Dynamic, Functional Language designed for building scalable and maintainable application

Elixir is a dynamic, functional language designed for building scalable and maintainable applications. Elixir leverages the Erlang VM, known for running low-latency, distributed and fault-tolerant systems, while also being successfully used in web development and the embedded software domain.

placid - A REST toolkit for building highly-scalable and fault-tolerant HTTP APIs with Elixir

A REST toolkit for building highly-scalable and fault-tolerant HTTP APIs with Elixir

NBitcoin.Indexer - Scalable and fault-tolerant block chain indexer for Azure cloud platform.

Leverage Azure, powershell and NBitcoin for a fault tolerant and scalable block chain indexer.All information on codeproject.

Hyperdex - A Searchable Distributed Key-Value Store

HyperDex is a distributed, searchable key-value store. HyperDex provides a unique search primitive which enables searches over stored values. By design, HyperDex retains the performance of traditional key-value stores while enabling support for the search operation. It is fast, scalable, Consistent, Fault tolerant.

disque - Disque is a distributed message broker

Disque is an ongoing experiment to build a distributed, in-memory, message broker. Its goal is to capture the essence of the "Redis as a jobs queue" use case, which is usually implemented using blocking list operations, and move it into an ad-hoc, self-contained, scalable, and fault tolerant design, with simple to understand properties and guarantees, but still resembling Redis in terms of simplicity, performance, and implementation as a C non-blocking networked server.Currently (2 Jan 2016) the project is in release candidate state. People are encouraged to start evaluating it and report bugs and experiences.

JSBus - Fault tolerant messaging between browser and server

Fault tolerant messaging between browser and server

distributed-ftp - Byzantine Fault tolerant distributed file system

Byzantine Fault tolerant distributed file system

FlockDB - A distributed, fault-tolerant graph database from Twitter

FlockDB is much simpler than other graph databases such as neo4j because it tries to solve fewer problems. It scales horizontally and is designed for on-line, low-latency, high throughput environments such as web-sites. Twitter uses FlockDB to store social graphs (who follows whom, who blocks whom) and secondary indices. As of April 2010, the Twitter FlockDB cluster stores 13+ billion edges and sustains peak traffic of 20k writes/second and 100k reads/second.

Gearman - Application Framework to farm out work to other Machines

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel by doing load balancing. It also supports to call functions between languages. It is the nervous system for how distributed processing communicates. It is fault tolerant.

ejabberd - Fault-tolerant distributed Jabber/XMPP server

Fault-tolerant distributed Jabber/XMPP server

mfs - MooseFS - distributed, fault tolerant file system

MooseFS - distributed, fault tolerant file system

Shoggoth - Simple distributed fault-tolerant file system

Simple distributed fault-tolerant file system