dvid - Distributed, Versioned, Image-oriented Dataservice

  •        35

Status: In production use at Janelia. See wiki page for outside lab use of DVID. See the DVID Wiki for more information including installation and examples of use.

https://github.com/janelia-flyem/dvid

Tags
Implementation
License
Platform

   




Related Projects

Apache Accumulo - Key Value Store based on Google BigTable

  •    Java

The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Apache Accumulo is based on Google's BigTable design and is built on top of Apache Hadoop, Zookeeper, and Thrift. Apache Accumulo features a few novel improvements on the BigTable design in the form of cell-based access control and a server-side programming mechanism that can modify key/value pairs at various points in the data management process.

Griddb - High performance, High scalability and High reliability database for big data

  •    C++

GridDB is an In-Memory NoSQL Database for highly scalable IoT applications . It has a KVS (Key-Value Store)-type data model that is suitable for sensor data stored in a timeseries. It is a database that can be easily scaled-out according to the number of sensors. High Reliability It is equipped with a structure to spread out the replication of key value data among fellow nodes so that in the event of a node failure, automatic failover can be carried out in a matter of seconds by using the replication function of other nodes.

huststore - High-performance Distributed Storage

  •    C

huststore is a open source high performance distributed database system. It not only provides key-value storage service with extremely high performance, up to 100 thousand QPS, but also supports data structures like hash, set, sorted set, etc. Also, it can store binary data as value from a key-value pair, and thus can be used as an alternative of Redis.In addtion, huststore implements a distributed message queue by integrating a special HA module, features including message Push Stream, and message Publish-SubScribe, these features can be used as replacements of the corresponding features in rabbitmq and gearman.

membase - distributed key-value database

  •    C

Membase is an distributed, key-value database management system optimized for storing data behind interactive web applications. These applications must service many concurrent users, creating, storing, retrieving, aggregating, manipulating and presenting data in real-time. Supporting these requirements, membase processes data operations with quasi-deterministic low latency and high sustained throughput.

Pika - nosql compatible with redis, it is developed by Qihoo's DBA and infrastructure team

  •    C++

Pika is a persistent huge storage service , compatible with the vast majority of redis interfaces (details), including string, hash, list, zset, set and management interfaces. With the huge amount of data stored, redis may suffer for a capacity bottleneck, and pika was born for solving it. Except huge storage capacity, pika also support master-slave mode by slaveof command, including full and partial synchronization.


fastdfs - FastDFS is an open source high performance distributed file system (DFS)

  •    C

FastDFS is an open source high performance distributed file system. It's major functions include: file storing, file syncing and file accessing (file uploading and file downloading), and it can resolve the high capacity and load balancing problem. FastDFS should meet the requirement of the website whose service based on files such as photo sharing site and video sharing site. FastDFS has two roles: tracker and storage. The tracker takes charge of scheduling and load balancing for file access. The storage store files and it's function is file management including: file storing, file syncing, providing file access interface. It also manage the meta data which are attributes representing as key value pair of the file. For example: width=1024, the key is "width" and the value is "1024".

daos - DAOS Storage Engine

  •    C

The Distributed Asynchronous Object Storage (DAOS) is an open-source software-defined object store designed from the ground up for massively distributed Non Volatile Memory (NVM). DAOS takes advantage of next generation NVM technology like Storage Class Memory (SCM) and NVM express (NVMe) while presenting a key-value storage interface and providing features such as transactional non-blocking I/O, advanced data protection with self-healing on top of commodity hardware, end-to-end data integrity, fine-grained data control and elastic storage to optimize performance and cost. DAOS is licensed under the BSD-2-Clause Plus Patent License. Please see the LICENSE & NOTICE files for more information.

RocksDB - A persistent key-value store for fast storage environments

  •    C++

RocksDB is an embeddable persistent key-value store for fast storage. RocksDB builds on LevelDB to be scalable to run on servers with many CPU cores, to efficiently use fast storage, to support IO-bound, in-memory and write-once workloads, and to be flexible to allow for innovation. It is especially suited for storing data on flash drives. It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Apache Gora - Provides persistence to column stores, key value stores, document stores and RDBMS

  •    Java

The Apache Gora open source framework provides an in-memory data model and persistence for big data. Gora supports persisting to column stores, key value stores, document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce support.

Project-voldemort - A distributed database, Clone of Amazon's Dynamo

  •    Java

Voldemort is a distributed key-value storage system. Data is automatically replicated over multiple servers. Data is automatically partitioned so each server contains only a subset of the total data. Server failure is handled transparently. It is used at LinkedIn for certain high-scalability storage problems where simple functional partitioning is not sufficient.

mapreduce - C++ MapReduce Library for efficient multi-threading on single-machine

  •    C++

The MapReduce C++ Library implements a single-machine platform for programming using the the Google MapReduce idiom. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the Google paper. The developer is required to write two classes; MapTask implements a mapping function to process key/value pairs generate a set of intermediate key/value pairs and ReduceTask that implements a reduce function to merges all intermediate values associated with the same intermediate key. In addition, there are three optional template parameters that can be used to modify the default implementation behavior; Datasource that implements a mechanism to feed data to the Map Tasks - on request of the MapReduce library, Combiner that can be used to partially consolidate results of the Map Task before they are passed to the Reduce Tasks, and IntermediateStore that handles storage, merging and sorting of intermediate results between the Map and Reduce phases. The MapTask class must define four data types; the key/value types for the inputs to the Map Tasks and the intermediate types.

OpenStack - Software for building Private and Public Clouds

  •    Python

OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.

Keyv - Simple key-value storage with support for multiple backends

  •    Javascript

Keyv provides a consistent interface for key-value storage across multiple backends via storage adapters. It supports TTL based expiry, making it suitable as a cache or a persistent key-value store. It works with any storage that implements the Map API. It handles all JSON types plus Buffer. It has wide range of efficient, well tested storage adapters.

ForestDB - A Fast Key-Value Storage Engine Based on Hierarchical B+-Tree Trie

  •    C++

ForestDB is a key-value storage engine developed by Couchbase Caching and Storage Team, and its main index structure is built from Hierarchical B+-Tree based Trie, called HB+-Trie. ForestDB paper has been published in IEEE Transactions on Computers.

LargeLocalStorage - Problem: You need to store a large amount of key-value based data in IE, Chrome, Safari, AND Firefox

  •    Javascript

Problem: You need a large key-value store in the browser. LargeLocalStorage bridges all of that to give you a large capacity (up to several GB when authorized by the user) key-value store in the browser (IE 10, Chrome, Safari 6+, Firefox, Opera).

godown - Distributed, fault-tolerant key-value storage written in go.

  •    Go

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List. You can find binaries on the Github releases page.

PumpkinDB - Event Sourcing Database Engine

  •    Rust

PumpkinDB is a event sourcing database engine that doesn't overwrite data. It features include Immutable key/value storage, ACID transactions, Binary keys and values (allows any encoding to be used: JSON, XML, Protobuf, Cap'n Proto, etc.), An embedded programming language (PumpkinScript), A range of event indexing and querying primitives.

nitro - A high performance in-memory index storage engine

  •    Go

A high performance in-memory key-value item storage engine written in golang. The storage engine is based lock-free data structures and scales well with multicore CPUs.

etcd - Distributed reliable key-value store for the most critical data of a distributed system

  •    Go

etcd is a distributed, consistent key-value store for shared configuration and service discovery. It is simple, secure, fast and reliable. it uses the Raft consensus algorithm to manage a highly-available replicated log.






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.