SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, SeaweedFS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".
distributed-storage distributed-systems filesystem distributed-filesystemhuststore is a open source high performance distributed database system. It not only provides key-value storage service with extremely high performance, up to 100 thousand QPS, but also supports data structures like hash, set, sorted set, etc. Also, it can store binary data as value from a key-value pair, and thus can be used as an alternative of Redis.In addtion, huststore implements a distributed message queue by integrating a special HA module, features including message Push Stream, and message Publish-SubScribe, these features can be used as replacements of the corresponding features in rabbitmq and gearman.
architecture benchmark redis database-engine nginx message-queue distributed-storageAn industrial-grade C++ implementation of RAFT consensus algorithm and replicated state machine based on brpc. braft is designed and implemented for scenarios demanding for high workload and low overhead of latency, with the consideration for easy-to-understand concepts so that engineers inside Baidu can build their own distributed systems individually and correctly. Build brpc which is the main dependency of braft.
raft-consensus-algorithm raft raft-protocol raft-cpp raft-cxx distributed-storage distributed-consensus state-machine-replicationDragonboat is a high performance multi-group Raft consensus library in Go with C++11 binding support. Consensus algorithms such as Raft provides fault-tolerance by alllowing a system continue to operate as long as the majority member servers are available. For example, a Raft cluster of 5 servers can make progress even if 2 servers fail. It also appears to clients as a single node with strong data consistency always provided. All running servers can be used to initiate read requests for aggregated read throughput.
raft paxos distributed-systems raft-protocol raft-algorithm consensus replicated-state-machines raft-cpp raft-cxx distributed-storageAlluxio (formerly known as Tachyon) is a virtual distributed storage system. It bridges the gap between computation frameworks and storage systems, enabling computation applications to connect to numerous storage systems through a common interface.
distributed-storage big-data memory-speed hadoop spark virtual-file-system presto tensorflow storage object-storeAn industrial-grade C++ implementation of RAFT consensus algorithm and replicated state machine based on brpc. braft is designed and implemented for scenarios demanding for high workload and low overhead of latency, with the consideration for easy-to-understand concepts so that engineers inside Baidu can build their own distributed systems individually and correctly. Build brpc which is the main dependency of braft.
raft distributed-storage raft-protocol raft-consensus-algorithm distributed-consensus state-machine-replication raft-cpp raft-cxxSheepdog is a distributed object storage system for volume and container services and manages the disks and nodes intelligently. Sheepdog features ease of use, simplicity of code and can scale out to thousands of nodes. The block level volume abstraction can be attached to QEMU virtual machines and Linux SCSI Target and supports advanced volume management features such as snapshot, cloning, and thin provisioning.
qemu qemu-kvm iscsi iscsi-target blockdevice storage distributed-storage virtualization vmm software-defined-storage sdnCeph provides seamless access to objects using native language bindings or radosgw, a REST interface that’s compatible with applications written for S3 and Swift. Ceph’s RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster. Ceph provides a POSIX-compliant network file system that aims for high performance, large data storage, and maximum compatibility with legacy applications.
object-store block-device file-system distributed block filesystem distributed-filesystem distributed-storageXtreemFS is a distributed, replicated and fault-tolerant file system for federated IT infrastructures. XtreemFS replicates your file data across multiple storage servers, which can be distributed worldwide. It scales according to your needs within minutes, simply by adding new standard hardware. Starting with a single server, growing to a cluster and across datacenters.
filesystem distributed-filesystem storage distributed-storage fault-tolerantOrangeFS is a scale-out network file system designed for use on high-end computing (HEC) systems that provides very high-performance access to multi-server-based disk storage, in parallel. The OrangeFS server and client are user-level code, making them very easy to install and manage. OrangeFS has optimized MPI-IO support for parallel and distributed applications, and it is leveraged in production installations and used as a research platform for distributed and parallel storage.
filesystem distributed-filesystem storage distributed-storage parallel-virtual-filesystem nasAll of our code for Storj v3 is open source. Have a code change you think would make Storj better? Please send a pull request along! Make sure to sign our Contributor License Agreement (CLA) first. See our license section for more details. Have comments, bug reports, or suggestions? Want to propose a PR before hand-crafting it? Jump on to our Rocketchat and join the #dev channel to say hi to the developer community and to talk to the Storj core team.
distributed-storage distributed-systems distributed storage object-storageMooseFS is a Petabyte Open Source Network Distributed File System. It is easy to deploy and maintain, fault tolerant, highly performing, easily scalable, POSIX compliant. MooseFS Linux Client uses FUSE. MooseFS macOS Client uses FUSE for macOS.
dfs software-defined-storage posix filesystem file-system distributed-file-system clustering distributed-storage distributed-computing fuse big-data snapshot storage-tiering high-availability scalability storage moosefs hadoop posix-compliantPravega is an open source distributed storage service implementing Streams. It offers Stream as the main primitive for the foundation of reliable storage systems: a high-performance, durable, elastic, and unlimited append-only byte stream with strict ordering and consistency.
streaming streaming-data distributed-storage real-time-data data-ingestion analyticsUpload static website to IPFS pinning services and optionally update DNS. The goal of ipfs-deploy is to make it as easy as possible to deploy a static website to IPFS.
ipfs static-site-generator dns distributed-storage dweb npm-package cloudflare-ipfs pinata infura pinning dnslink nodejs npm cli executable binaries zeroconfig 0cjs ipfs-pinning gatsby deployLizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.
distributed-filesystem filesystem nas distributed-systems distributed-computing distributed-storage fault-tolerance snapshot qos erasure-coding replication replicas geo-replication hsm hierarchical-storage hadoopBeeGFS (formerly FhGFS) is the leading parallel cluster file system, developed with a strong focus on performance and designed for very easy installation and management. It transparently spreads user data across multiple servers. By increasing the number of servers and disks in the system, you can simply scale performance and capacity of the file system to the level that you need, seamlessly from small clusters up to enterprise-class systems with thousands of nodes.
filesystem distributed-filesystem storage distributed-storageThe Distributed Asynchronous Object Storage (DAOS) is an open-source software-defined object store designed from the ground up for massively distributed Non Volatile Memory (NVM). DAOS takes advantage of next generation NVM technology like Storage Class Memory (SCM) and NVM express (NVMe) while presenting a key-value storage interface and providing features such as transactional non-blocking I/O, advanced data protection with self-healing on top of commodity hardware, end-to-end data integrity, fine-grained data control and elastic storage to optimize performance and cost. DAOS is licensed under the BSD-2-Clause Plus Patent License. Please see the LICENSE & NOTICE files for more information.
kvstore object-store distributed-storage software-defined-storage daos spdk hyperconverged nvme-storage pmdk hyperscale storage-as-a-servicestoroom is a simple distributed storage system, it uses key-value pair to store data. contains client/clientlib stornode, node to store data to physical storage system storoom, controller of stornode<s> or storoom<s> develop in vb.net
distributed-storage distributed-systems storageAwesome list of distributed systems resources
awesome awesome-list distributed-systems distributed distributed-database distributed-computing distributed-storage computer-science computer-engineeringThe Eclipse zenoh: Zero Overhead Pub/sub, Store/Query and Compute. Eclipse zenoh (pronounce /zeno/) unifies data in motion, data in-use, data at rest and computations. It carefully blends traditional pub/sub with geo-distributed storages, queries and computations, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks.
iot distributed-systems embedded networking robotics storage protocol messaging distributed-computing distributed distributed-storage network-programming ros2 edge-computing zenoh geo-distributed-storages
We have large collection of open source products. Follow the tags from
Tag Cloud >>
Open source products are scattered around the web. Please provide information
about the open source projects you own / you use.
Add Projects.