rusix - Distributed filesystem in Rust

  •        53

This project written in Rust because it is giving real guaranties for preventing data races and completely handles memory management without garbage collection, which is giving huge performance improvements and low memory usage. Want to contribute to Rusix? Awesome! Check out the contributing guide.

https://github.com/cholcombe973/rusix

Tags
Implementation
License
Platform

   




Related Projects

SeaweedFS - Simple and highly scalable distributed file system

  •    Go

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, SeaweedFS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

fastdfs - FastDFS is an open source high performance distributed file system (DFS)

  •    C

FastDFS is an open source high performance distributed file system. It's major functions include: file storing, file syncing and file accessing (file uploading and file downloading), and it can resolve the high capacity and load balancing problem. FastDFS should meet the requirement of the website whose service based on files such as photo sharing site and video sharing site. FastDFS has two roles: tracker and storage. The tracker takes charge of scheduling and load balancing for file access. The storage store files and it's function is file management including: file storing, file syncing, providing file access interface. It also manage the meta data which are attributes representing as key value pair of the file. For example: width=1024, the key is "width" and the value is "1024".

bfs - The Baidu File System.

  •    C++

The Baidu File System (BFS) is a distributed file system designed to support real-time applications. Like many other distributed file systems, BFS is highly fault-tolerant. But different from others, BFS provides low read/write latency while maintaining high throughput rates. Together with Galaxy and Tera, BFS supports many real-time products in Baidu, including Baidu webpage database, Baidu incremental indexing system, Baidu user behavior analysis system, etc.


Hadoop Common

  •    Java

Apache Hadoop is a framework for running applications on large clusters built of commodity hardware. Hadoop common supports other Hadoop subprojects

tfs - TFS (Taobao File System) is a distributed file system similar to GFS.

  •    C++

TFS (Taobao File System) is a distributed file system similar to GFS.

BeeGFS - Parallel Cluster File System

  •    C

BeeGFS (formerly FhGFS) is the leading parallel cluster file system, developed with a strong focus on performance and designed for very easy installation and management. It transparently spreads user data across multiple servers. By increasing the number of servers and disks in the system, you can simply scale performance and capacity of the file system to the level that you need, seamlessly from small clusters up to enterprise-class systems with thousands of nodes.

Ceph - Distributed Object Store

  •    C++

Ceph provides seamless access to objects using native language bindings or radosgw, a REST interface that’s compatible with applications written for S3 and Swift. Ceph’s RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster. Ceph provides a POSIX-compliant network file system that aims for high performance, large data storage, and maximum compatibility with legacy applications.

OrangeFS - Scale-out Network File System

  •    C

OrangeFS is a scale-out network file system designed for use on high-end computing (HEC) systems that provides very high-performance access to multi-server-based disk storage, in parallel. The OrangeFS server and client are user-level code, making them very easy to install and manage. OrangeFS has optimized MPI-IO support for parallel and distributed applications, and it is leveraged in production installations and used as a research platform for distributed and parallel storage.

bfs - distributed file system(small file storage) writen by golang.

  •    Go

distributed file system(small file storage) writen by golang.

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo

  •    Java

A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC.

Archon

  •    Python

It is a web based distributed system controller used to build/manage linux boxes for Declum. @ Declum 2013

HPCC System - Hadoop alternative

  •    C++

HPCC is a proven and battle-tested platform for manipulating, transforming, querying and data warehousing Big Data. It supports two type of configuration. Thor is responsible for consuming vast amounts of data, transforming, linking and indexing that data. It functions as a distributed file system with parallel processing power spread across the nodes. Roxie, the Data Delivery Engine, provides separate high-performance online query processing and data warehouse capabilities.

LogDevice - Distributed storage for sequential data

  •    C++

LogDevice is a scalable and fault tolerant distributed log system. While a file-system stores and serves data organized as files, a log system stores and delivers data organized as logs. The log can be viewed as a record-oriented, append-only, and trimmable file. LogDevice is supported on Ubuntu 18.04 (Bionic Beaver) only. However, it should be possible to build it on any other Linux distribution without significant challenges.

Next Generation File System

  •    C

Next generation file system: Support great capability and big file, support distributed storage and network storage, support encrypt storage, easy to port to any OS. At the core of this file system is the support of Universal File System Interface(UFSI)

Apache Tajo - A big data warehouse system on Hadoop

  •    Java

Apache Tajo is a robust big data relational and distributed data warehouse system for Apache Hadoop. Tajo is designed for low-latency and scalable ad-hoc queries, online aggregation, and ETL (extract-transform-load process) on large-data sets stored on HDFS (Hadoop Distributed File System) and other data sources.

Distributed Chat+File Excng Over Tib/Rv

  •    Java

A distributed client for the increased collaboration of distributed project teams. Written in Java and running over the Tibco Tib/Rv messaging system this project encompasses distributed chat, file sharing and document/code collaboration.

Dragonfly - Dragonfly is an intelligent P2P based file distribution system.

  •    Java

Dragonfly is an intelligent P2P based file distribution system. It resolved issues like low-efficiency,low-success rate,waste of network bandwidth you faced in large-scale file distribution scenarios such as application deployment, large-scale cache file distribution, data file distribution, images distribution etc. In Alibaba, the system transferred 2 billion times and distributed 3.4PB data every month, it becomes one of the most important infrastructure in Alibaba. The reliability is up to 99.9999%. DevOps takes a lot of benefits from container technologies . but at the same time, it also bring a lot of challenges: the efficiency of image distribution, especially when you have a lot of applications and require image distribution at the same time. Dragonfly works extremely well with both Docker and Pouch, and actually we compatible with any other container technologies without any modifications of container engine.

QConf - Qihoo Distributed Configuration Management System

  •    C++

We have build another interesting proect pika. Pika is a nosql compatible with redis protocol with huge storage space. You can have a try.QConf is a Distributed Configuration Management System! A better replacement of the traditional configuration file. As designed, configuration items which is constantly accessed and modified should be completely separated with application code, and QConf is where it should be.





We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.