aiofiles - File support for asyncio

•    Python

---

aiofiles is an Apache2 licensed library, written in Python, for handling local disk files in asyncio applications. Ordinary local file IO is blocking, and cannot easily and portably made asynchronous. This means doing file IO may interfere with asyncio applications, which shouldn't block the executing thread. aiofiles helps with this by introducing asynchronous versions of files that support delegating operations to a separate thread pool.

asyncio

SeaweedFS - Simple and highly scalable distributed file system

•    Go

---

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, SeaweedFS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

distributed-storage distributed-systems filesystem distributed-filesystem

goofys - a high-performance, POSIX-ish Amazon S3 file system written in Go

•    Go

---

Goofys allows you to mount an S3 bucket as a filey system. It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have a on disk data cache (checkout catfs), and consistency model is close-to-open.

s3-bucket s3 posix filesystem posix-ish aws-s3 cloud-storage fuse gcs google-cloud-storage openstack-swift fuse-filesystem

Robinhood Policy Engine

•    C

---

Policy engine and reporting tool for large file systems. It maintains a replicate of filesystem medatada in a database that can be queried at will. It makes it possible to schedule mass action on filesystem entries by defining attribute-based policies, provides fast 'find' and 'du' enhanced clones, gives to administrators an overall view of filesystem contents through its web UI and command line tools. It supports any POSIX filesystem and implements advanced features for Lustre filesystems

OrangeFS - Scale-out Network File System

•    C

---

OrangeFS is a scale-out network file system designed for use on high-end computing (HEC) systems that provides very high-performance access to multi-server-based disk storage, in parallel. The OrangeFS server and client are user-level code, making them very easy to install and manage. OrangeFS has optimized MPI-IO support for parallel and distributed applications, and it is leveraged in production installations and used as a research platform for distributed and parallel storage.

filesystem distributed-filesystem storage distributed-storage parallel-virtual-filesystem nas

moosefs - MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System

•    C

---

MooseFS is a Petabyte Open Source Network Distributed File System. It is easy to deploy and maintain, fault tolerant, highly performing, easily scalable, POSIX compliant. MooseFS Linux Client uses FUSE. MooseFS macOS Client uses FUSE for macOS.

dfs software-defined-storage posix filesystem file-system distributed-file-system clustering distributed-storage distributed-computing fuse big-data snapshot storage-tiering high-availability scalability storage moosefs hadoop posix-compliant

Gluster Filesystem - Scalable Network Filesystem

•    C

---

Gluster is a software defined distributed storage that can scale to several petabytes. It provides interfaces for object, block and file storage. It is a distributed scale-out filesystem that allows rapid provisioning of additional storage based on your storage consumption needs. It incorporates automatic failover as a primary feature.

distributed-filesystem filesystem distributed storage scalable-storage network-storage

SystemRescueCd

•    

---

SystemRescueCd is a Linux system rescue disk available as a bootable CD-ROM or USB stick for administrating or repairing your system and data after a crash. It aims to provide an easy way to carry out admin tasks on your computer, such as creating and editing the hard disk partitions. It comes with a lot of software such as disk management tools (parted, partimage, fsarchiver, filesystem tools, ...), network administration programs and simple text editors . It can be used for both Linux an

diskover - File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch

•    Python

---

diskover is an open source file system crawler and disk space usage software that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organize files and system administrators are able to manage storage infrastructure, efficiently provision storage, monitor and report on storage use, and effectively make decisions about new infrastructure purchases. As the amount of file data generated by business' continues to expand, the stress on expensive storage infrastructure, users and system administrators, and IT budgets continues to grow.

elasticsearch crawler filesystem-visualization filesystem-analysis filesystem-indexer disk-space disk-usage storage-analytics storage filesystem file-indexing duplicatefilefinder metadata duplicate-files botnet file-tagging analytics aws-s3 tree-walker

lcfs - LCFS Graph driver for Docker

•    C

---

tl;dr: Every time you build, pull or destroy a Docker container, you are using a storage driver. Current storage drivers like Device Mapper, AUFS, and Overlay2 implement container behavior using file systems designed to run a full OS. We are open-sourcing a file system that is purpose-built for the container lifecycle. We call this new file system Layer Cloning File System (LCFS). Because it is designed only for containers, it is up to 2.5x faster to build an image and up to almost 2x faster to pull an image. We're looking forward to working with the container community to improve and expand this new tool. Layer Cloning FileSystem (LCFS) is a new filesystem purpose-built to be a Docker storage driver. All Docker images are constructed of layers using storage drivers (graph drivers) like AUFS, OverlayFS, and Device Mapper. As a design principle, LCFS focuses on layers as the first-class citizen. The LCFS filesystem operates directly on top of block devices, as opposed to merging separate filesystems. Thereby, LCFS aims to directly manage at the container image’s layer level, eliminate the overhead of having a second filesystem that then is merged, and to optimize for density.

Ceph - Distributed Object Store

•    C++

---

Ceph provides seamless access to objects using native language bindings or radosgw, a REST interface that’s compatible with applications written for S3 and Swift. Ceph’s RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster. Ceph provides a POSIX-compliant network file system that aims for high performance, large data storage, and maximum compatibility with legacy applications.

object-store block-device file-system distributed block filesystem distributed-filesystem distributed-storage

cascadb - Yet another write-optimized storage engine, using buffered B-tree algorithm inspired by TokuDB

•    C++

---

Yet another write-optimized storage engine, using buffered B-tree algorithm inspired by TokuDB. CascaDB can have better performance if libaio and google snappy library 're installed. Otherwise Posix AIO (simulate AIO with multiple threads which is not true asynchronous) is used instead and data blocks're not compressed.

docker-volume-netshare - Docker NFS, AWS EFS, Ceph & Samba/CIFS Volume Plugin

•    Go

---

Mount NFS v3/4, AWS EFS or CIFS inside your docker containers. This is a docker plugin which enables these volume types to be directly mounted within a container.NFS needs to be installed on Linux systems in order to properly mount NFS mounts.

docker volume-plugin nfs efs cifs cifs-volume-plugin samba docker-nfs

aiomysql - Library for accessing a MySQL database from the asyncio

•    Python

---

aiomysql is a "driver" for accessing a MySQL database from the asyncio (PEP-3156/tulip) framework. It depends on and reuses most parts of PyMySQL . aiomysql tries to be like awesome aiopg library and preserve same api, look and feel.Internally aiomysql is copy of PyMySQL, underlying io calls switched to async, basically yield from and asyncio.coroutine added in proper places)). sqlalchemy support ported from aiopg.

asyncio mysql mariadb sqlalchemy mysql-driver mysql-client

tinydir - Lightweight, portable and easy to integrate C directory and file reader

•    C

---

Lightweight, portable and easy to integrate C directory and file reader. TinyDir wraps dirent for POSIX and FindFirstFile for Windows. Windows unicode is supported by defining UNICODE and _UNICODE before including tinydir.h.

dir directory file reader filesystem

POSIX Overlay Filesystem

•    C

---

A FUSE filesystem that provides POSIX functionality - UNIX-style permissions, ownership, special files - for filesystems that do not have such, e.g. vfat. It is a modern equivalent of the UMSDOS fs.

libsqlfs - a library that implements a POSIX style filesystem on top of an SQLite database

•    C

---

a library that implements a POSIX style filesystem on top of an SQLite database

LizardFS - Software Defined Storage, Distributed, Parallel, Scalable, Fault-tolerant, Geo-Redundant and Highly available file system

•    C++

---

LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.

distributed-filesystem filesystem nas distributed-systems distributed-computing distributed-storage fault-tolerance snapshot qos erasure-coding replication replicas geo-replication hsm hierarchical-storage hadoop

BeeGFS - Parallel Cluster File System

•    C

---

BeeGFS (formerly FhGFS) is the leading parallel cluster file system, developed with a strong focus on performance and designed for very easy installation and management. It transparently spreads user data across multiple servers. By increasing the number of servers and disks in the system, you can simply scale performance and capacity of the file system to the level that you need, seamlessly from small clusters up to enterprise-class systems with thousands of nodes.

filesystem distributed-filesystem storage distributed-storage

Emerald Data Store

•    Java

---

Emerald Data Store this is a network file store which implement three (or more) network file systems as FTP, NFS and SMB/CIFS. File system in this data store is a model based on relational database structure. As the database we use Firebird.