diskover - File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch

  •        51

diskover is an open source file system crawler and disk space usage software that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organize files and system administrators are able to manage storage infrastructure, efficiently provision storage, monitor and report on storage use, and effectively make decisions about new infrastructure purchases. As the amount of file data generated by business' continues to expand, the stress on expensive storage infrastructure, users and system administrators, and IT budgets continues to grow.

https://shirosaidev.github.io/diskover
https://github.com/shirosaidev/diskover

Tags
Implementation
License
Platform

   




Related Projects

goofys - a high-performance, POSIX-ish Amazon S3 file system written in Go

  •    Go

Goofys allows you to mount an S3 bucket as a filey system. It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have a on disk data cache (checkout catfs), and consistency model is close-to-open.

SeaweedFS - Simple and highly scalable distributed file system

  •    Go

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, SeaweedFS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

OrangeFS - Scale-out Network File System

  •    C

OrangeFS is a scale-out network file system designed for use on high-end computing (HEC) systems that provides very high-performance access to multi-server-based disk storage, in parallel. The OrangeFS server and client are user-level code, making them very easy to install and manage. OrangeFS has optimized MPI-IO support for parallel and distributed applications, and it is leveraged in production installations and used as a research platform for distributed and parallel storage.

PartImage - Disk Backup Software

  •    C

Partimage is opensource disk backup software. It saves partitions having a supported filesystem on a sector basis to an image file. Although it runs under Linux, Windows and most Linux filesystems are supported. The image file can be compressed to save disk space and transfer time and can be split into multiple files to be copied to CDs or DVDs.

lcfs - LCFS Graph driver for Docker

  •    C

tl;dr: Every time you build, pull or destroy a Docker container, you are using a storage driver. Current storage drivers like Device Mapper, AUFS, and Overlay2 implement container behavior using file systems designed to run a full OS. We are open-sourcing a file system that is purpose-built for the container lifecycle. We call this new file system Layer Cloning File System (LCFS). Because it is designed only for containers, it is up to 2.5x faster to build an image and up to almost 2x faster to pull an image. We're looking forward to working with the container community to improve and expand this new tool. Layer Cloning FileSystem (LCFS) is a new filesystem purpose-built to be a Docker storage driver. All Docker images are constructed of layers using storage drivers (graph drivers) like AUFS, OverlayFS, and Device Mapper. As a design principle, LCFS focuses on layers as the first-class citizen. The LCFS filesystem operates directly on top of block devices, as opposed to merging separate filesystems. Thereby, LCFS aims to directly manage at the container image’s layer level, eliminate the overhead of having a second filesystem that then is merged, and to optimize for density.


Gluster Filesystem - Scalable Network Filesystem

  •    C

Gluster is a software defined distributed storage that can scale to several petabytes. It provides interfaces for object, block and file storage. It is a distributed scale-out filesystem that allows rapid provisioning of additional storage based on your storage consumption needs. It incorporates automatic failover as a primary feature.

TrueCrypt - Disk Encryption Software

  •    C

TrueCrypt is software for establishing and maintaining an on-the-fly-encrypted volume (data storage device). On-the-fly encryption means that data is automatically encrypted right before it is saved and decrypted right after it is loaded, without any user intervention. No data stored on an encrypted volume can be read (decrypted) without using the correct password/keyfile(s) or correct encryption keys.

TMSU - TMSU lets you tags your files and then access them through a nifty virtual filesystem from any other application

  •    Go

TMSU is a tool for tagging your files. It provides a simple command-line utility for applying tags and a virtual filesystem to give you a tag-based view of your files from any other program. TMSU does not alter your files in any way: they remain unchanged on disk, or on the network, wherever your put them. TMSU maintains its own database and you simply gain an additional view, which you can mount where you like, based upon the tags you set up.

multer-s3 - multer storage engine for amazon s3

  •    Javascript

Streaming multer storage engine for AWS S3. This project is mostly an integration piece for existing code samples from Multer's storage engine documentation with s3fs as the substitution piece for file system. Existing solutions I found required buffering the multipart uploads into the actual filesystem which is difficult to scale.

Ceph - Distributed Object Store

  •    C++

Ceph provides seamless access to objects using native language bindings or radosgw, a REST interface that’s compatible with applications written for S3 and Swift. Ceph’s RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster. Ceph provides a POSIX-compliant network file system that aims for high performance, large data storage, and maximum compatibility with legacy applications.

BrowserFS - BrowserFS is an in-browser filesystem that emulates the Node JS filesystem API and supports storing and retrieving files from various backends

  •    Javascript

BrowserFS is an in-browser file system that emulates the Node JS file system API and supports storing and retrieving files from various backends. BrowserFS also integrates nicely into the Emscripten file system. More backends can be defined by separate libraries, so long as they extend the BaseFileSystem class. Multiple backends can be active at once at different locations in the directory hierarchy.

simplefs - A simple, kernel-space, on-disk filesystem from the scratch

  •    C

A simple, kernel-space, on-disk filesystem from the scratch

Marketstore - DataFrame Server for Financial Timeseries Data

  •    Go

MarketStore is a database server optimized for financial timeseries data. You can think of it as an extensible DataFrame service that is accessible from anywhere in your system, at higher scalability. It is designed from the ground up to address scalability issues around handling large amounts of financial market data used in algorithmic trading backtesting, charting, and analyzing price history with data spanning many years, including tick-level for the all US equities or the exploding crypto currencies space. If you are struggling with managing lots of HDF5 files, this is perfect solution to your problem.

Dump/Restore ext2/3/4 filesystem backup

  •    C

Dump examines files in a filesystem, determines which ones need to be backed up, and copies those files to a specified disk, tape or other storage medium. The restore command performs the inverse function of dump

Afero - A FileSystem Abstraction System for Go

  •    Go

Afero is an filesystem framework providing a simple, uniform and universal API interacting with any filesystem, as an abstraction layer providing interfaces, types and methods. Afero has an exceptionally clean interface and simple design without needless constructors or initialization methods.Afero is also a library providing a base set of interoperable backend filesystems that make it easy to work with afero while retaining all the power and benefit of the os and ioutil packages.

riofs - Userspace S3 filesystem

  •    C

RioFS is an userspace filesystem for Amazon S3 buckets for servers that run on Linux and MacOSX. It supports versioned and non-versioned buckets in all AWS regions. RioFS development started at Skoobe as a storage backend for legacy daemons which cannot talk natively to S3. It handles buckets with many thousands of keys and highly concurrent access gracefully.

mc - Minio Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage

  •    Go

Minio Client (mc) provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff, find etc. It supports filesystems and Amazon S3 compatible cloud storage service (AWS Signature v2 and v4).then use the mc config command.

svfs - The Swift Virtual File System

  •    Go

SVFS is a Virtual File System over Openstack Swift built upon fuse. It is compatible with hubiC, OVH Public Cloud Storage and basically every endpoint using a standard Openstack Swift setup. It brings a layer of abstraction over object storage, making it as accessible and convenient as a filesystem, without being intrusive on the way your data is stored. This is not an official project of the Openstack community.





We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.