goofys - a high-performance, POSIX-ish Amazon S3 file system written in Go

We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.

goofys - a high-performance, POSIX-ish Amazon S3 file system written in Go

Goofys allows you to mount an S3 bucket as a filey system. It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have a on disk data cache (checkout catfs), and consistency model is close-to-open.

https://github.com/kahing/goofys

Tags	s3-bucket s3 posix filesystem posix-ish aws-s3 cloud-storage fuse gcs google-cloud-storage openstack-swift fuse-filesystem
Implementation	Go
License	Apache
Platform	Windows MacOS Linux

Related Projects

Rclone - rsync for cloud storage like Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers like Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex.

golang sync encryption dropbox amazon-drive ftp s3 google-cloud-storage onedrive google-drive sftp webdav cloud-storage openstack-swift rclone hubic fuse-filesystem azure-blob backblaze-b2 file-synchronization file-sync

rclone - "rsync for cloud storage" - Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Cloudfiles, Google Cloud Storage, Yandex Files

See the home page for installation, usage, documentation, changelog and configuration walkthroughs. This is free software under the terms of MIT the license (check the COPYING file included in this package).

cloud-storage s3 openstack-swift google-drive azure-blob amazon-drive backblaze-b2 sftp ftp rclone hubic dropbox google-cloud-storage onedrive sync fuse-filesystem encryption webdav

svfs - The Swift Virtual File System

SVFS is a Virtual File System over Openstack Swift built upon fuse. It is compatible with hubiC, OVH Public Cloud Storage and basically every endpoint using a standard Openstack Swift setup. It brings a layer of abstraction over object storage, making it as accessible and convenient as a filesystem, without being intrusive on the way your data is stored. This is not an official project of the Openstack community.

cloud-storage openstack-swift hubic ovh fuse filesystem

mc - Minio Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage

Minio Client (mc) provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff, find etc. It supports filesystems and Amazon S3 compatible cloud storage service (AWS Signature v2 and v4).then use the mc config command.

storage google-cloud-storage aws-s3 filesystem s3 tools coreutils

S3 - Node.js implementation of a server handling the Amazon S3 protocol

Javascript

CloudServer (formerly S3 Server) is an open-source Amazon S3-compatible object storage server that is part of Zenko, Scality’s Open Source Multi-Cloud Data Controller.CloudServer provides a single AWS S3 API interface to access multiple backend data storage both on-premise or public in the cloud.

aws-s3 docker storage cloud s3-storage cloud-storage cloud-native object-storage nodejs aws-s3-alernative

s3proxy - Access other storage backends via the S3 API

Java

Access other storage backends via the S3 API

s3 proxy openstack-swift azure aws-s3 google-cloud-storage backblaze-b2 atmos

Cyberduck - Libre server and cloud storage browser for Mac and Windows

Java

Cyberduck is a libre server and cloud storage browser for Mac and Windows with support for FTP, SFTP, WebDAV, Amazon S3, OpenStack Swift, Backblaze B2, Microsoft Azure & OneDrive, Google Drive and Dropbox.

storage-browser ftp webdav aws-s3 google-drive dropbox one-drive backblaze

chartmuseum - Helm Chart Repository with support for Amazon S3, Google Cloud Storage, Microsoft Azure Blob Storage, Alibaba Cloud OSS Storage, and Openstack Object Storage

ChartMuseum is an open-source Helm Chart Repository written in Go (Golang), with support for cloud storage backends, including Google Cloud Storage, Amazon S3, Microsoft Azure Blob Storage, Alibaba Cloud OSS Storage and Openstack Object Storage. Works as a valid Helm Chart Repository, and also provides an API for uploading new chart packages to storage etc.

helm kubernetes chartmuseum

cloudserver - Zenko CloudServer, an open-source Node

Javascript

CloudServer (formerly S3 Server) is an open-source Amazon S3-compatible object storage server that is part of Zenko, Scality’s Open Source Multi-Cloud Data Controller. CloudServer provides a single AWS S3 API interface to access multiple backend data storage both on-premise or public in the cloud.

aws-s3 docker multiple-backends storage cloud s3-storage cloud-storage cloud-native object-storage nodejs s3 server

meteor-slingshot - Upload files directly to AWS S3, Google Cloud Storage and others in meteor

Javascript

Direct and secure file-uploads to AWS S3, Google Cloud Storage and others. There are many many packages out there that allow file uploads to S3, Google Cloud and other cloud storage services, but they usually rely on the meteor apps' server to relay the files to the cloud service, which puts the server under unnecessary load.

Minio - Open source object storage server compatible with Amazon S3 APIs

Minio is an object storage server, It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB. Minio server is light enough to be bundled with the application stack, similar to NodeJS, Redis and MySQL.

aws-s3 storage cloud s3-storage cloud-storage object-storage cloud-native object-store file-storage blob

moosefs - MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System

MooseFS is a Petabyte Open Source Network Distributed File System. It is easy to deploy and maintain, fault tolerant, highly performing, easily scalable, POSIX compliant. MooseFS Linux Client uses FUSE. MooseFS macOS Client uses FUSE for macOS.

dfs software-defined-storage posix filesystem file-system distributed-file-system clustering distributed-storage distributed-computing fuse big-data snapshot storage-tiering high-availability scalability storage moosefs hadoop posix-compliant

Ceph - Distributed Object Store

Ceph provides seamless access to objects using native language bindings or radosgw, a REST interface that’s compatible with applications written for S3 and Swift. Ceph’s RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster. Ceph provides a POSIX-compliant network file system that aims for high performance, large data storage, and maximum compatibility with legacy applications.

object-store block-device file-system distributed block filesystem distributed-filesystem distributed-storage

lakeFS - Git-like capabilities for your object storage

lakeFS is an open source layer that delivers resilience and manageability to object-storage based data lakes. With lakeFS you can build repeatable, atomic and versioned data lake operations - from complex ETL jobs to data science and analytics.

apache-spark aws-s3 google-cloud-storage data-engineering data-lake object-storage datalake hadoop-filesystem data-quality data-versioning azure-blob-storage apache-sparksql git-for-data lakefs datalakes

yas3fs - YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3

Python

YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster through the YAS3FS Console project. This is a personal project. No relation whatsoever exists between this project and my employer.

Quickwit - Fast and cost-efficient distributed search engine for immutable data

Rust

Quickwit is a distributed search engine built from the ground up to offer cost-efficiency and high reliability. By mere mortals for mere mortals, Quickwit's architecture is as simple as possible. Quickwit is written in Rust and built on top of the mighty tantivy library. It designed to index big datasets straight from object storage like AWS S3 in a stateless manner.

search-engine distributed-systems searchengine full-text-search

multer-s3 - multer storage engine for amazon s3

Javascript

Streaming multer storage engine for AWS S3. This project is mostly an integration piece for existing code samples from Multer's storage engine documentation with s3fs as the substitution piece for file system. Existing solutions I found required buffering the multipart uploads into the actual filesystem which is difficult to scale.

multer s3 amazon aws

diskover - File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch

Python

diskover is an open source file system crawler and disk space usage software that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organize files and system administrators are able to manage storage infrastructure, efficiently provision storage, monitor and report on storage use, and effectively make decisions about new infrastructure purchases. As the amount of file data generated by business' continues to expand, the stress on expensive storage infrastructure, users and system administrators, and IT budgets continues to grow.

elasticsearch crawler filesystem-visualization filesystem-analysis filesystem-indexer disk-space disk-usage storage-analytics storage filesystem file-indexing duplicatefilefinder metadata duplicate-files botnet file-tagging analytics aws-s3 tree-walker

s3cmd - Official s3cmd repo -- Command line tool for managing Amazon S3 and CloudFront services

Python

S3cmd requires Python 2.6 or newer. Python 3+ is also supported starting with S3cmd version 2. S3cmd (s3cmd) is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects. It is best suited for power users who are familiar with command line programs. It is also ideal for batch scripts and automated backup to S3, triggered from cron, etc.

MyDrive - Open Source Cloud Server (Similar To Google Drive)

Javascript

MyDrive is an Open Source Cloud Server (Similar To Google Drive), the service uses mongoDB to store file/folder metadata, and supports multiple databases to store the file chunks, such as Amazon S3, the Filesystem, or just MongoDB. MyDrive is built using Node.js, and Typescript. The service now even supports Docker images!

cloud-storage google-drive-alternative drive file-sharing dropbox-alternative