s3-write-stream - Pipe data straight to an S3 key of your choice

  •        594

Pipe data straight to an S3 key of your choice.This is a writeable stream that takes data and uploads it to Amazon S3 using its multipart upload API. This is ideal for handling generated content without needing to know the content's length ahead of time, and without resorting to file system hacks or buffering everything before the upload.

https://github.com/hughsk/s3-write-stream

Dependencies:

aws-sdk : ~2.0.0-rc8
through2 : ~0.4.0
backoff : ~2.3.0
bl : ~0.7.0

Tags
Implementation
License
Platform

   




Related Projects

s3-upload-stream - A Node.js module for streaming data to Amazon S3 via the multipart upload API

  •    Javascript

A pipeable write stream which uploads to Amazon S3 using the multipart file upload API. NOTE: This module is deprecated after the 2.1.0 release of the AWS SDK on Dec 9, 2014, which added S3.upload(). I highly recommend switching away from this module and using the official method supported by AWS.

node-s3-client - high level amazon s3 client for node.js

  •    Javascript

See also the companion CLI tool which is meant to be a drop-in replacement for s3cmd: s3-cli.This contains a reference to the aws-sdk module. It is a valid use case to use both this module and the lower level aws-sdk module in tandem.

smart_open - Utils for streaming large files (S3, HDFS, gzip, bz2...)

  •    Python

There are a few optional keyword arguments that are useful only for S3 access. These are both passed to boto.s3_connect() as keyword arguments. The S3 reader supports gzipped content, as long as the key is obviously a gzipped file (e.g. ends with ".gz").

django-s3direct - Add direct uploads to S3 with a progress bar to file input fields

  •    Javascript

Add direct uploads to AWS S3 functionality with a progress bar to file input fields. With 1.0.0 supporting multipart-upload, most of the internals have been changed, a new endpoint has been added, and support has been dropped for old style positional settings. There are also new requirements to allow GET and HEAD cross-origin requests to S3, as well as the ETAG header. Django compatibility has been raised to >=1.8.

skipper - Streaming multi-uploads for Sails/Express - supports disk, S3, gridfs, and custom file adapters

  •    Javascript

Skipper makes it easy to implement streaming file uploads to disk, S3, or any supported file upload adapters.The following example assumes skipper is already installed as the body parser in your Express or Sails app. It receives one or more files from a file parameter named avatar using the default, built-in file adapter (skipper-disk). This streams the file(s) to the default upload directory .tmp/uploads/ on the server's local disk.


multer-s3 - multer storage engine for amazon s3

  •    Javascript

Streaming multer storage engine for AWS S3. This project is mostly an integration piece for existing code samples from Multer's storage engine documentation with s3fs as the substitution piece for file system. Existing solutions I found required buffering the multipart uploads into the actual filesystem which is difficult to scale.

gulp-awspublish - gulp plugin to publish files to amazon s3

  •    Javascript

Note: If you follow the aws-sdk suggestions for providing your credentials you don't need to pass them in to create the publisher. create a through stream, that gzip file and add Content-Encoding header.

bucket-stream - Find interesting Amazon S3 Buckets by watching certificate transparency logs.

  •    Python

Find interesting Amazon S3 Buckets by watching certificate transparency logs. This tool simply listens to various certificate transparency logs (via certstream) and attempts to find public S3 buckets from permutations of the certificates domain name.

busboy - A streaming parser for HTML form data for node.js

  •    Javascript

A node.js module for parsing incoming HTML form data.file(< string >fieldname, < ReadableStream >stream, < string >filename, < string >transferEncoding, < string >mimeType) - Emitted for each new file form field found. transferEncoding contains the 'Content-Transfer-Encoding' value for the file stream. mimeType contains the 'Content-Type' value for the file stream.

streaming-benchmarks - Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink,

  •    Java

Code licensed under the Apache 2.0 license. See LICENSE file for terms.At Yahoo we have adopted Apache Storm as our stream processing platform of choice. But that was in 2012 and the landscape has changed significantly since then. Because of this we really want to know what Storm is good at, where it needs to be improved compared to other systems, and what its limitations are compared to other tools so we can recommend the best tool for the job to our customers. To do this we started to look for stream processing benchmarks that we could use to do this evaluation, but all of them ended up lacking in several fundamental areas. Primarily they did not test anything close to a read world use case, so we decided to write a simple one. This is the first round of these tests. The tool here is not polished and only covers three tools and one specific use case. We hope to expand this in the future in terms of the tools tested, the variety of processing tested, and the metrics gathered.

hstream - The database built for IoT streaming data storage and real-time stream processing.

  •    Haskell

The database built for IoT streaming data storage and real-time stream processing. By subscribing to streams in HStreamDB, any update of the data stream will be pushed to your apps in real-time, and this promotes your apps to be more responsive.

react-native-aws3 - Pure JavaScript React Native library for uploading to AWS S3

  •    Javascript

React Native AWS3 is a module for uploading files to S3. Unlike other libraries out there, there are no native dependencies. Upload a file to S3.

s3-plugin-webpack - Uploads files to s3 after complete

  •    Javascript

I notice a lot of people are setting the directory option when the files are part of their build. Please don't set directory if you're uploading your build. Using the directory option reads the files after compilation to upload instead of from the build process. include and exclude rules behave similarly to Webpack's loader options. In addition to a RegExp you can pass a function which will be called with the path as its first argument. Returning a truthy value will match the rule. You can also pass an Array of rules, all of which must pass for the file to be included or excluded.

Hosebird client - A Java HTTP client for consuming Twitter's Streaming API

  •    Java

A Java HTTP client for consuming Twitter's Streaming API. It has GZip support, OAuth support, Partitioning support, Automatic reconnections with appropriate backfill counts, Access to raw bytes payload, Proper backoffs/retry schemes, Relevant statistics/events, Control stream support for sitestreams.

Flajaxian S3 Amazon Service Uploader

  •    CSharp

Flajaxian S3 Amazon Service Uploader is adapter for the Flajaxian FileUploader .NET web control designed for asynchronous file upload to an Amazon Service of multiple files at the same time, without a page post back and with a progress bar indicating the current upload progres...

tinys3 - A simple python S3 upload library. Inspired by requests

  •    Python

A simple Python S3 upload library. Inspired by one of my favorite packages, requests. tinys3 is used at Smore to upload more than 1.5 million keys to S3 every month.

C# Library and Code for Amazon S3

  •    

An advanced C# library for interfacing with the Amazon S3 system. Among its powerful features are: - Full support for data streaming. No need to load data into memory before sending to S3. - Data encryption. - Thread safety and live statistics. Perform multiple simultaneous up...

Streamlabs OBS - Streaming software built on OBS and Electron

  •    Typescript

Simple, powerful, and efficient live streaming software built on Electron and OBS. Broadcast your live stream to Twitch, YouTube, or Facebook. Choose the sources that appear in your live stream or recorded video. Effortlessly save replays of your live stream or replay clips for your viewers.

Stream-2-Stream

  •    Java

Stream-2-Stream's multicast+ is the next generation of streaming. Multicast+ is more efficient and requires less bandwidth than direct streaming (e.g. shoutcast/icecast). Stream-2-Stream stations can be set up without paying a fortune for bandwidth.

JS3tream

  •    Java

JS3tream is a simple STDIO stream director between your machine an Amazon S3. Allowing you to stream limitless information to and from Amazon. Great for backups using TAR or ZIP. Main Site http://js3tream.sourceforge.net






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.