11 best open source big data projects.

We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.

Displaying 1 to 11 from 11 results

poseidon - A search engine which can hold 100 trillion lines of log data.

A search engine which can hold 100 trillion lines of log data.

poseidon search-engine big-data map-reduce

pachyderm - Reproducible Data Science at Scale!

Pachyderm is a tool for production data pipelines. If you need to chain together data scraping, ingestion, cleaning, munging, wrangling, processing, modeling, and analysis in a sane way, then Pachyderm is for you. If you have an existing set of scripts which do this in an ad-hoc fashion and you're looking for a way to "productionize" them, Pachyderm can make this easy for you. Install Pachyderm locally or deploy on AWS/GCE/Azure in about 5 minutes.

pachyderm docker analytics big-data containers distributed-systems kubernetes data-science data-analysis

TrailDB - Efficient tool for storing and querying series of events

TrailDB is a library, implemented in C, which allows you to query series of events at blazing speed. TrailDB is also optimized for speed of development: Use its simple API with your favorite language, in your favorite environment. TrailDB's secret sauce is data compression. It leverages predictability of time-based data to compress your data to a fraction of its original size. In contrast to traditional compression, you can query the encoded data directly, decompressing only the parts you need.

database data-analytics event-data big-data time-series time-series-database

dvid - Distributed, Versioned, Image-oriented Dataservice

Status: In production use at Janelia. See wiki page for outside lab use of DVID. See the DVID Wiki for more information including installation and examples of use.

dataservice image-storage http-service connectomics big-data neuroscience key-value

warp - Convert and analyze large data sets at light speed, on Mac and iOS.

Swift

Warp allows you to convert and analyze (very) large databases with ease at the speed of light. In Warp, you work on a small subset of the data, after which Warp repeats your actions on the entire dataset. Unlike most data analysis apps, you do not have to type any codes in Warp. Effortlessly juggle around data between files and databases by simply dragging-and-dropping! Load CSV files into MySQL or transfer a PostgreSQL table to a RethinkDB table by just dragging one to the other.

rethinkdb mysql postgresql data-analysis big-data sqlite

hazelcast-go-client - Hazelcast IMDG Go Client

Go client implementation for Hazelcast, the open source in-memory data grid. Go client is implemented using the Hazelcast Open Binary Client Protocol.

hazelcast in-memory datagrid big-data clustering scalability distributed caching imdg hazelcast-client golang-library

k8s-ingress-claim - An admission control policy that safeguards against accidental duplicate claiming of Hosts/Domains

k8s-ingress-claim provides an admission control policy that safeguards against accidental duplicate claiming of Hosts/Domains by ingresses that have already been claimed by existing ingresses. This is implemented as an External Admission Webhook with the k8s-ingress-claim service running as a deployment on each cluster.

kubernetes admission-controllers ingress big-data