Alibaba JStorm is an enterprise fast and stable streaming process engine. It runs program up to 4x faster than Apache Storm. It is easy to switch from record mode to mini-batch mode. It is not only a streaming process engine. It means one solution for real time requirement, whole realtime ecosystem.
stream-processing batch-processing real-time data-processing distributedA curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources. Inspired by other awesome projects.
awesome-list stream-processing awesome listWallaroo is a fast, elastic data processing engine that rapidly takes you from prototype to production by eliminating infrastructure complexity. Wallaroo is a fast and elastic data processing engine that rapidly takes you from prototype to production.
stream-processing stream-processor stream-processing-engine pony-language framework apiFaust is a stream processing library, porting the ideas from Kafka Streams to Python. It is used at Robinhood to build high performance distributed systems and real-time data pipelines that process billions of events every day.
kafka-streams kafka asyncio distributed-systems stream-processingGoJay is a performant JSON encoder/decoder for Golang (currently the most performant, see benchmarks). It has a simple API and doesn't use reflection. It relies on small interfaces to decode/encode structures and slices.
decoder encoder perfomance json stream-decoder stream-processingVector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space. To get started, follow our quickstart guide or install Vector.
parser events router pipeline metrics vector logs stream-processing forwarder observabilityVector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space. To get started, follow our quickstart guide or install Vector.
parser events router pipeline metrics vector logs stream-processing forwarder observabilityStorm is a distributed real time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real time processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more.
real-time-computation analytics real-time stream-processing distributed-rpc data-processingSiddhi, high performing Complex Event Processing Engine
wso2 cep stream-processing complex-event-processing online-learningHazelcast Jet is a distributed computing platform built for high-performance stream processing and fast batch processing. It embeds Hazelcast In-Memory Data Grid (IMDG) to provide a lightweight, simple-to-deploy package that includes scalable in-memory storage. Hazelcast Jet performs parallel execution to enable data-intensive applications to operate in near real-time.
in-memory data-grid big-data stream-processing data-processing real-time streams batch-processingSigNoz is an opensource observability platform. SigNoz uses distributed tracing to gain visibility into your systems and powers data using Kafka (to handle high ingestion rate and backpressure) and Apache Druid (Apache Druid is a high performance real-time analytics database), both proven in the industry to handle scale.
metrics prometheus self-hosted tracing stream-processing druid kafka-streams observability distributed-tracing application-monitoring jaegertracing opentelemetry open-telemetry datadog-alternative newrelic-alternativeFluent Bit is a fast Log Processor and Forwarder, it allows to collect log events or metrics from different sources, process them and deliver them to different backends such as Fluentd, Elasticsearch, Splunk, DataDog, Kafka, New Relic, Azure services, AWS services, Google services, NATS, InfluxDB or any custom HTTP end-point. It also comes with full SQL Stream Processing capabilities: data manipulation and analytics using SQL queries.
fluentd logging data-collector fluent-bit log forwarder cloudnative log-forwarder logstash-alternative stream-processingYoMo is an open-source Streaming Serverless Framework for building Low-latency Edge Computing applications. Built atop QUIC Transport Protocol and Functional Reactive Programming interface. makes real-time data processing reliable, secure, and easy. Congratulations! You have done your first YoMo Stream Function.
iot networking serverless realtime stream-processing functional-reactive-programming low-latency quic metaverse 5g edge-computing geodistributedsystems edge-ai distributed-cloud edge-mesh metaverse-infrastructureHazelcast Jet is a distributed computing platform built for high-performance stream processing and fast batch processing. It embeds Hazelcast In Memory Data Grid (IMDG) to provide a lightweight package of a processor and a scalable in-memory storage. It supports distributed java.util.stream API support for Hazelcast data structures such as IMap and IList, Distributed implementations of java.util.{Queue, Set, List, Map} data structures highly optimized to be used for the processing
data-grid data-processing data-streaming in-memory batch-processing stream-processingThis repository contains code examples for Apache Kafka and the Confluent Platform.
confluent kafka kafka-streams kafka-clients example examples demo stream-processing ccloudApache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.
data-processing data-streaming batch-processing stream-processing distributed big-dataRu brings Ruby's expressiveness, cleanliness, and readability to the command line. It lets you avoid looking up pesky options in man pages and Googling how to write a transformation in bash that would take you approximately 1s to write in Ruby.
bash stream-processingThis project is currently in Beta. The API is ~95% stable so you can expect only minor breaking changes. For an introduction to Kasper and the motivation behind it, you can read our introductory blog post.
golang-library kafka stream-processingPlease refer to Wormhole用户手册.
wormhole stream-processing spark-streamingApache Hudi (pronounced Hoodie) stands for Hadoop Upserts Deletes and Incrementals. Hudi manages the storage of large analytical datasets on DFS (Cloud stores, HDFS or any Hadoop FileSystem compatible storage). As an organization, Hudi can help you build an efficient data lake, solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker.
bigdata stream-processing data-integration datalake spark apachehudi incremental-processing data-lake etl
We have large collection of open source products. Follow the tags from
Tag Cloud >>
Open source products are scattered around the web. Please provide information
about the open source projects you own / you use.
Add Projects.