jstorm - Enterprise Stream Process Engine

•    Java

Alibaba JStorm is an enterprise fast and stable streaming process engine. It runs program up to 4x faster than Apache Storm. It is easy to switch from record mode to mini-batch mode. It is not only a streaming process engine. It means one solution for real time requirement, whole realtime ecosystem.

awesome-streaming - a curated list of awesome streaming frameworks, applications, etc

A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources. Inspired by other awesome projects.

wallaroo - Build and scale real-time data applications as easily as writing a Python script

•    Pony

Wallaroo is a fast, elastic data processing engine that rapidly takes you from prototype to production by eliminating infrastructure complexity. Wallaroo is a fast and elastic data processing engine that rapidly takes you from prototype to production.

faust - Python Stream Processing

•    Python

Faust is a stream processing library, porting the ideas from Kafka Streams to Python. It is used at Robinhood to build high performance distributed systems and real-time data pipelines that process billions of events every day.

gojay - fastest JSON encoder/decoder with powerful stream API for Golang

•    Go

GoJay is a performant JSON encoder/decoder for Golang (currently the most performant, see benchmarks). It has a simple API and doesn't use reflection. It relies on small interfaces to decode/encode structures and slices.

vector - A high-performance, highly reliable, observability data pipeline.

•    Rust

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space. To get started, follow our quickstart guide or install Vector.

vector - A high-performance observability data pipeline.

•    Rust

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space. To get started, follow our quickstart guide or install Vector.

Apache Storm - Distributed and fault-tolerant realtime computation

•    Java

Storm is a distributed real time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real time processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more.

siddhi - Stream Processing and Complex Event Processing Engine

•    Java

Siddhi, high performing Complex Event Processing Engine

Hazelcast Jet - A general purpose distributed data processing engine, built on top of Hazelcast.

•    Java

Hazelcast Jet is a distributed computing platform built for high-performance stream processing and fast batch processing. It embeds Hazelcast In-Memory Data Grid (IMDG) to provide a lightweight, simple-to-deploy package that includes scalable in-memory storage. Hazelcast Jet performs parallel execution to enable data-intensive applications to operate in near real-time.

Signoz - Open-source Observability platform and an alternative to DataDog, NewRelic

•    Javascript

SigNoz is an opensource observability platform. SigNoz uses distributed tracing to gain visibility into your systems and powers data using Kafka (to handle high ingestion rate and backpressure) and Apache Druid (Apache Druid is a high performance real-time analytics database), both proven in the industry to handle scale.

Fluent Bit - Fast and Lightweight Logs and Metrics processor

•    C

Fluent Bit is a fast Log Processor and Forwarder, it allows to collect log events or metrics from different sources, process them and deliver them to different backends such as Fluentd, Elasticsearch, Splunk, DataDog, Kafka, New Relic, Azure services, AWS services, Google services, NATS, InfluxDB or any custom HTTP end-point. It also comes with full SQL Stream Processing capabilities: data manipulation and analytics using SQL queries.

yomo - 🦖 Serverless Streaming Framework for Low-latency Edge Computing applications, running atop QUIC protocol, as Metaverse infrastructure, engaging 5G technology

•    Go

YoMo is an open-source Streaming Serverless Framework for building Low-latency Edge Computing applications. Built atop QUIC Transport Protocol and Functional Reactive Programming interface. makes real-time data processing reliable, secure, and easy. Congratulations! You have done your first YoMo Stream Function.

Hazelcast Jet - Distributed data processing engine, built on top of Hazelcast

•    Java

Hazelcast Jet is a distributed computing platform built for high-performance stream processing and fast batch processing. It embeds Hazelcast In Memory Data Grid (IMDG) to provide a lightweight package of a processor and a scalable in-memory storage. It supports distributed java.util.stream API support for Hazelcast data structures such as IMap and IList, Distributed implementations of java.util.{Queue, Set, List, Map} data structures highly optimized to be used for the processing

examples - Demo applications and code examples for Confluent Platform and Apache Kafka

•    Java

This repository contains code examples for Apache Kafka and the Confluent Platform.

Apache Beam - Unified model for defining both batch and streaming data-parallel processing pipelines

•    Java

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.

ru - Ruby in your shell!

•    Ruby

Ru brings Ruby's expressiveness, cleanliness, and readability to the command line. It lets you avoid looking up pesky options in man pages and Googling how to write a transformation in bash that would take you approximately 1s to write in Ruby.

kasper - Kasper is a lightweight library for processing Kafka topics.

•    Go

This project is currently in Beta. The API is ~95% stable so you can expect only minor breaking changes. For an introduction to Kasper and the motivation behind it, you can read our introductory blog post.

wormhole - Wormhole is a SPaaS (Stream Processing as a Service) Platform

•    Javascript

Please refer to Wormhole用户手册.

Apache Hudi - Streaming Data Lake Platform

•    Java

Apache Hudi (pronounced Hoodie) stands for Hadoop Upserts Deletes and Incrementals. Hudi manages the storage of large analytical datasets on DFS (Cloud stores, HDFS or any Hadoop FileSystem compatible storage). As an organization, Hudi can help you build an efficient data lake, solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker.