Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

awesome-streaming

a curated list of awesome streaming frameworks, applications, etc
https://github.com/manuzhang/awesome-streaming

Last synced: 2 days ago
JSON representation

  • Table of Contents

    • Applications and Tools

      • Streamdal - badge]</sub> <sub>![TypeScript][language-typescript]</sub> - Code-native data privacy controls for detecting PII in application data flows.

    • Benchmark

      • storm-perf-test - a simple storm performance/stress test.
      • streaming-benchmarks - Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink, etc.
      • flotilla - Automated message queue orchestration for scaled-up benchmarking.

    • Closed Source

      • Amazon Kinesis Streams - real-time, fully managed and scalable data stream engine provided by AWS.
      • concord - a distributed stream processing framework built in C++ on top of Apache.
      • jubatus - distributed processing framework and streaming machine learning library.
      • millwheel - framework for building low-latency data-processing applications that is widely used at Google.
      • NVIDIA Deep Stream - a platform for real-time image, video and audio processing, preferably using on edge devices or cloud.
      • Azure Stream Analytics - time, data stream engine provided by Microsoft Azure.
      • IBM Streams - platform for distributed processing and real-time analytics. Provides toolkits for advanced analytics like geospatial, time series, etc. out of the box.
      • Azure Stream Analytics - time, data stream engine provided by Microsoft Azure.
      • jubatus - distributed processing framework and streaming machine learning library.
      • Cloud Dataflow - Google's managed stream and batch data processing engine. Supports running Beam pipelines.
      • concord - a distributed stream processing framework built in C++ on top of Apache.

    • Data Integration and Pipelines

      • Apache Flume - java]</sub> - Distributed service for collecting, aggregating, and moving large amounts of log-like data.
      • Camus - badge]</sub> <sub>![Java][language-java]</sub> - LinkedIn's retired, previous-generation Kafka-to-HDFS pipeline.
      • faucet-stream - rust]</sub> - Config-driven platform and embeddable library for ETL, CDC, and streaming data movement.
      • Redpanda Connect - go]</sub> - Declarative stream processor for moving, enriching, transforming, and filtering data between sources and sinks.
      • yasdb - go]</sub> - Durable Streams protocol server backed by SlateDB object storage, with offset-based catch-up reads and SSE or long-poll tailing.

    • Data Pipeline

      • LogDevice - a high-performant distributed system by Facebook for streaming and storing sequential data, using a log structure.
      • brooklin - a distributed system intended for streaming data between various heterogeneous source and destination systems with high reliability and throughput at scale from Linkedin (replaced databus).
      • Apache Kafka - distributed, partitioned, replicated commit log service, which provides the functionality of a messaging system, but with a unique design.
      • Apache RocketMQ - distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.
      • databus - Linkedin's source-agnostic distributed change data capture system.
      • fluvio - Real-time programmable data streaming platform with in-line computation capabilities.
      • Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms.
      • metaq - Taobao's high available, high performance distributed messaging system
      • NATS streaming - fast disk-backed messaging solution
      • nsq - realtime distributed messaging platform designed to operate at scale, handling billions of messages per day.
      • Redpanda - Redpanda is Kafka compatible, ZooKeeper-free, JVM-free and source available.
      • RudderStack - an open source customer data infrastructure (segment, mparticle alternative).
      • suro - data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data.
      • StreamSets Data Collector - continuous big data ingestion infrastructure that reads from and writes to a large number of end-points, including S3, JDBC, Hadoop, Kafka, Cassandra and many others.
      • AutoMQ - cloud-first alternative to Kafka by decoupling durability to S3 and EBS. 100% Kafka compatible. 10x cost-effective. Autoscale in seconds. Single-digit ms latency.
      • flume - distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.
      • Apache Pulsar - distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
      • brooklin - a distributed system intended for streaming data between various heterogeneous source and destination systems with high reliability and throughput at scale from Linkedin (replaced databus).
      • camus - Linkedin's Kafka -> HDFS pipeline.
      • Bruin - End-to-end data pipeline tool combining ingestion from 50+ sources, SQL/Python transformations, and built-in data quality checks in a single CLI.
      • ingestr - CLI tool to copy data between any source and destination with a single command. Supports 50+ connectors including databases, SaaS apps, and data warehouses.
      • faucet-stream - Config-driven data-movement platform for ETL, CDC, and streaming: pluggable source and sink connectors, run declaratively from YAML by a single binary or embedded as a Rust library.

    • DSL

      • Apache Beam - unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs), open sourced by Google.
      • coast - a DSL that builds DAGs on top of Samza and provides exactly-once semantics.
      • Esper - component for complex event processing (CEP) and event series analysis.
      • summingbird - library that lets you write MapReduce programs that look like native Scala or Java collection transformations and execute them on a number of well-known distributed MapReduce platforms, including Storm and Scalding.
      • Streamparse - lets you run Python code against real-time streams of data via Apache Storm.

    • Engines and Platforms

      • Aeron - java-cpp]</sub> - Reliable UDP unicast, multicast, and IPC message transport.
      • Amazon Kinesis Data Streams - Fully managed service for ingesting and processing real-time data streams on AWS.
      • Apache Pulsar - java]</sub> - Distributed pub-sub messaging and event streaming platform.
      • Apache StreamPipes - java-python-typescript]</sub> - Self-service industrial IoT platform for connecting, analyzing, and exploring data streams.
      • AthenaX - badge]</sub> <sub>![Java][language-java]</sub> - Uber's retired SQL-based streaming analytics platform.
      • Azure Stream Analytics - sql]</sub> - Fully managed service for serverless real-time analytics in the cloud and at the edge.
      • eKuiper - go]</sub> - Lightweight data stream processing engine for resource-constrained IoT edge devices.
      • Fluvio - rust-wasm]</sub> - Composable, stateful data streaming system with programmable in-line computation.
      • IBM Streams - badge]</sub> <sub>![Python/Java/Scala][language-python-java-scala]</sub> - Discontinued platform for distributed stream processing and real-time analytics.
      • LogDevice - badge]</sub> <sub>![C++][language-cpp]</sub> - Facebook's archived distributed storage system for sequential data.
      • S4 - badge]</sub> <sub>![Java][language-java]</sub> - Retired distributed platform for processing continuous unbounded data streams.
      • Tigon - badge]</sub> <sub>![C++/Java][language-cpp-java]</sub> - High-throughput real-time stream processing framework built on Hadoop and HBase.
      • Trill - badge]</sub> <sub>![C#][language-csharp]</sub> - Single-node query processor for temporal and streaming data from Microsoft Research.
      • Wally - badge]</sub> <sub>![Pony][language-pony]</sub> - Distributed stream processing framework formerly named Wallaroo.
      • Wingfoil - rust-python-typescript]</sub> - Graph-based stream processing engine for latency-critical systems.
      • SQLFlow - go-python]</sub> - Stream processing engine that runs DuckDB SQL over Kafka, WebSocket, and webhook streams.

    • IoT

      • sensorbee - lightweight stream processing engine for IoT.
      • Apache Edgent - a programming model and runtime that enables continuous streaming analytics on gateways and edge devices which can work with centralized systems to provide efficient and timely analytics across the whole IoT ecosystem: from the center to the edge, opens sourced by IBM.

    • Libraries, SDKs, and Programming Models

      • Akka - scala-java]</sub> - Toolkit and runtime for concurrent, distributed, resilient applications on the JVM.
      • Apache Edgent - badge]</sub> <sub>![Java][language-java]</sub> - Retired programming model and runtime for streaming analytics on gateways and edge devices.
      • Apache Pekko - scala-java]</sub> - Open-source toolkit for concurrent, distributed, resilient applications, forked from Akka 2.6.
      • CapyMOA - python-java]</sub> - Machine learning library for data streams with a Python API and MOA backend.
      • DataSketches - java]</sub> - Apache library of stochastic streaming algorithms known as sketches.
      • FastStream - python]</sub> - Asynchronous framework for building event-driven applications with Kafka, RabbitMQ, NATS, Redis, and MQTT.
      • FS2 - scala]</sub> - Compositional streaming I/O library for Scala.
      • Jubatus - badge]</sub> <sub>![C++][language-cpp]</sub> - Distributed processing framework and library for online machine learning.
      • MediaPipe - cpp-python-java-typescript]</sub> - Cross-platform, customizable ML solutions for live and streaming media.
      • Numalogic - python]</sub> - Collection of machine learning models and tools for anomaly detection and forecasting on operational time-series data.
      • Pulsar - badge]</sub> <sub>![Python][language-python]</sub> - Actor-based event-driven concurrency framework for Python.
      • Streamparse - badge]</sub> <sub>![Python][language-python]</sub> - Python API, command-line tools, and topology DSL for Apache Storm.
      • Tributary - python]</sub> - Python library for constructing synchronous, reactive, and lazy dataflow graphs.
      • WindFlow - cpp]</sub> - Data stream processing parallel library for multi-core CPUs and GPUs.

    • Online Machine Learning

      • Apache Samoa - distributed streaming machine learning (ML) framework that contains a programing abstraction for distributed streaming ML algorithms.
      • River - online machine learning library.
      • streamDM - mining Big Data streams using Spark Streaming from Huawei.
      • StreamingBandit - Provides a webserver to quickly setup and evaluate possible solutions to contextual multi-armed bandit (cMAB) problems.
      • StormCV - enables the use of Apache Storm for video processing by adding computer vision (CV) specific operations and data model.
      • trident-ml - realtime online machine learning library based on Trident.
      • yurita - Anomaly detection framework built on Spark Structured Streaming from Paypal.
      • DataSketches - sketches library from Yahoo!.

    • Readings

    • Streaming Application

      • javactrl-kafka - An application of a stateful stream processing for workflow as Java code (microservices orchestration, business process automation, and more).
      • straw - A platform for real-time streaming search.
      • Zilla - Cross-platform, API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT and the native Kafka protocol.
      • storm-crawler - Web crawler SDK based on Apache Storm.
      • storm-crawler - Web crawler SDK based on Apache Storm.

    • Streaming Engine

      • Apache Apex - unified platform for big data stream and batch processing.
      • Apache Flink - system for high-throughput, low-latency data stream processing that supports stateful computation, data-driven windowing semantics and iterative stream processing.
      • Apache Heron (incubating) - a realtime, distributed, fault-tolerant stream processing engine from Twitter.
      • Apache Samza - distributed stream processing framework that build on Kafka(messaging, storage) and YARN(fault tolerance, processor isolation, security and resource management).
      • Apache Spark Streaming - makes it easy to build scalable fault-tolerant streaming applications.
      • Apache Storm - distributed real-time computation system. Storm is to stream processing what Hadoop is to batch processing.
      • Bytewax - data parallel, distributed, stateful stream processing framework.
      • Faust - stream processing library, porting the ideas from Kafka Streams to Python
      • Gearpump - lightweight real-time distributed streaming engine built on Akka.
      • Hazelcast Jet - A general purpose distributed data processing engine, built on top of Hazelcast.
      • hailstorm - distributed stream processing with exactly-once semantics based on Storm.
      • Maki Nage - A stream processing framework for data scientists, based on Kafka and ReactiveX.
      • mantis - Netflix's platform to build an ecosystem of realtime stream processing applications
      • mupd8(muppet) - mapReduce-style framework for processing fast/streaming data.
      • Numaflow - Kubernetes native stream processing platform with language agnostic framework. Scalable and cost-efficient
      • Onyx - Distributed, masterless, high performance, fault tolerant data processing.
      • Pathway - The fastest data processing engine supporting unified workflows for batch, streaming data, and LLM applications.
      • SABER - Window-Based Hybrid CPU/GPU Stream Processing Engine.
      • Scramjet Cloud Platform - data processing engine for running multiple data processing apps (sequences) written in Python, JavaScript or TypeScript
      • SPQR - dynamic framework for processing high volumn data streams through pipelines.
      • Teknek - Simple elegant stream processing with interactive prototying shell SOL (Stream Operator Language)
      • Trill - Trill is a high-performance one-pass in-memory streaming analytics engine from Microsoft Research.
      • LightSaber - Multi-core Window-Based Stream Processing Engine. LightSaber uses code generation for efficient window aggregation.
      • HStreamDB - The streaming database built for IoT data storage and real-time processing.
      • WindFlow - A C++17 Data Stream Processing Parallel Library for Multicores and GPUs
      • Arroyo - a distributed stream processing engine. Supports SQL and Rust pipelines. Scales up to millions of events per second. Supports stateful operations like windows and joins, state checkpointing for fault-tolerance and recovery of pipelines. Uses the Timely Dataflow model.
      • RisingWave - A PostgreSQL-compatible streaming database that is designed to build event-driven applications, real-time ETL pipelines, continuous analytics services, and feature stores for AI applications. It excels in extracting fresh and consistent insights from real-time event streams, database CDC, and time series data within sub-seconds. It unifies streaming and batch processing, enabling users to ingest, join, and analyze both live and historical data at a cloud scale.
      • CocoIndex - ETL framework to build fresh index for AI, with realtime incremental updates.
      • Wallaroo - A fast, stream-processing framework. Wallaroo makes it easy to react to data in real-time. By eliminating infrastructure complexity, going from prototype to production has never been simpler.
      • ArkFlow - High-performance Rust stream processing engine, providing powerful data stream processing capabilities, supporting multiple input/output sources and processors.
      • NebulaStream - High-performance, general-purpose, end-to-end data-management system for cloud-edge-sensor environments.
      • AthenaX - Uber's Stream Analytics Framework used in production
      • s4 - general-purpose, distributed, scalable, fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.
      • tigon - high throughput real-time streaming processing framework built on Hadoop and HBase.
      • Kuiper - An edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices.
      • beava - Single-binary feature server. Push events over HTTP or TCP, query fresh per-entity counters and aggregates inline, no broker in between. For fraud, recommendations, LLM guardrails, and in-product analytics.

    • Streaming Library

      • Streamiz - a .Net Stream Processing Library for Apache Kafka
      • Daggy - real-time streams aggregation and catching.
      • monix - high-performance Scala / Scala.js library for composing asynchronous and event-based programs.
      • Quix Streams - a streaming library originally designed for the McLaren Formula 1 racing team that can process high volumes of time-series data with up to nanosecond precision using Apache Kafka as a message broker.
      • Scramjet Node.js - [Node.js] functional reactive stream programming framework written on top of Node.js object streams + [the legacy Scramjet.js version](https://github.com/scramjetorg/scramjet)
      • Scramjet Python - [Python] functional reactive stream programming framework written from scratch operating on object, string and buffer streams.
      • Scramjet C++ - [C++] functional reactive stream programming framework written on top of Node.js object streams.
      • Streamline - Stream Analytics Framework by Hortonworks, designed as a wrapper around existing streaming solutions like Storm. Aimed to allow users to drag-and-drop streaming components to focus on business logic.
      • StreamAlert - Airbnb's Real-time Data Analysis and Alerting.
      • Swave - A lightweight Reactive Streams Infrastructure Toolkit for Scala.
      • Streamz - A lightweight library for building pipelines to manage continuous streams of data; supports complex pipelines that involve branching, joining, flow control, feedback, back pressure, and so on.
      • Stream Ops - A fully embeddable data streaming engine and stream processing API for Java.
      • Substation - Substation is a cloud native data pipeline and transformation toolkit written in Go.
      • SwimOS - A framework for building real-time streaming data processing applications written in Rust.
      • YoMo - An open source Streaming Serverless Framework for building Low-latency Geo-distributed system. YoMo Built atop [QUIC Transport Protocol](https://en.wikipedia.org/wiki/QUIC) and Functional Reactive Programming interface.
      • Streamiz - a .Net Stream Processing Library for Apache Kafka
      • Benthos - Benthos is a high performance and resilient message streaming service, able to connect various sources and sinks and perform arbitrary actions, transformations and filters on payloads
      • Tributary - A python library for constructing dataflow graphs. Supports synchronous, reactive data streams built using python generators that mimic complex event processors, as well as lazily-evaluated acyclic graphs and functional currying streams.
      • FS2(prev. 'Scalaz-Stream') - Compositional, streaming I/O library for Scala.
      • Mediapipe - Cross-platform, customizable ML solutions for live and streaming media.
      • Kzmlabs StateFun Actors - Stateful actors on Apache Flink 2.x with durable per-key state, exactly-once messaging, and Kafka/Kinesis I/O. Continuation of Apache Stateful Functions on Flink 2.2 + Java 21.

    • Streaming SQL

      • pipelinedb - An open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.
      • squall - Squall executes SQL queries on top of Storm for doing online processing.
      • StreamCQL - Continuous Query Language on RealTime Computation System.
      • ksqlDB - A cloud-native, source-available [database](https://ksqldb.io/) purpose-built for stream processing applications
      • Siddhi - A cloud native Streaming and Complex Event Processing engine that understands Streaming SQL queries in order to capture events from diverse data sources, process them, detect complex conditions, and publish output to various endpoints in real time.
      • Proton - A unified streaming and historical data analytics database in a single binary, powered by ClickHouse.
      • laminardb - A open-source streaming SQL engine, embed it in your process or run it standalone (single binary). Supports tumbling, hopping, and session windows, watermarks, ASOF joins, stream-to-stream temporal joins, and materialized views. Connectors include Kafka, NATS, WebSocket, Postgres CDC, Apache Iceberg, and Delta Lake. Built on Apache Arrow and DataFusion

    • Toolkit

      • akka - toolkit and runtime for building highly concurrent, distributed, and resilient message-driven application on the JVM.
      • StreamFlow - stream processing tool designed to help build and monitor processing workflows.
      • samza-luwak - uses Luwak, a stored-query engine built on Lucene, to implement full-text search on streams.
      • Turbine - tool for aggregating streams of Server-Sent Event (SSE) JSON data into a single stream.
      • Nussknacker - A visual tool to define and run real-time decision algorithms.
      • Apache Pekko - Fork of Akka 2.6.x, prior to the Akka project's adoption of the Business Source License.
      • aeron - efficient reliable unicast and multicast message transport.
      • pulsar - Actor based event driven concurrent framework for Python.
      • Eventum - Data generation platform for producing synthetic event streams based on templates, scripts or log samples.

  • Website

Programming Languages
Java 48 Python 19 Scala 15 Go 13 Rust 12 C++ 11 C# 3 TypeScript 3 Pony 2 Jupyter Notebook 2
Categories
Table of Contents 172 License 1 Website 1
Sub Categories
Streaming Engine 36 Data Pipeline 22 Streaming Library 21 Engines and Platforms 16 Libraries, SDKs, and Programming Models 14 Closed Source 11 Toolkit 9 Readings 8 Online Machine Learning 8 Streaming SQL 7 Data Integration and Pipelines 5 Streaming Application 5 DSL 5 Benchmark 3 IoT 2 Applications and Tools 1
Keywords
stream-processing 36 streaming 31 kafka 26 python 22 streaming-data 15 distributed-systems 14 java 14 rust 13 sql 12 real-time 12 machine-learning 9 cloud-native 9 data-engineering 9 stream 8 scala 8 messaging 7 go 7 big-data 7 etl 7 serverless 6 golang 6 analytics 6 actor-model 6 data-integration 6 stream-processing-engine 5 flink 5 high-performance 5 event-driven-architecture 5 data-pipeline 5 data-processing 5 stream-processor 5 storm 4 nats 4 distributed-actors 4 concurrency 4 kubernetes 4 c-plus-plus 4 reactive 4 cpp 4 kafka-streams 4 deep-learning 4 postgresql 4 iot 4 streaming-data-processing 4 distributed 4 reactive-programming 4 data-science 4 api 4 event-streaming 4 data-pipelines 4