{"id":283,"url":"https://github.com/manuzhang/awesome-streaming","name":"awesome-streaming","description":"a curated list of awesome streaming frameworks, applications, etc","projects_count":143,"last_synced_at":"2026-10-10T21:00:24.571Z","repository":{"id":24750985,"uuid":"28163761","full_name":"manuzhang/awesome-streaming","owner":"manuzhang","description":"a curated list of awesome streaming frameworks, applications, etc","archived":false,"fork":false,"pushed_at":"2026-10-08T23:02:40.000Z","size":3168,"stargazers_count":3021,"open_issues_count":7,"forks_count":327,"subscribers_count":132,"default_branch":"master","last_synced_at":"2026-10-09T12:35:27.331Z","etag":null,"topics":["awesome","awesome-list","list","stream-processing"],"latest_commit_sha":null,"homepage":"https://manuzhang.github.io/awesome-streaming","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manuzhang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"claude":null,"gemini":null,"cursor":null,"copilot":null,"dco":null,"cla":null,"disclosure":null}},"created_at":"2014-12-18T01:47:00.000Z","updated_at":"2026-10-08T19:12:23.000Z","dependencies_parsed_at":"2026-09-27T23:04:21.446Z","dependency_job_id":"50795cbe-12e8-4790-a508-2a7408140bef","html_url":"https://github.com/manuzhang/awesome-streaming","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/manuzhang/awesome-streaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuzhang%2Fawesome-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuzhang%2Fawesome-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuzhang%2Fawesome-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuzhang%2Fawesome-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manuzhang","download_url":"https://codeload.github.com/manuzhang/awesome-streaming/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuzhang%2Fawesome-streaming/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":343680568,"owners_count":38146847,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-09T02:00:07.585Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-04T17:20:10.515Z","updated_at":"2026-10-10T21:00:24.571Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Table of Contents","Website","License"],"sub_categories":["Closed Source","Readings","Engines and Platforms","Data Pipeline","Libraries, SDKs, and Programming Models","Streaming Engine","Toolkit","Applications and Tools","IoT","DSL","Data Integration and Pipelines","Streaming SQL","Online Machine Learning","Streaming Library","Streaming Application","Benchmark"],"readme":"## Awesome Streaming  [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome) [![Build Status](https://github.com/manuzhang/awesome-streaming/workflows/build/badge.svg)](https://github.com/manuzhang/awesome-streaming/actions)\n\nA curated list of awesome [streaming (stream processing)](http://radar.oreilly.com/2015/08/the-world-beyond-batch-streaming-101.html) frameworks, applications, readings and other resources. Inspired by [other awesome projects](https://github.com/sindresorhus/awesome). \n\n## Website\n\n[https://manuzhang.github.io/awesome-streaming/](https://manuzhang.github.io/awesome-streaming/) is a more dynamic website where you can find **updates** of the awesome projects here.\n\n## Table of Contents\n\n- [Engines and Platforms](#engines-and-platforms)\n- [Libraries, SDKs, and Programming Models](#libraries-sdks-and-programming-models)\n- [Data Integration and Pipelines](#data-integration-and-pipelines)\n- [Applications and Tools](#applications-and-tools)\n- [Managed and Closed Source](#managed-and-closed-source)\n- [Benchmarks](#benchmarks)\n- [Readings](#readings)\n\n### Engines and Platforms\n\n- [Aeron](https://github.com/aeron-io/aeron) \u003csub\u003e![Java/C++][language-java-cpp]\u003c/sub\u003e - Reliable UDP unicast, multicast, and IPC message transport.\n- [Apache Apex](https://github.com/apache/apex-core) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Unified platform for big data stream and batch processing.\n- [Apache Flink](https://github.com/apache/flink) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Distributed engine for stateful computation over bounded and unbounded data streams.\n- [Apache Heron](https://github.com/apache/incubator-heron) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Retired distributed, fault-tolerant stream processing engine originally developed at Twitter.\n- [Apache Kafka](https://github.com/apache/kafka) \u003csub\u003e![Java/Scala][language-java-scala]\u003c/sub\u003e - Distributed event streaming platform for high-performance data pipelines and applications.\n- [Apache Pulsar](https://github.com/apache/pulsar) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Distributed pub-sub messaging and event streaming platform.\n- [Apache RocketMQ](https://github.com/apache/rocketmq) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Cloud-native messaging and streaming platform for event-driven applications.\n- [Apache Samza](https://github.com/apache/samza) \u003csub\u003e![Java/Scala][language-java-scala]\u003c/sub\u003e - Distributed stream processing framework built on Apache Kafka with standalone and YARN deployment options.\n- [Apache Spark Streaming](https://github.com/apache/spark) \u003csub\u003e![Scala/Java/Python/R][language-scala-java-python-r]\u003c/sub\u003e - Scalable, fault-tolerant stream processing on Apache Spark.\n- [Apache Storm](https://github.com/apache/storm) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Distributed real-time computation system for reliably processing unbounded streams of data.\n- [Apache StreamPipes](https://github.com/apache/streampipes) \u003csub\u003e![Java/Python/TypeScript][language-java-python-typescript]\u003c/sub\u003e - Self-service industrial IoT platform for connecting, analyzing, and exploring data streams.\n- [ArkFlow](https://github.com/arkflow-rs/arkflow) \u003csub\u003e![Rust][language-rust]\u003c/sub\u003e - High-performance stream processing engine with pluggable sources, processors, and sinks.\n- [Arroyo](https://github.com/ArroyoSystems/arroyo) \u003csub\u003e![Rust][language-rust]\u003c/sub\u003e - Distributed stream processing engine for stateful SQL computations over unbounded data.\n- [AthenaX](https://github.com/uber-archive/AthenaX) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Uber's retired SQL-based streaming analytics platform.\n- [AutoMQ](https://github.com/AutoMQ/automq) \u003csub\u003e![Java/Scala][language-java-scala]\u003c/sub\u003e - Diskless Kafka-compatible streaming platform that stores durable data in object storage.\n- [Bytewax](https://github.com/bytewax/bytewax) \u003csub\u003e![Python/Rust][language-python-rust]\u003c/sub\u003e - Community-maintained Python framework with a Rust-based distributed engine for stateful stream processing.\n- [eKuiper](https://github.com/lf-edge/ekuiper) \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Lightweight data stream processing engine for resource-constrained IoT edge devices.\n- [Esper](https://github.com/espertechinc/esper) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Complex event processing, Streaming SQL, and event series analysis engine.\n- [Fluvio](https://github.com/fluvio-community/fluvio) \u003csub\u003e![Rust/WASM][language-rust-wasm]\u003c/sub\u003e - Composable, stateful data streaming system with programmable in-line computation.\n- [Gazette](https://github.com/gazette/core) \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Cloud-storage-backed streaming infrastructure that combines SQL, batch, and millisecond-latency stream processing.\n- [Gearpump](https://github.com/gearpump/gearpump) \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - Lightweight real-time distributed streaming engine built on Akka.\n- [hailstorm](https://github.com/hailstorm-hs/hailstorm) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Haskell][language-haskell]\u003c/sub\u003e - Distributed stream processing with exactly-once semantics based on Storm.\n- [Hazelcast Jet](https://github.com/hazelcast/hazelcast-jet) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Stream and batch processing engine whose development moved into Hazelcast Platform.\n- [HStreamDB](https://github.com/hstreamdb/hstream) \u003csub\u003e![Haskell][language-haskell]\u003c/sub\u003e - Cloud-native streaming database for IoT data storage and real-time processing.\n- [ksqlDB](https://github.com/confluentinc/ksql) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Source-available database purpose-built for stream processing applications.\n- [LaminarDB](https://github.com/laminardb/laminardb) \u003csub\u003e![Rust][language-rust]\u003c/sub\u003e - Embeddable streaming SQL engine built on Apache Arrow and DataFusion.\n- [LightSaber](https://github.com/lsds/LightSaber) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Multi-core stream processing engine using code generation for window aggregation.\n- [LogDevice](https://github.com/facebookarchive/LogDevice) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Facebook's archived distributed storage system for sequential data.\n- [Maki Nage](https://github.com/maki-nage/makinage) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Stream processing framework for data scientists based on Kafka and ReactiveX.\n- [Mantis](https://github.com/Netflix/mantis) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Netflix platform for building real-time, operations-focused stream processing applications.\n- [Materialize](https://materialize.com) \u003csub\u003e![Rust][language-rust]\u003c/sub\u003e - Incremental SQL engine for maintaining continuously updated views over changing data.\n- [Metaq](https://github.com/killme2008/Metamorphosis) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Taobao distributed messaging system.\n- [mupd8 (Muppet)](https://github.com/walmartlabs/mupd8) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Scala/Java][language-scala-java]\u003c/sub\u003e - MapReduce-style framework for processing fast-moving data streams.\n- [NATS Streaming](https://github.com/nats-io/nats-streaming-server) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Deprecated disk-backed messaging system superseded by NATS JetStream.\n- [NebulaStream](https://github.com/nebulastream/nebulastream) \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Data management system for cloud-edge-sensor environments.\n- [NSQ](https://github.com/nsqio/nsq) \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Real-time distributed messaging platform designed to operate at scale.\n- [Numaflow](https://github.com/numaproj/numaflow) \u003csub\u003e![Rust/Go/TypeScript][language-rust-go-typescript]\u003c/sub\u003e - Kubernetes-native, language-agnostic platform for scalable event-driven applications.\n- [Onyx](https://github.com/onyx-platform/onyx) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Clojure][language-clojure]\u003c/sub\u003e - Distributed, masterless, fault-tolerant data processing platform.\n- [PipelineDB](https://github.com/pipelinedb/pipelinedb) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![C][language-c]\u003c/sub\u003e - Unmaintained PostgreSQL extension for continuous SQL aggregation over time-series data.\n- [Proton](https://github.com/timeplus-io/proton) \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Unified streaming and historical data analytics database powered by ClickHouse.\n- [Redpanda](https://github.com/redpanda-data/redpanda) \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Kafka API-compatible streaming data platform without ZooKeeper or a JVM.\n- [RisingWave](https://github.com/risingwavelabs/risingwave) \u003csub\u003e![Rust][language-rust]\u003c/sub\u003e - PostgreSQL-compatible streaming database for event-driven applications, real-time ETL, and continuous analytics.\n- [S4](https://github.com/apache/incubator-retired-s4) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Retired distributed platform for processing continuous unbounded data streams.\n- [SABER](https://github.com/lsds/Saber) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java/C][language-java-c]\u003c/sub\u003e - Window-based hybrid CPU/GPU stream processing engine.\n- [Scramjet Cloud Platform](https://github.com/scramjetorg/transform-hub) \u003csub\u003e![TypeScript/Python][language-typescript-python]\u003c/sub\u003e - Distributed runtime for running and managing data processing programs on edge, server, and cloud infrastructure.\n- [SensorBee](https://github.com/sensorbee/sensorbee) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Discontinued lightweight stream processing engine for IoT.\n- [Siddhi](https://github.com/siddhi-io/siddhi) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Streaming SQL and complex event processing engine.\n- [SPQR](https://github.com/ottogroup/SPQR) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Dynamic framework for processing high-volume data streams through pipelines.\n- [SQLFlow](https://github.com/turbolytics/sql-flow) \u003csub\u003e![Go/Python][language-go-python]\u003c/sub\u003e - Stream processing engine that runs DuckDB SQL over Kafka, WebSocket, and webhook streams.\n- [Squall](https://github.com/epfldata/squall) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Online SQL query processing engine built on Apache Storm.\n- [StreamCQL](https://github.com/Zhiqiang-He/StreamCQL) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Storm-based continuous query language for real-time computation.\n- [Teknek](https://github.com/edwardcapriolo/teknek-core) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Stream processing platform with the interactive Stream Operator Language shell.\n- [Tigon](https://github.com/cdapio/tigon) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![C++/Java][language-cpp-java]\u003c/sub\u003e - High-throughput real-time stream processing framework built on Hadoop and HBase.\n- [Trill](https://github.com/microsoft/Trill) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![C#][language-csharp]\u003c/sub\u003e - Single-node query processor for temporal and streaming data from Microsoft Research.\n- [Wally](https://github.com/WallarooLabs/wally) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Pony][language-pony]\u003c/sub\u003e - Distributed stream processing framework formerly named Wallaroo.\n- [Wingfoil](https://github.com/wingfoil-io/wingfoil) \u003csub\u003e![Rust/Python/TypeScript][language-rust-python-typescript]\u003c/sub\u003e - Graph-based stream processing engine for latency-critical systems.\n\n### Libraries, SDKs, and Programming Models\n\n- [Akka](https://github.com/akka/akka-core) \u003csub\u003e![Scala/Java][language-scala-java]\u003c/sub\u003e - Toolkit and runtime for concurrent, distributed, resilient applications on the JVM.\n- [Akka Streams](https://github.com/akka/akka-core) \u003csub\u003e![Scala/Java][language-scala-java]\u003c/sub\u003e - Reactive Streams implementation built on Akka actors.\n- [Apache Beam](https://github.com/apache/beam) \u003csub\u003e![Java/Python/Go][language-java-python-go]\u003c/sub\u003e - Unified programming model and language-specific SDKs for batch and streaming data processing.\n- [Apache Edgent](https://github.com/apache/incubator-retired-edgent) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Retired programming model and runtime for streaming analytics on gateways and edge devices.\n- [Apache Kafka Streams](https://github.com/apache/kafka) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Stream processing library included with Apache Kafka.\n- [Apache Pekko](https://github.com/apache/pekko) \u003csub\u003e![Scala/Java][language-scala-java]\u003c/sub\u003e - Open-source toolkit for concurrent, distributed, resilient applications, forked from Akka 2.6.\n- [Apache SAMOA](https://github.com/apache/incubator-samoa) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Retired distributed streaming machine learning framework.\n- [Apache StormCrawler](https://github.com/apache/stormcrawler) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Scalable web crawler SDK based on Apache Storm.\n- [CapyMOA](https://github.com/adaptive-machine-learning/CapyMOA) \u003csub\u003e![Python/Java][language-python-java]\u003c/sub\u003e - Machine learning library for data streams with a Python API and MOA backend.\n- [coast](https://github.com/bkirwi/coast) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - DSL that builds exactly-once dataflow graphs on top of Apache Samza.\n- [Daggy](https://github.com/synacker/daggy) \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Utility and developer library for aggregating and capturing data streams.\n- [DataSketches](https://github.com/apache/datasketches-java) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Apache library of stochastic streaming algorithms known as sketches.\n- [FastStream](https://github.com/ag2ai/faststream) \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Asynchronous framework for building event-driven applications with Kafka, RabbitMQ, NATS, Redis, and MQTT.\n- [Faust](https://github.com/robinhood/faust) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Deprecated Python stream processing library inspired by Kafka Streams.\n- [FS2](https://github.com/typelevel/fs2) \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - Compositional streaming I/O library for Scala.\n- [Jubatus](https://github.com/jubatus/jubatus) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Distributed processing framework and library for online machine learning.\n- [Kzmlabs StateFun Actors](https://github.com/kzmlabs/flink-statefun) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Stateful actors on Apache Flink 2.x with durable per-key state, exactly-once messaging, and Kafka and Kinesis I/O.\n- [MediaPipe](https://github.com/google-ai-edge/mediapipe) \u003csub\u003e![C++/Python/Java/TypeScript][language-cpp-python-java-typescript]\u003c/sub\u003e - Cross-platform, customizable ML solutions for live and streaming media.\n- [Monix](https://github.com/monix/monix) \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - High-performance Scala and Scala.js library for asynchronous and event-based programs.\n- [Numalogic](https://github.com/numaproj/numalogic) \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Collection of machine learning models and tools for anomaly detection and forecasting on operational time-series data.\n- [Pulsar](https://github.com/quantmind/pulsar) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Actor-based event-driven concurrency framework for Python.\n- [PySAD](https://github.com/selimfirat/pysad) \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Python library for anomaly detection on streaming data, with online detectors, stream simulators, and evaluators.\n- [Quix Streams](https://github.com/quixio/quix-streams) \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Python framework for real-time data engineering, analytics, and machine learning on Apache Kafka.\n- [River](https://github.com/online-ml/river) \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Online machine learning library for Python.\n- [samza-luwak](https://github.com/romseygeek/samza-luwak) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Integration of Apache Samza with the Luwak stored-query engine for full-text stream search.\n- [Scramjet C++](https://github.com/scramjetorg/framework-cpp) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Reactive stream programming framework for C++.\n- [Scramjet Python](https://github.com/scramjetorg/framework-python) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Reactive stream programming framework for Python.\n- [Scramjet TypeScript](https://github.com/scramjetorg/framework-js) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![TypeScript][language-typescript]\u003c/sub\u003e - Reactive stream programming framework for asynchronous JavaScript and TypeScript applications.\n- [StormCV](https://github.com/sensorstorm/StormCV) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Computer vision operations and data models for Apache Storm.\n- [Stream Ops](https://github.com/nanosai/stream-ops-java) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Embeddable data streaming engine and stream processing API for Java.\n- [streamDM](https://github.com/huawei-noah/streamDM) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - Huawei library for mining data streams with Spark Streaming.\n- [Streamiz.Kafka.Net](https://github.com/LGouellec/streamiz) \u003csub\u003e![C#][language-csharp]\u003c/sub\u003e - .NET stream processing library for Apache Kafka.\n- [Streamparse](https://github.com/pystorm/streamparse) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Python API, command-line tools, and topology DSL for Apache Storm.\n- [Streamz](https://github.com/python-streamz/streamz) \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Library for building continuous data pipelines with branching, joining, flow control, and back pressure.\n- [Summingbird](https://github.com/twitter/summingbird) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - Retired library for expressing streaming MapReduce computations over Storm and Scalding.\n- [Swave](https://github.com/sirthias/swave) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - Lightweight Reactive Streams infrastructure toolkit for Scala.\n- [Tributary](https://github.com/1kbgz/tributary) \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Python library for constructing synchronous, reactive, and lazy dataflow graphs.\n- [trident-ml](https://github.com/pmerienne/trident-ml) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Online machine learning library based on Apache Storm Trident.\n- [WindFlow](https://github.com/ParaGroup/WindFlow) \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Data stream processing parallel library for multi-core CPUs and GPUs.\n- [Yurita](https://github.com/paypal/yurita) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - PayPal anomaly detection framework built on Spark Structured Streaming.\n\n### Data Integration and Pipelines\n\n- [Apache Flume](https://github.com/apache/logging-flume) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Distributed service for collecting, aggregating, and moving large amounts of log-like data.\n- [Brooklin](https://github.com/linkedin/Brooklin) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Distributed system for reliable nearline data streaming between heterogeneous systems at scale.\n- [Bruin](https://github.com/bruin-data/bruin) \u003csub\u003e![Go][language-go]\u003c/sub\u003e - End-to-end data pipeline tool combining ingestion, SQL and Python transformations, and data quality checks.\n- [Camus](https://github.com/LinkedInAttic/camus) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - LinkedIn's retired, previous-generation Kafka-to-HDFS pipeline.\n- [CocoIndex](https://github.com/cocoindex-io/cocoindex) \u003csub\u003e![Rust/Python][language-rust-python]\u003c/sub\u003e - Incremental data transformation engine for continuously updated AI and agent workloads.\n- [Databus](https://github.com/linkedin/databus) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - LinkedIn source-agnostic distributed change data capture system.\n- [faucet-stream](https://github.com/faucet-hq/faucet-stream) \u003csub\u003e![Rust][language-rust]\u003c/sub\u003e - Config-driven platform and embeddable library for ETL, CDC, and streaming data movement.\n- [inGestr](https://github.com/bruin-data/ingestr) \u003csub\u003e![Go/Python][language-go-python]\u003c/sub\u003e - Command-line application and Python SDK for copying data between databases, SaaS applications, and data warehouses.\n- [Pathway](https://github.com/pathwaycom/pathway) \u003csub\u003e![Python/Rust][language-python-rust]\u003c/sub\u003e - Python ETL framework for stream processing, real-time analytics, and AI data pipelines.\n- [Redpanda Connect](https://github.com/redpanda-data/connect) \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Declarative stream processor for moving, enriching, transforming, and filtering data between sources and sinks.\n- [RudderStack](https://github.com/rudderlabs/rudder-server) \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Developer-focused customer data platform for event streaming and cloud-to-warehouse data pipelines.\n- [StreamPulse](https://github.com/Yacine-ai-tech/StreamPulse) \u003csub\u003e![Python/TypeScript][language-python-typescript]\u003c/sub\u003e - Real-time Kafka telemetry ingestion and stream processing pipeline with Redis sliding-window aggregations and live dashboard.\n- [Suro](https://github.com/Netflix/suro) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Netflix data pipeline for collecting, aggregating, and dispatching application events.\n- [yasdb](https://github.com/JayJamieson/yasdb) \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Durable Streams protocol server backed by SlateDB object storage, with offset-based catch-up reads and SSE or long-poll tailing.\n\n### Applications and Tools\n\n- [beava](https://github.com/beava-dev/beava) \u003csub\u003e![Rust/Python][language-rust-python]\u003c/sub\u003e - Single-binary feature server for querying fresh per-entity counters and aggregates without a message broker.\n- [Eventum](https://github.com/eventum-generator/eventum) \u003csub\u003e![Python/TypeScript][language-python-typescript]\u003c/sub\u003e - Data generation platform for producing synthetic event streams.\n- [javactrl-kafka](https://github.com/javactrl/javactrl-kafka) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Code-first distributed workflow engine for microservice orchestration and business process automation.\n- [Nussknacker](https://github.com/TouK/nussknacker) \u003csub\u003e![Scala][language-scala]\u003c/sub\u003e - Visual tool for defining and running real-time decision algorithms.\n- [straw](https://github.com/rwalk/straw) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Python/Java][language-python-java]\u003c/sub\u003e - Platform for real-time streaming search.\n- [StreamAlert](https://github.com/airbnb/streamalert) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Airbnb serverless framework for real-time security log analysis and alerting.\n- [Streamdal](https://github.com/streamdal/streamdal) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![TypeScript][language-typescript]\u003c/sub\u003e - Code-native data privacy controls for detecting PII in application data flows.\n- [StreamFlow](https://github.com/lmco/streamflow) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Tool for building and monitoring stream processing workflows.\n- [StreamingBandit](https://github.com/Nth-iteration-labs/streamingbandit) \u003csub\u003e![Python][language-python]\u003c/sub\u003e - Web application for setting up and evaluating contextual multi-armed bandit experiments.\n- [Streamline](https://github.com/hortonworks/streamline) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Hortonworks visual stream analytics framework built around engines such as Apache Storm.\n- [Substation](https://github.com/brexhq/substation) \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Toolkit for routing, normalizing, and enriching security event and audit logs.\n- [Turbine](https://github.com/Netflix/Turbine) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Netflix tool for aggregating Server-Sent Event JSON streams.\n- [Zilla](https://github.com/aklivity/zilla) \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Multi-protocol gateway for connecting applications, APIs, agents, and devices to event streams.\n\n### Managed and Closed Source\n\n- [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/) - Fully managed service for ingesting and processing real-time data streams on AWS.\n- [Azure Stream Analytics](https://azure.microsoft.com/en-us/products/stream-analytics) \u003csub\u003e![SQL][language-sql]\u003c/sub\u003e - Fully managed service for serverless real-time analytics in the cloud and at the edge.\n- [Concord](https://www.slideshare.net/concord-io/may-2016-data-by-the-bay-concord-simple-flexible-stream-processing-on-apache-mesos) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![C++][language-cpp]\u003c/sub\u003e - Historical distributed stream processing framework built on Apache Mesos.\n- [Google Cloud Dataflow](https://cloud.google.com/dataflow/) \u003csub\u003e![Java/Python/Go][language-java-python-go]\u003c/sub\u003e - Fully managed service for running Apache Beam batch and streaming pipelines.\n- [IBM Streams](https://www.ibm.com/support/pages/ibm-streams-life-cycle-guidance) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Python/Java/Scala][language-python-java-scala]\u003c/sub\u003e - Discontinued proprietary platform for distributed stream processing and real-time analytics.\n- [NVIDIA DeepStream SDK](https://developer.nvidia.com/deepstream-sdk) \u003csub\u003e![C/C++/Python][language-c-cpp-python]\u003c/sub\u003e - GStreamer-based toolkit with open-source components and proprietary NVIDIA libraries for real-time AI streaming analytics and multi-sensor processing.\n\n### Benchmarks\n\n- [Flotilla](https://github.com/tylertreat/Flotilla) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Go][language-go]\u003c/sub\u003e - Automated message queue orchestration for scaled-up benchmarking.\n- [storm-perf-test](https://github.com/yahoo/storm-perf-test) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Apache Storm performance and stress test.\n- [streaming-benchmarks](https://github.com/yahoo/streaming-benchmarks) \u003csub\u003e![Archived][archived-badge]\u003c/sub\u003e \u003csub\u003e![Java][language-java]\u003c/sub\u003e - Benchmarks for low-latency stream processing systems including Storm, Spark, and Flink.\n\n### Readings\n\n1. [In-Stream Big Data Processing](https://highlyscalable.wordpress.com/2013/08/20/in-stream-big-data-processing/)\n2. [The world beyond batch: Streaming 101](http://radar.oreilly.com/2015/08/the-world-beyond-batch-streaming-101.html) by Tyler Akidau. \n3. [Real Time Analytics: Algorithms and Systems (VLDB 2015)](https://arxiv.org/abs/1708.02621)\n4. [Grokking Streaming Systems](https://www.manning.com/books/grokking-streaming-systems) by Josh Fischer \u0026 Ning Wang\n5. [Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing](https://www.oreilly.com/library/view/streaming-systems/9781491983867/) by Reuven Lax, Slava Chernyak, and Tyler Akidau\n6. [Data Pipelines with Apache Airflow](https://www.manning.com/books/data-pipelines-with-apache-airflow) by Bas P. Harenslak and Julian Rutger de Ruiter\n7. [MillWheel: Fault-Tolerant Stream Processing at Internet Scale](https://research.google/pubs/millwheel-fault-tolerant-stream-processing-at-internet-scale/)\n\n## License\n\n![Creative Commons License](https://i.creativecommons.org/l/by-sa/4.0/80x15.png)\n\nLicensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-sa/4.0/)\n\n[archived-badge]: https://img.shields.io/badge/status-archived-lightgrey\n[language-c]: https://img.shields.io/badge/language-C-blue\n[language-c-cpp-python]: https://img.shields.io/badge/language-C%2FC%2B%2B%2FPython-blue\n[language-csharp]: https://img.shields.io/badge/language-C%23-blue\n[language-cpp]: https://img.shields.io/badge/language-C%2B%2B-blue\n[language-cpp-java]: https://img.shields.io/badge/language-C%2B%2B%2FJava-blue\n[language-cpp-python-java-typescript]: https://img.shields.io/badge/language-C%2B%2B%2FPython%2FJava%2FTypeScript-blue\n[language-clojure]: https://img.shields.io/badge/language-Clojure-blue\n[language-go]: https://img.shields.io/badge/language-Go-blue\n[language-go-python]: https://img.shields.io/badge/language-Go%2FPython-blue\n[language-haskell]: https://img.shields.io/badge/language-Haskell-blue\n[language-java]: https://img.shields.io/badge/language-Java-blue\n[language-java-c]: https://img.shields.io/badge/language-Java%2FC-blue\n[language-java-cpp]: https://img.shields.io/badge/language-Java%2FC%2B%2B-blue\n[language-java-python-go]: https://img.shields.io/badge/language-Java%2FPython%2FGo-blue\n[language-java-python-typescript]: https://img.shields.io/badge/language-Java%2FPython%2FTypeScript-blue\n[language-java-scala]: https://img.shields.io/badge/language-Java%2FScala-blue\n[language-pony]: https://img.shields.io/badge/language-Pony-blue\n[language-python]: https://img.shields.io/badge/language-Python-blue\n[language-python-java]: https://img.shields.io/badge/language-Python%2FJava-blue\n[language-python-java-scala]: https://img.shields.io/badge/language-Python%2FJava%2FScala-blue\n[language-python-rust]: https://img.shields.io/badge/language-Python%2FRust-blue\n[language-python-typescript]: https://img.shields.io/badge/language-Python%2FTypeScript-blue\n[language-rust]: https://img.shields.io/badge/language-Rust-blue\n[language-rust-go-typescript]: https://img.shields.io/badge/language-Rust%2FGo%2FTypeScript-blue\n[language-rust-python]: https://img.shields.io/badge/language-Rust%2FPython-blue\n[language-rust-python-typescript]: https://img.shields.io/badge/language-Rust%2FPython%2FTypeScript-blue\n[language-rust-wasm]: https://img.shields.io/badge/language-Rust%2FWASM-blue\n[language-scala]: https://img.shields.io/badge/language-Scala-blue\n[language-scala-java]: https://img.shields.io/badge/language-Scala%2FJava-blue\n[language-scala-java-python-r]: https://img.shields.io/badge/language-Scala%2FJava%2FPython%2FR-blue\n[language-sql]: https://img.shields.io/badge/language-SQL-blue\n[language-typescript]: https://img.shields.io/badge/language-TypeScript-blue\n[language-typescript-python]: https://img.shields.io/badge/language-TypeScript%2FPython-blue\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/manuzhang%2Fawesome-streaming/projects"}