Projects in Awesome Lists tagged with stream-processing

https://github.com/google-ai-edge/mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

android audio-processing c-plus-plus calculator computer-vision deep-learning framework graph-based graph-framework inference machine-learning mediapipe mobile-development perception pipeline-framework stream-processing video-processing

Last synced: 12 May 2025

https://google.github.io/mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

android audio-processing c-plus-plus calculator computer-vision deep-learning framework graph-based graph-framework inference machine-learning mediapipe mobile-development perception pipeline-framework stream-processing video-processing

Last synced: 02 Apr 2025

https://github.com/pathwaycom/pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

batch-processing data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics kafka machine-learning-algorithms pathway python real-time rust stream-processing streaming time-series-analysis

Last synced: 08 Jan 2026

https://github.com/vectordotdev/vector

A high-performance observability data pipeline.

events forwarder logs metrics observability parser pipeline router rust stream-processing vector

Last synced: 12 May 2025

https://github.com/zhisheng17/flink-learning

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

clickhouse elasticsearch flink hbase influxdb kafka loki mysql opentsdb rabbitmq redis rocketmq spark stream-processing streaming

Last synced: 14 May 2025

https://github.com/onurakpolat/awesome-bigdata

A curated list of awesome big data frameworks, ressources and other awesomeness.

awesome awesome-list bigdata data data-analytics data-science data-stream data-visualization data-warehouse database distributed-database series-database stream-processing streaming-data visualize-data

Last synced: 22 Mar 2025

https://github.com/redpanda-data/redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

containers cpp event-driven go kafka kubernetes microservices realtime redpanda seastar storage-engine stream-processing streaming

Last synced: 26 Dec 2025

https://github.com/redpanda-data/connect

Fancy stream processing made operationally mundane

amqp cqrs data-engineering data-ops etl event-sourcing go golang kafka logs message-bus message-queue nats rabbitmq stream-processing stream-processor streaming-data

Last synced: 22 Jan 2026

https://github.com/threedotslabs/watermill

Building event-driven applications the easy way in Go.

cqrs event-driven event-sourcing events go golang kafka nats rabbitmq reactive sagas stream-processing watermill

Last synced: 13 May 2025

https://github.com/Jeffail/benthos

Fancy stream processing made operationally mundane

amqp cqrs data-engineering data-ops etl event-sourcing go golang kafka logs message-bus message-queue nats rabbitmq stream-processing stream-processor streaming-data

Last synced: 25 Mar 2025

https://github.com/risingwavelabs/risingwave

Stream processing and management platform.

data-engineering database kafka materialized-view postgresql rust stream-processing

Last synced: 13 May 2025

https://github.com/singularity-data/risingwave

Stream processing and management platform.

data-engineering database kafka materialized-view postgresql rust stream-processing

Last synced: 15 Apr 2025

https://github.com/ThreeDotsLabs/watermill

Building event-driven applications the easy way in Go.

cqrs event-driven event-sourcing events go golang kafka nats rabbitmq reactive sagas stream-processing watermill

Last synced: 14 Mar 2025

https://github.com/RisingWaveLabs/risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.

analytics big-data cloud-native data-engineering database distributed-database etl flink kafka ksqldb materialized-view postgres postgresql real-time real-time-analytics rust serverless spark-streaming sql stream-processing

Last synced: 29 Mar 2025

https://github.com/fluent/fluent-bit

Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows

c cloudnative data-collector fluent-bit fluentd forwarder logging logs metrics opentelemetry prometheus sql-queries stream-processing traces

Last synced: 06 May 2025

https://github.com/robinhood/faust

Python Stream Processing

asyncio distributed-systems kafka kafka-streams python stream-processing

Last synced: 13 May 2025

https://github.com/hazelcast/hazelcast

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

big-data caching data-in-motion data-insights distributed distributed-computing distributed-systems hacktoberfest hazelcast in-memory java low-latency real-time scalability stream-processing

Last synced: 09 Sep 2025

https://github.com/materializeinc/materialize

Real-time Data Integration and Transformation: use SQL to transform, deliver, and act on fast-changing data.

data-store database distributed-systems kafka materialized-view operational-data-store postgresql postgresql-dialect rust sql stream-processing streaming streaming-data

Last synced: 09 Sep 2025

https://github.com/MaterializeInc/materialize

The Cloud Operational Data Store: use SQL to transform, deliver, and act on fast-changing data.

data-warehouse database distributed-systems kafka materialized-view operational-data-warehouse postgresql postgresql-dialect rust sql stream-processing streaming streaming-data

Last synced: 28 Mar 2025

https://github.com/apache/hudi

Upserts, Deletes And Incremental Processing on Big Data.

apacheflink apachehudi apachespark bigdata data-integration datalake hudi incremental-processing stream-processing

Last synced: 12 May 2025

https://github.com/online-ml/river

🌊 Online machine learning in Python

concept-drift data-science incremental-learning machine-learning online-learning online-machine-learning online-statistics python real-time-processing stream-processing streaming streaming-data

Last synced: 12 Dec 2025

https://github.com/infinyon/fluvio

🦀 event stream processing for developers to stream and process data in motion to power responsive data intensive applications.

cloud-native data-analytics data-flow data-integration data-pipelines distributed-systems event-driven-architecture real-time rust serverless stateful stream-processing stream-processing-engine streaming streaming-analytics streaming-data streaming-data-pipelines streaming-data-processing webassembly

Last synced: 13 May 2025

https://github.com/javascriptdata/danfojs

Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.

danfojs data-analysis data-analytics data-manipulation data-science dataframe javascript pandas plotting-charts stream-data stream-processing table tensorflow tensors

Last synced: 14 May 2025

https://github.com/ag2ai/faststream

FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.

asyncapi asyncio distributed-systems fastkafka faststream kafka nats propan python rabbitmq redis stream-processing

Last synced: 14 May 2025

https://github.com/ArroyoSystems/arroyo

Distributed stream processing engine in Rust

data data-stream-processing dev-tools infrastructure kafka rust sql stream-processing stream-processing-engine

Last synced: 26 Mar 2025

https://github.com/memgraph/memgraph

Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.

cypher graph graph-algorithms graph-analysis graph-database kafka kafka-streams nosql opencypher stream-processing streaming-data

Last synced: 14 May 2025

https://github.com/airtai/faststream

FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.

asyncapi asyncio distributed-systems fastkafka faststream kafka nats propan python rabbitmq redis stream-processing

Last synced: 25 Mar 2025

https://manuzhang.github.io/awesome-streaming/

a curated list of awesome streaming frameworks, applications, etc

awesome awesome-list list stream-processing

Last synced: 27 Mar 2025

https://github.com/douban/dpark

Python clone of Spark, a MapReduce alike framework in Python

bigdata dpark mapreduce python spark stream-processing

Last synced: 29 Oct 2025

https://github.com/pipelinedb/pipelinedb

High-performance time-series aggregation for PostgreSQL

aggregation analytics pipelinedb postgresql push realtime sql stream-processing time-series

Last synced: 18 Mar 2025

https://github.com/peerdb-io/peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

bigquery cdc clickhouse cloud-native distributed-systems etl eventhubs kafka postgres postgresql realtime rust s3 snowflake sql stream-processing

Last synced: 21 Jan 2026

https://github.com/numaproj/numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

data-processing hacktoberfest k8s kubernetes map-reduce pipeline stream-processing

Last synced: 18 Jan 2026

https://github.com/PeerDB-io/peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

bigquery cdc clickhouse cloud-native distributed-systems etl eventhubs kafka postgres postgresql realtime rust s3 snowflake sql stream-processing

Last synced: 28 Mar 2025

https://github.com/francoispqt/gojay

high performance JSON encoder/decoder with stream API for Golang

decoder encoder go golang json perfomance stream-decoder stream-processing

Last synced: 13 May 2025

https://github.com/reugn/go-streams

A lightweight stream processing library for Go

aerospike data-pipeline data-stream etl kafka kafka-streams low-code nats-streaming pipeline pulsar redis stream-processing stream-processor streaming-api streaming-data streams throttling websocket windowing workflow

Last synced: 14 May 2025

https://github.com/timeplus-io/proton

High-performance, low-footprint SQL database written in C++. Process millions of rows per second from Kafka/Pulsar, Iceberg, or ClickHouse, and seamlessly write results back. Supports powerful features like JOIN, CDC, UPSERT, and LOOKUP, enabling real-time analytics and ETL at scale.

analytics clickhouse confluent cpp flink-alternative high-performance iceberg kakfa ksqldb-alternative redpanda simd single-binary sql stream-processing streaming-sql udf

Last synced: 14 May 2025

https://github.com/yomorun/yomo

🦖 Stateful Serverless Framework for Geo-distributed Edge AI Infra. with function calling support, write once, run on any model.

chatgpt distributed-cloud edge-computing function-calling gemini geodistributedsystems gpt-4o low-latency openai quic realtime serverless stateful-serverless stream-processing webassembly yomo

Last synced: 13 May 2025

https://github.com/bytewax/bytewax

Python Stream Processing

data-engineering data-processing data-science dataflow machine-learning python rust stream-processing streaming-data

Last synced: 13 May 2025

https://github.com/nerevu/riko

A Python stream processing engine modeled after Yahoo! Pipes

asynchronous cli data etl featured functional-programming library parallelism rss stream-processing

Last synced: 15 May 2025

https://github.com/siddhi-io/siddhi

Stream Processing and Complex Event Processing Engine

cep cloud cloud-native cncf complex-event-processing library online-learning stream-processing streaming-data-processing

Last synced: 16 May 2025

https://github.com/WallarooLabs/wallaroo

Distributed Stream Processing

api framework go golang linux pony-language python stream-processing stream-processing-engine stream-processor

Last synced: 25 Mar 2025

https://github.com/wallaroolabs/wally

Distributed Stream Processing

api framework go golang linux pony-language python stream-processing stream-processing-engine stream-processor

Last synced: 16 May 2025

https://github.com/WallarooLabs/wally

Distributed Stream Processing

api framework go golang linux pony-language python stream-processing stream-processing-engine stream-processor

Last synced: 07 May 2025

https://github.com/quixio/quix-streams

Python Streaming DataFrames for Kafka

data-engineering data-intensive-applications data-science event-driven-architecture kafka machine-learning python real-time-data-processing stream-processing stream-processor streaming-data streaming-data-pipelines streaming-data-processing time-series-data

Last synced: 13 May 2025

https://github.com/halaxa/json-machine

Efficient, easy-to-use, and fast PHP JSON stream parser

json-iterator json-parser json-stream parsing php stream-processing

Last synced: 13 May 2025

https://github.com/spring-attic/spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes

batch-processing cloud-native datapipelines microservices-architecture orchestration predictive-analytics stream-processing

Last synced: 14 Jun 2025

https://github.com/spring-cloud/spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes

batch-processing cloud-native datapipelines microservices-architecture orchestration predictive-analytics stream-processing

Last synced: 12 May 2025

https://github.com/hazelcast/hazelcast-jet

Distributed Stream and Batch Processing

batch-processing big-data cdc event-processing hacktoberfest java kafka low-latency stream-processing

Last synced: 17 Oct 2025

https://github.com/halaxa/json-machine?utm_source=gold_browser_extension

Efficient, easy-to-use, and fast PHP JSON stream parser

json-iterator json-parser json-stream parsing php stream-processing

Last synced: 25 Mar 2025

https://github.com/uhop/stream-json

The micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API.

javascript-objects parse-json-files parser stream-components stream-processing streaming-json

Last synced: 13 May 2025

https://github.com/spring-cloud/spring-cloud-stream

Framework for building Event-Driven Microservices

cloud-native event-driven-microservices messaging microservices reactive stream-processing

Last synced: 24 Nov 2025

https://github.com/clj-commons/manifold

A compatibility layer for event-driven abstractions

async clojure future hacktoberfest stream stream-processing

Last synced: 12 Dec 2025

https://github.com/edp963/wormhole

Wormhole is a SPaaS (Stream Processing as a Service) Platform

spark-streaming stream-processing wormhole

Last synced: 16 May 2025

https://github.com/ververica/flink-sql-cookbook

The Apache Flink SQL Cookbook is a curated collection of examples, patterns, and use cases of Apache Flink SQL. Many of the recipes are completely self-contained and can be run in Ververica Platform as is.

apache-flink flink flink-sql sql stream-processing

Last synced: 08 Oct 2025

https://github.com/nodefluent/kafka-streams

equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:

big-data kafka kafka-streams node nodejs stream-processing streams

Last synced: 15 May 2025

https://github.com/vladimirvivien/automi

A stream processing API for Go (alpha)

data-stream go golang stream-processing streaming-api

Last synced: 15 May 2025

https://github.com/gearpump/gearpump

Lightweight real-time big data streaming engine over Akka

akka bigdata scala stream-processing

Last synced: 11 Jan 2026

https://github.com/pipeless-ai/pipeless

An open-source computer vision framework to build and deploy apps in minutes

artificial-intelligence cloud computer-vision deep-learning ffmpeg gstreamer inference inference-server machine-learning multimedia multimedia-applications object-detection perception pipeline-framework python stream-processing video video-processing vision-framework yolo

Last synced: 14 Dec 2025

https://github.com/gazette/core

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

brokers event-sourcing golang stream-processing streaming-platforms

Last synced: 21 Apr 2025

https://github.com/lw-lin/streaming-readings

Streaming System 相关的论文读物

apache-spark dataflow drizzle flink heron millwheel s4 spark-streaming spe storm stream-processing stream-processing-engine streaming streaming-engine

Last synced: 04 Apr 2025

https://github.com/hstreamdb/hstream

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

data-processing database distributed-database distributed-systems financial-analysis haskell hstreamdb iot iot-database kafka materialized-view real-time realtime-database scale sql stream-processing streaming streaming-data streaming-database

Last synced: 15 May 2025

https://github.com/touk/nussknacker

Low-code tool for automating actions on real time data | Stream processing for the users.

apache-flink automation big-data data-streaming decision-engine decision-making decisioning flink flink-kafka gui kafka low-code lowcode real-time rules-engine scala stream-processing streaming touk

Last synced: 14 May 2025

https://github.com/TouK/nussknacker

Low-code tool for automating actions on real time data | Stream processing for the users.

apache-flink automation big-data data-streaming decision-engine decision-making decisioning flink flink-kafka gui kafka low-code lowcode real-time rules-engine scala stream-processing streaming touk

Last synced: 28 Mar 2025

https://github.com/apache/streampipes

Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

analytics dashboard edge iiot iot iot-analytics iot-platform microservices opcua s7 self-service stream-processing

Last synced: 11 Dec 2025

https://github.com/microsoft/psi

Platform for Situated Intelligence

artificial-intelligence component-library framework human-robot-interaction multimodal multimodal-interactions perception pipelines stream-processing streaming

Last synced: 13 Apr 2025

https://github.com/leesf/hudi-resources

汇总Apache Hudi相关资料

apache apachehudi bigdata data-integration datalake hudi hudi-resources incremental-processing stream-processing

Last synced: 27 Mar 2025

https://github.com/deep5050/radio-active

Play any radios around the globe right from the terminal :zap:

audio cli commandline-interface favorite-station music music-bot music-player online-radio pyradio pyradios radio radios stations stream-processing streaming

Last synced: 15 May 2025

https://github.com/constellation-rs/amadeus

Harmonious distributed data analysis in Rust.

data-analysis data-processing distributed-computing parallel-computing rust stream-processing

Last synced: 08 Apr 2025

https://github.com/nmaquet/kasper

Kasper is a lightweight library for processing Kafka topics.

golang-library kafka stream-processing

Last synced: 15 Dec 2025

https://github.com/smooks/smooks

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

analytics chunking enterprise-integration etl event-driven java pipelines sax smooks stream-processing xml

Last synced: 13 May 2025

https://github.com/futuresdr/futuresdr

An Async SDR Runtime for Heterogeneous Architectures

dsp message-passing radio research runtime rust sdr stream-processing wireless

Last synced: 16 May 2025

https://github.com/kurento/kurento

Kurento WebRTC Media Server

stream-processing video-streaming webrtc

Last synced: 16 May 2025

https://github.com/graphform/swim-rust

Self-contained distributed software platform for building stateful, massively real-time streaming applications in Rust.

actor-model async data-integration decentralized-applications distributed-systems framework kafka real-time rust serverless stateful stream-processing streaming streaming-data-pipelines web

Last synced: 29 Jul 2025

https://github.com/AICoE/log-anomaly-detector

Log Anomaly Detection - Machine learning to detect abnormal events logs

aiops anomaly-detection artificial-intelligence gensim kubernetes log machine-learning-algorithms som stream-processing word2vec

Last synced: 06 Apr 2025

https://github.com/axiscommunications/media-stream-library-js

JavaScript library to handle media streams on the command line (Node.js) and in the browser.

aac axis camera h264 ip-camera javascript jpeg media-pipelines mp4 rtp rtp-client rtp-player rtp-streaming rtsp rtsp-client stream-processing typescript video-streaming

Last synced: 14 May 2025

https://github.com/memgraph/mage

MAGE - Memgraph Advanced Graph Extensions :crystal_ball:

cypher graph-algorithms graph-database memgraph open-source real-time-analytics stream-processing

Last synced: 25 Jun 2025

https://github.com/Point72/csp

csp is a high performance reactive stream processing library, written in C++ and Python

cpp python reactive reactive-programming stream-processing streaming

Last synced: 14 May 2025

https://github.com/vectorengine/vectorsql

VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.

clickhouse column-store distributed-database iot olap pipeline reactive stream-processing

Last synced: 17 Apr 2025

https://github.com/streamnative/pulsar-flink

Elastic data processing with Apache Pulsar and Apache Flink

apache-flink apache-pulsar batch-processing catalog data-processing flink flink-connector flink-stream-processing pulsar schema schema-registry sql stream-processing

Last synced: 14 May 2025

https://github.com/FutureSDR/FutureSDR

An Async SDR Runtime for Heterogeneous Architectures

dsp message-passing radio research runtime rust sdr stream-processing wireless

Last synced: 07 Apr 2025

https://github.com/raystack/dagger

Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.

apache-flink apache-kafka dataops framework influxdb prometheus real-time-analytics real-time-processing stream-processing