Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with stream-processing

A curated list of projects in awesome lists tagged with stream-processing .

https://github.com/vasco-jofra/dadstorm

A simplified implementation of a fault-tolerant real-time distributed stream processing system.

distributed-systems stream-processing

Last synced: 22 Apr 2025

https://github.com/iferdel/sensor-data-streaming-pubsub

Hybrid IoT system using PubSub as backbone by means of Go/RabbitMQ, PostgreSQL with TimeScaleDB extension for storage, Alloy and Loki for logs, and Grafana for real-time visualization. Interaction via a CLI tool, which communicates with an HTTP REST API server. Deployed on Kubernetes through GitOps.

alloy amqp api-keys cli dba distributed-systems go grafana loki mqtt postgresql pubsub rabbitmq rest-api stream-processing time-series timescaledb

Last synced: 20 Apr 2026

https://github.com/narius2030/lakehouse-solution-imcp

An end-to-end MLOps pipeline to develop, train, and deploy an Image Caption model that automatically generates captions for images based on diverse datasets

apache-airflow apache-kafka batch-processing lakehouse mlflow-tracking mlops polars spark-streaming stream-processing

Last synced: 28 Feb 2025

https://github.com/akincenk/tradewatch

A modular trade monitoring & alerting system built with Python. Analyzes market activity and detects unusual behavior. Real-time cryptocurrency trade analytics platform

crypto dash data-visualization fastapi plotly python real-time stream-processing trading websocket

Last synced: 01 May 2026

https://github.com/dcs-chalmers/haren

Scheduling framework for stream processing systems

scheduling stream-processing

Last synced: 14 Jan 2026

https://github.com/abhirockzz/accs-ehcs-stream-processing

Stream Processing application using Apache Kafka, Kafka Streams and Redis (Oracle Code 2017 San Francisco)

javaee kafka kafka-streams oracle-application-container-cloud oracle-cloud oracle-jet redis stream-processing

Last synced: 20 May 2026

https://github.com/bartke/tributary

Simple, flow based event stream processor using Go and Lua.

cep dataflow-programming event-driven flow-based-programming stream-processing

Last synced: 18 Jan 2026

https://github.com/irc-sphere/hyperstreamviewer

Web-app visualisation for viewing HyperStream output

flask hyperstream stream-processing visualisation web-application

Last synced: 02 Apr 2025

https://github.com/notwoods/chrome-supersize-reports

Demo HTML reports from Chromium's supersize analysis tool.

chrome google python stream-processing treeview worker

Last synced: 13 Apr 2026

https://github.com/gabrielmbmb/flowtastic

🎏 Python Stream Processing (Faust like!) backed by pydantic.

async asyncio framework kafka kafka-streams python python-types stream-processing

Last synced: 06 Sep 2025

https://github.com/supernomad/protond

A simple data stream parser, mangler, and alert system

alerting data-manipulation data-processing stream-processing

Last synced: 17 Jan 2026

https://github.com/ibmstreams/streamsx.anomalydetection

(Incubation) Providing functions related to anomaly detection algorithms

anomalydetection ibm-streams machine-learning stream-processing

Last synced: 05 Jul 2025

https://github.com/whossname/stream_stats

Concurrent calculation of count, mean and standard deviation

elixir standard-deviation statistics stream-processing

Last synced: 30 Oct 2025

https://github.com/arkady-emelyanov/kafka-streams-session-window

:leftwards_arrow_with_hook: Kafka Streams DSL: session window example

example-app kafka kafka-consumer kafka-streams kafka-streams-test stream-processing

Last synced: 08 Jul 2025

https://github.com/davidblewett/transistor

Match disparate input and output sources.

async kafka sqs stream-processing tornado zmq

Last synced: 27 Mar 2025

https://github.com/zahidgalea/cdc-with-debezium-in-minikube

Demo of a CDC Application using Debezium, Kafka, SpringBoot app with Oracle as Backend - Running in a minikube context - Currently in DRAFT.

kafka kubernetes pipeline stream-processing

Last synced: 19 May 2026

https://github.com/abitofhelp/optimized_adaptive_pipeline_rs

Adaptive Rust pipeline for high-throughput file processing—dynamic chunking, parallelism, AES/ChaCha encryption, backpressure, and Prometheus/tracing.

adaptive-concurrency backpressure chunking concurrency data-pipeline encryption file-processing metrics observability opentelemetry parallelism prometheus rust stream-processing tracing

Last synced: 05 Oct 2025

https://github.com/avestura/debs2015-wso2-si

🚕 A WSO2 Streaming Integrator solution to DEBS2015 (New York Taxi) challange

distributed-systems iust stream-processing streaming-integrator university wso2

Last synced: 06 Oct 2025

https://github.com/ibmstreams/streamsx.websocket

[Incubation] C++ WebSocket toolkit for IBM Streams

ibm-streams stream-processing toolkit websocket

Last synced: 08 Oct 2025

https://github.com/ibmstreams/sample.forecast_with_r

Sample application using Streams Flow demonstrating the usage of R

r samples stream-processing watson-studio

Last synced: 17 Mar 2025

https://github.com/abitofhelp/adaptive_pipeline

Adaptive Rust pipeline for high-throughput file processing—dynamic chunking, parallelism, AES/ChaCha encryption, backpressure, and Prometheus/tracing.

adaptive-concurrency backpressure chunking concurrency data-pipeline encryption file-processing metrics observability opentelemetry parallelism prometheus rust stream-processing tracing

Last synced: 17 May 2026

https://github.com/easydarwin/easystreamclient

EasyStreamClient是一款支持File、RTSP、RTMP、HTTP等各种流媒体协议的音视频提取功能组件，支持H.264、H.265、PCM、AAC等各种音视频编码格式输出，支持Windows、Linux、Android、iOS、ARM等各种平台。EasyStreamClient support all kinds of streaming protocol, callback video & audio data，you can use this as easy ffmpeg !

ffmpeg-client ffmpeg-sdk stream-client stream-processing

Last synced: 19 Jan 2026

https://github.com/glassflow/pipelines-push-action

This Github Action lets you automate GlassFlow pipelines deployments as code

data data-processing datastreaming deployment github-actions glassflow python real-time stream-processing

Last synced: 19 May 2026

https://github.com/ibmstreams/streamsx.messagehub

Repository to provide easy integration with IBM MessageHub Bluemix Service

ibm-cloud ibm-streams stream-processing toolkit

Last synced: 17 Oct 2025

https://github.com/afsalthaj/fp-generator

The library that solves all your data generation problem !

fs2 functional-programming generator scala stream-processing

Last synced: 16 Jun 2026

https://github.com/dagronf/dsfvaluestream

Generate a stream of values over time

combine random stream-processing swift

Last synced: 19 Oct 2025

https://github.com/havva-nur-ezginci/goodplace

🚀 📅 The Habit Tracker project is a comprehensive Flutter application designed to help users in 🔥 effectively tracking their habits and staying motivated. 🎯

android calendar-widget carousel-slider chat-history chatgpt cloud-firestore dio fcm-notifications firebase-auth firebase-messaging flutter flutter-dotenv flutter-examples http lottie-animation openai provider-state-management stream-processing streaming unsplash

Last synced: 03 Jan 2026

https://github.com/sumolari/stream-size

Get the size of a stream and abort it if threshold is reached

content-length nodejs stream-processing typescript

Last synced: 16 Mar 2026

https://github.com/mre/stream-processors

Just a list of stream-processors

logstash stream-processing

Last synced: 07 Feb 2026

https://github.com/ibmstreams/streamsx.utility

(Incubation) Contains utilities for IBM Streams

ibm-streams stream-processing

Last synced: 27 Feb 2026

https://github.com/ibmstreams/streamsx.opencv

(incubation) Toolkit for processing videos in a Streams application

ibm-streams opencv stream-processing video-processing

Last synced: 17 Mar 2025

https://github.com/ibmstreams/streamsx.shell

Utility toolkit to execute shell commands in a Streams application

ibm-streams stream-processing toolkit

Last synced: 17 Mar 2025

https://github.com/streamshub/streamshub-site

The StreamsHub website source code

event-driven flink kafka stream-processing strimzi

Last synced: 05 Mar 2026

https://github.com/technicalguru/csv

CSV/Excel Utility - Easy CSV and Excel handling for Java applications

csv excel java stream-processing

Last synced: 06 Mar 2026

https://github.com/ibmstreams/streamsx.watsonexplorer

Toolkit for accessing data and analytics results from IBM Watson Explorer (formerly known as IBM Infosphere Data Explorer)

analytics ibm-streams nlp stream-processing text-analysis toolkit watson

Last synced: 17 Mar 2025

https://github.com/ibmstreams/streamsx.resourcemanager.mesos

Repository for resource manager integration with Apache Mesos

ibm-streams mesos resource-manager stream-processing

Last synced: 17 Mar 2025

https://github.com/angeligareta/spark-kafka-cassandra-overview

Second lab for Data-Intensive Computing course at KTH where we use Apache Kafka, Spark, and Cassandra to practice stream processing.

apache-kafka apache-spark cassandra cassandra-server data-intensive id2221 kafka kafka-topic kth scala spark stream-processing

Last synced: 18 Apr 2026

https://github.com/ibmstreams/streamsx.ps

(Incubation) This Streams toolkit allows a simple way for the SPL and C++ operators that are fused inside a single PE (Processing Element) to share the application specific state information.

ibm-streams stream-processing toolkit

Last synced: 17 Mar 2025

https://github.com/narius2030/mlops-image-captioning

An end-to-end MLOps pipeline to develop, train, and deploy an Image Caption model that automatically generates captions for images based on diverse datasets

apache-airflow apache-kafka batch-processing lakehouse mlflow-tracking mlops polars spark-streaming stream-processing

Last synced: 29 Apr 2026

https://github.com/dyzio18/linux_stream_server

Asynchronous stream files using Unix socket and signals.

c file-streaming linux signals socket socket-client socket-server stream-processing unix-socket

Last synced: 29 Apr 2026

https://github.com/manhtdxxx/batch-and-stream-pipeline-via-lakehouse

This project demonstrates a modern Lakehouse architecture supporting both streaming and batch data pipelines, built on Apache Iceberg tables.

airflow batch-processing data-engineering data-visualization docker elt-pipeline hive-metastore iceberg kafka lakehouse medallion-architecture spark stream-processing superset trino

Last synced: 08 May 2026

https://github.com/lerouxrgd/squirrel

Flink window word count example in Clojure

clojure flink stream-processing wordcount

Last synced: 02 Aug 2025

https://github.com/d0ugur4su/kafka-fundamentals-for-java-developers

Studies related about Kafka-fundamentals-for-java-developers Udemy course

kafka kafka-streams stream-processing

Last synced: 24 Feb 2025

https://github.com/souravray/oktavhefts

kafka + node streams - a proof of concept

kafka kafka-node stream-processing

Last synced: 14 Oct 2025

https://github.com/jaehyeon-kim/oml-digital-twin-hotrolling

A streaming Digital Twin of a steel hot rolling mill demonstrating Online Machine Learning (OML) with Apache Kafka, Apache Flink and MOA to handle real-time concept drift.

apache-flink apache-kafka concept-drift digital-twin discrete-event-simulation dynamic-des industry-4-0 kotlin massive-online-analysis moa online-machine-learning python stream-processing

Last synced: 05 Jun 2026

https://github.com/broneq/sql-stream-queue

SQL stream queue implementation- inspired on redis

mysql php php5 sql stream stream-processing

Last synced: 16 May 2026

https://github.com/bernardini687/hack_assembler

assembler for nand2tetris (part 1) final project

hackassembler nand2tetris nand2tetris-assembler stream-processing

Last synced: 31 Aug 2025

https://github.com/ibmstreams/pypi.streamsx.standard

Python packages for IBM Streams Standard Toolkit

ibm-streams pypi python python-package stream-processing

Last synced: 15 Jul 2025

https://github.com/ibmstreams/streamsx.mail

Toolkit for sending or receiving emails in a Streams application.

email ibm-streams stream-processing toolkit

Last synced: 17 Mar 2025

https://github.com/ibmstreams/sample.edge-mnist-flows

MNIST digit recognition sample for Streams Flows

edge-computing samples stream-processing

Last synced: 17 Mar 2025

https://github.com/msachi/kafka-streams-demo

Build a demo app with Kafka Streams

kafka kafka-streams stream-processing streams-application

Last synced: 21 Jul 2025

https://github.com/ibmstreams/streamsx.sttgateway

This toolkit does Speech To Text transcription using an external provider such as the IBM Watson STT cloud service.

ibm-cloud ibm-cloud-private ibm-streams speech-to-text stream-processing stt toolkit watson-speech-to-text

Last synced: 17 Mar 2025

https://github.com/ymtszw/elm-broker

Data stream buffer for Elm application, inspired by Apache Kafka

broker buffer elm elm-lang kafka stream-processing

Last synced: 23 Mar 2025

https://github.com/kzmlabs/flink-statefun

Actively maintained continuation of Apache Flink Stateful Functions — updated for Flink 2.2.0 and Java 21, published to Maven Central as io.github.kzmlabs.flinkstatefun. Stateful serverless stream processing on Kubernetes.

actor-model apache-flink distributed-systems event-driven event-driven-architecture flink flink-kubernetes-operator java java21 kafka kubernetes maven-central rocksdb serverless stateful-functions stateful-stream-processing statefun stream-processing

Last synced: 29 Apr 2026

https://github.com/mariuszpaluch2001/stream-processing-graph-framework

Framework for analysis and stream processing of dynamic graphs.

framework graph-algorithms graphs stream-processing

Last synced: 11 Jun 2026

https://github.com/austinv11/pypeline

A simple data pipeline builder for Python 3+

data leveldb pypeline python python3 stream-processing

Last synced: 20 Aug 2025

https://github.com/coot/speedy

Super fast execution of gulp tasks.

gulp-tasks nodejs stream-processing vim

Last synced: 12 Apr 2026

https://github.com/ori88c/data-oriented-slim-deque

A slim and efficient in-memory Deque (doubly-ended queue) for Node.js projects, designed with Data-Oriented Design principles and implemented using a cyclic buffer to optimize memory layout through sequential item allocation. A notable real-world use case is maintaining the minimum or maximum item in a data stream.

circular-buffer cyclic-buffer data-oriented-design data-structure deque dequeue doubly-ended-queue es2020 in-memory javascript js monotonic-stack node-js nodejs sequential stream-processing ts typescript

Last synced: 28 Jan 2026

https://github.com/aymendaoudi/electric-vehicle-charging-simulator

Simulation, Ingestion and ETL-ing data of millions of EV charging sessions by thousands of EVs in thousands of stations around the world.

apache-airflow apache-spark batch-processing data-engineering data-lake data-warehouse kafka kafka-connect lake-fs minio mongodb postgresql python3 simpy spark stream-processing

Last synced: 15 Apr 2026

https://github.com/ibmstreams/streamsx.weather

Toolkit for accessing data from the Insights for Weather Bluemix service

ibm-streams stream-processing watson weather weather-bluemix-service

Last synced: 17 Mar 2025

https://github.com/hhandoko/streaming-applications

Code repository for "Streaming Applications with Akka Streams" talk at geekcamp Indonesia 2017

akka akka-streams flow-based-programming stream-data stream-processing

Last synced: 16 May 2026

https://github.com/theveryhim/stream-data-analysis

Filtering, sampling and analysis on a news data set using stream simulation.

pyspark stream-data-analysis stream-processing

Last synced: 18 Jul 2025

https://github.com/ibmstreams/streamsx.slack

Toolkit to enable IBM Streams application developer to integrate Streams application with Slack

slack slack-webhooks stream-processing toolkit

Last synced: 15 Apr 2026

https://github.com/jbloch100/scalable-analytics-pipeline

Python-based real-time data pipeline using Kafka and Spark for streaming analytics

big-data data-pipeline kafka python spark stream-processing

Last synced: 19 May 2026

https://github.com/vubacktracking/stream-data-processing

Streaming data processing pipeline using Spark, PostgreSQL, Debezium, Kafka, Minio, Delta Lake, Trino and DBeaver

dbeaver debezium delta-lake kafka spark spark-streaming stream-processing trino

Last synced: 31 Jan 2026

https://github.com/arivum/linebuf

This module allows JSON stream processing via line-buffered JSON objects.

golang json library serialization stream-processing

Last synced: 11 Jan 2026

https://github.com/gabraganca/udacity-data-streaming-project-1

Udacity Data Streaming - Project Optimizing Public Transportation

kafka kafka-streams stream-processing udacity

Last synced: 15 Apr 2026

https://github.com/earthdaily/analyticstreamformater

The Analytic Stream Formater is listening agricultural field related (new satellite image, new analytic generated) events coming from <geosys/> platform triggering analytic fetching, formatting and publication to an Azure Blob storage.

agricultural agriculture dotnet-core dotnetcore earth-observation geospatial geospatial-analysis geospatial-data geospatial-processing satellite satellite-data satellite-imagery satellite-imagery-analysis stream-processing

Last synced: 25 Apr 2026

https://github.com/mohamedawnallah/learning-apache-flink

Document my Apache Flink learning experience

apacheflink batch-processing java stream-processing

Last synced: 28 Jun 2025

https://github.com/will-white/dahua-companion

Small, simple, and fast event driven Dahua (Amcrest) doorbell processor for home automation integration

containerization event-driven golang mqtt pubsub stream-processing

Last synced: 12 May 2026

https://github.com/kathiravanmuthaiah/reactive-mqtt-file-pipeline

Reactive File Processing Pipeline in Java using MQTT, Spring Boot (Project Reactor) & Quarkus (Mutiny). Used for reactive streams, microservices file ingestion demo.

developer-demo file-pipeline java microservices mqtt mutiny project-reactor quarkus reactive-programming spring-boot stream-processing

Last synced: 15 Apr 2026

https://github.com/reubano/devcraft-workshop

Materials for the DevCraft workshop on stream processing

data functional-programming meza python riko stream-processing tutorial

Last synced: 04 May 2026

https://github.com/michaeljklein/stream-timers

A Haskell package for working with high-performance streamed operations: some folds and experiments

haskell-library performance stream-processing

Last synced: 14 Feb 2026

https://github.com/firoz-ahmad-likhon/kafka-flink-clickstream

Production-grade real-time streaming pipeline using Apache Kafka and Apache Flink to simulate real-world streaming workflows.

apache-flink apache-kafka clickstream flink kafka stream-processing streaming

Last synced: 01 May 2026

https://github.com/e2fyi/streams

A library of custom NodeJS streams to do various things.

auto-increment mongoose nodejs stream stream-processing

Last synced: 20 Apr 2026

https://github.com/nasim-raj-laskar/real-time-aircraft-engine-predictive-maintenance-system

GRU-based aircraft engine Remaining Useful Life (RUL) forecasting platform with Redis-backed online feature serving, streaming telemetry ingestion, batched WebSocket inference, Prometheus/Grafana observability, Evidently drift monitoring, and containerized MLOps infrastructure.

docker evidently-ai grafana gru mlops prometheus pyflink redis rul s3 stream-processing tensorflow time-series vuejs websocket

Last synced: 11 Jun 2026

https://github.com/jerrinot/flink-connector-questdb

Apache Flink Table & SQL Connector for QuestDB

flink questdb stream-processing streaming time-series

Last synced: 16 Jan 2026

https://github.com/irc-sphere/hyperstream-dockerfiles

Docker containers for HyperStream (and its dependencies)

docker docker-image hyperstream stream-processing

Last synced: 02 May 2026