Projects in Awesome Lists tagged with dataflow

https://github.com/pathwaycom/pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

batch-processing data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics kafka machine-learning-algorithms pathway python real-time rust stream-processing streaming time-series-analysis

Last synced: 08 Jan 2026

https://github.com/marimo-team/marimo

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. All in a modern, AI-native editor.

artificial-intelligence dag data-science data-visualization dataflow developer-tools machine-learning notebooks pipeline python reactive sql web-app

Last synced: 23 Jan 2026

https://github.com/jerosoler/drawflow

Simple flow library 🖥️🖱️

dataflow dataflow-programming drawflow editor flow flow-based-programming flowchart graph-editor javascript javascript-library nodebased visual-programming vue

Last synced: 13 May 2025

https://github.com/jerosoler/Drawflow

Simple flow library 🖥️🖱️

dataflow dataflow-programming drawflow editor flow flow-based-programming flowchart graph-editor javascript javascript-library nodebased visual-programming vue

Last synced: 14 Mar 2025

https://github.com/thi-ng/umbrella

⛱ Broadly scoped ecosystem & mono-repository of 206 TypeScript projects (and ~185 examples) for general purpose, functional, data driven development

color data-structures dataflow dsl functional-programming geometry html monorepo parser-combinators reactive-programming shadergraph streams transducers typescript ui vectors visualization webassembly webgl ziglang

Last synced: 14 May 2025

https://github.com/nextflow-io/nextflow

A DSL for data-driven computational pipelines

aws bioinformatics cloud dataflow docker groovy hello hpc nextflow pipeline pipeline-framework reproducible-research reproducible-science sge singularity singularity-containers slurm workflow-engine

Last synced: 13 May 2025

https://github.com/joernio/joern

Open-source code analysis platform for C/C++/Java/Binary/Javascript/Python/Kotlin based on code property graphs. Discord https://discord.gg/vv4MH284Hc

binary c code-analysis code-browser code-property-graph controlflow cpg cpp dataflow fuzzy-parsing ghidra graph java javabytecode javascript llvm query-language scala syntax-tree

Last synced: 24 Jan 2026

https://github.com/spotify/scio

A Scala API for Apache Beam and Google Cloud Dataflow.

batch beam bigquery data dataflow google-cloud ml scala scio streaming

Last synced: 23 Apr 2025

https://github.com/Bearer/bearer

Code security scanning tool (SAST) to discover, filter and prioritize security and privacy risks.

appsec code-quality compliance dataflow devsecops devsecops-tools gdpr owasp privacy sast security security-audit security-automation security-scanner security-tools static-analysis static-code-analysis vulnerabilities vulnerability

Last synced: 01 Apr 2025

https://github.com/bearer/bearer

Code security scanning tool (SAST) to discover, filter and prioritize security and privacy risks.

appsec code-quality compliance dataflow devsecops devsecops-tools gdpr owasp privacy sast security security-audit security-automation security-scanner security-tools static-analysis static-code-analysis vulnerabilities vulnerability

Last synced: 29 Jan 2026

https://github.com/python-security/pyt

A Static Analysis Tool for Detecting Security Vulnerabilities in Python Web Applications

abstract-syntax abstract-syntax-tree control-flow-graph dataflow dataflow-analysis fixed-point fixed-point-analysis flask program-analysis pyt python python3 security static-analysis static-code-analysis taint taint-analysis

Last synced: 14 May 2025

https://github.com/dora-rs/dora

DORA (Dataflow-Oriented Robotic Architecture) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed dataflow capabilities. Applications are modeled as directed graphs, also referred to as pipelines.

dataflow embodied-ai low-latency robotics rust

Last synced: 08 Jan 2026

https://github.com/willcrichton/flowistry

Flowistry is an IDE plugin for Rust that helps you focus on relevant code.

dataflow rust static-analysis vscode

Last synced: 13 May 2025

https://github.com/newcat/baklavajs

Graph / node editor in the browser using VueJS

dataflow dataflow-programming editor flow flow-based-programming flow-control graph node node-editor vuejs

Last synced: 13 May 2025

https://github.com/bytewax/bytewax

Python Stream Processing

data-engineering data-processing data-science dataflow machine-learning python rust stream-processing streaming-data

Last synced: 13 May 2025

https://github.com/serverless/event-gateway

React to any event with serverless functions across clouds

dataflow event-driven event-router functions-as-a-service golang pubsub serverless

Last synced: 05 Aug 2025

https://github.com/trustmaster/goflow

Flow-based and dataflow programming library for Go (golang)

data-flow dataflow fbp go goflow

Last synced: 15 May 2025

https://github.com/ipyflow/ipyflow

A reactive Python kernel for Jupyter notebooks.

dataflow developer-tools highlighting ipyflow jupyter jupyter-notebooks jupyterlab lineage nbsafety notebooks pypi python reactivity static-analysis static-code-analysis tracing

Last synced: 12 Dec 2025

https://github.com/scipipe/scipipe

Robust, flexible and resource-efficient pipelines using Go and the commandline

bioinformatics bioinformatics-pipeline cheminformatics dataflow fbp go golang pipeline scientific-workflows scipipe workflow workflow-engine

Last synced: 15 Dec 2025

https://github.com/cocoindex-io/cocoindex

ETL framework to turn your data AI-ready - with realtime incremental updates and support custom logic like lego.

ai change-data-capture data data-engineering data-indexing data-infrastructure data-processing dataflow etl help-wanted indexing knowledge-graph llm pipeline python rag real-time rust semantic-search streaming

Last synced: 31 Jan 2026

https://github.com/shaiwz/data-platform-open

🔥🔥🔥可视化拖拽式大数据集成平台、大数据平台、大数据，包含数据流、数据源、数据对齐、查询模板、完善的监控等。像画流程图一样且无代码方式同步、清洗数据。

big-data dataflow doris java kafka starrocks

Last synced: 20 Nov 2025

https://github.com/raftlib/raftlib

The RaftLib C++ library, streaming/dataflow concurrency via C++ iostream-like operators

c-plus-plus cmake dataflow dataflow-programming dataflow-structure dataflows dsl hpc ipc machine opencv parallel pthreads qthread-library qthreads raftlib runtime streaming thread thread-library

Last synced: 16 May 2025

https://github.com/OWASP/pytm

A Pythonic framework for threat modeling

data-flow-diagram dataflow dfd diagram pythonic-framework secure-development sequence-diagram threat-modeling threat-modeling-from-code threats

Last synced: 18 Oct 2025

https://github.com/RaftLib/RaftLib

The RaftLib C++ library, streaming/dataflow concurrency via C++ iostream-like operators

c-plus-plus cmake dataflow dataflow-programming dataflow-structure dataflows dsl hpc ipc machine opencv parallel pthreads qthread-library qthreads raftlib runtime streaming thread thread-library

Last synced: 15 Mar 2025

https://github.com/composewell/streamly

High performance, concurrent functional programming abstractions

arrays async concurrency dataflow filesystem folds frp haskell logic-programming loops modular network non-determinism parsers pipes reactive-programming stream-fusion streaming unfolds unicode

Last synced: 13 Apr 2025

https://github.com/xilinx/finn

Dataflow compiler for QNN inference on FPGAs

compiler dataflow fpga neural-network quantization

Last synced: 11 Oct 2025

https://github.com/Xilinx/finn

Dataflow compiler for QNN inference on FPGAs

compiler dataflow fpga neural-network quantization

Last synced: 20 Mar 2025

https://github.com/nipy/nipype

Workflows and interfaces for neuroimaging packages

big-data brain-imaging brainweb data-science dataflow dataflow-programming neuroimaging python workflow-engine

Last synced: 21 Oct 2025

https://github.com/lw-lin/streaming-readings

Streaming System 相关的论文读物

apache-spark dataflow drizzle flink heron millwheel s4 spark-streaming spe storm stream-processing stream-processing-engine streaming streaming-engine

Last synced: 04 Apr 2025

https://github.com/jaystack/repatch

Dispatch reducers

async-actions data-management dataflow dispatch middleware react reducer redux redux-thunk thunk

Last synced: 26 Oct 2025

https://github.com/google/shenzhen-go

Experimental visual Go environment

dataflow flow flowchart go golang graph visual

Last synced: 15 Dec 2025

https://github.com/asg017/dataflow

An experimental self-hosted Observable notebook editor, with support for FileAttachments, Secrets, custom standard libraries, and more!

dataflow dataflow-notebooks observable-notebooks observablehq

Last synced: 23 Sep 2025

https://github.com/chigraph/chigraph

A visual systems language for beginners compiled using LLVM

chigraph dataflow dataflow-programming language language-learning learn-to-code llvm

Last synced: 05 Apr 2025

https://github.com/wotbrew/relic

Functional relational programming for Clojure(Script).

clojure dataflow incremental-view-maintenance logic relational-algebra relations sql

Last synced: 22 Oct 2025

https://github.com/pothosware/pothoscore

The Pothos data-flow framework

cpp11 dataflow flowframework networking parallel-computing pothos pothos-framework sdr

Last synced: 17 Dec 2025

https://github.com/pothosware/PothosCore

The Pothos data-flow framework

cpp11 dataflow flowframework networking parallel-computing pothos pothos-framework sdr

Last synced: 05 Apr 2025

https://github.com/spotify/pythonflow

:snake: Dataflow programming for python.

dataflow machine-learning python

Last synced: 05 Apr 2025

https://github.com/ghostiam/vue-blocks

Vue2 dataflow graph editor

blocks blueprints dataflow editor graph visual-programming vuejs2 workflow

Last synced: 03 Apr 2025

https://github.com/chaffelson/nipyapi

A convenient Python wrapper for Apache NiFi

api-wrapper automation dataflow nifi python

Last synced: 17 Jan 2026

https://github.com/Chaffelson/nipyapi

A convenient Python wrapper for Apache NiFi

api-wrapper automation dataflow nifi python

Last synced: 28 Mar 2025

https://github.com/meemoo/meemooapp

Creative apps to use, build, share, and hack in the browser.

animation dataflow media visual-programming vj web-art webapp

Last synced: 03 Apr 2025

https://github.com/googlecloudplatform/data-analytics-golden-demo

An end to end demo of Google's Cloud data and analytic stack.

bigdata bigquery composer dataflow dataproc gcp

Last synced: 16 May 2025

https://github.com/googlecloudplatform/fraudfinder

Fraudfinder: A comprehensive lab series on how to build a real-time fraud detection system on Google Cloud

bigquery bigquery-ml dataflow google-cloud-platform machine-learning mlops mlpipelines vertex-ai

Last synced: 16 May 2025

https://nnmer.github.io/azure-services-map/

A visual representation and reference to Azure services

azure azure-services dataflow interconnection service-data-flow

Last synced: 11 May 2025

https://github.com/microflo/microflo

Live dataflow programming for microcontrollers and embedded

arduino dataflow fbp fbp-runtime flowhub microcontroller

Last synced: 07 Apr 2025

https://github.com/erdos-project/erdos

Dataflow system for building self-driving car and robotics applications.

autonomous-vehicles dataflow drone robot robot-framework robot-programming robotics ros rust self-driving-car

Last synced: 05 Apr 2025

https://github.com/skydoves/chamber

⚖️ A lightweight Android lifecycle-aware and thread-safe pipeline for communicating between components with custom scopes.

android architecture chamber dataflow kotlin lifecycle-aware scope skydoves

Last synced: 13 Apr 2025

https://github.com/skydoves/Chamber

⚖️ A lightweight Android lifecycle-aware and thread-safe pipeline for communicating between components with custom scopes.

android architecture chamber dataflow kotlin lifecycle-aware scope skydoves

Last synced: 13 Apr 2025

https://github.com/googlecloudplatform/df-ml-anomaly-detection

Streaming Anomaly Detection Solution by using Pub/Sub, Dataflow, BQML & Cloud DLP

anomaly-detection bqml cybersecurity dataflow dlp kmeans-clustering log network pubsub

Last synced: 09 Apr 2025

https://github.com/maestro-project/maestro

An analytical cost model evaluating DNN mappings (dataflows and tiling).

dataflow deep-learning deep-neural-networks

Last synced: 27 Mar 2025

https://github.com/GoogleCloudPlatform/df-ml-anomaly-detection

Streaming Anomaly Detection Solution by using Pub/Sub, Dataflow, BQML & Cloud DLP

anomaly-detection bqml cybersecurity dataflow dlp kmeans-clustering log network pubsub

Last synced: 26 Mar 2025

https://github.com/cda-group/arcon

State-first Streaming Applications in Rust

arrow data-analytics dataflow distributed-computing rust stream-processor

Last synced: 27 Apr 2025

https://github.com/flowbase/flowbase

A Flow-based Programming inspired micro-framework / un-framework for Go (Golang)

dataflow fbp fbp-frameworks flow-based-programming golang micro-framework

Last synced: 17 Jan 2026

https://blocks-editor.github.io/blocks/

Blocks. An online drag-and-drop smart contract builder.

blockchain cryptocurrency dapps-development dataflow decentralized-applications defi dfinity editor graph icp internet-computer javascript low-code motoko motoko-language no-code online react smart-contracts

Last synced: 07 May 2025

https://github.com/Blocks-Editor/blocks

Blocks. An online drag-and-drop smart contract builder.

blockchain cryptocurrency dapps-development dataflow decentralized-applications defi dfinity editor graph icp internet-computer javascript low-code motoko motoko-language no-code online react smart-contracts

Last synced: 01 Apr 2025

https://github.com/googlecloudplatform/bigquery-data-lineage

Reference implementation for real-time Data Lineage tracking for BigQuery using Audit Logs, ZetaSQL and Dataflow.

bigdata bigquery data-catalog data-governance data-lineage data-management dataflow zetasql

Last synced: 20 Oct 2025

https://github.com/msgflo/msgflo

Distributed Flow-Based Programming via message queues

amqp dataflow distributed fbp fbp-runtime flowhub iot-platform mqtt pubsub

Last synced: 07 Apr 2025

https://github.com/googlegenomics/gcp-variant-transforms

GCP Variant Transforms

beam bigquery dataflow vcf-files

Last synced: 07 May 2025

https://github.com/google/megalista

First Party data integration solution built for marketing teams to enable audience and conversion onboarding into Google Marketing products (Google Ads, Campaign Manager, Google Analytics).

audience-targeting audiences bigquery conversions customermatch data-integration dataflow google googleads googleanalytics python

Last synced: 24 Sep 2025

https://github.com/yubowenok/visflow

Web-based Dataflow Framework for Visual Data Exploration

dataflow tabular-data visflow visualization visualization-framework

Last synced: 11 Apr 2025

https://github.com/allegro/bigflow

A Python framework for data processing on GCP.

airflow-dag beam bigquery composer dag dataflow dataproc gcp python python-framework workflows

Last synced: 08 Apr 2025

https://github.com/anna-geller/dataflow-ops

Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate

analytics analytics-engineering automation aws cicd data data-engineering data-engineering-infrastructure data-engineering-pipeline data-science dataflow dataflow-ops infrastructure-as-code observability orchestration pipeline prefect python serverless

Last synced: 24 Mar 2025

https://github.com/martinklepsch/derivatives

🌱 Your companion to create derived values from a single source (atom)

clojurescript dataflow reactive rum

Last synced: 05 May 2025

https://github.com/anna-geller/prefect-deployment-patterns

Code examples showing flow deployment to various types of infrastructure

automation aws data data-engineering data-engineering-infrastructure data-engineering-pipeline data-engineering-team data-products data-science dataflow dataflow-ops orchestration pipeline prefect python serverless serverless-framework

Last synced: 22 Jun 2025

https://github.com/radiantone/entangle

A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs.

artificial-intelligence containers dataflow dataflow-engine decorator-composition devops gpu gpu-computing hpc parallel parallel-processes parallel-workflows python3 scripting supercomputing workflow-composition workflow-managers

Last synced: 20 Jul 2025

https://github.com/Refefer/Dampr

Python Data Processing library

batch-processing dataflow machine-learning mapreduce

Last synced: 19 Jul 2025

https://github.com/anna-geller/prefect-dataplatform

Example repository showing how to build a data platform with Prefect, dbt and Snowflake

analytics analytics-engineering automation data-engineering data-platform data-warehousing dataflow dataflow-ops dbt orchestration prefect python snowflake sql

Last synced: 25 Oct 2025

https://github.com/JasonThon/lightflus

A Lightweight, Cloud-Native Stateful Distributed Dataflow Engine

dataflow deno distributed-computing rust streaming typescript

Last synced: 27 Apr 2025

https://github.com/asyncvlsi/act

ACT hardware description language and core tools.

asynchronous-circuits asynchronous-vlsi cad chp circuit-simulator communicating-hardware-processes dataflow dataflow-programming design-automation eda hardware-description-language hdl language production-rules prs vlsi vlsi-cad

Last synced: 14 Mar 2025

https://github.com/googlecloudplatform/dlp-dataflow-deidentification

Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP

beam bigquery data dataflow dlp pii tokenization

Last synced: 11 Apr 2025

https://github.com/googlecloudplatform/google-cloud-eclipse

Google Cloud Platform plugin for Eclipse

app-engine dataflow eclipse eclipse-ide eclipse-neon eclipse-oxygen eclipse-plugin google google-cloud-platform google-cloud-sdk google-cloud-tools

Last synced: 20 Oct 2025

https://github.com/lilactown/flex

flex is a reactive signal library for Clojure(Script)

clojure clojurescript dataflow reactive reactive-programming reactivity signals state-management

Last synced: 07 Apr 2025

https://github.com/hasielhassan/plumbermanager

A helper tool to design CG Pipeline interactive diagrams and data flow documentation

animation cgi dataflow diagrams documentation flowchart graph-editor graph-editor-gui node-based-ui node-editor nodebased nodes pipeline pyside python qt vfx vfx-graph vfx-pipeline workflow

Last synced: 10 Aug 2025

https://github.com/DFiantHDL/DFHDL

DFiant HDL (DFHDL): A Dataflow Hardware Descripition Language

asic dataflow dataflow-programming fpga hdl

Last synced: 11 May 2025

https://github.com/kestra-io/docs

Documentation for Kestra — an event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

dataflow hacktoberfest orchestration pipeline workflow workflow-engine

Last synced: 28 Jan 2026

https://github.com/automaticanalysis/automaticanalysis

Automatic Analysis (aa)

big-data dataflow diffusion eeg fmri matlab neuroimaging parallel-computing pipeline workflow-automation workflow-engine

Last synced: 12 Jan 2026

https://github.com/pothosware/pothosflow

GUI frontend and designer tool for the Pothos framework

block-design dataflow graphical gui pothos pothos-framework qt sdr

Last synced: 12 Apr 2025

https://github.com/pothosware/PothosFlow

GUI frontend and designer tool for the Pothos framework

block-design dataflow graphical gui pothos pothos-framework qt sdr

Last synced: 05 Apr 2025

https://github.com/reactivesets/toubkal

Fully reactive programming for nodejs and the browser

dataflow javascript publish-subscribe reactive reactive-dataflows transactional

Last synced: 13 May 2025

https://github.com/fibo/dflow

is a minimal Dataflow programming engine

dataflow dataflow-programming

Last synced: 13 Apr 2025

https://github.com/googlecloudplatform/dataflow-contact-center-speech-analysis

Speech Analysis Framework, a collection of components and code from Google Cloud that you can use to transcribe audio files to create analytics.

cloud-functions data-loss-prevention dataflow google-cloud natural-language-processing speech-to-text

Last synced: 20 Oct 2025

https://github.com/google/grizzly

End-to-end DataOps platform deployed by Terraform.

airflow bigquery cloud-sql cloud-storage composer data-catalog data-lineage data-loss-prevention dataflow dataops dataops-platform gcp git google-cloud google-cloud-platform pubsub spanner terraform

Last synced: 27 Apr 2025

https://github.com/vectaport/flowgraph

Flowgraph package for scalable asynchronous system development

dataflow distributed scalable

Last synced: 30 Dec 2025

https://github.com/daanvdh/javadataflow

Creating Data Flow Graphs from java input classes

control-flow-graph controlflow controlflowgraph data-flow-analysis data-flow-graph dataflow java javadataflow-graph javaparser

Last synced: 14 Jan 2026

https://github.com/mchmarny/github-activity-counter

Cloud Run service for GitHub event Webhook to monitor repo or org activity in real-time in Stackdriver and analyze activity through ad-hoc SQL queries in BigQuery

bigquery cloudrun dataflow github pubsub stackdriver webhook