{"id":15723697,"url":"https://github.com/andrewdavidmackenzie/flow","last_synced_at":"2025-04-11T00:51:58.600Z","repository":{"id":37706400,"uuid":"47028388","full_name":"andrewdavidmackenzie/flow","owner":"andrewdavidmackenzie","description":"Exploration of a data-flow programming paradigm","archived":false,"fork":false,"pushed_at":"2025-04-10T14:46:00.000Z","size":1748691,"stargazers_count":30,"open_issues_count":188,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-11T00:51:47.540Z","etag":null,"topics":["data-flow","distributed-computing","flow","iced","webassembly"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andrewdavidmackenzie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"ko_fi":"andrew","patreon":"andrewmackenzie"}},"created_at":"2015-11-28T15:21:34.000Z","updated_at":"2025-03-20T06:59:21.000Z","dependencies_parsed_at":"2023-10-16T08:44:07.145Z","dependency_job_id":"d2a62083-27ae-48ad-9876-d89eedbbf39e","html_url":"https://github.com/andrewdavidmackenzie/flow","commit_stats":{"total_commits":3706,"total_committers":4,"mean_commits":926.5,"dds":0.04533189422558015,"last_synced_commit":"095d19f985f55459cf29a06bad0ef62d549bf190"},"previous_names":[],"tags_count":265,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewdavidmackenzie%2Fflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewdavidmackenzie%2Fflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewdavidmackenzie%2Fflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewdavidmackenzie%2Fflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andrewdavidmackenzie","download_url":"https://codeload.github.com/andrewdavidmackenzie/flow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322609,"owners_count":21084336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-flow","distributed-computing","flow","iced","webassembly"],"created_at":"2024-10-03T22:12:53.334Z","updated_at":"2025-04-11T00:51:58.560Z","avatar_url":"https://github.com/andrewdavidmackenzie.png","language":"Rust","funding_links":["https://ko-fi.com/andrew","https://patreon.com/andrewmackenzie","https://www.patreon.com/andrewmackenzie"],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/andrewdavidmackenzie/flow.svg?branch=master)](https://travis-ci.org/andrewdavidmackenzie/flow)\n[![codecov](https://codecov.io/gh/andrewdavidmackenzie/flow/branch/master/graph/badge.svg)](https://codecov.io/gh/andrewdavidmackenzie/flow)\n[![Generic badge](https://img.shields.io/badge/macos-supported-Green.svg)](https://shields.io/)\n[![Generic badge](https://img.shields.io/badge/linux-supported-Green.svg)](https://shields.io/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n# Welcome!\nWelcome to `flow`, for defining, compiling and running parallel, \n[dataflow programs](https://en.wikipedia.org/wiki/Dataflow_programming) like the one below (that is a visual \nrepresentation generated by the compiler from the flow definition and rendered with graphviz) of a \nflow program to generate a sequence of fibonacci numbers.\n\nIf you are a programmer, your intuition will probably tell you a lot already about how `flow` works\nwithout any explanation.\n![First flow](first.svg)\nThis flow program generates a fibonacci series on standard output.\nIt is one of the examples ([fibonacci](flowr/examples/fibonacci/DESCRIPTION.md)) in the `flowr` crate \nthat is part of the`flow` project, and the first thing I got working (much to my own delight!).\n\nThe two inputs to `add` (`i1` and `i2`) are initialized \"once\" (at startup) with 0 and 1. \nThe output (`sum`) is then fed back to input `i2` and the value presented at input `i2` previously is fed back to\ninput `i1`. \nThe output (`sum`) is also sent to the default (unnamed) input of the `stdout` function which prints\nthe value to standard output.\nThe program runs until integer overflow causes no output to be produced and it stops.\n\n\nHere you can see it being executed by the `flowrgui` graphical flow runner:\n\n![Fibonacci Series](book/images/fibonacci.gif)\n\n\n## What is a `dataflow program`?\nA data flow program consists of a graph of processes (hierarchical in this case, as a process within it can be another\ngraph of processes, and so on down) that act on data that flow between them on defined connections. \n- it is declarative and defines what processes are used, and how they are connected\n- processes are small and single purpose and \"pure\". They get a series of inputs, execute an algorithm\n  (probably written in some procedural language) and produce an output.\n- The application used to run a flow (a \"flow runner\") provides ways for it to interact with the\nexecution environment via \"impure\" functions, for things like Stdio, File System, etc.\n\n## What characteristics do they have?\nWhy is writing a `dataflow program` something interesting to explore in the first place? \n\nWell, data flow programs define the program in terms of the processing steps that needs to be done on data\nand the dependencies between the data, making them inherently parallelizable and distributable (and in\nmy mind, kind of the minimal essence or expression of the algorithm). \n\nProcesses only run on data when it is available, making them \"event driven\" (where the \"event\" is the availability\nof data...or alternatively, the data expresses an event that needs processing done on it and some output created).\nThey are not focussed so much on the procedural steps that need to be done and the control flow of the same, \nbut on the required transformations to the data and on data flow _through_ the program.\n\n## What does the `flow` project include?\nCurrently, flows are defined declaratively in a text file (toml, json or yaml are supported) that is then compiled\nto a flow manifest, which is executed.\n\nThe `flow` project includes:\n- Compiler: a library and a binary (`flowclib` and `flowc`) for compiling flows\n- Runner: a library (`flowrlib`) and two binaries for running flows:\n  - `flowrcli` - default command line runner and debugger to use from a terminal\n  - `flowrgui` - a GUI application for running and debugging flows\n- Job executor: `flowrex` binary can be discovered (on same machine or local network) \nby a runner and used to execute jobs, distributing execution in a basic fashion\n- Standard Library: `flowstdlib` library of pre-defined flows and functions that can be re-used in flows\n- Examples: A set of example flows to illustrate flow programming (more to come!)\n   - Here is a screenshot of the mandlebrot example rendering a monochrome mandlebrot set using `flowrgui`\n\n![Mandlebrot set](book/images/mandlebrot.png)\n\n- The [book](SUMMARY.md) covers defining flows, the runtime semantics, command line tool options and how to use them, \nthe `flowstdlib` library functions and flows, `flowrcli/flowrgui`'s context functions and more. It is published online \n[here](https://mackenzie-serres.net/flow/book/book_intro.html).\n \n## What made me want to do it?\nYou can read more about what made me want to do this project, based on ideas gathered over a few decades\non and off (combined with looking for a \"real\" project to use to learn rust!) in the book's \n[Inspirations for flow](book/introduction/inspirations.md) section. The core reason is: I wanted to know\nif I could do it and make it work, having stopped being a Software Engineer many years ago, based on rough ideas \nand intuition I had in my head (no real formal knowledge in this area or reading of books and papers - \nthat came later *after* I did it).\n\nI implemented the runtime \"semantics\" as needed as I implemented the examples. It's been a journey of discovery:\nof writing something like this (for me), learning rust in the process and learning how such a programming \nparadigm could work. I learned it could work, but requires a change in how you think about programming \n(with procedural programming so ingrained in us). Sometimes I struggled to think about relatively simple\nalgorithms in a completely new way. This reminded me of when I got stuck trying to write a loop in Prolog, in\nUniversity. If you're trying to write a loop ....\"you're thinking about it wrong\".\n\n## Installing\nYou can install many of the crates from crates.io, but due to unresolved issues in packaging\nnon-source files, a total working installation cannot yet be achieved using `cargo install`.\n\nThe workaround in the meantime is to clone the repo and build all from source (see below).\n\n## Building `flow`\nFor more details on how to build flow locally and contribute to it, please see\n[building flow](book/developing/building.md)\nInstall the dependencies with `make config`, then run `make`, which builds everything and installs the `flowc` and\n`flowr` and`flowrex` binaries.\n\nNOTE: Building of `flowstdlib` the first time will take a long time, as it is compiling many rust functions to \nWebAssembly.\n\n## Running your first 'flow'\nWith `flowc` and `flowr` installed, you can run the 'fibonacci' example flow using:\n\n`cargo run --example fibonacci`\n\nYou should get a fibonacci series of numbers output to the terminal.\n\nThe [first flow](book/first_flow/first_flow.md) section of the book walks you through it.\n\n## Tech decisions\n### Job/Work Distribution - with Threads\nFlow was started before async landed in rust, and so it uses a manually managed thread pool for executing \n\"jobs\" (functions with their set of inputs). Rewriting in async rust would make sense in some areas but\nbe quite a chunk of disruptive work, so I haven't done it yet.\n\n### Message Passing - with Zero MQ\nI started with channels for distributing Jobs and results between threads.\nI wanted to enable distributing work across the (local for now) network and so moved to ZeroMQ message queues \nand passing messages. This is used for inter-thread and inter-process message passing indistinctly. \nZeroMQ rust bindings don't support all socket types (at the time of writing) so I had to use the REQ/REP\npattern, which has some restrictions on the protocol, and which end writes first - which I also had to\nwork around. For a while I kept \n\n### Discovery - with mDNS and beacons\nTo discover \"executors\" (processes with threads, able to execute flow jobs) on the network I wrote my own \nsmall discovery crate (as I couldn't get libp2p mDNS or other mDNS crates to work). Not very happy with it\nas it frequently ties up ports and other issues I have had to work around.\n\n### Portability - with WebAssembly (WASM)\nLibrary functions are compiled both to native and optionally linked statically to a flow runner with a \nfeature, AND compiled to wasm (and their size optimized to around 110KB) and described in a library manifest.\nLibraries are referenced from a flow's compiled manifest and if the library is already statically linked then\nthe native implementations can be used, or the WASM supplied files can be used, under control of an option.\nI have used this to have a flow program running on my mac, and with the flowrex job executor running on a\nconnected RaspberryPi running native or WASM. \n\nWhen a user writes a new flow and includes a \"provided implementation\" (a custom function used in the flow), \nthey write it in rust and it is compiled to WASM and loaded at run time.\n\n### Client - Server\nI knew I wanted to be able to distriubute flow execution between processes, and I know I wanted to have the\nability to have a background process coordinate execution and execute jobs, and have different UIs (CLI, GUI)\nand be able to use standard input/output from CLI. So, the \"context functions\" (impure functions that interact\nwith the environment where a flow runs) are implemented in the \"runner\" and can be CLI or GUI implementations.\nThat lead to some messy client/server message passing, that is now pretty stable and works on both CLI and GUI\nwith the same backend (in ºflowrlibº) coordinating a flow and executing jobs - but with some complexity.\n\n### Testing\nTesting coverage is about 85%-90% and I try to keep it high. There are simple unit tests for functions, a\nlot of tests around the compiler semantics, integration tests of flow compile/run errors, and of flows\nthat compile \u0026 run correctly, and the examples all have supplied test inputs files and expected output files\nand they are all tested to work correctly on every build. Additionally there are some integration \ntests of the debugger and of executing a flow in client-server mode (separate processes for each) and\nthe distribution of job execution using `flowrex`.\n\n## GUI (`flowrgui`)\nData-flow programming, declaratively defining a graph of `processes` (nodes) and `connections` (edges), fits\nnaturally with visualization of a graph. \nThe ability to define a flow, execute it and view its execution and debug it with a visual tool would be great! \nThis tool would avoid the \"hard work\" of writing flow definition text files, just producing the flow definition \nfiles formats to be compiled by the `flowc` compiler. \n\nI have started work on a native rust GUI using the `Iced` \ntoolkit. Initially, it is focussed only on running flows and replaces the Terminal based stdio, file output \nand image operations with visual equivalents.\n\nI hope to add flow design and programming to it, using `flowrclib`, either in a single\nbinary, or in a second compile-time-only tool.\n\n## Docs\nApart from this README, I have written pretty extensive documentation in a book \n([Table of Contents](SUMMARY.md)), using Markdown\nand mdbook. That describes writing flows, using flow, general ideas, the standard library functions,\nthe examples etc. It's hard to keep up to date but I try and am always generating GH issues for myself to\nimprove it! It doesn't go into two many details on the implementation, to reduce the burden of keeping it up to date.\n\nThe book is rebuilt on every PR and Merge (to make sure it is not broken) and the re-published \n[here](https://mackenzie-serres.net/flow/book/book_intro.html) as part of every release.\n\n## What's next?\nI generate ideas for ways to improve the project faster than I can implement things in my spare time,\nso over time I accumulated many issues in GitHub, and had to organize them into a\n[GitHub project](https://github.com/users/andrewdavidmackenzie/projects/2/views/1) with columns and \nto attack them kanban-style, to stop me going mad. \nI still have plenty left and continue to generate new ones all the time.\n\nProbably the main areas of work short-term will be on the GUI, enabling me to learn `Iced`\nin the process.\n\nOther main themes of items in the [GitHub project](https://github.com/users/andrewdavidmackenzie/projects/2/views/1) \nare related to adding web functionality, better example programs, packaging/distribution/install,\nand true distributed flow execution with flow partitioning based on resources and constraints.\n\n## Discuss\nI have started using the [discussions](https://github.com/andrewdavidmackenzie/flow/discussions) feature of\nGitHub, although it's just me here for now. If you have suggestions, questions or want to chat about the \nproject or learn more, just go there and let me know!\n\n## Contributing\nRefer to the [contributing](book/developing/contributing.md) section of the book.\n\n## Feedback and/or Encouragement\nYou can open an issue or email me to let me know what you think.\n\nIf you want to encourage me, even with a \"token gesture\", you can\n[\"patreonize me\"](https://www.patreon.com/andrewmackenzie)\n\nThanks for Reading this far!\n\nAndrew\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewdavidmackenzie%2Fflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandrewdavidmackenzie%2Fflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewdavidmackenzie%2Fflow/lists"}