https://github.com/vectaport/flowgraph
Flowgraph package for scalable asynchronous system development
https://github.com/vectaport/flowgraph
dataflow distributed scalable
Last synced: 9 months ago
JSON representation
Flowgraph package for scalable asynchronous system development
- Host: GitHub
- URL: https://github.com/vectaport/flowgraph
- Owner: vectaport
- License: bsd-3-clause
- Created: 2018-08-29T21:45:26.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2021-04-24T16:09:30.000Z (over 4 years ago)
- Last Synced: 2024-07-31T20:51:07.362Z (over 1 year ago)
- Topics: dataflow, distributed, scalable
- Language: Go
- Size: 164 KB
- Stars: 58
- Watchers: 2
- Forks: 8
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go - flowgraph - flow-based programming package. (Distributed Systems / Search and Analytic Databases)
- awesome-go-cn - flowgraph - based programming package。 [![近三年未更新][Y]](https://github.com/vectaport/flowgraph) [![godoc][D]](https://godoc.org/github.com/vectaport/flowgraph) (分布式系统 / 检索及分析资料库)
- awesome-go - flowgraph - Flowgraph package for scalable asynchronous system development - ★ 2 (Distributed Systems)
- awesome-go-plus - flowgraph - flow-based programming package.  (Distributed Systems / Search and Analytic Databases)
- fucking-awesome-go - :octocat: flowgraph - MPI-style ready-send coordination layer. :star: 14 :fork_and_knife: 0 (Distributed Systems / Advanced Console UIs)
- awesome-go - flowgraph - based programming package. | - | - | - | (Distributed Systems / Advanced Console UIs)
- awesome-go - flowgraph - flow-based programming package. (Distributed Systems / Search and Analytic Databases)
- awesome-go - flowgraph - flow-based programming package. - :arrow_down:5 - :star:19 (Distributed Systems / Advanced Console UIs)
- awesome-go-with-stars - flowgraph - flow-based programming package. (Distributed Systems / Search and Analytic Databases)
- fucking-awesome-go - flowgraph - flow-based programming package. (Distributed Systems / Search and Analytic Databases)
- awesome-go-cn - flowgraph
- awesome-go - flowgraph - flow-based programming package. (Distributed Systems / Advanced Console UIs)
- awesome-go - flowgraph - flow-based programming package. (Distributed Systems / Search and Analytic Databases)
- awesome-go-extra - flowgraph - 08-29T21:45:26Z|2021-04-24T16:09:30Z| (Distributed Systems / Advanced Console UIs)
- awesome-go - flowgraph - MPI风格的即时发送协调层。 (<span id="分布式系统-distributed-systems">分布式系统 Distributed Systems</span> / <span id="高级控制台用户界面-advanced-console-uis">高级控制台用户界面 Advanced Console UIs</span>)
- awesome-go - flowgraph - MPI-style ready-send coordination layer. (Distributed Systems / Advanced Console UIs)
- awesome-go - flowgraph - flow-based programming package. (Distributed Systems / Search and Analytic Databases)
- awesome-go - flowgraph - flow-based programming package. (Distributed Systems / Search and Analytic Databases)
- awesome-Char - flowgraph - flow-based programming package. (Distributed Systems / Advanced Console UIs)
- awesome-go-cn - flowgraph - style ready-send coordination layer.) (分布式系统 / Advanced Console UIs)
- awesome-go-cn - flowgraph - based programming package。 [![godoc][D]](https://godoc.org/github.com/vectaport/flowgraph) (分布式系统 / 检索及分析资料库)
README
# flowgraph
### Getting Started
```
go get -u github.com/vectaport/flowgraph
go test
```
### Links
* [](https://godoc.org/github.com/vectaport/flowgraph)
* [Wiki](https://github.com/vectaport/flowgraph/wiki)
* [Slides](https://www.youtube.com/watch?v=awAsZUBncG8) from [Minneapolis Golang Meetup, May 22nd 2019](https://www.meetup.com/Minneapolis-Golang/events/259276080/)
### Overview
Flowgraphs are built out of hubs interconnected by streams. The hubs are implemented with goroutines that use select to wait on incoming data or back-pressure handshakes. The data and handshakes travel on streams implemented with channels of empty interfaces for forward flow (interface{}) and channels of empty structs for back-pressure (struct{}).
The user of this package is completely isolated from the details of using goroutines, channels, and select, and only has to provide the empty interface functions that transform incoming data into outgoing data as needed for each hub of the flowgraph under construction. It includes the ability to log each data flow and transformation at the desired level of detail for debugging and monitoring purposes.
The package allows for correct-by-construction dataflow systems that avoid deadlock and gridlock by using back-pressure to manage empty space. It also supports looping constructs that can operate at the same efficiency as pipeline structures using channel buffering within the loop.
All of this is made available with an API designed to directly underlie a future HDL for a flowgraph language.