https://github.com/noneback/go-taskflow
A pure go General-purpose Task-parallel Programming Framework with integrated visualizer and profiler
https://github.com/noneback/go-taskflow
Last synced: 8 months ago
JSON representation
A pure go General-purpose Task-parallel Programming Framework with integrated visualizer and profiler
- Host: GitHub
- URL: https://github.com/noneback/go-taskflow
- Owner: noneback
- License: apache-2.0
- Created: 2024-03-06T04:53:14.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-04-07T05:55:12.000Z (8 months ago)
- Last Synced: 2025-04-11T23:18:03.188Z (8 months ago)
- Language: Go
- Homepage:
- Size: 142 KB
- Stars: 495
- Watchers: 2
- Forks: 21
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go-cn - go-taskflow - like General-purpose Task-parallel Programming Framework with integrated visualizer and profiler. [![godoc][D]](https://godoc.org/github.com/noneback/go-taskflow) (Workflow Frameworks / 路由器)
- awesome-go-plus - go-taskflow - A taskflow-like General-purpose Task-parallel Programming Framework with integrated visualizer and profiler.  (Workflow Frameworks / Routers)
- fucking-awesome-go - go-taskflow - A taskflow-like General-purpose Task-parallel Programming Framework with integrated visualizer and profiler. (Workflow Frameworks / Routers)
- awesome-workflow-engines - go-taskflow - taskflow.svg)](https://github.com/noneback/go-taskflow) - A pure go General-purpose Task-parallel Programming Framework with integrated visualizer and profiler (Library (embedded usage))
- awesome-go - noneback/go-taskflow - purpose Task-parallel Programming Framework with integrated visualizer and profiler ☆`598` (Workflow Frameworks / Utility/Miscellaneous)
- awesome-go - go-taskflow - A taskflow-like General-purpose Task-parallel Programming Framework with integrated visualizer and profiler. (Workflow Frameworks / Routers)
- awesome-go - go-taskflow - A taskflow-like General-purpose Task-parallel Programming Framework with integrated visualizer and profiler. (Workflow Frameworks / Routers)
- awesome-go-with-stars - go-taskflow - A taskflow-like General-purpose Task-parallel Programming Framework with integrated visualizer and profiler. (Workflow Frameworks / Routers)
README
# Go-Taskflow
[](https://codecov.io/github/noneback/go-taskflow)
[](https://pkg.go.dev/github.com/noneback/go-taskflow)
[](https://goreportcard.com/report/github.com/noneback/go-taskflow)
[](https://github.com/avelino/awesome-go)
A General-purpose Task-parallel Programming Framework for Go, inspired by [taskflow-cpp](https://github.com/taskflow/taskflow), with Go's native capabilities and simplicity, suitable for complex dependency management in concurrent tasks.
## Feature
- **High extensibility**: Easily extend the framework to adapt to various specific use cases.
- **Native Go's concurrency model**: Leverages Go's goroutines to manage concurrent task execution effectively.
- **User-friendly programming interface**: Simplify complex task dependency management using Go.
- **Static\Subflow\Conditional\Cyclic tasking**: Define static tasks, condition nodes, nested subflows and cyclic flow to enhance modularity and programmability.
| Static | Subflow | Condition | Cyclic |
|:-----------|:------------:|------------:|------------:|
|  |  |  |  |
- **Priority Task Schedule**: Define tasks' priority, higher priority tasks will be scheduled first.
- **Built-in visualization & profiling tools**: Generate visual representations of tasks and profile task execution performance using integrated tools, making debugging and optimization easier.
## Use Cases
- **Data Pipeline**: Orchestrate data processing stages that have complex dependencies.
- **AI Agent Workflow Automation**: Define and run AI Agent automation workflows where tasks have a clear sequence and dependency structure.
- **Parallel Graph Tasking**: Execute Graph-based tasks concurrently to fully utilize CPU resources.
## Example
import latest version: `go get -u github.com/noneback/go-taskflow`
```go
package main
import (
"fmt"
"log"
"math/rand"
"os"
"slices"
"strconv"
"sync"
gtf "github.com/noneback/go-taskflow"
)
// merge sorted src to sorted dest
func mergeInto(dest, src []int) []int {
size := len(dest) + len(src)
tmp := make([]int, 0, size)
i, j := 0, 0
for i < len(dest) && j < len(src) {
if dest[i] < src[j] {
tmp = append(tmp, dest[i])
i++
} else {
tmp = append(tmp, src[j])
j++
}
}
if i < len(dest) {
tmp = append(tmp, dest[i:]...)
} else {
tmp = append(tmp, src[j:]...)
}
return tmp
}
func main() {
size := 100
radomArr := make([][]int, 10)
sortedArr := make([]int, 0, 10*size)
mutex := &sync.Mutex{}
for i := 0; i < 10; i++ {
for j := 0; j < size; j++ {
radomArr[i] = append(radomArr[i], rand.Int())
}
}
sortTasks := make([]*gtf.Task, 10)
tf := gtf.NewTaskFlow("merge sort")
done := tf.NewTask("Done", func() {
if !slices.IsSorted(sortedArr) {
log.Fatal("Failed")
}
fmt.Println("Sorted")
fmt.Println(sortedArr[:1000])
})
for i := 0; i < 10; i++ {
sortTasks[i] = tf.NewTask("sort_"+strconv.Itoa(i), func() {
arr := radomArr[i]
slices.Sort(arr)
mutex.Lock()
defer mutex.Unlock()
sortedArr = mergeInto(sortedArr, arr)
})
}
done.Succeed(sortTasks...)
executor := gtf.NewExecutor(1000)
executor.Run(tf).Wait()
if err := tf.Dump(os.Stdout); err != nil {
log.Fatal("V->", err)
}
if err := executor.Profile(os.Stdout); err != nil {
log.Fatal("P->", err)
}
}
```
[more code examples](https://github.com/noneback/go-taskflow/tree/main/examples)
## Benchmark
We provide a basic benchmark to give a rough estimate of performance. However, most realistic workloads are I/O-bound, and their performance cannot be accurately reflected by the benchmark results. So, don’t take it too seriously.
If you really care about CPU Performance, we strongly recommend [taskflow-cpp](https://github.com/taskflow/taskflow).
```plaintext
goos: linux
goarch: amd64
pkg: github.com/noneback/go-taskflow/benchmark
cpu: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
BenchmarkC32-4 17964 68105 ns/op 7368 B/op 226 allocs/op
BenchmarkS32-4 5848 195952 ns/op 6907 B/op 255 allocs/op
BenchmarkC6-4 53138 22913 ns/op 1296 B/op 46 allocs/op
BenchmarkC8x8-4 6099 194579 ns/op 16956 B/op 503 allocs/op
PASS
ok github.com/noneback/go-taskflow/benchmark 5.802s
```
## Understand Condition Task Correctly
Condition Node is special in [taskflow-cpp](https://github.com/taskflow/taskflow). It not only enrolls in Condition Control but also in Looping.
Our repo keeps almost the same behavior. You should read [ConditionTasking](https://taskflow.github.io/taskflow/ConditionalTasking.html) to avoid common pitfalls.
## Error Handling in go-taskflow
`errors` in golang are values. It is the user's job to handle it correctly.
Only unrecovered `panic` needs to be addressed by the framework. Now, if it happens, the whole parent graph will be canceled, leaving the rest tasks undone. This behavior may evolve someday. If you have any good thoughts, feel free to let me know.
If you prefer not to interrupt the whole taskflow when panics occur, you can also handle panics manually while registering tasks.
Eg:
```go
tf.NewTask("not interrupt", func() {
defer func() {
if r := recover(); r != nil {
// deal with it.
}
}()
// user functions.
)
```
## How to use visualize taskflow
```go
if err := tf.Dump(os.Stdout); err != nil {
log.Fatal(err)
}
```
`tf.Dump` generates raw strings in dot format, use `dot` to draw a Graph svg.
## How to use profile taskflow
```go
if err :=exector.Profile(os.Stdout);err != nil {
log.Fatal(err)
}
```
`Profile` generates raw strings in flamegraph format, use `flamegraph` to draw a flamegraph svg.
## Stargazer
[](https://star-history.com/#noneback/go-taskflow&Date)