{"id":13491055,"url":"https://github.com/s8sg/goflow","last_synced_at":"2025-05-15T21:03:04.785Z","repository":{"id":37629219,"uuid":"277050358","full_name":"s8sg/goflow","owner":"s8sg","description":"A Golang based high performance, scalable and distributed workflow framework","archived":false,"fork":false,"pushed_at":"2023-12-19T00:15:35.000Z","size":4813,"stargazers_count":1259,"open_issues_count":27,"forks_count":150,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-05-12T10:57:38.519Z","etag":null,"topics":["distributed-computing","framework","golang","workflow","workflow-engine","workload-automation"],"latest_commit_sha":null,"homepage":"","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s8sg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-04T06:10:37.000Z","updated_at":"2025-05-11T15:47:05.000Z","dependencies_parsed_at":"2024-05-29T12:13:33.371Z","dependency_job_id":"f202f506-b91e-41e4-8e23-8ace48cb6112","html_url":"https://github.com/s8sg/goflow","commit_stats":{"total_commits":52,"total_committers":6,"mean_commits":8.666666666666666,"dds":"0.11538461538461542","last_synced_commit":"5757115a21fd51c6c87634667eaa7d4ed7b1756d"},"previous_names":["faasflow/goflow"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fgoflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fgoflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fgoflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s8sg%2Fgoflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s8sg","download_url":"https://codeload.github.com/s8sg/goflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254422754,"owners_count":22068678,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","framework","golang","workflow","workflow-engine","workload-automation"],"created_at":"2024-07-31T19:00:53.170Z","updated_at":"2025-05-15T21:03:04.715Z","avatar_url":"https://github.com/s8sg.png","language":"CSS","funding_links":[],"categories":["开源类库","CSS","Open source library"],"sub_categories":["流处理","Stream Processing"],"readme":"# Go-Flow  \u0026nbsp; [![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=Start%20writing%20your%20distributed%20workflow%20in%20Golang%20with%20GoFlow\u0026url=https://github.com/s8sg/goflow\u0026hashtags=golang,workflow,distributedcomputing,framework)\n\n[![GoDoc](https://godoc.org/github.com/s8sg/goflow?status.svg)](https://godoc.org/github.com/s8sg/goflow)\n![Build](https://github.com/s8sg/goflow/workflows/GO-Flow-Build/badge.svg) \n[![Go Report Card](https://goreportcard.com/badge/github.com/s8sg/goflow)](https://goreportcard.com/report/github.com/s8sg/goflow)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\n\n![Gopher staring_at flow](doc/goflow-gopher.png)\n\nA Golang based high performance, scalable and distributed workflow framework\n\nIt allows to programmatically author distributed workflow as Directed Acyclic Graph (DAG) of tasks. \nGoFlow executes your tasks on an array of workers by uniformly distributing the loads \n\n## Stability and Compatibility\n\n**Status**: The library is currently undergoing **heavy development** with frequent, breaking API changes.\n\n\u003e ☝️ **Important Note**: Current major version is zero (`v0.x.x`) to accommodate rapid development and fast iteration. The public API could change without a major version update before `v1.0.0` release.\n\n\n## Install It \nInstall GoFlow\n```sh\ngo mod init myflow\ngo get github.com/s8sg/goflow@master\n```\n\n## Write First Flow\n\u003e Library to Build Flow `github.com/s8sg/goflow/flow/v1`\n\n[![GoDoc](https://godoc.org/github.com/faasflow/goflow/flow?status.svg)](https://godoc.org/github.com/faasflow/goflow/flow)\n\nMake a `flow.go` file\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\tgoflow \"github.com/s8sg/goflow/v1\"\n\tflow \"github.com/s8sg/goflow/flow/v1\"\n)\n\n// Workload function\nfunc doSomething(data []byte, option map[string][]string) ([]byte, error) {\n\treturn []byte(fmt.Sprintf(\"you said \\\"%s\\\"\", string(data))), nil\n}\n\n// Define provide definition of the workflow\nfunc DefineWorkflow(workflow *flow.Workflow, context *flow.Context) error {\n    dag := workflow.Dag()\n    dag.Node(\"test\", doSomething)\n    return nil\n}\n\nfunc main() {\n    fs := \u0026goflow.FlowService{\n        Port:                8080,\n        RedisURL:            \"localhost:6379\",\n        OpenTraceUrl:        \"localhost:5775\",\n        WorkerConcurrency:   5,\n        EnableMonitoring:    true,\n    }\n    fs.Register(\"myflow\", DefineWorkflow)\n    fs.Start()\n}\n```\n\u003e `Start()` runs a HTTP Server that listen on the provided Port. It also runs a flow worker that handles the workload\n\n## Run It \nStart goflow stack\n```sh\ndocker-compose up\n```\nThis will start the required services \n* redis\n* jaeger\n* dashboard\n\n\nRun the Flow\n```sh\ngo build -o goflow\n./goflow\n```\n\n## Invoke It\n\n### Using curl\n```sh\ncurl -d hallo localhost:8080/flow/myflow\n```\n\n### Using Client\n\nUsing the goflow client you can request the flow directly. \nThe requests are always async and gets queued for the workers to pick up\n```go\nfs := \u0026goflow.FlowService{\n    RedisURL: \"localhost:6379\",\n}\nfs.Execute(\"myflow\", \u0026goflow.Request{\n    Body: []byte(\"hallo\")\n})\n```\n\n### Using Dashboard\nDashboard visualize the flow and provides observability\n![Dashboard](doc/dashboard.png)\n\n## Scale It\nGoFlow scale horizontally, you can distribute the load by just adding more instances\n\n#### Worker Mode\nAlternatively you can start your GoFlow in worker mode. As a worker, GoFlow only handles the workload instead\nof running an HTTP server. If required you can only scale the workers \n```go\nfs := \u0026goflow.FlowService{\n    RedisURL:            \"localhost:6379\",\n    OpenTraceUrl:        \"localhost:5775\",\n    WorkerConcurrency:   5,\n}\nfs.Register(\"myflow\", DefineWorkflow)\nfs.StartWorker()\n```\n\n#### Register Multiple Flow\n`Register()` allows user to bind multiple flows onto single flow service. \nThis way one instance of server/worker can be used for more than one flows\n```go\nfs.Register(\"createUser\", DefineCreateUserFlow)\nfs.Register(\"deleteUser\", DefineDeleteUserFlow)\n```` \n\u003cbr /\u003e\n\n## Creating More Complex DAG\n\nThe initial example is a single vertex DAG.\nSingle vertex DAG are great for synchronous task\n\nUsing [GoFlow's DAG construct](https://godoc.org/github.com/faasflow/lib/goflow#Dag) one can achieve more complex compositions\nwith multiple vertexes and connect them using edges.\n\n### Multi Nodes\n\nA multi-vertex flow is always asynchronous in nature where each nodes gets\ndistributed across the workers\n\nBelow is an example of a simple multi vertex flow to validate a KYC image of a user and mark the user according to the result.\nThis is a asynchronous flow with three steps \n![Async Flow](doc/goflow-sequential.png)\n```go\nfunc DefineWorkflow(f *flow.Workflow, context *flow.Context) error {\n    dag := f.Dag()\n    dag.Node(\"get-kyc-image\", getPresignedURLForImage)\n    dag.Node(\"face-detect\", detectFace)\n    dag.Node(\"mark-profile\", markProfileBasedOnStatus)\n    dag.Edge(\"get-kyc-image\", \"face-detect\")\n    dag.Edge(\"face-detect\", \"mark-profile\")\n    return nil\n}\n```\n\n\n### Branching\nBranching are great for parallelizing independent workloads in separate branches\n\nBranching can be achieved with simple vertex and edges. GoFlow provides a special operator [Aggregator](https://godoc.org/github.com/faasflow/lib/goflow#Aggregator) to aggregate result of multiple branch on a converging node\n\nWe are extending our earlier example to include a new requirement to match the face with existing data \nand we are performing the operation in parallel to reduce time\n![Branching](doc/goflow-branching.png)\n\n```go\nfunc DefineWorkflow(f *flow.Workflow, context *flow.Context) error {\n    dag := f.Dag()\n    dag.Node(\"get-kyc-image\", getPresignedURLForImage)\n    dag.Node(\"face-detect\", detectFace)\n    dag.Node(\"face-match\", matchFace)\n    // Here mark-profile depends on the result from face-detect and face-match, \n    // we are using a aggregator to create unified results\n    dag.Node(\"mark-profile\", markProfileBasedOnStatus, flow.Aggregator(func(responses map[string][]byte) ([]byte, error) {\n       status := validateResults(responses[\"face-detect\"],  responses[\"face-match\"])\n       return []byte(status), nil\n    }))\n    dag.Edge(\"get-kyc-image\", \"face-detect\")\n    dag.Edge(\"get-kyc-image\", \"face-match\")\n    dag.Edge(\"face-detect\", \"mark-profile\")\n    dag.Edge(\"face-match\", \"mark-profile\")\n    return nil\n}\n```\n\n### Subdag\nSubdag allows to reuse existing DAG by embedding it into DAG with wider functionality\n\n[SubDag](https://godoc.org/github.com/faasflow/lib/goflow#Dag.SubDag) is available as a GoFlow DAG construct which takes\na separate DAG as an input and composite it within a vertex, where the vertex completion depends on the embedded DAG's \ncompletion\n```go\nfunc (currentDag *Dag) SubDag(vertex string, dag *Dag)\n```\n\nSay we have a separate flow that needs the same set of steps to validate a user. \nWith our earlier example we can separate out the validation process into subdag and put it \nin a library that can be shared across different flows\n![Subdag](doc/goflow-subdag.png)\n```go\nfunc KycImageValidationDag() *flow.Dag {\n    dag := flow.NewDag()\n    dag.Node(\"verify-url\", s3DocExists)\n    dag.Node(\"face-detect\", detectFace)\n    dag.Node(\"face-match\", matchFace)\n    dag.Node(\"generate-result\", func(data []byte, option map[string][]string) ([]byte, error) {\n                 return data, nil\n            }, \n            flow.Aggregator(func(responses map[string][]byte) ([]byte, error) {\n                status := validateResults(responses[\"face-detect\"],  responses[\"face-match\"])\n                status = \"failure\"\n                if status {\n                   status = \"success\"\n                }\n                return []byte(status), nil\n            }\n    ))\n    dag.Edge(\"verify-url\", \"face-detect\")\n    dag.Edge(\"verify-url\", \"face-match\")\n    dag.Edge(\"face-detect\", \"generate-result\")\n    dag.Edge(\"face-match\", \"generate-result\")\n    return dag\n}\n```\nOur existing flow embeds the `KycImageValidation` DAG\n```go\nfunc DefineWorkflow(f *flow.Workflow, context *flow.Context) error {\n    dag := f.Dag()\n    dag.Node(\"get-image\", getPresignedURLForImage)\n    dag.SubDag(\"verify-image\", common.KycImageValidationDag)\n    dag.Node(\"mark-profile\", markProfileBasedOnStatus)\n    dag.Edge(\"get-image\", \"verify-image\")\n    dag.Edge(\"verify-image\", \"mark-profile\")\n    return nil\n}\n```\n\n\n### Conditional Branching\nConditional branching is a great way to choose different execution path dynamically\n\nGoFlow provides a DAG component called [ConditionalBranch](https://godoc.org/github.com/faasflow/lib/goflow#Dag.ConditionalBranch).\nConditionalBranch creates a vertex that composites different conditional branches as an individual subdags, each \nidentified with a unique key resemble the condition\n\n```\nfunc (currentDag *Dag) ConditionalBranch(vertex string, conditions []string, condition sdk.Condition,\n    options ...BranchOption) (conditiondags map[string]*Dag)\n```\n\n[Condition](https://godoc.org/github.com/faasflow/sdk#Condition) is a special handler that allows user to dynamically choose one or more\nexecution path based on the result from earlier node and return a set of condition Keys \n\nUser gets the condition branches as a response where each branch specific dags are mapped against the specific condition.\nUser can farther define each branch using the DAG constructs\n\nBelow is the updated example with a conditional Branch where we are trying to call face-match only when face-detect passes\n![Conditional](doc/goflow-conditional-branching-1.png)\n```go\nfunc KycImageValidationDag() *flow.Dag {\n    dag := flow.NewDag()\n    dag.Node(\"verify-url\", s3DocExists)\n    dag.Node(\"face-detect\", detectFace)\n    // here face match happen only when face-detect is success\n    branches = dag.ConditionalBranch(\"handle-face-detect-response\", []string{\"pass\"}, func(response []byte) []string {\n        response := ParseFaceDetectResponse(response)\n        if response[0] == \"pass\" { return []string{\"pass\"}  }\n        return []string{}\n    })\n\n    // On the pass branch we are performing the `face-match` . If condition `pass` \n    // is not matched execution of next node `generate-result` is continued\n\n    branches[\"pass\"].Node(\"face-match\", matchFace)\n    dag.Node(\"generate-result\", generateResult)\n    dag.Edge(\"verify-url\", \"face-detect\")\n    dag.Edge(\"face-detect\", \"handle-face-detect-response\")\n    dag.Edge(\"handle-face-detect-response\", \"generate-result\")\n    return dag\n}\n```\n\nYou can also have multiple conditional branch in a workflow and different nodes corresponding to each branch\n\nBelow is the updated example with two conditional Branches where we are trying to call face-match or create-user based on response from previous node\n![Conditional](doc/goflow-conditional-branching-2.png)\n```go\nfunc KycImageValidationDag() *flow.Dag {\n    dag := flow.NewDag()\n    dag.Node(\"verify-url\", s3DocExists)\n    dag.Node(\"face-detect\", detectFace)\n    // here face match happen only when face-detect is success\n    // otherwise create-user is called\n    branches = dag.ConditionalBranch(\"handle-face-detect-response\", []string{\"pass\", \"fail\"}, \n        func(response []byte) []string {\n           response := ParseFaceDetectResponse(response)\n           if response.isSuccess() { return []string{\"pass\"}  }\n           return []string{\"fail\"}\n    })\n    // On the pass branch we are performing the `face-match`\n    branches[\"pass\"].Node(\"face-match\", matchFace)\n    // on the fail branch we are performing `create-user`\n    branches[\"fail\"].Node(\"create-user\", createUser)\n  \n    dag.Node(\"generate-result\", generateResult)\n    dag.Edge(\"verify-url\", \"face-detect\")\n    dag.Edge(\"face-detect\", \"handle-face-detect-response\")\n    dag.Edge(\"handle-face-detect-response\", \"generate-result\")\n    return dag\n}\n```\n\n### Foreach Branching\nForeach branching allows user to iteratively perform a certain set of task for a range of values\n\nGoFlow provides a DAG component called [ForEachBranch](https://godoc.org/github.com/faasflow/lib/goflow#Dag.ForEachBranch).\nForEachBranch creates a vertex composites of a subdag that defines the flow within the iteration\n\n```\nfunc (currentDag *Dag) ForEachBranch(vertex string, foreach sdk.ForEach, options ...BranchOption) (dag *Dag)\n```\n\n[ForEach](https://godoc.org/github.com/faasflow/sdk#ForEach) is a special handler that allows user to dynamically \nreturn a set of key and values. For each of the items in the returned set, the user defined dag will get executed \n\nUser gets the foreach branch as a response and can define the flow using the DAG constructs\n\nWe are updating our flow to execute over a set of user that has been listed for possible fraud\n![Foreach](doc/goflow-foreach-branching.png)\n```go\nfunc DefineWorkflow(f *flow.Workflow, context *flow.Context) error {\n    dag := f.Dag()\n    dag.Node(\"get-users\", getListedUsers)\n    verifyDag = dag.ForEachBranch(\"for-each-user-verify\", func(data []byte) map[string][]byte {\n       users := ParseUsersList(data)\n       forEachSet := make(map[string][]byte)\n       for _, user := range users {\n           forEachSet[user.id] = []byte(user.GetKycImageUrl())\n       }\n       return forEachSet\n    })\n    verifyDag.SubDag(\"verify-image\", KycImageValidationDag)\n    verifyDag.Node(\"mark-profile\", markProfileBasedOnStatus)\n    verifyDag.Edge(\"verify-image\", \"mark-profile\")\n\n    dag.Edge(\"get-users\", \"for-each-user-verify\")\n    return nil\n}\n```\n\n\n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs8sg%2Fgoflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs8sg%2Fgoflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs8sg%2Fgoflow/lists"}