{"id":13413044,"url":"https://github.com/earthboundkid/flowmatic","last_synced_at":"2025-04-10T05:08:26.653Z","repository":{"id":176690619,"uuid":"657680779","full_name":"earthboundkid/flowmatic","owner":"earthboundkid","description":"Structured concurrency made easy","archived":false,"fork":false,"pushed_at":"2024-09-30T16:59:24.000Z","size":168,"stargazers_count":380,"open_issues_count":1,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-10T05:08:21.245Z","etag":null,"topics":["concurrency","golang","goroutine","structured-concurrency","task-runner"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/earthboundkid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"carlmjohnson"}},"created_at":"2023-06-23T15:36:05.000Z","updated_at":"2025-04-10T03:49:02.000Z","dependencies_parsed_at":"2024-04-29T13:06:20.285Z","dependency_job_id":null,"html_url":"https://github.com/earthboundkid/flowmatic","commit_stats":null,"previous_names":["carlmjohnson/flowmatic","earthboundkid/flowmatic"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthboundkid%2Fflowmatic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthboundkid%2Fflowmatic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthboundkid%2Fflowmatic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthboundkid%2Fflowmatic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/earthboundkid","download_url":"https://codeload.github.com/earthboundkid/flowmatic/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248161272,"owners_count":21057555,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","golang","goroutine","structured-concurrency","task-runner"],"created_at":"2024-07-30T20:01:32.750Z","updated_at":"2025-04-10T05:08:26.614Z","avatar_url":"https://github.com/earthboundkid.png","language":"Go","readme":"# Flowmatic [![GoDoc](https://pkg.go.dev/badge/github.com/carlmjohnson/flowmatic)](https://pkg.go.dev/github.com/carlmjohnson/flowmatic) [![Coverage Status](https://coveralls.io/repos/github/carlmjohnson/flowmatic/badge.svg)](https://coveralls.io/github/carlmjohnson/flowmatic) [![Go Report Card](https://goreportcard.com/badge/github.com/carlmjohnson/flowmatic)](https://goreportcard.com/report/github.com/carlmjohnson/flowmatic) [![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)\n\n![Flowmatic logo](https://github.com/carlmjohnson/flowmatic/assets/222245/c14936e9-bb35-405b-926e-4cfeb8003439)\n\n\nFlowmatic is a generic Go library that provides a [structured approach](https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/) to concurrent programming. It lets you easily manage concurrent tasks in a manner that is simple, yet effective and flexible.\n\nFlowmatic has an easy to use API with functions for handling common concurrency patterns. It automatically handles spawning workers, collecting errors, and propagating panics.\n\nFlowmatic requires Go 1.20+.\n\n## Features\n\n- Has a simple API that improves readability over channels/waitgroups/mutexes\n- Handles a variety of concurrency problems such as heterogenous task groups, homogenous execution of a task over a slice, and dynamic work spawning\n- Aggregates errors\n- Properly propagates panics across goroutine boundaries\n- Has helpers for context cancelation\n- Few dependencies\n- Good test coverage\n\n## How to use Flowmatic\n\n### Execute heterogenous tasks\nOne problem that Flowmatic solves is managing the execution of multiple tasks in parallel that are independent of each other. For example, let's say you want to send data to three different downstream APIs. If any of the sends fail, you want to return an error. With traditional Go concurrency, this can quickly become complex and difficult to manage, with Goroutines, channels, and `sync.WaitGroup`s to keep track of. Flowmatic makes it simple.\n\nTo execute heterogenous tasks, just use `flowmatic.Do`:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003eflowmatic\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003estdlib\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nerr := flowmatic.Do(\n    func() error {\n        return doThingA(),\n    },\n    func() error {\n        return doThingB(),\n    },\n    func() error {\n        return doThingC(),\n    })\n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nvar wg sync.WaitGroup\nvar errs []error\nerrChan := make(chan error)\n\nwg.Add(3)\ngo func() {\n    defer wg.Done()\n    if err := doThingA(); err != nil {\n        errChan \u003c- err\n    }\n}()\ngo func() {\n    defer wg.Done()\n    if err := doThingB(); err != nil {\n        errChan \u003c- err\n    }\n}()\ngo func() {\n    defer wg.Done()\n    if err := doThingC(); err != nil {\n        errChan \u003c- err\n    }\n}()\n\ngo func() {\n    wg.Wait()\n    close(errChan)\n}()\n\nfor err := range errChan {\n    errs = append(errs, err)\n}\n\nerr := errors.Join(errs...)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nTo create a context for tasks that is canceled on the first error,\nuse `flowmatic.All`.\nTo create a context for tasks that is canceled on the first success,\nuse `flowmatic.Race`.\n\n```go\n// Make variables to hold responses\nvar pageA, pageB, pageC string\n// Race the requests to see who can answer first\nerr := flowmatic.Race(ctx,\n\tfunc(ctx context.Context) error {\n\t\tvar err error\n\t\tpageA, err = request(ctx, \"A\")\n\t\treturn err\n\t},\n\tfunc(ctx context.Context) error {\n\t\tvar err error\n\t\tpageB, err = request(ctx, \"B\")\n\t\treturn err\n\t},\n\tfunc(ctx context.Context) error {\n\t\tvar err error\n\t\tpageC, err = request(ctx, \"C\")\n\t\treturn err\n\t},\n)\n```\n\n### Execute homogenous tasks\n`flowmatic.Each` is useful if you need to execute the same task on each item in a slice using a worker pool:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003eflowmatic\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003estdlib\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nthings := []someType{thingA, thingB, thingC}\n\nerr := flowmatic.Each(numWorkers, things,\n    func(thing someType) error {\n        foo := thing.Frobincate()\n        return foo.DoSomething()\n    })\n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nthings := []someType{thingA, thingB, thingC}\n\nwork := make(chan someType)\nerrs := make(chan error)\n\nfor i := 0; i \u003c numWorkers; i++ {\n    go func() {\n        for thing := range work {\n            // Omitted: panic handling!\n            foo := thing.Frobincate()\n            errs \u003c- foo.DoSomething()\n        }\n    }()\n}\n\ngo func() {\n    for _, thing := range things {\n            work \u003c- thing\n    }\n\n    close(tasks)\n}()\n\nvar collectedErrs []error\nfor i := 0; i \u003c len(things); i++ {\n    collectedErrs = append(collectedErrs, \u003c-errs)\n}\n\nerr := errors.Join(collectedErrs...)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nUse `flowmatic.Map` to map an input slice to an output slice.\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"2\"\u003e\n\n```go\nfunc main() {\n\tresults, err := Google(context.Background(), \"golang\")\n\tif err != nil {\n\t\tfmt.Fprintln(os.Stderr, err)\n\t\treturn\n\t}\n\tfor _, result := range results {\n\t\tfmt.Println(result)\n\t}\n}\n```\n\n\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003eflowmatic\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ca href=\"https://pkg.go.dev/golang.org/x/sync/errgroup#example-Group-Parallel\"\u003e\u003ccode\u003ex/sync/errgroup\u003c/code\u003e\u003c/a\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\n\n```go\nfunc Google(ctx context.Context, query string) ([]Result, error) {\n\tsearches := []Search{Web, Image, Video}\n\treturn flowmatic.Map(ctx, flowmatic.MaxProcs, searches,\n\t\tfunc(ctx context.Context, search Search) (Result, error) {\n\t\t\treturn search(ctx, query)\n\t\t})\n}\n```\n\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nfunc Google(ctx context.Context, query string) ([]Result, error) {\n\tg, ctx := errgroup.WithContext(ctx)\n\n\tsearches := []Search{Web, Image, Video}\n\tresults := make([]Result, len(searches))\n\tfor i, search := range searches {\n\t\ti, search := i, search // https://golang.org/doc/faq#closures_and_goroutines\n\t\tg.Go(func() error {\n\t\t\tresult, err := search(ctx, query)\n\t\t\tif err == nil {\n\t\t\t\tresults[i] = result\n\t\t\t}\n\t\t\treturn err\n\t\t})\n\t}\n\tif err := g.Wait(); err != nil {\n\t\treturn nil, err\n\t}\n\treturn results, nil\n}\n```\n\n\u003c/td\u003e\n\u003c/table\u003e\n\n### Manage tasks that spawn new tasks\nFor tasks that may create more work, use `flowmatic.ManageTasks`.\nCreate a manager that will be serially executed,\nand have it save the results\nand examine the output of tasks to decide if there is more work to be done.\n\n```go\n// Task fetches a page and extracts the URLs\ntask := func(u string) ([]string, error) {\n    page, err := getURL(ctx, u)\n    if err != nil {\n        return nil, err\n    }\n    return getLinks(page), nil\n}\n\n// Map from page to links\n// Doesn't need a lock because only the manager touches it\nresults := map[string][]string{}\nvar managerErr error\n\n// Manager keeps track of which pages have been visited and the results graph\nmanager := func(req string, links []string, err error) ([]string, bool) {\n    // Halt execution after the first error\n    if err != nil {\n        managerErr = err\n        return nil, false\n    }\n    // Save final results in map\n    results[req] = urls\n\n    // Check for new pages to scrape\n    var newpages []string\n    for _, link := range links {\n        if _, ok := results[link]; ok {\n            // Seen it, try the next link\n            continue\n        }\n        // Add to list of new pages\n        newpages = append(newpages, link)\n        // Add placeholder to map to prevent double scraping\n        results[link] = nil\n    }\n    return newpages, true\n}\n\n// Process the tasks with as many workers as GOMAXPROCS\nflowmatic.ManageTasks(flowmatic.MaxProcs, task, manager, \"http://example.com/\")\n// Check if anything went wrong\nif managerErr != nil {\n    fmt.Println(\"error\", managerErr)\n}\n```\n\nNormally, it is very difficult to keep track of concurrent code because any combination of events could occur in any order or simultaneously, and each combination has to be accounted for by the programmer. `flowmatic.ManageTasks` makes it simple to write concurrent code because everything follows a simple rule: **tasks happen concurrently; the manager runs serially**.\n\nCentralizing control in the manager makes reasoning about the code radically simpler. When writing locking code, if you have M states and N methods, you need to think about all N states in each of the M methods, giving you an M × N code explosion. By centralizing the logic, the N states only need to be considered in one location: the manager.\n\n### Advanced patterns with TaskPool\n\nFor very advanced uses, `flowmatic.TaskPool` takes the boilerplate out of managing a pool of workers. Compare Flowmatic to [this example from x/sync/errgroup](https://pkg.go.dev/golang.org/x/sync/errgroup#example-Group-Pipeline):\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"2\"\u003e\n\n```go\nfunc main() {\n\tm, err := MD5All(context.Background(), \".\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\tfor k, sum := range m {\n\t\tfmt.Printf(\"%s:\\t%x\\n\", k, sum)\n\t}\n}\n```\n\n\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003eflowmatic\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003ex/sync/errgroup\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\n\n\n```go\n// MD5All reads all the files in the file tree rooted at root\n// and returns a map from file path to the MD5 sum of the file's contents.\n// If the directory walk fails or any read operation fails,\n// MD5All returns an error.\nfunc MD5All(ctx context.Context, root string) (map[string][md5.Size]byte, error) {\n\t// Make a pool of 20 digesters\n\tin, out := flowmatic.TaskPool(20, digest)\n\n\tm := make(map[string][md5.Size]byte)\n\t// Open two goroutines:\n\t// one for reading file names by walking the filesystem\n\t// one for recording results from the digesters in a map\n\terr := flowmatic.All(ctx,\n\t\tfunc(ctx context.Context) error {\n\t\t\treturn walkFilesystem(ctx, root, in)\n\t\t},\n\t\tfunc(ctx context.Context) error {\n\t\t\tfor r := range out {\n\t\t\t\tif r.Err != nil {\n\t\t\t\t\treturn r.Err\n\t\t\t\t}\n\t\t\t\tm[r.In] = *r.Out\n\t\t\t}\n\t\t\treturn nil\n\t\t},\n\t)\n\n\treturn m, err\n}\n\nfunc walkFilesystem(ctx context.Context, root string, in chan\u003c- string) error {\n\tdefer close(in)\n\n\treturn filepath.Walk(root, func(path string, info os.FileInfo, err error) error {\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif !info.Mode().IsRegular() {\n\t\t\treturn nil\n\t\t}\n\t\tselect {\n\t\tcase in \u003c- path:\n\t\tcase \u003c-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\t}\n\n\t\treturn nil\n\t})\n}\n\nfunc digest(path string) (*[md5.Size]byte, error) {\n\tdata, err := os.ReadFile(path)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\thash := md5.Sum(data)\n\treturn \u0026hash, nil\n}\n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\ntype result struct {\n\tpath string\n\tsum  [md5.Size]byte\n}\n\n// MD5All reads all the files in the file tree rooted at root and returns a map\n// from file path to the MD5 sum of the file's contents. If the directory walk\n// fails or any read operation fails, MD5All returns an error.\nfunc MD5All(ctx context.Context, root string) (map[string][md5.Size]byte, error) {\n\t// ctx is canceled when g.Wait() returns. When this version of MD5All returns\n\t// - even in case of error! - we know that all of the goroutines have finished\n\t// and the memory they were using can be garbage-collected.\n\tg, ctx := errgroup.WithContext(ctx)\n\tpaths := make(chan string)\n\n\tg.Go(func() error {\n\t\tdefer close(paths)\n\t\treturn filepath.Walk(root, func(path string, info os.FileInfo, err error) error {\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tif !info.Mode().IsRegular() {\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\tselect {\n\t\t\tcase paths \u003c- path:\n\t\t\tcase \u003c-ctx.Done():\n\t\t\t\treturn ctx.Err()\n\t\t\t}\n\t\t\treturn nil\n\t\t})\n\t})\n\n\t// Start a fixed number of goroutines to read and digest files.\n\tc := make(chan result)\n\tconst numDigesters = 20\n\tfor i := 0; i \u003c numDigesters; i++ {\n\t\tg.Go(func() error {\n\t\t\tfor path := range paths {\n\t\t\t\tdata, err := ioutil.ReadFile(path)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn err\n\t\t\t\t}\n\t\t\t\tselect {\n\t\t\t\tcase c \u003c- result{path, md5.Sum(data)}:\n\t\t\t\tcase \u003c-ctx.Done():\n\t\t\t\t\treturn ctx.Err()\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn nil\n\t\t})\n\t}\n\tgo func() {\n\t\tg.Wait()\n\t\tclose(c)\n\t}()\n\n\tm := make(map[string][md5.Size]byte)\n\tfor r := range c {\n\t\tm[r.path] = r.sum\n\t}\n\t// Check whether any of the goroutines failed. Since g is accumulating the\n\t// errors, we don't need to send them (or check for them) in the individual\n\t// results sent on the channel.\n\tif err := g.Wait(); err != nil {\n\t\treturn nil, err\n\t}\n\treturn m, nil\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Note on panicking\n\nIn Go, if there is a panic in a goroutine, and that panic is not recovered, then the whole process is shutdown. There are pros and cons to this approach. The pro is that if the panic is the symptom of a programming error in the application, no further damage can be done by the application. The con is that in many cases, this leads to a shutdown in a situation that might be recoverable.\n\nAs a result, although the Go standard HTTP server will catch panics that occur in one of its HTTP handlers and continue serving requests, a standard Go HTTP server cannot catch panics that occur in separate goroutines, and these will cause the whole server to go offline.\n\nFlowmatic fixes this problem by catching a panic that occurs in one of its worker goroutines and repropagating it in the parent goroutine, so the panic can be caught and logged at the appropriate level.\n","funding_links":["https://github.com/sponsors/carlmjohnson"],"categories":["Goroutines","Go"],"sub_categories":["Search and Analytic Databases"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fearthboundkid%2Fflowmatic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fearthboundkid%2Fflowmatic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fearthboundkid%2Fflowmatic/lists"}