{"id":18955695,"url":"https://github.com/creachadair/taskgroup","last_synced_at":"2025-09-07T14:40:19.742Z","repository":{"id":57486085,"uuid":"191209857","full_name":"creachadair/taskgroup","owner":"creachadair","description":"A Go package for managing a group of collaborating goroutines.","archived":false,"fork":false,"pushed_at":"2025-03-13T23:15:19.000Z","size":195,"stargazers_count":30,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T19:42:34.905Z","etag":null,"topics":["concurrency","go","golang","goroutines"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/creachadair.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-10T17:00:02.000Z","updated_at":"2025-03-13T23:15:23.000Z","dependencies_parsed_at":"2023-02-19T01:10:15.433Z","dependency_job_id":"6d0ba845-75f8-454d-8147-63afdd226ad2","html_url":"https://github.com/creachadair/taskgroup","commit_stats":{"total_commits":191,"total_committers":1,"mean_commits":191.0,"dds":0.0,"last_synced_commit":"fb5fb44ac3ccf2431f7963d5f26f994a96a3f906"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/creachadair%2Ftaskgroup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/creachadair%2Ftaskgroup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/creachadair%2Ftaskgroup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/creachadair%2Ftaskgroup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/creachadair","download_url":"https://codeload.github.com/creachadair/taskgroup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247608153,"owners_count":20965952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","go","golang","goroutines"],"created_at":"2024-11-08T13:49:51.497Z","updated_at":"2025-04-07T07:11:34.160Z","avatar_url":"https://github.com/creachadair.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# taskgroup\n\n[![GoDoc](https://img.shields.io/static/v1?label=godoc\u0026message=reference\u0026color=khaki)](https://pkg.go.dev/github.com/creachadair/taskgroup)\n[![CI](https://github.com/creachadair/taskgroup/actions/workflows/go-presubmit.yml/badge.svg?event=push\u0026branch=main)](https://github.com/creachadair/taskgroup/actions/workflows/go-presubmit.yml)\n\nA `*taskgroup.Group` represents a group of goroutines working on related tasks.\nNew tasks can be added to the group at will, and the caller can wait until all\ntasks are complete. Errors are automatically collected and delivered\nsynchronously to a user-provided callback.  This does not replace the full\ngenerality of Go's built-in features, but it simplifies some of the plumbing\nfor common concurrent tasks.\n\nHere is a [working example in the Go Playground](https://go.dev/play/p/miyrtp4PyOc).\n\n## Contents\n\n- [Rationale](#rationale)\n- [Overview](#overview)\n- [Filtering Errors](#filtering-errors)\n- [Controlling Concurrency](#controlling-concurrency)\n- [Solo Tasks](#solo-tasks)\n- [Gathering Results](#gathering-results)\n\n## Rationale\n\nGo provides powerful concurrency primitives, including\n[goroutines](http://golang.org/ref/spec#Go_statements),\n[channels](http://golang.org/ref/spec#Channel_types),\n[select](http://golang.org/ref/spec#Select_statements), and the standard\nlibrary's [sync](http://godoc.org/sync) package. In some common situations,\nhowever, managing goroutine lifetimes can become unwieldy using only what is\nbuilt in.\n\nFor example, consider the case of copying a large directory tree: Walk through\na source directory recursively, creating a parallel target directory structure\nand starting a goroutine to copy each of the files concurrently.  In outline:\n\n```go\nfunc copyTree(source, target string) error {\n    err := filepath.Walk(source, func(path string, fi os.FileInfo, err error) error {\n        adjusted := adjustPath(path)\n        if fi.IsDir() {\n            return os.MkdirAll(adjusted, 0755)\n        }\n        go copyFile(adjusted, target)\n        return nil\n    })\n    if err != nil {\n        // ... clean up the output directory ...\n    }\n    return err\n}\n```\n\nThis solution is deficient, however, as it does not provide any way to detect\nwhen all the file copies are finished. To do that we will typically use a\n`sync.WaitGroup`:\n\n```go\nvar wg sync.WaitGroup\n...\nwg.Add(1)\ngo func() {\n    defer wg.Done()\n    copyFile(adjusted, target)\n}()\n...\nwg.Wait() // block until all the tasks signal done\n```\n\nIn addition, we need to handle errors. Copies might fail (the disk may fill, or\nthere might be a permissions error). For some applications it might suffice to\nlog the error and continue, but usually in case of error we should back out and\nclean up the partial state.\n\nTo do that, we need to capture the return value from the function inside the\ngoroutine―and that will require us either to add a lock or plumb in another\nchannel:\n\n```go\nerrs := make(chan error)\n...\ngo copyFile(adjusted, target, errs)\n```\n\nSince multiple operations can be running in parallel, we will also need another\ngoroutine to drain the errors channel and accumulate the results somewhere:\n\n```go\nvar failures []error\ngo func() {\n    for e := range errs {\n        failures = append(failures, e)\n    }\n}()\n...\nwg.Wait()\nclose(errs)\n```\n\nOnce the work is finished, we must also detect when the error collector is\ndone, so we can examine the `failures` without a data race.  We'll need another\nchannel or wait group to signal for this:\n\n```go\nvar failures []error\nedone := make(chan struct{})\ngo func() {\n    defer close(edone)\n    for e := range errs {\n        failures = append(failures, e)\n    }\n}()\n...\nwg.Wait()   // all the workers are done\nclose(errs) // signal the error collector to stop\n\u003c-edone     // wait for the error collector to be done\n```\n\nAnother issue is, if one of the file copies fails, we don't necessarily want to\nwait around for all the copies to finish before reporting the error―we want to\nstop everything and clean up the whole operation. Typically we would do this\nusing a `context.Context`:\n\n```go\n\tctx, cancel := context.WithCancel(context.Background())\n\tdefer cancel()\n\t...\n\tcopyFile(ctx, adjusted, target, errs)\n```\n\nNow `copyFile` will have to check for `ctx` to be finished:\n\n```go\nfunc copyFile(ctx context.Context, source, target string, errs chan\u003c- error) {\n\tif ctx.Err() != nil {\n\t\treturn\n\t}\n \t// ... do the copy as normal, or propagate an error\n}\n```\n\nFinally, we want the ability to to limit the number of concurrent copies. Even\nif the host has plenty of memory and CPU, unbounded concurrency is likely to\nrun us out of file descriptors.  To handle this we might use a\n[semaphore](https://godoc.org/golang.org/x/sync/semaphore) or a throttling\nchannel:\n\n```go\nthrottle := make(chan struct{}, 64) // allow up to 64 concurrent copies\ngo func() {\n    throttle \u003c- struct{}{} // block until the throttle has a free slot\n    defer func() { wg.Done(); \u003c-throttle }()\n    copyFile(ctx, adjusted, target, errs)\n}()\n```\n\nSo far, we're up to four channels (errs, edone, context, and throttle) plus a\nwait group.  The point to note is that while these tools are quite able to\nexpress what we want, it can be tedious to wire them all together and keep\ntrack of the current state of the system.\n\nThe `taskgroup` package exists to handle the plumbing for the common case of a\ngroup of tasks that are all working on a related outcome (_e.g.,_ copying a\ndirectory structure), and where an error on the part of any _single_ task may\nbe grounds for terminating the work as a whole.\n\nThe package provides a `taskgroup.Group` type that has built-in support for\nsome of these concerns:\n\n- Limiting the number of active goroutines.\n- Collecting and filtering errors.\n- Waiting for completion and delivering status.\n\nA `taskgroup.Group` collects error values from each task and can deliver them\nto a user-provided callback. The callback can filter them or take other actions\n(such as cancellation). Invocations of the callback are all done from a single\ngoroutine so it is safe to manipulate local resources without a lock.\n\nA group does not directly support cancellation, but integrates cleanly with the\nstandard [context](https://godoc.org/context) package. A `context.CancelFunc`\ncan be used as a trigger to signal the whole group when an error occurs.\n\n## Overview\n\nA task is expressed as a `func() error`, and is added to a group using the `Go`\nmethod:\n\n```go\nvar g taskgroup.Group\ng.Go(myTask)\n```\n\nAny number of tasks may be added, and it is safe to do so from multiple\ngoroutines concurrently.  To wait for the tasks to finish, use:\n\n```go\nerr := g.Wait()\n```\n\n`Wait` blocks until all the tasks in the group have returned, and then reports\nthe first non-nil error returned by any of the worker tasks.\n\nAn implementation of this example can be found in `examples/copytree/copytree.go`.\n\n## Filtering Errors\n\nThe `taskgroup.New` function takes an optional callback to be invoked for each\nnon-nil error reported by a task in the group. The callback may choose to\npropagate, replace, or discard the error. For example, suppose we want to\nignore \"file not found\" errors from a copy operation:\n\n```go\ng := taskgroup.New(func(err error) error {\n    if os.IsNotExist(err) {\n        return nil // ignore files that do not exist\n    }\n    return err\n})\n```\n\nThis mechanism can also be used to trigger a context cancellation if a task\nfails, for example:\n\n```go\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\n\ng := taskgroup.New(cancel)\n```\n\nNow, if a task in `g` reports an error, it will cancel the context, allowing\nany other running tasks to observe a context cancellation and bail out.\n\n## Controlling Concurrency\n\nThe `Limit` method supports limiting the number of concurrently _active_\ngoroutines in the group. It returns a `StartFunc` that adds goroutines to the\ngroup, but will will block when the limit of goroutines is reached until some\nof the goroutines already running have finished.\n\nFor example:\n\n```go\n// Allow at most 3 concurrently-active goroutines in the group.\ng, start := taskgroup.New(nil).Limit(3)\n\n// Start tasks by calling the function returned by taskgroup.Limit:\nstart(task1)\nstart(task2)\nstart(task3)\nstart(task4) // blocks until one of the previous tasks is finished\n// ...\n```\n\n## Solo Tasks\n\nIn some cases it is useful to start a single background task to handle an\nisolated concern (elsewhere sometimes described as a \"promise\" or a \"future\").\n\nFor example, suppose we want to run some expensive background cleanup task\nwhile we take care of other work. Rather than create a whole group for a single\ngoroutine we can create a solo task using the `Go` or `Run` functions:\n\n```go\ns := taskgroup.Go(func() error {\n    for _, v := range itemsToClean {\n        if err := cleanup(v); err != nil {\n            return err\n        }\n    }\n    return nil\n})\n```\n\nOnce we're ready, we can `Wait` for this task to collect its result:\n\n```go\nif err := s.Wait(); err != nil {\n    log.Printf(\"WARNING: Cleanup failed: %v\", err)\n}\n```\n\nSolo tasks are also helpful for functions that return a value. For example,\nsuppose we want to read a file while we handle other matters. The `Call`\nfunction creates a solo task from such a function:\n\n```go\ns := taskgroup.Call(func() ([]byte, error) {\n    return os.ReadFile(filePath)\n})\n```\n\nAs before, we can `Wait` for the result when we're ready:\n\n```go\n// N.B.: Wait returns a taskgroup.Result, whose Get method unpacks\n// it into a value and an error like a normal function call.\ndata, err := s.Wait().Get()\nif err != nil {\n    log.Fatalf(\"Read configuration: %v\", err)\n}\ndoThingsWith(data)\n```\n\n## Gathering Results\n\nOne common use for a background task is accumulating the results from a batch\nof concurrent workers. This could be handled by a solo task, as described\nabove, but it is a common enough pattern that the library provides a `Gatherer`\ntype to handle it specifically.\n\nTo use it, pass a function to `Gather` to receive the values:\n\n```go\nvar g taskgroup.Group\n\nvar sum int\nc := taskgroup.Gather(g.Go, func(v int) { sum += v })\n```\n\nThe `Call`, `Run`, and `Report` methods of `c` can now be used to start tasks\nin `g` that yield values, and deliver those values to the accumulator:\n\n- `c.Call` takes a `func() (T, error)`, returning a value and an error.\n  If the task reports an error, that error is returned as usual.  Otherwise,\n  its non-error value is gathered by the callback.\n\n- `c.Run` takes a `func() T`, returning only a value, which is gathered by the\n  callback.\n\n- `c.Report` takes a `func(func(T)) error`, which allows a task to report\n  _multiple_ values to the gatherer via a \"report\" callback. The task itself\n  returns only an `error`, but it may call its argument any number of times to\n  gather values.\n\nCalls to the callback are serialized so that it is safe to access state without\nadditional locking:\n\n```go\n// Report an error, no value is gathered.\nc.Call(func() (int, error) {\n    return -1, errors.New(\"bad\")\n})\n\n// No error, send gather the value 25.\nc.Call(func() (int, error) {\n    return 25, nil\n})\n\n// Gather a random integer.\nc.Run(func() int { return rand.Intn(1000) })\n\n// Gather the values 10, 20, and 30.\n//\n// Note that even if the function reports an error, any values it sent\n// before returning are still gathered.\nc.Report(func(report func(int)) error {\n    report(10)\n    report(20)\n    report(30)\n    return nil\n})\n```\n\nOnce all the tasks passed to the gatherer are complete, it is safe to access\nthe values accumulated by the callback:\n\n```go\ng.Wait()  // wait for tasks to finish\n\n// Now you can access the values accumulated by c.\nfmt.Println(sum)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcreachadair%2Ftaskgroup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcreachadair%2Ftaskgroup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcreachadair%2Ftaskgroup/lists"}