{"id":13413086,"url":"https://github.com/neilotoole/errgroup","last_synced_at":"2025-04-06T10:14:33.896Z","repository":{"id":41083328,"uuid":"275085993","full_name":"neilotoole/errgroup","owner":"neilotoole","description":"errgroup with goroutine worker limits","archived":false,"fork":false,"pushed_at":"2023-01-10T16:32:04.000Z","size":25,"stargazers_count":159,"open_issues_count":6,"forks_count":11,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-07-31T20:51:54.125Z","etag":null,"topics":["errgroup","go","golang","goroutine","goroutine-pool","pool","sync","workerpool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neilotoole.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-26T06:07:39.000Z","updated_at":"2024-06-26T18:07:19.000Z","dependencies_parsed_at":"2023-02-08T19:31:36.890Z","dependency_job_id":null,"html_url":"https://github.com/neilotoole/errgroup","commit_stats":null,"previous_names":["neilotoole/errgroupn"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Ferrgroup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Ferrgroup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Ferrgroup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Ferrgroup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neilotoole","download_url":"https://codeload.github.com/neilotoole/errgroup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247464226,"owners_count":20942970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["errgroup","go","golang","goroutine","goroutine-pool","pool","sync","workerpool"],"created_at":"2024-07-30T20:01:33.284Z","updated_at":"2025-04-06T10:14:33.867Z","avatar_url":"https://github.com/neilotoole.png","language":"Go","funding_links":[],"categories":["Goroutines","Go","Goroutines `goroutines的管理和使用`","Relational Databases"],"sub_categories":["Search and Analytic Databases","Advanced Console UIs","SQL 查询语句构建库","检索及分析资料库"],"readme":"\n[![Actions Status](https://github.com/neilotoole/errgroup/workflows/Go/badge.svg)](https://github.com/neilotoole/errgroup/actions?query=workflow%3AGo)\n[![Go Report Card](https://goreportcard.com/badge/neilotoole/errgroup)](https://goreportcard.com/report/neilotoole/errgroup)\n[![release](https://img.shields.io/badge/release-v0.1.5-green.svg)](https://github.com/neilotoole/errgroup/releases/tag/v0.1.5)\n[![Coverage](https://gocover.io/_badge/github.com/neilotoole/errgroup)](https://gocover.io/github.com/neilotoole/errgroup)\n[![GoDoc](https://godoc.org/github.com/golang/gddo?status.svg)](https://pkg.go.dev/github.com/neilotoole/errgroup)\n[![license](https://img.shields.io/github/license/neilotoole/errgroup)](./LICENSE)\n\n# neilotoole/errgroup\n`neilotoole/errgroup` is a drop-in alternative to Go's wonderful\n[`sync/errgroup`](https://pkg.go.dev/golang.org/x/sync/errgroup) but\nlimited to `N` goroutines. This is useful for interaction with rate-limited\nAPIs, databases, and the like.\n\n\n\u003e **Note**\n\u003e The `sync/errgroup` package [now](https://github.com/neilotoole/errgroup/issues/14) has a [Group.SetLimit](https://pkg.go.dev/golang.org/x/sync/errgroup#Group.SetLimit) method,\n\u003e which eliminates the need for `neilotoole/errgroup`. This package will no longer be maintained. Use `sync/errgroup` instead.\n\n\n## Overview\nIn effect, `neilotoole/errgroup` is `sync/errgroup` but with a worker pool\nof `N` goroutines. The exported API is identical but for an additional\nfunction `WithContextN`, which allows the caller\nto specify the maximum number of goroutines (`numG`) and the capacity\nof the queue channel (`qSize`) used to hold work before it is picked\nup by a worker goroutine. The zero `Group` and the `Group` returned\nby `WithContext` have `numG` and `qSize` equal to `runtime.NumCPU`.\n\n\n## Usage\nThe exported API of this package mirrors the `sync/errgroup` package.\nThe only change needed is the import path of the package, from:\n\n```go\nimport (\n  \"golang.org/x/sync/errgroup\"\n)\n```\n\nto\n\n```go\nimport (\n  \"github.com/neilotoole/errgroup\"\n)\n```\n\nThen use in the normal manner. See the [godoc](https://pkg.go.dev/github.com/neilotoole/errgroup) for more.\n\n```go\ng, ctx := errgroup.WithContext(ctx)\ng.Go(func() error {\n    // do something\n    return nil\n})\n\nerr := g.Wait()\n```\n\nMany users will have no need to tweak the `numG` and `qCh` params. However, benchmarking\nmay suggest particular values for your workload. For that you'll need `WithContextN`:\n\n```go\nnumG, qSize := 8, 4\ng, ctx := errgroup.WithContextN(ctx, numG, qSize)\n\n```\n\n## Performance\nThe motivation for creating `neilotoole/errgroup` was to provide rate-limiting while\nmaintaining the lovely `sync/errgroup` semantics. Sacrificing some\nperformance vs `sync/errgroup` was assumed. However, benchmarking\nsuggests that this implementation can be more effective than `sync/errgroup` \nwhen tuned for a specific workload.\n\nBelow is a selection of benchmark results. How to read this: a workload is _X_ tasks\nof _Y_ complexity. The workload is executed for:\n \n- `sync/errgroup`, listed as `sync_errgroup`\n- a non-parallel implementation (`sequential`)\n- various `{numG, qSize}` configurations of `neilotoole/errgroup`, listed as `errgroupn_{numG}_{qSize}`\n\n```\nBenchmarkGroup_Short/complexity_5/tasks_50/errgroupn_default_16_16-16         \t   25574\t     46867 ns/op\t     688 B/op\t      12 allocs/op\nBenchmarkGroup_Short/complexity_5/tasks_50/errgroupn_4_4-16                   \t   24908\t     48926 ns/op\t     592 B/op\t      12 allocs/op\nBenchmarkGroup_Short/complexity_5/tasks_50/errgroupn_16_4-16                  \t   24895\t     48313 ns/op\t     592 B/op\t      12 allocs/op\nBenchmarkGroup_Short/complexity_5/tasks_50/errgroupn_32_4-16                  \t   24853\t     48284 ns/op\t     592 B/op\t      12 allocs/op\nBenchmarkGroup_Short/complexity_5/tasks_50/sync_errgroup-16                   \t   18784\t     65826 ns/op\t    1858 B/op\t      55 allocs/op\nBenchmarkGroup_Short/complexity_5/tasks_50/sequential-16                      \t   10000\t    111483 ns/op\t       0 B/op\t       0 allocs/op\n\nBenchmarkGroup_Short/complexity_20/tasks_50/errgroupn_default_16_16-16        \t    3745\t    325993 ns/op\t    1168 B/op\t      27 allocs/op\nBenchmarkGroup_Short/complexity_20/tasks_50/errgroupn_4_4-16                  \t    5186\t    227034 ns/op\t    1072 B/op\t      27 allocs/op\nBenchmarkGroup_Short/complexity_20/tasks_50/errgroupn_16_4-16                 \t    3970\t    312816 ns/op\t    1076 B/op\t      27 allocs/op\nBenchmarkGroup_Short/complexity_20/tasks_50/errgroupn_32_4-16                 \t    3715\t    320757 ns/op\t    1073 B/op\t      27 allocs/op\nBenchmarkGroup_Short/complexity_20/tasks_50/sync_errgroup-16                  \t    2739\t    432093 ns/op\t    1862 B/op\t      55 allocs/op\nBenchmarkGroup_Short/complexity_20/tasks_50/sequential-16                     \t    2306\t    520947 ns/op\t       0 B/op\t       0 allocs/op\n\nBenchmarkGroup_Short/complexity_40/tasks_250/errgroupn_default_16_16-16       \t     354\t   3602666 ns/op\t    1822 B/op\t      47 allocs/op\nBenchmarkGroup_Short/complexity_40/tasks_250/errgroupn_4_4-16                 \t     420\t   2468605 ns/op\t    1712 B/op\t      47 allocs/op\nBenchmarkGroup_Short/complexity_40/tasks_250/errgroupn_16_4-16                \t     334\t   3581349 ns/op\t    1716 B/op\t      47 allocs/op\nBenchmarkGroup_Short/complexity_40/tasks_250/errgroupn_32_4-16                \t     310\t   3890316 ns/op\t    1712 B/op\t      47 allocs/op\nBenchmarkGroup_Short/complexity_40/tasks_250/sync_errgroup-16                 \t     253\t   4740462 ns/op\t    8303 B/op\t     255 allocs/op\nBenchmarkGroup_Short/complexity_40/tasks_250/sequential-16                    \t     200\t   5924693 ns/op\t       0 B/op\t       0 allocs/op\n```\n\nThe overall impression is that `neilotoole/errgroup` can provide higher\nthroughput than `sync/errgroup` for these (CPU-intensive) workloads,\nsometimes significantly so. As always, these benchmark results should\nnot be taken as gospel: your results may vary.\n\n\n## Design Note\nWhy require an explicit `qSize` limit?\n\nIf the number of calls to `Group.Go` results in `qCh` becoming\nfull, the `Go` method will block until worker goroutines relieve `qCh`.\nThis behavior is in contrast to `sync/errgroup`'s `Go` method, which doesn't block.\nWhile `neilotoole/errgroup` aims to be as much of a behaviorally similar\n\"drop-in\" alternative to `sync/errgroup` as possible, this blocking behavior\nis a conscious deviation.\n\nNoting that the capacity of `qCh` is controlled by `qSize`, it's probable an\nalternative implementation could be built that uses a (growable) slice\nacting - if `qCh` is full - as a buffer for functions passed to `Go`.\nConsideration of this potential design led to this [issue](https://github.com/golang/go/issues/20352)\nregarding _unlimited capacity channels_, or perhaps better characterized\nin this particular case as \"_growable capacity channels_\". If such a\nfeature existed in the language, it's possible that this implementation might\nhave taken advantage of it, at least in the first-pass release (benchmarking notwithstanding).\nHowever benchmarking seems to suggest that a relatively\nsmall `qSize` has performance benefits for some workloads, so it's possible\nthat the explicit `qSize` requirement is a better design choice regardless.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneilotoole%2Ferrgroup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneilotoole%2Ferrgroup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneilotoole%2Ferrgroup/lists"}