{"id":13413034,"url":"https://github.com/sourcegraph/conc","last_synced_at":"2025-05-13T20:03:06.355Z","repository":{"id":65163523,"uuid":"584556174","full_name":"sourcegraph/conc","owner":"sourcegraph","description":"Better structured concurrency for go","archived":false,"fork":false,"pushed_at":"2024-04-26T15:28:57.000Z","size":280,"stargazers_count":9873,"open_issues_count":22,"forks_count":334,"subscribers_count":65,"default_branch":"main","last_synced_at":"2025-05-06T19:51:50.725Z","etag":null,"topics":["concurrency","go","golang","goroutines"],"latest_commit_sha":null,"homepage":"https://about.sourcegraph.com/blog/building-conc-better-structured-concurrency-for-go","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sourcegraph.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-02T22:52:07.000Z","updated_at":"2025-05-05T15:56:27.000Z","dependencies_parsed_at":"2023-11-12T16:07:52.525Z","dependency_job_id":"8d68bd17-3552-4579-a2d0-20ad2582ae9d","html_url":"https://github.com/sourcegraph/conc","commit_stats":{"total_commits":164,"total_committers":18,"mean_commits":9.11111111111111,"dds":"0.41463414634146345","last_synced_commit":"06d3061eb7cc571d833a2c0e75220ca1add74a19"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcegraph%2Fconc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcegraph%2Fconc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcegraph%2Fconc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcegraph%2Fconc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sourcegraph","download_url":"https://codeload.github.com/sourcegraph/conc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254020472,"owners_count":22000749,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","go","golang","goroutines"],"created_at":"2024-07-30T20:01:32.604Z","updated_at":"2025-05-13T20:03:06.304Z","avatar_url":"https://github.com/sourcegraph.png","language":"Go","readme":"![conch](https://user-images.githubusercontent.com/12631702/210295964-785cc63d-d697-420c-99ff-f492eb81dec9.svg)\n\n# `conc`: better structured concurrency for go\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/sourcegraph/conc.svg)](https://pkg.go.dev/github.com/sourcegraph/conc)\n[![Sourcegraph](https://img.shields.io/badge/view%20on-sourcegraph-A112FE?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAYAAAAeP4ixAAAEZklEQVRoQ+2aXWgUZxSG3292sxtNN43BhBakFPyhxSujRSxiU1pr7SaGXqgUxOIEW0IFkeYighYUxAuLUlq0lrq2iCDpjWtmFVtoG6QVNOCFVShVLyxIk0DVjZLMxt3xTGTccd2ZOd/8JBHci0CY9zvnPPN+/7sCIXwKavOwAcy2QgngQiIztDSE0OwQlDPYR1ebiaH6J5kZChyfW12gRG4QVgGTBfMchMbFP9Sn5nlZL2D0JjLD6710lc+z0NfqSGTXQRQ4bX07Mq423yoBL3OSyHSvUxirMuaEvgbJWrdcvkHMoJwxYuq4INUhyuWvQa1jvdMGxAvCxJlyEC9XOBCWL04wwRzpbDoDQ7wfZJzIQLi5Eggk6DiRhZgWIAbE3NrM4A3LPT8Q7UgqAqLqTmLSHLGPkyzG/qXEczhd0q6RH+zaSBfaUoc4iQx19pIClIscrTkNZzG6gd7qMY6eC2Hqyo705ZfTf+eqJmhMzcSbYtQpOXc92ZsZjLVAL4YNUQbJ5Ttg4CQrQdGYj44Xr9m1XJCzmZusFDJOWNpHjmh5x624a2ZFtOKDVL+uNo2TuXE3bZQQZUf8gtgqP31uI94Z/rMqix+IGiRfWw3xN9dCgVx+L3WrHm4Dju6PXz/EkjuXJ6R+IGgyOE1TbZqTq9y1eo0EZo7oMo1ktPu3xjHvuiLT5AFNszUyDULtWpzE2/fEsey8O5TbWuGWwxrs5rS7nFNMWJrNh2No74s9Ec4vRNmRRzPXMP19fBMSVsGcOJ98G8N3Wl2gXcbTjbX7vUBxLaeASDQCm5Cu/0E2tvtb0Ea+BowtskFD0wvlc6Rf2M+Jx7dTu7ubFr2dnKDRaMQe2v/tcIrNB7FH0O50AcrBaApmRDVwFO31ql3pD8QW4dP0feNwl/Q+kFEtRyIGyaWXnpy1OO0qNJWHo1y6iCmAGkBb/Ru+HenDWIF2mo4r8G+tRRzoniSn2uqFLxANhe9LKHVyTbz6egk9+x5w5fK6ulSNNMhZ/Feno+GebLZV6isTTa6k5qNl5RnZ5u56Ib6SBvFzaWBBVFZzvnERWlt/Cg4l27XChLCqFyLekjhy6xJyoytgjPf7opIB8QPx7sYFiMXHPGt76m741MhCKMZfng0nBOIjmoJPsLqWHwgFpe6V6qtfcopxveR2Oy+J0ntIN/zCWkf8QNAJ7y6d8Bq4lxLc2/qJl5K7t432XwcqX5CrI34gzATWuYILQtdQPyePDK3iuOekCR3Efjhig1B1Uq5UoXEEoZX7d1q535J5S9VOeFyYyEBku5XTMXXKQTToX5Rg7OI44nbW5oKYeYK4EniMeF0YFNSmb+grhc84LyRCEP1/OurOcipCQbKxDeK2V5FcVyIDMQvsgz5gwFhcWWwKyRlvQ3gv29RwWoDYAbIofNyBxI9eDlQ+n3YgsgCWnr4MStGXQXmv9pF2La/k3OccV54JEBM4yp9EsXa/3LfO0dGPcYq0Y7DfZB8nJzZw2rppHgKgVHs8L5wvRwAAAABJRU5ErkJggg==)](https://sourcegraph.com/github.com/sourcegraph/conc)\n[![Go Report Card](https://goreportcard.com/badge/github.com/sourcegraph/conc)](https://goreportcard.com/report/github.com/sourcegraph/conc)\n[![codecov](https://codecov.io/gh/sourcegraph/conc/branch/main/graph/badge.svg?token=MQZTEA1QWT)](https://codecov.io/gh/sourcegraph/conc)\n[![Discord](https://img.shields.io/badge/discord-chat-%235765F2)](https://discord.gg/bvXQXmtRjN)\n\n`conc` is your toolbelt for structured concurrency in go, making common tasks\neasier and safer.\n\n```sh\ngo get github.com/sourcegraph/conc\n```\n\n# At a glance\n\n- Use [`conc.WaitGroup`](https://pkg.go.dev/github.com/sourcegraph/conc#WaitGroup) if you just want a safer version of `sync.WaitGroup`\n- Use [`pool.Pool`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#Pool) if you want a concurrency-limited task runner\n- Use [`pool.ResultPool`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#ResultPool) if you want a concurrent task runner that collects task results\n- Use [`pool.(Result)?ErrorPool`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#ErrorPool) if your tasks are fallible\n- Use [`pool.(Result)?ContextPool`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#ContextPool) if your tasks should be canceled on failure\n- Use [`stream.Stream`](https://pkg.go.dev/github.com/sourcegraph/conc/stream#Stream) if you want to process an ordered stream of tasks in parallel with serial callbacks\n- Use [`iter.Map`](https://pkg.go.dev/github.com/sourcegraph/conc/iter#Map) if you want to concurrently map a slice\n- Use [`iter.ForEach`](https://pkg.go.dev/github.com/sourcegraph/conc/iter#ForEach) if you want to concurrently iterate over a slice\n- Use [`panics.Catcher`](https://pkg.go.dev/github.com/sourcegraph/conc/panics#Catcher) if you want to catch panics in your own goroutines\n\nAll pools are created with\n[`pool.New()`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#New)\nor\n[`pool.NewWithResults[T]()`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#NewWithResults),\nthen configured with methods:\n\n- [`p.WithMaxGoroutines()`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#Pool.MaxGoroutines) configures the maximum number of goroutines in the pool\n- [`p.WithErrors()`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#Pool.WithErrors) configures the pool to run tasks that return errors\n- [`p.WithContext(ctx)`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#Pool.WithContext) configures the pool to run tasks that should be canceled on first error\n- [`p.WithFirstError()`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#ErrorPool.WithFirstError) configures error pools to only keep the first returned error rather than an aggregated error\n- [`p.WithCollectErrored()`](https://pkg.go.dev/github.com/sourcegraph/conc/pool#ResultContextPool.WithCollectErrored) configures result pools to collect results even when the task errored\n\n# Goals\n\nThe main goals of the package are:\n1) Make it harder to leak goroutines\n2) Handle panics gracefully\n3) Make concurrent code easier to read\n\n## Goal #1: Make it harder to leak goroutines\n\nA common pain point when working with goroutines is cleaning them up. It's\nreally easy to fire off a `go` statement and fail to properly wait for it to\ncomplete.\n\n`conc` takes the opinionated stance that all concurrency should be scoped.\nThat is, goroutines should have an owner and that owner should always\nensure that its owned goroutines exit properly.\n\nIn `conc`, the owner of a goroutine is always a `conc.WaitGroup`. Goroutines\nare spawned in a `WaitGroup` with `(*WaitGroup).Go()`, and\n`(*WaitGroup).Wait()` should always be called before the `WaitGroup` goes out\nof scope.\n\nIn some cases, you might want a spawned goroutine to outlast the scope of the\ncaller. In that case, you could pass a `WaitGroup` into the spawning function.\n\n```go\nfunc main() {\n    var wg conc.WaitGroup\n    defer wg.Wait()\n\n    startTheThing(\u0026wg)\n}\n\nfunc startTheThing(wg *conc.WaitGroup) {\n    wg.Go(func() { ... })\n}\n```\n\nFor some more discussion on why scoped concurrency is nice, check out [this\nblog\npost](https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/).\n\n## Goal #2: Handle panics gracefully\n\nA frequent problem with goroutines in long-running applications is handling\npanics. A goroutine spawned without a panic handler will crash the whole process\non panic. This is usually undesirable.\n\nHowever, if you do add a panic handler to a goroutine, what do you do with the\npanic once you catch it? Some options:\n1) Ignore it\n2) Log it\n3) Turn it into an error and return that to the goroutine spawner\n4) Propagate the panic to the goroutine spawner\n\nIgnoring panics is a bad idea since panics usually mean there is actually\nsomething wrong and someone should fix it.\n\nJust logging panics isn't great either because then there is no indication to the spawner\nthat something bad happened, and it might just continue on as normal even though your\nprogram is in a really bad state.\n\nBoth (3) and (4) are reasonable options, but both require the goroutine to have\nan owner that can actually receive the message that something went wrong. This\nis generally not true with a goroutine spawned with `go`, but in the `conc`\npackage, all goroutines have an owner that must collect the spawned goroutine.\nIn the conc package, any call to `Wait()` will panic if any of the spawned goroutines\npanicked. Additionally, it decorates the panic value with a stacktrace from the child\ngoroutine so that you don't lose information about what caused the panic.\n\nDoing this all correctly every time you spawn something with `go` is not\ntrivial and it requires a lot of boilerplate that makes the important parts of\nthe code more difficult to read, so `conc` does this for you.\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003estdlib\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003econc\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\ntype caughtPanicError struct {\n    val   any\n    stack []byte\n}\n\nfunc (e *caughtPanicError) Error() string {\n    return fmt.Sprintf(\n        \"panic: %q\\n%s\",\n        e.val,\n        string(e.stack)\n    )\n}\n\nfunc main() {\n    done := make(chan error)\n    go func() {\n        defer func() {\n            if v := recover(); v != nil {\n                done \u003c- \u0026caughtPanicError{\n                    val: v,\n                    stack: debug.Stack()\n                }\n            } else {\n                done \u003c- nil\n            }\n        }()\n        doSomethingThatMightPanic()\n    }()\n    err := \u003c-done\n    if err != nil {\n        panic(err)\n    }\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nfunc main() {\n    var wg conc.WaitGroup\n    wg.Go(doSomethingThatMightPanic)\n    // panics with a nice stacktrace\n    wg.Wait()\n}\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Goal #3: Make concurrent code easier to read\n\nDoing concurrency correctly is difficult. Doing it in a way that doesn't\nobfuscate what the code is actually doing is more difficult. The `conc` package\nattempts to make common operations easier by abstracting as much boilerplate\ncomplexity as possible.\n\nWant to run a set of concurrent tasks with a bounded set of goroutines? Use\n`pool.New()`. Want to process an ordered stream of results concurrently, but\nstill maintain order? Try `stream.New()`. What about a concurrent map over\na slice? Take a peek at `iter.Map()`.\n\nBrowse some examples below for some comparisons with doing these by hand.\n\n# Examples\n\nEach of these examples forgoes propagating panics for simplicity. To see\nwhat kind of complexity that would add, check out the \"Goal #2\" header above.\n\nSpawn a set of goroutines and waiting for them to finish:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003estdlib\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003econc\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nfunc main() {\n    var wg sync.WaitGroup\n    for i := 0; i \u003c 10; i++ {\n        wg.Add(1)\n        go func() {\n            defer wg.Done()\n            // crashes on panic!\n            doSomething()\n        }()\n    }\n    wg.Wait()\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nfunc main() {\n    var wg conc.WaitGroup\n    for i := 0; i \u003c 10; i++ {\n        wg.Go(doSomething)\n    }\n    wg.Wait()\n}\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nProcess each element of a stream in a static pool of goroutines:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003estdlib\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003econc\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nfunc process(stream chan int) {\n    var wg sync.WaitGroup\n    for i := 0; i \u003c 10; i++ {\n        wg.Add(1)\n        go func() {\n            defer wg.Done()\n            for elem := range stream {\n                handle(elem)\n            }\n        }()\n    }\n    wg.Wait()\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nfunc process(stream chan int) {\n    p := pool.New().WithMaxGoroutines(10)\n    for elem := range stream {\n        elem := elem\n        p.Go(func() {\n            handle(elem)\n        })\n    }\n    p.Wait()\n}\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nProcess each element of a slice in a static pool of goroutines:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003estdlib\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003econc\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nfunc process(values []int) {\n    feeder := make(chan int, 8)\n\n    var wg sync.WaitGroup\n    for i := 0; i \u003c 10; i++ {\n        wg.Add(1)\n        go func() {\n            defer wg.Done()\n            for elem := range feeder {\n                handle(elem)\n            }\n        }()\n    }\n\n    for _, value := range values {\n        feeder \u003c- value\n    }\n    close(feeder)\n    wg.Wait()\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nfunc process(values []int) {\n    iter.ForEach(values, handle)\n}\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nConcurrently map a slice:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003estdlib\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003econc\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nfunc concMap(\n    input []int,\n    f func(int) int,\n) []int {\n    res := make([]int, len(input))\n    var idx atomic.Int64\n\n    var wg sync.WaitGroup\n    for i := 0; i \u003c 10; i++ {\n        wg.Add(1)\n        go func() {\n            defer wg.Done()\n\n            for {\n                i := int(idx.Add(1) - 1)\n                if i \u003e= len(input) {\n                    return\n                }\n\n                res[i] = f(input[i])\n            }\n        }()\n    }\n    wg.Wait()\n    return res\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nfunc concMap(\n    input []int,\n    f func(*int) int,\n) []int {\n    return iter.Map(input, f)\n}\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nProcess an ordered stream concurrently:\n\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003estdlib\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003econc\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```go\nfunc mapStream(\n    in chan int,\n    out chan int,\n    f func(int) int,\n) {\n    tasks := make(chan func())\n    taskResults := make(chan chan int)\n\n    // Worker goroutines\n    var workerWg sync.WaitGroup\n    for i := 0; i \u003c 10; i++ {\n        workerWg.Add(1)\n        go func() {\n            defer workerWg.Done()\n            for task := range tasks {\n                task()\n            }\n        }()\n    }\n\n    // Ordered reader goroutines\n    var readerWg sync.WaitGroup\n    readerWg.Add(1)\n    go func() {\n        defer readerWg.Done()\n        for result := range taskResults {\n            item := \u003c-result\n            out \u003c- item\n        }\n    }()\n\n    // Feed the workers with tasks\n    for elem := range in {\n        resultCh := make(chan int, 1)\n        taskResults \u003c- resultCh\n        tasks \u003c- func() {\n            resultCh \u003c- f(elem)\n        }\n    }\n\n    // We've exhausted input.\n    // Wait for everything to finish\n    close(tasks)\n    workerWg.Wait()\n    close(taskResults)\n    readerWg.Wait()\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```go\nfunc mapStream(\n    in chan int,\n    out chan int,\n    f func(int) int,\n) {\n    s := stream.New().WithMaxGoroutines(10)\n    for elem := range in {\n        elem := elem\n        s.Go(func() stream.Callback {\n            res := f(elem)\n            return func() { out \u003c- res }\n        })\n    }\n    s.Wait()\n}\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n# Status\n\nThis package is currently pre-1.0. There are likely to be minor breaking\nchanges before a 1.0 release as we stabilize the APIs and tweak defaults.\nPlease open an issue if you have questions, concerns, or requests that you'd\nlike addressed before the 1.0 release. Currently, a 1.0 is targeted for \nMarch 2023.\n","funding_links":[],"categories":["Popular","Go","开源类库","Goroutines","\u003ca name=\"Go\"\u003e\u003c/a\u003eGo","Go 程序设计"],"sub_categories":["协程/线程","检索及分析资料库","网络服务_其他","Search and Analytic Databases"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsourcegraph%2Fconc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsourcegraph%2Fconc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsourcegraph%2Fconc/lists"}