{"id":48657102,"url":"https://github.com/cmitsakis/workerpool-go","last_synced_at":"2026-04-10T09:19:48.825Z","repository":{"id":43913842,"uuid":"481650842","full_name":"cmitsakis/workerpool-go","owner":"cmitsakis","description":"auto-scaling worker pool (work queue) in Go, using generics","archived":false,"fork":false,"pushed_at":"2023-08-14T11:51:55.000Z","size":84,"stargazers_count":56,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-06-19T05:55:00.783Z","etag":null,"topics":["goroutine-pool","pool","work-queue","worker","worker-pool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cmitsakis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-14T15:13:50.000Z","updated_at":"2024-01-26T12:41:14.000Z","dependencies_parsed_at":"2024-06-19T05:35:44.101Z","dependency_job_id":null,"html_url":"https://github.com/cmitsakis/workerpool-go","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cmitsakis/workerpool-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmitsakis%2Fworkerpool-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmitsakis%2Fworkerpool-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmitsakis%2Fworkerpool-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmitsakis%2Fworkerpool-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cmitsakis","download_url":"https://codeload.github.com/cmitsakis/workerpool-go/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmitsakis%2Fworkerpool-go/sbom","scorecard":{"id":294216,"data":{"date":"2025-08-11","repo":{"name":"github.com/cmitsakis/workerpool-go","commit":"1fafdade59da826c0e0eea32a2421ad4aa091a9c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-17T19:04:56.376Z","repository_id":43913842,"created_at":"2025-08-17T19:04:56.376Z","updated_at":"2025-08-17T19:04:56.376Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31636294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T07:40:12.752Z","status":"ssl_error","status_checked_at":"2026-04-10T07:40:11.664Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["goroutine-pool","pool","work-queue","worker","worker-pool"],"created_at":"2026-04-10T09:19:47.949Z","updated_at":"2026-04-10T09:19:48.816Z","avatar_url":"https://github.com/cmitsakis.png","language":"Go","readme":"# workerpool\n\nWorker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines. Uses Go 1.18 generics.\n\n**Notable differences from other worker pool libraries:**\n\n- Each worker runs init/deinit functions (if set) when it starts/stops respectively,\n  and stores the value returned by the init function (e.g. a connection, or external process) for the duration of the time it is active.\n  This way you can easily create a **connection pool** for a crawler or email sender.\n- You don't submit a function for each job. Instead you pass a handler function at the creation of the pool and then you submit a value (payload) for each job.\n- You can connect worker pools into a **pipeline**. This way you can increase performance by separating IO-intensive from CPU-intensive tasks (see [crawler example](#crawler-pipeline-example)), or IO tasks of different parallelizability (e.g. crawling and saving to disk).\n\n**backpressure:**\nThe pool includes a queue with limited capacity.\nIf the queue is full, job submissions block until they can be put in queue.\n\n**auto-scaling:**\nIf too many jobs are in queue, new workers are started (if available).\nIf there are more active workers than needed, some workers (and their respective goroutines) are stopped.\nYou can disable auto-scaling for CPU intensive tasks.\n\n**steady-state behavior:**\nIf the rate of job submissions is constant, the number of active workers will quickly become almost constant, and the output rate will be equal to the input (submission) rate.\n\n## Installation\n\nRequires Go 1.18+\n\n```sh\ngo get go.mitsakis.org/workerpool\n```\n\n## API Reference\n\n\u003chttps://pkg.go.dev/go.mitsakis.org/workerpool\u003e\n\n## Usage\n\n### Type parameters\n\nType `Pool[I, O, C any]` uses three type parameters:\n\n- `I`: input (job payload) type\n- `O`: output (result) type\n- `C`: type returned by the `workerInit()` function (e.g. a connection)\n\n### Constructors\n\nYou might not need all three type parameters so for convenience you can create a pool by using a constructor that hides some type parameters.\nThat's why there are four constructors of increasing complexity:\n\n```go\nNewPoolSimple(\n\tnumOfWorkers int,\n\thandler func(job Job[I], workerID int) error,\n\t...)\n\nNewPoolWithInit(\n\tnumOfWorkers int,\n\thandler func(job Job[I], workerID int, connection C) error,\n\tworkerInit func(workerID int) (C, error),\n\tworkerDeinit func(workerID int, connection C) error,\n\t...)\n\nNewPoolWithResults(\n\tnumOfWorkers int,\n\thandler func(job Job[I], workerID int) (O, error),\n\t...)\n\nNewPoolWithResultsAndInit(\n\tnumOfWorkers int,\n\thandler func(job Job[I], workerID int, connection C) (O, error),\n\tworkerInit func(workerID int) (C, error),\n\tworkerDeinit func(workerID int, connection C) error,\n\t...)\n```\n\n### Simple pool\n\nIn the most simple case you create a pool (`p`), submit the job payloads, and then call `p.StopAndWait()` to signal that you are done submitting jobs.\n`StopAndWait()` blocks until all submitted jobs have run and workers have stopped.\n```go\np, _ := workerpool.NewPoolSimple(...)\nfor i := 0; i \u003c 100; i++ {\n\tp.Submit(i)\n}\np.StopAndWait()\n```\n\n### Pool with results\n\nIf you create a pool with results (using the constructors `NewPoolWithResults()` or `NewPoolWithResultsAndInit()`), the pool writes each job's result to the `p.Results` channel.\n\n**Warning:** You *must* read from `p.Results`. Submissions and reading from `p.Results` channel *must* happen concurrently (from different goroutines) otherwise there will be a deadlock.\n```go\np, _ := workerpool.NewPoolWithResults(10, func(job workerpool.Job[float64], workerID int) (float64, error) {\n\treturn math.Sqrt(job.Payload), nil\n})\ngo func() {\n\tfor i := 0; i \u003c 100; i++ {\n\t\tp.Submit(float64(i))\n\t}\n\tp.StopAndWait()\n}()\nfor result := range p.Results {\n}\n```\n\n### Pool with worker init/deinit\n\nYou can create a pool using the constructors `NewPoolWithInit()` or `NewPoolWithResultsAndInit()`, if you want each worker to:\n- run an `init()` function when it starts, and store the value returned by `init()`\n- run a `deinit()` when it stops\n\nThis is useful, if you need each worker to have a connection or an external process.\n```go\np, _ := workerpool.NewPoolWithInit(5, func(job workerpool.Job[string], workerID int, tr *http.Transport) error {\n    // handler function\n}, func(workerID int) (*http.Transport, error) { // init\n    return \u0026http.Transport{}, nil\n}, func(workerID int, tr *http.Transport) error { // deinit\n    tr.CloseIdleConnections()\n})\n```\n\nNotice that the `http.Transport` created by the init function is accessible in the handler function as a parameter.\n\n### Pipeline\n\nYou can also connect pools of compatible type (results of `p1` are the same type as inputs to `p2`) into a pipeline by using the `ConnectPools(p1, p2, handleError)` function like this:\n```go\nworkerpool.ConnectPools(p1, p2, func(result workerpool.Result[string, []byte]) {\n\t// log error\n})\n```\nBy connecting two pools, results of `p1` that have no error are submitted to `p2`, and those with an error are handled by the `handleError()` function.\n\n### Retrying failed jobs\n\nThe pool will retry failed jobs, if in the handler function you wrap the error with `ErrorWrapRetryable()` like this:\n```go\np, _ := workerpool.NewPoolSimple(4, func(job workerpool.Job[float64], workerID int) error {\n\t...\n\tif err != nil {\n\t\treturn workerpool.ErrorWrapRetryable(fmt.Errorf(\"error: %w\", err))\n\t}\n\treturn nil\n})\n```\n\nBy default a failed job will be retried only once.\nYou can increase the number of retries by creating a pool with `Retries()` like this:\n```go\np, _ := workerpool.NewPoolSimple(4, func(job workerpool.Job[float64], workerID int) error {\n\t...\n}, workerpool.Retries(3))\n```\n\n### Examples\n\n#### Simple example\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"math\"\n\n\t\"go.mitsakis.org/workerpool\"\n)\n\nfunc main() {\n\tp, _ := workerpool.NewPoolSimple(4, func(job workerpool.Job[float64], workerID int) error {\n\t\tresult := math.Sqrt(job.Payload)\n\t\tfmt.Println(\"result:\", result)\n\t\treturn nil\n\t})\n\tfor i := 0; i \u003c 100; i++ {\n\t\tp.Submit(float64(i))\n\t}\n\tp.StopAndWait()\n}\n```\n\n#### Pipeline example\n\nA more complicated example with three pools connected into a pipeline.\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"math\"\n\n\t\"go.mitsakis.org/workerpool\"\n)\n\nfunc main() {\n\t// stage 1: calculate square root\n\tp1, _ := workerpool.NewPoolWithResults(10, func(job workerpool.Job[float64], workerID int) (float64, error) {\n\t\treturn math.Sqrt(job.Payload), nil\n\t})\n\n\t// stage 2: negate number\n\tp2, _ := workerpool.NewPoolWithResults(10, func(job workerpool.Job[float64], workerID int) (float64, error) {\n\t\treturn -job.Payload, nil\n\t})\n\n\t// stage 3: convert float to string\n\tp3, _ := workerpool.NewPoolWithResults(10, func(job workerpool.Job[float64], workerID int) (string, error) {\n\t\treturn fmt.Sprintf(\"%.3f\", job.Payload), nil\n\t})\n\n\t// connect p1, p2, p3 into a pipeline\n\tworkerpool.ConnectPools(p1, p2, nil)\n\tworkerpool.ConnectPools(p2, p3, nil)\n\n\tgo func() {\n\t\tfor i := 0; i \u003c 100; i++ {\n\t\t\tp1.Submit(float64(i))\n\t\t}\n\t\tp1.StopAndWait()\n\t}()\n\n\tfor result := range p3.Results {\n\t\tfmt.Println(\"result:\", result.Value)\n\t}\n}\n```\n\n#### Crawler pipeline example\n\nA real world example with two pools.\nThe first pool (p1) downloads URLs and the second (p2) processes the downloaded documents.\nEach worker has its own http.Transport that is reused between requests.\n```go\npackage main\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"go.mitsakis.org/workerpool\"\n)\n\nfunc main() {\n\t// pool p1 downloads URLs\n\tp1, _ := workerpool.NewPoolWithResultsAndInit(5, func(job workerpool.Job[string], workerID int, tr *http.Transport) ([]byte, error) {\n\t\tclient := \u0026http.Client{\n\t\t\tTransport: tr,\n\t\t\tTimeout:   30 * time.Second,\n\t\t}\n\t\tresp, err := client.Get(job.Payload)\n\t\tif err != nil {\n\t\t\t// mark error as retryable\n\t\t\treturn nil, workerpool.ErrorWrapRetryable(fmt.Errorf(\"client.Get failed: %w\", err))\n\t\t}\n\t\tif resp.StatusCode \u003c 200 || resp.StatusCode \u003e 399 {\n\t\t\treturn nil, fmt.Errorf(\"HTTP status code %d\", resp.StatusCode)\n\t\t}\n\t\tbody, err := io.ReadAll(resp.Body)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to read response body: %w\", err)\n\t\t}\n\t\treturn body, nil\n\t}, func(workerID int) (*http.Transport, error) { // worker init function\n\t\treturn \u0026http.Transport{}, nil\n\t}, func(workerID int, tr *http.Transport) error { // worker deinit function\n\t\ttr.CloseIdleConnections()\n\t\treturn nil\n\t}, workerpool.Retries(2)) // retry twice if error is retryable\n\n\t// pool p2 processes the content of the URLs downloaded by p1\n\tp2, _ := workerpool.NewPoolWithResults(1, func(job workerpool.Job[[]byte], workerID int) (int, error) {\n\t\tnumOfLines := bytes.Count(job.Payload, []byte(\"\\n\"))\n\t\treturn numOfLines, nil\n\t}, workerpool.FixedWorkers()) // we use a fixed number of workers (1) because it's a CPU intensive task\n\n\t// connect pools p1, p2 into a pipeline.\n\t// documents downloaded by p1 are submitted to p2 for further processing.\n\tworkerpool.ConnectPools(p1, p2, func(result workerpool.Result[string, []byte]) {\n\t\tif result.Error != nil {\n\t\t\tlog.Printf(\"failed to download URL %v - error: %v\", result.Job.Payload, result.Error)\n\t\t}\n\t})\n\n\tgo func() {\n\t\turls := []string{\n\t\t\t\"http://example.com/\",\n\t\t\t// add your own URLs\n\t\t}\n\t\tfor _, u := range urls {\n\t\t\ttime.Sleep(100 * time.Millisecond)\n\t\t\tlog.Println(\"submitting URL:\", u)\n\t\t\tp1.Submit(u)\n\t\t}\n\t\tp1.StopAndWait()\n\t}()\n\n\tfor result := range p2.Results {\n\t\tlog.Printf(\"web page has %d lines\\n\", result.Value)\n\t}\n}\n```\n\n## Contributing\n\n### Non-code Contributions\n\nBug reports, and ideas to improve the API or the auto-scaling behavior, are welcome.\n\n### Code Contributions\n\nBug fixes, and improvements to auto-scaling (implementation or tests), are welcome.\n\nCorrectness tests (`go test -run Correctness`) must pass, and auto-scaling behavior tests (`go test -run Autoscaling -v -timeout 30m`) should not become worse.\n\n## License\n\nCopyright (C) 2022 Charalampos Mitsakis (go.mitsakis.org/workerpool)\n\nThis software is licensed under the terms of the [Apache License, Version 2.0](LICENSE)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmitsakis%2Fworkerpool-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmitsakis%2Fworkerpool-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmitsakis%2Fworkerpool-go/lists"}