{"id":13582526,"url":"https://github.com/shomali11/parallelizer","last_synced_at":"2025-04-15T12:19:48.465Z","repository":{"id":45901770,"uuid":"95328668","full_name":"shomali11/parallelizer","owner":"shomali11","description":"Simplifies the parallelization of function calls.","archived":false,"fork":false,"pushed_at":"2022-07-17T17:32:26.000Z","size":80,"stargazers_count":73,"open_issues_count":1,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-15T12:19:15.614Z","etag":null,"topics":["function","functions","go","golang","job","jobs","parallel","parallelism","parallelization","parallelize","pool","timeout","worker","worker-pool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shomali11.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-24T23:51:23.000Z","updated_at":"2025-03-09T04:18:48.000Z","dependencies_parsed_at":"2022-08-06T10:01:30.862Z","dependency_job_id":null,"html_url":"https://github.com/shomali11/parallelizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shomali11%2Fparallelizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shomali11%2Fparallelizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shomali11%2Fparallelizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shomali11%2Fparallelizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shomali11","download_url":"https://codeload.github.com/shomali11/parallelizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249067788,"owners_count":21207397,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["function","functions","go","golang","job","jobs","parallel","parallelism","parallelization","parallelize","pool","timeout","worker","worker-pool"],"created_at":"2024-08-01T15:02:47.868Z","updated_at":"2025-04-15T12:19:48.441Z","avatar_url":"https://github.com/shomali11.png","language":"Go","readme":"# parallelizer [![Build Status](https://travis-ci.com/shomali11/parallelizer.svg?branch=master)](https://travis-ci.com/shomali11/parallelizer) [![Go Report Card](https://goreportcard.com/badge/github.com/shomali11/parallelizer)](https://goreportcard.com/report/github.com/shomali11/parallelizer) [![GoDoc](https://godoc.org/github.com/shomali11/parallelizer?status.svg)](https://godoc.org/github.com/shomali11/parallelizer) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nSimplifies creating a pool of workers that execute jobs in parallel\n\n## Features\n\n* Easy to use\n* Context Support\n* Fail fast with errors\n* Customizable Pool Size\n    * Default number of workers is 10\n* Customizable Job Queue Size\n    * Default size is 100\n\n# Examples\n\n## Example 1\n\nRunning multiple function calls in parallel without a timeout.\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/shomali11/parallelizer\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup()\n\tdefer group.Close()\n\n\tgroup.Add(func() error {\n\t\tfor char := 'a'; char \u003c 'a'+3; char++ {\n\t\t\tfmt.Printf(\"%c \", char)\n\t\t}\n\t\treturn nil\n\t})\n\n\tgroup.Add(func() error {\n\t\tfor number := 1; number \u003c 4; number++ {\n\t\t\tfmt.Printf(\"%d \", number)\n\t\t}\n\t\treturn nil\n\t})\n\n\terr := group.Wait()\n\n\tfmt.Println()\n\tfmt.Println(\"Done\")\n\tfmt.Printf(\"Error: %v\", err)\n}\n```\n\nOutput:\n\n```text\na 1 b 2 c 3 \nDone\nError: \u003cnil\u003e\n```\n\n## Example 2\n\nRunning multiple slow function calls in parallel with a context with a short timeout.\n_Note: The timeout will not kill the routines. It will just stop waiting for them to finish_\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"github.com/shomali11/parallelizer\"\n\t\"time\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup()\n\tdefer group.Close()\n\n\tgroup.Add(func() error {\n\t\ttime.Sleep(2 * time.Second)\n\n\t\tfmt.Println(\"Finished work 1\")\n\n\t\treturn nil\n\t})\n\n\tgroup.Add(func() error {\n\t\ttime.Sleep(2 * time.Second)\n\n\t\tfmt.Println(\"Finished work 2\")\n\n\t\treturn nil\n\t})\n\n\tctx, cancel := context.WithTimeout(context.Background(), time.Second)\n\tdefer cancel()\n\n\terr := group.Wait(parallelizer.WithContext(ctx))\n\n\tfmt.Println(\"Done\")\n\tfmt.Printf(\"Error: %v\", err)\n\tfmt.Println()\n\n\ttime.Sleep(2 * time.Second)\n}\n```\n\nOutput:\n\n```text\nDone\nError: context deadline exceeded\nFinished work 2\nFinished work 1\n```\n\n## Example 3\n\nRunning multiple function calls in parallel with a large enough worker pool.\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/shomali11/parallelizer\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup(parallelizer.WithPoolSize(10))\n\tdefer group.Close()\n\n\tfor i := 1; i \u003c= 10; i++ {\n\t\ti := i\n\t\tgroup.Add(func() error {\n\t\t\tfmt.Print(i, \" \")\n\t\t\treturn nil\n\t\t})\n\t}\n\n\terr := group.Wait()\n\n\tfmt.Println()\n\tfmt.Println(\"Done\")\n\tfmt.Printf(\"Error: %v\", err)\n}\n```\n\nOutput:\n\n```text\n7 6 3 2 8 9 5 10 1 4  \nDone\nError: \u003cnil\u003e\n```\n\n## Example 4\n\nRunning multiple function calls with 1 worker. _Note: the functions are no longer executed in parallel but sequentially_\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/shomali11/parallelizer\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup(parallelizer.WithPoolSize(1))\n\tdefer group.Close()\n\n\tfor i := 1; i \u003c= 10; i++ {\n\t\ti := i\n\t\tgroup.Add(func() error {\n\t\t\tfmt.Print(i, \" \")\n\t\t\treturn nil\n\t\t})\n\t}\n\n\terr := group.Wait()\n\n\tfmt.Println()\n\tfmt.Println(\"Done\")\n\tfmt.Printf(\"Error: %v\", err)\n}\n```\n\nOutput:\n\n```text\n1 2 3 4 5 6 7 8 9 10 \nDone\nError: \u003cnil\u003e\n```\n\n## Example 5\n\nRunning multiple function calls in parallel with a small worker pool and job queue size. _Note: the `Add` call blocks until there is space to push into the Job Queue_\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/shomali11/parallelizer\"\n\t\"time\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup(parallelizer.WithPoolSize(1), parallelizer.WithJobQueueSize(1))\n\tdefer group.Close()\n\n\tfor i := 1; i \u003c= 10; i++ {\n\t\tgroup.Add(func() error {\n\t\t\ttime.Sleep(time.Second)\n\t\t\treturn nil\n\t\t})\n\n\t\tfmt.Println(\"Job added at\", time.Now().Format(\"04:05\"))\n\t}\n\n\terr := group.Wait()\n\n\tfmt.Println()\n\tfmt.Println(\"Done\")\n\tfmt.Printf(\"Error: %v\", err)\n}\n```\n\nOutput:\n\n```text\nJob added at 00:12\nJob added at 00:13\nJob added at 00:14\nJob added at 00:15\nJob added at 00:16\nJob added at 00:17\nJob added at 00:18\nJob added at 00:19\nJob added at 00:20\nJob added at 00:21\n\nDone\nError: \u003cnil\u003e\n```\n\n## Example 6\n\nRunning multiple function calls in parallel with a large enough worker pool and job queue size. _Note: In here the `Add` calls did not block because there was plenty of space in the Job Queue_\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/shomali11/parallelizer\"\n\t\"time\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup(parallelizer.WithPoolSize(10), parallelizer.WithJobQueueSize(10))\n\tdefer group.Close()\n\n\tfor i := 1; i \u003c= 10; i++ {\n\t\tgroup.Add(func() error {\n\t\t\ttime.Sleep(time.Second)\n\t\t\treturn nil\n\t\t})\n\n\t\tfmt.Println(\"Job added at\", time.Now().Format(\"04:05\"))\n\t}\n\n\terr := group.Wait()\n\n\tfmt.Println()\n\tfmt.Println(\"Done\")\n\tfmt.Printf(\"Error: %v\", err)\n}\n```\n\nOutput:\n\n```text\nJob added at 00:30\nJob added at 00:30\nJob added at 00:30\nJob added at 00:30\nJob added at 00:30\nJob added at 00:30\nJob added at 00:30\nJob added at 00:30\nJob added at 00:30\nJob added at 00:30\n\nDone\nError: \u003cnil\u003e\n```\n\n## Example 7\n\nShowing an example without calling `Wait`\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/shomali11/parallelizer\"\n\t\"time\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup()\n\tdefer group.Close()\n\n\tgroup.Add(func() error {\n\t\tfmt.Println(\"Finished work\")\n\t\treturn nil\n\t})\n\n\tfmt.Println(\"We did not wait!\")\n\n\ttime.Sleep(time.Second)\n}\n```\n\nOutput:\n\n```text\nWe did not wait!\nFinished work\n```\n\n## Example 8\n\nShowing an example with a mixture of `Add` and `Wait` calls.\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/shomali11/parallelizer\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup()\n\tdefer group.Close()\n\n\tgroup.Add(func() error {\n\t\tfmt.Println(\"Worker 1\")\n\t\treturn nil\n\t})\n\n\tgroup.Add(func() error {\n\t\tfmt.Println(\"Worker 2\")\n\t\treturn nil\n\t})\n\n\tfmt.Println(\"Waiting for workers 1 and 2 to finish\")\n\n\tgroup.Wait()\n\n\tfmt.Println(\"Workers 1 and 2 have finished\")\n\n\tgroup.Add(func() error {\n\t\tfmt.Println(\"Worker 3\")\n\t\treturn nil\n\t})\n\n\tfmt.Println(\"Waiting for worker 3 to finish\")\n\n\tgroup.Wait()\n\n\tfmt.Println(\"Worker 3 has finished\")\n}\n\n```\n\nOutput:\n\n```text\nWaiting for workers 1 and 2 to finish\nWorker 1\nWorker 2\nWorkers 1 and 2 have finished\nWaiting for worker 3 to finish\nWorker 3\nWorker 3 has finished\n```\n\n## Example 9\n\nShowing an example with a failed task.\n\n```go\npackage main\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/shomali11/parallelizer\"\n)\n\nfunc main() {\n\tgroup := parallelizer.NewGroup()\n\tdefer group.Close()\n\n\tgroup.Add(func() error {\n\t\treturn errors.New(\"something went wrong\")\n\t})\n\n\tgroup.Add(func() error {\n\t\ttime.Sleep(10 * time.Second)\n\t\treturn nil\n\t})\n\n\terr := group.Wait()\n\n\tfmt.Println()\n\tfmt.Println(\"Done\")\n\tfmt.Printf(\"Error: %v\", err)\n}\n```\n\nOutput:\n\n```text\nDone\nError: something went wrong\n```","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshomali11%2Fparallelizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshomali11%2Fparallelizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshomali11%2Fparallelizer/lists"}