{"id":13413062,"url":"https://github.com/catmullet/go-workers","last_synced_at":"2025-03-14T19:31:14.565Z","repository":{"id":45033782,"uuid":"301775263","full_name":"catmullet/go-workers","owner":"catmullet","description":"👷 Library for safely running groups of workers concurrently or consecutively that require input and output through channels","archived":true,"fork":false,"pushed_at":"2022-01-13T07:41:18.000Z","size":997,"stargazers_count":164,"open_issues_count":3,"forks_count":16,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-07-31T20:51:53.648Z","etag":null,"topics":["concurrency","go","go-concurrency","go-worker","go-workerpool","golang","golang-library","multiple-workers","pool","pools","worker-functions","workers"],"latest_commit_sha":null,"homepage":"https://github.com/catmullet/go-workers/wiki/Getting-Started","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/catmullet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-06T15:39:43.000Z","updated_at":"2024-06-01T00:31:58.000Z","dependencies_parsed_at":"2022-08-27T16:31:57.380Z","dependency_job_id":null,"html_url":"https://github.com/catmullet/go-workers","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catmullet%2Fgo-workers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catmullet%2Fgo-workers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catmullet%2Fgo-workers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catmullet%2Fgo-workers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/catmullet","download_url":"https://codeload.github.com/catmullet/go-workers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221498713,"owners_count":16833053,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","go","go-concurrency","go-worker","go-workerpool","golang","golang-library","multiple-workers","pool","pools","worker-functions","workers"],"created_at":"2024-07-30T20:01:32.982Z","updated_at":"2024-10-26T05:30:18.041Z","avatar_url":"https://github.com/catmullet.png","language":"Go","readme":"![go workers](https://raw.githubusercontent.com/catmullet/go-workers/assets/constworker_header_anim.gif)\n\n[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge-flat.svg)](https://github.com/avelino/awesome-go#goroutines)\n[![Maintainability](https://api.codeclimate.com/v1/badges/402fee86fbd1e24defb2/maintainability)](https://codeclimate.com/github/catmullet/go-workers/maintainability)\n[![GoCover](http://gocover.io/_badge/github.com/catmullet/go-workers)](http://gocover.io/github.com/catmullet/go-workers)\n[![Go Reference](https://pkg.go.dev/badge/github.com/catmullet/go-workers.svg)](https://pkg.go.dev/github.com/catmullet/go-workers)\n\n# Examples\n* [Quickstart](https://github.com/catmullet/go-workers/blob/master/examples/quickstart/quickstart.go)\n* [Multiple Go Workers](https://github.com/catmullet/go-workers/blob/master/examples/multiple_workers/multipleworkers.go)\n* [Passing Fields](https://github.com/catmullet/go-workers/blob/master/examples/passing_fields/passingfields.go)\n# Getting Started\n### Pull in the dependency\n```zsh\ngo get github.com/catmullet/go-workers\n```\n\n### Add the import to your project\ngiving an alias helps since go-workers doesn't exactly follow conventions.    \n_(If you're using a JetBrains IDE it should automatically give it an alias)_\n```go\nimport (\n    workers \"github.com/catmullet/go-workers\"\n)\n```\n### Create a new worker \u003cimg src=\"https://raw.githubusercontent.com/catmullet/go-workers/assets/constworker.png\" alt=\"worker\" width=\"35\"/\u003e\nThe NewWorker factory method returns a new worker.    \n_(Method chaining can be performed on this method like calling .Work() immediately after.)_\n```go\ntype MyWorker struct {}\n\nfunc NewMyWorker() Worker {\n\treturn \u0026MyWorker{}\n}\n\nfunc (my *MyWorker) Work(in interface{}, out chan\u003c- interface{}) error {\n\t// work iteration here\n}\n\nrunner := workers.NewRunner(ctx, NewMyWorker(), numberOfWorkers)\n```\n### Send work to worker\nSend accepts an interface.  So send it anything you want.\n```go\nrunner.Send(\"Hello World\")\n```\n### Wait for the worker to finish and handle errors\nAny error that bubbles up from your worker functions will return here.\n```go\nif err := runner.Wait(); err != nil {\n    //Handle error\n}\n```\n\n## Working With Multiple Workers\n### Passing work form one worker to the next \n\nBy using the InFrom method you can tell `workerTwo` to accept output from `workerOne`\n```go\nrunnerOne := workers.NewRunner(ctx, NewMyWorker(), 100).Work()\nrunnerTwo := workers.NewRunner(ctx, NewMyWorkerTwo(), 100).InFrom(workerOne).Work()\n```\n### Accepting output from multiple workers\nIt is possible to accept output from more than one worker but it is up to you to determine what is coming from which worker.  (They will send on the same channel.)\n```go\nrunnerOne := workers.NewRunner(ctx, NewMyWorker(), 100).Work()\nrunnerTwo := workers.NewRunner(ctx, NewMyWorkerTwo(), 100).Work()\nrunnerThree := workers.NewRunner(ctx, NewMyWorkerThree(), 100).InFrom(workerOne, workerTwo).Work()\n```\n\n## Passing Fields To Workers\n### Adding Values\nFields can be passed via the workers object. Be sure as with any concurrency in Golang that your variables are concurrent safe.  Most often the golang documentation will state the package or parts of it are concurrent safe.  If it does not state so there is a good chance it isn't.  Use the sync package to lock and unlock for writes on unsafe variables.  (It is good practice NOT to defer in the work function.)\n\n\u003cimg src=\"https://raw.githubusercontent.com/catmullet/go-workers/assets/constworker2.png\" alt=\"worker\" width=\"35\"/\u003e **ONLY** use the `Send()` method to get data into your worker. It is not shared memory unlike the worker objects values.\n\n```go\ntype MyWorker struct {\n\tmessage string\n}\n\nfunc NewMyWorker(message string) Worker {\n\treturn \u0026MyWorker{message}\n}\n\nfunc (my *MyWorker) Work(in interface{}, out chan\u003c- interface{}) error {\n\tfmt.Println(my.message)\n}\n\nrunner := workers.NewRunner(ctx, NewMyWorker(), 100).Work()\n```\n\n### Setting Timeouts or Deadlines\nIf your workers needs to stop at a deadline or you just need to have a timeout use the SetTimeout or SetDeadline methods. (These must be in place before setting the workers off to work.)\n```go\n // Setting a timeout of 2 seconds\n timeoutRunner.SetTimeout(2 * time.Second)\n\n // Setting a deadline of 4 hours from now\n deadlineRunner.SetDeadline(time.Now().Add(4 * time.Hour))\n\nfunc workerFunction(in interface{}, out chan\u003c- interface{} error {\n\tfmt.Println(in)\n\ttime.Sleep(1 * time.Second)\n}\n```\n\n\n## Performance Hints\n### Buffered Writer\nIf you want to write out to a file or just stdout you can use SetWriterOut(writer io.Writer).  The worker will have the following methods available\n```go\nrunner.Println()\nrunner.Printf()\nrunner.Print()\n```\nThe workers use a buffered writer for output and can be up to 3 times faster than the fmt package.  Just be mindful it won't write out to the console as quickly as an unbuffered writer.  It will sync and eventually flush everything at the end, making it ideal for writing out to a file.\n\n### Using GOGC env variable\nIf your application is based solely around using workers, consider upping the percentage of when the scheduler will garbage collect. (ex. GOGC=200) 200% -\u003e 300% is a good starting point. Make sure your machine has some good memory behind it.\nBy upping the percentage your application will interupt the workers less, meaning they get more work done.  However, be aware of the rest of your applications needs when modifying this variable.\n\n### Using GOMAXPROCS env variable\nFor workers that run quick bursts of lots of simple data consider lowering the GOMAXPROCS.  Be carfeful though, this can affect your entire applicaitons performance.  Profile your application and benchmark it.  See where your application runs best.\n","funding_links":[],"categories":["开源类库","Goroutines","Open source library","Relational Databases"],"sub_categories":["协程/线程","Advanced Console UIs","Coroutine/Thread","Search and Analytic Databases","检索及分析资料库"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcatmullet%2Fgo-workers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcatmullet%2Fgo-workers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcatmullet%2Fgo-workers/lists"}