{"id":15564942,"url":"https://github.com/proullon/workerpool","last_synced_at":"2025-04-28T13:32:41.297Z","repository":{"id":45948432,"uuid":"252841094","full_name":"proullon/workerpool","owner":"proullon","description":"auto scaling generic worker pool","archived":false,"fork":false,"pushed_at":"2023-07-09T07:10:16.000Z","size":101,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-30T10:04:37.604Z","etag":null,"topics":["auto-scaling","autoscaler","autoscaling","go","golang","scaling","worker","worker-management","worker-pool","workerpool"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proullon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-03T21:02:13.000Z","updated_at":"2024-09-12T04:03:16.000Z","dependencies_parsed_at":"2024-06-21T20:17:48.910Z","dependency_job_id":"82dec030-092e-40fe-8f5c-cac5fef8344a","html_url":"https://github.com/proullon/workerpool","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proullon%2Fworkerpool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proullon%2Fworkerpool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proullon%2Fworkerpool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proullon%2Fworkerpool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proullon","download_url":"https://codeload.github.com/proullon/workerpool/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251319825,"owners_count":21570458,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auto-scaling","autoscaler","autoscaling","go","golang","scaling","worker","worker-management","worker-pool","workerpool"],"created_at":"2024-10-02T16:46:02.380Z","updated_at":"2025-04-28T13:32:41.254Z","avatar_url":"https://github.com/proullon.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# workerpool\n\nAuto scaling generic worker pool. WorkerPool will adapt the number of goroutine to find the optimal velocity by recording operation per second for each specified pool percentil.\n\n## Example\n\n```go\nfunc work(db *sql.DB) error {\n  job := \u0026Job{db:db}\n  wp, err := workerpool.New(job.execute)\n  if err != nil {\n    return err\n  }\n\n  for i := 0; i \u003c 100000; i++ {\n    wp.Feed(i)\n  }\n\n  // wait for completion\n  wp.Wait()\n\n  return nil\n}\n\ntype Job struct {\n  db *sql.DB\n}\n\nfunc (j *Job) execute(p interface{}) (interface{}, error) {\n  payload := p.(int)\n  f := func(p int) (int, error) {\n    // do stuff\n    var id int\n    err := config.db.Exec(`INSERT into example (value, event_date) ($1, NOW()) RETURNING id`, p).Scan(\u0026id)\n    if err != nil {\n      return nil, err\n    }\n    return id, nil\n  }\n  return f(payload)\n}\n```\n\nExample behaviour with network and database operation:\n```\nVelocity:\n1% : 10op/s\n2% : 21op/s\n3% : 31op/s\n4% : 43op/s\n5% : 50op/s\n6% : 55op/s\n7% : 66op/s\n8% : 80op/s\n9% : 84op/s\n10% : 85op/s\n11% : 92op/s\n12% : 95op/s\n13% : 101op/s\n14% : 103op/s\n15% : 108op/s\n16% : 141op/s\n17% : 163op/s\n18% : 143op/s\n19% : 145op/s\n20% : 145op/s\n21% : 119op/s\nCurrent velocity: 17% -\u003e 163 op/s\n```\n\n## Options\n\n```go\nfunc work(config *Config) {\n  wp, err := workerpool.New(jobfnc,\n    workerpool.WithMaxWorker(1000),\n    workerpool.WithSizePercentil(workerpool.LogSizesPercentil),\n    workerpool.EvaluationTime(1),\n    workerpool.MaxDuration(3 * time.Second),\n  )\n}\n```\n\n`MaxWorker` defines the maximum parallelisation possible, `SizePercentil` defines all the pool size possible by reference to `MaxWorker`. Default value for MaxWorker is `runtime.MaxCPU`\n\nWith 1000 max worker and default size percentil array, possible values are:\n  * 1% -\u003e 10 workers\n  * 10% -\u003e 100 workers\n  * 20% -\u003e 200 workers\n  * 30% -\u003e 300 workers\n  * 40% -\u003e 400 workers\n  * 50% -\u003e 500 workers\n  * 60% -\u003e 600 workers\n  * 70% -\u003e 700 workers\n  * 80% -\u003e 800 workers\n  * 90% -\u003e 900 workers\n  * 100% -\u003e 1000 workers\n\nWorkerPool will measure the velocity to find the most effective number of workers to achieve the best performance, increasing or decreasing the number of worker depending of the recorded velocity.\n\nThis means increasing the number of worker too find the highest op/s possible. It also means reducing the number of worker if the job takes longer at some point for some reason (network traffic, database load, etc) then increasing again as soon as possible. `EvaluationTime` defines the sampling period and the duration between WorkerPool size change. `MaxDuration` parameter ensures `WorkerPool` won't overload the client resource.\n\n## Automatic scaling mode\n\nScaling is affected by `MaxWorker`, `MaxDuration` and scaling pattern array `SizePercentil`. WorkerPool computes possible number ofworkers using `SizePercentil` of `MaxWorker`.\n\n`MaxDuration` defines the maximum duration wanted to execute a job. If a job takes longer than `MaxDuration`, worker will exit, affecting down current velocity.\n\nOne can defines specific `SizePercentil` pattern using `WithSizePercentil` functionnal option to `New`.\nPredefined pattern are:\n  * DefaultSizesPercentil: Regular increase ten by ten from 1 to 100. Allows fast scaling.\n  * AllSizesPercentil: All percentils from 1 to 100. it allows WorkerPool to find the perfect sizing fo optimal velocity. Only worth for long running operation.\n  * LogSizesPercentil: Logarithmic distribution from 1 to 100. Perfect for job targeting client sensible to load.\n  * AllInSizesPercentil: Only 100% of workerpool, meaning WorkerPool will always use MaxWorker goroutines.\n\n## Response channel\n\nWorker response are stored in an internal list. This allows user to read responses whenever ready without impacting worker, they will not lock.\n\nCall to `Stop()` will close `ReturnChannel` once all stored responses have been read.\n\n```go\nfunc Work() {\n  wp, _ := New(testJob, workerpool.WithMaxWorker(10), workerpool.WithEvaluationTime(1))\n\n  for i := 0; i \u003c 100; i++ {\n    wp.Feed(i)\n  }\n\n  wp.Wait()\n\n  n := wp.AvailableResponses() // n = 100\n\n  wp.Stop()\n\t\n  // read all responses\n  var count int\n  for r := range wp.ReturnChannel {\n    count++\n    if r.Err != nil {\n      panic(\"Expected all errors to be nil\")\n    }\n  }\n\n  // count = 100\n  n = wp.AvailableResponses() // n = 0\n}\n```\n## Retry\n\nWhen specified using `WithRetry`, WorkerPool will run again jobs failing with given\npayload until success or specified retry value.\n\nNote: only error from the last try is returned via `ReturnChannel`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproullon%2Fworkerpool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproullon%2Fworkerpool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproullon%2Fworkerpool/lists"}