Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/proullon/workerpool
auto scaling generic worker pool
https://github.com/proullon/workerpool
auto-scaling autoscaler autoscaling go golang scaling worker worker-management worker-pool workerpool
Last synced: 6 days ago
JSON representation
auto scaling generic worker pool
- Host: GitHub
- URL: https://github.com/proullon/workerpool
- Owner: proullon
- License: mit
- Created: 2020-04-03T21:02:13.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2023-07-09T07:10:16.000Z (over 1 year ago)
- Last Synced: 2024-12-13T02:11:42.064Z (10 days ago)
- Topics: auto-scaling, autoscaler, autoscaling, go, golang, scaling, worker, worker-management, worker-pool, workerpool
- Language: Go
- Size: 98.6 KB
- Stars: 6
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# workerpool
Auto scaling generic worker pool. WorkerPool will adapt the number of goroutine to find the optimal velocity by recording operation per second for each specified pool percentil.
## Example
```go
func work(db *sql.DB) error {
job := &Job{db:db}
wp, err := workerpool.New(job.execute)
if err != nil {
return err
}
for i := 0; i < 100000; i++ {
wp.Feed(i)
}
// wait for completion
wp.Wait()
return nil
}
type Job struct {
db *sql.DB
}
func (j *Job) execute(p interface{}) (interface{}, error) {
payload := p.(int)
f := func(p int) (int, error) {
// do stuff
var id int
err := config.db.Exec(`INSERT into example (value, event_date) ($1, NOW()) RETURNING id`, p).Scan(&id)
if err != nil {
return nil, err
}
return id, nil
}
return f(payload)
}
```
Example behaviour with network and database operation:
```
Velocity:
1% : 10op/s
2% : 21op/s
3% : 31op/s
4% : 43op/s
5% : 50op/s
6% : 55op/s
7% : 66op/s
8% : 80op/s
9% : 84op/s
10% : 85op/s
11% : 92op/s
12% : 95op/s
13% : 101op/s
14% : 103op/s
15% : 108op/s
16% : 141op/s
17% : 163op/s
18% : 143op/s
19% : 145op/s
20% : 145op/s
21% : 119op/s
Current velocity: 17% -> 163 op/s
```
## Options
```go
func work(config *Config) {
wp, err := workerpool.New(jobfnc,
workerpool.WithMaxWorker(1000),
workerpool.WithSizePercentil(workerpool.LogSizesPercentil),
workerpool.EvaluationTime(1),
workerpool.MaxDuration(3 * time.Second),
)
}
```
`MaxWorker` defines the maximum parallelisation possible, `SizePercentil` defines all the pool size possible by reference to `MaxWorker`. Default value for MaxWorker is `runtime.MaxCPU`
With 1000 max worker and default size percentil array, possible values are:
* 1% -> 10 workers
* 10% -> 100 workers
* 20% -> 200 workers
* 30% -> 300 workers
* 40% -> 400 workers
* 50% -> 500 workers
* 60% -> 600 workers
* 70% -> 700 workers
* 80% -> 800 workers
* 90% -> 900 workers
* 100% -> 1000 workers
WorkerPool will measure the velocity to find the most effective number of workers to achieve the best performance, increasing or decreasing the number of worker depending of the recorded velocity.
This means increasing the number of worker too find the highest op/s possible. It also means reducing the number of worker if the job takes longer at some point for some reason (network traffic, database load, etc) then increasing again as soon as possible. `EvaluationTime` defines the sampling period and the duration between WorkerPool size change. `MaxDuration` parameter ensures `WorkerPool` won't overload the client resource.
## Automatic scaling mode
Scaling is affected by `MaxWorker`, `MaxDuration` and scaling pattern array `SizePercentil`. WorkerPool computes possible number ofworkers using `SizePercentil` of `MaxWorker`.
`MaxDuration` defines the maximum duration wanted to execute a job. If a job takes longer than `MaxDuration`, worker will exit, affecting down current velocity.
One can defines specific `SizePercentil` pattern using `WithSizePercentil` functionnal option to `New`.
Predefined pattern are:
* DefaultSizesPercentil: Regular increase ten by ten from 1 to 100. Allows fast scaling.
* AllSizesPercentil: All percentils from 1 to 100. it allows WorkerPool to find the perfect sizing fo optimal velocity. Only worth for long running operation.
* LogSizesPercentil: Logarithmic distribution from 1 to 100. Perfect for job targeting client sensible to load.
* AllInSizesPercentil: Only 100% of workerpool, meaning WorkerPool will always use MaxWorker goroutines.
## Response channel
Worker response are stored in an internal list. This allows user to read responses whenever ready without impacting worker, they will not lock.
Call to `Stop()` will close `ReturnChannel` once all stored responses have been read.
```go
func Work() {
wp, _ := New(testJob, workerpool.WithMaxWorker(10), workerpool.WithEvaluationTime(1))
for i := 0; i < 100; i++ {
wp.Feed(i)
}
wp.Wait()
n := wp.AvailableResponses() // n = 100
wp.Stop()
// read all responses
var count int
for r := range wp.ReturnChannel {
count++
if r.Err != nil {
panic("Expected all errors to be nil")
}
}
// count = 100
n = wp.AvailableResponses() // n = 0
}
```
## Retry
When specified using `WithRetry`, WorkerPool will run again jobs failing with given
payload until success or specified retry value.
Note: only error from the last try is returned via `ReturnChannel`.