https://github.com/matthewoestreich/workerpoolxt
Concurrency limiting goroutine pool without upper limit on queue length. Extends github.com/gammazero/workerpool
https://github.com/matthewoestreich/workerpoolxt
concurrency multithreading worker-pool
Last synced: about 2 months ago
JSON representation
Concurrency limiting goroutine pool without upper limit on queue length. Extends github.com/gammazero/workerpool
- Host: GitHub
- URL: https://github.com/matthewoestreich/workerpoolxt
- Owner: matthewoestreich
- License: mit
- Created: 2020-10-07T05:53:30.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2025-11-13T21:58:00.000Z (about 2 months ago)
- Last Synced: 2025-11-13T23:27:02.671Z (about 2 months ago)
- Topics: concurrency, multithreading, worker-pool
- Language: Go
- Homepage:
- Size: 199 KB
- Stars: 15
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
workerpoolxt
Worker pool library that extends https://github.com/gammazero/workerpool
go get github.com/matthewoestreich/workerpoolxt
---
## Synopsis
We wrap the func(s) that get passed to `workerpool`, which we call "jobs". Job results will be returned to you after all jobs have completed.
You have the ability to give each job a name. You can access job results, job runtime duration, or any job errors within job results. In `gammazero/workerpool` this is not possible - you do not have the ability to get any sort of result or error data from a "job".
## Important
**Breaking changes in `v1.5.0`!
You still retain access to all `gammazero/workerpool` members, but **you must use `pool.StopWaitXT()` if you submit jobs via `pool.SubmitXT(..)`!**
## Generics
If you want to use this package with type-safe generics, [please see here](/generic).
## Import
```go
import wpxt "github.com/matthewoestreich/workerpoolxt"
```
## Examples
### Hello, world!
[Playground](https://go.dev/play/p/4oKDsprC8dC)
```go
numWorkers := 5
pool := wpxt.New(numWorkers)
// Or if you have an existing |*workerpool.WorkerPool| instance
pool := wpxt.WithWorkerPool(existingWorkerPool)
helloWorldJob := &wpxt.Job{
Name: "Hello world job",
// Function signature must be |func() (any, error)|
Function: func() (any, error) {
pretendError := nil
if pretendError != nil {
// To demonstrate how you would return an error
return nil, theError
}
return "Hello, world!", nil
},
}
// Submit job
pool.SubmitXT(helloWorldJob)
// Block main thread until all jobs are done.
results := pool.StopWaitXT()
// Results also accessable via
sameResults := pool.Results()
// Grab first result
r := results[0]
// Print it to console
fmt.Printf(
"Name: %s\nData: %v\nDuration: %v\nError?: %v",
r.Name, r.Data, r.Duration, r.Error,
)
/*
Name: Hello world job
Data: Hello, world!
Duration: 3.139µs
Error?:
*/
```
### Using Context
Works with timeouts, cancelled context, etc..
The point is: you have full control over every job.
```go
&wpxt.Job{
Name: "Job using context",
Function: func() (any, error) {
timeout := 10 * time.Second
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
result, err := LongRunningTaskWithContext(ctx)
if err != nil {
return nil, err
}
return result, nil
},
}
```
### Using Retry
You can use something like [backoff](https://github.com/cenkalti/backoff) for this (just an example).
The point is: you have full control over every job.
```go
&wpxt.Job{
Name: "Job using retry",
Function: func() (any, error) {
work := func() (string, error) {
if /* Some result is an error */ {
return "", theError
}
return "IT SUCCEEDED", nil
}
expBackoff := backoff.WithBackOff(backoff.NewExponentialBackOff())
result, err := backoff.Retry(ctx, work, expBackoff)
if err != nil {
return nil, err
}
return result, nil
},
}
```