Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/catmullet/go-workers
👷 Library for safely running groups of workers concurrently or consecutively that require input and output through channels
https://github.com/catmullet/go-workers
concurrency go go-concurrency go-worker go-workerpool golang golang-library multiple-workers pool pools worker-functions workers
Last synced: 14 days ago
JSON representation
👷 Library for safely running groups of workers concurrently or consecutively that require input and output through channels
- Host: GitHub
- URL: https://github.com/catmullet/go-workers
- Owner: catmullet
- License: mit
- Archived: true
- Created: 2020-10-06T15:39:43.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2022-01-13T07:41:18.000Z (almost 3 years ago)
- Last Synced: 2024-07-31T20:51:53.648Z (3 months ago)
- Topics: concurrency, go, go-concurrency, go-worker, go-workerpool, golang, golang-library, multiple-workers, pool, pools, worker-functions, workers
- Language: Go
- Homepage: https://github.com/catmullet/go-workers/wiki/Getting-Started
- Size: 974 KB
- Stars: 164
- Watchers: 3
- Forks: 16
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go - go-workers - Easily and safely run workers for large data processing pipelines. (Goroutines / Search and Analytic Databases)
- go-awesome - go-workers - Safely run a group of workers simultaneously, input and output across channels (Open source library / Coroutine/Thread)
- awesome-go-extra - go-workers - 10-06T15:39:43Z|2022-01-13T07:41:18Z| (Goroutines / Advanced Console UIs)
README
[](https://github.com/avelino/awesome-go#goroutines)
[](https://codeclimate.com/github/catmullet/go-workers/maintainability)
[](http://gocover.io/github.com/catmullet/go-workers)
[](https://pkg.go.dev/github.com/catmullet/go-workers)
# Examples
* [Quickstart](https://github.com/catmullet/go-workers/blob/master/examples/quickstart/quickstart.go)
* [Multiple Go Workers](https://github.com/catmullet/go-workers/blob/master/examples/multiple_workers/multipleworkers.go)
* [Passing Fields](https://github.com/catmullet/go-workers/blob/master/examples/passing_fields/passingfields.go)
# Getting Started
### Pull in the dependency
```zsh
go get github.com/catmullet/go-workers
```
### Add the import to your project
giving an alias helps since go-workers doesn't exactly follow conventions.
_(If you're using a JetBrains IDE it should automatically give it an alias)_
```go
import (
workers "github.com/catmullet/go-workers"
)
```
### Create a new worker 
The NewWorker factory method returns a new worker.
_(Method chaining can be performed on this method like calling .Work() immediately after.)_
```go
type MyWorker struct {}
func NewMyWorker() Worker {
return &MyWorker{}
}
func (my *MyWorker) Work(in interface{}, out chan<- interface{}) error {
// work iteration here
}
runner := workers.NewRunner(ctx, NewMyWorker(), numberOfWorkers)
```
### Send work to worker
Send accepts an interface. So send it anything you want.
```go
runner.Send("Hello World")
```
### Wait for the worker to finish and handle errors
Any error that bubbles up from your worker functions will return here.
```go
if err := runner.Wait(); err != nil {
//Handle error
}
```
## Working With Multiple Workers
### Passing work form one worker to the next
By using the InFrom method you can tell `workerTwo` to accept output from `workerOne`
```go
runnerOne := workers.NewRunner(ctx, NewMyWorker(), 100).Work()
runnerTwo := workers.NewRunner(ctx, NewMyWorkerTwo(), 100).InFrom(workerOne).Work()
```
### Accepting output from multiple workers
It is possible to accept output from more than one worker but it is up to you to determine what is coming from which worker. (They will send on the same channel.)
```go
runnerOne := workers.NewRunner(ctx, NewMyWorker(), 100).Work()
runnerTwo := workers.NewRunner(ctx, NewMyWorkerTwo(), 100).Work()
runnerThree := workers.NewRunner(ctx, NewMyWorkerThree(), 100).InFrom(workerOne, workerTwo).Work()
```
## Passing Fields To Workers
### Adding Values
Fields can be passed via the workers object. Be sure as with any concurrency in Golang that your variables are concurrent safe. Most often the golang documentation will state the package or parts of it are concurrent safe. If it does not state so there is a good chance it isn't. Use the sync package to lock and unlock for writes on unsafe variables. (It is good practice NOT to defer in the work function.)
**ONLY** use the `Send()` method to get data into your worker. It is not shared memory unlike the worker objects values.
```go
type MyWorker struct {
message string
}
func NewMyWorker(message string) Worker {
return &MyWorker{message}
}
func (my *MyWorker) Work(in interface{}, out chan<- interface{}) error {
fmt.Println(my.message)
}
runner := workers.NewRunner(ctx, NewMyWorker(), 100).Work()
```
### Setting Timeouts or Deadlines
If your workers needs to stop at a deadline or you just need to have a timeout use the SetTimeout or SetDeadline methods. (These must be in place before setting the workers off to work.)
```go
// Setting a timeout of 2 seconds
timeoutRunner.SetTimeout(2 * time.Second)
// Setting a deadline of 4 hours from now
deadlineRunner.SetDeadline(time.Now().Add(4 * time.Hour))
func workerFunction(in interface{}, out chan<- interface{} error {
fmt.Println(in)
time.Sleep(1 * time.Second)
}
```
## Performance Hints
### Buffered Writer
If you want to write out to a file or just stdout you can use SetWriterOut(writer io.Writer). The worker will have the following methods available
```go
runner.Println()
runner.Printf()
runner.Print()
```
The workers use a buffered writer for output and can be up to 3 times faster than the fmt package. Just be mindful it won't write out to the console as quickly as an unbuffered writer. It will sync and eventually flush everything at the end, making it ideal for writing out to a file.
### Using GOGC env variable
If your application is based solely around using workers, consider upping the percentage of when the scheduler will garbage collect. (ex. GOGC=200) 200% -> 300% is a good starting point. Make sure your machine has some good memory behind it.
By upping the percentage your application will interupt the workers less, meaning they get more work done. However, be aware of the rest of your applications needs when modifying this variable.
### Using GOMAXPROCS env variable
For workers that run quick bursts of lots of simple data consider lowering the GOMAXPROCS. Be carfeful though, this can affect your entire applicaitons performance. Profile your application and benchmark it. See where your application runs best.