https://github.com/nikandfor/batch
Reliable batches with ease.
https://github.com/nikandfor/batch
batch batching concurrency go golang parallelism semaphore
Last synced: 3 months ago
JSON representation
Reliable batches with ease.
- Host: GitHub
- URL: https://github.com/nikandfor/batch
- Owner: nikandfor
- License: mit
- Created: 2023-12-16T21:31:15.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2025-04-01T01:30:56.000Z (about 1 year ago)
- Last Synced: 2025-04-01T02:29:02.711Z (about 1 year ago)
- Topics: batch, batching, concurrency, go, golang, parallelism, semaphore
- Language: Go
- Homepage:
- Size: 77.1 KB
- Stars: 6
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://pkg.go.dev/nikand.dev/go/batch?tab=doc)
[](https://github.com/nikandfor/batch/actions/workflows/go.yml)
[](https://circleci.com/gh/nikandfor/batch)
[](https://codecov.io/gh/nikandfor/batch)
[](https://goreportcard.com/report/nikand.dev/go/batch)
# batch
`batch` is a library to make concurrent work batcheable and reliable.
Each worker either has its work committed or gets an error.
> Hope is not a strategy. ([from Google SRE book](https://sre.google/sre-book/introduction/))
No more batch operations that add its data to a batch and go away hoping it would be committed.
This is all without timeouts, additional goroutines, allocations, and channels.
There is also [singleflight](./singleflight) lib in this module.
## How it works
* Each worker adds its work to a shared batch.
* If there are no more workers in the queue the last one executes commit, the others wait.
* Every worker in the batch gets the same result and error.
## Usage
```
// General pattern is
// Queue.In -> Enter -> defer Exit -> add work to the batch -> Commit/Cancel/panic/return
var sum int
bc := batch.Controller[int]{
Committer: func(ctx context.Context) (int, error) {
// commit sum
return sum, err
},
}
for j := 0; j < N; j++ {
go func(j int) {
bc.Queue().In() // let others know we are going to join
data := 1 // prepare data
idx := bc.Enter(true)
defer bc.Exit()
if idx == 0 { // we are first in the batch, reset it
sum = 0
}
sum += data // add data to the batch
res, err := bc.Commit(ctx)
if err != nil { // works the same as we had independent commit in each goroutine
_ = err
}
// batching is transparent for worker
_ = res // res is the sum returned by Commit
}(j)
}
```
See the all available options in [the doc](https://pkg.go.dev/nikand.dev/go/batch).
`batch` is error- and panic-proof, meaning that user code can return an error or panic at any point.
However, once all workers have exited the batch, its state is reset.
External state, such as `sum` in this case, remains the caller's responsibility and must be kept consistent.