https://github.com/pacedotdev/batch
Very simple batching API for Go.
https://github.com/pacedotdev/batch
golang
Last synced: about 1 month ago
JSON representation
Very simple batching API for Go.
- Host: GitHub
- URL: https://github.com/pacedotdev/batch
- Owner: pacedotdev
- License: mit
- Created: 2019-11-28T11:59:11.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2021-10-22T05:16:51.000Z (over 4 years ago)
- Last Synced: 2024-06-19T02:09:35.107Z (over 1 year ago)
- Topics: golang
- Language: Go
- Homepage: https://pace.dev/blog/2020/02/13/batching-operations-in-go-by-mat-ryer
- Size: 9.77 KB
- Stars: 90
- Watchers: 6
- Forks: 7
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# batch
Very simple batching API for Go.
This repo came out of a [tutorial blog post](https://pace.dev/blog/2020/02/13/batching-operations-in-go-by-mat-ryer).
## Install
You might as well just copy the [batch.go](https://github.com/pacedotdev/batch/blob/master/batch.go) file into your own project (and the [batch_test.go](https://github.com/pacedotdev/batch/blob/master/batch_test.go) while you're at it for future generations) rather than adding a dependency.
But it is maintained as a Go module which you can get with:
```bash
go get github.com/pacedotdev/batch
```
## Usage
Import it:
```go
import (
"github.com/pacedotdev/batch"
)
```
If we wanted to perform an operation in batches of ten, regardless of how many items were in the slice, we could use the `batch.All` function by passing in the total number of items `len(items)`, and the `batchSize`. The function is a callback that gets called for each batch, with `start` and `end` indexes which you can use to re-slice the `items`:
```go
items, err := getAllItemsFromRequest(r)
if err != nil {
return errors.Wrap(err, "getAllItemsFromRequest")
}
batchSize := 10
err := batch.All(len(items), batchSize, func(start, end int) error {
batchItems := items[start:end]
if err := performSomeRemoteThing(ctx, batchItems); err != nil {
return errors.Wrap(err, "performSomeRemoteThing")
}
})
if err != nil {
return err
}
```
In this example, if we got `105` items, the `performSomeRemoteThing` function would get called eleven times, each time with a different page of `10` items (the `batchSize`) except the last time, when it would be a slice of the remaining five items.
The mechanics are [fairly simple](https://github.com/pacedotdev/batch/blob/master/batch.go), but the code is encapsulated and well tested.
### Using context for cancellation
You can use a `context.Context` for cancellation by adding a check into the body of the `BatchFunc`. The following code will cancel batching if the user aborts the HTTP request.
```go
ctx := r.Context()
items, err := getAllItemsFromRequest(r)
if err != nil {
return errors.Wrap(err, "getAllItemsFromRequest")
}
batchSize := 10
err := batch.All(len(items), batchSize, func(start, end int) error {
if err := ctx.Err(); err != nil {
return err
}
batchItems := items[start:end]
if err := performSomeRemoteThing(ctx, batchItems); err != nil {
return errors.Wrap(err, "performSomeRemoteThing")
}
})
if err != nil {
return err
}
```