https://github.com/deeptiman/go-batch
A Simple Batch Processing library in Go
https://github.com/deeptiman/go-batch
batch-processing batch-reader concurrency concurrent-programming data-science go-library go-modules golang golang-channel golang-concurrency golang-library golang-tools parallel-computing parallel-processing parallel-programming supply-chain-data-science workers
Last synced: 2 months ago
JSON representation
A Simple Batch Processing library in Go
- Host: GitHub
- URL: https://github.com/deeptiman/go-batch
- Owner: Deeptiman
- License: mit
- Created: 2021-04-17T14:24:50.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2023-06-26T19:39:39.000Z (almost 2 years ago)
- Last Synced: 2025-04-17T17:37:17.628Z (2 months ago)
- Topics: batch-processing, batch-reader, concurrency, concurrent-programming, data-science, go-library, go-modules, golang, golang-channel, golang-concurrency, golang-library, golang-tools, parallel-computing, parallel-processing, parallel-programming, supply-chain-data-science, workers
- Language: Go
- Homepage:
- Size: 7.53 MB
- Stars: 23
- Watchers: 1
- Forks: 4
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# go-batch
go-batch is a batch processing library written in Go. The process execution has multiple stages to release a Batch to the client.
 [](https://goreportcard.com/report/github.com/Deeptiman/go-batch)
## Features
1. Client can use this library as an asynchronous batch processing for their application use case.
2. There are no restrictions on applying batch processing matrices to the library. The client can define the maximum no of items for a batch using the BatchOptions.
3. The library has a Workerpool that will faster the batch processing in concurrent scenarios.
## Demo
[](https://asciinema.org/a/2vi5gAHjsuTrB3tCBTGeSW6hq)
## Stages
1. Batch Reader receives the resource payload from the client and marshals the payload item into the BatchItem object.
``````````````````````````
type BatchItems struct {
Id int
BatchNo int
Item interface{}
}
``````````````````````````
2. BatchProducer has a Watcher channel that receives the marshal payload from the Batch reader. Watcher marks each BatchItem with a BatchNo and adds it to the []BatchItems array. After the batch itemCounter++ increases to the MaxItems [DefaultMaxItems: 100], the Batch gets
releases to the Consumer callback function.
3. BatchConsumer has a ConsumerFunc that gets invoke by BatchProducer as a callback function to send the prepared []BatchItems arrays. Then, the Consumer channel sends the
[]BatchItems to the Worker channel.
4. Workerline is the sync.WaitGroup synchronizes the workers to send the []BatchItems to the supply chain.
5. BatchSupplyChannel works as a bidirectional channel that requests for the []BatchItems to the Workerline and gets in the response.
6. ClientSupplyChannel is the delivery channel that works as a Supply line to sends the []BatchItems and the client receives by listening to the channel.
## Go Docs
Documentation at pkg.go.dev
## Installation
go get github.com/Deeptiman/go-batch
## Example
````````````````````````````````````````````````````
b := batch.NewBatch(batch.WithMaxItems(100))
go b.StartBatchProcessing()
for i := 1; i <= 1000; i++ {
b.Item <- &Resources{
id: i,
name: fmt.Sprintf("%s%d", "R-", i),
flag: false,
}
}
b.Close()
````````````````````````````````````````````````````
## Note
- In this version release, the library doesn't support starting concurrent BatchProcessing sessions.
## License
This project is licensed under the MIT License