https://github.com/bsv-blockchain/go-batcher
High-performance batch processing for Go applications
https://github.com/bsv-blockchain/go-batcher
batcher batching bitcoin bitcoinsv bsv golang teranode
Last synced: 21 days ago
JSON representation
High-performance batch processing for Go applications
- Host: GitHub
- URL: https://github.com/bsv-blockchain/go-batcher
- Owner: bsv-blockchain
- License: other
- Created: 2025-07-17T18:28:42.000Z (11 months ago)
- Default Branch: master
- Last Pushed: 2026-05-01T13:05:01.000Z (27 days ago)
- Last Synced: 2026-05-01T13:27:43.970Z (27 days ago)
- Topics: batcher, batching, bitcoin, bitcoinsv, bsv, golang, teranode
- Language: Go
- Homepage:
- Size: 1.02 MB
- Stars: 1
- Watchers: 0
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: .github/SECURITY.md
- Support: .github/SUPPORT.md
- Agents: .github/AGENTS.md
Awesome Lists containing this project
README
# ⚡ go-batcher
**High-performance batch processing for Go applications**
CI / CD
Quality
Security
Community
### Project Navigation
📦 Installation
🎯 What's Inside
📚 Documentation
🧪 Examples & Tests
⚡ Benchmarks
🛠️ Code Standards
🤖 AI Usage
🤝 Contributing
👥 Maintainers
📝 License
## 🎯 What's Inside
### Lightning-Fast Batch Processing in Action
```go
package main
import (
"fmt"
"time"
"github.com/bsv-blockchain/go-batcher"
)
func main() {
// Create a batcher that processes items every 100ms or when batch size hits 1000
b := batcher.New[string](
1000, // batch size
100*time.Millisecond, // timeout interval
func(batch []*string) { // processor function
fmt.Printf("⚡ Processing %d items in one go!\n", len(batch))
// Your batch processing logic here
for _, item := range batch {
fmt.Printf("Processing: %s\n", *item)
}
},
true, // background processing
)
// Feed items - they'll be intelligently batched
for i := 0; i < 5000; i++ {
item := fmt.Sprintf("item-%d", i)
b.Put(&item)
}
// Process any remaining items before shutdown
b.Trigger()
// Note: The batcher worker runs indefinitely - use context cancellation for cleanup
}
```
### Constructor Variants
The `go-batcher` library provides several constructor options to fit different use cases:
```go
// Basic batcher - simple batching with size and timeout triggers
b := batcher.New[string](100, time.Second, processFn, true)
// With slice pooling - reduces memory allocations for high-throughput scenarios
b := batcher.NewWithPool[string](100, time.Second, processFn, true)
// With automatic deduplication - filters duplicate items within a 1-minute window
b := batcher.NewWithDeduplication[string](100, time.Second, processFn, true)
// Combined pooling and deduplication - maximum performance with duplicate filtering
b := batcher.NewWithDeduplicationAndPool[string](100, time.Second, processFn, true)
```
### Why You'll Love This Batcher
* **Blazing Performance** – Process millions of items with minimal overhead ([benchmarks](#benchmark-results): 135 ns/op)
* **Smart Batching** – Auto-groups by size or time interval, whichever comes first
* **Optional Deduplication** – Built-in dedup variant ensures each item is processed only once within a time window
* **Memory Pool Optimization** – Optional slice pooling reduces GC pressure in high-throughput scenarios
* **Thread-Safe by Design** – Concurrent Put() from multiple goroutines without worry
* **Time-Partitioned Storage** – Efficient memory usage with automatic cleanup (dedup variant)
* **Minimal Dependencies** – Pure Go with only essential external dependencies
* **Flexible Configuration** – Multiple constructor variants for different use cases
* **Production-Ready** – Battle-tested with full test coverage and benchmarks
Perfect for high-throughput scenarios like log aggregation, metrics collection, event processing, or any situation where you need to efficiently batch operations for downstream systems.
## 📦 Installation
**go-batcher** requires a [supported release of Go](https://golang.org/doc/devel/release.html#policy).
```shell script
go get -u github.com/bsv-blockchain/go-batcher
```
## 📈 Observability
All constructors accept a trailing variadic `...Option` so you can attach a
logger, Prometheus metrics, an OpenTelemetry tracer, and a name without
breaking the existing API. Omit the options and you get the original zero-cost
behaviour — defaults are no-op.
```go
m := batcher.NewPrometheusMetrics(reg, "myservice", "batcher")
store := batcher.New(100, 5*time.Second, fn, true,
batcher.WithName("store"), // distinguishes batchers in metrics & traces
batcher.WithLogger(logger), // ulogger.Logger satisfies it directly
batcher.WithMetrics(m), // share one Metrics across all batchers
batcher.WithTracer(otel.Tracer("batcher")),
)
// Tracing-aware enqueue: the SpanContext from ctx is recorded as a link on
// the batch span when this item is flushed.
store.PutCtx(ctx, item)
```
When a service runs multiple batchers, share one `Metrics` provider and
distinguish them by `WithName`:
```go
m := batcher.NewPrometheusMetrics(reg, "myservice", "batcher")
storeBatcher := batcher.New(..., batcher.WithName("store"), batcher.WithMetrics(m))
getBatcher := batcher.New(..., batcher.WithName("get"), batcher.WithMetrics(m))
setMinedBatcher := batcher.New(..., batcher.WithName("set_mined"), batcher.WithMetrics(m))
```
All series carry a `batcher` label so each one can be graphed independently.
Series exported (under your chosen `__…`):
| Metric | Type | Labels |
|-----------------------------------|----------------|-------------------------|
| `enqueued_total` | counter | `batcher` |
| `enqueue_blocked_seconds` | histogram | `batcher` |
| `batches_total` | counter | `batcher`, `reason` |
| `batch_size` | histogram | `batcher` |
| `batch_duration_seconds` | histogram | `batcher` |
| `backpressure_wait_seconds` | histogram | `batcher` |
| `dedup_total` | counter | `batcher`, `result` |
| `panic_total` | counter | `batcher` |
`reason` ∈ `{size, timeout, manual, drain, shutdown}`; `result` ∈ `{hit, miss}`.
Each batch dispatch is wrapped in an OTel span named `.flush` with
attributes `batcher.name`, `batcher.batch_size`, `batcher.reason`. Span links
are added for every item enqueued via `PutCtx`. Panics in the user batch
function are recovered, logged, and recorded on both the panic counter and
the span (status = error) — the worker keeps running.
## 📚 Documentation
- **API Reference** – Dive into the godocs at [pkg.go.dev/github.com/bsv-blockchain/go-batcher](https://pkg.go.dev/github.com/bsv-blockchain/go-batcher)
- **Usage Examples** – Browse practical patterns either the [examples directory](examples) or view the [example functions](batcher_example_test.go)
- **Benchmarks** – Check the latest numbers in the [benchmark results](#benchmark-results)
- **Test Suite** – Review both the [unit tests](batcher_test.go) and [fuzz tests](batcher_fuzz_test.go) (powered by [`testify`](https://github.com/stretchr/testify))
Development Build Commands
Get the [MAGE-X](https://github.com/mrz1836/mage-x) build tool for development:
```shell script
go install github.com/mrz1836/mage-x/cmd/magex@latest
```
View all build commands
```bash script
magex help
```
Repository Features
This repository includes 25+ built-in features covering CI/CD, security, code quality, developer experience, and community tooling.
**[View the full Repository Features list →](.github/docs/repository-features.md)**
Library Deployment
This project uses [goreleaser](https://github.com/goreleaser/goreleaser) for streamlined binary and library deployment to GitHub. To get started, install it via:
```bash
brew install goreleaser
```
The release process is defined in the [.goreleaser.yml](.goreleaser.yml) configuration file.
Then create and push a new Git tag using:
```bash
magex version:bump push=true bump=patch branch=master
```
This process ensures consistent, repeatable releases with properly versioned artifacts and citation metadata.
Pre-commit Hooks
Set up the Go-Pre-commit System to run the same formatting, linting, and tests defined in [AGENTS.md](.github/AGENTS.md) before every commit:
```bash
go install github.com/mrz1836/go-pre-commit/cmd/go-pre-commit@latest
go-pre-commit install
```
The system is configured via [modular env files](.github/env/README.md) and provides 17x faster execution than traditional Python-based pre-commit hooks. See the [complete documentation](http://github.com/mrz1836/go-pre-commit) for details.
GitHub Workflows
All workflows are driven by modular configuration in [`.github/env/`](.github/env/README.md) — no YAML editing required.
**[View all workflows and the control center →](.github/docs/workflows.md)**
Updating Dependencies
To update all dependencies (Go modules, linters, and related tools), run:
```bash
magex deps:update
```
This command ensures all dependencies are brought up to date in a single step, including Go modules and any tools managed by [MAGE-X](https://github.com/mrz1836/mage-x). It is the recommended way to keep your development environment and CI in sync with the latest versions.
## 🧪 Examples & Tests
All unit tests and [examples](examples) run via [GitHub Actions](https://github.com/bsv-blockchain/go-batcher/actions) and use [Go version 1.25.x](https://go.dev/doc/go1.25). View the [configuration file](.github/workflows/fortress.yml).
Run all tests (fast):
```bash script
magex test
```
Run all tests with race detector (slower):
```bash script
magex test:race
```
## ⚡ Benchmarks
Run the Go [benchmarks](batcher_benchmark_test.go):
```bash script
magex bench
```
### Benchmark Results
| Benchmark | Description | ns/op | B/op | allocs/op |
|--------------------------------------------------------------------------------------|----------------------------------|--------:|------:|----------:|
| [BenchmarkBatcherPut](batcher_comprehensive_benchmark_test.go) | Basic Put operation | 135.1 | 8 | 0 |
| [BenchmarkBatcherPutParallel](batcher_comprehensive_benchmark_test.go) | Concurrent Put operations | 310.0 | 9 | 0 |
| [BenchmarkPutComparison/Put](benchmark_comparison_test.go) | Put operation (non-blocking) | 300.7 | 9 | 0 |
| [BenchmarkPutComparison/PutWithPool](benchmark_comparison_test.go) | Put with slice pooling | 309.9 | 1 | 0 |
| [BenchmarkWithPoolComparison/Batcher](benchmark_comparison_test.go) | Standard batcher | 171.2 | 18 | 1 |
| [BenchmarkWithPoolComparison/WithPool](benchmark_comparison_test.go) | Pooled batcher | 184.0 | 9 | 1 |
| [BenchmarkTimePartitionedMapSet](batcher_comprehensive_benchmark_test.go) | Map Set operation (bloom filter) | 366.7 | 147 | 6 |
| [BenchmarkTimePartitionedMapGet](batcher_comprehensive_benchmark_test.go) | Map Get operation (bloom filter) | 80.5 | 39 | 2 |
| [BenchmarkBatcherWithDedupPut](batcher_comprehensive_benchmark_test.go) | Put with deduplication | 740.1 | 166 | 7 |
| [BenchmarkBatcher](batcher_benchmark_test.go) | Full batch processing (1M items) | 1,081ms | 710MB | 1.9M |
| [BenchmarkBatcherWithDeduplication](batcher_benchmark_test.go) | Deduplication processing | 90.7 | 13 | 0 |
> Performance benchmarks for the core functions in this library, executed on an Apple M1 Max (ARM64).
> The benchmarks demonstrate excellent performance with minimal allocations for basic operations.
## 🛠️ Code Standards
Read more about this Go project's [code standards](.github/CODE_STANDARDS.md).
## 🤖 AI Usage & Assistant Guidelines
Read the [AI Usage & Assistant Guidelines](.github/tech-conventions/ai-compliance.md) for details on how AI is used in this project and how to interact with AI assistants.
## 👥 Maintainers
| [
](https://github.com/icellan) | [
](https://github.com/galt-tr) | [
](https://github.com/mrz1836) |
|:--------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------:|
| [Siggi](https://github.com/icellan) | [Dylan](https://github.com/galt-tr) | [MrZ](https://github.com/mrz1836) |
## 🤝 Contributing
View the [contributing guidelines](.github/CONTRIBUTING.md) and please follow the [code of conduct](.github/CODE_OF_CONDUCT.md).
### How can I help?
All kinds of contributions are welcome :raised_hands:!
The most basic way to show your support is to star :star2: the project, or to raise issues :speech_balloon:.
[](https://github.com/bsv-blockchain/go-batcher/stargazers)
## 📝 License
[](LICENSE)