https://github.com/woorui/async-buffer
Generic Asynchronous data buffer with Automatic and Manual flushing
https://github.com/woorui/async-buffer
async buffer flush go golang goroutine telegraf
Last synced: 4 months ago
JSON representation
Generic Asynchronous data buffer with Automatic and Manual flushing
- Host: GitHub
- URL: https://github.com/woorui/async-buffer
- Owner: woorui
- License: mit
- Created: 2022-04-24T12:43:34.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-08-21T17:14:36.000Z (about 2 years ago)
- Last Synced: 2025-05-02T02:39:02.545Z (5 months ago)
- Topics: async, buffer, flush, go, golang, goroutine, telegraf
- Language: Go
- Homepage:
- Size: 29.3 KB
- Stars: 14
- Watchers: 1
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# async-buffer
[](https://github.com/woorui/async-buffer/actions/workflows/go.yml)
[](https://codecov.io/gh/woorui/async-buffer)
[](https://pkg.go.dev/github.com/woorui/async-buffer)
The async-buffer buffer data that can be flushed when reach threshold or duration limit. It is multi-goroutinue safe.
**It only support go1.18 or later**
## Why you need it?
### An Usecase:
You have a message queue subscriber server.
The Server receives messages one by one and inserts them into your database,
But there is a big performance gap between one by one insertion and batch insertion to your database.
So that to use async-buffer to buffer data then find timing to batch insert them.
## Installation
```
go get -u github.com/woorui/async-buffer
```
## Quick start
The `Write`, `Flush`, `Close` api are goroutinue-safed.
```go
package main
import (
"context"
"fmt"
"time"
buffer "github.com/woorui/async-buffer"
)
// pp implements Flusher interface
type pp struct{}
func (p *pp) Flush(strs []string) error {
return print(strs)
}
func print(strs []string) error {
fmt.Printf("print: %v \n", strs)
return nil
}
func main() {
// can also call buffer.FlushFunc` to adapt a function to Flusher
buf := buffer.New[string](&pp{}, buffer.Option[string]{
Threshold: 5,
FlushInterval: 3 * time.Second,
WriteTimeout: time.Second,
FlushTimeout: time.Second,
ErrHandler: func(err error, t []string) { fmt.Printf("err: %v, ele: %v", err, t) },
})
// data maybe loss if Close() is not be called
defer buf.Close()
// 1. flush at threshold
buf.Write("a", "b", "c", "d", "e", "f")
// 2. time to flush automatically
buf.Write("aaaaa")
buf.Write("bbbbb")
buf.Write("ccccc", "ddddd")
time.Sleep(5 * time.Second)
// 3. flush manually and write call `WriteWithContext`
buf.WriteWithContext(context.Background(), "eeeee", "fffff")
buf.Flush()
}
```
## License
[MIT License](https://github.com/woorui/async-buffer/blob/main/LICENSE)