Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/a8m/kinesis-producer
An aggregated records producer for Amazon Kinesis
https://github.com/a8m/kinesis-producer
amazon-kinesis go kinesis kinesis-producer kpl kpl-aggregation producer
Last synced: 5 days ago
JSON representation
An aggregated records producer for Amazon Kinesis
- Host: GitHub
- URL: https://github.com/a8m/kinesis-producer
- Owner: a8m
- License: mit-0
- Created: 2016-03-19T16:29:35.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2024-09-16T12:06:06.000Z (4 months ago)
- Last Synced: 2024-10-20T22:53:20.577Z (3 months ago)
- Topics: amazon-kinesis, go, kinesis, kinesis-producer, kpl, kpl-aggregation, producer
- Language: Go
- Homepage:
- Size: 49.8 KB
- Stars: 148
- Watchers: 6
- Forks: 51
- Open Issues: 14
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Amazon kinesis producer [![Build status][travis-image]][travis-url] [![License][license-image]][license-url] [![GoDoc][godoc-img]][godoc-url]
> A KPL-like batch producer for Amazon Kinesis built on top of the official Go AWS SDK
and using the same aggregation format that [KPL][kpl-url] use.
### Useful links
- [Documentation][godoc-url]
- [Aggregation format][aggregation-format-url]
- [Considerations When Using KPL Aggregation][kpl-aggregation]
- [Consumer De-aggregation][de-aggregation]
### Example
```go
package main
import (
"time"
"github.com/sirupsen/logrus"
"github.com/a8m/kinesis-producer"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/kinesis"
)
func main() {
client := kinesis.New(session.New(aws.NewConfig()))
pr := producer.New(&producer.Config{
StreamName: "test",
BacklogCount: 2000,
Client: client
})
pr.Start()
// Handle failures
go func() {
for r := range pr.NotifyFailures() {
// r contains `Data`, `PartitionKey` and `Error()`
log.Error(r)
}
}()
go func() {
for i := 0; i < 5000; i++ {
err := pr.Put([]byte("foo"), "bar")
if err != nil {
log.WithError(err).Fatal("error producing")
}
}
}()
time.Sleep(3 * time.Second)
pr.Stop()
}
```
#### Specifying logger implementation
`producer.Config` takes an optional `logging.Logger` implementation.
##### Using a custom logger
```go
customLogger := &CustomLogger{}
&producer.Config{
StreamName: "test",
BacklogCount: 2000,
Client: client,
Logger: customLogger,
}
```
#### Using logrus
```go
import (
"github.com/sirupsen/logrus"
producer "github.com/a8m/kinesis-producer"
"github.com/a8m/kinesis-producer/loggers"
)
log := logrus.New()
&producer.Config{
StreamName: "test",
BacklogCount: 2000,
Client: client,
Logger: loggers.Logrus(log),
}
```
kinesis-producer ships with three logger implementations.
- `producer.Standard` used the standard library logger
- `loggers.Logrus` uses logrus logger
- `loggers.Zap` uses zap logger
### License
MIT
[godoc-url]: https://godoc.org/github.com/a8m/kinesis-producer
[godoc-img]: https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square
[kpl-url]: https://github.com/awslabs/amazon-kinesis-producer
[de-aggregation]: http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-kpl-consumer-deaggregation.html
[kpl-aggregation]: http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-producer-adv-aggregation.html
[aggregation-format-url]: https://github.com/a8m/kinesis-producer/blob/master/aggregation-format.md
[license-image]: https://img.shields.io/badge/license-MIT-blue.svg?style=flat-square
[license-url]: LICENSE
[travis-image]: https://img.shields.io/travis/a8m/kinesis-producer.svg?style=flat-square
[travis-url]: https://travis-ci.org/a8m/kinesis-producer