https://github.com/streamdal/kng
High throughput oriented kafka wrapper lib
https://github.com/streamdal/kng
Last synced: about 1 year ago
JSON representation
High throughput oriented kafka wrapper lib
- Host: GitHub
- URL: https://github.com/streamdal/kng
- Owner: streamdal
- Created: 2021-04-07T01:24:26.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2022-12-28T22:44:08.000Z (over 3 years ago)
- Last Synced: 2025-03-24T20:11:12.500Z (about 1 year ago)
- Language: Go
- Size: 5.35 MB
- Stars: 5
- Watchers: 6
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
kng
===
`kng` is a kafka client wrapper lib for Go that is geared towards facilitating
high throughput message production and consumption.
It wraps both [segmentio/kafka-go](https://github.com/segmentio/kafka-go) and
[Shopify/sarama](https://github.com/Shopify/sarama) libs for different
functionality.
## Special bits
The publisher does a lot of stuff under the hood which dramatically improves
write performance.
* Automatically creates a dedicated publisher for the given topic IF a
publisher does not already exist.
* Starts a background publisher in a goroutine that will clear its queue on
an interval defined by `PublishInterval`.
* Publisher goroutine will be stopped if it is idle for longer than
`WorkerIdleTimeout`.
* To avoid kafka from rejecting a batch containing too many messages, the
publisher will automatically divide the batch into "sub-batches" (whose
size is defined by `DefaultSubBatchSize`.
## Why multiple libs?
The segment lib works great for consuming and producing (and has internal
batching mechanisms) but doesn't work as well when dealing with topic
management and other administrative tasks.
We also prefer the `kafka-go` interface but some things just work better with
the `sarama` lib.
## Name
`kng` is "kafka next generation" which is _mildly_ poking fun at the late 90's
software naming pattern that included "next generation" in its name (such as
`syslog-ng`) :smiley: