https://github.com/h12w/kpax
A modular & idiomatic Kafka client in Go
https://github.com/h12w/kpax
go golang golang-package golang-tools kafka
Last synced: 5 months ago
JSON representation
A modular & idiomatic Kafka client in Go
- Host: GitHub
- URL: https://github.com/h12w/kpax
- Owner: h12w
- License: bsd-2-clause
- Created: 2015-07-23T10:48:49.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2018-05-05T16:19:20.000Z (about 8 years ago)
- Last Synced: 2024-06-19T02:05:18.544Z (about 2 years ago)
- Topics: go, golang, golang-package, golang-tools, kafka
- Language: Go
- Homepage:
- Size: 378 KB
- Stars: 21
- Watchers: 4
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
kpax: a modular & idiomatic Kafka client in Go
==============================================
[](https://godoc.org/h12.io/kpax)
Install
-------
```
go get -u h12.io/kpax.v1
```
### Compatibility
h12.io/kpax.v1 is compatible with Kafka Server 0.8.2.
Design
------
The client is built on top of Kafka Wire Protocol (i.e. low-level API). The protocol related types & marshal/unmarshal functions are automatically generated by [wipro](https://github.com/h12w/wipro) from [the HTML spec](https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol)).
`(-)` means to be done.
### Sub packages
* **model** is an abstraction model for request, response, broker and cluster
* **broker** is a lazy, asynchronous and recoverable round tripper that talks to a single Kafka broker
* **cluster** is a metadata manager that talks to a Kafka cluster
* **proto** contains both low level API and a "middle" level facade
* **producer**: fault tolerant high-level producer (batching and partitioning strategy)
* **consumer**: fault tolerant high-level consumer (consumer group and offset commit)
* **log**: replaceable global logger
* **cmd**
- **kpax**: command line tool to help with Kafka programming
### Error Handling
* broker
+ fail fast: timeout returns error immediately
+ release resources carefully
+ reconnect when requested next time
* client
+ metadata reload lazily (only when a leader/coordinator cannot be found in cache)
+ leader/coordinator should be deleted on error
* producer
+ fail over to another partition
+ failed partition will be retried again after a period of time
+ partition expand (-)
* consumer
+ just loop & wait on error
+ partition expand (-)
* graceful shutdown (-)
### Efficiency
* efficiency
+ batching
- consumer response
- consumer request (-)
- producer (-)
+ decompression
- snappy
- gzip (-)
+ compression
- snappy (-)
- gzip (-)
Author
------
[Hǎiliàng Wáng](https://github.com/h12w)
Contributors
------------
* [Tao Huang](https://github.com/AnotherGoogleFans)