Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/aws/go-kafka-event-source
Go/Kafka client library for developing event sourcing applications
https://github.com/aws/go-kafka-event-source
event-sourcing go golang kafka kafka-consumer kafka-producer kafka-streams
Last synced: 2 months ago
JSON representation
Go/Kafka client library for developing event sourcing applications
- Host: GitHub
- URL: https://github.com/aws/go-kafka-event-source
- Owner: aws
- License: apache-2.0
- Created: 2022-08-28T19:02:04.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-18T22:58:12.000Z (about 1 year ago)
- Last Synced: 2024-10-10T21:28:49.305Z (3 months ago)
- Topics: event-sourcing, go, golang, kafka, kafka-consumer, kafka-producer, kafka-streams
- Language: Go
- Homepage:
- Size: 298 KB
- Stars: 13
- Watchers: 4
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
## Go Kafka Event Source
GKES (Go Kafka Event Source) attempts to fill the gaps ub the Go/Kafka library ecosystem. It supplies EOS (Exactly Once Semantics),
local state stores and incremental consumer rebalancing to Go Kafka consumer applications, making it a viable alternative to
a traditional Kafka Streams application written in Java.
## Getting Started
This repository is organized into several modules. The primary module being "streams".
To use GKES in your go program, simply add it as you would any other Go module.
```
go get github.com/aws/go-kafka-event-source/streams
```
If you are using AWS MSK, you may find the provided "msk" module useful for cluster connectivity, though this is not required.
```
go get github.com/aws/go-kafka-event-source/msk
```
For API documentation on these modules, see https://pkg.go.dev/github.com/aws/go-kafka-event-source/streams and https://pkg.go.dev/github.com/aws/go-kafka-event-source/msk. The [docs](./docs) folder also contains detailed examples.
## Project Status
GKES is usable in it's current form but there are a few things slated for the very near future. This library is being used by AWS for internal workloads.
- [X] More comprehensive documentation for and async processing - [Complete](./docs/asyncprocessing.md)
- [ ] More comprehensive documentation for StateStores - In Progress
- [X] Per topic partitioner support for EventContext.Forward() and Producer/BatchProducer APIs, Currently, only the Java default murmur2 partitioner is supported - Complete
- [ ] Instructions for testing locally - In progress
- [ ] More robust functional tests for async processing. Though this is tested extenisively using a local test harness, this process needs to be more repeatable and available to contributors - In progress
## Compatibility Notes
It is recommnded tp use Go v1.19.2 or greater for GKES. There was a known compiler issue in previous versions of Go 1.19 which prevented modules using GKES from compiling. There was a back-port fix made to previous versions of Go, but it probably simpler and safer to update your Go environment to the latest available if you run into this issue.
GKES has been extensively test with Kafka 3.2 but should be fine to use with any Kafka version > 2.5.1. Kafka versions < 2.5.1 are not likely to be compatible with GKES due to transaction semantics, and they have not been tested.
## Security
See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
## License
This project is licensed under the Apache-2.0 License.