https://github.com/karafka/waterdrop
Standalone Karafka library for producing Kafka messages
https://github.com/karafka/waterdrop
apache-kafka apache-kafka-producer kafka karafka karafka-application karafka-framework ruby ruby-kafka rubygem rubygems waterdrop
Last synced: 7 months ago
JSON representation
Standalone Karafka library for producing Kafka messages
- Host: GitHub
- URL: https://github.com/karafka/waterdrop
- Owner: karafka
- License: other
- Created: 2015-07-30T15:10:32.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2025-06-23T11:53:20.000Z (8 months ago)
- Last Synced: 2025-07-05T02:46:08.264Z (8 months ago)
- Topics: apache-kafka, apache-kafka-producer, kafka, karafka, karafka-application, karafka-framework, ruby, ruby-kafka, rubygem, rubygems, waterdrop
- Language: Ruby
- Homepage: https://karafka.io
- Size: 1.11 MB
- Stars: 265
- Watchers: 9
- Forks: 40
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# WaterDrop
[](https://github.com/karafka/waterdrop/actions?query=workflow%3Aci)
[](http://badge.fury.io/rb/waterdrop)
[](https://slack.karafka.io)
WaterDrop is a standalone gem that sends messages to Kafka easily with an extra validation layer. It is a part of the [Karafka](https://github.com/karafka/karafka) ecosystem.
It:
- Is thread-safe
- Supports sync and async producing
- Supports transactions
- Supports buffering
- Supports producing to multiple clusters
- Supports multiple delivery policies
- Supports per-topic configuration alterations (variants)
- Works with Kafka `1.0+` and Ruby `3.1+`
- Works with and without Karafka
## Documentation
Karafka ecosystem components documentation, including WaterDrop, can be found [here](https://karafka.io/docs/#waterdrop).
## Getting Started
If you want to both produce and consume messages, please use [Karafka](https://github.com/karafka/karafka/). It integrates WaterDrop automatically.
To get started with WaterDrop:
1. Add it to your Gemfile:
```bash
bundle add waterdrop
```
2. Create and configure a producer:
```ruby
producer = WaterDrop::Producer.new do |config|
config.deliver = true
config.kafka = {
'bootstrap.servers': 'localhost:9092',
'request.required.acks': 1
}
end
```
3. Use it as follows:
```ruby
# sync producing
producer.produce_sync(topic: 'my-topic', payload: 'my message')
# or for async
producer.produce_async(topic: 'my-topic', payload: 'my message')
# or in sync batches
producer.produce_many_sync(
[
{ topic: 'my-topic', payload: 'my message'},
{ topic: 'my-topic', payload: 'my message'}
]
)
# and async batches
producer.produce_many_async(
[
{ topic: 'my-topic', payload: 'my message'},
{ topic: 'my-topic', payload: 'my message'}
]
)
# transactions
producer.transaction do
producer.produce_async(topic: 'my-topic', payload: 'my message')
producer.produce_async(topic: 'my-topic', payload: 'my message')
end
```