https://github.com/dropwizard/dropwizard-kafka
A convenience library for Apache Kafka integration in a Dropwizard service.
https://github.com/dropwizard/dropwizard-kafka
dropwizard-kafka kafka kafka-client kafka-consumers
Last synced: about 1 year ago
JSON representation
A convenience library for Apache Kafka integration in a Dropwizard service.
- Host: GitHub
- URL: https://github.com/dropwizard/dropwizard-kafka
- Owner: dropwizard
- License: apache-2.0
- Created: 2019-05-09T20:59:02.000Z (almost 7 years ago)
- Default Branch: release/4.0.x
- Last Pushed: 2024-10-29T17:27:40.000Z (over 1 year ago)
- Last Synced: 2024-10-29T18:50:35.984Z (over 1 year ago)
- Topics: dropwizard-kafka, kafka, kafka-client, kafka-consumers
- Language: Java
- Homepage:
- Size: 469 KB
- Stars: 21
- Watchers: 11
- Forks: 12
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# dropwizard-kafka
[](https://github.com/dropwizard/dropwizard-kafka/actions?query=workflow%3ABuild)
[](https://sonarcloud.io/dashboard?id=dropwizard_dropwizard-kafka)
[](https://maven-badges.herokuapp.com/maven-central/io.dropwizard.modules/dropwizard-kafka/)
Provides easy integration for Dropwizard applications with the Apache Kafka client.
This bundle comes with out-of-the-box support for:
* YAML Configuration integration
* Producer and Consumer lifecycle management
* Producer and Cluster connection health checks
* Metrics integration for the Kafka client
* An easier way to create/configure Kafka consumers/producers than is offered by the base Kafka client
* Distributed tracing integration, using the [Brave Kafka client instrumentation library](https://github.com/openzipkin/brave/tree/master/instrumentation/kafka-clients).
For more information on Kafka, take a look at the official documentation here: http://kafka.apache.org/documentation/
## Dropwizard Version Support Matrix
| dropwizard-kafka | Dropwizard v1.3.x | Dropwizard v2.0.x | Dropwizard v2.1.x | Dropwizard v3.0.x | Dropwizard v4.0.x |
|------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
| v1.3.x | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: |
| v1.4.x | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: |
| v1.5.x | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: |
| v1.6.x | :x: | :white_check_mark: | :white_check_mark: | :x: | :x: |
| v1.7.x | :x: | :white_check_mark: | :white_check_mark: | :x: | :x: |
| v1.8.x | :x: | :x: | :white_check_mark: | :x: | :x: |
| v3.0.x | :x: | :x: | :x: | :white_check_mark: | :x: |
| v4.0.x | :x: | :x: | :x: | :x: | :white_check_mark: |
## Usage
Add dependency on library.
Maven:
```xml
io.dropwizard.modules
dropwizard-kafka
1.7.0
```
Gradle:
```groovy
compile "io.dropwizard.modules:dropwizard-kafka:$dropwizardVersion"
```
### Basic Kafka Producer
In your Dropwizard `Configuration` class, configure a `KafkaProducerFactory`:
```java
@Valid
@NotNull
@JsonProperty("producer")
private KafkaProducerFactory kafkaProducerFactory;
```
Then, in your `Application` class, you'll want to do something similar to the following:
```java
private final KafkaProducerBundle kafkaProducer = new KafkaProducerBundle() {
@Override
public KafkaProducerFactory getKafkaProducerFactory(ExampleConfiguration configuration) {
return configuration.getKafkaProducerFactory();
}
};
@Override
public void initialize(Bootstrap bootstrap) {
bootstrap.addBundle(kafkaProducer);
}
@Override
public void run(ExampleConfiguration config, Environment environment) {
final PersonEventProducer personEventProducer = new PersonEventProducer(kafkaProducer.getProducer());
environment.jersey().register(new PersonEventResource(personEventProducer));
}
```
Configure your factory in your `config.yml` file:
```yaml
producer:
type: basic
bootstrapServers:
- 127.0.0.1:9092
- 127.0.0.1:9093
- 127.0.0.1:9094
name: producerNameToBeUsedInMetrics
keySerializer:
type: string
valueSerializer:
type: string
acks: all
retries: 2147483647 # int max value
maxInFlightRequestsPerConnection: 1
maxPollBlockTime: 10s
security:
securityProtocol: sasl_ssl
sslProtocol: TLSv1.2
saslMechanism: PLAIN
saslJaas: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";"
```
### Basic Kafka Consumer
In your Dropwizard `Configuration` class, configure a `KafkaConsumerFactory`:
```java
@Valid
@NotNull
@JsonProperty("consumer")
private KafkaConsumerFactory kafkaConsumerFactory;
```
Then, in your `Application` class, you'll want to do something similar to the following:
```java
private final KafkaConsumerBundle kafkaConsumer = new KafkaConsumerBundle() {
@Override
public KafkaConsumerFactory getKafkaConsumerFactory(ExampleConfiguration configuration) {
return configuration.getKafkaConsumerFactory();
}
};
@Override
public void initialize(Bootstrap bootstrap) {
bootstrap.addBundle(kafkaConsumer);
}
@Override
public void run(ExampleConfiguration config, Environment environment) {
final PersonEventConsumer personEventConsumer = new PersonEventConsumer(kafkaConsumer.getConsumer());
personEventConsumer.startConsuming();
}
```
Configure your factory in your `config.yml` file:
```yaml
consumer:
type: basic
bootstrapServers:
- 127.0.0.1:9092
- 127.0.0.1:9093
- 127.0.0.1:9094
consumerGroupId: consumer1
name: consumerNameToBeUsedInMetrics
keyDeserializer:
type: string
valueDeserializer:
type: string
security:
securityProtocol: sasl_ssl
sslProtocol: TLSv1.2
saslMechanism: PLAIN
saslJaas: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";"
```
## Using an older version of the Kafka Client
This library *should* remain backwards compatible, such that you can override the version of the kafka client that this library includes. If this becomes no longer possible, we will need to create separate branches for differing major versions of the Kafka client.
For example, say you would like to use version `1.1.1` of the Kafka client. One option would be to explicitly define a dependency on version `1.1.1` of `kafka-clients` before you declare a dependency on `dropwizard-kafka`.
```xml
org.apache.kafka
kafka-clients
1.1.1
io.dropwizard
dropwizard-kafka
${dropwizard.version}
```
## Adding support for additional serializers and/or deserializers
In order to support additional serializers or deserializers, you'll need to create a new factory:
```java
@JsonTypeName("my-serializer")
public class MySerializerFactory extends SerializerFactory {
@NotNull
@JsonProperty
private String someConfig;
public String getSomeConfig() {
return someConfig;
}
public void setSomeConfig(final String someConfig) {
this.someConfig = someConfig;
}
@Override
public Class extends Serializer> getSerializerClass() {
return MySerializer.class;
}
}
```
Then you will need to add the following files to your `src/main/resources/META-INF/services` directory in order to support Jackson
polymorphic serialization:
File named `io.dropwizard.jackson.Discoverable`:
```
io.dropwizard.kafka.serializer.SerializerFactory
```
File named `io.dropwizard.kafka.serializer.SerializerFactory`:
```
package.name.for.your.MySerializerFactory
```