https://github.com/daniel-shuy/kafka-protobuf-serde
Serializer/Deserializer for Kafka to serialize/deserialize Protocol Buffers messages
https://github.com/daniel-shuy/kafka-protobuf-serde
deserialization kafka protobuf protocol-buffers serde serialization
Last synced: 5 months ago
JSON representation
Serializer/Deserializer for Kafka to serialize/deserialize Protocol Buffers messages
- Host: GitHub
- URL: https://github.com/daniel-shuy/kafka-protobuf-serde
- Owner: daniel-shuy
- License: mit
- Created: 2018-05-07T08:19:24.000Z (about 8 years ago)
- Default Branch: develop
- Last Pushed: 2025-12-13T08:59:16.000Z (6 months ago)
- Last Synced: 2025-12-14T23:18:32.182Z (6 months ago)
- Topics: deserialization, kafka, protobuf, protocol-buffers, serde, serialization
- Language: Java
- Homepage:
- Size: 633 KB
- Stars: 65
- Watchers: 3
- Forks: 15
- Open Issues: 29
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# kafka-protobuf-serde
| Branch | Travis CI | CodeFactor | Codacy | Better Code Hub | Coverall |
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Master | [](https://travis-ci.org/daniel-shuy/kafka-protobuf-serde) | [](https://www.codefactor.io/repository/github/daniel-shuy/kafka-protobuf-serde/overview/master) | [](https://www.codacy.com/app/daniel-shuy/kafka-protobuf-serde?utm_source=github.com&utm_medium=referral&utm_content=daniel-shuy/kafka-protobuf-serde&utm_campaign=Badge_Grade) | [](https://bettercodehub.com/) | [](https://coveralls.io/github/daniel-shuy/kafka-protobuf-serde?branch=master) |
| Develop | [](https://travis-ci.org/daniel-shuy/kafka-protobuf-serde) | [](https://www.codefactor.io/repository/github/daniel-shuy/kafka-protobuf-serde/overview/develop) | [](https://www.codacy.com/app/daniel-shuy/kafka-protobuf-serde?utm_source=github.com&utm_medium=referral&utm_content=daniel-shuy/kafka-protobuf-serde&utm_campaign=Badge_Grade) | [](https://bettercodehub.com/) | [](https://coveralls.io/github/daniel-shuy/kafka-protobuf-serde?branch=develop) |
Serializer/Deserializer for Kafka to serialize/deserialize Protocol Buffers
messages
## Requirements
| Dependency | Version |
| ---------- | ------- |
| Kafka | 2.X.X |
| Protobuf | 3.X.X |
| Java | 8+ |
## Usage
Add the following to your Maven dependency list:
```xml
com.github.daniel-shuy
kafka-protobuf-serde
${kafka-protobuf-serde.version}
```
Override the `protobuf-java` dependency with the version of Protobuf you wish to use:
```xml
com.google.protobuf
protobuf-java
${protobuf.version}
```
Override the `kafka-clients` dependency version with the version of Kafka you wish to use:
```xml
org.apache.kafka
kafka-clients
${kafka.version}
```
### Kafka Producer
```java
Properties props = new Properties();
// props.put(..., ...);
Producer producer = new KafkaProducer<>(props,
new StringSerializer(),
new KafkaProtobufSerializer<>());
producer.send(new ProducerRecord<>("topic", new MyValue()));
```
### Kafka Consumer
```java
Properties props = new Properties();
// props.put(..., ...);
Consumer consumer = new KafkaConsumer<>(props,
new StringDeserializer(),
new KafkaProtobufDeserializer<>(MyValue.parser()));
consumer.subscribe(Collections.singleton("topic"));
ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
records.forEach(record -> {
String key = record.key();
MyValue value = record.value();
// ...
});
```
### Kafka Streams
```java
Serde stringSerde = Serdes.String();
Serde myValueSerde = new KafkaProtobufSerde<>(MyValue.parser());
Properties config = new Properties();
// config.put(..., ...);
StreamsBuilder builder = new StreamsBuilder();
KStream myValues = builder.stream("input_topic", Consumed.with(stringSerde, myValueSerde));
KStream filteredMyValues = myValues.filter((key, value) -> {
// ...
});
filteredMyValues.to("output_topic", Produced.with(stringSerde, myValueSerde));
Topology topology = builder.build();
KafkaStreams streams = new KafkaStreams(topology, config);
streams.setUncaughtExceptionHandler((thread, throwable) -> {
// ...
});
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
streams.start();
```
### Spring for Apache Kafka (spring-kafka)
#### Kafka Producer
```java
@Configuration
public class KafkaConfig {
@Bean
public ProducerFactory producerFactory() {
Map props = new HashMap<>();
// props.put(..., ...);
return new DefaultKafkaProducerFactory<>(producerProps,
new StringSerializer(),
new KafkaProtobufSerializer<>());
}
@Bean
public KafkaTemplate kafkaTemplate() {
return new KafkaTemplate(producerFactory());
}
}
```
#### Kafka Consumer
```java
@Configuration
@EnableKafka
public class KafkaConfig {
@Bean
KafkaListenerContainerFactory>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
@Bean
public ConsumerFactory consumerFactory() {
Map props = new HashMap<>();
// props.put(..., ...);
return new DefaultKafkaConsumerFactory<>(props,
new StringDeserializer(),
new KafkaProtobufDeserializer<>(MyValue.parser()));
}
}
public class Listener {
@KafkaListener(id = "foo", topics = "annotated1")
public void listen1(String foo) {
// ...
}
}
```