Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/thake/avro4k-kafka-serializer

A Kafka Serializer and SerDe implementation based on Avro4k serializers
https://github.com/thake/avro4k-kafka-serializer

avro kafka-serializer kotlin schema-registry serde

Last synced: 26 days ago
JSON representation

A Kafka Serializer and SerDe implementation based on Avro4k serializers

Host: GitHub
URL: https://github.com/thake/avro4k-kafka-serializer
Owner: thake
License: apache-2.0
Created: 2020-02-28T05:48:52.000Z (almost 5 years ago)
Default Branch: main
Last Pushed: 2022-08-29T05:35:38.000Z (over 2 years ago)
Last Synced: 2023-03-02T00:55:46.032Z (almost 2 years ago)
Topics: avro, kafka-serializer, kotlin, schema-registry, serde
Language: Kotlin
Homepage:
Size: 162 KB
Stars: 19
Watchers: 5
Forks: 7
Open Issues: 2
Metadata Files:
- Readme: Readme.md
- License: LICENSE

Awesome Lists containing this project

README

# Kafka avro4k serializer / deserializer
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.thake.avro4k/avro4k-kafka-serializer/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.github.thake.avro4k/avro4k-kafka-serializer)

This project implements a Kafka serializer / deserializer that integrates with the confluent schema registry and
leverages [avro4k](https://github.com/avro-kotlin/avro4k). It is based on confluent's [Kafka Serializer]((https://github.com/confluentinc/schema-registry/tree/master/avro-serializer)).
As such, this implementations can be used to in several projects (i.e. spring)

This SerDe supports retrying of failed calls to the schema registry (i.e. due to flaky network). Confluent's Serde does not implement this yet.
See https://github.com/confluentinc/schema-registry/issues/928.

## Confluent Versions

Version 0.10.x is compatible with Apache Kafka 2.5.x / Confluent 5.5.x and avro4k < 1.0

Version 0.11.x+ is compatible with Apache Kafka 2.6.x / Confluent 6.0.0 and avro4k < 1.0

Version > 0.13 is compatible with Apache Kafka 2.6.x/3.x (Confluent 6.x/7.x) and avro4k 1.x

## Example usage

You can find an example configuration of a Kafka Consumer, Kafka Producer and Kafka Streams application in the [ConfluentIT](./src/integrationTest/kotlin/com/github/thake/kafka/avro4k/serializer/ConfluentIT.kt) integration test.

### Spring Cloud Stream with Kafka

```javascript
spring:
application:
name:
kafka:
bootstrap-servers:
-

cloud:
stream:
default:
consumer:
useNativeDecoding: true
producer:
useNativeEncoding: true
kafka:
streams:
binder:
brokers:
configuration:
schema.registry.url:
schema.registry.retry.attemps: 3
schema.registry.retry.jitter.base: 10
schema.registry.retry.jitter.max: 5000
record.packages:
default.key.serde: com.github.thake.kafka.avro4k.serializer.Avro4kSerde
default.value.serde: com.github.thake.kafka.avro4k.serializer.Avro4kSerde
default:
producer:
keySerde: com.github.thake.kafka.avro4k.serializer.Avro4kSerde
valueSerde: com.github.thake.kafka.avro4k.serializer.Avro4kSerde
consumer:
keySerde: com.github.thake.kafka.avro4k.serializer.Avro4kSerde
valueSerde: com.github.thake.kafka.avro4k.serializer.Avro4kSerde
...
```

### Maven

```xml

com.github.thake.avro4k
avro4k-kafka-serializer
VERSION

```

### Configuration options

- `schema.registry.url`
Comma-separated list of URLs for schema registry instances that can
be used to register or look up schemas. If you wish to get a
connection to a mocked schema registry for testing, you can specify
a scope using the `mock://` pseudo-protocol. For example,
`mock://my-scope-name` corresponds to
`MockSchemaRegistry.getClientForScope("my-scope-name")`.

- Type: list
- Importance: high

- `record.packages`
The packages in which record types annotated with `@AvroName`,
`@AvroAlias` and `@AvroNamespace` can be found. Packages are separated
by a comma `,`. Only needed for deserialization.

- Type: string
- Default: null
- Importance: high

- `schema.registry.ssl.key.password`
The password of the private key in the key store file or the PEM key
specified in `ssl.keystore.key`. This is required for clients only
if two-way authentication is configured.