https://github.com/anupkumarsharma/kafkamirrormaker.extensions
Extensions for Kafka MirrorMaker - Change Schema ID of a message
https://github.com/anupkumarsharma/kafkamirrormaker.extensions
apache-kafka kafka kafkatopic mirrormaker schemaregistry
Last synced: 3 months ago
JSON representation
Extensions for Kafka MirrorMaker - Change Schema ID of a message
- Host: GitHub
- URL: https://github.com/anupkumarsharma/kafkamirrormaker.extensions
- Owner: anupkumarsharma
- Created: 2018-10-13T05:40:24.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2018-10-13T06:08:17.000Z (over 6 years ago)
- Last Synced: 2025-01-12T11:36:15.825Z (5 months ago)
- Topics: apache-kafka, kafka, kafkatopic, mirrormaker, schemaregistry
- Language: Java
- Homepage:
- Size: 6.84 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
KafkaMirrorMaker.Extensions
======================================
This allows to create multiple extensions to manipulate the payloads as they are copied from source Kafka cluster to target cluster.
Kafka Mirror is preferred for end to end kafka replication without heavy serdes requirement. This extension allows to alter the schema id
based on the schema registry server of target destination.
* Logic - It takes the message from Kafka uses the wire format to read the bytes. It then reads the schema id from schema registry server for the target topic.
The target schema id is then stuffed into message and sent to destination.
The argument need to be passed is like - registry,http://schema-registry.com/;topic1,topic1. This specifies the target schema registry server url and topic1:topic1 = sourcetopicname:destinationtopicname
* Update the classpath
`export CLASSPATH=/etc/kafka-mirror-maker/KafkaMirrorMaker.Extensions-1.0-SNAPSHOT-jar-with-dependencies.jar`
* Start MirrorMaker. Specify the Handler class in "--message.handler" and any arguments in "--message.handler.args". For example:
`/usr/hdp/current/kafka-broker/bin/kafka-mirror-maker.sh --whitelist topic1 --abort.on.send.failure true --new.consumer --producer.config /tmp/mirror-maker/producer.config --consumer.config /tmp/mirror-maker/consumer.config --message.handler com.extensions.SchemaRegistryIdSwitcher --message.handler.args "registry,http://schema-registry.com/;topic1,topic1`
* Recommended docker image which can work seamlessly - https://hub.docker.com/r/ambuds/mirror-maker/
* Thanks for other authors who has similar extensions - https://github.com/opencore/mirrormaker_topic_rename
* To Build
`mvn clean; mvn build; mvn package`