Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/alokkusingh/kafka-experimental
Kaka Experimental with Apache Avro
https://github.com/alokkusingh/kafka-experimental
avro avro-kafka avro-schema docker-service jenkinsfile kafka kafka-consumer kafka-producer
Last synced: 24 days ago
JSON representation
Kaka Experimental with Apache Avro
- Host: GitHub
- URL: https://github.com/alokkusingh/kafka-experimental
- Owner: alokkusingh
- Created: 2020-08-22T09:01:27.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2024-04-14T19:36:48.000Z (9 months ago)
- Last Synced: 2024-04-21T19:05:25.476Z (9 months ago)
- Topics: avro, avro-kafka, avro-schema, docker-service, jenkinsfile, kafka, kafka-consumer, kafka-producer
- Language: Java
- Homepage:
- Size: 137 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
[](https://travis-ci.org/github/alokkusingh/kafka-experimental)
[](https://github.com/alokkusingh/kafka-experimental/issues)
[](https://github.com/alokkusingh/kafka-experimental/issues?q=is%3Aissue+is%3Aclosed)
# kafka-experimental
Kafka Experimental using Spring Kafka and Apache Avro.
Using `Apache Avro` for model generation for Producer and Consumer.
Using my own `Avro Serializer` utility library [https://github.com/alokkusingh/avro-serializer/packages/373458](https://github.com/alokkusingh/avro-serializer/packages/373458) to Serialize/Deserialize Avro model objects without using `Schema Registry`.
Table of Contents
=================
* [Current Deployment](#current-deployment)
* [TO-DO](#to-do)
* [Deployment](#deployment)
## Current Deployment
- 1 Zookeeper Instance
- 1 Kafka Broker
- 2 Producers - you may scale up using `docker-compose` scale command
- 1 instance of Rain Sensor
- 1 instance of Temperature Sensor - you may scale up using `docker-compose` scale command
- 1 Partition
- 2 Apps (`consumer`s from 2 diffrenet `consumer group`) - you may scale up using `docker-compose` scale command. This will be useless uless we have multiple `Partition`s.
- `app-one` - subscribed to Rain Sensor and Temperature Sensor Topics. Consumer group: `app-one`
- `app-two` - subscribed to Rain Sensor Topic. Consumer group: `app-two`
- Apache Avro used as schema definition to serialize and deserialize the message. Avro schema definition is maintained locally, no schema registry was used.
## TO-DO
- Add `Docker Swarm` Deployment Steps
- Add Multiple `Kafka Broker`
- Add Multiple `Partition`
- Add Multiple `Instances` of Each Apps
- Add `Apache Avro` Schema Registry - currently Avro is used only for object mapping. Without schema registry schema versioning won't be possible
## Few Notes
- Producer creates topic on startup using "NewTopic" by providing "Number of Partitions" and "Replication Factor"
## Deployment
### Build the Maven Artifact
````
mvn clean package
````
### Deploy the docker services
````
docker-compose -d -f docker-compose.yml up --build
````
### Stop the docker services
````
docker-compose stop
````
### Scale up/down/stop one or more docker service
````
docker-compose -f docker-compose.yml scale app-two=0
````