{"id":18458000,"url":"https://github.com/alokkusingh/kafka-experimental","last_synced_at":"2025-04-23T12:19:59.973Z","repository":{"id":45404213,"uuid":"289454813","full_name":"alokkusingh/kafka-experimental","owner":"alokkusingh","description":"Kaka Experimental with Apache Avro","archived":false,"fork":false,"pushed_at":"2024-04-14T19:36:48.000Z","size":140,"stargazers_count":1,"open_issues_count":15,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-04-21T19:05:25.476Z","etag":null,"topics":["avro","avro-kafka","avro-schema","docker-service","jenkinsfile","kafka","kafka-consumer","kafka-producer"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alokkusingh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-22T09:01:27.000Z","updated_at":"2024-06-27T08:31:06.391Z","dependencies_parsed_at":"2023-10-01T16:19:07.846Z","dependency_job_id":"699705a6-e11a-41bd-abdb-a43abebd7fd9","html_url":"https://github.com/alokkusingh/kafka-experimental","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alokkusingh%2Fkafka-experimental","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alokkusingh%2Fkafka-experimental/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alokkusingh%2Fkafka-experimental/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alokkusingh%2Fkafka-experimental/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alokkusingh","download_url":"https://codeload.github.com/alokkusingh/kafka-experimental/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250430784,"owners_count":21429350,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-kafka","avro-schema","docker-service","jenkinsfile","kafka","kafka-consumer","kafka-producer"],"created_at":"2024-11-06T08:16:22.238Z","updated_at":"2025-04-23T12:19:59.955Z","avatar_url":"https://github.com/alokkusingh.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/alokkusingh/kafka-experimental.svg?branch=master)](https://travis-ci.org/github/alokkusingh/kafka-experimental)\n[![GitHub issues](https://img.shields.io/github/issues/alokkusingh/kafka-experimental.svg)](https://github.com/alokkusingh/kafka-experimental/issues)\n[![GitHub issues closed](https://img.shields.io/github/issues-closed-raw/alokkusingh/kafka-experimental.svg?maxAge=2592000)](https://github.com/alokkusingh/kafka-experimental/issues?q=is%3Aissue+is%3Aclosed)\n\n# kafka-experimental\nKafka Experimental using Spring Kafka and Apache Avro.\n\nUsing `Apache Avro` for model generation for Producer and Consumer. \n\nUsing my own `Avro Serializer` utility library [https://github.com/alokkusingh/avro-serializer/packages/373458](https://github.com/alokkusingh/avro-serializer/packages/373458) to Serialize/Deserialize Avro model objects without using `Schema Registry`.\n\nTable of Contents\n=================\n\n   * [Current Deployment](#current-deployment)\n   * [TO-DO](#to-do)\n   * [Deployment](#deployment)\n\n## Current Deployment \n- 1 Zookeeper Instance\n- 1 Kafka Broker\n- 2 Producers - you may scale up using `docker-compose` scale command\n    - 1 instance of Rain Sensor \n    - 1 instance of Temperature Sensor - you may scale up using `docker-compose` scale command\n- 1 Partition\n- 2 Apps (`consumer`s from 2 diffrenet `consumer group`) - you may scale up using `docker-compose` scale command. This will be useless uless we have multiple `Partition`s.\n    - `app-one` - subscribed to Rain Sensor and Temperature Sensor Topics. Consumer group: `app-one`\n    - `app-two` - subscribed to Rain Sensor Topic. Consumer group: `app-two`\n- Apache Avro used as schema definition to serialize and deserialize the message. Avro schema definition is maintained locally, no schema registry was used.\n\n## TO-DO\n- Add `Docker Swarm` Deployment Steps\n- Add Multiple `Kafka Broker`\n- Add Multiple `Partition`\n- Add Multiple `Instances` of Each Apps\n- Add `Apache Avro` Schema Registry - currently Avro is used only for object mapping. Without schema registry schema versioning won't be possible\n\n## Few Notes\n- Producer creates topic on startup using \"NewTopic\" by providing \"Number of Partitions\" and \"Replication Factor\"\n\n## Deployment\n\n### Build the Maven Artifact\n````\nmvn clean package\n````\n\n### Deploy the docker services\n````\ndocker-compose -d -f docker-compose.yml up --build\n````\n\n### Stop the docker services\n````\ndocker-compose stop\n````\n\n### Scale up/down/stop one or more docker service\n````\ndocker-compose -f docker-compose.yml scale app-two=0\n````\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falokkusingh%2Fkafka-experimental","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falokkusingh%2Fkafka-experimental","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falokkusingh%2Fkafka-experimental/lists"}