{"id":16111633,"url":"https://github.com/mayankgupta-dev08/the-kafka-project","last_synced_at":"2025-10-12T22:12:47.260Z","repository":{"id":241953121,"uuid":"808256871","full_name":"MayankGupta-dev08/the-kafka-project","owner":"MayankGupta-dev08","description":"Kafka POC","archived":false,"fork":false,"pushed_at":"2024-08-09T14:49:16.000Z","size":4706,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-12T12:15:02.276Z","etag":null,"topics":["java","kafka","kafka-consumer","kafka-producer","spring-boot","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MayankGupta-dev08.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-30T17:38:23.000Z","updated_at":"2024-08-09T14:49:19.000Z","dependencies_parsed_at":"2024-06-08T13:26:50.542Z","dependency_job_id":"f012a7a5-446a-4383-95bd-d8d96e6cbcab","html_url":"https://github.com/MayankGupta-dev08/the-kafka-project","commit_stats":null,"previous_names":["mayankgupta-dev08/the-kafka-project"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MayankGupta-dev08%2Fthe-kafka-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MayankGupta-dev08%2Fthe-kafka-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MayankGupta-dev08%2Fthe-kafka-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MayankGupta-dev08%2Fthe-kafka-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MayankGupta-dev08","download_url":"https://codeload.github.com/MayankGupta-dev08/the-kafka-project/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247444240,"owners_count":20939806,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java","kafka","kafka-consumer","kafka-producer","spring-boot","zookeeper"],"created_at":"2024-10-09T19:44:00.138Z","updated_at":"2025-10-12T22:12:47.172Z","avatar_url":"https://github.com/MayankGupta-dev08.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KAFKA\n\n---\n\n## Pre-requisites\n\n1. install apache kafka (open source) or confluent kafka (commercial distribution)\n    - Important Dirs:\n        - \"E:\\kafka\\bin\\windows\"\n        - \"E:\\kafka\\config\"\n2. install kafka offset explorer (visualiser)\n    - connect by creating a new cluster\n        - give cluster name\n        - enable zookeeper access\n        - provide zookeeper host and port\n3. install docker\n\n---\n\n## KAFKA CLI - 101\n\n### Kafka Producer-Consumer Flow Steps:\n\n1. **Start Zookeeper**\n    - The default port of zookeeper is **2181**\n\n```shell\n.\\bin\\windows\\zookeeper-server-start.bat .\\config\\zookeeper.properties\n```\n\n2. **Start Kafka Server**\n    - The default port of kafka server/broker is **9092**\n\n```shell\n.\\bin\\windows\\kafka-server-start.bat .\\config\\server.properties\n```\n\n3. **Create a Topic**\n    - The default port of kafka server/broker is **9092**\n\n```shell\n.\\bin\\windows\\kafka-topics.bat --bootstrap-server localhost:9092 --create --topic my-1st-topic --partitions 3 --replication-factor 1\n.\\bin\\windows\\kafka-topics.bat --bootstrap-server localhost:9092 --create --topic my-2nd-topic --partitions 3 --replication-factor 1\n```\n\n```shell\n.\\bin\\windows\\kafka-topics.bat --bootstrap-server localhost:9092 --list\n.\\bin\\windows\\kafka-topics.bat --bootstrap-server localhost:9092 --describe --topic \u003ctopicName\u003e\n```\n\n- **Partition count**\n- **Replication factor**\n\n```mermaid\ngraph TD\n;\n    A[Producer] --\u003e|Sends 10 messages| B[Broker / Kafka Server]\n    B --\u003e C[Consumer]\n    subgraph Zookeeper\n        D[Start Zookeeper]\n    end\n    subgraph Kafka Server\n        E[Start Kafka Server]\n    end\n    subgraph Topic\n        F[Create a Topic]\n        F --\u003e G[Partition count]\n        F --\u003e H[Replication factor]\n    end\n    B --\u003e I[Partition 1]\n    B --\u003e J[Partition 2]\n    B --\u003e K[Partition 3]\n    I --\u003e|Message 1| L[Message]\n    I --\u003e|Message 2| M[Message]\n    I --\u003e|Message 3| N[Message]\n    I --\u003e|Message 4| O[Message]\n    I --\u003e|Message 5| P[Message]\n    J --\u003e|Message 1| Q[Message]\n    J --\u003e|Message 2| R[Message]\n    K --\u003e|Message 1| S[Message]\n    K --\u003e|Message 2| T[Message]\n    K --\u003e|Message 3| U[Message]\n```\n\n4. To start the producer using CLI\n\n```shell\n.\\bin\\windows\\kafka-console-producer.bat --broker-list localhost:9092 --topic my-1st-topic\n```\n\n```shell\nGet-Content E:\\Coding\\the-kafka-project\\src\\main\\resources\\csv\\customers-10000.csv | .\\bin\\windows\\kafka-console-producer.bat --broker-list localhost:9092 --topic my-1st-topic\n```\n\n5. To start the consumer using CLI\n\n```shell\n.\\bin\\windows\\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my-1st-topic --from-beginning\n```\n\n---\n\n## KAFKA using Docker\n\n```shell\ndocker compose -f .\\docker\\docker-compose.yml up -d\n```\n\n- For using the bash inside the kafka container\n```shell\ndocker exec -it \u003ckafka_container_name\u003e bash\n```\n\n- navigate to the `bin` dir inside the kafka container --\u003e /opt/kafka_version/bin/\n  \n- To create a kafka topic\n```shell\nkafka-topics.sh --zookeeper zookeeper:2181 --create --topic my-1st-topic --partitions 3 --replication-factor 1\n```\n\n- To start a kafka producer for a specific topic\n```shell\nkafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-1st-topic\n```\n\n- To start a kafka consumer for a specific topic\n```shell\nkafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-1st-topic --from-beginning\n```\n\n---\n\n## Kafka + Spring Boot Application (Producer-Consumer Example)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmayankgupta-dev08%2Fthe-kafka-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmayankgupta-dev08%2Fthe-kafka-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmayankgupta-dev08%2Fthe-kafka-project/lists"}