{"id":26345923,"url":"https://github.com/vedantwork/kafka-notifier","last_synced_at":"2025-06-29T01:37:48.051Z","repository":{"id":280574832,"uuid":"942452484","full_name":"vedantwork/kafka-notifier","owner":"vedantwork","description":"This project implements a Kafka-based notification system using Python with confluent-kafka. It includes a Kafka producer to publish messages and a Kafka consumer to process them.","archived":false,"fork":false,"pushed_at":"2025-03-11T11:30:54.000Z","size":18,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-11T12:31:09.811Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vedantwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-04T06:04:59.000Z","updated_at":"2025-03-11T11:30:57.000Z","dependencies_parsed_at":"2025-03-04T07:20:40.682Z","dependency_job_id":"0a8c51b2-7232-42d2-b514-ccecb8f9be8b","html_url":"https://github.com/vedantwork/kafka-notifier","commit_stats":null,"previous_names":["vedantwork/kafka-notifier"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vedantwork/kafka-notifier","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vedantwork%2Fkafka-notifier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vedantwork%2Fkafka-notifier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vedantwork%2Fkafka-notifier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vedantwork%2Fkafka-notifier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vedantwork","download_url":"https://codeload.github.com/vedantwork/kafka-notifier/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vedantwork%2Fkafka-notifier/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262520751,"owners_count":23323762,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-16T06:37:31.815Z","updated_at":"2025-06-29T01:37:48.027Z","avatar_url":"https://github.com/vedantwork.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📌 Kafka Notification System\n\nThis project implements a Kafka-based notification system using Python with `confluent-kafka`. It includes a Kafka producer to publish messages and a Kafka consumer to process them.\n\n---\n\n## 📜 Table of Contents\n1. 📝 [Overview](#-overview)\n2. 📂 [Project Structure](#-project-structure)\n3. 🚀 [Kafka Producer (Publishing Messages)](#-kafka-producer-publishing-messages)\n4. 🎯 [Kafka Consumer (Processing Messages)](#-kafka-consumer-processing-messages)\n5. 🛠 [Kafka Topic Management](#-kafka-topic-management)\n6. ▶️ [How to Run](#-how-to-run)\n7. ⚠️ [Error Handling](#-error-handling)\n8. 🔗 [Download \u0026 Setup Links](#-download--setup-links)\n\n---\n\n## 📝 Overview\nThis system consists of two main components:\n1. **Kafka Producer** - Publishes messages to a Kafka topic.\n2. **Kafka Consumer** - Consumes messages from the Kafka topic and processes them.\n\nEach component ensures that Kafka topics exist before producing or consuming messages. The messages are serialized as JSON before publishing.\n\n![Kafka Workflow](https://upload.wikimedia.org/wikipedia/commons/0/05/Apache_kafka.svg)\n\n---\n\n## 📂 Project Structure\n```\nproject/\n│── publish_to_kafka.py  # Defines code to publish the payload\n│── kafka_service.py     # Defines the Kafka Producer\n│── consumer.py          # Defines the Kafka Consumer\n│── .env                 # Environment variables\n│── config.py            # Configuration settings\n│── README.md            # Documentation\n```\n\n---\n\n## 🚀 Kafka Producer (Publishing Messages)\nThe `kafka_service.py` file defines a producer that publishes messages to a Kafka topic.\n\n### 🔹 Key Functions:\n1. **`publish_to_kafka(payload)`**: \n   - Ensures the topic exists using `ensure_kafka_topic()`.\n   - Serializes the payload as JSON.\n   - Publishes the message to Kafka.\n   - Calls `flush()` to ensure the message is sent immediately.\n\n2. **`ensure_kafka_topic(topic_name)`**: \n   - Checks if the Kafka topic exists.\n   - If not, creates the topic with default partitions and replication factor.\n\n3. **`acked(err, msg)`**: \n   - Callback function to check if a message was successfully published.\n\n#### 📌 Sample Code for Publishing Messages:\n```python\ntry:\n    publish_to_kafka(payload)\nexcept Exception as e:\n    logger.error(\"Kafka error while publishing payload: \", e)\n```\n\n---\n\n## 🎯 Kafka Consumer (Processing Messages)\nThe `consumer.py` file defines a consumer that reads messages from the Kafka topic.\n\n### 🔹 Key Functions:\n1. **`consume_messages()`**: \n   - Ensures the topic exists before consuming.\n   - Subscribes to the topic and continuously polls for new messages.\n   - Calls `process_message()` for each received message.\n   - If successful, commits the message offset.\n\n2. **`process_message(message)`**: \n   - Decodes and processes the message.\n   - Logs any errors that occur during processing.\n\n3. **`commit_offsets(message)`**: \n   - Commits the message offset after successful processing.\n\n#### 📌 Sample Code for Consuming Messages:\n```python\nif __name__ == \"__main__\":\n    asyncio.run(consume_messages())\n```\n\n---\n\n## 🛠 Kafka Topic Management\nBoth producer and consumer include the **`ensure_kafka_topic()`** function to:\n- ✅ Verify if the topic exists.\n- ➕ If not, create it dynamically.\n\n---\n\n## ▶️ How to Run\n### 🔧 1. Set Up Kafka\nEnsure that Kafka is installed and running.\n\n```sh\n# Start Zookeeper in a new terminal\nbin/zookeeper-server-start.sh config/zookeeper.properties\n\n# Start Kafka Broker in another terminal\nbin/kafka-server-start.sh config/server.properties\n```\n\n### 🔧 2. Configure Environment Variables\nCreate a `.env` file and set the following variables:\n```env\nKAFKA_BOOTSTRAP_SERVERS=localhost:9092\nTOPIC_NAME=notifications\n```\n\n### 🔧 3. Manage Kafka Topics\n```sh\n# Delete an existing topic\nbin/kafka-topics.sh --delete --topic notifications --bootstrap-server localhost:9092\n\n# Create a new topic\nbin/kafka-topics.sh --create --topic notifications --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1\n```\n\n### 🔧 4. Run the Kafka Producer\n```sh\npython kafka_service.py\n```\n\n### 🔧 5. Run the Kafka Consumer\n```sh\npython consumer.py\n```\n\n### 🔧 6. Consume Messages from Kafka Console\n```sh\nbin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic notifications --from-beginning\n```\n\n---\n\n## ⚠️ Error Handling\n1. **Producer Errors**:\n   - ❌ If message publishing fails, an error is logged.\n   - 🔄 The producer flushes messages to ensure they are sent.\n\n2. **Consumer Errors**:\n   - ❌ If message processing fails, it is logged and offset is not committed.\n   - ⚙️ Kafka errors are logged and handled appropriately.\n\n---\n\n## 🔗 Download \u0026 Setup Links\n1. **Download Apache Kafka**: [Kafka Official Download](https://kafka.apache.org/downloads)\n2. **Install Java (Required for Kafka)**: [Download Java](https://www.oracle.com/java/technologies/javase-jdk17-downloads.html)\n3. **Install Python Dependencies**:\n   ```sh\n   pip install confluent-kafka python-dotenv\n   ```\n4. **Kafka Documentation**: [Kafka Official Docs](https://kafka.apache.org/documentation/)\n\n---\n\n## 🎯 Conclusion\nThis system provides a robust way to publish and consume messages using Kafka. It ensures that topics exist before publishing and consuming messages and includes error handling mechanisms to handle failures gracefully.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvedantwork%2Fkafka-notifier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvedantwork%2Fkafka-notifier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvedantwork%2Fkafka-notifier/lists"}