{"id":34851447,"url":"https://github.com/shiningflash/kafka-python-messaging-engine","last_synced_at":"2025-12-25T19:20:25.584Z","repository":{"id":295232754,"uuid":"985966544","full_name":"shiningflash/kafka-python-messaging-engine","owner":"shiningflash","description":"A Python-based Kafka messaging engine with Avro serialization, Schema Registry integration, and Redpanda support for real-time data streaming.","archived":false,"fork":false,"pushed_at":"2025-05-25T23:20:50.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-25T23:27:53.154Z","etag":null,"topics":["avro","confluent-kafka","data-pipeline","data-streaming","event-driven","event-driven-architecture","kafka","kafka-python","python","real-time-data","real-time-messaging","redpanda","schema-registry"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shiningflash.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-18T22:13:53.000Z","updated_at":"2025-05-25T23:20:53.000Z","dependencies_parsed_at":"2025-05-25T23:27:54.727Z","dependency_job_id":null,"html_url":"https://github.com/shiningflash/kafka-python-messaging-engine","commit_stats":null,"previous_names":["shiningflash/python-kafka","shiningflash/kafka-python-messaging-engine"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/shiningflash/kafka-python-messaging-engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiningflash%2Fkafka-python-messaging-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiningflash%2Fkafka-python-messaging-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiningflash%2Fkafka-python-messaging-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiningflash%2Fkafka-python-messaging-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shiningflash","download_url":"https://codeload.github.com/shiningflash/kafka-python-messaging-engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiningflash%2Fkafka-python-messaging-engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28035466,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-25T02:00:05.988Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","confluent-kafka","data-pipeline","data-streaming","event-driven","event-driven-architecture","kafka","kafka-python","python","real-time-data","real-time-messaging","redpanda","schema-registry"],"created_at":"2025-12-25T19:19:59.678Z","updated_at":"2025-12-25T19:20:25.579Z","avatar_url":"https://github.com/shiningflash.png","language":"Python","readme":"# kafka-python-messaging-engine\n\nA modular, schema-aware messaging engine built in **Python** with **Kafka** and **Avro**, using **Redpanda** as the Kafka backend.\n\nThis project includes:\n- ✅ Avro serialization with Schema Registry\n- ✅ Producer and consumer interfaces\n- ✅ Batch support, delivery reports, tombstone messages\n- ✅ Fully configurable via `.env` (see `.env.example`)\n- ✅ Managed using [Poetry](https://python-poetry.org)\n\n---\n\n## Tech Stack\n\n- **Python 3.11+**\n- **Poetry** for dependency management\n- **Redpanda** (Kafka-compatible streaming platform)\n- **Confluent Kafka Python Client**\n- **Avro + Schema Registry**\n\n---\n\n## Getting Started (with Docker + Redpanda)\n\nRedpanda is used to run a Kafka-compatible stack locally with minimal effort.\n\n### 1. Clone the repo\n\n```bash\ngit clone https://github.com/shiningflash/kafka-python-messaging-engine.git\ncd kafka-python-messaging-engine\n````\n\n### 2. Start Redpanda and Schema Registry (via Docker)\n\n```bash\ndocker compose up -d\n```\n\n\u003e This starts:\n\u003e\n\u003e * Redpanda Kafka Broker at `localhost:19092`\n\u003e * Redpanda Schema Registry at `http://localhost:18081`\n\u003e * Redpanda Console UI at `http://localhost:8080`\n\n### 3. Check Services\n\n* **Kafka Broker**: `localhost:19092`\n* **Schema Registry**: `http://localhost:18081`\n* **Redpanda Console**: [http://localhost:8080](http://localhost:8080)\n\n---\n\n## Environment Configuration\n\nCopy and modify your environment variables:\n\n```bash\ncp .env.example .env\n```\n\nEdit `.env` to suit your local or dev settings.\n\n---\n\n## Schema Location\n\nUser Avro schema lives in:\n\n```\nsrc/schemas/user.avsc\n```\n\nUpdate the schema and `.env` accordingly if you modify fields.\n\n---\n\n## Python Setup (via Poetry)\n\n### Install Dependencies\n\n```bash\npoetry install\n```\n\n### Activate the Virtual Env\n\n```bash\npoetry env activate\n```\n\n---\n\n## 🚀 Run Producer App\n\n```bash\npython -m src.producer_app\n```\n\nThis will:\n\n* Connect to Redpanda\n* Prompt you to input user data\n* Send serialized Avro messages to Kafka\n\n---\n\n## 🎧 Run Consumer App\n\n```bash\npython -m src.consumer_app\n```\n\nThis will:\n\n* Connect to the same topic\n* Consume and deserialize Avro messages\n* Support batch reads (configurable in `.env`)\n\n---\n\n## Learn More\n\nTo get familiar with Kafka, its role in streaming pipelines, and design concepts, check the beginner-friendly guide:\n\n👉 [intro\\_to\\_kafka.md](intro_to_kafka.md)\n\n---\n\n## Contributing\n\nPull requests welcome! Please:\n\n* Keep code clean and modular\n* Write meaningful commit messages\n* Follow PEP8 and type hinting conventions\n\n---\n\n## 📝 License\n\nMIT License — feel free to use, fork, and contribute.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshiningflash%2Fkafka-python-messaging-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshiningflash%2Fkafka-python-messaging-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshiningflash%2Fkafka-python-messaging-engine/lists"}