{"id":19435482,"url":"https://github.com/anshumanpattnaik/python-kafka-messaging","last_synced_at":"2026-02-09T21:36:39.974Z","repository":{"id":41266000,"uuid":"317341239","full_name":"anshumanpattnaik/python-kafka-messaging","owner":"anshumanpattnaik","description":"Python Kafka Messaging","archived":false,"fork":false,"pushed_at":"2020-12-03T19:44:28.000Z","size":422,"stargazers_count":9,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-25T14:49:02.987Z","etag":null,"topics":["flask-restful","jwt-authentication","kafka","kafka-consumer","kafka-producer","kafka-producer-consumer","microservices","mongodb","python","python-3","twilio-python","twilio-sms"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anshumanpattnaik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-30T20:43:48.000Z","updated_at":"2023-07-13T17:43:37.000Z","dependencies_parsed_at":"2022-09-20T23:24:00.600Z","dependency_job_id":null,"html_url":"https://github.com/anshumanpattnaik/python-kafka-messaging","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/anshumanpattnaik/python-kafka-messaging","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fpython-kafka-messaging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fpython-kafka-messaging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fpython-kafka-messaging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fpython-kafka-messaging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anshumanpattnaik","download_url":"https://codeload.github.com/anshumanpattnaik/python-kafka-messaging/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fpython-kafka-messaging/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29281976,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-09T19:05:41.198Z","status":"ssl_error","status_checked_at":"2026-02-09T19:05:37.449Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flask-restful","jwt-authentication","kafka","kafka-consumer","kafka-producer","kafka-producer-consumer","microservices","mongodb","python","python-3","twilio-python","twilio-sms"],"created_at":"2024-11-10T15:06:38.196Z","updated_at":"2026-02-09T21:36:39.958Z","avatar_url":"https://github.com/anshumanpattnaik.png","language":"Python","readme":"## Python Kafka Messaging\nThe idea behind this project is to demonstrate the chat messaging service using [Apache Kafka](https://kafka.apache.org/) and this proof of concept follows the architecture of WhatsApp messaging, (i.e. one to one messaging and group messaging). Apache Kafka messaging framework works based on producer and consumer architecture and to implement these core concepts it provides us to create Kafka topics, send messages using KafkaProducer and receive messages using KafkaConsumer based on the subscribed topics, so the Kafka framework is quite helpful while building distributed streaming-based system.\n\n\u003cimg src=\"screenshots/WhatsApp_Kafka_Python.png\"/\u003e\n\n## Technical Overview\nThe proof of concept written using python and it follows the core \u0026 common architecture of WhatsApp Messenger and to replicate the similar messaging concepts it uses [kafka-python](https://pypi.org/project/kafka-python/) library to send/receive messages using KafkaProducer/KafkaConsumer. It also implements [Twilio](https://www.twilio.com/) messaging framework for user authentication by verifying their registered phone no using the sent OTP and to cache all the user data and chat messages it uses mongodb database for the storage.\n\nIn this messaging system, there are three different types of Microservices designed using [python flask framework](https://flask.palletsprojects.com/en/1.1.x/).\n\n1. Users\n2. Kafka Messaging\n3. Kafka Group Messaging\n\nTo run the complete project, first of all, you need to install Kafka in your local system, I have installed [kafka_2.12-2.6.0](https://kafka.apache.org/downloads) latest version in my ubuntu-18.04 machine.\n\n### Swagger API Documentation\nThe Swagger API docs can be accessible via [http://127.0.0.1:5000/api/docs](http://127.0.0.1:5000/api/docs) and to test the API endpoints you need to authorize yourself using your jwt access token.\n\n\u003cimg src=\"screenshots/swagger.png\"/\u003e\n\n## Installation\n``````````````````````````````````````````````````````````````````\ngit clone https://github.com/anshumanpattnaik/python-kafka-messaging\ncd python-kafka-messaging\npip install -r requirements.txt\n``````````````````````````````````````````````````````````````````\n\n## Twilio Configurations\nTo run the Twilio messaging framework, first of all, you need to sign up an account in [Twilio developer console](https://www.twilio.com/console) and once you have the credentials then follow the below steps to configure the environment variables for secret keys.\n\n```````````````````````````````````````````````````````````````````\n$ nano ~/.bash_profile\n\n// Add the below variables to your profile\nexport ACCOUNT_SID='TWILIO_ACCOUNT_SID'\nexport AUTH_TOKEN='TWILIO_AUTH_TOKEN'\nexport TWILIO_PHONE_NO='TWILIO_SENDER_PHONE_NO'\n\n$ source ~/.bash_profile\n```````````````````````````````````````````````````````````````````\n\n## Start Kafka Server\nWhen you have Kafka set up in your local machine before starting the project first you need to start Kafka server in your localhost.\n\n### Execute the below command in terminal-1\n``````````````````````````````````````````````````````````````\n$ cd kafka_2.12-2.6.0\n$ bin/kafka-server-start.sh config/server.properties\n``````````````````````````````````````````````````````````````\n\n### Start the python kafka project in terminal-2\n``````````````````````````````````````````````````````````````\n$ cd python-kafka-messaging\n$ source venv/bin/activate\n$ python3 run.py\n\nOpen http://127.0.0.1:5000 to view in the browser.\n``````````````````````````````````````````````````````````````\n\n### Start the python kafka consumer script in terminal-3\nOnce your Kafka server up and running then go inside the project and start the [kafka-consumer.py](https://github.com/anshumanpattnaik/python-kafka-messaging/blob/main/kafka_consumer.py) script to receive an incoming message from the producer.\n\n#### Important\nIn case of one to one messaging you can comment out `subscribe` code at line no [21](https://github.com/anshumanpattnaik/python-kafka-messaging/blob/b651fbbdd34650408a6ae639b7ce1a536e0d121e/kafka_consumer.py#L21) because it's only required in a group conversation.\n\n````````````````````````````````````````\n# consumer.subscribe(group_topic) // in case of one to one conversation\n````````````````````````````````````````\n\n`````````````````````````````````````````````````````````````````\n$ cd python-kafka-messaging\n$ python3 kafka_consumer.py -p YOUR_PARTICIPANT_PHONE_NO\n`````````````````````````````````````````````````````````````````\n\n#### Note\nIt's quite important to understand basic Kafka concepts (i.e Topics/Producer/Consumer) so I'll highly recommend [kafka official documentation](https://kafka.apache.org/intro) to understand the Kafka architecture in depth.\n\n### License\nThis project is licensed under the [MIT License](LICENSE)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanshumanpattnaik%2Fpython-kafka-messaging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanshumanpattnaik%2Fpython-kafka-messaging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanshumanpattnaik%2Fpython-kafka-messaging/lists"}