{"id":22026585,"url":"https://github.com/harshit98/retail-updates-streamer","last_synced_at":"2026-05-09T14:44:44.864Z","repository":{"id":101158331,"uuid":"374369152","full_name":"harshit98/Retail-Updates-Streamer","owner":"harshit98","description":"EuroPython 2021 - High Performance Data Processing using Python, Elasticsearch and Kafka.","archived":false,"fork":false,"pushed_at":"2021-07-12T15:28:52.000Z","size":1310,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-20T01:16:14.655Z","etag":null,"topics":["elasticsearch","fastapi","kafka","pycon","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harshit98.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-06T13:36:13.000Z","updated_at":"2021-10-13T16:58:15.000Z","dependencies_parsed_at":"2023-03-13T15:28:33.338Z","dependency_job_id":null,"html_url":"https://github.com/harshit98/Retail-Updates-Streamer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshit98%2FRetail-Updates-Streamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshit98%2FRetail-Updates-Streamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshit98%2FRetail-Updates-Streamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshit98%2FRetail-Updates-Streamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harshit98","download_url":"https://codeload.github.com/harshit98/Retail-Updates-Streamer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245090875,"owners_count":20559298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","fastapi","kafka","pycon","python3"],"created_at":"2024-11-30T07:28:28.534Z","updated_at":"2026-05-09T14:44:44.817Z","avatar_url":"https://github.com/harshit98.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Retail Updates Streamer\n\nEuroPython 2021 - High Performance Data Processing using Python, Kafka and Elasticsearch.\n\n## Prerequisites\n\n- Python \u003e= 3.6\n- FastAPI\n- Kafka\n- Elasticsearch\n\n## Workflow Architecture\n\nOverall Flow:\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n    \u003cimg alt=\"workflow-architecture\" src=\"assets/producer-consumer.png\"\u003e \n\u003c/p\u003e\n\nCode Explanation:\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n    \u003cimg alt=\"workflow-kafka\" src=\"assets/kafka-workflow.png\"\u003e \n\u003c/p\u003e\n\n## Setup\n\n1. Make sure Python \u003e= 3.6 is installed on your local machine and create a virtual environment.\n\n   ```bash\n   python3 -m venv retail-updates-streamer\n   ```\n\n2. Activate your virtual environment.\n\n   ```bash\n   source retail-updates-streamer/bin/activate\n   ```\n\n3. Install application packages.\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. Add your producer-consumer related config in a `.env` file.\n\n5. For docker setup, add global environment constant in your `zshrc` or `bashrc` depending on shell.\n\n   ```bash\n   export DOCKER_KAFKA_HOST = $(ipconfig getifaddr en0)\n   ```\n\n6. Please note that I've commented out `elasticsearch` and `kibana` in `docker-compose.yml` as they need a lot of virtual environment memory. You can run these two stacks on local by downloading them as zip from official elastic site.\n\n## Sample CURLs\n\n### Producer\n\n1. Health Check\n\n   Request\n\n   ```bash\n   curl --location --request GET 'http://0.0.0.0:8001/ping'\n   ```\n\n   Response\n\n   ```json\n   {\n     \"ping\": \"pong!\"\n   }\n   ```\n\n2. Produce message to Kafka\n\n   Request\n\n   ```bash\n   curl --location --request POST 'http://127.0.0.1:8001/producer/retail-product' \\\n   --header 'Content-Type: application/json' \\\n   --data-raw '{\n      \"name\": \"Funny Farm House Ketchup\",\n      \"category\" : \"Dips and Ketchups\",\n      \"price\" : 15,\n      \"stock\" : 3,\n      \"product_id\": 10,\n      \"timestamp\": \"\"\n   }'\n   ```\n\n   Response\n\n   ```json\n   {\n     \"name\": \"Funny Farm House Ketchup\",\n     \"message_id\": \"Funny Farm House Ketchup_e5c0f1b0-ac1e-44c7-92c1-1f86728a36dc\",\n     \"topic\": \"retail-product\",\n     \"timestamp\": \"2021-07-12 15:17:20.314894\"\n   }\n   ```\n\n### Consumer\n\n1. Health Check\n\n   Request\n\n   ```bash\n   curl --location --request GET 'http://0.0.0.0:8000/ping'\n   ```\n\n   Response\n\n   ```json\n   {\n     \"ping\": \"pong!\"\n   }\n   ```\n\n2. Consume messages from Kafka\n\n   Request\n\n   ```bash\n   curl --location --request GET 'http://127.0.0.1:8000/consumer/retail-product'\n   ```\n\n   Response\n\n   ```json\n   {\n     \"topic\": \"retail-product\",\n     \"timestamp\": \"2021-07-12 20:47:21.067802\",\n     \"product_name\": \"Funny Farm House Ketchup\",\n     \"product_id\": 10,\n     \"success\": true\n   }\n   ```\n\n## Author\n\n👤 Harshit Prasad\n\nTwitter: [@HarshitPrasad8](https://twitter.com/HarshitPrasad8)\\\nGithub: [@harshit98](https://github.com/harshit98)\\\nWebsite: [harshitprasad.com](http://harshitprasad.com)\\\nLinkedIn: [harshit-prasad](https://www.linkedin.com/in/harshit-prasad/)\n\n## 🤝 Contributing\n\nContributions, issues and feature requests are welcome!\n\n## Show your support\n\nGive a ⭐️ if you think this project is awesome!\n\n## 📝 License\n\nCopyright © 2021 [Harshit Prasad](https://github.com/harshit98)\\\nThis project is [Apache License](https://github.com/harshit98/personalized-search/blob/master/LICENSE) licensed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharshit98%2Fretail-updates-streamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharshit98%2Fretail-updates-streamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharshit98%2Fretail-updates-streamer/lists"}