{"id":24446225,"url":"https://github.com/cloaky233/datastreams","last_synced_at":"2025-10-11T01:17:55.166Z","repository":{"id":256945455,"uuid":"856897170","full_name":"CLoaKY233/DataStreams","owner":"CLoaKY233","description":"Real-time cat/dog image classifier using Kafka and CNN. Sender uploads images to Kafka, receiver processes with pre-trained model, returns predictions via Kafka. Demonstrates distributed, scalable image processing with instant feedback. Uses TensorFlow, OpenCV, and Kafka for efficient, asynchronous communication.","archived":false,"fork":false,"pushed_at":"2024-10-17T07:57:57.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-14T04:44:18.227Z","etag":null,"topics":["binary-classification","data-streaming","kafka","kafka-basic"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CLoaKY233.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-13T12:18:34.000Z","updated_at":"2024-11-02T14:53:05.000Z","dependencies_parsed_at":"2024-09-14T02:33:04.374Z","dependency_job_id":"37a31e2b-2c23-4534-9bf9-86c4e2ddf369","html_url":"https://github.com/CLoaKY233/DataStreams","commit_stats":null,"previous_names":["cloaky233/datastreams"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CLoaKY233/DataStreams","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLoaKY233%2FDataStreams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLoaKY233%2FDataStreams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLoaKY233%2FDataStreams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLoaKY233%2FDataStreams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CLoaKY233","download_url":"https://codeload.github.com/CLoaKY233/DataStreams/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLoaKY233%2FDataStreams/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279005927,"owners_count":26083984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary-classification","data-streaming","kafka","kafka-basic"],"created_at":"2025-01-20T23:42:20.775Z","updated_at":"2025-10-11T01:17:55.153Z","avatar_url":"https://github.com/CLoaKY233.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka-based Real-time Image Classifier\n\nThis project demonstrates a real-time cat/dog image classification system using Kafka and a pre-trained CNN model.\n\n## Prerequisites\n\n- Python 3.7+\n- Apache Kafka\n- Git\n\n## Setup\n\n1. Clone the repository:\n   ```\n   git clone https://github.com/yourusername/kafka-image-classifier.git\n   cd kafka-image-classifier\n   ```\n\n2. Set up Kafka:\n   Follow the instructions in [KafkaSetup.md](KafkaSetup.md) to install and configure Kafka.\n\n3. Create and activate a virtual environment:\n   ```\n   python -m venv venv\n   source venv/bin/activate  # On Windows, use `venv\\Scripts\\activate`\n   ```\n\n4. Install dependencies:\n   ```\n   pip install -r requirements.txt\n   ```\n\n5. Ensure `dogcatclassifier.h5` (the pre-trained model) is in the project root directory.\n\n## Components\n\nThe system consists of three main components:\n\n1. **Receiver-Predictor**: Processes images and makes predictions.\n2. **Sender**: Allows you to send images for classification.\n3. **Result Consumer**: (Optional) Displays classification results in real-time.\n\n## Usage\n\n### 1. Start the Receiver-Predictor\n\nIn a terminal, run:\n```\npython receiver_predictor.py\n```\n\nThis will continuously wait for images on the Kafka topic.\n\n### 2. Run the Sender\n\nIn another terminal, run:\n```\npython sender.py\n```\n\n- You'll be prompted to enter paths to image files.\n- Enter the full path to an image when prompted.\n- Type 'quit' to exit the sender program.\n\n### 3. (Optional) Run the Result Consumer\n\nTo see predictions in real-time, open a third terminal and run:\n```\npython result_consumer.py\n```\n\n## Example Workflow\n\n1. Ensure Kafka is running (refer to KafkaSetup.md).\n\n2. Open three terminal windows and navigate to the project directory in each.\n\n3. In Terminal 1, start the receiver-predictor:\n   ```\n   python receiver_predictor.py\n   ```\n\n4. In Terminal 2, start the sender:\n   ```\n   python sender.py\n   ```\n\n5. (Optional) In Terminal 3, start the result consumer:\n   ```\n   python result_consumer.py\n   ```\n\n6. In the sender terminal (Terminal 2), enter an image path when prompted:\n   ```\n   Enter the path to the image (or 'quit' to exit): /path/to/your/cat_image.jpg\n   ```\n\n7. View the results in Terminal 1 (receiver-predictor) or Terminal 3 (result consumer).\n\n8. Repeat step 6 with different images as desired.\n\n9. Type 'quit' in Terminal 2 (sender) to exit.\n\n## Troubleshooting\n\n- Verify Kafka is running correctly (check KafkaSetup.md).\n- Ensure `dogcatclassifier.h5` is present in the project root.\n- Verify all dependencies are installed (`pip list`).\n- Check that the image paths you enter exist and are accessible.\n\n## Customization\n\n- To use a different model, replace `dogcatclassifier.h5` and update `receiver_predictor.py` if needed.\n- Kafka topics and server addresses can be modified in the scripts if required.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloaky233%2Fdatastreams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloaky233%2Fdatastreams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloaky233%2Fdatastreams/lists"}