https://github.com/cloaky233/datastreams
Real-time cat/dog image classifier using Kafka and CNN. Sender uploads images to Kafka, receiver processes with pre-trained model, returns predictions via Kafka. Demonstrates distributed, scalable image processing with instant feedback. Uses TensorFlow, OpenCV, and Kafka for efficient, asynchronous communication.
https://github.com/cloaky233/datastreams
binary-classification data-streaming kafka kafka-basic
Last synced: 8 months ago
JSON representation
Real-time cat/dog image classifier using Kafka and CNN. Sender uploads images to Kafka, receiver processes with pre-trained model, returns predictions via Kafka. Demonstrates distributed, scalable image processing with instant feedback. Uses TensorFlow, OpenCV, and Kafka for efficient, asynchronous communication.
- Host: GitHub
- URL: https://github.com/cloaky233/datastreams
- Owner: CLoaKY233
- License: mit
- Created: 2024-09-13T12:18:34.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-10-17T07:57:57.000Z (over 1 year ago)
- Last Synced: 2025-03-14T04:44:18.227Z (over 1 year ago)
- Topics: binary-classification, data-streaming, kafka, kafka-basic
- Language: Python
- Homepage:
- Size: 7.81 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: Readme.md
- License: LICENSE
Awesome Lists containing this project
README
# Kafka-based Real-time Image Classifier
This project demonstrates a real-time cat/dog image classification system using Kafka and a pre-trained CNN model.
## Prerequisites
- Python 3.7+
- Apache Kafka
- Git
## Setup
1. Clone the repository:
```
git clone https://github.com/yourusername/kafka-image-classifier.git
cd kafka-image-classifier
```
2. Set up Kafka:
Follow the instructions in [KafkaSetup.md](KafkaSetup.md) to install and configure Kafka.
3. Create and activate a virtual environment:
```
python -m venv venv
source venv/bin/activate # On Windows, use `venv\Scripts\activate`
```
4. Install dependencies:
```
pip install -r requirements.txt
```
5. Ensure `dogcatclassifier.h5` (the pre-trained model) is in the project root directory.
## Components
The system consists of three main components:
1. **Receiver-Predictor**: Processes images and makes predictions.
2. **Sender**: Allows you to send images for classification.
3. **Result Consumer**: (Optional) Displays classification results in real-time.
## Usage
### 1. Start the Receiver-Predictor
In a terminal, run:
```
python receiver_predictor.py
```
This will continuously wait for images on the Kafka topic.
### 2. Run the Sender
In another terminal, run:
```
python sender.py
```
- You'll be prompted to enter paths to image files.
- Enter the full path to an image when prompted.
- Type 'quit' to exit the sender program.
### 3. (Optional) Run the Result Consumer
To see predictions in real-time, open a third terminal and run:
```
python result_consumer.py
```
## Example Workflow
1. Ensure Kafka is running (refer to KafkaSetup.md).
2. Open three terminal windows and navigate to the project directory in each.
3. In Terminal 1, start the receiver-predictor:
```
python receiver_predictor.py
```
4. In Terminal 2, start the sender:
```
python sender.py
```
5. (Optional) In Terminal 3, start the result consumer:
```
python result_consumer.py
```
6. In the sender terminal (Terminal 2), enter an image path when prompted:
```
Enter the path to the image (or 'quit' to exit): /path/to/your/cat_image.jpg
```
7. View the results in Terminal 1 (receiver-predictor) or Terminal 3 (result consumer).
8. Repeat step 6 with different images as desired.
9. Type 'quit' in Terminal 2 (sender) to exit.
## Troubleshooting
- Verify Kafka is running correctly (check KafkaSetup.md).
- Ensure `dogcatclassifier.h5` is present in the project root.
- Verify all dependencies are installed (`pip list`).
- Check that the image paths you enter exist and are accessible.
## Customization
- To use a different model, replace `dogcatclassifier.h5` and update `receiver_predictor.py` if needed.
- Kafka topics and server addresses can be modified in the scripts if required.