{"id":51062175,"url":"https://github.com/sebsop/realtime-entity-classifier","last_synced_at":"2026-06-23T03:31:31.003Z","repository":{"id":288507417,"uuid":"968329365","full_name":"sebsop/realtime-entity-classifier","owner":"sebsop","description":"Real-time video system for detecting and classifying people and pets, using enhanced MobileNetV2/V3 segmentation. Optimized for consumer hardware and developed in Python with PyTorch.","archived":false,"fork":false,"pushed_at":"2025-08-13T11:52:37.000Z","size":16852,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-12T09:37:16.249Z","etag":null,"topics":["artificial-intelligence","computer-vision","convolutional-neural-networks","jupyter-notebook","machine-learning","mobilenet","python","transfer-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sebsop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-17T22:32:23.000Z","updated_at":"2025-10-05T21:51:35.000Z","dependencies_parsed_at":"2025-04-18T13:23:00.831Z","dependency_job_id":"0e40c4ed-0cbd-4b79-a727-14689bba5f42","html_url":"https://github.com/sebsop/realtime-entity-classifier","commit_stats":null,"previous_names":["dosqas/realtime-entity-classifier","sebsop/realtime-entity-classifier"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sebsop/realtime-entity-classifier","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebsop%2Frealtime-entity-classifier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebsop%2Frealtime-entity-classifier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebsop%2Frealtime-entity-classifier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebsop%2Frealtime-entity-classifier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sebsop","download_url":"https://codeload.github.com/sebsop/realtime-entity-classifier/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebsop%2Frealtime-entity-classifier/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34674702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","computer-vision","convolutional-neural-networks","jupyter-notebook","machine-learning","mobilenet","python","transfer-learning"],"created_at":"2026-06-23T03:31:30.200Z","updated_at":"2026-06-23T03:31:30.998Z","avatar_url":"https://github.com/sebsop.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🔍 Real-Time Entity Classification System\n\nA smart computer vision system that identifies and classifies people and pets in real-time using advanced deep learning techniques.\n\n![Project Demo](./demo/project_demo.gif)  \n*\"Watch the app nail the purr-fect prediction — correctly spotting me and my friend’s cat, Felix!\"*\n\n---\n\n## ✨ Key Features\n\n- **Four-Class Detection**: Accurately identifies `owner`, `pet`, `other person`, and `background` classes\n- **Adaptive Processing**: Automatically switches between classification and segmentation for improved accuracy\n- **Real-Time Performance**: ~33 FPS on consumer hardware (NVIDIA RTX 3050)\n- **Privacy-Focused**: All processing happens locally on your device\n- **Interactive Controls**: Toggle segmentation mode and visualize confidence scores\n- **Memory Efficient**: Optimized for resource-constrained environments\n\n---\n\n## 🧠 Technical Overview\n\nThis project combines transfer learning with efficient model deployment to create a responsive computer vision system that runs smoothly on mid-range hardware:\n\n- **Base Architecture**: MobileNetV2 (finetuned from ImageNet weights)\n- **Enhancement**: LRASPP MobileNetV3 segmentation model for challenging cases\n- **Confidence Threshold**: Auto-switching between models at 0.7 confidence level\n- **Training Method**: Transfer learning with frozen feature extraction layers\n- **Performance**: 99.4% accuracy in ideal conditions, 84.2% in low light\n\n---\n\n## 📊 Model Specifications\n\n| Attribute | Value |\n|-----------|-------|\n| Architecture | MobileNetV2 (finetuned) |\n| Input Resolution | 224x224 (resized from 640x480) |\n| Output Classes | `['owner', 'pet', 'other person', 'background']` |\n| Model Format | `.pth` |\n| Model Size | ~10 MB (quantized) |\n| Inference Speed | 33 FPS @ 640x480 |\n| Hardware Tested | NVIDIA RTX 3050, CUDA 11.8 |\n| Framework | PyTorch 3.13.2 |\n\n---\n\n## 🚀 Getting Started\n\n### Prerequisites\n\n```bash\n# Clone the repository\ngit clone https://github.com/sebsop/Realtime-Entity-Classifier.git\ncd Realtime-Entity-Classifier\n\n# Create and activate virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### Running the Classifier\n\n```bash\npython src/realtime_classifier.py\n```\n\n### Controls\n\n- Press `s` to toggle forced segmentation mode\n- Press `q` to quit\n\n---\n\n## 🏗️ Architecture Details\n\n### Classification Model (MobileNetV2)\n\nThe system uses a modified MobileNetV2 architecture with:\n\n```python\nmodel.classifier[1] = nn.Sequential(\n    nn.Linear(model.classifier[1].in_features, 256),\n    nn.ReLU(),\n    nn.Dropout(0.2),\n    nn.Linear(256, 4)\n)\n```\n\n- **Intermediate Layer (256 neurons)**: Enhanced representational capacity\n- **ReLU Activation**: Efficient non-linearity\n- **Dropout (0.2)**: Prevents overfitting\n- **Xavier Initialization**: Improves convergence speed\n\n### Training Configuration\n\n- **Optimizer**: Adam with selective training\n- **Learning Rate**: 5e-5\n- **Weight Decay**: 1e-5\n- **Loss Function**: CrossEntropyLoss with label smoothing (0.1)\n- **Epochs**: 10 (converged early)\n\n### Segmentation Enhancement\n\nWhen classification confidence drops below threshold:\n\n1. An LRASPP MobileNetV3 segmentation model identifies people and pets\n2. Segmentation mask isolates the subject from the background\n3. Classification is re-run on the masked input\n4. System returns to normal mode after confidence improves\n\n---\n\n## 📈 Performance Metrics\n\n### Training Progress\n\n| Epoch | Loss   | Accuracy | Δ Accuracy |\n|-------|--------|----------|------------|\n| 1     | 0.1600 | 95.01%   | +0%        |\n| 2     | 0.0404 | 98.74%   | +3.73%     |\n| 3     | 0.0293 | 99.11%   | +0.37%     |\n| 4     | 0.0238 | 99.17%   | +0.06%     |\n| 5     | 0.0209 | 99.34%   | +0.17%     |\n| 6     | 0.0217 | 99.26%   | -0.08%     |\n| 7     | 0.0194 | 99.35%   | +0.09%     |\n| 8     | 0.0187 | 99.35%   | +0.00%     |\n| 9     | 0.0158 | 99.48%   | +0.13%     |\n| 10    | 0.0153 | 99.44%   | -0.04%     |\n\n### Dataset Overview\n\n- **Total Samples**: 34,575  \n- **Class Distribution**:\n  - `owner`: 8,750 samples (25.3%)  \n    - Sourced from a **2:20 min video** of myself walking around the house in varied lighting conditions, angles and backgrounds.\n  - `pet`: 4,575 samples (13.2%)  \n    - Extracted from a **30-second video** of my friend Bogdan’s cat, **Felix**.\n  - `other person`: 12,500 samples (36.2%)  \n    - Includes **2,500 cropped face images** from the [Human Faces Kaggle dataset](https://www.kaggle.com/datasets/ashwingupta3012/human-faces?resource=download).\n  - `background`: 8,750 samples (25.3%)  \n    - Captured from a **30-second video** of walking around the house with no subject in focus.\n   \n---\n\n## 🧪 Known Limitations\n\n1. **Pet Detection**:\n   - Accuracy drops when \u003c30% of the pet's body is visible\n   - Low lighting reduces confidence by ~40%\n\n2. **Person Identification**:\n   - Needs ≥92% confidence to reliably classify \"owner\" vs \"other person\"\n   - False positives with reflections (mirrors, glass)\n   - May struggle with diverse \"other person\" examples\n  \n---\n\n## 🔧 Customization\n\n### Setting Custom Confidence Threshold\n\n```python\n# In realtime_classifier.py\nCONFIDENCE_THRESHOLD = 0.7  # Default\n```\n\n### Toggling Pet Detection\n\n```python\n# In realtime_classifier.py\nPET_MASK_ENABLED = True  # Set to False to disable generic pet detection\n```\n\n---\n\n## 📂 Project Structure\n\n```\nrealtime-entity-classifier/\n├── demo/\n│   └── project_demo.gif             # Project demo GIF\n├── data/                            # Dataset used for training and evaluation\n│   ├── owner/                       # Images and optional video of the owner\n│   │   ├── images/                  # Folder containing image samples\n│   │   └── owner.mp4 (optional)     # Optional video for data generation\n│   ├── pet/                         # Images and optional video of pets (e.g., cat, dog)\n│   │   ├── images/\n│   │   └── pet.mp4 (optional)\n│   ├── other_people/               # Images and optional video of non-owners\n│   │   ├── images/\n│   │   └── other_people.mp4 (optional)\n│   └── background/                 # Background-only scenes\n│       ├── images/\n│       └── background.mp4 (optional)\n├── models/                          # Trained model weights\n│   └── entity_classifier.pth        # Main classifier model\n├── notebooks/                       # Jupyter notebooks\n│   └── classifier_build_and_train.ipynb\n├── reports/                         # Reports and visualizations\n│   ├── TEST_RESULTS.md              # Full test performance summary\n│   └── training_plots/\n│       └── mobilenetv2_4class_finetune_20250420.jpg  # Training progress plot\n├── src/\n│   └── realtime_classifier.py       # Main application script\n├── requirements.txt                 # Python dependencies\n└── README.md                        # Project overview and usage guide\n```\n\n---\n\n## 🙏 Acknowledgments\n\n- **[PyTorch](https://pytorch.org/)** – for the powerful and flexible deep learning framework  \n- **[TorchVision](https://pytorch.org/vision/stable/index.html)** – for pre-trained models and helpful computer vision utilities  \n- **[OpenCV](https://opencv.org/)** – for enabling efficient image and video processing  \n- **[Human Faces Dataset (Kaggle)](https://www.kaggle.com/datasets/ashwingupta3012/human-faces?resource=download)** – used for training on diverse human faces for the \"other person\" class\n- **My friend Bogdan and his cat Felix** - for helping me with data to train the model for the \"pet\" class\n\n---\n\n## 📄 License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n\n---\n\n## 💡 Contact\n\nQuestions, feedback, or ideas? Reach out anytime at [sebastian.soptelea@proton.me](mailto:sebastian.soptelea@proton.me).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsebsop%2Frealtime-entity-classifier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsebsop%2Frealtime-entity-classifier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsebsop%2Frealtime-entity-classifier/lists"}