{"id":38147010,"url":"https://github.com/kudosscience/fer_and_ter_model","last_synced_at":"2026-01-16T22:57:26.016Z","repository":{"id":331342836,"uuid":"1028552339","full_name":"kudosscience/fer_and_ter_model","owner":"kudosscience","description":"A comprehensive real-time emotion recognition system combining facial and textual analysis with Furhat robot integration for social robotics applications.","archived":false,"fork":false,"pushed_at":"2026-01-01T18:25:59.000Z","size":10777,"stargazers_count":4,"open_issues_count":14,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-06T16:52:18.522Z","etag":null,"topics":["computer-vision","deep-learning","distilbert","emotion-recognition","facial-emotion-detection","fer2013","furhat-robot","machine-learning","multimodal-fusion","natural-language-processing","opencv","pytorch","real-time-processing","social-robotics","text-emotion-detection"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kudosscience.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-29T17:42:41.000Z","updated_at":"2025-12-01T14:11:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kudosscience/fer_and_ter_model","commit_stats":null,"previous_names":["kudosscience/fer_and_ter_model"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/kudosscience/fer_and_ter_model","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kudosscience%2Ffer_and_ter_model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kudosscience%2Ffer_and_ter_model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kudosscience%2Ffer_and_ter_model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kudosscience%2Ffer_and_ter_model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kudosscience","download_url":"https://codeload.github.com/kudosscience/fer_and_ter_model/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kudosscience%2Ffer_and_ter_model/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28486964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T22:54:02.790Z","status":"ssl_error","status_checked_at":"2026-01-16T22:50:10.344Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","distilbert","emotion-recognition","facial-emotion-detection","fer2013","furhat-robot","machine-learning","multimodal-fusion","natural-language-processing","opencv","pytorch","real-time-processing","social-robotics","text-emotion-detection"],"created_at":"2026-01-16T22:57:25.418Z","updated_at":"2026-01-16T22:57:26.008Z","avatar_url":"https://github.com/kudosscience.png","language":"Jupyter Notebook","readme":"# Multimodal Emotion Recognition System\n\nA comprehensive, real-time emotion recognition system that combines **Facial Emotion Recognition (FER)** and **Textual Emotion Recognition (TER)** with advanced multimodal fusion capabilities and **Furhat robot integration**. This system provides accurate emotion detection through computer vision and natural language processing, with support for interactive robotics applications.\n\n## 🌟 Key Features\n\n- **🎭 Facial Emotion Recognition**: Real-time emotion detection from camera feed using CNN models trained on FER2013\n- **💬 Textual Emotion Recognition**: Voice-to-text emotion analysis using DistilBERT-based models  \n- **🔀 Multimodal Fusion**: Advanced fusion strategies (confidence-based, weighted average, and formula-based)\n- **🤖 Furhat Integration**: Complete social robot platform integration with real-time emotion feedback\n- **⚡ Real-time Processing**: Live emotion recognition with interactive GUI interfaces\n- **🛡️ Robust Architecture**: Comprehensive error handling and fallback mechanisms\n- **📊 Comprehensive Testing**: Full test suite with system health checks and validation\n\n## 🧪 Testing \u0026 Validation\n\nThe system includes comprehensive testing capabilities to ensure reliability:\n\n### Run All Tests\n\n```bash\n# Run comprehensive system tests\npython tests/test_multimodal_system.py    # Complete system validation\npython tests/test_fer_model.py           # FER component testing\npython tests/test_furhat_integration.py  # Furhat integration testing\npython tests/test_formula_fusion.py      # Fusion algorithm testing\npython tests/test_corrected_formula.py   # Mathematical validation\n```\n\n### System Health Check\n\n```bash\n# Validate system configuration and dependencies\npython demos/demo_multimodal_usage.py\n```\n\nThe test suite validates:\n\n- Model loading and initialization\n- Camera and microphone availability  \n- Emotion fusion algorithm accuracy\n- Robot integration functionality\n- Error handling and fallback mechanisms\n\n## 📁 Project Architecture\n\n```text\nfer_and_ter_model/\n├── src/                          # Source code\n│   ├── fer/                      # Facial Emotion Recognition\n│   │   └── camera_fer_inference.py\n│   ├── ter/                      # Textual Emotion Recognition  \n│   │   ├── voice_ter_inference.py\n│   │   └── setup_voice_ter.py\n│   ├── multimodal/               # Multimodal Fusion\n│   │   └── multimodal_emotion_inference.py\n│   ├── furhat/                   # Furhat Robot Integration\n│   │   └── furhat_multimodal_emotion_inference.py\n│   └── utils/                    # Shared utilities\n├── models/                       # Trained models\n│   ├── fer2013_final_model.pth\n│   └── ter_distilbert_model/\n├── datasets/                     # Dataset files\n│   └── multimodal_emotion_dataset.json\n├── notebooks/                    # Jupyter notebooks\n│   ├── fer2013_model_training.ipynb\n│   ├── multimodal_emotion_fusion.ipynb\n│   ├── multimodal_emotion_recognition.ipynb\n│   └── textual_emotion_recognition_distilbert.ipynb\n├── tests/                        # Test scripts\n│   ├── test_corrected_formula.py\n│   ├── test_fer_model.py\n│   ├── test_formula_fusion.py\n│   ├── test_furhat_integration.py\n│   └── test_multimodal_system.py\n├── demos/                        # Demo scripts\n│   ├── demo_furhat_usage.py\n│   ├── demo_multimodal_usage.py\n│   ├── demo_usage.py\n│   └── demo_voice_ter.py\n├── docs/                         # Documentation\n│   ├── DATASET_SETUP.md\n│   ├── FER_PROJECT_SUMMARY.md\n│   ├── FURHAT_INTEGRATION_SUMMARY.md\n│   ├── IMPLEMENTATION_SUMMARY.md\n│   ├── JUNIE.md\n│   └── README_*.md files\n├── requirements/                 # Requirements files\n│   ├── requirements_backup.txt\n│   ├── requirements_camera_inference.txt\n│   ├── requirements_furhat.txt\n│   ├── requirements_multimodal.txt\n│   └── requirements_voice_ter.txt\n└── README.md                     # This file\n```\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Python 3.8+ (3.11+ recommended)\n- Webcam for facial emotion recognition\n- Microphone for voice/text emotion recognition\n- GPU support recommended for optimal performance\n\n### Installation\n\n1. **Clone the repository**:\n\n   ```bash\n   git clone https://github.com/kudosscience/fer_and_ter_model.git\n   cd fer_and_ter_model\n   ```\n\n2. **Install the package**:\n\n   ```bash\n   # Install with all components\n   pip install -e \".[all]\"\n   \n   # OR install specific components:\n   pip install -e \".[multimodal]\"    # For multimodal processing\n   pip install -e \".[furhat]\"        # For Furhat robot integration\n   pip install -e \".[camera_inference]\"  # For FER only\n   pip install -e \".[voice_ter]\"     # For TER only\n   ```\n\n3. **Alternative: Install from requirements files**:\n\n   ```bash\n   # Choose based on your use case:\n   pip install -r requirements/requirements_multimodal.txt     # Recommended\n   pip install -r requirements/requirements_furhat.txt         # For robot integration\n   pip install -r requirements/requirements_camera_inference.txt  # FER only\n   pip install -r requirements/requirements_voice_ter.txt      # TER only\n   ```\n\n### Quick Demo\n\n```bash\n# Run the comprehensive multimodal demo\npython demos/demo_multimodal_usage.py\n\n# Try individual components\npython demos/demo_usage.py           # Basic FER demo\npython demos/demo_voice_ter.py       # TER demo  \npython demos/demo_furhat_usage.py    # Furhat integration demo\n```\n\n### Console Commands\n\nAfter installation, you can use these console commands:\n\n```bash\n# Run individual components\nfer-camera              # Launch facial emotion recognition\nter-voice              # Launch textual emotion recognition  \nmultimodal-emotion     # Launch multimodal system\nfurhat-emotion         # Launch Furhat integration\n```\n\n## 🧩 System Components\n\n### 🎭 FER (Facial Emotion Recognition)\n\n- **Real-time Processing**: Live camera feed emotion detection with GUI overlay\n- **CNN Architecture**: Deep learning model trained on FER2013 dataset  \n- **7 Emotion Classes**: Happy, Sad, Angry, Fear, Surprise, Disgust, Neutral\n- **High Accuracy**: Optimized model with robust preprocessing pipeline\n- **Fallback Support**: Graceful handling of camera unavailability\n\n**Usage**:\n\n```bash\n# Direct script execution\npython src/fer/camera_fer_inference.py\n\n# Console command\nfer-camera\n```\n\n### 💬 TER (Textual Emotion Recognition)\n\n- **Voice-to-Text Pipeline**: Real-time speech recognition and emotion analysis\n- **DistilBERT Model**: State-of-the-art transformer-based emotion classification\n- **Multi-emotion Support**: Comprehensive emotion category detection\n- **Robust Processing**: Advanced text preprocessing and normalization\n- **Audio Fallback**: Multiple audio input handling strategies\n\n**Usage**:\n\n```bash\n# Direct script execution  \npython src/ter/voice_ter_inference.py\n\n# Console command\nter-voice\n```\n\n### 🔀 Multimodal Fusion System\n\nThe core innovation of this system - combines FER and TER for superior accuracy:\n\n- **Multiple Fusion Strategies**:\n  - **Confidence-based**: Selects prediction with highest confidence score\n  - **Weighted Average**: Combines predictions with 60% facial, 40% textual weighting\n  - **Formula-based**: Advanced mathematical fusion using custom algorithm\n\n- **Real-time Integration**: Simultaneous processing of visual and audio streams\n- **Adaptive Fallback**: Works with single modality when needed\n- **Interactive GUI**: Live visualization of both streams and fusion results\n\n**Usage**:\n\n```bash\n# Default multimodal processing\npython src/multimodal/multimodal_emotion_inference.py\n\n# With specific fusion strategy\npython src/multimodal/multimodal_emotion_inference.py --fusion confidence_based\npython src/multimodal/multimodal_emotion_inference.py --fusion weighted_average  \npython src/multimodal/multimodal_emotion_inference.py --fusion formula_based\n\n# Console command\nmultimodal-emotion\n```\n\n### 🤖 Furhat Robot Integration\n\nComplete social robotics platform integration for interactive emotion recognition:\n\n- **Furhat Remote API**: Official SDK integration for robot communication\n- **Interactive Responses**: Robot gestures, speech, and LED feedback based on detected emotions\n- **Voice Integration**: Uses robot's microphone for natural interaction\n- **Real-time Feedback**: Immediate emotional responses and social cues\n- **Robust Connection**: Graceful fallback when robot unavailable\n\n**Robot Capabilities**:\n\n- Emotional gesture mapping (BigSmile, Frown, Surprised, etc.)\n- LED color changes reflecting emotional states  \n- Speech synthesis for emotion acknowledgment\n- Interactive conversation flow\n\n**Usage**:\n\n```bash\n# Furhat integration (requires robot connection)\npython src/furhat/furhat_multimodal_emotion_inference.py\n\n# With fusion strategy\npython src/furhat/furhat_multimodal_emotion_inference.py --fusion formula_based\n\n# Console command  \nfurhat-emotion\n```\n\n## 📚 Comprehensive Documentation\n\nDetailed documentation for each component is available in the `docs/` directory:\n\n### Core Documentation\n\n- **[IMPLEMENTATION_SUMMARY.md](docs/IMPLEMENTATION_SUMMARY.md)** - Complete technical implementation overview\n- **[FER_PROJECT_SUMMARY.md](docs/FER_PROJECT_SUMMARY.md)** - Facial emotion recognition deep dive  \n- **[FURHAT_INTEGRATION_SUMMARY.md](docs/FURHAT_INTEGRATION_SUMMARY.md)** - Robot integration guide\n- **[DATASET_SETUP.md](docs/DATASET_SETUP.md)** - Dataset preparation and training\n\n### Component-Specific Guides\n\n- **[README_multimodal.md](docs/README_multimodal.md)** - Multimodal system usage and configuration\n- **[README_furhat.md](docs/README_furhat.md)** - Furhat robot setup and integration\n- **[README_camera_inference.md](docs/README_camera_inference.md)** - FER system configuration\n- **[README_voice_ter.md](docs/README_voice_ter.md)** - Voice/text emotion recognition setup\n\n## ⚙️ Advanced Configuration\n\n### Custom Model Usage\n\n```bash\n# Use custom FER model\npython src/multimodal/multimodal_emotion_inference.py \\\n    --fer_model ./path/to/custom_fer_model.pth\n\n# Use custom TER model  \npython src/multimodal/multimodal_emotion_inference.py \\\n    --ter_model ./path/to/custom_ter_model/\n```\n\n### Fusion Strategy Selection\n\n```bash\n# Confidence-based fusion (chooses most confident prediction)\npython src/multimodal/multimodal_emotion_inference.py --fusion confidence_based\n\n# Weighted average fusion (60% facial, 40% textual)  \npython src/multimodal/multimodal_emotion_inference.py --fusion weighted_average\n\n# Formula-based fusion (mathematical optimization)\npython src/multimodal/multimodal_emotion_inference.py --fusion formula_based\n```\n\n### Performance Optimization\n\n```bash\n# GPU acceleration (if available)\nexport CUDA_VISIBLE_DEVICES=0\npython src/multimodal/multimodal_emotion_inference.py\n\n# CPU-only mode\nexport CUDA_VISIBLE_DEVICES=\"\"\npython src/multimodal/multimodal_emotion_inference.py\n```\n\n## 🔧 Development \u0026 Extension\n\n### Project Structure Overview\n\nThe system follows a modular architecture allowing easy extension and modification:\n\n- **`src/`** - Core source code with modular components\n- **`models/`** - Pre-trained models (FER CNN, TER DistilBERT)\n- **`datasets/`** - Training and evaluation datasets\n- **`tests/`** - Comprehensive test suite\n- **`demos/`** - Usage examples and demonstrations\n- **`docs/`** - Detailed technical documentation\n- **`requirements/`** - Component-specific dependency management\n\n### Adding New Components\n\n1. Create module in appropriate `src/` subdirectory\n2. Add requirements to relevant requirements file\n3. Create tests in `tests/` directory\n4. Add demo in `demos/` directory\n5. Update documentation in `docs/`\n\n### Contributing\n\nThis project uses professional development practices:\n\n- **Modular Design**: Each component is self-contained and reusable\n- **Comprehensive Testing**: Full test coverage with validation scripts\n- **Documentation**: Extensive documentation for all components\n- **Package Management**: Proper Python packaging with setuptools\n- **Console Integration**: Command-line tools for easy usage\n\n## 🛠️ Troubleshooting\n\n### Common Issues\n\n**Camera not detected**:\n\n```bash\n# Test camera access\npython -c \"import cv2; print('Camera available:', cv2.VideoCapture(0).isOpened())\"\n```\n\n**Microphone issues**:\n\n```bash\n# Test microphone access  \npython -c \"import speech_recognition as sr; print('Microphone available:', len(sr.Microphone.list_microphone_names()) \u003e 0)\"\n```\n\n**Model loading errors**:\n\n- Ensure models are downloaded and in `models/` directory\n- Check file permissions and paths\n- Verify CUDA availability for GPU models\n\n**Furhat connection issues**:\n\n- Verify robot IP address and port\n- Check network connectivity\n- Ensure Furhat Remote API service is running\n\n### Performance Tips\n\n- Use GPU acceleration when available\n- Close other applications using camera/microphone\n- Ensure adequate lighting for facial recognition\n- Use external microphone for better voice recognition\n- Run system health check before important usage\n\n## 📊 Model Information\n\n### FER Model (Facial Emotion Recognition)\n\n- **Architecture**: Custom CNN trained on FER2013\n- **Input**: 48x48 grayscale facial images\n- **Output**: 7 emotion classes (Happy, Sad, Angry, Fear, Surprise, Disgust, Neutral)\n- **Accuracy**: Optimized for real-time performance with robust preprocessing\n\n### TER Model (Textual Emotion Recognition)\n\n- **Architecture**: DistilBERT-based transformer\n- **Input**: Text transcribed from speech\n- **Output**: Multi-dimensional emotion classification\n- **Features**: Advanced text preprocessing and normalization\n\n### Fusion Algorithms\n\n- **Confidence-based**: Selects highest confidence prediction\n- **Weighted Average**: Optimized 60/40 facial/textual weighting\n- **Formula-based**: Mathematical fusion using correlation analysis\n\n## 🎯 Use Cases\n\n- **Research**: Emotion recognition research and experimentation\n- **Education**: Teaching multimodal AI and emotion recognition\n- **Healthcare**: Patient emotion monitoring and therapy assistance  \n- **Human-Computer Interaction**: Emotional interfaces and feedback systems\n- **Social Robotics**: Interactive robots with emotional intelligence\n- **Accessibility**: Emotion-aware assistive technologies\n\n## 📈 Performance Metrics\n\nThe system has been validated across multiple scenarios:\n\n- **Real-time Processing**: \u003c 100ms latency for combined FER+TER\n- **Accuracy**: Improved performance through multimodal fusion\n- **Robustness**: Graceful degradation with single modality\n- **Scalability**: Modular architecture supports easy extension\n\n## 🔗 Related Projects\n\n- [FER2013 Dataset](https://www.kaggle.com/deadskull7/fer2013)\n- [Furhat Robot Platform](https://furhatrobotics.com/)\n- [DistilBERT Model](https://huggingface.co/distilbert-base-uncased)\n- [OpenCV Computer Vision](https://opencv.org/)\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 👤 Author\n\nHenry Ward\n\n- GitHub: [@kudosscience](https://github.com/kudosscience)\n- Email: [45144290+kudosscience@users.noreply.github.com](mailto:45144290+kudosscience@users.noreply.github.com)\n\n## 🙏 Acknowledgments\n\n- FER2013 dataset contributors\n- Hugging Face Transformers library\n- OpenCV community\n- Furhat Robotics platform\n- PyTorch and scikit-learn teams\n\n## 📞 Support\n\nFor issues, questions, or contributions:\n\n1. Check the [Issues](https://github.com/kudosscience/fer_and_ter_model/issues) page\n2. Review the comprehensive documentation in `docs/`\n3. Run the system health check: `python demos/demo_multimodal_usage.py`\n4. Create a new issue with detailed information\n\n---\n\nBuilt with ❤️ for multimodal emotion recognition research and applications\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkudosscience%2Ffer_and_ter_model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkudosscience%2Ffer_and_ter_model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkudosscience%2Ffer_and_ter_model/lists"}