{"id":49244099,"url":"https://github.com/code-master-harsh/eris","last_synced_at":"2026-04-26T22:00:45.108Z","repository":{"id":353325415,"uuid":"1218909656","full_name":"code-master-harsh/ERIS","owner":"code-master-harsh","description":"Emotion-aware Robotic Interaction System","archived":false,"fork":false,"pushed_at":"2026-04-23T11:08:29.000Z","size":13294,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-25T21:02:42.821Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/code-master-harsh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-23T10:37:26.000Z","updated_at":"2026-04-25T17:38:50.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/code-master-harsh/ERIS","commit_stats":null,"previous_names":["code-master-harsh/eris"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/code-master-harsh/ERIS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-master-harsh%2FERIS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-master-harsh%2FERIS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-master-harsh%2FERIS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-master-harsh%2FERIS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/code-master-harsh","download_url":"https://codeload.github.com/code-master-harsh/ERIS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-master-harsh%2FERIS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32314116,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T21:09:39.134Z","status":"ssl_error","status_checked_at":"2026-04-26T21:09:21.240Z","response_time":129,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-24T20:40:18.639Z","updated_at":"2026-04-26T22:00:45.068Z","avatar_url":"https://github.com/code-master-harsh.png","language":"Jupyter Notebook","readme":"# ERIS — Emotion-Aware Robotic Interaction System\n\nERIS is a real-time multimodal system that recognises human emotion from speech and facial expressions, fuses both signals with a learned gate, generates empathetic LLM responses, and can drive an emotion-aware quadruped robot simulation.\n\nThe project exists in two tightly related layers:\n\n1. **Multimodal emotion understanding**  \n   A live Gradio application that continuously processes webcam and microphone input, predicts emotion, tracks emotional context, and generates short supportive responses.\n\n2. **Embodied robotic expression**  \n   A ROS 2 / Gazebo / CHAMP-based quadruped control stack that converts the predicted emotion into physically distinct gait, posture, and motion behaviour.\n\n---\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Core Features](#core-features)\n- [System Architecture](#system-architecture)\n- [Emotion Classes](#emotion-classes)\n- [Project Structure](#project-structure)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Running the Project](#running-the-project)\n- [Robot Integration](#robot-integration)\n- [Model Notes](#model-notes)\n- [Acknowledgements](#acknowledgements)\n- [License](#license)\n\n---\n\n## Overview\n\nERIS started as a multimodal emotion recognition system and was extended into a complete interaction pipeline. The final system can:\n\n- capture **live audio and video**\n- predict one of four emotions: **angry, happy, neutral, sad**\n- combine audio and visual evidence using a **learned gated fusion module**\n- keep short-term emotional context across successive predictions\n- generate empathetic responses with an LLM\n- export the predicted emotional state to a robot simulation\n- modulate a quadruped’s gait and posture to match the detected emotion\n\nThe goal is not only to classify emotion, but to make the response feel more natural, contextual, and embodied.\n\n---\n\n## Core Features\n\n### Live multimodal emotion analysis\n- Continuous webcam and microphone capture in background threads\n- Rolling multimodal windows for repeated inference\n- Real-time prediction display\n- Emotion timeline / history across the session\n- Live gate readout showing the balance between audio and visual modalities\n\n### Empathetic conversational response\n- Context-aware short replies generated through Groq-hosted Llama 3.1 8B\n- Response behaviour that can adapt to:\n  - the current emotion\n  - the detected intensity\n  - recent emotion shifts\n- Optional response suppression when the emotional state is stable or not strong enough to require a reply\n\n### Offline / upload mode\n- Analyse a pre-recorded video file\n- Run the same multimodal inference pipeline\n- Display class probabilities and gate behaviour\n- Skip the live conversational layer when not needed\n\n### Robot expression\n- Emotion state written to a shared file for the robot process\n- Quadruped gait changes based on the predicted emotion\n- Smooth transitions between states using cosine interpolation\n- Emotion-specific posture, velocity, and motion profiles\n\n---\n\n## System Architecture\n\n```text\nAudio stream (microphone)\n        │\n        ▼\n   WavLM-Base+\n        │\n   attention pooling\n        │\n        ├───────────────┐\n        │               │\n        ▼               ▼\n   Learned gate     ViT-Face-Expression\n        │               │\n        └───────┬───────┘\n                ▼\n        Multimodal classifier\n                ▼\n     angry / happy / neutral / sad\n                ▼\n   ┌────────────┴────────────┐\n   ▼                         ▼\nEmpathetic LLM reply     Robot emotion state\n   ▼                         ▼\nUser-facing response    Quadruped gait control\n```\n\n---\n\n## Emotion Classes\n\nERIS currently predicts four emotions:\n\n- **angry**\n- **happy**\n- **neutral**\n- **sad**\n\nThese classes are used consistently across the multimodal model, the live response layer, and the robot behaviour mapping.\n\n---\n\n## Project Structure\n\nA typical project layout is:\n\n```text\nERIS/\n├── app.py                         # Gradio UI, live inference, LLM integration\n├── model.py                       # Multimodal model architecture\n├── multimodal_final_v4.pt         # Trained model checkpoint\n├── emotion_state.json             # Shared emotion state for the robot\n├── blaze_face_short_range.tflite   # MediaPipe face detector model\n├── .env                           # API keys and local configuration\n├── .gitignore\n├── README.md\n├── Stride_bot-main/               # Robot simulation / gait control\n│   ├── bot_walk.py                # Robot runtime loop\n│   ├── stridebot.urdf             # Robot description\n│   ├── src/\n│   │   ├── emotion_gait.py        # Emotion-specific gait parameters\n│   │   ├── gaitPlanner.py         # Base trot gait planner\n│   │   ├── kinematic_model.py     # IK model\n│   │   └── ...\n│   └── LICENSE\n└── ros2_quad_ws/ or similar       # ROS 2 integration workspace, if used\n    ├── src/\n    ├── launch/\n    └── ...\n```\n\n---\n\n## Requirements\n\n### For the multimodal app\n- Python 3.10 or newer\n- `ffmpeg`\n- Webcam\n- Microphone\n\n### For the robot integration stack\n- Ubuntu 22.04\n- ROS 2 Humble\n- Gazebo Fortress\n- CHAMP-based quadruped control stack\n- PyTorch, OpenCV, Gradio, MediaPipe, SoundDevice\n\n---\n\n## Installation\n\n### 1. Clone the repository\n\n```bash\ngit clone \u003cyour-repo-url\u003e\ncd ERIS\n```\n\n### 2. Create and activate a Python environment\n\n```bash\npython3 -m venv .venv\nsource .venv/bin/activate\n```\n\n### 3. Install Python dependencies\n\n```bash\npip install torch torchvision transformers mediapipe opencv-python \\\n            sounddevice soundfile gradio groq python-dotenv \\\n            matplotlib pybullet numpy\n```\n\n### 4. Download the MediaPipe face detector model\n\n```bash\ncurl -L -o blaze_face_short_range.tflite \\\n  \"https://storage.googleapis.com/mediapipe-models/face_detector/blaze_face_short_range/float16/1/blaze_face_short_range.tflite\"\n```\n\n### 5. Create a `.env` file\n\n```env\nGROQ_API_KEY=your_groq_api_key_here\nHF_TOKEN=your_huggingface_token_here\n```\n\n---\n\n## Running the Project\n\n### Multimodal emotion recognition app\n\n```bash\npython app.py\n```\n\nThe Gradio app runs locally and provides a live interface for webcam and microphone input.\n\n### Robot companion\n\nIn a separate terminal, run the robot simulation:\n\n```bash\ncd Stride_bot-main\npython bot_walk.py\n```\n\nThe robot reads the current emotion from `emotion_state.json` and changes gait accordingly.\n\n### ROS 2 / Gazebo behaviour stack\n\nIf you are using the ROS 2 integration variant, start the simulator and behaviour nodes in separate terminals:\n\n```bash\nsource /opt/ros/humble/setup.bash\nsource ~/ros2_ws/install/setup.bash\n\nros2 launch go2_config gazebo.launch.py\nros2 launch go2_behaviour emotion_motion.launch.py\n```\n\nYou can also dispatch manual commands from the CLI:\n\n```bash\nros2 run go2_behaviour behaviour_commander --action move --value 3.0 --emotion happy\nros2 run go2_behaviour behaviour_commander --action turn --value 90.0 --emotion angry\nros2 run go2_behaviour behaviour_commander --action move --value 1.5 --emotion sad\n```\n\n---\n\n## Robot Integration\n\nThe robot side of ERIS is designed to turn emotion into visible motion rather than a simple preset animation.\n\n### Behaviour mapping\n- **Happy**: energetic, bouncy, prancing gait\n- **Sad**: slower movement with a hunched posture\n- **Angry**: faster, more forceful motion with sharper turns\n- **Neutral**: stable, calm baseline locomotion\n\n### Motion design\n- Smooth interpolation between emotion states\n- Safe posture updates to avoid inverse kinematics instability\n- Dynamic adjustment of gait parameters such as stance duration and swing height\n- Heading correction using ground-truth odometry in the ROS 2 variant\n\n### Bridge between app and robot\nThe multimodal app can publish the current emotion state directly to the robot control layer, allowing the perception pipeline and the robot behaviour pipeline to operate as one connected system.\n\n---\n\n## Model Notes\n\nThe multimodal network uses:\n\n- **WavLM-Base+** for speech emotion features\n- **ViT-Face-Expression** for facial emotion features\n- **Temporal pooling / transformer-based aggregation** for sequence-level video understanding\n- **Learned gated fusion** to balance audio and video on a per-sample basis\n- **Final classifier** to predict the emotion label\n\nThis design makes the model more robust than using either modality alone.\n\n---\n\n## Acknowledgements\n\n- **WavLM-Base+** for speech feature extraction\n- **ViT-Face-Expression** for facial emotion classification\n- **Groq / Llama 3.1 8B** for low-latency empathetic response generation\n- **MediaPipe** for face detection\n- **Stride Bot** for the quadruped simulation and gait framework\n- **ROS 2**, **Gazebo Fortress**, and **CHAMP** for the embodied robotics pipeline\n\n---\n\n## License\n\nThis project contains components derived from the Stride Bot quadruped simulation, which retains its original GPL-3.0 licence in `Stride_bot-main/LICENSE`. Any additional project code should be licensed according to your repository policy.\n\n---\n\n## Summary\n\nERIS is a real-time emotion-aware interaction system that combines speech and facial emotion recognition, learned multimodal fusion, contextual LLM responses, and emotion-driven quadruped motion into a single integrated pipeline.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcode-master-harsh%2Feris","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcode-master-harsh%2Feris","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcode-master-harsh%2Feris/lists"}