{"id":18237486,"url":"https://github.com/eesunmoon/on-device_multimodal_er","last_synced_at":"2026-02-08T06:39:10.397Z","repository":{"id":258148675,"uuid":"872638205","full_name":"EesunMoon/On-device_Multimodal_ER","owner":"EesunMoon","description":"[Research - MINES Lab] Multimodal Emotion Recognition for On-device AI","archived":false,"fork":false,"pushed_at":"2024-12-13T00:14:25.000Z","size":58,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-25T08:02:08.450Z","etag":null,"topics":["artificial-intelligence","data-analysis","deep-learning","embedded-systems","emotion-recognition","heart-rate-analysis","multimodal-fusion","npu","on-device","python","speech-processing","speech-recognition","tensorflow","wearable-devices"],"latest_commit_sha":null,"homepage":"https://home.sejong.ac.kr/~hyungkim/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EesunMoon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-14T19:44:26.000Z","updated_at":"2025-01-30T05:30:33.000Z","dependencies_parsed_at":"2024-12-13T01:18:49.042Z","dependency_job_id":"92f3c5cb-0d55-402f-b0ba-3c4009f1c08a","html_url":"https://github.com/EesunMoon/On-device_Multimodal_ER","commit_stats":{"total_commits":6,"total_committers":2,"mean_commits":3.0,"dds":"0.16666666666666663","last_synced_commit":"c4214071f275838282ab0091c564a7a51b3bb79c"},"previous_names":["eesunmoon/on-device_multimodal_er"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EesunMoon%2FOn-device_Multimodal_ER","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EesunMoon%2FOn-device_Multimodal_ER/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EesunMoon%2FOn-device_Multimodal_ER/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EesunMoon%2FOn-device_Multimodal_ER/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EesunMoon","download_url":"https://codeload.github.com/EesunMoon/On-device_Multimodal_ER/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248385507,"owners_count":21094900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","data-analysis","deep-learning","embedded-systems","emotion-recognition","heart-rate-analysis","multimodal-fusion","npu","on-device","python","speech-processing","speech-recognition","tensorflow","wearable-devices"],"created_at":"2024-11-05T02:04:44.775Z","updated_at":"2026-02-08T06:39:10.340Z","avatar_url":"https://github.com/EesunMoon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# On-device Multimodal Emotion Recognition on Neural Processing Unit (NPU)\n**Optimizing AI for low latency and power consumption in real-time applications.**\n\n---\n\n## 🔍 Project Summary\n\nAs part of a government agency project, I led the development of an **on-device multimodal emotion recognition system** on NPUs (Neural Processing Units). The project focused on optimizing real-time AI applications for **high emotion classification accuracy**, **low latency**, and **power efficiency**, addressing constraints typical of edge systems, such as limited model size and computational resources.\n\n### Objectives\n1. **Enhancing emotion recognition performance** by leveraging multimodal data sources, including:\n   - Heart rate (HR)\n   - EEG\n   - Speech\n   - Images\n\n2. **Implementing a scalable real-time system** by embedding models on NPUs to reduce latency and power consumption.\n\n---\n\n## 🛠 Project Workflow\n### Overall Architecture\n![figure1](https://github.com/user-attachments/assets/e22babde-a2ad-42d1-bf5e-509ebed0e3f7)\n\n### Detailed Structures of Emotion Recognition Models\n![figure2](https://github.com/user-attachments/assets/ed881ac7-39db-447f-a180-429580abd3cd)\n\n### 1. Model Design and Optimization\n- **Simplified Architectures**: Developed deep learning models using architectures like CNNs and dense layers to balance performance and complexity.\n- **Hyperparameter Tuning**: Conducted ablation studies to fine-tune parameters such as optimizer type, number of epochs, batch size, and loss functions.\n- **Multimodal Fusion**: Adopted a **score-based fusion method** to combine outputs from multiple models at the decision level, avoiding additional neural network complexity.\n\n### 2. NPU Deployment\n- Converted models into **ONNX format** and compiled them using the **MXQ compiler** for compatibility with Mobilint’s NPU chips.\n- Applied **quantization techniques** (Max, Percentile, and Max-Percentile) to compress models, optimizing based on an efficiency metric combining:\n  - Matrix: Accuracy-increase ratio x Compression ratio\n\n\n## 📊 Optimization Methods\n\n### Multimodal Fusion and Simplified Models\n- Built individual models for HR, EEG, speech, and image data.\n- Focused on reducing model parameters while maintaining relative performance using simple architectures like CNN and dense layers.\n- Used **score-based fusion** to integrate outputs without additional network complexity.\n\n### Quantization Techniques\n- Converted models into NPU-compatible formats via **ONNX** and the **MXQ compiler**.\n- Applied three quantization methods to determine the best compression:\n  - **MAX**: Clipping ranges based on minimum and maximum values.\n  - **Percentile**: Clipping ranges based on top percentile values.\n  - **Max-Percentile**: Clipping ranges based on the top percentile of maximum values.\n\n---\n\n## ⚙️ Evaluation Metrics\n### Emotion Classification Accuracy\n- Achieved an impressive **99.68% accuracy**, ensuring reliable and robust emotion recognition in real-time applications.\n\n### Latency\n- Compared model size before and after compression.\n- Achieved **1.47x reduction** in model size.\n\n### Power Consumption\n- Measured power usage with an outlet power meter.\n- Found **3.12x reduction** in power consumption for NPU-based models compared to GPU-based models.\n\n## 📝 Key Findings\n\n- The system achieved significant improvements in **efficiency and scalability**, making it suitable for real-time AI applications.\n- Successfully implemented at the **Korean Institute of Science and Technology** as part of a government initiative.\n- Findings were presented at an academic conference, and a related paper is currently under review.\n- Reinforced my passion for developing **efficient, real-world AI systems**.\n\n## 🤔 Insights on Clipping Range for Quantization\n- **MAX**: Activations clipped using minimum and maximum values.\n- **Percentile**: Activations clipped using the top percentile of values.\n- **Max-Percentile**: Activations clipped using the top percentile of maximum values.\n\n---\n\n## 🌟 Conclusion\n\nThis project demonstrated the viability of deploying **real-time AI systems on edge devices** by optimizing multimodal emotion recognition models for **low latency** and **power efficiency**. It solidified my passion for creating **practical and scalable AI solutions** for real-world applications.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feesunmoon%2Fon-device_multimodal_er","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feesunmoon%2Fon-device_multimodal_er","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feesunmoon%2Fon-device_multimodal_er/lists"}