{"id":25952414,"url":"https://github.com/junayed-hasan/spontaneous-smile-recognition","last_synced_at":"2026-02-03T09:41:46.330Z","repository":{"id":229569091,"uuid":"777063508","full_name":"junayed-hasan/spontaneous-smile-recognition","owner":"junayed-hasan","description":"A deep learning framework for distinguishing spontaneous from posed smiles in videos. It uses a multi-task approach, leveraging Duchenne Marker features through transformer networks. The model simultaneously predicts smile types and D-Markers, achieving state-of-the-art results on major datasets without D-Marker computation during inference.","archived":false,"fork":false,"pushed_at":"2025-01-01T00:14:51.000Z","size":71,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-07T05:42:22.077Z","etag":null,"topics":["computer-vision","feature-extraction","neural-network","pattern-recognition","transformers","video-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/junayed-hasan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-25T05:52:25.000Z","updated_at":"2025-01-01T00:14:55.000Z","dependencies_parsed_at":"2025-03-04T14:51:13.890Z","dependency_job_id":"1d97acb4-877b-4b63-a8c1-9884cda976ab","html_url":"https://github.com/junayed-hasan/spontaneous-smile-recognition","commit_stats":null,"previous_names":["junayed-hasan/spontaneous-smile-recognition"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/junayed-hasan/spontaneous-smile-recognition","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2Fspontaneous-smile-recognition","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2Fspontaneous-smile-recognition/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2Fspontaneous-smile-recognition/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2Fspontaneous-smile-recognition/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/junayed-hasan","download_url":"https://codeload.github.com/junayed-hasan/spontaneous-smile-recognition/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2Fspontaneous-smile-recognition/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29039825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-03T09:33:44.148Z","status":"ssl_error","status_checked_at":"2026-02-03T09:33:43.343Z","response_time":96,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","feature-extraction","neural-network","pattern-recognition","transformers","video-processing"],"created_at":"2025-03-04T14:51:11.829Z","updated_at":"2026-02-03T09:41:46.314Z","avatar_url":"https://github.com/junayed-hasan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DeepMarkerNet: Leveraging Supervision from the Duchenne Marker for Spontaneous Smile Recognition \n\n(Paper Accepted for Publication in **Pattern Recognition Letters** on 26 September, 2024)\n\n\n\n[![Python 3.8](https://img.shields.io/badge/python-3.8-blue.svg)](https://www.python.org/downloads/release/python-380/)\n[![PyTorch](https://img.shields.io/badge/PyTorch-1.11.0-red.svg)](https://pytorch.org/)\n\n## Abstract\n\nDistinguishing between spontaneous and posed smiles from videos poses a significant challenge in pattern classification literature. This project introduces DeepMarkerNet, a novel deep learning solution that leverages Duchenne Marker (D-Marker) features to improve spontaneous smile recognition performance. Our multi-task learning framework integrates a transformer network with the utilization of facial D-Markers for accurate smile classification.\n\nUnlike past methods, our approach simultaneously predicts the class of the smile and associated facial D-Markers using two different feed-forward neural networks, creating a symbiotic relationship that enriches the learning process. The novelty lies in incorporating supervisory signals from pre-calculated D-Markers, harmonizing the loss functions through a weighted average. This allows our training to utilize the benefits of D-Markers without requiring their computation during inference.\n\n## Key Features\n\n- Multi-task learning framework combining smile classification and D-Marker prediction\n- Transformer network integration for improved feature extraction\n- Utilization of D-Marker features as supervisory signals\n- State-of-the-art performance on multiple smile datasets\n\n## Dependencies\n\n- Python 3.8\n- numpy 1.21.5\n- Pillow 9.0.1\n- dlib 19.24.0\n- opencv-python 4.6.0.66\n- torch 1.11.0\n- torchvision 0.12.0\n- vidaug 1.5\n- einops 0.6.0\n- tqdm 4.64.1\n- colorama 0.4.6\n\n## Datasets\n\nTo reproduce our results, obtain the following datasets:\n\n1. [UVA-NEMO database](https://www.uva-nemo.org)\n2. [MMI database](https://mmifacedb.eu)\n3. [BBC database](https://www.bbc.co.uk/science/humanbody/mind/surveys/smiles/)\n4. [SPOS database](https://www.oulu.fi/cmvs/node/41317)\n\nNote: Only landmarks data from UVA-NEMO are provided in this repository. For other datasets, you can extract landmarks using the code from [MediaPipe Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh.html).\n\n## Usage\n\n### Training DeepMarkerNet\n\nTo train the model on the UVA-NEMO dataset:\n\n```bash\npython smile_point.py --fold 0\n```\n\nThis command trains the model using fold 0 of the UVA-NEMO dataset. The trained model weights will be saved in the `uva/labels` folder.\n\n## Results\n\nOur model achieves state-of-the-art results on four well-known smile datasets: UvA-NEMO, BBC, MMI facial expression, and SPOS datasets. Detailed results and comparisons can be found in our paper (link to be added upon publication).\n\n## Citation\n```bash\n@article{HASAN2024,\ntitle = {DeepMarkerNet: Leveraging supervision from the Duchenne Marker for spontaneous smile recognition},\njournal = {Pattern Recognition Letters},\nyear = {2024},\nissn = {0167-8655},\ndoi = {https://doi.org/10.1016/j.patrec.2024.09.015},\nurl = {https://www.sciencedirect.com/science/article/pii/S0167865524002770},\nauthor = {Mohammad Junayed Hasan and Kazi Rafat and Fuad Rahman and Nabeel Mohammed and Shafin Rahman}\n}\n```\n\n## Contributing\n\nWe welcome contributions to improve DeepMarkerNet. Please feel free to submit issues and pull requests.\n\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Contact\n\nFor any queries regarding the code or paper, please open an issue or contact Mohammad Junayed Hasan (junayedhasan100@gmail.com).\n\n---\n\nCopyright © 2024 Mohammad Junayed Hasan. All rights reserved.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjunayed-hasan%2Fspontaneous-smile-recognition","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjunayed-hasan%2Fspontaneous-smile-recognition","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjunayed-hasan%2Fspontaneous-smile-recognition/lists"}