{"id":20065492,"url":"https://github.com/cvi-szu/me-graphau","last_synced_at":"2025-10-15T16:17:32.732Z","repository":{"id":40776211,"uuid":"484957236","full_name":"CVI-SZU/ME-GraphAU","owner":"CVI-SZU","description":"[IJCAI 2022] Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition, Pytorch code","archived":false,"fork":false,"pushed_at":"2025-08-21T10:33:51.000Z","size":27085,"stargazers_count":186,"open_issues_count":8,"forks_count":40,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-10T07:38:54.094Z","etag":null,"topics":["facial-action-unit-detection","facial-action-units","graph-neural-network","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CVI-SZU.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-04-24T07:30:08.000Z","updated_at":"2025-09-08T00:27:31.000Z","dependencies_parsed_at":"2024-11-13T13:51:06.646Z","dependency_job_id":"bccb404b-2d36-4713-912c-5cb24b921577","html_url":"https://github.com/CVI-SZU/ME-GraphAU","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CVI-SZU/ME-GraphAU","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FME-GraphAU","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FME-GraphAU/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FME-GraphAU/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FME-GraphAU/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CVI-SZU","download_url":"https://codeload.github.com/CVI-SZU/ME-GraphAU/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FME-GraphAU/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279091683,"owners_count":26101557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-15T02:00:07.814Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["facial-action-unit-detection","facial-action-units","graph-neural-network","pytorch"],"created_at":"2024-11-13T13:50:57.055Z","updated_at":"2025-10-15T16:17:32.727Z","avatar_url":"https://github.com/CVI-SZU.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/learning-multi-dimensional-edge-feature-based/facial-action-unit-detection-on-bp4d)](https://paperswithcode.com/sota/facial-action-unit-detection-on-bp4d?p=learning-multi-dimensional-edge-feature-based)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/learning-multi-dimensional-edge-feature-based/facial-action-unit-detection-on-disfa)](https://paperswithcode.com/sota/facial-action-unit-detection-on-disfa?p=learning-multi-dimensional-edge-feature-based)\n\n\n📢 News\n=\n\n* **20/08/2025 — We released [AU-Canvas](https://github.com/awakening-ai/AUCanvas), a visualization tool that offers an intuitive UI for facial action unit (FAU) detection and enhanced visualization.**\n\n### Example running on a RTX 3090 GPU (Avg. FPS\u003e50):\n\u003ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\"\u003e\n\u003c!--   \u003ctr\u003e --\u003e\n\u003c!--       \u003ctd\u003e --\u003e\n          \u003cvideo src=\"https://github.com/user-attachments/assets/69737b41-f215-4b69-9c10-08ae26ba5dc3\" width=\"100%\" controls autoplay loop\u003e\u003c/video\u003e\n          \u003cvideo src=\"https://github.com/user-attachments/assets/7a672e51-5a0a-4730-b02e-e5015570ba1b\" width=\"100%\" controls autoplay loop\u003e\u003c/video\u003e\n\u003c/table\u003e\n\n\n* 12/11/2022 We released an [OpenGraphAU](OpenGraphAU) or [OpenGraphAU](https://github.com/lingjivoo/OpenGraphAU) version of our code and models trained on a large-scale hybrid dataset of over 2,000k images and 41 action unit categories. \n\nLearning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition\n=\nThis is an official release of the paper  \n\u003e \n\u003e **\"Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition\"**, \n\u003e **IJCAI-ECAI 2022**\n\u003e \n\u003e [[Paper]](https://arxiv.org/abs/2205.01782) [[Project]](https://www.chengluo.cc/projects/ME-AU/)\n\u003e \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"img/intro.png\" width=\"70%\" /\u003e\n\u003c/p\u003e\n\n\u003eThe main novelty of the proposed approach in comparison to pre-defined AU graphs and deep learned facial display-specific graphs are illustrated in this figure.\n\n\nhttps://user-images.githubusercontent.com/35754447/169745317-40f76ec9-4bfd-4206-8f1e-4ab4a9bf464d.mp4\n\n\n🔧 Requirements\n=\n- Python 3\n- PyTorch\n\n\n- Check the required python packages in `requirements.txt`.\n```\npip install -r requirements.txt\n```\n\nData and Data Prepareing Tools\n=\nThe Datasets we used:\n  * [BP4D](http://www.cs.binghamton.edu/~lijun/Research/3DFE/3DFE_Analysis.html)\n  * [DISFA](http://mohammadmahoor.com/disfa-contact-form/)\n\nWe provide tools for prepareing data in ```tool/```.\nAfter Downloading raw data files, you can use these tools to process them, aligning with our protocals.\nMore details have been described in [tool/README.md](tool/README.md).\n\n\n**Training with ImageNet pre-trained models**\n\nMake sure that you download the ImageNet pre-trained models to `checkpoints/` (or you alter the checkpoint path setting in `models/resnet.py` or `models/swin_transformer.py`)\n\nThe download links of pre-trained models are in `checkpoints/checkpoints.txt`\n\nThanks to the offical Pytorch and [Swin Transformer](https://github.com/microsoft/Swin-Transformer)\n\nTraining and Testing\n=\n- to train the first stage of our approach (ResNet-50) on BP4D Dataset, run:\n```\npython train_stage1.py --dataset BP4D --arc resnet50 --exp-name resnet50_first_stage -b 64 -lr 0.0001 --fold 1 \n```\n\n- to train the second stage of our approach (ResNet-50) on BP4D Dataset, run:\n```\npython train_stage2.py --dataset BP4D --arc resnet50 --exp-name resnet50_second_stage  --resume results/resnet50_first_stage/bs_64_seed_0_lr_0.0001/xxxx_fold1.pth --fold 1 --lam 0.05\n```\n\n- to train the first stage of our approach (Swin-B) on DISFA Dataset, run:\n```\npython train_stage1.py --dataset DISFA --arc swin_transformer_base --exp-name swin_transformer_base_first_stage -b 64 -lr 0.0001 --fold 2\n```\n\n- to train the second stage of our approach (Swin-B) on DISFA Dataset, run:\n```\npython train_stage2.py --dataset DISFA --arc swin_transformer_base --exp-name swin_transformer_base_second_stage  --resume results/swin_transformer_base_first_stage/bs_64_seed_0_lr_0.0001/xxxx_fold2.pth -b 64 -lr 0.000001 --fold 2 --lam 0.01 \n```\n\n- to test the performance on DISFA Dataset, run:\n```\npython test.py --dataset DISFA --arc swin_transformer_base --exp-name test_fold2  --resume results/swin_transformer_base_second_stage/bs_64_seed_0_lr_0.000001/xxxx_fold2.pth --fold 2\n```\n\n\n### Pretrained models\n\nBP4D\n|arch_type|GoogleDrive link| Average F1-score|\n| :--- | :---: |  :---: |\n|`Ours (ResNet-18)`| -| - |\n|`Ours (ResNet-50)`| [link](https://drive.google.com/file/d/1gYVHRjIj6ounxtTdXMje4WNHFMfCTVF9/view?usp=sharing) | 64.7 |\n|`Ours (ResNet-101)`| [link](https://drive.google.com/file/d/1i-ra0dtoEhwIep6goZ55PvEgwE3kecbl/view?usp=sharing) | 64.8 |\n|`Ours (Swin-Tiny)`| [link](https://drive.google.com/file/d/1BT4n7_5Wr6bGxHWVf3WrT7uBT0Zg9B5c/view?usp=sharing)| 65.6 |\n|`Ours (Swin-Small)`| [link](https://drive.google.com/file/d/1EiQd6q7x1bEO6JBLi3s2y5348EuVdP3L/view?usp=sharing) | 65.1 |\n|`Ours (Swin-Base)`|[link](https://drive.google.com/file/d/1Ti0auMA5o94toJfszuHoMlSlWUumm9L8/view?usp=sharing)| 65.5 |\n\nDISFA\n|arch_type|GoogleDrive link| Average F1-score|\n| :--- | :---: |  :---: |\n|`Ours (ResNet-18)`| -| - |\n|`Ours (ResNet-50)`| [link](https://drive.google.com/file/d/1V-imbmhg-OgcP2d9SETT5iswNtCA0f8_/view?usp=sharing) | 63.1 |\n|`Ours (ResNet-101)`| -| - |\n|`Ours (Swin-Tiny)`| -| - |\n|`Ours (Swin-Small)`| -| - |\n|`Ours (Swin-Base)`| [link](https://drive.google.com/file/d/1T44KPDaUhi4J_C-fWa6RxXNkY3yoDwIi/view?usp=sharing) | 62.4 |\n\nDownload these files (e.g. ```ME-GraphAU_swin_base_BP4D.zip```) and unzip them, each of which involves the checkpoints of three folds.\n\n\n📝 Main Results\n=\n**BP4D**\n\n|   Method  | AU1 | AU2 | AU4 | AU6 | AU7 | AU10 | AU12 | AU14 | AU15 | AU17 | AU23 | AU24 | Avg. |\n| :-------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|   EAC-Net  | 39.0 | 35.2 | 48.6 | 76.1 | 72.9 | 81.9 | 86.2 | 58.8 | 37.5 | 59.1 |  35.9 | 35.8 | 55.9 |\n|   JAA-Net  |  47.2 | 44.0 |54.9 |77.5 |74.6 |84.0 |86.9 |61.9 |43.6 |60.3 |42.7 |41.9 |60.0|\n|   LP-Net |  43.4  | 38.0  | 54.2  | 77.1  | 76.7  | 83.8  | 87.2  |63.3  |45.3  |60.5  |48.1  |54.2  |61.0|\n|   ARL | 45.8 |39.8 |55.1 |75.7 |77.2 |82.3 |86.6 |58.8 |47.6 |62.1 |47.4 |55.4 |61.1|\n|   SEV-Net | 58.2 |50.4 |58.3 |81.9 |73.9 |87.8 |87.5 |61.6 |52.6 |62.2 |44.6 |47.6 |63.9|\n|   FAUDT | 51.7 |49.3 |61.0 |77.8 |79.5 |82.9 |86.3 |67.6 |51.9 |63.0 |43.7 |56.3 |64.2 |\n|   SRERL | 46.9 |45.3 |55.6 |77.1 |78.4 |83.5 |87.6 |63.9 |52.2 |63.9  |47.1 |53.3 |62.9 |\n|   UGN-B | 54.2  |46.4  |56.8  |76.2  |76.7  |82.4  |86.1  |64.7  |51.2  |63.1  |48.5  |53.6  |63.3 |\n|   HMP-PS | 53.1 |46.1 |56.0 |76.5 |76.9 |82.1 |86.4 |64.8 |51.5 |63.0 |49.9 | 54.5  |63.4 |\n|   Ours (ResNet-50) | 53.7 |46.9 |59.0 |78.5 |80.0 |84.4 |87.8 |67.3 |52.5 |63.2 |50.6 |52.4 |64.7 |\n|   Ours (Swin-B) | 52.7 |44.3 |60.9 |79.9 |80.1| 85.3 |89.2| 69.4| 55.4| 64.4| 49.8 |55.1 |65.5|\n\n**DISFA**\n\n|   Method  | AU1 | AU2 | AU4 | AU6 | AU9 | AU12 | AU25 | AU26 | Avg. |\n| :-------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|   EAC-Net  |41.5 |26.4 |66.4 |50.7 |80.5 |89.3| 88.9 |15.6 |48.5 |\n|   JAA-Net  | 43.7 |46.2 |56.0 |41.4 |44.7 |69.6 |88.3 |58.4 |56.0|\n|   LP-Net |  29.9 |24.7 |72.7 |46.8 |49.6 |72.9 |93.8 |65.0 |56.9|\n|   ARL | 43.9 |42.1 |63.6 |41.8 |40.0 |76.2 |95.2| 66.8 |58.7|\n|   SEV-Net | 55.3 |53.1|61.5 |53.6 |38.2 |71.6 |95.7| 41.5 |58.8|\n|   FAUDT | 46.1 |48.6| 72.8 |56.7 |50.0 |72.1 |90.8 |55.4 |61.5 |\n|   SRERL | 45.7  |47.8  |59.6  |47.1  |45.6  |73.5  |84.3  |43.6  |55.9 |\n|   UGN-B |43.3  |48.1  |63.4  |49.5  |48.2  |72.9  |90.8  |59.0  |60.0 |\n|   HMP-PS | 38.0 |45.9 |65.2 |50.9 |50.8 |76.0 |93.3 |67.6 |61.0|\n|   Ours (ResNet-50) | 54.6 |47.1 |72.9 |54.0 |55.7 |76.7 |91.1 |53.0 |63.1|\n|   Ours (Swin-B) | 52.5 |45.7 |76.1 |51.8 |46.5 |76.1 |92.9 |57.6 |62.4|\n\n\n\n🎓 Citation\n=\nif the code or method help you in the research, please cite the following paper:\n```\n\n@inproceedings{luo2022learning,\n  title     = {Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition},\n  author    = {Luo, Cheng and Song, Siyang and Xie, Weicheng and Shen, Linlin and Gunes, Hatice},\n  booktitle = {Proceedings of the Thirty-First International Joint Conference on\n               Artificial Intelligence, {IJCAI-22}},\n  pages     = {1239--1246},\n  year      = {2022}\n}\n\n\n@article{song2022gratis,\n    title={Gratis: Deep learning graph representation with task-specific topology and multi-dimensional edge features},\n    author={Song, Siyang and Song, Yuxin and Luo, Cheng and Song, Zhiyuan and Kuzucu, Selim and Jia, Xi and Guo, Zhijiang and Xie, Weicheng and Shen, Linlin and Gunes, Hatice},\n    journal={arXiv preprint arXiv:2211.12482},\n    year={2022}\n}\n\n\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvi-szu%2Fme-graphau","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcvi-szu%2Fme-graphau","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvi-szu%2Fme-graphau/lists"}