{"id":19186824,"url":"https://github.com/donydchen/causal_emotion","last_synced_at":"2025-08-13T11:44:23.629Z","repository":{"id":47079106,"uuid":"515823672","full_name":"donydchen/causal_emotion","owner":"donydchen","description":"☯︎[ACMMM'22] Official PyTorch Implementation of Towards Unbiased Visual Emotion Recognition via Causal Intervention","archived":false,"fork":false,"pushed_at":"2022-07-20T04:43:01.000Z","size":220,"stargazers_count":19,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-20T05:32:33.216Z","etag":null,"topics":["acmmm2022","causal-inference","causality","confounding","emotion-recognition","facial-expression-recognition","pytorch"],"latest_commit_sha":null,"homepage":"https://github.com/donydchen/causal_emotion","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/donydchen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-20T03:45:35.000Z","updated_at":"2024-10-18T19:56:46.000Z","dependencies_parsed_at":"2022-09-17T05:41:00.320Z","dependency_job_id":null,"html_url":"https://github.com/donydchen/causal_emotion","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donydchen%2Fcausal_emotion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donydchen%2Fcausal_emotion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donydchen%2Fcausal_emotion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donydchen%2Fcausal_emotion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/donydchen","download_url":"https://codeload.github.com/donydchen/causal_emotion/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252979971,"owners_count":21835140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acmmm2022","causal-inference","causality","confounding","emotion-recognition","facial-expression-recognition","pytorch"],"created_at":"2024-11-09T11:16:50.056Z","updated_at":"2025-05-08T01:24:50.922Z","avatar_url":"https://github.com/donydchen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Towards Unbiased Visual Emotion Recognition via Causal Intervention\n\nOfficial PyTorch implementation of [ACM Multimedia 2022] *Towards Unbiased Visual Emotion Recognition via Causal Intervention* by [Yuedong Chen](https://donydchen.github.io/), [Xu Yang](https://scholar.google.com.sg/citations?user=SqdxMH0AAAAJ), [Tat-Jen Cham](https://personal.ntu.edu.sg/astjcham/) and [Jianfei Cai](https://jianfei-cai.github.io/).\n\n\u003ca href=\"https://arxiv.org/abs/2107.12096\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2107.12096-b31b1b.svg\" height=22.5\u003e\u003c/a\u003e \n\u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" height=22.5\u003e\u003c/a\u003e \n\n## Abstract\n\nAlthough much progress has been made in visual emotion recognition, researchers have realized that modern deep networks tend to exploit dataset characteristics to learn spurious statistical associations between the input and the target. Such dataset characteristics are usually treated as dataset bias, which damages the robustness and generalization performance of these recognition systems. In this work, we scrutinize this problem from the perspective of causal inference, where such dataset characteristic is termed as a **confounder** which misleads the system to learn the spurious correlation. To alleviate the negative effects brought by the dataset bias, we propose a novel **Interventional Emotion Recognition Network (IERN)** to achieve the backdoor adjustment, which is one fundamental deconfounding technique in causal inference. Specifically, IERN starts by disentangling the dataset-related context feature from the actual emotion feature, where the former forms the confounder. The emotion feature will then be forced to see each confounder stratum equally before being fed into the classifier. A series of designed tests validate the efficacy of IERN, and experiments on three emotion benchmarks demonstrate that IERN outperforms state-of-the-art approaches for unbiased visual emotion recognition.\n\n\u003cimg src=\"docs/archi.png\" width=\"1024\"\u003e\n\n----\n\n## Installation\n\n### Setup Environment\n\nWe recommend to use [Anaconda](https://www.anaconda.com) to create the running environment for the project, and all related dependencies are provided in `environment/iern.yml`, kindly run\n\n```bash\ngit clone https://github.com/donydchen/causal_emotion.git\ncd causal_emotion\nconda env create -f environment/iern.yml\nconda activate iern\n```\n\n**Note**: The above environment contains *PyTorch 1.4 with CUDA 10*, if it does not work on your machine, please refer to [environment/README.md](https://github.com/donydchen/causal_emotion/blob/main/environment/README.md) for manual installation and trouble shootings.\n\n### Download Pretrained Weights\n\nDownload the pretrained models from [here](https://drive.google.com/drive/folders/1TCb6oYzLSVKHs040K09y-BcE-ER9VOuU), and save them to `pretrained_models/`.\n\n\n## Dataset Preparation\n\nNOTE: All dataset preprocessing scripts are mainly *for reference* only. Generally they should work out fine, but you are expected to quickly go through them, and uncomment some functions (if needed) to complete the preprocessings step-by-step.\n\n* Download the CK+, MMI and OuluCASIA datasets, and store them to `datasets/`.\n* Download the face and landmark detector from [here](https://drive.google.com/drive/folders/1IjJTmrEMgVN1ei9weku1p1o6UeT1klc0), and save them to `datasets/face_process/`.\n* Check out the scripts `ckplus.py, mmi.py, oulucasia.py` in `dataset/face_process` for how to extract and align the face. \n* Refer to `combine_labels.py` in `dataset/face_process` for combining all labels and dumping them into one file.\n* That's it, you are good to go. Your `datasets` should have the following structures at the end.\n\n```bash\ndatasets/\n    |__ face_process/\n    |__ MixBasic/\n        |__ CKPlus/\n            |__ images/\n        |__ MMI/\n            |__ images/\n        |__ OuluCASIA/\n            |__ images/\n        |__ emotion_labels.pkl\n        |__ test_ids_bias90v1.csv\n        |__ test_ids_bias90v2.csv\n        |__ ...\n        |__ train_ids_bias100v3.csv\n```\n\n\n## Experiments\n\nNote that we design a *three-folds* cross validation setting for the Mix-Dataset experiments. Training and testing configurations are provided in `datasets/*.csv`, kindly refer to *Section 4.1 Dataset settings* in the paper for more details.\n\nBesides, `bias100` refers to the designed *o.o.d* setting (results reported in Table 2 in the paper), while `bias90` refers to the modified setting (results reported in Table 3 in the paper). `v1` refers to Fold1, and so on.\n\n### Inference\n\nTo test IERN with the pretrained_models downloaded above, kindly run the following command.\n\n```bash\npython test.py \\\n--dataroot=datasets/MixBasic \\\n--checkpoints_dir=pretrained_models/mixbasic/fold_bias100v1 \\\n--epoch=160 \\\n--test_conf_name=test_ids_bias100v1.csv\n```\n\nYou can also change the checkpoints path and configuration path to test on other folds.\n\n### Training\n\nTo train IERN, use the following command,\n\n```bash\npython train.py \\\n--dataroot=datasets/MixBasic \\\n--batch_size=8 \\\n--train_conf_name=train_ids_bias100v1.csv \\\n--test_conf_name=test_ids_bias100v1.csv\n```\n\nYou can also tune some training options, so as to get better performance, *e.g.*, `lr_policy, n_epochs_warmup, lr_policy_after, n_epochs, n_epochs_decay`, *etc*. Kindly run `python train.py --help` to check out more tuning options.\n\n## Misc\n\n### Citations\n\nIf you use this project for your research, please cite our paper.\n\n```bibtex\n@article{chen2021towards,\n  title={Towards Unbiased Visual Emotion Recognition via Causal Intervention},\n  author={Chen, Yuedong and Yang, Xu and Cham, Tat-Jen and Cai, Jianfei},\n  journal={arXiv preprint arXiv:2107.12096},\n  year={2021}\n}\n```\n\n### Acknowledgments\n\nOur implementation borrowed the code structure provided by [pytorch-CycleGAN-and-pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix), we would like to thank them for releasing such a great project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonydchen%2Fcausal_emotion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdonydchen%2Fcausal_emotion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonydchen%2Fcausal_emotion/lists"}