Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/phamquiluan/residualmaskingnetwork
ICPR 2020: Facial Expression Recognition using Residual Masking Network
https://github.com/phamquiluan/residualmaskingnetwork
emotion-detection emotion-recognition facial-expression-recognition fer2013 pytorch residual-masking-network
Last synced: 5 days ago
JSON representation
ICPR 2020: Facial Expression Recognition using Residual Masking Network
- Host: GitHub
- URL: https://github.com/phamquiluan/residualmaskingnetwork
- Owner: phamquiluan
- License: mit
- Created: 2020-01-09T07:14:07.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2024-08-04T01:32:33.000Z (6 months ago)
- Last Synced: 2025-01-14T19:29:25.719Z (12 days ago)
- Topics: emotion-detection, emotion-recognition, facial-expression-recognition, fer2013, pytorch, residual-masking-network
- Language: Python
- Homepage: https://ieeexplore.ieee.org/document/9411919
- Size: 2.46 MB
- Stars: 465
- Watchers: 8
- Forks: 95
- Open Issues: 17
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Facial Expression Recognition using Residual Masking Network
The code for my undergraduate thesis.
[](https://pepy.tech/project/rmn)
[](https://pypi.org/project/rmn)
[](https://app.circleci.com/pipelines/github/phamquiluan/ResidualMaskingNetwork)
[](https://github.com/phamquiluan/ResidualMaskingNetwork/actions/workflows/python-package.yml)
[](https://github.com/phamquiluan/residualmaskingnetwork)
[](https://paperswithcode.com/sota/facial-expression-recognition-on-fer2013?p=facial-expression-recognition-using-residual)
# Inference:
[](https://colab.research.google.com/drive/1ZEbJ6rJuGZ6UzmOFE1XCzSgacTUJiN-H?usp=sharing)
1. Install from pip
```bash
pip install rmn
# or build from source
git clone [email protected]:phamquiluan/ResidualMaskingNetwork.git
cd ResidualMaskingNetwork
pip install -e .
```
2. Run demo in Python (with webcam available)
```python
from rmn import RMN
m = RMN()
m.video_demo()
```
3. Detect emotions from an image
```python
image = cv2.imread("some-image-path.png")
results = m.detect_emotion_for_single_frame(image)
print(results)
image = m.draw(image, results)
cv2.imwrite("output.png", image)
```
### Table of Contents
- Recent Update
- Benchmarking on FER2013
- Benchmarking on ImageNet
- Installation
- Download datasets
- Training on FER2013
- Training on ImageNet
- Evaluation results
- Download dissertation and slide
## Recent Update
- [07/03/2023] Re-structure, update Readme
- [05/05/2021] Release ver 2, add colab
- [27/02/2021] Add paper
- [14/01/2021] Packaging Project and publish `rmn` on Pypi
- [27/02/2020] Update Tensorboard visualizations and Overleaf source
- [22/02/2020] Test-time augmentation implementation.
- [21/02/2020] Imagenet training code and trained weights released.
- [21/02/2020] Imagenet evaluation results released.
- [10/01/2020] Checking demo stuff and training procedure works on another machine
- [09/01/2020] First time upload
## Benchmarking on FER2013
We benchmark our code thoroughly on two datasets: FER2013 and VEMO. Below are the results and trained weights:
| Model | Accuracy |
| --------------------------------------------------------------------------------------------------------- | -------- |
| [VGG19](https://drive.google.com/file/d/196qnnULQpFK5O5Md_YfvsUC2OQRd5LrQ/view?usp=sharing) | 70.80 |
| [EfficientNet_b2b](https://drive.google.com/file/d/1ZqvGcqYZXXbMKFwSaoHlo6LaKVYptXxx/view?usp=sharing) | 70.80 |
| [Googlenet](https://drive.google.com/file/d/1FNiNS7qqkd4dPQX9APiFsEyylb-ZHN8f/view?usp=sharing) | 71.97 |
| [Resnet34](https://drive.google.com/file/d/1GFSWnjA2tvsVel3kz0EyYW_el7Nyu8Gc/view?usp=sharing) | 72.42 |
| [Inception_v3](https://drive.google.com/file/d/10LpGJEmSTikidqaiv32rcJYQc3Cx_n6-/view?usp=sharing) | 72.72 |
| [Bam_Resnet50](https://drive.google.com/file/d/1RlVX2eiaQrUUmpng1PCyhK_hJdfJRcoU/view?usp=sharing) | 73.14 |
| [Densenet121](https://drive.google.com/file/d/1VWZ96Ar_n_OQD4Rk5ThZbixApMQ5eeYW/view?usp=sharing) | 73.16 |
| [Resnet152](https://drive.google.com/file/d/1A-kUqrP9u3ZCXVBtE6AdUN5afPwQjTHl/view?usp=sharing) | 73.22 |
| [Cbam_Resnet50](https://drive.google.com/file/d/1_u_B2NcxqCMPdWmPLMYKVzAenwBlwoFi/view?usp=sharing) | 73.39 |
| [ResMaskingNet](https://drive.google.com/file/d/1dmOycqZACsLh0eyDSR2ssn4g9xh36zMp/view?usp=sharing) | 74.14 |
| [**ResMaskingNet + 6**](https://drive.google.com/drive/folders/1Nt7y1T99HpmF93peYxMg-i6BUqdzDBve?usp=sharing) | **76.82** |
Results in VEMO dataset could be found in my thesis or slide (attached below)
## Benchmarking on ImageNet
We also benchmark our model on ImageNet dataset.
| Model | Top-1 Accuracy | Top-5 Accuracy |
| -------------------------------------------------------------------------------------------- | -------------- | -------------- |
| [Resnet34](https://drive.google.com/open?id=16lErBAk7K3WswKP0wyE9S0dNrr7AF6wd) | 72.59 | 90.92 |
| [CBAM Resnet34](https://drive.google.com/open?id=16lErBAk7K3WswKP0wyE9S0dNrr7AF6wd) | 73.77 | 91.72 |
| [**ResidualMaskingNetwork**](https://drive.google.com/open?id=1myjp4_XL8mNJlAbz0TFjYKUc7B0N64eb) | **74.16** | **91.91** |
## Installation
- Install [PyTorch](http://pytorch.org/) by selecting your environment on the website and running the appropriate command.
- Clone this repository and install package [prerequisites](#prerequisites) below.
- Then download the dataset by following the [instructions](#datasets) below.
## Datasets
- [FER2013 Dataset](https://drive.google.com/drive/folders/1Nt7y1T99HpmF93peYxMg-i6BUqdzDBve?usp=sharing) (locate it in `saved/data/fer2013` like `saved/data/fer2013/train.csv`)
- [ImageNet 1K Dataset](http://image-net.org/download-images) (ensure it can be loaded by torchvision.datasets.Imagenet)
## Training on FER2013
[](https://colab.research.google.com/drive/1IEQ091jBeJrOKHJe4wNhodH-bUGbLHSE?usp=sharing)
- To train the networks, you need to specify the model name and other hyperparameters in the config file (located at configs/\*) then ensure it is loaded in main file, then run training procedure by simply run main file, for example:
```Shell
python main_fer.py # Example for fer2013_config.json file
```
- The best checkpoints will chosen at term of best validation accuracy, located at `saved/checkpoints`
- The TensorBoard training logs are located at `saved/logs`, to open it, use `tensorboard --logdir saved/logs/`
- By default, it will train `alexnet` model, you can switch to another model by edit `configs/fer2013\_config.json` file (to `resnet18` or `cbam\_resnet50` or my network `resmasking\_dropout1`.
## Training on the Imagenet dataset
To perform training resnet34 on 4 V100 GPUs on a single machine:
```Shell
python ./main_imagenet.py -a resnet34 --dist-url 'tcp://127.0.0.1:12345' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0
```
## Evaluation
For students, who should take care of the font family of the confusion matrix and would like to write things in LaTeX, below is an example for generating a striking confusion matrix.
(Read [this article](https://matplotlib.org/3.1.1/tutorials/text/usetex.html) for more information, there will be some bugs if you blindly run the code without reading).
```Shell
python cm_cbam.py
```
## Ensemble method
I used the no-weighted sum average ensemble method to fuse 7 different models together, to reproduce results, you need to do some steps:
1. Download all needed trained weights and locate them on the `./saved/checkpoints/` directory. The link to download can be found in the Benchmarking section.
2. Edit file `gen_results` and run it to generate result offline for **each** model.
3. Run the `gen_ensemble.py` file to generate accuracy for example methods.
## Dissertation and Slide
- [Dissertation PDF (in Vietnamese)](https://drive.google.com/drive/folders/1Nt7y1T99HpmF93peYxMg-i6BUqdzDBve?usp=sharing)
- [Dissertation Overleaf Source](https://www.overleaf.com/read/qdyhnzjmbscd)
- [Presentation slide PDF (in English) with full appendix](https://drive.google.com/drive/folders/1Nt7y1T99HpmF93peYxMg-i6BUqdzDBve?usp=sharing)
- [Presentation slide Overleaf Source](https://www.overleaf.com/read/vxdhjvhvgwdn)
- [ICPR Paper](docs/paper.pdf)
- [ICPR Poster Overleaf Source](https://www.overleaf.com/read/jjqwfrsdcdwh#566470)
## Authors
- [**Luan Pham**](https://github.com/phamquiluan)
- [**Tuan Anh Tran**](https://github.com/phamquiluan)
## Citation
Pham Luan, The Huynh Vu, and Tuan Anh Tran. "Facial Expression Recognition using Residual Masking Network". In: Proc. ICPR. 2020.
```
@inproceedings{pham2021facial,
title={Facial expression recognition using residual masking network},
author={Pham, Luan and Vu, The Huynh and Tran, Tuan Anh},
booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
pages={4513--4519},
year={2021},
organization={IEEE}
}
```
## Star History
[](https://star-history.com/#phamquiluan/residualmaskingnetwork&Date)