{"id":19125347,"url":"https://github.com/ryangawei/cnn-facial-expression-recognition","last_synced_at":"2025-05-05T20:14:52.986Z","repository":{"id":68816732,"uuid":"172022897","full_name":"ryangawei/CNN-Facial-Expression-Recognition","owner":"ryangawei","description":"Facial Expression Recognition (FER) based on VGG16","archived":false,"fork":false,"pushed_at":"2019-08-30T03:40:04.000Z","size":1504,"stargazers_count":18,"open_issues_count":0,"forks_count":7,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-05-05T20:14:47.901Z","etag":null,"topics":["computer-vision","convolutional-neural-networks","deep-learning","facial-expression-recognition","keras","keras-tensorflow","vgg16"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryangawei.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-22T08:11:23.000Z","updated_at":"2024-11-08T06:04:31.000Z","dependencies_parsed_at":"2023-09-14T07:47:37.212Z","dependency_job_id":null,"html_url":"https://github.com/ryangawei/CNN-Facial-Expression-Recognition","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryangawei%2FCNN-Facial-Expression-Recognition","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryangawei%2FCNN-Facial-Expression-Recognition/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryangawei%2FCNN-Facial-Expression-Recognition/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryangawei%2FCNN-Facial-Expression-Recognition/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryangawei","download_url":"https://codeload.github.com/ryangawei/CNN-Facial-Expression-Recognition/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252569648,"owners_count":21769517,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","convolutional-neural-networks","deep-learning","facial-expression-recognition","keras","keras-tensorflow","vgg16"],"created_at":"2024-11-09T05:35:30.576Z","updated_at":"2025-05-05T20:14:52.963Z","avatar_url":"https://github.com/ryangawei.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CNN Facial Expression Classification\nFacial Expression Rcognition(FER) based on deep convolutional neural network(CNN).\n\n## Dataset\nFER2013   [Challenges in Representation Learning: Facial Expression Recognition Challenge](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge)  \nDownload `fer2013.csv` and put it into `./data`\n## Model\nBased on VGG16, with GaussianNoise, and simpler fully-connected layers in the top. \n```python\nmain_input = layers.Input([config.img_size, config.img_size, 1])\n\nx = layers.BatchNormalization()(main_input)\nx = layers.GaussianNoise(0.01)(x)\n\nbase_model = VGG16(weights=None, input_tensor=x, include_top=False)\n\n# flatten = layers.GlobalAveragePooling2D()(base_model.output)\nflatten = Flatten()(base_model.output)\n\nfc = Dense(2048, activation='relu',\n           kernel_regularizer=l2(0.001),\n           bias_regularizer=l2(0.001),\n           )(flatten)\nfc = Dropout(dropout_rate)(fc)\nfc = Dense(2048, activation='relu',\n           kernel_regularizer=l2(0.001),\n           bias_regularizer=l2(0.001),\n           )(fc)\nfc = Dropout(dropout_rate)(fc)\n\npredictions = Dense(config.class_num, activation=\"softmax\")(fc)\n\nmodel = keras.Model(inputs=main_input, outputs=predictions, name='vgg16')\n\noptimizer = keras.optimizers.Adam(lr)\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['categorical_accuracy'])\nreturn model\n```\n## Preprocessing\nUsage: `python preprocess.py`  \nOrigin csv file is converted to images.  \nFacial part of the image is detected and extracted by Dlib and Opencv2.\nThen resize to original size(48*48).  \nSeveral pickle files contain each image path and label.\n```python\ndef crop_face_area(detector, landmark_predictor, image, img_size):\n    p_img = Image.fromarray(image).convert(mode='RGB')\n    cv_img = cv2.cvtColor(np.asarray(p_img), cv2.COLOR_RGB2GRAY)\n    faces = detector(cv_img, 1)\n    all_landmarks = []\n    all_faces = []\n    if len(faces) \u003e 0:\n        for face in faces:\n            shape = landmark_predictor(cv_img, face)\n            landmarks = np.ndarray(shape=[68, 2])\n            for i in range(68):\n                landmarks[i] = (shape.part(i).x, shape.part(i).y)\n            all_landmarks.append(landmarks)\n            x1, y1, x2, y2 = face.left(), face.top(), face.right(), face.bottom()\n            if x1 \u003c 0:\n                x1 = 0\n            if x1 \u003e cv_img.shape[1]:\n                x1 = cv_img.shape[1]\n            if x2 \u003c 0:\n                x2 = 0\n            if x2 \u003e cv_img.shape[1]:\n                x2 = cv_img.shape[1]\n            if y1 \u003c 0:\n                y1 = 0\n            if y1 \u003e cv_img.shape[0]:\n                y1 = cv_img.shape[0]\n            if y2 \u003c 0:\n                y2 = 0\n            if y2 \u003e cv_img.shape[0]:\n                y2 = cv_img.shape[0]\n            img = cv2.resize(cv_img[y1:y2, x1:x2], (img_size, img_size))\n            all_faces.append(img)\n        return np.asarray(all_faces), np.asarray(all_landmarks)\n    else:\n        return None, None\n```\nAfter running the script, you should get the following files.\n`shape_predictor_68_face_landmarks.dat` is downloaded from [davisking/dlib-models\n](https://github.com/davisking/dlib-models)\n```\n│  fer2013.csv\n│  shape_predictor_68_face_landmarks.dat\n│  test.pickle\n│  test_landmark.npz\n│  train.pickle\n│  train_landmark.npz\n│  valid.pickle\n│  valid_landmark.npz\n│  __init__.py\n├─test\n├─train\n└─valid\n```\n## Image Augmentation\n```python\ntrain_datagen = ImageDataGenerator(\n    samplewise_center=True,\n    samplewise_std_normalization=True,\n    brightness_range=(0.8, 1.2),\n    rotation_range=10,\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    zoom_range=0.1,\n    horizontal_flip=True,\n)\n```\n\n## Training\nTraining and validation is performed on FER2013's Train and PrivateTest samples.  \n\n* LR: 1e-4\n* Batch size: 128\n* Optimizer: Adam\n* Dropout rate: 0.5\n* Early stopping: monitor on validation loss with patience 6\n* Learning rate reduce: factor 0.1 with patience 4  \n\nUsage:\n```\npython train.py -h\n\nusage: train.py [-h] [-dropout DROPOUT_RATE] [-lr LEARNING_RATE]\n                [-batch_size BATCH_SIZE] [-model MODEL_NAME]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -dropout DROPOUT_RATE, --dropout_rate DROPOUT_RATE\n                        The dropout rate for the last dense layers.Default\n                        0.5.\n  -lr LEARNING_RATE, --learning_rate LEARNING_RATE\n                        Learning rate. Default 1e-3.\n  -batch_size BATCH_SIZE, --batch_size BATCH_SIZE\n                        Batch size. Default 128.\n  -model MODEL_NAME, --model_name MODEL_NAME\n                        The classification model. Default vgg16.\n```\n\n\n## Testing\nTest the performance on FER2013's PublicTest samples.\n* Test-time augmentation: 10\n\nUsage\n```\npython test.py -h\n\nusage: test.py [-h] [-tta TTA] [-batch_size BATCH_SIZE] [-model MODEL_NAME]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -tta TTA, --tta TTA   Test-time augmentation times. Default 5.\n  -batch_size BATCH_SIZE, --batch_size BATCH_SIZE\n                        Batch size. Default 128.\n  -model MODEL_NAME, --model_name MODEL_NAME\n                        The classification model. Default vgg16.\n```\n## Test on camera or images\nUsage:\n```\npython detect.py -h\n\nusage: detect.py [-h] [-tta TTA] [-model MODEL_NAME] [-cam CAMERA]\n                 [-path IMAGE_PATH]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -tta TTA, --tta TTA   Test-time augmentation times. Default 5.\n  -model MODEL_NAME, --model_name MODEL_NAME\n                        The classification model. Default vgg16.\n  -cam CAMERA, --camera CAMERA\n                        Whether to detect face using camera. Default False\n  -path IMAGE_PATH, --image_path IMAGE_PATH\n                        The path of the image. Only useful when -cam=false.\n```\n\n## Result\n![accuracy](data/acc.png)\n![loss](data/loss.png)\n![lr](data/lr.png)\n\n![cm](data/cm.jpg)\n```\nf1 score: 0.6667545793929432, acc: 0.6682750301568154, recall: 0.6682750301568154\n```\n\n## Example\n![angry_result](data/demo/angry_result.png)\n![sad_score](data/demo/sad_result.png)\n![surprise_result](data/demo/surprise_result.png)\n\n## Environment\n* python 3.6.5\n* tensorflow 1.13.1\n* keras 2.2.4\n* dlib \n* opencv\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryangawei%2Fcnn-facial-expression-recognition","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryangawei%2Fcnn-facial-expression-recognition","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryangawei%2Fcnn-facial-expression-recognition/lists"}