{"id":19070289,"url":"https://github.com/amogh7joshi/engagement-detection","last_synced_at":"2025-04-28T14:14:05.604Z","repository":{"id":138929764,"uuid":"314635860","full_name":"amogh7joshi/engagement-detection","owner":"amogh7joshi","description":"Engagement Detection, including facial detection and emotion recognition, using CNNs/LSTMs.","archived":false,"fork":false,"pushed_at":"2021-02-25T18:22:21.000Z","size":5238,"stargazers_count":26,"open_issues_count":1,"forks_count":8,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-28T14:14:00.144Z","etag":null,"topics":["cnn","computer-vision","emotion-recognition","engagement-prediction","facial-detection","fer2013","keras","lstm","neural-networks","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amogh7joshi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-20T18:20:36.000Z","updated_at":"2025-03-10T10:52:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"9c940d31-3484-4b5a-b115-6d466338419d","html_url":"https://github.com/amogh7joshi/engagement-detection","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amogh7joshi%2Fengagement-detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amogh7joshi%2Fengagement-detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amogh7joshi%2Fengagement-detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amogh7joshi%2Fengagement-detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amogh7joshi","download_url":"https://codeload.github.com/amogh7joshi/engagement-detection/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251326851,"owners_count":21571636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cnn","computer-vision","emotion-recognition","engagement-prediction","facial-detection","fer2013","keras","lstm","neural-networks","tensorflow"],"created_at":"2024-11-09T01:17:56.589Z","updated_at":"2025-04-28T14:14:05.579Z","avatar_url":"https://github.com/amogh7joshi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Engagement Detection\n\nThis repository contains the source code for neural networks used in facial detection, emotion recognition,\nand the overarching framework of engagement detection. \n\nCurrently, emotion detection has been implemented from the `fer2013` dataset, and can be used for image and live \nvideo classification, in the `videoclassification.py` and `emotionclassification.py` scripts. \nThe `facevideo.py` file contains live facial detection from the computer webcam. The comments\non the top of the file contain more information on usage of the different detectors. The `facedetect.py` also contains \nfacial detection, however it detects faces from inputted images rather than a live video feed.\n\nThe repository also contains multiple convolutional neural networks for facial emotion recognition.\nThey are still in progress, but the general usage is as follows: Train the model from the `trainmodel.py` file,\nand test the model using the `testmodel.py` file. \n\nFor information on the neural network models being used, see the `Neural Network Information` section below.\n\n**NOTE:** Before using anything in this repository, please visit the `data` directory and read the instructions\nthere on downloading any necessary files and the location of saved files.\n\n## Installation\n\nTo use the repository, it can be directly cloned from the command line:\n\n```shell script\ngit clone --recurse-submodules https://github.com/amogh7joshi/engagement-detection.git\n```\n\n### Setup\n\nFor setup, a Makefile is provided:\n\n```shell script\nmake install\n```\n\nOr, you can manually run:\n\n```shell script\n# Install System Requirements\npython3 -m pip install -r requirements.txt\n```\n\nIn either case, you should delete the `.cloud` directory. Either the Makefile will do it for you or \nyou can manually delete it. It contains operations that I use personally when working with the Google API,\nso unless you are working with any of the same Google APIs, you should delete it.\n\n### Data Acquisition\n\nThen, use the scripts provided in the `scripts` directory to install the necessary data:\n1. To install the model and caffemodel files for the DNN, use the `getdata.sh` script. \n2. Download the `fer2013.csv` file from [here](https://www.kaggle.com/deadskull7/fer2013), follow the directions in the `data`\n subdirectory.\n3. Optionally, also download the `ck+` dataset from [here](https://www.kaggle.com/shawon10/ckplus), and follow the directions\nin the `data` subdirectory.\n4. Run the `preprocess.sh` script. It may take a couple of minutes.\n\n### Dataset Usage\n\nOnce the datasets are preprocessed, they can be called through the following functions:\n\n```python\nfrom data.load_data import get_fer2013_data\nfrom data.load_data import get_ckplus_data\n\n# Load the training, validation, and testing data (repeat with other datasets).\nX_train, X_validation, X_test, y_train, y_validation, y_test = get_fer2013_data()\n```\n\nFor more information, visit the `data` subdirectory.\n\nThe other tools in the Makefile are for convenience purposes only when committing to this repository, \nin addition to the `editconstant.sh` script. Do not use them unless you are committing to your own repository.\n\nThe `info.json` file contains the relevant locations of the cascade classifiers and DNN model files.\nYou can replace the current locations with those on your computer, and then load the detectors as follows.\n\n```python\nfrom util.info import load_info\n\n# Set the `eyes` option to true if you want to load the eye cascade.\ncascade_face, cascade_eyes, net = load_info(eyes = True)\n```\n\n## Usage\n\n![GitHub Workflow Status](https://img.shields.io/github/workflow/status/amogh7joshi/chemsolve/CodeQL)\n\nCurrently, all models have been configured to work with the `fer2013` and `ck+` datasets.\n\n**Model Training**: Run the `trainmodel.py` script. You can edit the number of epochs in the argparse argument\nat the top of the file. Alternatively, you can run itt from the command line using the flags as mentioned by the \nargparse arguments. Model weights will be saved to the `data/model` directory, and at the completion of the training,\nthe best model will be moved into the `data/savedmodels` directory. The json file containing the model\narchitecture will also be saved there. You can control what models to keep in the `data/savedmodels` directory manually.\n\n**Model Testing**: Run the `testmodel.py` script. You can edit which model weights and architecture you want to use at the \nlocation at the top of the file. From there, you can run `model.evaluate` on the pre-loaded training and testing data, \nyou can run `model.predict` on any custom images you want to test, or run any other operations with the model. \nA confusion matrix is also present, which will display if `plt.show()` is uncommented.\n\n**Live Emotion Detection**: Run the `videoclassification.py` script. If you already have a trained model, set it at the top of the \nscript, and it will detect emotions live. For just facial detection, run the `facevideo.py` script. You can choose which detector you\nwant to use, as described at the top of the file. If you want to save images, set the `-s` flag to `True`, and they will save to a \ncustom directory `imageruntest` at the top-level. More information is included at the top of the file. \n\n**Image Emotion Detection**: Run the `emotionclassification.py` script. Choose the images you want to detect emotions on and place their paths in \nthe `userimages` variable. If running from the command line, then write out the paths to each of the images when running the script. Optionally, if you\njust want facial detection,  run the `facedetect.py` script. If running from the command line, then read the argument information at the top of the file. \nOtherwise, insert the paths of the images that you want to detect faces from into a list called `user_images` midway through the file. The changed images will save\nto a custom directory called `modded`, but you can change that from the `savedir` variable. For each image inputted, the script will output the same image\nwith a bounding box around the faces detected from the image.\n\n## Neural Network Information\n\n![Model Architectures](examples/images/model3.png)\n\nThe model architecture I am currently using (architecture is the model on the upper right) for the emotion recognition convolutional neural network uses a very basic inception architecture,\nwith each block containing a triple convolution and max pooling branch and an average pooling branch. \n\nPreviously I had used a model (architecture is on the upper left) roughly developed as a miniature version of the \nXception model [\\[1\\]](https://arxiv.org/abs/1610.02357). It contains three branches at each convolution layer: the first with two convolutions and max pooling,\nthe second with only convolution, and a third with one convolution and average pooling. \n\nInitially, I had chosen to use one similar to the likes of VGG16 and VGG19 \n[\\[2\\]](http://arxiv.org/abs/1409.1556), but chose against it due to issues which arose during training, and its\nlack of any residual connections. These models can be seen in the `models/pastmodels.py` file, specifically models 1-3.\n\nSince the model has a convolutional architecture, fully-connected layers have been replaced with a global average pooling layer. \nIn general, it yields better results. The 2-D convolution layers can also be replaced with separable 2-D convolution layers,\nalthough regular convolution layers seem to yield better results with the image sizes of the `fer2013` dataset.\n\nThe deep neural network for face detection makes use of a pre-trained model using the  ResNet architecture \n[\\[3\\]](http://arxiv.org/abs/1512.03385).\n\n## Data Pipelines\n\nThe directories in this repository are integrated for a seamless transition sequence. All necessary data\nincluding model architecture, weights, cascade classifiers, and other necessary files will be saved to necessary \nsubdirectories in the `data` directory. *Please visit the `data` directory for usage instructions.*\n\nFollowing a training sequence, models are saved to the `data/savedmodels` directory. They can then be loaded \neither through the architecture + weights files, or just from the weight files, as follows:\n\n```python\nfrom models.model_factory import *\n\n# Load from architecture + weights files.\nmodel = load_json_model('\u003cinsert-model-name\u003e', compile = 'default')\n# Load just from weights file.\nmodel = load_keras_model('\u003cinsert-model-name\u003e', compile = False)\n```\n\n## License and Contributions\n\n![GitHub](https://img.shields.io/github/license/amogh7joshi/engagement-detection)\n\nThe code in this repository is available under the [MIT License](https://github.com/amogh7joshi/fer/blob/master/LICENSE). Although you are welcome to download the \nrepository and work with it, contributions will not be accepted. However, if you notice an issue with the system, feel free to create an issue for me to take a look at. \n\n## References\n[1]: Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. \nArXiv:1610.02357 [Cs]. http://arxiv.org/abs/1610.02357\n\n[2]: Simonyan, K., and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. ArXiv:1409.1556 [Cs]. http://arxiv.org/abs/1409.1556\n\n[3]: He, K., Zhang, X., Ren, S., \u0026 Sun, J. (2015). Deep residual learning for image recognition. ArXiv:1512.03385 [Cs]. http://arxiv.org/abs/1512.03385","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famogh7joshi%2Fengagement-detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famogh7joshi%2Fengagement-detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famogh7joshi%2Fengagement-detection/lists"}