{"id":20096576,"url":"https://github.com/hendrycks/emodiversity","last_synced_at":"2025-07-08T04:34:53.506Z","repository":{"id":45742696,"uuid":"501733202","full_name":"hendrycks/emodiversity","owner":"hendrycks","description":"Wellbeing and Emotion Prediction (NeurIPS 2022)","archived":false,"fork":false,"pushed_at":"2022-10-19T01:46:15.000Z","size":83,"stargazers_count":9,"open_issues_count":1,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-09T08:51:17.815Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hendrycks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-09T16:40:39.000Z","updated_at":"2025-02-21T15:55:09.000Z","dependencies_parsed_at":"2023-01-20T05:31:28.145Z","dependency_job_id":null,"html_url":"https://github.com/hendrycks/emodiversity","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Femodiversity","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Femodiversity/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Femodiversity/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Femodiversity/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hendrycks","download_url":"https://codeload.github.com/hendrycks/emodiversity/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252629071,"owners_count":21779137,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T16:59:15.766Z","updated_at":"2025-05-06T05:31:42.718Z","avatar_url":"https://github.com/hendrycks.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Emodiversity\n\nThis is the repository for \"[How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios](https://arxiv.org/abs/2210.10039)\" by [Mantas Mazeika*](https://www.linkedin.com/in/mmazeika/), [Eric Tang*](https://andyzoujm.github.io/), [Andy Zou](https://andyzoujm.github.io/), [Steven Basart](https://stevenbas.art/resources/resume/), [Jun Shern Chan](https://junshern.notion.site/junshern/Chan-Jun-Shern-304311c71e9d4376a27175ed6f876db4), [Dawn Song](https://people.eecs.berkeley.edu/~dawnsong/), [David Forsyth](http://luthuli.cs.uiuc.edu/~daf/), [Jacob Steinhardt](https://www.stat.berkeley.edu/~jsteinhardt/), and [Dan Hendrycks*](https://danhendrycks.com/).\n\nThe Video Cognitive Empathy (VCE) and Video to Valence (V2V) datasets [are available here](https://drive.google.com/drive/folders/1sRKitbXpLZ4pwXTONjiA-X0Y1z4I2o4X?usp=sharing). \n\nThis repository contains submodules. To clone the full repository along with submodules (required for reproducing training/results), please use\n```\ngit clone --recurse-submodules https://github.com/hendrycks/emodiversity.git\n```\n\n## Video Cognitive Empathy (VCE) dataset\nThe VCE dataset contains 61,046 videos, each labelled for the intensity of 27 emotions. The dataset is split into\n1. `train`: 50,000 videos\n2. `test`: 11,046 videos\n\nThe dataset is structured as follows:\n\n```bash\nvce_dataset/\n├── metadata.json\n├── train_labels.json\n├── test_labels.json\n├── videos/00000.mp4\n├── videos/00001.mp4\n...\n```\nwhere\n- `videos/` contains MP4 video files from both the train and test splits \n- `train_labels.json` and `test_labels.json` contain emotion labels corresponding to the videos. For example:\n```\n{\n    \"00000\": {\n        \"emotions\": { # This is the average rated \"intensity\" score (rated 0-1) for each emotion\n            \"Admiration\": 0.4416666666666667,\n            \"Adoration\": 0.23333333333333334,\n            \"Aesthetic Appreciation\": 0.0,\n            \"Amusement\": 0.06666666666666667,\n            \"Anger\": 0.0,\n            \"Anxiety\": 0.06666666666666667,\n            \"Awe (or Wonder)\": 0.225,\n            \"Awkwardness\": 0.0,\n            \"Boredom\": 0.0,\n            \"Calmness\": 0.0,\n            \"Confusion\": 0.0,\n            \"Craving\": 0.0,\n            \"Disgust\": 0.0,\n            \"Empathic Pain\": 0.0,\n            \"Entrancement\": 0.0,\n            \"Excitement\": 0.13333333333333333,\n            \"Fear\": 0.0,\n            \"Horror\": 0.0,\n            \"Interest\": 0.125,\n            \"Joy\": 0.08333333333333334,\n            \"Nostalgia\": 0.0,\n            \"Relief\": 0.0,\n            \"Romance\": 0.0,\n            \"Sadness\": 0.0,\n            \"Satisfaction\": 0.14166666666666666,\n            \"Sexual Desire\": 0.0,\n            \"Surprise\": 0.20833333333333334\n        },\n        \"file\": \"videos/00000.mp4\",\n        \"topK\": [ # The 3 highest-intensity emotions, from most intense to least intense\n            \"Admiration\",\n            \"Adoration\",\n            \"Awe (or Wonder)\"\n        ]\n    },\n    \"00001\": {\n    ...\n}\n```\n- `metadata.json` contains helpful metadata for all videos, e.g.\n```\n{\n    \"00000\": {\n        \"codec_name\": \"h264\",\n        \"duration\": \"12.000000\",\n        \"file\": \"videos/00000.mp4\",\n        \"frame_rate\": \"30/1\",\n        \"height\": 320,\n        \"number_of_frames\": \"360\",\n        \"width\": 256\n    },\n    \"00001\": {\n    ...\n}\n```\n\n\n## Video to Valence (V2V) Dataset\nThe V2V dataset consists preference annotations over a total of 26,670 videos, split into:\n- `train`: 11,038 pairwise comparisons on 16,125 unique videos\n- `test`: 3,189 pairwise comparisons on 6,223 unique videos\n- `listwise`: 1,758 listwise ranked comparisons on 4,322 unique videos (up to 4 videos per list)\n\nNote that `train`, `test`, `listwise` videos are mutually exclusive.\n\nThe dataset has the form:\n```bash\nv2v_dataset/\n├── metadata.json\n├── train_labels.json\n├── test_labels.json\n├── listwise_labels.json\n├── videos/00000.mp4\n├── videos/00001.mp4\n...\n```\nwhere\n- `videos/` contains 26,670 MP4 video files (this is a subset of the VCE dataset videos that have V2V labels)\n- `metadata.json` contains helpful metadata for all videos (this is the same as the metadata file in the VCE dataset)\n- `train_labels.json` and `test_labels.json` each contain a list of pairwise video comparisons, where the second video is more preferred:\n```\n{\n    \"comparisons\": [\n        [\"52694\", \"15036\"],\n        [\"49134\", \"56215\"],\n        [\"34304\", \"31620\"],\n        ...\n    ]\n}\n```\n- `listwise_labels.json` contains a list of listwise video comparisons (ordered from least-preferred to most-preferred):\n```\n{\n    \"comparisons\": [\n        [\"50986\", \"05956\", \"42507\"],\n        [\"52542\", \"52733\", \"53157\", \"50334\"],\n        [\"38647\", \"11277\", \"53157\", \"50334\"],\n        [\"56616\", \"33536\", \"39234\"],\n        ...\n    ]\n}\n```\n\n\n## Reproducibility\n\n### VideoMAE\n\nTo reproduce our results with the VideoMAE models, make sure you have [our fork of the VideoMAE repository](https://github.com/JunShern/VideoMAE) in this repository as a submodule. If `Emodiversity/VideoMAE` does not exist, you can run the following command from the root of this repository to get it:\n```\ngit submodule update --init\n```\n\n#### Pretrained models\nWe finetune our models on top of the VideoMAE models pretrained on the [Kinetics-400](https://www.deepmind.com/open-source/kinetics) dataset. Download the pretrained model \"Kinetics-400, ViT-B, Epoch 1600, Pre-train checkpoint\" from [this page](https://github.com/JunShern/VideoMAE/blob/main/MODEL_ZOO.md) and save it to `emodiversity/VideoMAE/models/kinetics400-ViTB-1600-16x5x3-pretrain.pth`.\n\nIf you have [gdown](https://github.com/wkentaro/gdown) installed, this command does the above for you:\n```\ngdown 1tEhLyskjb755TJ65ptsrafUG2llSwQE1 --output emodiversity/VideoMAE/models/kinetics400-ViTB-1600-16x5x3-pretrain.pth\n```\n\n\n#### Fine-tuning models\nTo finetune the VideoMAE models on our dataset, update the relevant filepaths in the `VideoMae/scripts/finetune_vce.sh` and `VideoMae/scripts/finetune_v2v.sh` scripts to match your system, and run them:\n```\nbash VideoMAE/scripts/emodiversity/finetune_vce.sh\n```\nand\n```\nbash VideoMAE/scripts/emodiversity/finetune_v2v.sh\n```\n\nIn practice, we use sbatch scripts with a SLURM cluster to train our models. If you would like to replicate this, please refer to `finetune_vce.sbatch` and `finetune_v2v.sbatch`.\n\n## Citation\n\nIf you find this useful in your research, please consider citing\n\n    @article{hendrycks2022wellbeing,\n      title={How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios},\n      author={Mantas Mazeika and Eric Tang and Andy Zou and Steven Basart and Jun Shern Chan and Dawn Song and David Forsyth and Jacob Steinhardt and Dan Hendrycks},\n      journal={NeurIPS},\n      year={2022}\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhendrycks%2Femodiversity","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhendrycks%2Femodiversity","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhendrycks%2Femodiversity/lists"}