{"id":20435522,"url":"https://github.com/praveena2j/dynamic-crossattention","last_synced_at":"2025-10-16T04:21:46.207Z","repository":{"id":208506003,"uuid":"721804664","full_name":"praveena2j/Dynamic-CrossAttention","owner":"praveena2j","description":"IEEE ICME : \"Cross-Attention is not always needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition\"","archived":false,"fork":false,"pushed_at":"2024-12-05T15:56:07.000Z","size":2374,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-15T19:36:29.221Z","etag":null,"topics":["affective-computing","ai","attention","attention-mechanism","attention-model","audio-visual-learning","computer-vision","cross-attention","emotion-recognition","multimodal-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/praveena2j.png","metadata":{"files":{"readme":"ReadMe.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-21T20:01:51.000Z","updated_at":"2024-12-05T16:13:30.000Z","dependencies_parsed_at":"2023-11-21T21:27:22.477Z","dependency_job_id":"a377901b-47b9-455a-b6f4-83e197c32164","html_url":"https://github.com/praveena2j/Dynamic-CrossAttention","commit_stats":null,"previous_names":["praveena2j/dynamic-crossattention"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/praveena2j%2FDynamic-CrossAttention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/praveena2j%2FDynamic-CrossAttention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/praveena2j%2FDynamic-CrossAttention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/praveena2j%2FDynamic-CrossAttention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/praveena2j","download_url":"https://codeload.github.com/praveena2j/Dynamic-CrossAttention/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241982478,"owners_count":20052529,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["affective-computing","ai","attention","attention-mechanism","attention-model","audio-visual-learning","computer-vision","cross-attention","emotion-recognition","multimodal-learning"],"created_at":"2024-11-15T08:35:07.326Z","updated_at":"2025-10-16T04:21:41.159Z","avatar_url":"https://github.com/praveena2j.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Audio–Visual Fusion for Emotion Recognition in the Valence–Arousal Space Using Joint Cross-Attention\nCode for our paper \"Cross-Attention is not always needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition\" accepted to IEEE ICME 2024. Our paper can be found [here](https://ieeexplore.ieee.org/document/10687371).\n\n## Citation\n\nIf you find this code useful for your research, please cite our paper.\n\n```\n@INPROCEEDINGS{10095234,\n  author={Praveen, R. Gnana and Alam, Jahangir},\n  journal={2024 IEEE International Conference on Multimedia and Expo (ICME)}, \n  title={Cross-Attention is not always needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition}, \n  year={2024},\n}\n```\n\nThis code uses the Affwild2 dataset to validate the proposed approach for Dimensional Emotion Recognition. There are three major blocks in this repository to reproduce the results of our paper. This code uses Mixed Precision Training (torch.cuda.amp). The dependencies and packages required to reproduce the environment of this repository can be found in the `environment.yml` file. \n\n### Creating the environment\nCreate an environment using the `environment.yml` file\n\n`conda env create -f environment.yml`\n\n### Models\nThe pre-trained models of audio and visual backbones are obtained [here](https://github.com/kuhnkeF/ABAW2020TNT)\n\nThe fusion models trained using our fusion approach can be found [here]()\n\n```\ndcacam_model.pt:  Fusion model trained using our approach on the Affwild2 dataset\n```\n\n# Table of contents \u003ca name=\"Table_of_Content\"\u003e\u003c/a\u003e\n\n+ [Preprocessing](#DP) \n    + [Step One: Download the dataset](#PD)\n    + [Step Two: Preprocess the visual modality](#PV) \n    + [Step Three: Preprocess the audio modality](#PA)\n    + [Step Four: Preprocess the annotations](#PL)\n+ [Training](#Training) \n    + [Training the fusion model](#TE) \n+ [Inference](#R)\n    + [Generating the results](#GR)\n \n## Preprocessing \u003ca name=\"DP\"\u003e\u003c/a\u003e\n[Return to Table of Content](#Table_of_Content)\n\n### Step One: Download the dataset \u003ca name=\"PD\"\u003e\u003c/a\u003e\n[Return to Table of Content](#Table_of_Content)\nPlease download the following.\n  + The dataset for the valence-arousal track can be downloaded [here](https://ibug.doc.ic.ac.uk/resources/aff-wild2/)\n\n### Step Two: Preprocess the visual modality \u003ca name=\"PD\"\u003e\u003c/a\u003e\n[Return to Table of Content](#Table_of_Content)\n  + The cropped-aligned images are necessary. They are used to form the visual input. They are already provided by the dataset organizers. Otherwise, you may choose to use [OpenFace toolkit](https://github.com/TadasBaltrusaitis/OpenFace/releases) to extract the cropped-aligned images. However, the per-frame success rate is lower compared to the database-provided version.\n\n### Step Three: Preprocess the audio modality \u003ca name=\"PD\"\u003e\u003c/a\u003e\n[Return to Table of Content](#Table_of_Content)\n  + The audio files are extracted and segmented to generate the corresponding audio files in alignment with the visual files using [mkvextract](https://mkvtoolnix.download/). To generate these audio files, you can use the file Preprocessing/audio_preprocess.py. \n\n### Step Four: Preprocess the annotations \u003ca name=\"PD\"\u003e\u003c/a\u003e\n[Return to Table of Content](#Table_of_Content)\n  + The annotations provided by the dataset organizers are preprocessed to obtain the labels of aligned audio and visual files. To generate these audio files, you can use the file Preprocessing/preprocess_labels.py. \n\n## Training \u003ca name=\"DP\"\u003e\u003c/a\u003e\n[Return to Table of Content](#Table_of_Content)\n  + After obtaining the preprocessed audio and visual files along with annotations, we can train the model using the proposed fusion approach using the main.py script.\n\n## Inference \u003ca name=\"DP\"\u003e\u003c/a\u003e\n[Return to Table of Content](#Table_of_Content)\n  + The results of the proposed model can be reproduced using the trained model. In order to obtain the predictions on the test set using our proposed model, we can use the test.py.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpraveena2j%2Fdynamic-crossattention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpraveena2j%2Fdynamic-crossattention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpraveena2j%2Fdynamic-crossattention/lists"}