{"id":18832954,"url":"https://github.com/declare-lab/mm-align","last_synced_at":"2025-04-14T04:31:48.973Z","repository":{"id":112025918,"uuid":"556342599","full_name":"declare-lab/MM-Align","owner":"declare-lab","description":"[EMNLP 2022] This repository contains the official implementation of the paper \"MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences\"","archived":false,"fork":false,"pushed_at":"2024-03-10T07:21:05.000Z","size":291,"stargazers_count":28,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-27T18:21:40.582Z","etag":null,"topics":["machine-learning","multimodal-deep-learning","multimodal-sentiment-analysis","natural-language-processing","optimal-transport"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/declare-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-10-23T16:30:19.000Z","updated_at":"2025-03-18T07:35:16.000Z","dependencies_parsed_at":"2024-03-10T08:27:48.192Z","dependency_job_id":"96d6d89f-25ff-452c-9274-f72b598c0d09","html_url":"https://github.com/declare-lab/MM-Align","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FMM-Align","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FMM-Align/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FMM-Align/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FMM-Align/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/declare-lab","download_url":"https://codeload.github.com/declare-lab/MM-Align/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248821747,"owners_count":21166950,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","multimodal-deep-learning","multimodal-sentiment-analysis","natural-language-processing","optimal-transport"],"created_at":"2024-11-08T01:59:39.561Z","updated_at":"2025-04-14T04:31:48.573Z","avatar_url":"https://github.com/declare-lab.png","language":"Python","readme":"# MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences\n\nThis repository contains the official implementation of the paper: [MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences](https://arxiv.org/pdf/2210.12798v1.pdf), published at EMNLP 2022.\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"assets/Problem_setting.png\" width=400/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\n\n## Setup\n### Conda Environemnt\n```bash\nconda env create -f environment.yml\nconda activate mmalign\npython -m spacy download en_core_web_sm\n```\n\n### CMU-MOSI and CMU-MOSEI\nPlease refer to [this repository](https://github.com/declare-lab/BBFN) to get the `.pkl` files that store the extracted features (by CMU-MMSDK with integrated COVAREP and P2FA) of the two datasets.\n\n### MELD dataset\nYou can download the processed dataset (`.pkl`) from [here](https://drive.google.com/file/d/1RjrYSMpXxg_6r_nUQaysaPyMsldLpMcb/view?usp=sharing).\nAlternatively, if you'd like to extract the features by yourself, you can download the raw dataset from [here](http://web.eecs.umich.edu/~mihalcea/downloads/MELD.Raw.tar.gz). Then you can extract the visual and audio features with [ResNet101](https://github.com/v-iashin/video_features) (FPS=25) and [Wave2Vec2.0](https://huggingface.co/docs/transformers/model_doc/wav2vec2). Additionally, you need to manually gather text and extracted feature vectors by their IDs and split them into `(train/dev/test).pkl` files.\n\nNext, split the processed dataset into complete/incomplete partitions using `scripts/split_dataset.py`\n```bash\npython split_dataset.py --data_path \u003cpath_to_pickle_files\u003e --seed \u003cseed\u003e --group_id \u003cgroup_id\u003e --complete_ratio \u003ccomplete_ratio\u003e --split \u003csplit\u003e\n```\nWe provide an example script `script/run_split.sh`, which automatically generates 5 different partitions for a given dataset under the seed 2020-2024.\n\n\n## Train and Test\n```bash\ncd src\npython main.py --dataset \u003cdataset_name\u003e --data_path \u003cpath_to_dataset\u003e --group_id \u003cgroup_to_experiment\u003e --modals \u003cmodality_pairs\u003e --save_name \u003cname_prefix\u003e\n```\n\nThe best test results are automatically saved under `results/\u003csave_name\u003e_\u003cmodality_pairs\u003e.tsv`\n\n## Citation\nPlease cite our paper if you find that useful for your research:\n```bibtex\n@inproceedings{han2022mmalign,\n  title={MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences},\n  author={Han, Wei and Chen, Hui and Kan Min-Yen and Poria, Soujanya},\n  booktitle={Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},\n  year={2022}\n}\n```\n\n## Contact \nShould you have any question, feel free to contact me through [henryhan88888@gmail.com](henryhan88888@gmail.com)\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fmm-align","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeclare-lab%2Fmm-align","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fmm-align/lists"}