{"id":18887134,"url":"https://github.com/thunlp-mt/modelcompose","last_synced_at":"2025-04-14T22:33:46.007Z","repository":{"id":226942296,"uuid":"759931175","full_name":"THUNLP-MT/ModelCompose","owner":"THUNLP-MT","description":"Official code for our paper \"Model Composition for Multimodal Large Language Models\" (ACL 2024)","archived":false,"fork":false,"pushed_at":"2025-01-08T09:26:28.000Z","size":3505,"stargazers_count":25,"open_issues_count":3,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T10:54:14.739Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-19T15:59:51.000Z","updated_at":"2025-03-20T06:33:12.000Z","dependencies_parsed_at":"2024-11-08T07:44:48.083Z","dependency_job_id":null,"html_url":"https://github.com/THUNLP-MT/ModelCompose","commit_stats":null,"previous_names":["thunlp-mt/modelcompose"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FModelCompose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FModelCompose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FModelCompose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FModelCompose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/ModelCompose/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248973018,"owners_count":21191898,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T07:34:38.875Z","updated_at":"2025-04-14T22:33:40.996Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Model Composition for Multimodal Large Language Models\n\n## Contents\n- [Install](#install)\n- [Preparation](#preparation)\n- [Train](#train)\n- [Evaluation](#evaluation)\n\n## Install\n\n1. Clone this repository and navigate to ModelCompose folder\n```bash\ngit clone https://github.com/THUNLP-MT/ModelCompose.git\ncd ModelCompose\n```\n\n2. Install Package\n```bash\nconda create -n modelcompose python=3.10 -y\nconda activate modelcompose\npip install -r requirements.txt\n```\n\n## Preparation\n\n1. Data\n\nBefore training or evaluation, please prepare the datasets based on your need. Json files can be downloaded from [Hugging Face](https://huggingface.co/datasets/Adu2021/ModelCompose/tree/main).\n\nYou can organize them under `./data` as follows:\n```\ndata\n├── test\n│   └── [json files]\n├── train\n│   └── [json files]\n├── evaluation_datasets\n├── audiocaps\n│   ├── train\n|   └── test\n├── clotho\n│   └── audio\n├── coco\n│   └── train2017\n├── gqa\n│   └── images\n├── ocr_vqa\n│   └── images\n├── textvqa\n│   └── train_images\n├── activitynet\n├── PointCloud\n│   └── 8192_npy\n├── WavCaps\n│   └── audios\n|       ├── AudioSet_SL_flac\n|       ├── BBC_Sound_Effects_flac\n|       ├── FreeSound_flac\n|       └── SoundBible_flac\n└── vg\n    ├── VG_100K\n    └── VG_100K_2\n```\n\n2. Base model\n\nWe use vicuna-7b-v1.5 as our base model. [Download](https://huggingface.co/lmsys/vicuna-7b-v1.5/tree/main)\n\n## Train\n\nWe apply a two-stage training paradigm. Find pretrain and finetune scripts under `./scripts/model_composition/train`. Please modify the following parameters in the scripts: `BASE_PATH`, `ROOT`, and `MODEL_BASE`.\n\nNote that we use Video-LLaVA-Pretrain-7B as pretrained checkpoint for text-video modalities. Download pretrained checkpoints for [Video-LLaVA-Pretrain-7B](https://huggingface.co/LanguageBind/Video-LLaVA-Pretrain-7B/tree/main) if needed.\n\nWe have released our trained checkpoints at [Hugging Face](https://huggingface.co/Adu2021/ModelCompose/tree/main).\n\n## Evaluation\n\n1. Merge checkpoints\n\nSeperately trained checkpoints should be merged before evaluation. Specify *parameter adjustment coefficient* in `--strategy` param starts with `online-merge-reset-default-`. Use *vision, video, audio, point* for each modality.\n\n```bash\npython scripts/model_composition/merge_unimodal_modelcompose.py \\\n                checkpoints/multimodal-vicuna-7b-v1.5-video-damc \\\n                checkpoints/multimodal-vicuna-7b-v1.5-audio-damc \\\n                checkpoints/multimodal-vicuna-7b-v1.5-vision-damc \\\n                -o checkpoints/multimodal-pdt-video-image-audio \\\n                --strategy online-merge-reset-default-video=0.333,default-audio=0.333,default-vision=0.333\n```\n\n2. Run evaluation\n\nNote that the basename of the checkpoint should contain \"multimodal\" to load correctly. Replace \"multimodal-checkpoint-name\" in the following command with your merged checkpoint. \n\n**AVQA**\n\n```bash\nbash scripts/model_composition/test/avqa.sh \\\n    0,1,2,3,4,5,6,7 \\\n    multimodal-checkpoint-name \\\n    [modal] \\\n    path/to/vicuna-7b-v1.5\n    \n```\n\nChoose [modal] from `[audio, image, video, image+audio, image+video, video+audio, video+image+audio]`.\n\nReplace \"0,1,2,3,4,5,6,7\" with actual available gpu.\n\n**MUSIC-AVQA**\n\n```bash\nbash scripts/model_composition/test/music_avqa_video+image+audio.sh \\\n    0,1,2,3,4,5,6,7 \\\n    multimodal-checkpoint-name \\\n    path/to/vicuna-7b-v1.5\n```\n\nFind scripts for other modalities combinations under `scripts/model_composition/test`.\n\n**MCUB**\n\n\n```bash\nbash scripts/model_composition/test/MCUB-4.sh \\\n    0,1,2,3,4,5,6,7 \\\n    multimodal-checkpoint-name \\\n    path/to/vicuna-7b-v1.5\n\nbash scripts/model_composition/test/MCUB-3.sh \\\n    0,1,2,3,4,5,6,7 \\\n    multimodal-checkpoint-name \\\n    [modal] \\\n    path/to/vicuna-7b-v1.5\n```\n\nChoose [modal] from `[image-audio-video, audio-video-pointcloud, image-audio-pointcloud, image-video-pointcloud]`.\n\n## Citation\n\nIf you find our work useful, please consider giving this repository a star and citing our paper.\n\n```\n@misc{chen2024model,\n      title={Model Composition for Multimodal Large Language Models}, \n      author={Chi Chen and Yiyang Du and Zheng Fang and Ziyue Wang and Fuwen Luo and Peng Li and Ming Yan and Ji Zhang and Fei Huang and Maosong Sun and Yang Liu},\n      year={2024},\n      eprint={2402.12750},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n## Acknowledgement\n\n[LLaVA](https://github.com/haotian-liu/LLaVA): the codebase we built upon, and it offers strong language \u0026 vision abilities.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fmodelcompose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Fmodelcompose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fmodelcompose/lists"}