{"id":28677467,"url":"https://github.com/thunlp-mt/adamms","last_synced_at":"2026-02-20T19:31:23.931Z","repository":{"id":286608350,"uuid":"948360476","full_name":"THUNLP-MT/AdaMMS","owner":"THUNLP-MT","description":"Official Repository for \"AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization\" [CVPR2025]","archived":false,"fork":false,"pushed_at":"2025-04-07T12:06:27.000Z","size":2915,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-14T00:06:03.611Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-14T07:42:46.000Z","updated_at":"2025-05-30T05:06:32.000Z","dependencies_parsed_at":"2025-04-07T13:34:10.248Z","dependency_job_id":null,"html_url":"https://github.com/THUNLP-MT/AdaMMS","commit_stats":null,"previous_names":["thunlp-mt/adamms"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/THUNLP-MT/AdaMMS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FAdaMMS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FAdaMMS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FAdaMMS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FAdaMMS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/AdaMMS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FAdaMMS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29661594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-20T16:33:43.953Z","status":"ssl_error","status_checked_at":"2026-02-20T16:33:43.598Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-14T00:06:03.528Z","updated_at":"2026-02-20T19:31:23.761Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e🚀 AdaMMS: Adaptive Model Merging for Heterogeneous Multimodal LLMs\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/CVPR-2025-blue.svg\" alt=\"CVPR 2025\"/\u003e\n  \u003cimg src=\"https://img.shields.io/github/stars/THUNLP-MT/AdaMMS?style=social\" alt=\"GitHub stars\"/\u003e\n  \u003cimg src=\"https://img.shields.io/github/issues/THUNLP-MT/AdaMMS\" alt=\"Issues\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg\" alt=\"License\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e🔥 Accepted to CVPR 2025! \u003ca href=\"https://arxiv.org/abs/2503.23733\"\u003e[Paper]\u003c/a\u003e\u003c/p\u003e\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003e🌐 Language:\u003c/b\u003e\n  \u003ca href=\"https://github.com/THUNLP-MT/AdaMMS/blob/main/README.md\"\u003e🇺🇸 English\u003c/a\u003e | \n  \u003ca href=\"https://github.com/THUNLP-MT/AdaMMS/blob/main/README_CH.md\"\u003e🇨🇳 中文\u003c/a\u003e\n\u003c/p\u003e\n\n## Introduction\n\nRecent advancements in model merging have shown great potential in combining capabilities from multiple large language models (LLMs). However, existing methods primarily focus on merging **homogeneous models** with identical architectures, struggling when applied to **heterogeneous Multimodal Large Language Models (MLLMs)** that differ in both architecture and parameter space.\n\nWe propose **AdaMMS**: **Ada**ptive **M**apping, **M**erging, and **S**earching — a novel unsupervised model merging framework tailored for heterogeneous MLLMs. AdaMMS tackles the challenges in three steps:\n\n1. 🧠 **Mapping**  \n   Establish a mapping function between different model architectures.\n\n2. ⚖️ **Merging**  \n   Perform weighted linear interpolation to accommodate asymmetries in parameter space.\n\n3. 🔍 **Searching**  \n   Introduce an unsupervised hyperparameter search method to determine optimal merging coefficients.\n\n📊 Extensive experiments show that AdaMMS consistently outperforms previous model merging methods on various vision-language benchmarks.\n\nHere is the illustration of three steps in AdaMMS:\n\n\u003cimg src=\"assets/crop_head.jpg\" alt=\"!Result\" style=\"zoom: 33%;\" /\u003e\n\nHere is the average results from different mnerging methods:\n\n\u003cimg src=\"assets/radar_compare.jpg\" alt=\"!Result\" style=\"zoom: 33%;\" /\u003e\n\nThis is a visualization of the model outputs obtained with different alpha values：\n\n\u003cimg src=\"assets/crop_diff.jpg\" alt=\"Result\" style=\"zoom: 33%;\" /\u003e\n\n\n\n---\n\n## 🛠️ Environment Setup\n\n\u003e ⚠️ It's recommended to set up environments **individually for each model**, then install the `lmms-eval` evaluation framework.\n\u003e\n\u003e \n\n~~~markdown\n### ✅ Example: CogVLM\n\n```bash\nconda create -n lmms-cogvlm python=3.10\nconda activate lmms-cogvlm\n\nwget https://github.com/THUDM/CogVLM/blob/main/requirements.txt --no-check-certificate\npip install -r requirements.txt\npython -m spacy download en_core_web_sm\n\ngit clone https://github.com/EvolvingLMMs-Lab/lmms-eval\ncd lmms-eval \u0026\u0026 pip install -e .\n\nconda install openjdk=8\n\n########################\n### ✅ Example: mPLUG-Owl\nconda create -n lmms-mplug python=3.10\nconda activate lmms-mplug\n\ngit clone https://github.com/X-PLUG/mPLUG-Owl.git\ncd mPLUG-Owl/mPLUG-Owl2\npip install --upgrade pip \u0026\u0026 pip install -e .\n\ngit clone https://github.com/EvolvingLMMs-Lab/lmms-eval\ncd lmms-eval \u0026\u0026 pip install -e .\n\nconda install openjdk=8\npip install deepspeed  # Optional for inference acceleration\n~~~\n\n### \n\n------\n\n## 🔄 Merge Scripts\n\n\u003e Naming convention: `xxx2yyy.py` indicates merging model `xxx` into architecture `yyy`.\n\n### 📈 Linear Interpolation Scripts\n\n| Source Model         | Target Model | Script File            |\n| -------------------- | ------------ | ---------------------- |\n| LLaVA                | CogVLM       | `llava2cogvlm.py`      |\n| mPLUG-Owl            | CogVLM       | `mplugowl2cogvlm.py`   |\n| LLaVA-OneVision-Qwen | QwenVL2      | `llava-qwen2qwenvl.py` |\n\n### 🧬 Non-Linear Merging (Baseline)\n\n| Source Model         | Target Model | Script File                         |\n| -------------------- | ------------ | ----------------------------------- |\n| LLaVA                | CogVLM       | `llava2cogvlm_ties_merging.py`      |\n| mPLUG-Owl            | CogVLM       | `mplugowl2cogvlm_ties_merging.py`   |\n| LLaVA-OneVision-Qwen | QwenVL2      | `llava-qwen2qwenvl_ties_merging.py` |\n\n------\n\n## ⚙️ Merging + Inference\n\n\u003e 📝 Refer to `runs/` for example scripts. Logging results helps identify the best alpha. You can find more details for inference in https://github.com/EvolvingLMMs-Lab/lmms-eva .\n\n### 🧪 Run Merge Script\n\n```bash\nconda activate lmms-cogvlm\npython $MERGE_SCRIPT --output $ckpt_path --alpha $alpha \\\n       --base $BASE_MODEL_PATH --base_llava $LLAVA_PATH \\\n       --interpolation\n```\n\n### 🚀 Batch Evaluation for Multiple Alphas (0.4~1.0)\n\n```bash\n#!/bin/bash\n\nfor alpha in 1.0 0.9 0.8 0.7 0.6 0.5 0.4; do\n    echo \"===\u003e Alpha: $alpha\"\n    \n    # Merge\n    python3 $MERGE_SCRIPT --output $ckpt_path --alpha $alpha --interpolation \\\n        --base COGVLM_PATH --llava_base LLAVA_PATH\n\n    # Evaluate\n    for task in \"mme\" \"mmmu_val\" \"nocaps_val\" \"vizwiz_vqa_val\" \"seedbench\"  \"gqa\" \"ok_vqa\" \"refcoco_bbox_testA\" \"refcocog_bbox_test\" \"refcoco+_bbox_testA\" \"mmbench\" \"ocrbench\" ; do\n        CUDA_VISIBLE_DEVICES=$GPU accelerate launch \\\n            --num_processes=1 \\\n            -m lmms_eval \\\n            --model cogvlm \\\n            --model_args pretrained=$ckpt_path,... \\\n            --tasks $task \\\n            --log_samples \\\n            --output_path $output_path\n    done\n\n    rm -rf $ckpt_path\ndone\n```\n\n------\n\n## 🔍 Alpha Selection\n\nAfter evaluating different alphas, run the following script to auto-select the best one:\n\n```bash\npython search/view_log_delta_perdata_search_limit.py\n```\n\nThis will output the best `alpha` and its performance logs.\n\n------\n\n## 🧩 Merge Logic (Example: `llava2cogvlm.py`)\n\n### 1️⃣ Load Parameters\n\n- Check if parameter should be merged: `need_merge(key)`\n- Scale base model:\n\n```python\ncogvlm_diff[key] = (cogvlm_chat[key] * alpha)\n```\n\n### 2️⃣ Merge Parameters\n\n- **Linear**:\n\n```python\ncogvlm_diff['lm_head.weight'] += llava['lm_head.weight']\n```\n\n- **Non-linear**: Call `do_merging()` or `do_merging_strategy()` from `ties_merging.py`.\n\n### 3️⃣ Save Parameters\n\n- Compatible with both `torch` and `safetensors`.\n- For `safetensors`, metadata is required.\n\n------\n\n## 🤝 Contributions\n\nWe welcome PRs and issues! 🌟\n AdaMMS aims to improve the efficiency of heterogeneous multimodal model merging and support your research in MLLMs.\n\n------\n\n## 📄 Citation\n\nIf you find this project helpful, please cite:\n\n```bibtex\n@misc{du2025adamms,\n      title={AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization}, \n      author={Yiyang Du and Xiaochen Wang and Chi Chen and Jiabo Ye and Yiru Wang and Peng Li and Ming Yan and Ji Zhang and Fei Huang and Zhifang Sui and Maosong Sun and Yang Liu},\n      year={2025},\n      eprint={2503.23733},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2503.23733}, \n}\n```\n\n------\n\n## 🇨🇳 中文版说明\n\n👉 请点击此链接跳转 [中文版 README](https://github.com/THUNLP-MT/AdaMMS/blob/main/README_CH.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fadamms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Fadamms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fadamms/lists"}