{"id":13430864,"url":"https://github.com/PKU-YuanGroup/MoE-LLaVA","last_synced_at":"2025-03-16T06:31:38.035Z","repository":{"id":219408615,"uuid":"731413021","full_name":"PKU-YuanGroup/MoE-LLaVA","owner":"PKU-YuanGroup","description":"Mixture-of-Experts for Large Vision-Language Models","archived":false,"fork":false,"pushed_at":"2024-05-15T14:50:56.000Z","size":17352,"stargazers_count":1937,"open_issues_count":58,"forks_count":123,"subscribers_count":24,"default_branch":"main","last_synced_at":"2024-10-10T17:41:33.054Z","etag":null,"topics":["large-vision-language-model","mixture-of-experts","moe","multi-modal"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2401.15947","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PKU-YuanGroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-14T02:59:50.000Z","updated_at":"2024-10-08T09:40:36.000Z","dependencies_parsed_at":"2024-03-16T03:33:19.770Z","dependency_job_id":"0768b1aa-7ec7-4a6c-96f2-d9b341f58c11","html_url":"https://github.com/PKU-YuanGroup/MoE-LLaVA","commit_stats":null,"previous_names":["pku-yuangroup/moe-llava"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-YuanGroup%2FMoE-LLaVA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-YuanGroup%2FMoE-LLaVA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-YuanGroup%2FMoE-LLaVA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-YuanGroup%2FMoE-LLaVA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PKU-YuanGroup","download_url":"https://codeload.github.com/PKU-YuanGroup/MoE-LLaVA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221656454,"owners_count":16858774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["large-vision-language-model","mixture-of-experts","moe","multi-modal"],"created_at":"2024-07-31T02:00:58.527Z","updated_at":"2025-03-16T06:31:38.028Z","avatar_url":"https://github.com/PKU-YuanGroup.png","language":"Python","funding_links":[],"categories":["4 Reasoning Techniques","Python","Mixture of Experts (Sparse MoE)","其他_机器视觉","Repos","\u003cspan id=\"visual\"\u003eVLM (Visual)\u003c/span\u003e"],"sub_categories":["4.4 Mixture of Experts (MoE)","网络服务_其他","\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e"],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://s11.ax1x.com/2023/12/28/piqvDMV.png\" width=\"250\" style=\"margin-bottom: 0.2;\"/\u003e\n\u003cp\u003e\n\u003ch2 align=\"center\"\u003e \u003ca href=\"https://arxiv.org/abs/2401.15947\"\u003eMoE-LLaVA: Mixture of Experts for Large Vision-Language Models\u003c/a\u003e\u003c/h2\u003e\n\u003ch5 align=\"center\"\u003e If you like our project, please give us a star ⭐ on GitHub for latest update.  \u003c/h2\u003e\n\n\u003ch5 align=\"center\"\u003e\n    \n\n\n[![hf_space](https://img.shields.io/badge/🤗-Open%20In%20Spaces-blue.svg)](https://huggingface.co/spaces/LanguageBind/MoE-LLaVA)\n[![Replicate demo and cloud API](https://replicate.com/camenduru/moe-llava/badge)](https://replicate.com/camenduru/moe-llava)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/camenduru/MoE-LLaVA-jupyter/blob/main/MoE_LLaVA_jupyter.ipynb)\n[![hf_space](https://img.shields.io/badge/🤗-Paper%20In%20HF-red.svg)](https://huggingface.co/papers/2401.15947)\n[![arXiv](https://img.shields.io/badge/Arxiv-2401.15947-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2401.15947) \n[![youtube](https://img.shields.io/badge/-YouTube-000000?logo=youtube\u0026logoColor=FF0000)](https://www.youtube.com/watch?v=uYb38g-weEY)\n[![jiqizhixin](https://img.shields.io/badge/-WeChat@机器之心-000000?logo=wechat\u0026logoColor=07C160)](https://mp.weixin.qq.com/s/ICylR6n2LhqQRS0CAHFI1A)\n[![License](https://img.shields.io/badge/License-Apache%202.0-yellow)](https://github.com/PKU-YuanGroup/MoE-LLaVA/blob/main/LICENSE) \n[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FPKU-YuanGroup%2FMoE-LLaVA\u0026count_bg=%2379C83D\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=Visitor\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n[![GitHub issues](https://img.shields.io/github/issues/PKU-YuanGroup/MoE-LLaVA?color=critical\u0026label=Issues)](https://github.com/PKU-YuanGroup/MoE-LLaVA/issues?q=is%3Aopen+is%3Aissue)\n[![GitHub closed issues](https://img.shields.io/github/issues-closed/PKU-YuanGroup/MoE-LLaVA?color=success\u0026label=Issues)](https://github.com/PKU-YuanGroup/MoE-LLaVA/issues?q=is%3Aissue+is%3Aclosed)  \u003cbr\u003e\n\u003c/h5\u003e\n\n\u003cdetails open\u003e\u003csummary\u003e💡 I also have other vision-language projects that may interest you ✨. \u003c/summary\u003e\u003cp\u003e\n\u003c!--  may --\u003e\n\n\u003e [**Open-Sora Plan: Open-Source Large Video Generation Model**](https://arxiv.org/abs/2412.00131) \u003cbr\u003e\n\u003e Bin Lin and Yunyang Ge and Xinhua Cheng and Zongjian Li and Bin Zhu and Shaodong Wang and Xianyi He and Yang Ye and Shenghai Yuan and Liuhan Chen and Tanghui Jia and Junwu Zhang and Zhenyu Tang and Yatian Pang and Bin She and Cen Yan and Zhiheng Hu and Xiaoyi Dong and Lin Chen and Zhang Pan and Xing Zhou and Shaoling Dong and Yonghong Tian and Li Yuan \u003cbr\u003e\n[![github](https://img.shields.io/badge/-Github-black?logo=github)](https://github.com/PKU-YuanGroup/Open-Sora-Plan)  [![github](https://img.shields.io/github/stars/PKU-YuanGroup/Open-Sora-Plan.svg?style=social)](https://github.com/PKU-YuanGroup/Open-Sora-Plan) [![arXiv](https://img.shields.io/badge/Arxiv-2412.00131-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2412.00131) \u003cbr\u003e\n\n\u003e [**Video-LLaVA: Learning United Visual Representation by Alignment Before Projection**](https://arxiv.org/abs/2311.10122) \u003cbr\u003e\n\u003e Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, Li Yuan \u003cbr\u003e\n[![github](https://img.shields.io/badge/-Github-black?logo=github)](https://github.com/PKU-YuanGroup/Video-LLaVA)  [![github](https://img.shields.io/github/stars/PKU-YuanGroup/Video-LLaVA.svg?style=social)](https://github.com/PKU-YuanGroup/Video-LLaVA) [![arXiv](https://img.shields.io/badge/Arxiv-2311.10122-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2311.10122) \u003cbr\u003e\n\n\u003e [**LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment**](https://arxiv.org/abs/2310.01852) \u003cbr\u003e\n\u003e Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, Li Yuan \u003cbr\u003e\n[![github](https://img.shields.io/badge/-Github-black?logo=github)](https://github.com/PKU-YuanGroup/LanguageBind)  [![github](https://img.shields.io/github/stars/PKU-YuanGroup/LanguageBind.svg?style=social)](https://github.com/PKU-YuanGroup/LanguageBind)  [![arXiv](https://img.shields.io/badge/Arxiv-2310.01852-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2310.01852) \u003cbr\u003e\n\n\u003c/p\u003e\u003c/details\u003e\n\n\n## 📣 News\n* **[2024.03.16]**  🎉 We release all stage2 models, cheching our [model zoo](#-model-zoo).\n* **[2024.02.03]**  🎉 We release a stronger [MoE-LLaVA-StableLM](https://huggingface.co/LanguageBind/MoE-LLaVA-StableLM-1.8B-4e-384). The average performance is close to LLaVA-1.5-7B by using **2.0B** sparse activated parameters, checking our [model zoo](#-model-zoo).\n* **[2024.02.02]**  🤝 Enjoying the [![Replicate demo and cloud API](https://replicate.com/camenduru/moe-llava/badge)](https://replicate.com/camenduru/moe-llava) and [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/camenduru/MoE-LLaVA-jupyter/blob/main/MoE_LLaVA_jupyter.ipynb), created by [@camenduru](https://github.com/camenduru), who generously supports our research!\n* **[2024.02.01]**  🔥 People who cannot access HF can now download the model through the \u003cimg src=\"https://github.com/PKU-YuanGroup/MoE-LLaVA/raw/main/assets/modelscope_logo.png\" width=\"20px\" style=\"max-width: 100%;\"\u003e model scope, checking our [model zoo](#-model-zoo).\n* **[2024.01.30]**  🔥 We release a stronger [MoE-LLaVA-Phi2](https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-2.7B-4e-384). The average performance **surpasses LLaVA-1.5-7B by using 3.6B** sparse activated parameters, checking our [model zoo](#-model-zoo).\n* **[2024.01.27]**  🤗 [Hugging Face demo](https://huggingface.co/spaces/LanguageBind/MoE-LLaVA) and **all codes \u0026 datasets** are available now! Welcome to **watch** 👀 this repository for the latest updates.\n\n## 😮 Highlights\n\nMoE-LLaVA shows excellent performance in multi-modal learning.\n\n### 🔥 High performance, but with fewer parameters\n- with just **3B sparsely activated parameters**, MoE-LLaVA demonstrates performance comparable to the LLaVA-1.5-7B on various visual understanding datasets and even surpasses the LLaVA-1.5-13B in object hallucination benchmarks.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/intro0.jpg\" width=55%\u003e\n\u003c/p\u003e\n\n### 🚀 Simple baseline, learning multi-modal interactions with sparse pathways.\n- With the addition of **a simple MoE tuning stage**, we can complete the training of MoE-LLaVA on **8 A100 GPUs** within 1 days.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/intro.jpg\" width=65%\u003e\n\u003c/p\u003e\n\n## 🤗 Demo\n\n### Gradio Web UI  \u003ca href='https://github.com/gradio-app/gradio'\u003e\u003cimg src='https://img.shields.io/github/stars/gradio-app/gradio'\u003e\u003c/a\u003e \n\nHighly recommend trying out our web demo by the following command, which incorporates all features currently supported by MoE-LLaVA. We also provide [online demo](https://huggingface.co/spaces/LanguageBind/MoE-LLaVA) in Huggingface Spaces.\n```bash\n# use phi2\ndeepspeed --include localhost:0 moellava/serve/gradio_web_server.py --model-path \"LanguageBind/MoE-LLaVA-Phi2-2.7B-4e\" \n# use qwen\ndeepspeed --include localhost:0 moellava/serve/gradio_web_server.py --model-path \"LanguageBind/MoE-LLaVA-Qwen-1.8B-4e\" \n# use stablelm\ndeepspeed --include localhost:0 moellava/serve/gradio_web_server.py --model-path \"LanguageBind/MoE-LLaVA-StableLM-1.6B-4e\" \n```\n\n\n\nhttps://github.com/PKU-YuanGroup/MoE-LLaVA/assets/62638829/8541aac6-9ef6-4fde-aa94-80d0375b9bdb\n\n\n\n### CLI Inference\n\n```bash\n# use phi2\ndeepspeed --include localhost:0 moellava/serve/cli.py --model-path \"LanguageBind/MoE-LLaVA-Phi2-2.7B-4e\"  --image-file \"image.jpg\"\n# use qwen\ndeepspeed --include localhost:0 moellava/serve/cli.py --model-path \"LanguageBind/MoE-LLaVA-Qwen-1.8B-4e\"  --image-file \"image.jpg\"\n# use stablelm\ndeepspeed --include localhost:0 moellava/serve/cli.py --model-path \"LanguageBind/MoE-LLaVA-StableLM-1.6B-4e\"  --image-file \"image.jpg\"\n```\n\n\u003cimg src=\"assets/imagecli.gif\" /\u003e\n\n## 🐳 Model Zoo\n\n| Model | Activated Param | Transformers(HF) | ModelScope(HF) | Avg | VQAv2 | GQA | VizWiz | SQA-IMG | T-VQA | POPE | MME | MM-Bench | MM-Vet |\n|----------|-----------|-----------|---|---|---|---|---|---|---|---|---|---|---|\n| MoE-LLaVA-1.6B×4-Top2 | 2.0B | [🤗LanguageBind/MoE-LLaVA-StableLM-1.6B-4e](https://huggingface.co/LanguageBind/MoE-LLaVA-StableLM-1.6B-4e) | [\u003cimg src=\"https://github.com/PKU-YuanGroup/MoE-LLaVA/raw/main/assets/modelscope_logo.png\" width=\"20px\" style=\"max-width: 100%;\"\u003ePKU-YuanLab/MoE-LLaVA-StableLM-1.6B-4e](https://modelscope.cn/models/PKU-YuanLab/MoE-LLaVA-StableLM-1.6B-4e) | 57.3 | 76.7 | 60.3 | 36.2 | 62.6 | 50.1 | 85.7 | 1318.1 | 60.2 | 26.9 |\n| MoE-LLaVA-1.8B×4-Top2 | 2.2B | [🤗LanguageBind/MoE-LLaVA-Qwen-1.8B-4e](https://huggingface.co/LanguageBind/MoE-LLaVA-Qwen-1.8B-4e) | [\u003cimg src=\"https://github.com/PKU-YuanGroup/MoE-LLaVA/raw/main/assets/modelscope_logo.png\" width=\"20px\" style=\"max-width: 100%;\"\u003ePKU-YuanLab/MoE-LLaVA-Qwen-1.8B-4e](https://modelscope.cn/models/PKU-YuanLab/MoE-LLaVA-Qwen-1.8B-4e) | 56.7 | 76.2 | 61.5 | 32.6 | 63.1 | 48.0 | 87.0 | 1291.6 | 59.6 | 25.3 |\n| MoE-LLaVA-2.7B×4-Top2 | 3.6B | [🤗LanguageBind/MoE-LLaVA-Phi2-2.7B-4e](https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-2.7B-4e) | [\u003cimg src=\"https://github.com/PKU-YuanGroup/MoE-LLaVA/raw/main/assets/modelscope_logo.png\" width=\"20px\" style=\"max-width: 100%;\"\u003ePKU-YuanLab/MoE-LLaVA-Phi2-2.7B-4e](https://modelscope.cn/models/PKU-YuanLab/MoE-LLaVA-Phi2-2.7B-4e) | 61.1 | 77.6 | 61.4 | 43.9 | 68.5 | 51.4 | 86.3 | 1423.0 | 65.2 | 34.3 |\n| MoE-LLaVA-1.6B×4-Top2-384 | 2.0B | [🤗LanguageBind/MoE-LLaVA-StableLM-1.6B-4e-384](https://huggingface.co/LanguageBind/MoE-LLaVA-StableLM-1.6B-4e-384) | [\u003cimg src=\"https://github.com/PKU-YuanGroup/MoE-LLaVA/raw/main/assets/modelscope_logo.png\" width=\"20px\" style=\"max-width: 100%;\"\u003ePKU-YuanLab/MoE-LLaVA-StableLM-1.6B-4e-384](https://modelscope.cn/models/PKU-YuanLab/MoE-LLaVA-StableLM-1.6B-4e-384) | 60.0 | 78.6 | 61.5 | 40.5 | 63.9 | 54.3 | 85.9 | 1335.7 | 63.3 | 32.3 |\n| MoE-LLaVA-2.7B×4-Top2-384 | 3.6B | [🤗LanguageBind/MoE-LLaVA-Phi2-2.7B-4e-384](https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-2.7B-4e-384) | [\u003cimg src=\"https://github.com/PKU-YuanGroup/MoE-LLaVA/raw/main/assets/modelscope_logo.png\" width=\"20px\" style=\"max-width: 100%;\"\u003ePKU-YuanLab/MoE-LLaVA-Phi2-2.7B-4e-384](https://modelscope.cn/models/PKU-YuanLab/MoE-LLaVA-Phi2-2.7B-4e-384) | **62.9** | 79.9 | 62.6 | 43.7 | 70.3 | 57.0 | 85.7 | 1431.3 | 68.0 | 35.9 |\n| LLaVA-1.5 | 7B | [🤗liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b) | - | 62.0 | 78.5 | 62.0 | 50.0 | 66.8 | 58.2 | 85.9 | 1510.7 | 64.3 | 30.5 |\n\n\u003c!--\n| LLaVA-1.5 | 13B | [liuhaotian/llava-v1.5-13b](https://huggingface.co/liuhaotian/llava-v1.5-13b) | 64.9 | 80.0 | 63.3 | 53.6 | 71.6 | 61.3 | 85.9 | 1531.3 | 67.7 | 35.4 |\n--\u003e\n\n\u003cdetails\u003e\n\n\n🚨 **Please know https://github.com/PKU-YuanGroup/MoE-LLaVA/issues/27.**\n\n\n\u003csummary\u003eStage2 Model\u003c/summary\u003e\n\n\n    \n| Model  | Checkpoint |\n|----------|-----------|\n| MoE-LLaVA-1.6B×4-Top2 | [LanguageBind/MoE-LLaVA-StableLM-Stage2](https://huggingface.co/LanguageBind/MoE-LLaVA-StableLM-Stage2) |\n| MoE-LLaVA-1.6B×4-Top2-384 | [LanguageBind/MoE-LLaVA-StableLM-Stage2-384](https://huggingface.co/LanguageBind/MoE-LLaVA-StableLM-Stage2-384) |\n| MoE-LLaVA-1.8B×4-Top2 | [LanguageBind/MoE-LLaVA-Qwen-Stage2](https://huggingface.co/LanguageBind/MoE-LLaVA-Qwen-Stage2) |\n| MoE-LLaVA-2.7B×4-Top2 | [LanguageBind/MoE-LLaVA-Phi2-Stage2](https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-Stage2) |\n| MoE-LLaVA-2.7B×4-Top2-384 | [LanguageBind/MoE-LLaVA-Phi2-Stage2-384](https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-Stage2-384) |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003ePretrain Model\u003c/summary\u003e\n\n| Model  | Checkpoint |\n|----------|-----------|\n| MoE-LLaVA-1.6B×4-Top2 | [LanguageBind/MoE-LLaVA-StableLM-Pretrain](https://huggingface.co/LanguageBind/MoE-LLaVA-StableLM-Pretrain) |\n| MoE-LLaVA-1.6B×4-Top2-384 | [LanguageBind/MoE-LLaVA-StableLM-384-Pretrain](https://huggingface.co/LanguageBind/MoE-LLaVA-StableLM-384-Pretrain) |\n| MoE-LLaVA-1.8B×4-Top2 | [LanguageBind/MoE-LLaVA-Qwen-Pretrain](https://huggingface.co/LanguageBind/MoE-LLaVA-Qwen-Pretrain) |\n| MoE-LLaVA-2.7B×4-Top2 | [LanguageBind/MoE-LLaVA-Phi2-Pretrain](https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-Pretrain) |\n| MoE-LLaVA-2.7B×4-Top2-384 | [LanguageBind/MoE-LLaVA-Phi2-384-Pretrain](https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-384-Pretrain) |\n\n\n\u003c/details\u003e\n\n## ⚙️ Requirements and Installation\nWe recommend the requirements as follows.\n* Python == 3.10\n* Pytorch == 2.0.1\n* CUDA Version \u003e= 11.7\n* **Transformers == 4.37.0**\n* **Tokenizers==0.15.1**\n* Install required packages:\n```bash\ngit clone https://github.com/PKU-YuanGroup/MoE-LLaVA\ncd MoE-LLaVA\nconda create -n moellava python=3.10 -y\nconda activate moellava\npip install --upgrade pip  # enable PEP 660 support\npip install -e .\npip install -e \".[train]\"\npip install flash-attn --no-build-isolation\n\n# Below are optional. For Qwen model.\ngit clone https://github.com/Dao-AILab/flash-attention\ncd flash-attention \u0026\u0026 pip install .\n# Below are optional. Installing them might be slow.\n# pip install csrc/layer_norm\n# If the version of flash-attn is higher than 2.1.1, the following is not needed.\n# pip install csrc/rotary\n```\n\n\u003e [!Warning]\n\u003e \u003cdiv align=\"left\"\u003e\n\u003e \u003cb\u003e\n\u003e 🚨 We find that using flash attention2 makes performance degradation.\n\u003e \u003c/b\u003e\n\u003e \u003c/div\u003e\n\n## 🗝️ Training \u0026 Validating\nThe training \u0026 validating instruction is in [TRAIN.md](docs/TRAIN.md) \u0026 [EVAL.md](docs/EVAL.md).\n\n## 💡 Customizing your MoE-LLaVA\nThe instruction is in [CUSTOM.md](docs/CUSTOM.md).\n\n## 😍 Visualization\nThe instruction is in [VISUALIZATION.md](docs/VISUALIZATION.md).\n\n## 🤖 API\n**We open source all codes.** If you want to load the model (e.g. ```LanguageBind/MoE-LLaVA-Phi2-2.7B-4e```) on local, you can use the following code snippets.\n\n**Using the following command to run the code.**\n\n```bash\ndeepspeed --include localhost:0 predict.py\n```\n\n```python\nimport torch\nfrom PIL import Image\nfrom moellava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN\nfrom moellava.conversation import conv_templates, SeparatorStyle\nfrom moellava.model.builder import load_pretrained_model\nfrom moellava.utils import disable_torch_init\nfrom moellava.mm_utils import tokenizer_image_token, get_model_name_from_path, KeywordsStoppingCriteria\n\ndef main():\n    disable_torch_init()\n    image = 'moellava/serve/examples/extreme_ironing.jpg'\n    inp = 'What is unusual about this image?'\n    model_path = 'LanguageBind/MoE-LLaVA-Phi2-2.7B-4e'  # LanguageBind/MoE-LLaVA-Qwen-1.8B-4e or LanguageBind/MoE-LLaVA-StableLM-1.6B-4e\n    device = 'cuda'\n    load_4bit, load_8bit = False, False  # FIXME: Deepspeed support 4bit or 8bit?\n    model_name = get_model_name_from_path(model_path)\n    tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, load_8bit, load_4bit, device=device)\n    image_processor = processor['image']\n    conv_mode = \"phi\"  # qwen or stablelm\n    conv = conv_templates[conv_mode].copy()\n    roles = conv.roles\n    image_tensor = image_processor.preprocess(Image.open(image).convert('RGB'), return_tensors='pt')['pixel_values'].to(model.device, dtype=torch.float16)\n\n    print(f\"{roles[1]}: {inp}\")\n    inp = DEFAULT_IMAGE_TOKEN + '\\n' + inp\n    conv.append_message(conv.roles[0], inp)\n    conv.append_message(conv.roles[1], None)\n    prompt = conv.get_prompt()\n    input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).cuda()\n    stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2\n    keywords = [stop_str]\n    stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)\n\n    with torch.inference_mode():\n        output_ids = model.generate(\n            input_ids,\n            images=image_tensor,\n            do_sample=True,\n            temperature=0.2,\n            max_new_tokens=1024,\n            use_cache=True,\n            stopping_criteria=[stopping_criteria])\n\n    outputs = tokenizer.decode(output_ids[0, input_ids.shape[1]:], skip_special_tokens=True).strip()\n    print(outputs)\n\nif __name__ == '__main__':\n    main()\n```\n\n## 🙌 Related Projects\n* [Video-LLaVA](https://github.com/PKU-YuanGroup/Video-LLaVA) This framework empowers the model to efficiently utilize the united visual tokens.\n* [LanguageBind](https://github.com/PKU-YuanGroup/LanguageBind) An open source five modalities language-based retrieval framework.\n\n## 👍 Acknowledgement\n* [LLaVA](https://github.com/haotian-liu/LLaVA) The codebase we built upon and it is an efficient large language and vision assistant.\n\n## 🔒 License\n* The majority of this project is released under the Apache 2.0 license as found in the [LICENSE](https://github.com/PKU-YuanGroup/MoE-LLaVA/blob/main/LICENSE) file.\n* The service is a research preview intended for non-commercial use only, subject to the model [License](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) of LLaMA, [Terms of Use](https://openai.com/policies/terms-of-use) of the data generated by OpenAI, and [Privacy Practices](https://chrome.google.com/webstore/detail/sharegpt-share-your-chatg/daiacboceoaocpibfodeljbdfacokfjb) of ShareGPT. Please contact us if you find any potential violation.\n\n\n\n## ✏️ Citation\nIf you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.\n\n```BibTeX\n@article{lin2024moe,\n  title={MoE-LLaVA: Mixture of Experts for Large Vision-Language Models},\n  author={Lin, Bin and Tang, Zhenyu and Ye, Yang and Cui, Jiaxi and Zhu, Bin and Jin, Peng and Zhang, Junwu and Ning, Munan and Yuan, Li},\n  journal={arXiv preprint arXiv:2401.15947},\n  year={2024}\n}\n```\n\n```BibTeX\n@article{lin2023video,\n  title={Video-LLaVA: Learning United Visual Representation by Alignment Before Projection},\n  author={Lin, Bin and Zhu, Bin and Ye, Yang and Ning, Munan and Jin, Peng and Yuan, Li},\n  journal={arXiv preprint arXiv:2311.10122},\n  year={2023}\n}\n```\n\n\n\n## ✨ Star History\n[![Star History](https://api.star-history.com/svg?repos=PKU-YuanGroup/MoE-LLaVA\u0026type=Date)](https://star-history.com/#PKU-YuanGroup/MoE-LLaVA\u0026Date)\n\n\n## 🤝 Contributors\n\n\u003ca href=\"https://github.com/PKU-YuanGroup/MoE-LLaVA/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=PKU-YuanGroup/MoE-LLaVA\" /\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPKU-YuanGroup%2FMoE-LLaVA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPKU-YuanGroup%2FMoE-LLaVA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPKU-YuanGroup%2FMoE-LLaVA/lists"}