{"id":13642856,"url":"https://github.com/opengvlab/multi-modality-arena","last_synced_at":"2025-04-20T21:31:44.838Z","repository":{"id":163995899,"uuid":"638869686","full_name":"OpenGVLab/Multi-Modality-Arena","owner":"OpenGVLab","description":"Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!","archived":false,"fork":false,"pushed_at":"2024-04-21T11:14:46.000Z","size":22550,"stargazers_count":511,"open_issues_count":24,"forks_count":37,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-03T15:44:20.185Z","etag":null,"topics":["chat","chatbot","chatgpt","gradio","large-language-models","llms","multi-modality","vision-language-model","vqa"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenGVLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-10T09:26:08.000Z","updated_at":"2025-04-01T18:50:05.000Z","dependencies_parsed_at":"2024-08-01T13:28:14.305Z","dependency_job_id":null,"html_url":"https://github.com/OpenGVLab/Multi-Modality-Arena","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMulti-Modality-Arena","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMulti-Modality-Arena/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMulti-Modality-Arena/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMulti-Modality-Arena/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenGVLab","download_url":"https://codeload.github.com/OpenGVLab/Multi-Modality-Arena/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249965525,"owners_count":21352921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chat","chatbot","chatgpt","gradio","large-language-models","llms","multi-modality","vision-language-model","vqa"],"created_at":"2024-08-02T01:01:37.294Z","updated_at":"2025-04-20T21:31:39.827Z","avatar_url":"https://github.com/OpenGVLab.png","language":"Python","funding_links":[],"categories":["Chatbots"],"sub_categories":[],"readme":"### Multi-Modality Arena 🚀\n\n  \n\n\u003cp  align=\"center\"\u003e\u003cimg  src=\"./CVLAB/style/img/Opengvlab_LOGO.svg\"  width=\"400\"\u003e\u003c/p\u003e\n\n  \n\n\n\n\u003c!-- ## Description --\u003e\n\n**Multi-Modality Arena** is an evaluation platform for large multi-modality models. Following [Fastchat](https://chat.lmsys.org/), two anonymous models side-by-side are compared on a visual question-answering task. We release the  [**Demo**](http://vlarena.opengvlab.com)  and welcome the participation of everyone in this evaluation initiative.\n\n[![⚔️ LVLM Arena](https://img.shields.io/badge/LVLM%20Arena-blue.svg?label=%E2%9A%94%EF%B8%8F)](http://vlarena.opengvlab.com)\n[![arXiv](https://img.shields.io/badge/LVLM%20eHub-2306.09265-b31b1b.svg)](https://arxiv.org/abs/2306.09265)\n[![arXiv](https://img.shields.io/badge/Tiny%20LVLM%20eHub-2308.03729-b31b1b.svg)](https://arxiv.org/abs/2308.03729)\n[![GitHub Stars](https://img.shields.io/github/stars/OpenGVLab/Multi-modality-Arena.svg?style=social\u0026label=Star\u0026maxAge=60)](https://github.com/OpenGVLab/Multi-modality-Arena)🔥🔥🔥\n                \n\n \u003cp  align=\"center\"\u003e\u003cimg  src=\"./CVLAB/style/img/demo1.jpg\"  width=\"800\"\u003e\u003c/p\u003e\n\n## Holistic Evaluation of Large Multimodal Models\n\n### OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM\n- OmniMedVQA dataset: contains 118,010 images with 127,995 QA-items, covering 12 different modalities and referring to more than 20 human anatomical regions. Dataset could be download from [Here](https://openxlab.org.cn/datasets/GMAI/OmniMedVQA).\n- 12 models: 8 general-domain LVLMs and 4 medical-specialized LVLMs.\n\n### Tiny LVLM-eHub: Early Multimodal Experiments with Bard\n\n- Tiny datasets: only 50 randomly selected sampels for each dataset, i.e., 42 text-related visual benchmarks and 2.1K samples in total for ease of use.\n- More models: another 4 models, i.e., 12 models in total, including **Google Bard**.\n- *ChatGPT Ensemble Evalution*: improved agreement with human evaluation than previous word matching approach.\n\n\u003cp  align=\"center\"\u003e\u003cimg  src=\"./assets/tiny_lvlm_ehub_6_12.png\"  width=\"500\"\u003e\u003c/p\u003e\n\n### LVLM-eHub: An Evaluation Benchmark for Large Vision-Language Models 🚀\n\nLVLM-eHub is a comprehensive evaluation benchmark for publicly available large multimodal models (LVLM).  It extensively evaluates $8$ LVLMs in terms of $6$ categories of multimodal capabilities with $47$ datasets and $1$ arena online platform.\n \u003cp  align=\"center\"\u003e\u003cimg  src=\"./CVLAB/style/img/lvlm-ehub.png\"  width=\"800\"\u003e\u003c/p\u003e\n\n## LVLM Leaderboard\nThe LVLM Leaderboard systematically categorizes the datasets featured in the Tiny LVLM Evaluation according to their specific targeted abilities including visual perception, visual reasoning, visual commonsense, visual knowledge acquisition, and object hallucination.\nThis leaderboard includes recently released models to bolster its comprehensiveness.\n\nYou can download the benchmark from [here](https://drive.google.com/file/d/1PuFC612XzOmKwzRldtBb1CFZnIjiR7we/view?usp=sharing), and more details can be found in [here](https://github.com/OpenGVLab/Multi-Modality-Arena/tree/main/tiny_lvlm_evaluation).\n\n| Rank | Model | Version | Score |\n| :--: | :--: | :--: | :--: |\n| 🏅️ | **[InternVL](https://github.com/OpenGVLab/InternVL)** | InternVL-Chat | **327.61** |\n| 🥈 | **[InternLM-XComposer-VL](https://github.com/InternLM/InternLM-XComposer)** | InternLM-XComposer-VL-7B | **322.51** |\n| 🥉  | **[Bard](https://bard.google.com/)** | Bard | **319.59** |\n| 4 | [Qwen-VL-Chat](https://github.com/QwenLM/Qwen-VL) | Qwen-VL-Chat | 316.81 |\n| 5 | [LLaVA-1.5](https://github.com/haotian-liu/LLaVA) | Vicuna-7B | 307.17 |\n| 6 | [InstructBLIP](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip) | Vicuna-7B | 300.64 |\n| 7 | [InternLM-XComposer](https://github.com/InternLM/InternLM-XComposer) | InternLM-XComposer-7B | 288.89 |\n| 8 | [BLIP2](https://github.com/salesforce/LAVIS/tree/main/projects/blip2) | FlanT5xl | 284.72 |\n| 9 | [BLIVA](https://github.com/mlpc-ucsd/BLIVA) | Vicuna-7B | 284.17 |\n| 10 | [Lynx](https://github.com/bytedance/lynx-llm) | Vicuna-7B | 279.24 |\n| 11 | [Cheetah](https://github.com/DCDmllm/Cheetah) | Vicuna-7B | 258.91 |\n| 12 | [LLaMA-Adapter-v2](https://github.com/OpenGVLab/LLaMA-Adapter/tree/main/llama_adapter_v2_multimodal7b) | LLaMA-7B | 229.16 |\n| 13 | [VPGTrans](https://github.com/VPGTrans/VPGTrans) | Vicuna-7B | 218.91 |\n| 14 | [Otter-Image](https://github.com/Luodian/Otter) | Otter-9B-LA-InContext | 216.43 |\n| 15 | [VisualGLM-6B](https://github.com/THUDM/VisualGLM-6B) | VisualGLM-6B | 211.98 |\n| 16 | [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl) | LLaMA-7B | 209.40 |\n| 17 | [LLaVA](https://github.com/haotian-liu/LLaVA) | Vicuna-7B | 200.93 |\n| 18 | [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4) | Vicuna-7B | 192.62 |\n| 19 | [Otter](https://github.com/Luodian/Otter) | Otter-9B | 180.87 |\n| 20 | [OFv2_4BI](https://github.com/mlfoundations/open_flamingo) | RedPajama-INCITE-Instruct-3B-v1 | 176.37 |\n| 21 | [PandaGPT](https://github.com/yxuansu/PandaGPT) | Vicuna-7B | 174.25 |\n| 22 | [LaVIN](https://github.com/luogen1996/LaVIN) | LLaMA-7B | 97.51 |\n| 23 | [MIC](https://github.com/HaozheZhao/MIC) | FlanT5xl | 94.09 |\n\n\n\n\n## Update\n-  🔥 Mar. 31, 2024. We release OmniMedVQA, a large-scale comprehensive evaluation benchmark for medical LVLMs. Meanwhile, we 8 general-domain LVLMs and 4 medical-specialized LVLMs. For more details, please visit the [MedicalEval](https://github.com/OpenGVLab/Multi-Modality-Arena/tree/main/MedicalEval).\n-  🔥 Oct. 16, 2023. We present an ability-level dataset split derived from the LVLM-eHub, complemented by the inclusion of eight recently released models. For access to the dataset splits, evaluation code, model inference results, and comprehensive performance tables, please visit the [tiny_lvlm_evaluation](https://github.com/OpenGVLab/Multi-Modality-Arena/tree/main/tiny_lvlm_evaluation) ✅.\n- Aug. 8, 2023. We released [**[Tiny LVLM-eHub]**](https://arxiv.org/abs/2308.03729). Evaluation source codes and model inference results are open-sourced under [tiny_lvlm_evaluation](https://github.com/OpenGVLab/Multi-Modality-Arena/tree/main/tiny_lvlm_evaluation).\n- Jun. 15, 2023. We release [**[LVLM-eHub]**](https://arxiv.org/abs/2306.09265), an evaluation benchmark for large vision-language models. The code is coming soon.\n- Jun. 8, 2023. Thanks, Dr. Zhang, the author of VPGTrans, for his corrections. The authors of VPGTrans mainly come from NUS and Tsinghua University. We previously had some minor issues when re-implementing VPGTrans, but we found that its performance is actually better. For more model authors, please contact me for discussion at the [Email](shaowenqi@pjlab.org.cn). Also, please follow our model ranking list, where more accurate results will be available.\n- May. 22, 2023. Thanks, Dr. Ye, the author of mPLUG-Owl, for his corrections. We fix some minor issues in our implementation of mPLIG-Owl.\n## Supported Multi-modality Models\n\nThe following models are involving in randomized battles currently,\n\n - [**KAUST/MiniGPT-4**](https://github.com/Vision-CAIR/MiniGPT-4)\n - [**Salesforce/BLIP2**](https://github.com/salesforce/LAVIS/tree/main/projects/blip2)\n - [**Salesforce/InstructBLIP**](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip)\n - [**DAMO Academy/mPLUG-Owl**](https://github.com/X-PLUG/mPLUG-Owl)\n - [**NTU/Otter**](https://github.com/Luodian/otter)\n - [**University of Wisconsin-Madison/LLaVA**](https://github.com/haotian-liu/LLaVA)\n - [**Shanghai AI Lab/llama_adapter_v2**](https://github.com/OpenGVLab/LLaMA-Adapter/tree/main/llama_adapter_v2_multimodal)\n - [**NUS/VPGTrans**](https://github.com/VPGTrans/VPGTrans)\n\nMore details about these models can be found at ```./model_detail/.model.jpg```. We will try to schedule computing resources to host more multi-modality models in the arena.\n\n## Contact US at Wechat\nIf you are interested in any pieces of our VLarena platform, feel free to join the Wechat group.\n \u003cp  align=\"left\"\u003e\u003cimg  src=\"./CVLAB/style/img/Wechat.jpeg\"  width=\"200\"\u003e\u003c/p\u003e\n\n## Installation\n1. Create conda environment\n```bash\nconda create -n arena python=3.10\nconda activate arena\n```\n\n2. Install Packages required to run the controller and server\n```bash\npip install numpy gradio uvicorn fastapi\n```\n\n3. Then for each model, they may require conflicting versions of python packages, we recommend creating a specific environment for each model based on their GitHub repo.\n\n\n## Launch a Demo\n \nTo serve using the web UI, you need three main components: web servers that interface with users, model workers that host two or more models, and a controller to coordinate the webserver and model workers.\n\nHere are the commands to follow in your terminal:\n\n#### Launch the controller\n```bash\npython controller.py\n```\nThis controller manages the distributed workers.\n\n#### Launch the model worker(s)\n```bash\npython model_worker.py --model-name SELECTED_MODEL --device TARGET_DEVICE\n```\nWait until the process finishes loading the model and you see \"Uvicorn running on ...\". The model worker will register itself to the controller. For each model worker, you need to specify the model and the device you want to use.\n\n#### Launch the Gradio web server\n```bash\npython server_demo.py\n```\nThis is the user interface that users will interact with.\n\nBy following these steps, you will be able to serve your models using the web UI. You can open your browser and chat with a model now.\nIf the models do not show up, try to reboot the gradio web server.\n\n## Contribution Guidelines\nWe deeply value all contributions aimed at enhancing the quality of our evaluations. This section comprises two key segments: `Contributions to LVLM Evaluation` and `Contributions to LVLM Arena`.\n\n### Contributing to LVLM Evaluation\nYou can access the most recent version of our evaluation code in the LVLM_evaluation folder. This directory encompasses a comprehensive set of evaluation code, accompanied by the necessary datasets. If you're enthusiastic about partaking in the evaluation process, please don't hesitate to share your evaluation outcomes or the model inference API with us via email at xupeng@pjlab.org.cn.\n\n### Contributions to LVLM Arena\nWe extend our gratitude for your interest in integrating your model into our LVLM Arena! Should you wish to incorporate your model into our Arena, kindly prepare a model tester structured as follows:\n\n``` python\nclass ModelTester:\n    def __init__(self, device=None) -\u003e None:\n        # TODO: initialization of model and required pre processors\n    \n    def move_to_device(self, device) -\u003e None:\n        # TODO: this function is used to transfer the model between CPU and GPU (optional)\n    \n    def generate(self, image, question) -\u003e str:\n       # TODO: model inference code \n```\n\nFurthermore, we are open to online model inference links, such as those provided by platforms like Gradio. Your contributions are wholeheartedly appreciated.\n\n## Acknowledgement\nWe express our gratitude to the esteemed team at [ChatBot Arena](https://arena.lmsys.org/) and their paper [Judging LLM-as-a-judge](https://arxiv.org/abs/2306.05685) for their influential work, which served as inspiration for our LVLM evaluation endeavors. We would also like to extend our sincere appreciation to the providers of LVLMs, whose valuable contributions have significantly contributed to the progress and advancement of large vision-language models. Finally, we thank the providers of datasets used in our LVLM-eHub.\n\n## Term of Use\nThe project is an experimental research tool for non-commercial purposes only. It has limited safeguards and may generate inappropriate content. It cannot be used for anything illegal, harmful, violent, racist, or sexual. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopengvlab%2Fmulti-modality-arena","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopengvlab%2Fmulti-modality-arena","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopengvlab%2Fmulti-modality-arena/lists"}