{"id":13429542,"url":"https://github.com/OpenGVLab/InternVL","last_synced_at":"2025-03-16T03:31:54.632Z","repository":{"id":213529900,"uuid":"721995615","full_name":"OpenGVLab/InternVL","owner":"OpenGVLab","description":"[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o.  接近GPT-4o表现的开源多模态对话模型","archived":false,"fork":false,"pushed_at":"2024-11-20T04:09:57.000Z","size":37518,"stargazers_count":6068,"open_issues_count":151,"forks_count":471,"subscribers_count":53,"default_branch":"main","last_synced_at":"2024-11-21T03:45:13.333Z","etag":null,"topics":["gpt","gpt-4o","gpt-4v","image-classification","image-text-retrieval","llm","multi-modal","semantic-segmentation","video-classification","vision-language-model","vit-22b","vit-6b"],"latest_commit_sha":null,"homepage":"https://internvl.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenGVLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-22T08:08:08.000Z","updated_at":"2024-11-21T02:08:18.000Z","dependencies_parsed_at":"2024-03-07T21:31:11.144Z","dependency_job_id":"a905513d-153a-4392-addd-d891c220301a","html_url":"https://github.com/OpenGVLab/InternVL","commit_stats":{"total_commits":169,"total_committers":10,"mean_commits":16.9,"dds":"0.14792899408284022","last_synced_commit":"6a230b34cc04eb2ee51c3ea013362a57ab6a6dc9"},"previous_names":["opengvlab/internvl"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FInternVL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FInternVL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FInternVL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FInternVL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenGVLab","download_url":"https://codeload.github.com/OpenGVLab/InternVL/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243822309,"owners_count":20353496,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpt","gpt-4o","gpt-4v","image-classification","image-text-retrieval","llm","multi-modal","semantic-segmentation","video-classification","vision-language-model","vit-22b","vit-6b"],"created_at":"2024-07-31T02:00:41.513Z","updated_at":"2025-03-16T03:31:54.625Z","avatar_url":"https://github.com/OpenGVLab.png","language":"Python","funding_links":[],"categories":["2 Foundation Models","A01_文本生成_文本对话","Vision LLM for Generation","Papers","Repos","Python","Navigation Guide","2. LLM Backbones","💬 大语言模型（LLM）","Multi-Modality","Foundation VLMs","Health AI \u0026 Machine Learning","4. 机器学习项目 | ML","Vision-Language Models (VLMs)"],"sub_categories":["2.3 Multimodal Foundation Models","大语言对话模型及数据","Multimodal AI for Science","Blogs","多模态大模型","General-purpose open models","Multimodal \u0026 Imaging AI"],"readme":"\u003cdiv align=\"center\"\u003e\n\n# InternVL Family: Closing the Gap to Commercial Multimodal Models with Open-Source Suites —— A Pioneering Open-Source Alternative to GPT-4o\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"500\" alt=\"image\" src=\"https://github.com/user-attachments/assets/930e6814-8a9f-43e1-a284-118a5732daa4\"\u003e\n  \u003cbr\u003e\n\u003c/div\u003e\n\n[\\[🆕 Blog\\]](https://internvl.github.io/blog/)  [\\[🤔 FAQs\\]](https://internvl.readthedocs.io/en/latest/tutorials/faqs.html)   [\\[🗨️ Chat Demo\\]](https://internvl.opengvlab.com/)  [\\[🤗 HF Demo\\]](https://huggingface.co/spaces/OpenGVLab/InternVL)  [\\[📖 Document\\]](https://internvl.readthedocs.io/en/latest/)  [\\[🌐 API\\]](https://internlm.intern-ai.org.cn/api/document)  [\\[🚀 Quick Start\\]](#quick-start-with-huggingface)\n\n[\\[🔥 InternVL2.5 Report\\]](https://huggingface.co/papers/2412.05271) [\\[Mini-InternVL Paper\\]](https://arxiv.org/abs/2410.16261) [\\[InternVL2 Blog\\]](https://internvl.github.io/blog/2024-07-02-InternVL-2.0/)  [\\[📜 InternVL 1.5 Paper\\]](https://huggingface.co/papers/2404.16821)  [\\[📜 InternVL 1.0 Paper\\]](https://huggingface.co/papers/2312.14238)\n\n[\\[📖 2.0 中文解读\\]](https://zhuanlan.zhihu.com/p/706547971)  [\\[📖 1.5 中文解读\\]](https://zhuanlan.zhihu.com/p/699439759)  [\\[📖 1.0 中文解读\\]](https://zhuanlan.zhihu.com/p/702946079)\n\n[Switch to the Chinese version (切换至中文版)](/README_zh.md)\n\n\u003ca href=\"https://trendshift.io/repositories/9803\" target=\"_blank\"\u003e\u003cimg src=\"https://trendshift.io/api/badge/repositories/9803\" alt=\"OpenGVLab%2FInternVL | Trendshift\" style=\"width: 250px; height: 55px;\" width=\"250\" height=\"55\"/\u003e\u003c/a\u003e\n\u003cimg height=\"55\" alt=\"image\" src=\"https://github.com/user-attachments/assets/bd62ab46-f0ea-40c6-ab10-7fde671716cc\"\u003e\n\n![perform](https://github.com/user-attachments/assets/3059622e-e553-40ba-84ed-32d8f92a2f69)\n\n\u003c/div\u003e\n\n## News 🚀🚀🚀\n\n- `2024/12/20`: 🔥 We release the [InternVL2.5-MPO](https://internvl.github.io/blog/2024-12-20-InternVL-2.5-MPO/), which is finetuned with [Mixed Preference Optimization](https://huggingface.co/papers/2411.10442) on [MMPR-v1.1](https://huggingface.co/datasets/OpenGVLab/MMPR-v1.1). **The resulting models outperform their counterparts without MPO by an average of 2 points across all model scales on the OpenCompass leaderboard.** These models are available at [HF link](https://huggingface.co/collections/OpenGVLab/internvl25-mpo-6753fed98cd828219b12f849).\n- `2024/12/17`: 🚀 [InternVL2/2.5](https://github.com/PaddlePaddle/PaddleMIX/tree/develop/paddlemix/examples/internvl2) is supported in [PaddleMIX](https://github.com/PaddlePaddle/PaddleMIX) by Paddle Team.\n- `2024/12/05`: 🚀 We release the [InternVL2.5](https://huggingface.co/collections/OpenGVLab/internvl-25-673e1019b66e2218f68d7c1c), an advanced multimodal large language model (MLLM) series with parameter coverage ranging from 1B to 78B. [InternVL2_5-78B](https://huggingface.co/OpenGVLab/InternVL2_5-78B) is the first open-source MLLMs to achieve over **70%** on the **MMMU benchmark**, matching the performance of leading closed-source commercial models like GPT-4o. These models are available at [HF link](https://huggingface.co/collections/OpenGVLab/internvl-25-673e1019b66e2218f68d7c1c).\n- `2024/11/14`: We introduce [MMPR](https://huggingface.co/datasets/OpenGVLab/MMPR), a high-quality, large-scale multimodal reasoning preference dataset, and [MPO](https://github.com/OpenGVLab/InternVL/tree/main/internvl_chat/shell/internvl2.0_mpo), an effective preference optimization algorithm. The resulting model, [InternVL2-8B-MPO](https://huggingface.co/OpenGVLab/InternVL2-8B-MPO), achieves an accuracy of 67.0 on MathVista. Please refer to our [paper](https://arxiv.org/abs/2411.10442), [project page](https://internvl.github.io/blog/2024-11-14-InternVL-2.0-MPO/) and [document](https://internvl.readthedocs.io/en/latest/internvl2.0/preference_optimization.html) for more details.\n- `2024/10/21`: We release the Mini-InternVL series. These models achieve impressive performance with minimal size: the 4B model achieves 90% of the performance with just 5% of the model size. For more details, please check our [project page](https://github.com/OpenGVLab/InternVL/tree/main/internvl_chat/shell/mini_internvl) and [document](https://internvl.readthedocs.io/en/latest/internvl2.0/domain_adaptation.html).\n- `2024/08/01`: The [Chartmimic](https://chartmimic.github.io/) team evaluated the InternVL2 series models on their benchmark. The InternVL2-26B and 76B models achieved the top two performances among open-source models, with the InternVL2 76B model surpassing GeminiProVision and exhibiting comparable results to Claude-3-opus.\n- `2024/08/01`: InternVL2-Pro achieved the SOTA performance among open-source models on the [CharXiv](https://charxiv.github.io/#leaderboard) dataset, surpassing many closed-source models such as GPT-4V, Gemini 1.5 Flash, and Claude 3 Sonnet.\n- `2024/07/24`: The [MLVU](https://github.com/JUNJIE99/MLVU) team evaluated InternVL-1.5 on their benchmark. The average performance on the multiple-choice task was 50.4%, while the performance on the generative tasks was 4.02. The performance on the multiple-choice task ranked #1 among all open-source MLLMs.\n- `2024/07/04`: We release the [InternVL2 series](https://huggingface.co/collections/OpenGVLab/internvl-20-667d3961ab5eb12c7ed1463e). InternVL2-Pro achieved a 62.0% accuracy on the MMMU benchmark, matching the performance of leading closed-source commercial models like GPT-4o.\n\n\u003cdetails\u003e\n\u003csummary\u003eMore News\u003c/summary\u003e\n\n- `2024/07/18`: InternVL2-40B achieved SOTA performance among open-source models on the [Video-MME](https://github.com/BradyFU/Video-MME) dataset, scoring 61.2 when inputting 16 frames and 64.4 when inputting 32 frames. It significantly outperforms other open-source models and is the closest open-source model to GPT-4o mini.\n- `2024/07/18`: InternVL2-Pro achieved the SOTA performance on the [DocVQA](https://rrc.cvc.uab.es/?ch=17\u0026com=evaluation\u0026task=1) and [InfoVQA](https://rrc.cvc.uab.es/?ch=17\u0026com=evaluation\u0026task=3) benchmarks.\n- `2024/06/19`: We propose Needle In A Multimodal Haystack ([MM-NIAH](https://github.com/OpenGVLab/MM-NIAH)), the first benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.\n- `2024/05/30`: We release [ShareGPT-4o](https://sharegpt4o.github.io/), a large-scale dataset that we plan to open-source with 200K images, 10K videos, and 10K audios with detailed descriptions.\n- `2024/05/28`: Thanks to the [lmdeploy](https://github.com/InternLM/lmdeploy) team for providing AWQ quantization support. The 4-bit model is available at [OpenGVLab/InternVL-Chat-V1-5-AWQ](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5-AWQ).\n- `2024/05/13`: InternVL 1.0 can now be used as the [text encoder](https://huggingface.co/OpenGVLab/InternVL-14B-224px) for diffusion models to support multilingual generation natively in over 110 languages worldwide. See [MuLan](https://github.com/mulanai/MuLan) for more details.\n- `2024/04/18`: InternVL-Chat-V1-5 has been released at [HF link](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5), approaching the performance of GPT-4V and Gemini Pro on various benchmarks like MMMU, DocVQA, ChartQA, MathVista, etc.\n- `2024/02/27`: InternVL is accepted by CVPR 2024 (Oral)! 🎉\n- `2024/02/21`: [InternVL-Chat-V1-2-Plus](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-2-Plus) achieved SOTA performance on MathVista (59.9), MMBench (83.8), and MMVP (58.7). See our [blog](https://internvl.github.io/blog/2024-02-21-InternVL-1.2/) for more details.\n- `2024/02/12`: InternVL-Chat-V1-2 has been released. It achieves 51.6 on MMMU val and 82.3 on MMBench test. For more details, please refer to our [blog](https://internvl.github.io/blog/2024-02-21-InternVL-1.2/) and [SFT data](./internvl_chat#prepare-training-datasets). The model is now available on [HuggingFace](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-2), and both training / evaluation data and scripts are open-sourced.\n- `2024/01/24`: InternVL-Chat-V1-1 is released, it supports Chinese and has stronger OCR capability, see [here](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-1).\n- `2024/01/16`: We release our [customized mmcv/mmsegmentation/mmdetection code](https://github.com/OpenGVLab/InternVL-MMDetSeg), integrated with DeepSpeed, which can be used for training large-scale detection and segmentation models.\n\n\u003c/details\u003e\n\n## Documents\n\n### 🌟 **Get Started**\n\n- **Installation**: 🌱 [Installation Guide](https://internvl.readthedocs.io/en/latest/get_started/installation.html) | 📄 [requirements.txt](./requirements.txt)\n- **Chat Data Format**: 📝 [Meta File](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#meta-file) | ✏️ [Text](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#pure-text-data) | 🖼️ [Single-Image](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#single-image-data) | 🖼️🖼️ [Multi-Image](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#multi-image-data) | 🎥 [Video](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#video-data)\n- **Local Chat Demo**: 🤖 [Streamlit Demo](https://internvl.readthedocs.io/en/latest/get_started/local_chat_demo.html#streamlit-demo)\n- **InternVL-Chat API**: 🌐 [InternVL2.5 API](https://internlm.intern-ai.org.cn/api/document)\n- **Tutorials**: 🚀 [Enhancing InternVL2 on COCO Caption Using LoRA Fine-Tuning](https://internvl.readthedocs.io/en/latest/tutorials/coco_caption_finetune.html)\n\n### 🏆 **InternVL Family**\n\n- **InternVL 2.5**: 📖 [Intro](https://internvl.readthedocs.io/en/latest/internvl2.5/introduction.html) | ⚡ [Quick Start](https://internvl.readthedocs.io/en/latest/internvl2.5/quick_start.html) | ✨ [Finetune](https://internvl.readthedocs.io/en/latest/internvl2.5/finetune.html) | 📊 [Evaluate](https://internvl.readthedocs.io/en/latest/internvl2.5/evaluation.html) | 📦 [Deploy](https://internvl.readthedocs.io/en/latest/internvl2.5/deployment.html) | 🎯 [MPO](https://internvl.readthedocs.io/en/latest/internvl2.5/preference_optimization.html)\n- **InternVL 2.0**: 📖 [Intro](https://internvl.readthedocs.io/en/latest/internvl2.0/introduction.html) | ⚡ [Quick Start](https://internvl.readthedocs.io/en/latest/internvl2.0/quick_start.html) | ✨ [Finetune](https://internvl.readthedocs.io/en/latest/internvl2.0/finetune.html) | 📊 [Evaluate](https://internvl.readthedocs.io/en/latest/internvl2.0/evaluation.html) | 📦 [Deploy](https://internvl.readthedocs.io/en/latest/internvl2.0/deployment.html) | 🎯 [MPO](https://internvl.readthedocs.io/en/latest/internvl2.0/preference_optimization.html)\n- **InternVL 1.5**: 📖 [Intro](https://internvl.readthedocs.io/en/latest/internvl1.5/introduction.html) | ⚡ [Quick Start](https://internvl.readthedocs.io/en/latest/internvl1.5/quick_start.html) | ✨ [Finetune](https://internvl.readthedocs.io/en/latest/internvl1.5/finetune.html) | 📊 [Evaluate](https://internvl.readthedocs.io/en/latest/internvl1.5/evaluation.html) | 📦 [Deploy](https://internvl.readthedocs.io/en/latest/internvl1.5/deployment.html)\n- **InternVL 1.2**: 📖 [Intro](https://internvl.readthedocs.io/en/latest/internvl1.2/introduction.html) | ⚡ [Quick Start](https://internvl.readthedocs.io/en/latest/internvl1.2/quick_start.html) | ✨ [Finetune](https://internvl.readthedocs.io/en/latest/internvl1.2/finetune.html) | 📊 [Evaluate](https://internvl.readthedocs.io/en/latest/internvl1.2/evaluation.html)\n- **InternVL 1.1**: 📖 [Intro](https://internvl.readthedocs.io/en/latest/internvl1.1/introduction.html) | ⚡ [Quick Start](https://internvl.readthedocs.io/en/latest/internvl1.1/quick_start.html) | 📊 [Evaluation](https://internvl.readthedocs.io/en/latest/internvl1.1/evaluation.html)\n- **InternVL 1.0**: 🖼️ [Classification](https://internvl.readthedocs.io/en/latest/internvl1.0/classification.html) | 📊 [CLIP-Benchmark](https://internvl.readthedocs.io/en/latest/internvl1.0/clip_benchmark.html) | 🎨 [Segmentation](https://internvl.readthedocs.io/en/latest/internvl1.0/segmentation.html) | 💬 [Chat-LLaVA](https://internvl.readthedocs.io/en/latest/internvl1.0/internvl_chat_llava.html) | ✨ [InternVL-G](https://internvl.readthedocs.io/en/latest/internvl1.0/internvl_g.html)\n\n## Model Zoo\n\n#### Multimodal Large Language Model (InternVL 2.5)\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eModel Name\u003c/th\u003e\n    \u003cth\u003eVision Part\u003c/th\u003e\n    \u003cth\u003eLanguage Part\u003c/th\u003e\n    \u003cth\u003eHF\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eMS\u0026nbsp;Link\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-1B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003eInternViT\u0026#8209;300M\u0026#8209;448px\u0026#8209;V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct\"\u003eQwen2.5\u0026#8209;0.5B\u0026#8209;Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-1B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-1B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-2B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003eInternViT-300M-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2_5-1_8b-chat\"\u003einternlm2_5-1_8b-chat\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-2B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-2B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-4B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003eInternViT-300M-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2.5-3B-Instruct\"\u003eQwen2.5-3B-Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-4B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-4B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-8B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003eInternViT-300M-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2_5-7b-chat\"\u003einternlm2_5-7b-chat\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-8B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-8B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-26B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5\"\u003eInternViT-6B-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2_5-20b-chat\"\u003einternlm2_5-20b-chat\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-26B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-26B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-38B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5\"\u003eInternViT-6B-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct\"\u003eQwen2.5-32B-Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-38B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-38B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-78B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5\"\u003eInternViT-6B-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2.5-72B-Instruct\"\u003eQwen2.5-72B-Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-78B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-78B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eModel Name\u003c/th\u003e\n    \u003cth\u003eVision Part\u003c/th\u003e\n    \u003cth\u003eLanguage Part\u003c/th\u003e\n    \u003cth\u003eHF\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eMS\u0026nbsp;Link\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-1B-MPO\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003eInternViT\u0026#8209;300M\u0026#8209;448px\u0026#8209;V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct\"\u003eQwen2.5\u0026#8209;0.5B\u0026#8209;Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-1B-MPO\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-1B-MPO\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-2B-MPO\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003eInternViT-300M-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2_5-1_8b-chat\"\u003einternlm2_5-1_8b-chat\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-2B-MPO\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-2B-MPO\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-4B-MPO\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003eInternViT-300M-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2.5-3B-Instruct\"\u003eQwen2.5-3B-Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-4B-MPO\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-4B-MPO\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-8B-MPO\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003eInternViT-300M-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2_5-7b-chat\"\u003einternlm2_5-7b-chat\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-8B-MPO\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-8B-MPO\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-26B-MPO\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5\"\u003eInternViT-6B-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2_5-20b-chat\"\u003einternlm2_5-20b-chat\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-26B-MPO\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-26B-MPO\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-38B-MPO\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5\"\u003eInternViT-6B-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct\"\u003eQwen2.5-32B-Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-38B-MPO\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-38B-MPO\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2_5-78B-MPO\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5\"\u003eInternViT-6B-448px-V2_5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2.5-72B-Instruct\"\u003eQwen2.5-72B-Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2_5-78B-MPO\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2_5-78B-MPO\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n#### Multimodal Large Language Model (InternVL 2.0)\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eModel Name\u003c/th\u003e\n    \u003cth\u003eVision Part\u003c/th\u003e\n    \u003cth\u003eLanguage Part\u003c/th\u003e\n    \u003cth\u003eHF\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eMS\u0026nbsp;Link\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2-1B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px\"\u003eInternViT-300M-448px\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/Qwen/Qwen2-0.5B-Instruct\"\u003eQwen2-0.5B-Instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2-1B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2-1B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2-2B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px\"\u003eInternViT-300M-448px\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2-chat-1_8b\"\u003einternlm2-chat-1-8b\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2-2B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2-2B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2-4B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px\"\u003eInternViT-300M-448px\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/microsoft/Phi-3-mini-128k-instruct\"\u003ePhi\u0026#8209;3\u0026#8209;mini\u0026#8209;128k\u0026#8209;instruct\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2-4B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2-4B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2-8B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px\"\u003eInternViT-300M-448px\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2_5-7b-chat\"\u003einternlm2_5-7b-chat\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2-8B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2-8B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2-26B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5\"\u003eInternViT-6B-448px-V1-5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/internlm/internlm2-chat-20b\"\u003einternlm2-chat-20b\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2-26B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2-26B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2-40B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5\"\u003eInternViT\u0026#8209;6B\u0026#8209;448px\u0026#8209;V1\u0026#8209;5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B\"\u003eNous\u0026#8209;Hermes\u0026#8209;2\u0026#8209;Yi\u0026#8209;34B\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2-40B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2-40B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL2\u0026#8209;Llama3-76B\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5\"\u003eInternViT-6B-448px-V1-5\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B\"\u003eHermes‑2‑Theta‑\u003cbr\u003eLlama‑3‑70B\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL2-Llama3-76B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n#### Multimodal Large Language Model (InternVL 1.0-1.5)\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eModel\u003c/th\u003e\n    \u003cth\u003eDate\u003c/th\u003e\n    \u003cth\u003eHF\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eMS\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eNote\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eMini\u0026#8209;InternVL\u0026#8209;Chat\u0026#8209;4B\u0026#8209;V1\u0026#8209;5\u003c/td\u003e\n    \u003ctd\u003e2024.05.28\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/Mini-InternVL-Chat-4B-V1-5\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/Mini-InternVL-Chat-4B-V1-5\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e🚀🚀 16% of the model size, 90% of the performance\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eMini-InternVL-Chat-2B-V1-5\u003c/td\u003e\n    \u003ctd\u003e2024.05.19\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/Mini-InternVL-Chat-2B-V1-5\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/Mini-InternVL-Chat-2B-V1-5\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e🚀 8% of the model size, 80% of the performance\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL-Chat-V1-5\u003c/td\u003e\n    \u003ctd\u003e2024.04.18\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL-Chat-V1-5\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003esupport 4K image; super strong OCR; Approaching the performance of GPT-4V and Gemini Pro on various benchmarks like MMMU, DocVQA, ChartQA, MathVista, etc.\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL-Chat-V1-2-Plus\u003c/td\u003e\n    \u003ctd\u003e2024.02.21\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL-Chat-V1-2-Plus\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL-Chat-V1-2-Plus\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003emore SFT data and stronger\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL-Chat-V1-2\u003c/td\u003e\n    \u003ctd\u003e2024.02.11\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL-Chat-V1-2\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL-Chat-V1-2\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003escaling up LLM to 34B\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL-Chat-V1-1\u003c/td\u003e\n    \u003ctd\u003e2024.01.24\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL-Chat-V1-1\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL-Chat-V1-1\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003esupport Chinese and stronger OCR\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL-Chat-19B\u003c/td\u003e\n    \u003ctd\u003e2023.12.25\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL-Chat-ViT-6B-Vicuna-13B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL-Chat-ViT-6B-Vicuna-13B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003eEnglish multimodal dialogue\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL-Chat-13B\u003c/td\u003e\n    \u003ctd\u003e2023.12.25\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL-Chat-ViT-6B-Vicuna-7B\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL-Chat-ViT-6B-Vicuna-7B\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003eEnglish multimodal dialogue\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n#### CLIP-like Model (InternVL 1.0-2.5)\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eModel\u003c/th\u003e\n    \u003cth\u003eDate\u003c/th\u003e\n    \u003cth\u003eHF\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eMS\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eNote\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternViT-300M-448px-V2_5\u003c/td\u003e\n    \u003ctd\u003e2024.12.05\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternViT-300M-448px-V2_5\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e🚀🚀 A more powerful lightweight visual encoder. (🔥new)\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternViT-6B-448px-V2_5\u003c/td\u003e\n    \u003ctd\u003e2024.12.05\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternViT-6B-448px-V2_5\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e🚀🚀 A stronger visual encoder to extract visual features. (🔥new)\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternViT-300M-448px\u003c/td\u003e\n    \u003ctd\u003e2024.05.25\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-300M-448px\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternViT-300M-448px\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003edistilled small vision foundation model with 300M parameters \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternViT\u0026#8209;6B\u0026#8209;448px\u0026#8209;V1\u0026#8209;5\u003c/td\u003e\n    \u003ctd\u003e2024.04.20\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternViT-6B-448px-V1-5\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003esupport dynamic resolution and super strong OCR feature extraction capability by incremental pre-training\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternViT-6B-448px-V1-2\u003c/td\u003e\n    \u003ctd\u003e2024.02.11\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-2\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternViT-6B-448px-V1-2\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003esupport 448 resolution by incremental pre-training\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternViT-6B-448px-V1-0\u003c/td\u003e\n    \u003ctd\u003e2024.01.30\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-0\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternViT-6B-448px-V1-0\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003esupport 448 resolution by incremental pre-training\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternViT-6B-224px\u003c/td\u003e\n    \u003ctd\u003e2023.12.22\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternViT-6B-224px\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternViT-6B-224px\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003ethe first version of InternViT-6B, extracted from InternVL‑14B‑224px\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n#### Vision-Language Foundation Model (InternVL 1.0)\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eModel\u003c/th\u003e\n    \u003cth\u003eDate\u003c/th\u003e\n    \u003cth\u003eHF\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eMS\u0026nbsp;Link\u003c/th\u003e\n    \u003cth\u003eNote\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eInternVL\u0026#8209;14B\u0026#8209;224px\u003c/td\u003e\n    \u003ctd\u003e2023.12.22\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://huggingface.co/OpenGVLab/InternVL-14B-224px\"\u003e🤗 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://modelscope.cn/models/OpenGVLab/InternVL-14B-224px\"\u003e🤖 link\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003evision-language foundation model, InternViT-6B + QLLaMA, can be used for image-text retrieval like CLIP\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n## TODO List\n\n- [x] Release training / evaluation code for InternVL2.5 series\n- [x] Support liger kernels to save GPU memory\n- [x] Release the code, model, and data of MPO\n- [x] Support multimodal packed dataset\n- [ ] Support vLLM and Ollama\n- [ ] Support video and PDF input in online demo\n- [ ] Release InternVL2 with VisionLLMv2 integration\n- [x] Rebuild documents using readthedocs\n- [x] Support fine-tuning different LLMs with LoRA\n- [x] Release `requirements.txt` for InternVL2\n- [x] Release training / evaluation code for InternVL2 series\n- [x] Release Streamlit web UI for InternVL1.5 and InternVL2\n\n## What can InternVL do?\n\n\u003cdetails\u003e\n  \u003csummary\u003eVisual Perception (click to expand)\u003c/summary\u003e\n\n- Linear-Probe Image Classification [\\[see details\\]](./classification#-evaluation)\n\n  ViT-22B uses the private JFT-3B dataset.\n\n  | method              | #param | IN-1K | IN-ReaL | IN-V2 | IN-A | IN-R | IN-Sketch |\n  | ------------------- | :----: | :---: | :-----: | :---: | :--: | :--: | :-------: |\n  | OpenCLIP-G          |  1.8B  | 86.2  |  89.4   | 77.2  | 63.8 | 87.8 |   66.4    |\n  | DINOv2-g            |  1.1B  | 86.5  |  89.6   | 78.4  | 75.9 | 78.8 |   62.5    |\n  | EVA-01-CLIP-g       |  1.1B  | 86.5  |  89.3   | 77.4  | 70.5 | 87.7 |   63.1    |\n  | MAWS-ViT-6.5B       |  6.5B  | 87.8  |    -    |   -   |  -   |  -   |     -     |\n  | ViT-22B\\*           | 21.7B  | 89.5  |  90.9   | 83.2  | 83.8 | 87.4 |     -     |\n  | InternViT-6B (ours) |  5.9B  | 88.2  |  90.4   | 79.9  | 77.5 | 89.8 |   69.1    |\n\n- Semantic Segmentation [\\[see details\\]](./segmentation#-evaluation)\n\n  | method                | decoder | #param (train/total) | crop size | mIoU         |\n  | --------------------- | :-----: | :------------------: | :-------: | ------------ |\n  | OpenCLIP-G (frozen)   | Linear  |     0.3M / 1.8B      |    512    | 39.3         |\n  | ViT-22B (frozen)      | Linear  |     0.9M / 21.7B     |    504    | 34.6         |\n  | InternViT-6B (frozen) | Linear  |     0.5M / 5.9B      |    504    | 47.2 (+12.6) |\n  | ViT-22B (frozen)      | UperNet |     0.8B / 22.5B     |    504    | 52.7         |\n  | InternViT-6B (frozen) | UperNet |     0.4B / 6.3B      |    504    | 54.9 (+2.2)  |\n  | ViT-22B               | UperNet |    22.5B / 22.5B     |    504    | 55.3         |\n  | InternViT-6B          | UperNet |     6.3B / 6.3B      |    504    | 58.9 (+3.6)  |\n\n- Zero-Shot Image Classification [\\[see details\\]](./clip_benchmark#imagenet-variants-and-objectnet)\n\n  | method            | IN-1K | IN-A | IN-R | IN-V2 | IN-Sketch | ObjectNet |\n  | ----------------- | :---: | :--: | :--: | :---: | :-------: | :-------: |\n  | OpenCLIP-G        | 80.1  | 69.3 | 92.1 | 73.6  |   68.9    |   73.0    |\n  | EVA-02-CLIP-E+    | 82.0  | 82.1 | 94.5 | 75.7  |   71.6    |   79.6    |\n  | ViT-22B\\*         | 85.9  | 90.1 | 96.0 | 80.9  |     -     |   87.6    |\n  | InternVL-C (ours) | 83.2  | 83.8 | 95.5 | 77.3  |   73.9    |   80.6    |\n\n- Multilingual Zero-Shot Image Classification [\\[see details\\]](./clip_benchmark#multilingual-imagenet-1k)\n\n  EN: English, ZH: Chinese, JP: Japanese, Ar: Arabic, IT: Italian\n\n  | method            | IN-1K (EN) | IN-1K (ZH) | IN-1K (JP) | IN-1K (AR) | IN-1K (IT) |\n  | ----------------- | :--------: | :--------: | :--------: | :--------: | :--------: |\n  | Taiyi-CLIP-ViT-H  |     -      |    54.4    |     -      |     -      |     -      |\n  | WuKong-ViT-L-G    |     -      |    57.5    |     -      |     -      |     -      |\n  | CN-CLIP-ViT-H     |     -      |    59.6    |     -      |     -      |     -      |\n  | AltCLIP-ViT-L     |    74.5    |    59.6    |     -      |     -      |     -      |\n  | EVA-02-CLIP-E+    |    82.0    |     -      |     -      |     -      |    41.2    |\n  | OpenCLIP-XLM-R-H  |    77.0    |    55.7    |    53.1    |    37.0    |    56.8    |\n  | InternVL-C (ours) |    83.2    |    64.5    |    61.5    |    44.9    |    65.7    |\n\n- Zero-Shot Video Classification\n\n  | method            | #frame | K400 | K600 | K700 |\n  | ----------------- | :----: | :--: | :--: | :--: |\n  | OpenCLIP-G        |   1    | 65.9 | 66.1 | 59.2 |\n  | EVA-02-CLIP-E+    |   1    | 69.8 | 69.3 | 63.4 |\n  | InternVL-C (ours) |   1    | 71.0 | 71.3 | 65.7 |\n  | ViCLIP            |   8    | 75.7 | 73.5 | 66.4 |\n  | InternVL-C (ours) |   8    | 79.4 | 78.8 | 71.5 |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eCross-Modal Retrieval (click to expand)\u003c/summary\u003e\n\n- English Zero-Shot Image-Text Retrieval [\\[see details\\]](./clip_benchmark#flickr30k--coco)\n\n  \u003ctable\u003e\n    \u003ctr align=center\u003e\n        \u003ctd rowspan=\"3\" align=left\u003e\u003cb\u003emodel\u003c/b\u003e\u003c/td\u003e\n        \u003ctd colspan=\"6\" align=center\u003e\u003cb\u003eFlickr30K\u003c/b\u003e\u003c/td\u003e\n        \u003ctd colspan=\"6\" align=center\u003e\u003cb\u003eCOCO\u003c/b\u003e\u003c/td\u003e\n        \u003ctd rowspan=\"3\" align=center\u003e\u003cb\u003eavg\u003c/b\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n     \u003ctr align=center\u003e\n        \u003ctd colspan=\"3\" align=center\u003e\u003cb\u003eimage-to-text\u003c/b\u003e\u003c/td\u003e\n        \u003ctd colspan=\"3\" align=center\u003e\u003cb\u003etext-to-image\u003c/b\u003e\u003c/td\u003e\n         \u003ctd colspan=\"3\" align=center\u003e\u003cb\u003eimage-to-text\u003c/b\u003e\u003c/td\u003e\n        \u003ctd colspan=\"3\" align=center\u003e\u003cb\u003etext-to-image\u003c/b\u003e\u003c/td\u003e\n     \u003c/tr\u003e\n     \u003ctr\u003e\n        \u003ctd\u003eR@1\u003c/td\u003e\n        \u003ctd\u003eR@5\u003c/td\u003e\n        \u003ctd\u003eR@10\u003c/td\u003e\n        \u003ctd\u003eR@1\u003c/td\u003e\n        \u003ctd\u003eR@5\u003c/td\u003e\n        \u003ctd\u003eR@10\u003c/td\u003e\n        \u003ctd\u003eR@1\u003c/td\u003e\n        \u003ctd\u003eR@5\u003c/td\u003e\n        \u003ctd\u003eR@10\u003c/td\u003e\n        \u003ctd\u003eR@1\u003c/td\u003e\n        \u003ctd\u003eR@5\u003c/td\u003e\n        \u003ctd\u003eR@10\u003c/td\u003e\n     \u003c/tr\u003e\n  \u003ctr align=center\u003e\n        \u003ctd align=left\u003eOpenCLIP-G\u003c/td\u003e\n        \u003ctd\u003e92.9\u003c/td\u003e\n        \u003ctd\u003e99.3\u003c/td\u003e\n        \u003ctd\u003e99.8\u003c/td\u003e\n        \u003ctd\u003e79.5\u003c/td\u003e\n        \u003ctd\u003e95.0\u003c/td\u003e\n        \u003ctd\u003e97.1\u003c/td\u003e\n        \u003ctd\u003e67.3\u003c/td\u003e\n        \u003ctd\u003e86.9\u003c/td\u003e\n        \u003ctd\u003e92.6\u003c/td\u003e\n        \u003ctd\u003e51.4\u003c/td\u003e\n        \u003ctd\u003e74.9\u003c/td\u003e\n        \u003ctd\u003e83.0\u003c/td\u003e\n        \u003ctd\u003e85.0\u003c/td\u003e\n     \u003c/tr\u003e\n  \u003ctr align=center\u003e\n        \u003ctd align=left\u003eEVA-02-CLIP-E+\u003c/td\u003e\n        \u003ctd\u003e93.9\u003c/td\u003e\n        \u003ctd\u003e99.4\u003c/td\u003e\n        \u003ctd\u003e99.8\u003c/td\u003e\n        \u003ctd\u003e78.8\u003c/td\u003e\n        \u003ctd\u003e94.2\u003c/td\u003e\n        \u003ctd\u003e96.8\u003c/td\u003e\n        \u003ctd\u003e68.8\u003c/td\u003e\n        \u003ctd\u003e87.8\u003c/td\u003e\n        \u003ctd\u003e92.8\u003c/td\u003e\n        \u003ctd\u003e51.1\u003c/td\u003e\n        \u003ctd\u003e75.0\u003c/td\u003e\n        \u003ctd\u003e82.7\u003c/td\u003e\n        \u003ctd\u003e85.1\u003c/td\u003e\n     \u003c/tr\u003e\n    \u003ctr align=center\u003e\n        \u003ctd align=left\u003eEVA-CLIP-8B\u003c/td\u003e\n        \u003ctd\u003e95.6\u003c/td\u003e\n        \u003ctd\u003e99.6\u003c/td\u003e\n        \u003ctd\u003e99.9\u003c/td\u003e\n        \u003ctd\u003e80.8\u003c/td\u003e\n        \u003ctd\u003e95.5\u003c/td\u003e\n        \u003ctd\u003e97.6\u003c/td\u003e\n        \u003ctd\u003e70.3\u003c/td\u003e\n        \u003ctd\u003e89.3\u003c/td\u003e\n        \u003ctd\u003e93.9\u003c/td\u003e\n        \u003ctd\u003e53.0\u003c/td\u003e\n        \u003ctd\u003e76.0\u003c/td\u003e\n        \u003ctd\u003e83.4\u003c/td\u003e\n        \u003ctd\u003e86.2\u003c/td\u003e\n     \u003c/tr\u003e\n  \u003ctr align=center\u003e\n        \u003ctd align=left\u003eInternVL-C (ours)\u003c/td\u003e\n        \u003ctd\u003e94.7\u003c/td\u003e\n        \u003ctd\u003e99.6\u003c/td\u003e\n        \u003ctd\u003e99.9\u003c/td\u003e\n        \u003ctd\u003e81.7\u003c/td\u003e\n        \u003ctd\u003e96.0\u003c/td\u003e\n        \u003ctd\u003e98.2\u003c/td\u003e\n        \u003ctd\u003e70.6\u003c/td\u003e\n        \u003ctd\u003e89.0\u003c/td\u003e\n        \u003ctd\u003e93.5\u003c/td\u003e\n        \u003ctd\u003e54.1\u003c/td\u003e\n        \u003ctd\u003e77.3\u003c/td\u003e\n        \u003ctd\u003e84.6\u003c/td\u003e\n        \u003ctd\u003e86.6\u003c/td\u003e\n     \u003c/tr\u003e\n  \u003ctr align=center\u003e\n        \u003ctd align=left\u003eInternVL-G (ours)\u003c/td\u003e\n        \u003ctd\u003e95.7\u003c/td\u003e\n        \u003ctd\u003e99.7\u003c/td\u003e\n        \u003ctd\u003e99.9\u003c/td\u003e\n        \u003ctd\u003e85.0\u003c/td\u003e\n        \u003ctd\u003e97.0\u003c/td\u003e\n        \u003ctd\u003e98.6\u003c/td\u003e\n        \u003ctd\u003e74.9\u003c/td\u003e\n        \u003ctd\u003e91.3\u003c/td\u003e\n        \u003ctd\u003e95.2\u003c/td\u003e\n        \u003ctd\u003e58.6\u003c/td\u003e\n        \u003ctd\u003e81.3\u003c/td\u003e\n        \u003ctd\u003e88.0\u003c/td\u003e\n        \u003ctd\u003e88.8\u003c/td\u003e\n     \u003c/tr\u003e\n\n  \u003c/table\u003e\n\n- Chinese Zero-Shot Image-Text Retrieval [\\[see details\\]](./clip_benchmark#flickr30k-cn--coco-cn)\n\n  \u003ctable\u003e\n    \u003ctr  align=center\u003e\n        \u003ctd rowspan=\"3\" align=left\u003e\u003cb\u003emodel\u003c/b\u003e\u003c/td\u003e\n        \u003ctd colspan=\"6\" align=center\u003e\u003cb\u003eFlickr30K-CN\u003c/b\u003e\u003c/td\u003e\n        \u003ctd colspan=\"6\" align=center\u003e\u003cb\u003eCOCO-CN\u003c/b\u003e\u003c/td\u003e\n        \u003ctd rowspan=\"3\" align=center\u003e\u003cb\u003eavg\u003c/b\u003e\u003c/td\u003e\n\n  \u003c/tr\u003e\n     \u003ctr  align=center\u003e\n        \u003ctd colspan=\"3\" align=center\u003e\u003cb\u003eimage-to-text\u003c/b\u003e\u003c/td\u003e\n        \u003ctd colspan=\"3\" align=center\u003e\u003cb\u003etext-to-image\u003c/b\u003e\u003c/td\u003e\n         \u003ctd colspan=\"3\" align=center\u003e\u003cb\u003eimage-to-text\u003c/b\u003e\u003c/td\u003e\n        \u003ctd colspan=\"3\" align=center\u003e\u003cb\u003etext-to-image\u003c/b\u003e\u003c/td\u003e\n     \u003c/tr\u003e\n     \u003ctr\u003e\n        \u003ctd\u003eR@1\u003c/td\u003e\n        \u003ctd\u003eR@5\u003c/td\u003e\n        \u003ctd\u003eR@10\u003c/td\u003e\n        \u003ctd\u003eR@1\u003c/td\u003e\n        \u003ctd\u003eR@5\u003c/td\u003e\n        \u003ctd\u003eR@10\u003c/td\u003e\n        \u003ctd\u003eR@1\u003c/td\u003e\n        \u003ctd\u003eR@5\u003c/td\u003e\n        \u003ctd\u003eR@10\u003c/td\u003e\n        \u003ctd\u003eR@1\u003c/td\u003e\n        \u003ctd\u003eR@5\u003c/td\u003e\n        \u003ctd\u003eR@10\u003c/td\u003e\n     \u003c/tr\u003e\n\n  \u003ctr align=center\u003e\n        \u003ctd align=left\u003eCN-CLIP-ViT-H\u003c/td\u003e\n        \u003ctd\u003e81.6\u003c/td\u003e\n        \u003ctd\u003e97.5\u003c/td\u003e\n        \u003ctd\u003e98.8\u003c/td\u003e\n        \u003ctd\u003e71.2\u003c/td\u003e\n        \u003ctd\u003e91.4\u003c/td\u003e\n        \u003ctd\u003e95.5\u003c/td\u003e\n        \u003ctd\u003e63.0\u003c/td\u003e\n        \u003ctd\u003e86.6\u003c/td\u003e\n        \u003ctd\u003e92.9\u003c/td\u003e\n        \u003ctd\u003e69.2\u003c/td\u003e\n        \u003ctd\u003e89.9\u003c/td\u003e\n        \u003ctd\u003e96.1\u003c/td\u003e\n        \u003ctd\u003e86.1\u003c/td\u003e\n     \u003c/tr\u003e\n\n  \u003ctr align=center\u003e\n        \u003ctd align=left\u003eOpenCLIP-XLM-R-H\u003c/td\u003e\n        \u003ctd\u003e86.1\u003c/td\u003e\n        \u003ctd\u003e97.5\u003c/td\u003e\n        \u003ctd\u003e99.2\u003c/td\u003e\n        \u003ctd\u003e71.0\u003c/td\u003e\n        \u003ctd\u003e90.5\u003c/td\u003e\n        \u003ctd\u003e94.9\u003c/td\u003e\n        \u003ctd\u003e70.0\u003c/td\u003e\n        \u003ctd\u003e91.5\u003c/td\u003e\n        \u003ctd\u003e97.0\u003c/td\u003e\n        \u003ctd\u003e66.1\u003c/td\u003e\n        \u003ctd\u003e90.8\u003c/td\u003e\n        \u003ctd\u003e96.0\u003c/td\u003e\n        \u003ctd\u003e87.6\u003c/td\u003e\n     \u003c/tr\u003e\n\n  \u003ctr align=center\u003e\n        \u003ctd align=left\u003eInternVL-C (ours)\u003c/td\u003e\n        \u003ctd\u003e90.3\u003c/td\u003e\n        \u003ctd\u003e98.8\u003c/td\u003e\n        \u003ctd\u003e99.7\u003c/td\u003e\n        \u003ctd\u003e75.1\u003c/td\u003e\n        \u003ctd\u003e92.9\u003c/td\u003e\n        \u003ctd\u003e96.4\u003c/td\u003e\n        \u003ctd\u003e68.8\u003c/td\u003e\n        \u003ctd\u003e92.0\u003c/td\u003e\n        \u003ctd\u003e96.7\u003c/td\u003e\n        \u003ctd\u003e68.9\u003c/td\u003e\n        \u003ctd\u003e91.9\u003c/td\u003e\n        \u003ctd\u003e96.5\u003c/td\u003e\n        \u003ctd\u003e89.0\u003c/td\u003e\n     \u003c/tr\u003e\n  \u003ctr align=center\u003e\n        \u003ctd align=left\u003eInternVL-G (ours)\u003c/td\u003e\n        \u003ctd\u003e92.9\u003c/td\u003e\n        \u003ctd\u003e99.4\u003c/td\u003e\n        \u003ctd\u003e99.8\u003c/td\u003e\n        \u003ctd\u003e77.7\u003c/td\u003e\n        \u003ctd\u003e94.8\u003c/td\u003e\n        \u003ctd\u003e97.3\u003c/td\u003e\n        \u003ctd\u003e71.4\u003c/td\u003e\n        \u003ctd\u003e93.9\u003c/td\u003e\n        \u003ctd\u003e97.7\u003c/td\u003e\n        \u003ctd\u003e73.8\u003c/td\u003e\n        \u003ctd\u003e94.4\u003c/td\u003e\n        \u003ctd\u003e98.1\u003c/td\u003e\n        \u003ctd\u003e90.9\u003c/td\u003e\n     \u003c/tr\u003e\n\n  \u003c/table\u003e\n\n- Multilingual Zero-Shot Image-Text Retrieval on XTD [\\[see details\\]](./clip_benchmark#xtd)\n\n  | method            |  EN  |  ES  |  FR  |  ZH  |  IT  |  KO  |  RU  |  JP  | average |\n  | ----------------- | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :-----: |\n  | AltCLIP           | 95.4 | 94.1 | 92.9 | 95.1 | 94.2 | 94.4 | 91.8 | 91.7 |  93.7   |\n  | OpenCLIP-XLM-R-H  | 97.3 | 96.1 | 94.5 | 94.7 | 96.0 | 90.2 | 93.9 | 94.0 |  94.6   |\n  | InternVL-C (ours) | 97.3 | 95.7 | 95.1 | 95.6 | 96.0 | 92.2 | 93.3 | 95.5 |  95.1   |\n  | InternVL-G (ours) | 98.6 | 97.7 | 96.5 | 96.7 | 96.9 | 95.1 | 94.8 | 96.1 |  96.6   |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eMultimodal Dialogue\u003c/summary\u003e\n\n\u003c/details\u003e\n\n## Quick Start with HuggingFace\n\n\u003cdetails\u003e\n  \u003csummary\u003eusing InternViT-6B for visual feature extraction (click to expand)\u003c/summary\u003e\n\n```python\nimport torch\nfrom PIL import Image\nfrom transformers import AutoModel, CLIPImageProcessor\n\nmodel = AutoModel.from_pretrained(\n    'OpenGVLab/InternViT-6B-448px-V2_5',\n    torch_dtype=torch.bfloat16,\n    low_cpu_mem_usage=True,\n    trust_remote_code=True).cuda().eval()\n\nimage = Image.open('./examples/image1.jpg').convert('RGB')\n\nimage_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5')\n\npixel_values = image_processor(images=image, return_tensors='pt').pixel_values\npixel_values = pixel_values.to(torch.bfloat16).cuda()\n\noutputs = model(pixel_values)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eusing InternVL-C(ontrastive) and InternVL-G(enerative) for cross-modal retrieval (click to expand)\u003c/summary\u003e\n\n```python\nimport torch\nfrom PIL import Image\nfrom transformers import AutoModel, CLIPImageProcessor\nfrom transformers import AutoTokenizer\n\n\nmodel = AutoModel.from_pretrained(\n    'OpenGVLab/InternVL-14B-224px',\n    torch_dtype=torch.bfloat16,\n    low_cpu_mem_usage=True,\n    trust_remote_code=True).cuda().eval()\n\nimage_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternVL-14B-224px')\n\ntokenizer = AutoTokenizer.from_pretrained(\n    'OpenGVLab/InternVL-14B-224px', use_fast=False, add_eos_token=True)\ntokenizer.pad_token_id = 0  # set pad_token_id to 0\n\nimages = [\n    Image.open('./examples/image1.jpg').convert('RGB'),\n    Image.open('./examples/image2.jpg').convert('RGB'),\n    Image.open('./examples/image3.jpg').convert('RGB')\n]\nprefix = 'summarize:'\ntexts = [\n    prefix + 'a photo of a red panda',  # English\n    prefix + '一张熊猫的照片',  # Chinese\n    prefix + '二匹の猫の写真'  # Japanese\n]\n\npixel_values = image_processor(images=images, return_tensors='pt').pixel_values\npixel_values = pixel_values.to(torch.bfloat16).cuda()\ninput_ids = tokenizer(texts, return_tensors='pt', max_length=80,\n                      truncation=True, padding='max_length').input_ids.cuda()\n\n# InternVL-C\nlogits_per_image, logits_per_text = model(\n    image=pixel_values, text=input_ids, mode='InternVL-C')\nprobs = logits_per_image.softmax(dim=-1)\n# tensor([[9.9609e-01, 5.2185e-03, 6.0070e-08],\n#         [2.2949e-02, 9.7656e-01, 5.9903e-06],\n#         [3.2932e-06, 7.4863e-05, 1.0000e+00]], device='cuda:0',\n#        dtype=torch.bfloat16, grad_fn=\u003cSoftmaxBackward0\u003e)\n\n# InternVL-G\nlogits_per_image, logits_per_text = model(\n    image=pixel_values, text=input_ids, mode='InternVL-G')\nprobs = logits_per_image.softmax(dim=-1)\n# tensor([[9.9609e-01, 3.1738e-03, 3.6322e-08],\n#         [8.6060e-03, 9.9219e-01, 2.8759e-06],\n#         [1.7583e-06, 3.1233e-05, 1.0000e+00]], device='cuda:0',\n#        dtype=torch.bfloat16, grad_fn=\u003cSoftmaxBackward0\u003e)\n\n# please set add_eos_token to False for generation\ntokenizer.add_eos_token = False\nimage = Image.open('./examples/image1.jpg').convert('RGB')\npixel_values = image_processor(images=image, return_tensors='pt').pixel_values\npixel_values = pixel_values.to(torch.bfloat16).cuda()\n\ntokenized = tokenizer(\"English caption:\", return_tensors='pt')\npred = model.generate(\n    pixel_values=pixel_values,\n    input_ids=tokenized.input_ids.cuda(),\n    attention_mask=tokenized.attention_mask.cuda(),\n    num_beams=5,\n    min_new_tokens=8,\n)\ncaption = tokenizer.decode(pred[0].cpu(), skip_special_tokens=True).strip()\n# English caption: a red panda sitting on top of a wooden platform\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eusing InternVL 2.5 for multimodal chat (click to expand)\u003c/summary\u003e\n\nHere, we take the smaller `OpenGVLab/InternVL2_5-8B` as an example:\n\n```python\nimport numpy as np\nimport torch\nimport torchvision.transforms as T\nfrom decord import VideoReader, cpu\nfrom PIL import Image\nfrom torchvision.transforms.functional import InterpolationMode\nfrom transformers import AutoModel, AutoTokenizer\n\nIMAGENET_MEAN = (0.485, 0.456, 0.406)\nIMAGENET_STD = (0.229, 0.224, 0.225)\n\ndef build_transform(input_size):\n    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD\n    transform = T.Compose([\n        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),\n        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),\n        T.ToTensor(),\n        T.Normalize(mean=MEAN, std=STD)\n    ])\n    return transform\n\ndef find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):\n    best_ratio_diff = float('inf')\n    best_ratio = (1, 1)\n    area = width * height\n    for ratio in target_ratios:\n        target_aspect_ratio = ratio[0] / ratio[1]\n        ratio_diff = abs(aspect_ratio - target_aspect_ratio)\n        if ratio_diff \u003c best_ratio_diff:\n            best_ratio_diff = ratio_diff\n            best_ratio = ratio\n        elif ratio_diff == best_ratio_diff:\n            if area \u003e 0.5 * image_size * image_size * ratio[0] * ratio[1]:\n                best_ratio = ratio\n    return best_ratio\n\ndef dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):\n    orig_width, orig_height = image.size\n    aspect_ratio = orig_width / orig_height\n\n    # calculate the existing image aspect ratio\n    target_ratios = set(\n        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if\n        i * j \u003c= max_num and i * j \u003e= min_num)\n    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])\n\n    # find the closest aspect ratio to the target\n    target_aspect_ratio = find_closest_aspect_ratio(\n        aspect_ratio, target_ratios, orig_width, orig_height, image_size)\n\n    # calculate the target width and height\n    target_width = image_size * target_aspect_ratio[0]\n    target_height = image_size * target_aspect_ratio[1]\n    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]\n\n    # resize the image\n    resized_img = image.resize((target_width, target_height))\n    processed_images = []\n    for i in range(blocks):\n        box = (\n            (i % (target_width // image_size)) * image_size,\n            (i // (target_width // image_size)) * image_size,\n            ((i % (target_width // image_size)) + 1) * image_size,\n            ((i // (target_width // image_size)) + 1) * image_size\n        )\n        # split the image\n        split_img = resized_img.crop(box)\n        processed_images.append(split_img)\n    assert len(processed_images) == blocks\n    if use_thumbnail and len(processed_images) != 1:\n        thumbnail_img = image.resize((image_size, image_size))\n        processed_images.append(thumbnail_img)\n    return processed_images\n\ndef load_image(image_file, input_size=448, max_num=12):\n    image = Image.open(image_file).convert('RGB')\n    transform = build_transform(input_size=input_size)\n    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)\n    pixel_values = [transform(image) for image in images]\n    pixel_values = torch.stack(pixel_values)\n    return pixel_values\n\n# If you have an 80G A100 GPU, you can put the entire model on a single GPU.\n# Otherwise, you need to load a model using multiple GPUs, please refer to the `Multiple GPUs` section.\npath = 'OpenGVLab/InternVL2_5-8B'\nmodel = AutoModel.from_pretrained(\n    path,\n    torch_dtype=torch.bfloat16,\n    low_cpu_mem_usage=True,\n    trust_remote_code=True).eval().cuda()\ntokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)\n\n# set the max number of tiles in `max_num`\npixel_values = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()\ngeneration_config = dict(max_new_tokens=1024, do_sample=False)\n\n# pure-text conversation (纯文本对话)\nquestion = 'Hello, who are you?'\nresponse, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\nquestion = 'Can you tell me a story?'\nresponse, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\n# single-image single-round conversation (单图单轮对话)\nquestion = '\u003cimage\u003e\\nPlease describe the image shortly.'\nresponse = model.chat(tokenizer, pixel_values, question, generation_config)\nprint(f'User: {question}\\nAssistant: {response}')\n\n# single-image multi-round conversation (单图多轮对话)\nquestion = '\u003cimage\u003e\\nPlease describe the image in detail.'\nresponse, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\nquestion = 'Please write a poem according to the image.'\nresponse, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\n# multi-image multi-round conversation, combined images (多图多轮对话，拼接图像)\npixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()\npixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()\npixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)\n\nquestion = '\u003cimage\u003e\\nDescribe the two images in detail.'\nresponse, history = model.chat(tokenizer, pixel_values, question, generation_config,\n                               history=None, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\nquestion = 'What are the similarities and differences between these two images.'\nresponse, history = model.chat(tokenizer, pixel_values, question, generation_config,\n                               history=history, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\n# multi-image multi-round conversation, separate images (多图多轮对话，独立图像)\npixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()\npixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()\npixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)\nnum_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]\n\nquestion = 'Image-1: \u003cimage\u003e\\nImage-2: \u003cimage\u003e\\nDescribe the two images in detail.'\nresponse, history = model.chat(tokenizer, pixel_values, question, generation_config,\n                               num_patches_list=num_patches_list,\n                               history=None, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\nquestion = 'What are the similarities and differences between these two images.'\nresponse, history = model.chat(tokenizer, pixel_values, question, generation_config,\n                               num_patches_list=num_patches_list,\n                               history=history, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\n# batch inference, single image per sample (单图批处理)\npixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()\npixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()\nnum_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]\npixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)\n\nquestions = ['\u003cimage\u003e\\nDescribe the image in detail.'] * len(num_patches_list)\nresponses = model.batch_chat(tokenizer, pixel_values,\n                             num_patches_list=num_patches_list,\n                             questions=questions,\n                             generation_config=generation_config)\nfor question, response in zip(questions, responses):\n    print(f'User: {question}\\nAssistant: {response}')\n\n# video multi-round conversation (视频多轮对话)\ndef get_index(bound, fps, max_frame, first_idx=0, num_segments=32):\n    if bound:\n        start, end = bound[0], bound[1]\n    else:\n        start, end = -100000, 100000\n    start_idx = max(first_idx, round(start * fps))\n    end_idx = min(round(end * fps), max_frame)\n    seg_size = float(end_idx - start_idx) / num_segments\n    frame_indices = np.array([\n        int(start_idx + (seg_size / 2) + np.round(seg_size * idx))\n        for idx in range(num_segments)\n    ])\n    return frame_indices\n\ndef load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=32):\n    vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)\n    max_frame = len(vr) - 1\n    fps = float(vr.get_avg_fps())\n\n    pixel_values_list, num_patches_list = [], []\n    transform = build_transform(input_size=input_size)\n    frame_indices = get_index(bound, fps, max_frame, first_idx=0, num_segments=num_segments)\n    for frame_index in frame_indices:\n        img = Image.fromarray(vr[frame_index].asnumpy()).convert('RGB')\n        img = dynamic_preprocess(img, image_size=input_size, use_thumbnail=True, max_num=max_num)\n        pixel_values = [transform(tile) for tile in img]\n        pixel_values = torch.stack(pixel_values)\n        num_patches_list.append(pixel_values.shape[0])\n        pixel_values_list.append(pixel_values)\n    pixel_values = torch.cat(pixel_values_list)\n    return pixel_values, num_patches_list\n\nvideo_path = './examples/red-panda.mp4'\npixel_values, num_patches_list = load_video(video_path, num_segments=8, max_num=1)\npixel_values = pixel_values.to(torch.bfloat16).cuda()\nvideo_prefix = ''.join([f'Frame-{i+1}: \u003cimage\u003e\\n' for i in range(len(num_patches_list))])\nquestion = video_prefix + 'What is the red panda doing?'\n# Frame1: \u003cimage\u003e\\nFrame2: \u003cimage\u003e\\n...\\nFrame8: \u003cimage\u003e\\n{question}\nresponse, history = model.chat(tokenizer, pixel_values, question, generation_config,\n                               num_patches_list=num_patches_list, history=None, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n\nquestion = 'Describe this video in detail.'\nresponse, history = model.chat(tokenizer, pixel_values, question, generation_config,\n                               num_patches_list=num_patches_list, history=history, return_history=True)\nprint(f'User: {question}\\nAssistant: {response}')\n```\n\n\u003c/details\u003e\n\n## License\n\nThis project is released under the [MIT license](LICENSE). Parts of this project contain code and models from other sources, which are subject to their respective licenses.\n\n## Citation\n\nIf you find this project useful in your research, please consider cite:\n\n```BibTeX\n@article{chen2024expanding,\n  title={Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling},\n  author={Chen, Zhe and Wang, Weiyun and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Cui, Erfei and Zhu, Jinguo and Ye, Shenglong and Tian, Hao and Liu, Zhaoyang and others},\n  journal={arXiv preprint arXiv:2412.05271},\n  year={2024}\n}\n@article{wang2024mpo,\n  title={Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization},\n  author={Wang, Weiyun and Chen, Zhe and Wang, Wenhai and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Zhu, Jinguo and Zhu, Xizhou and Lu, Lewei and Qiao, Yu and Dai, Jifeng},\n  journal={arXiv preprint arXiv:2411.10442},\n  year={2024}\n}\n@article{gao2024mini,\n  title={Mini-InternVL: a flexible-transfer pocket multi-modal model with 5\\% parameters and 90\\% performance},\n  author={Gao, Zhangwei and Chen, Zhe and Cui, Erfei and Ren, Yiming and Wang, Weiyun and Zhu, Jinguo and Tian, Hao and Ye, Shenglong and He, Junjun and Zhu, Xizhou and others},\n  journal={Visual Intelligence},\n  volume={2},\n  number={1},\n  pages={1--17},\n  year={2024},\n  publisher={Springer}\n}\n@article{chen2024far,\n  title={How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites},\n  author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},\n  journal={Science China Information Sciences},\n  volume={67},\n  number={12},\n  pages={220101},\n  year={2024},\n  publisher={Springer}\n}\n@inproceedings{chen2024internvl,\n  title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},\n  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={24185--24198},\n  year={2024}\n}\n```\n\n## Acknowledgement\n\nInternVL is built with reference to the code of the following projects: [OpenAI CLIP](https://github.com/openai/CLIP), [Open CLIP](https://github.com/mlfoundations/open_clip), [CLIP Benchmark](https://github.com/LAION-AI/CLIP_benchmark), [EVA](https://github.com/baaivision/EVA/tree/master), [InternImage](https://github.com/OpenGVLab/InternImage), [ViT-Adapter](https://github.com/czczup/ViT-Adapter), [MMSegmentation](https://github.com/open-mmlab/mmsegmentation), [Transformers](https://github.com/huggingface/transformers), [DINOv2](https://github.com/facebookresearch/dinov2), [BLIP-2](https://github.com/salesforce/LAVIS/tree/main/projects/blip2), [Qwen-VL](https://github.com/QwenLM/Qwen-VL/tree/master/eval_mm), and [LLaVA-1.5](https://github.com/haotian-liu/LLaVA). Thanks for their awesome work!\n\n______________________________________________________________________\n\nScan the following QR Code, join our WeChat group.\n\n\u003cp align=\"center\"\u003e\u003cimg width=\"300\" alt=\"image\" src=\"https://github.com/user-attachments/assets/f776df09-ebba-4fd5-80c2-fec4ff1518be\"\u003e\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenGVLab%2FInternVL","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenGVLab%2FInternVL","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenGVLab%2FInternVL/lists"}