{"id":24009381,"url":"https://github.com/ictnlp/llava-mini","last_synced_at":"2025-05-16T04:05:56.429Z","repository":{"id":271441627,"uuid":"913468419","full_name":"ictnlp/LLaVA-Mini","owner":"ictnlp","description":"LLaVA-Mini is a unified large multimodal model (LMM) that can support the understanding of images, high-resolution images, and videos in an efficient manner. ","archived":false,"fork":false,"pushed_at":"2025-01-13T03:12:38.000Z","size":57210,"stargazers_count":474,"open_issues_count":24,"forks_count":21,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-05-16T04:05:47.555Z","etag":null,"topics":["efficient","gpt4o","gpt4v","large-language-models","large-multimodal-models","llama","llava","multimodal","multimodal-large-language-models","video","vision","vision-language-model","visual-instruction-tuning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ictnlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-07T18:37:05.000Z","updated_at":"2025-05-13T02:35:12.000Z","dependencies_parsed_at":"2025-02-25T13:50:52.925Z","dependency_job_id":null,"html_url":"https://github.com/ictnlp/LLaVA-Mini","commit_stats":null,"previous_names":["ictnlp/llava-mini"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FLLaVA-Mini","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FLLaVA-Mini/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FLLaVA-Mini/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FLLaVA-Mini/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ictnlp","download_url":"https://codeload.github.com/ictnlp/LLaVA-Mini/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254464895,"owners_count":22075570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["efficient","gpt4o","gpt4v","large-language-models","large-multimodal-models","llama","llava","multimodal","multimodal-large-language-models","video","vision","vision-language-model","visual-instruction-tuning"],"created_at":"2025-01-08T03:45:08.070Z","updated_at":"2025-05-16T04:05:51.420Z","avatar_url":"https://github.com/ictnlp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token\n\n[![arXiv](https://img.shields.io/badge/arXiv-2501.03895-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2501.03895)\n[![model](https://img.shields.io/badge/%F0%9F%A4%97%20huggingface%20-llava--mini--llama--3.1--8b-orange.svg)](https://huggingface.co/ICTNLP/llava-mini-llama-3.1-8b)\n[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fictnlp%2FLLaVA-Mini\u0026count_bg=%2379C83D\u0026title_bg=%23555555\u0026icon=awesomelists.svg\u0026icon_color=%23E7E7E7\u0026title=Vistors\u0026edge_flat=false)](https://github.com/ictnlp/LLaVA-Mini)\n\n\u003e **[Shaolei Zhang](https://zhangshaolei1998.github.io/), [Qingkai Fang](https://fangqingkai.github.io/), [Zhe Yang](https://nlp.ict.ac.cn/yjdw/xs/ssyjs/202210/t20221020_52708.html), [Yang Feng*](https://people.ucas.edu.cn/~yangfeng?language=en)**\n\n\nLLaVA-Mini is a unified large multimodal model that can support the understanding of images, high-resolution images, and videos in an efficient manner. Guided by the interpretability within LMM, LLaVA-Mini significantly improves efficiency while ensuring vision capabilities. [Model](https://huggingface.co/ICTNLP/llava-mini-llama-3.1-8b) and [demo](#-demo) of LLaVA-Mini are available now!\n\n\u003e [!Note]\n\u003e LLaVA-Mini only requires **1 token** to represent each image, which improves the efficiency of image and video understanding, including:\n\u003e - **Computational effort**: 77% FLOPs reduction\n\u003e - **Response latency**: reduce from 100 milliseconds to 40 milliseconds\n\u003e - **VRAM memory usage**: reduce from 360 MB/image to 0.6 MB/image, support 3-hour video processing\n\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/performance.png\" alt=\"performance\" style=\"width: 100%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n💡**Highlight**:\n1. **Good Performance**: LLaVA-Mini achieves performance comparable to LLaVA-v1.5 while using only 1 vision token instead of 576 (compression rate of 0.17%).\n2. **High Efficiency**: LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on the GPU hardware with 24GB of memory.\n3. **Insights**: To develop LLaVA-Mini, which reduces vision tokens while maintaining visual understanding, we conduct a preliminary analysis to explore how large multimodal models (LMMs) process visual tokens. Please refer to our [paper](https://arxiv.org/pdf/2501.03895) for a detailed analysis and our conclusions.\n\n## 🖥 Demo\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/llava_mini.gif\" alt=\"llava_mini\" style=\"width: 100%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n- Download LLaVA-Mini model from [here](https://huggingface.co/ICTNLP/llava-mini-llama-3.1-8b).\n\n- Run these scripts and Interact with LLaVA-Mini in your browser:\n\n  ```bash\n  # Launch a controller\n  python -m llavamini.serve.controller --host 0.0.0.0 --port 10000 \u0026\n\n  # Build the API of LLaVA-Mini, if the VRAM memory is less than 20GB, try using --load-8bit\n  CUDA_VISIBLE_DEVICES=0  python -m llavamini.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ICTNLP/llava-mini-llama-3.1-8b --model-name llava-mini \u0026\n\n  # Start the interactive interface\n  python -m llavamini.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload  --port 7860\n  ```\n\n## 🔥 Quick Start\n### Requirements\n- Install packages:\n\n  ```bash\n  conda create -n llavamini python=3.10 -y\n  conda activate llavamini\n  pip install -e .\n  pip install -e \".[train]\"\n  pip install flash-attn --no-build-isolation\n  ```\n\n### Command Interaction\n- Image understanding, using `--image-file`.\n- If the VRAM memory is less than 20GB, try using `--load-8bit`.\n\n  ```bash\n  # Image Understanding\n  CUDA_VISIBLE_DEVICES=0 python llavamini/eval/run_llava_mini.py \\\n      --model-path  ICTNLP/llava-mini-llama-3.1-8b \\\n      --image-file llavamini/serve/examples/baby_cake.png \\\n      --conv-mode llava_llama_3_1 --model-name \"llava-mini\" \\\n      --query \"What's the text on the cake?\"\n  ```\n\n- Video understanding, using `--video-file`:\n\n  ```bash\n  # Video Understanding\n  CUDA_VISIBLE_DEVICES=0 python llavamini/eval/run_llava_mini.py \\\n      --model-path  ICTNLP/llava-mini-llama-3.1-8b \\\n      --video-file llavamini/serve/examples/fifa.mp4 \\\n      --conv-mode llava_llama_3_1 --model-name \"llava-mini\" \\\n      --query \"What happened in this video?\"\n  ```\n\n### Reproduction and Evaluation\n\n- Refer to [Evaluation.md](docs/Evaluation.md) for the evaluation of LLaVA-Mini on image/video benchmarks.\n\n### Cases\n- LLaVA-Mini achieves high-quality image understanding and video understanding.\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/case1.png\" alt=\"case1\" style=\"width: 100%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eMore cases\u003c/summary\u003e\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/case2.png\" alt=\"case2\" style=\"width: 100%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/case3.png\" alt=\"case3\" style=\"width: 100%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/case4.png\" alt=\"case4\" style=\"width: 100%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n\u003c/details\u003e\n\n- LLaVA-Mini dynamically compresses image to capture important visual information (brighter areas are more heavily weighted during compression).\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/compression.png\" alt=\"compression\" style=\"width: 100%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n## 🤝 Acknowledgement\n- [LLaVA](https://github.com/haotian-liu/LLaVA): LLaVA-Mini is built upon LLaVA codebase, a large language and vision assistant.\n- [Video-ChatGPT](https://github.com/mbzuai-oryx/Video-ChatGPT): The training of LLaVA-Mini involves the video instruction data provided by Video-ChatGPT.\n- [LLaVA-OneVision](https://github.com/LLaVA-VL/LLaVA-NeXT): The training of LLaVA-Mini involves the image instruction data provided by LLaVA-OneVision.\n\n## 🖋Citation\n\nIf this repository is useful for you, please cite as:\n\n```\n@misc{llavamini,\n      title={LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token}, \n      author={Shaolei Zhang and Qingkai Fang and Zhe Yang and Yang Feng},\n      year={2025},\n      eprint={2501.03895},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2501.03895}, \n}\n```\n\nIf you have any questions, please feel free to submit an issue or contact `zhangshaolei20z@ict.ac.cn`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fictnlp%2Fllava-mini","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fictnlp%2Fllava-mini","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fictnlp%2Fllava-mini/lists"}