{"id":31745436,"url":"https://github.com/evolvinglmms-lab/llava-onevision-1.5","last_synced_at":"2025-12-26T04:21:36.191Z","repository":{"id":315064641,"uuid":"1058015331","full_name":"EvolvingLMMs-Lab/LLaVA-OneVision-1.5","owner":"EvolvingLMMs-Lab","description":"Fully Open Framework for Democratized Multimodal Training","archived":false,"fork":false,"pushed_at":"2025-09-30T11:40:33.000Z","size":5219,"stargazers_count":414,"open_issues_count":8,"forks_count":26,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-02T00:25:45.213Z","etag":null,"topics":["llava","llm","mllm","qwen3","vision-language-model"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EvolvingLMMs-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-16T14:05:47.000Z","updated_at":"2025-10-01T23:21:15.000Z","dependencies_parsed_at":"2025-09-16T14:42:18.869Z","dependency_job_id":"b874cb40-f013-429e-bb0e-255813c37b72","html_url":"https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5","commit_stats":null,"previous_names":["evolvinglmms-lab/llava-onevision-1.5"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EvolvingLMMs-Lab/LLaVA-OneVision-1.5","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvolvingLMMs-Lab%2FLLaVA-OneVision-1.5","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvolvingLMMs-Lab%2FLLaVA-OneVision-1.5/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvolvingLMMs-Lab%2FLLaVA-OneVision-1.5/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvolvingLMMs-Lab%2FLLaVA-OneVision-1.5/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EvolvingLMMs-Lab","download_url":"https://codeload.github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvolvingLMMs-Lab%2FLLaVA-OneVision-1.5/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001416,"owners_count":26083078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llava","llm","mllm","qwen3","vision-language-model"],"created_at":"2025-10-09T12:56:40.413Z","updated_at":"2025-12-26T04:21:36.185Z","avatar_url":"https://github.com/EvolvingLMMs-Lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"asset/llava_onevision_black.png\"\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"asset/llava_onevision_white.png\"\u003e\n    \u003cimg alt=\"LLaVA-OneVision-1.5\" src=\"output/llava_onevision_white.png\" width=\"600\" style=\"max-width: 100%;\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eFully Open Framework for Democratized Multimodal Training\u003c/strong\u003e\n\u003c/p\u003e\n\n\n\n\u003cdiv align=\"center\"\u003e\n\n🤗 **[Models and Datasets](https://huggingface.co/collections/lmms-lab/llava-onevision-15-68d385fe73b50bd22de23713)** |\n🖥️ **[Demo](https://huggingface.co/spaces/lmms-lab/LLaVA-OneVision-1.5)** |\n📄 **[Technical Report](https://arxiv.org/abs/2509.23661)** |\n📰 **[Zhihu](https://www.zhihu.com/question/1959577143697707446)** |\n📕 **[Xiaohongshu](http://xhslink.com/o/4nXL6EXDTqv)**\n\n\u003c/div\u003e\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003c!-- Mid-Training Dataset Downloads --\u003e\n  \u003ca href=\"https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Mid-Training-85M\"\u003e\n    \u003cimg alt=\"HF Mid-Training Dataset Downloads\" src=\"https://img.shields.io/badge/dynamic/json?url=https://huggingface.co/api/datasets/mvp-lab/LLaVA-OneVision-1.5-Mid-Training-85M\u0026amp;query=downloads\u0026amp;label=Mid%20Training%20DATA%20Downloads\u0026amp;color=green\u0026amp;logo=huggingface\u0026amp\"\u003e\n  \u003c/a\u003e\n  \u003c!-- Instruct Dataset Downloads --\u003e\n  \u003ca href=\"https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Instruct-Data\"\u003e\n    \u003cimg alt=\"HF Instruct Dataset Downloads\" src=\"https://img.shields.io/badge/dynamic/json?url=https://huggingface.co/api/datasets/mvp-lab/LLaVA-OneVision-1.5-Instruct-Data\u0026amp;query=downloads\u0026amp;label=Instruct%20DATA%20Downloads\u0026amp;color=blue\u0026amp;logo=huggingface\u0026amp\"\u003e\n  \u003c/a\u003e\n  \u003c!-- Model Downloads --\u003e\n  \u003ca href=\"https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct\"\u003e\n    \u003cimg alt=\"HF Model Downloads\" src=\"https://img.shields.io/badge/dynamic/json?url=https://huggingface.co/api/models/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct\u0026amp;query=downloads\u0026amp;label=OV-1.5-8B-Instruct%20Downloads\u0026amp;color=yellow\u0026amp;logo=huggingface\u0026amp\"\u003e\n  \u003c/a\u003e\n  \u003c!-- Training Cost --\u003e\n  \u003cimg alt=\"Training Cost\" src=\"https://img.shields.io/badge/Full%20Train%20Cost-~$16K-success\"\u003e\n  \u003c!-- License --\u003e\n  \u003ca href=\"LICENSE\"\u003e\n    \u003cimg alt=\"License\" src=\"https://img.shields.io/badge/License-Apache--2.0-blue.svg?logo=apache\u0026amp\"\u003e\n  \u003c/a\u003e\n  \u003c!-- PRs Welcome --\u003e\n  \u003ca href=\"https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5/pulls\"\u003e\n    \u003cimg alt=\"PRs Welcome\" src=\"https://img.shields.io/badge/PRs-welcome-brightgreen.svg?logo=github\u0026amp\"\u003e\n  \u003c/a\u003e\n  \u003c!-- Commit Activity --\u003e\n  \u003ca href=\"https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5/commits\"\u003e\n    \u003cimg alt=\"Commit Activity\" src=\"https://img.shields.io/github/commit-activity/m/EvolvingLMMs-Lab/LLaVA-OneVision-1.5?logo=github\u0026amp\"\u003e\n  \u003c/a\u003e\n  \u003c!-- Contributors --\u003e\n  \u003ca href=\"https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5/graphs/contributors\"\u003e\n    \u003cimg alt=\"Contributors\" src=\"https://img.shields.io/github/contributors/EvolvingLMMs-Lab/LLaVA-OneVision-1.5?logo=github\u0026amp\"\u003e\n  \u003c/a\u003e\n  \u003c!-- Megatron-LM Optimization --\u003e\n  \u003ca href=\"https://github.com/NVIDIA/Megatron-LM\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Megatron--LM-mcore%20optimized-1560b9?logo=nvidia\u0026amp\" alt=\"Megatron-LM mcore optimized\"\u003e\n  \u003c/a\u003e\n  \u003c!-- ModelScope Collection --\u003e\n  \u003ca href=\"https://www.modelscope.cn/collections/LLaVA-OneVision-15-ff6ede3d20a643\" target=\"_blank\"\u003e\n    \u003cimg alt=\"ModelScope Collection\" src=\"https://img.shields.io/badge/ModelScope-Collection-orange?logo=modelscope\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n\n## NEWS\n- 2025-12-11: Released [RL recipe for LLaVA-OneVision-1.5](https://mvp-ai-lab.github.io/LLaVA-OneVision-1.5-RL/) with [code](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5-RL), [data](https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-RL-Data), and [model](https://huggingface.co/mvp-lab/LLaVA-OneVision-1.5-8B-RL).\n- 2025-09-30: Released the [Offline Data Packing Guide](examples_offline_packing).\n- 2025-09-30: Released the LLaVA-OneVision-1.5 [Technical Report](https://arxiv.org/abs/2509.23661).\n\n\n## Contents\n\u003c!-- TOC (no expansion for Quick Start Guide / Fully Reproducing Guide) --\u003e\n- [Introduction](#introduction)\n- [Models](#models)\n- [Datasets](#datasets)\n- [Results](#evaluation-results)\n- [Quick Start with Hugging Face](#quick-start-with-huggingface)\n- [Evaluation](#evaluation)\n- [Quick Start For Training](#quick-start-guide)\n- [Fully Reproducing Guide](#fully-reproducing-guide)\n- [Citation](#citation)\n- [Acknowledgement](#acknowledgement)\n\n\n## Introduction\n**LLaVA-OneVision-1.5** introduces a family of fully open-source large multimodal models (LMMs) that operate on **native-resolution images**, achieve **state-of-the-art** performance, and require comparatively **lower training costs**.\n\n#### **Superior Performance**\n  - The model leads on multiple multimodal benchmarks and generally surpasses Qwen2.5-VL.\n  - Training on native-resolution images significantly improves its visual understanding.\n\n#### **High-Quality Data at Scale**\n  - The pretraining corpus comprises large-scale, concept-balanced, diverse, and high-quality captions curated with strict filtering and quality control.\n  - The instruction-tuning dataset is comprehensive and covers a wide range of tasks.\n\n#### **Ultra-Efficient Training Framework**\n  - The end-to-end training cost is about $16,000 on A100 GPUs at roughly $0.60 per GPU-hour.\n  - The system is built on Megatron-LM with support for MoE, FP8, and long-sequence parallelism, and the codebase is optimized for cost-effective scaling.\n\n#### **Fully Open Framework**\n  - The project releases high-quality pretraining and SFT datasets along with the complete training framework, configurations, and recipes.\n  - It also provides detailed training logs and metrics to enable reproducibility and community adoption.\n\n\n## Models\n\n| Model                    | HF Link                                                                                      | Training Log |\n|--------------------------|--------------------------------------------------------------------------------------------------------|-------------|\n| LLaVA-OneVision-1.5-4B-Instruct | [🤗 HF / 4B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct)                | [📈 TensorBoard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct/tensorboard) |\n| LLaVA-OneVision-1.5-8B-Instruct | [🤗 HF / 8B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct)                | [📈 TensorBoard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct/tensorboard) |\n| LLaVA-OneVision-1.5-4B-Base     | [🤗 HF / 4B-Base](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Base)                        | [📈 TensorBoard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct/tensorboard) |\n| LLaVA-OneVision-1.5-8B-Base     | [🤗 HF / 8B-Base](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Base)                        | [📈 TensorBoard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct/tensorboard) |\n## Datasets\n\n![Dataset Visualization](asset/dataset.jpg)\n\u003cp align=\"left\"\u003e\n  \u003cstrong\u003e(a)\u003c/strong\u003e The vocabulary coverage proportion in the LLaVA-OneVision-1.5 Mid-Training dataset before and after concept balancing.\n  \u003cstrong\u003e(b)\u003c/strong\u003e Distribution of data sources within the LLaVA-OneVision-1.5 Mid-Training dataset.\n  \u003cstrong\u003e(c)\u003c/strong\u003e Distribution of data sources within the LLaVA-OneVision-1.5 Instruct dataset.\n\u003c/p\u003e\n\n| Description        | Link                                                                                                   | Status      |\n|--------------------|--------------------------------------------------------------------------------------------------------|-------------|\n| LLaVA-OneVision-1.5-Mid-Training-85M   | [🤗HF / Mid-Training 85M](https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Mid-Training-85M) | Available  |\n| LLaVA-OneVision-1.5-Instruct           | [🤗HF / Instruct-Data](https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Instruct-Data)        | Available  |\n\n\n## Evaluation Results\n\n\nAll evaluations were conducted using [lmms_eval](https://github.com/EvolvingLMMs-Lab/lmms-eval).\n\n![](asset/performance.png)\n\n\n## Quick Start with HuggingFace\n\n```python\nfrom transformers import AutoTokenizer, AutoProcessor, AutoModelForCausalLM\nfrom qwen_vl_utils import process_vision_info\nmodel_path = \"lmms-lab/LLaVA-OneVision-1.5-8B-Instruct\"\n\n# default: Load the model on the available device(s)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_path, torch_dtype=\"auto\", device_map=\"auto\", trust_remote_code=True\n)\n\n# default processor\nprocessor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)\n\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"image\",\n                \"image\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg\",\n            },\n            {\"type\": \"text\", \"text\": \"Describe this image.\"},\n        ],\n    }\n]\n\n# Preparation for inference\ntext = processor.apply_chat_template(\n    messages, tokenize=False, add_generation_prompt=True\n)\nimage_inputs, video_inputs = process_vision_info(messages)\ninputs = processor(\n    text=[text],\n    images=image_inputs,\n    videos=video_inputs,\n    padding=True,\n    return_tensors=\"pt\",\n)\ninputs = inputs.to(\"cuda\")\n\n# Inference: Generation of the output\ngenerated_ids = model.generate(**inputs, max_new_tokens=1024)\ngenerated_ids_trimmed = [\n    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n]\noutput_text = processor.batch_decode(\n    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n)\nprint(output_text)\n\n```\n\n## Evaluation\n```\n# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git  \n\naccelerate launch --num_processes=8 --main_process_port 12399 -m lmms_eval \\\n    --model=llava_onevision1_5 \\\n    --model_args=pretrained=lmms-lab/LLaVA-OneVision-1.5-8B-Instruct,attn_implementation=flash_attention_2,max_pixels=3240000 \\\n    --tasks=mmmu_val,mmmu_pro_standard,mmbench_en_test,mmerealworld,mmerealworld_cn,ai2d,ai2d_no_mask,vstar_bench,chartqa,charxiv,docvqa_test,mathvista_testmini,mmstar,scienceqa \\\n    --batch_size=1\n```\n\n\n## Quick Start Guide\n\n### 1.🐳 Docker (Recommended)\n\nWe strongly recommend using the docker environment for a seamless experience. The following instructions are tailored for the A100 80GB GPU environment.\n\n\n```bash\n# Clone repository\ngit clone https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5.git\ncd LLaVA-OneVision-1.5\n\ndocker build -t llava_megatron:25.04 .\n\n# Run container with -w to set working directory directly to the mounted volume\ndocker run -it --gpus all \\\n    --ipc host --net host --privileged --cap-add IPC_LOCK \\\n    --ulimit memlock=-1 --ulimit stack=67108864 --rm \\\n    -v $(pwd):/workspace/LLaVA-OneVision-1.5 \\\n    -w /workspace/LLaVA-OneVision-1.5 \\\n    --name \"llava_megatron_container\" \\\n    llava_megatron:25.04 /bin/bash\n```\n\n### 2. Checkpoint and Format Conversion\n\nYou have two options to get started with LLaVA-OneVision-1.5-stage-0:\n\n#### Option 1: Download pre-trained model from Hugging Face\nDownload our `LLaVA-OneVision-1.5-4B-stage0` model directly from [Hugging Face](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-stage0).\n\n#### Option 2: Merge initial weights yourself\nAlternatively, you can merge the initial weights from the original ViT and LLM:\n```bash\npython ds/merge_model.py \\\n--vit_path DeepGlint-AI/rice-vit-large-patch14-560 \\\n--llm_path Qwen/Qwen3-4B-Instruct-2507 \\\n--output LLaVA-OneVision-1.5-4B-stage0\n```\nNote: When merging weights, the adapter component will be initialized with default values.\n\nConvert the model from Hugging Face format to Megatron format:\n\n```bash\nAIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 bash examples/llava_ov_1_5/convert/convert_4b_hf_to_mcore.sh \\\nLLaVA-OneVision-1.5-4B-stage0 \\\nLLaVA-OneVision-1.5-4B-stage0_mcore_tp1_pp1 \\\n1 1\n```\n\n### 3. Stage 1 Alignment-Training\n\nDownload LLaVA from [LLaVA-558K-Webdataset](https://huggingface.co/datasets/lmms-lab/LLaVA-558K-Webdataset).\n\n\n```bash\n# ============================================================\n# Required environment variables:\n#   AIAK_TRAINING_PATH  Root directory of the AIAK-Training-LLM project\n#   DATA_PATH           Directory with WebDataset shards (.tar) for pretraining\n#   TOKENIZER_PATH      Hugging Face tokenizer directory\n#   CHECKPOINT_PATH     Megatron-formatted checkpoint directory (e.g., mcore TP1/PP1)\n#   SAVE_CKPT_PATH      Output directory for saving training checkpoints\nAIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 \\\nDATA_PATH=LLaVA-558K-Webdataset \\\nTOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \\\nCHECKPOINT_PATH=LLaVA-OneVision-1.5-4B-stage0_mcore_tp1_pp1 \\\nbash examples/llava_ov_1_5/quick_start/stage_1_alignment_llava_ov_4b.sh\n```\n\n### 4. Stage 1.5 Mid-Training \n\nDownload our lightweight packed subset from [LLaVA-OneVision-1.5-Mid-Training-Quick-Start-3M-Webdataset](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-1.5-Mid-Training-Webdataset-Quick-Start-3M).\n\n```bash\n# ============================================================\n# Convert model to release format\nbash examples/llava_ov_1_5/convert/convert_4b_mcore_to_release.sh \\\nstage_1_alignment_llava_ov_4b/iter_0002500/ \\\nstage_1_alignment_llava_ov_4b_release 1 1\n# ============================================================\n# Launch\nAIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 \\\nDATA_PATH=LLaVA-OneVision-1.5-Mid-Training-Webdataset-Quick-Start-3M \\\nTOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \\\nCHECKPOINT_PATH=stage_1_alignment_llava_ov_4b_release \\\nbash examples/llava_ov_1_5/quick_start/stage_1.5_mid_training_llava_ov_4b.sh\n```\n\n\n### 5. Stage 2 Instruct-Training\n\nDownload LLaVA-NeXT-780k-webdataset at [LLaVA-NeXT-780K Dataset](https://huggingface.co/datasets/lmms-lab/LLaVA-NeXT-780k-webdataset).\n\n```bash\n# ============================================================\n# Convert model to release format\nbash examples/llava_ov_1_5/convert/convert_4b_mcore_to_release.sh \\\nstage_1.5_mid_training_llava_ov_4b/iter_0020000/ \\\nstage_1.5_mid_training_llava_ov_4b_release 1 1\n# ============================================================\n# # Launch\nAIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 \\\nDATA_PATH=LLaVA-NeXT-780k-Webdataset \\\nTOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \\\nCHECKPOINT_PATH=stage_1.5_mid_training_llava_ov_4b_release \\\nbash examples/llava_ov_1_5/quick_start/stage_2_instruct_llava_ov_4b.sh\n```\n\n\n### 6. Convert mcore to Hugging Face\n```bash\nAIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 \\\nbash examples/llava_ov_1_5/convert/convert_4b_mcore_to_hf.sh \\\nstage_2_instruct_llava_ov_4b/iter_0003500 \\\nLLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct \\\n1 1\n# Copy non-model files (e.g., tokenizer config) to the new directory\nfind LLaVA-OneVision-1.5-4B-stage0/ -type f -not -iname '*safetensors*' -exec cp {}  LLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct/ ';'\n```\n\n### 7. Evaluation\n```bash\n# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git\nCUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch \\\n--num_processes=4 --main_process_port 12399 -m lmms_eval --model=llava_onevision1_5 --batch_size=1 --tasks=mme \\\n--model_args=pretrained=/workspace/LLaVA-OneVision-1.5/LLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct,max_pixels=3240000\n```\n\n## Fully Reproducing Guide\n\n\u003e [!TIP]\n\u003e More detailed reproduction steps for the complete process will be provided after the dataset upload is completed.\n\n\n### Mid-Training\n\nTo improve model training efficiency, we implement offline sample packing:\n\n1. Download the [**Mid-Training-85M Dataset**](https://huggingface.co/datasets/lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M)\n2. Pack the data into WebDataset format, refer to [**Examples offlinepacking**](examples_offline_packing) and [**Offline Padding-Free Data Packing**](examples/llava_ov_1_5/sample_packing/README.md)\n\n\n### Instruct\n1. Download the [**LLaVA-OneVision-1.5-Instruct-Data**](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-1.5-Instruct-Data)\n2. Convert the data into WebDataset format, refer to [**Conversion for Mixed Instruction Data**](docs/sft_data_preprocessing.md)\n\n## Roadmap\n\nQ4 2025 Key Deliverables:\n\n1. **Ultra-efficient MoE Training**  \n2. **Full Video Input LLM**  \n\n\n## Contributors\nThanks so much to all of our amazing contributors!\n\n\u003c!-- readme: collaborators,contributors,jiankangdeng/- -start --\u003e\n\u003ctable\u003e\n\t\u003ctbody\u003e\n\t\t\u003ctr\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/fdcp\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/15667917?v=4\" width=\"80;\" alt=\"fdcp\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003efdcp\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/anxiangsir\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/31175974?v=4\" width=\"80;\" alt=\"anxiangsir\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003eanxiangsir\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/yiyexy\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/35927125?v=4\" width=\"80;\" alt=\"yiyexy\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003eyiyexy\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/wideyard\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/101321826?v=4\" width=\"80;\" alt=\"wideyard\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003ewideyard\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/Lornatang\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/31124350?v=4\" width=\"80;\" alt=\"Lornatang\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003eLornatang\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/chengzheng345\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/209475443?v=4\" width=\"80;\" alt=\"chengzheng345\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003echengzheng345\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/Luodian\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/15847405?v=4\" width=\"80;\" alt=\"Luodian\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003eLuodian\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/killTheHostage\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/16442720?v=4\" width=\"80;\" alt=\"killTheHostage\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003ekillTheHostage\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n\t\t\u003c/tr\u003e\n\t\t\u003ctr\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/mathCrazyy\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/20607153?v=4\" width=\"80;\" alt=\"mathCrazyy\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003emathCrazyy\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/wkzhang636\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/194186498?v=4\" width=\"80;\" alt=\"wkzhang636\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003ewkzhang636\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/yunglechao\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/7631185?v=4\" width=\"80;\" alt=\"yunglechao\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003eyunglechao\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd align=\"center\"\u003e\n                \u003ca href=\"https://github.com/RobitYadda\"\u003e\n                    \u003cimg src=\"https://avatars.githubusercontent.com/u/6811311?v=4\" width=\"80;\" alt=\"RobitYadda\"/\u003e\n                    \u003cbr /\u003e\n                    \u003csub\u003e\u003cb\u003eRobitYadda\u003c/b\u003e\u003c/sub\u003e\n                \u003c/a\u003e\n            \u003c/td\u003e\n\t\t\u003c/tr\u003e\n\t\u003ctbody\u003e\n\u003c/table\u003e\n\u003c!-- readme: collaborators,contributors,jiankangdeng/- -end --\u003e\n\n## Citation\n\nIf you find *LLaVA-OneVision-1.5* useful in your research, please consider to cite the following related papers:\n\n```\n@inproceedings{LLaVA-OneVision-1.5,\n  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},\n  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},\n  booktitle={arXiv},  \n  year={2025}\n }\n\n@inproceedings{xie2025region,\n  title={Region-based Cluster Discrimination for Visual Representation Learning},\n  author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},\n  booktitle={ICCV},\n  year={2025}\n}\n\n@article{lillava,\n  title={LLaVA-OneVision: Easy Visual Task Transfer},\n  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},\n  journal={Transactions on Machine Learning Research}\n  year={2024}\n}\n```\n\n## Acknowledgement\n\nWe extend our sincere gratitude to **AIAK team of the** [**Baige AI computing platform**](https://cloud.baidu.com/product/aihc.html) **from Baidu AI Cloud** for providing the exceptional training framework. The outstanding capabilities of AIAK-Training-LLM and AIAK-Megatron have significantly accelerated our training process with remarkable efficiency. These cutting-edge frameworks have been instrumental in achieving our research goals. `To get full AIAK support, you can contact Baidu Cloud.` \n\nWe acknowledge the support of [Synvo AI](https://synvo.ai/) for contributing to the partial data annotation in this work, and also thank the maintainers and contributors of the following open-source projects, whose work greatly inspired and supported our research:\n\n- LLaVA: Large Language-and-Vision Assistant — [LLaVA](https://github.com/haotian-liu/LLaVA)\n- LLaVA-NeXT: Next-generation multi-modal assistant — [LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT)\n- lmms-eval: A standardized evaluation framework for Large Multimodal Models — [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval)\n- Megatron-LM: Efficient, scalable training for large language models — [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)\n- Qwen2.5-VL: Strong vision-language foundation model — [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL)\n- InternVL: Open-source large-scale vision-language foundation model — [InternVL](https://github.com/OpenGVLab/InternVL)\n- Qwen3: Next-generation Qwen LLM — [Qwen](https://github.com/QwenLM/Qwen)\n- MetaCLIP: Scalable contrastive pretraining — [MetaCLIP](https://github.com/facebookresearch/MetaCLIP)\n- FineVision: Open Data Is All You Need — [FineVision](https://huggingface.co/spaces/HuggingFaceM4/FineVision)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevolvinglmms-lab%2Fllava-onevision-1.5","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevolvinglmms-lab%2Fllava-onevision-1.5","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevolvinglmms-lab%2Fllava-onevision-1.5/lists"}