{"id":14964522,"url":"https://github.com/nvlabs/eagle","last_synced_at":"2025-05-15T11:05:50.043Z","repository":{"id":255279880,"uuid":"820789280","full_name":"NVlabs/EAGLE","owner":"NVlabs","description":"Eagle Family: Exploring Model Designs, Data Recipes and Training Strategies for Frontier-Class Multimodal LLMs","archived":false,"fork":false,"pushed_at":"2025-04-22T09:17:52.000Z","size":80399,"stargazers_count":674,"open_issues_count":22,"forks_count":40,"subscribers_count":29,"default_branch":"main","last_synced_at":"2025-04-22T09:56:13.420Z","etag":null,"topics":["demo","eagle","gpt4","huggingface","large-language-models","llama","llama3","llava","llm","lmm","lvlm","mllm","nvdia"],"latest_commit_sha":null,"homepage":"https://nvlabs.github.io/EAGLE/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-06-27T07:22:52.000Z","updated_at":"2025-04-22T09:41:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"3704875e-3ea8-41ee-acee-e876427b06d7","html_url":"https://github.com/NVlabs/EAGLE","commit_stats":{"total_commits":73,"total_committers":7,"mean_commits":"10.428571428571429","dds":0.6575342465753424,"last_synced_commit":"fef95f103b5e9899acbbe2c237e5b99147ab7e8e"},"previous_names":["nvlabs/eagle"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FEAGLE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FEAGLE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FEAGLE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FEAGLE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVlabs","download_url":"https://codeload.github.com/NVlabs/EAGLE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254328385,"owners_count":22052632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["demo","eagle","gpt4","huggingface","large-language-models","llama","llama3","llava","llm","lmm","lvlm","mllm","nvdia"],"created_at":"2024-09-24T13:33:18.426Z","updated_at":"2025-05-15T11:05:50.035Z","avatar_url":"https://github.com/NVlabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Eagle Family:  Exploring Model Designs, Data Recipes and Training Strategies for Frontier-Class Multimodal LLMs\n\n\u003c/div\u003e\n\n[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/LICENSE)\n[![Model License](https://img.shields.io/badge/MODEL%20License-CC%20By%20NC%204.0-red.svg)](MODEL_LICENSE)\n\n\n\n[[📜 Eagle2.5 Paper](https://arxiv.org/abs/2504.15271)] [[📜 Eagle2 Paper](http://arxiv.org/abs/2501.14818)] [[📜 Eagle1 Paper](https://arxiv.org/pdf/2408.15998)] [[🤗 HF Models](https://huggingface.co/collections/nvidia/eagle-2-6764ba887fa1ef387f7df067)] [[🤗 HF Demo](https://huggingface.co/spaces/nvidia/Eagle2-Demo)] [[Faster Demo](https://4760e7b4534dd26c3e.gradio.live)] [[Project Page](https://nvlabs.github.io/EAGLE/)]\n\n\n\n## Introduction\n\nEagle 2.5 is a family of frontier vision-language models (VLMs) designed for long-context multimodal learning. While most existing VLMs focus on short-context tasks, Eagle 2.5 addresses the challenges of long video comprehension and high-resolution image understanding, providing a generalist framework for both. The Eagle 2.5 training framework introduces two key techniques—Automatic Degrade Sampling (ADS) and Image Area Preservation (IAP)—to preserve contextual integrity and visual details. Additionally, the training pipeline is optimized for efficient long-context data training.\n\nA major contribution of Eagle 2.5 is the introduction of Eagle-Video-110K, a novel dataset with both story-level and clip-level annotations, specifically curated for long video understanding. Eagle 2.5 demonstrates substantial improvements on long-context multimodal benchmarks, offering a robust solution to the limitations of existing VLMs. Notably, Eagle 2.5-8B achieves 72.4% on Video-MME with 512 input frames, matching the results of top-tier commercial models such as GPT-4o and large-scale open-source models like Qwen2.5-VL-72B and InternVL2.5-78B, despite having significantly fewer parameters.\n\n### Key Innovations\n\n- **Information-First Sampling**: \n  - *Image Area Preservation (IAP)*: Optimizes image tiling to retain most of the original image area and aspect ratio, preserving fine-grained details.\n  - *Automatic Degrade Sampling (ADS)*: Dynamically balances visual and textual input, ensuring complete text retention while maximizing visual content within context length constraints.\n- **Progressive Mixed Post-Training**: \n  - Gradually increases context length during training, enhancing the model's ability to process varying input sizes and improving information density over static sampling.\n- **Diversity-Driven Data Recipe**: \n  - Combines open-source data (human-annotated and synthetic) with the self-curated Eagle-Video-110K dataset, collected via a diversity-driven strategy and annotated with both story-level and clip-level QA pairs.\n\n\n\n \u003cdiv align=\"center\"\u003e\n \u003cimg src=\"./Eagle1/assets/eagle2.5_video_mme.jpg\" width=\"80%\"\u003e\n \u003c/div\u003e\n \n\n\n## Updates\n- [2025/04] 🔥 Release Eagle-2.5!\n- [2025/01] 🔥 Release Eagle-2!\n- [2025/01] 🔥 [Eagle-1](./Eagle1/README.md) is accepted by [ICLR 2025](https://iclr.cc).\n- [2024/08] Release [Eagle-1](./Eagle1/README.md).\n\n\n\n\n\n## Model Zoo\nWe provide the following models:\n\n| model name         | LLM  | Vision  | Max Length| HF Link|\n| ----------- | ------- |---------|-|-|\n| Eagle2-1B | [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) |  Siglip    | 16K| [🤗 link](https://huggingface.co/nvidia/Eagle2-1B)|\n| Eagle2-2B | [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) |  Siglip    | 16K| [🤗 link](https://huggingface.co/nvidia/Eagle2-2B)|\n| Eagle2-9B | [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) |  Siglip+ConvNext    | 16K| [🤗 link](https://huggingface.co/nvidia/Eagle2-9B)|\n\n## Benchmark Results\n\n\n### Eagle2.5 Video Benchmarks\n\n| Benchmark                                 | GPT-4o             | Gemini-1.5 Pro    | InternVL2.5-8B      | Qwen2.5-VL-8B       | **Eagle2.5-8B**     |\n|--------------------------------------------|--------------------|-------------------|---------------------|---------------------|---------------------|\n| MVBench\u003csub\u003etest\u003c/sub\u003e                     | -                  | -                 | 72.0                | 69.6                | **74.8**            |\n| Perception_test\u003csub\u003eval\u003c/sub\u003e              | -                  | -                 | -                   | 70.5                | **82.0**            |\n| EgoSchema\u003csub\u003efullset\u003c/sub\u003e                | -                  | 72.2              | -                   | 65.0                | **72.2**            |\n| MMB-Video                                 | 1.63               | 1.30              | 1.68                | 1.79                | **1.94**            |\n| MLVU\u003csub\u003eval\u003c/sub\u003e                         | -                  | -                 | 68.9                | 70.2                | **77.6**            |\n| LVBench\u003csub\u003eval\u003c/sub\u003e                      | 66.7               | 64.0              | 60.0                | 56.0                | **66.4**            |\n| Video-MME\u003csub\u003ew/o subtitle\u003c/sub\u003e           | 71.9               | 75.0              | 64.2                | 65.1                | **72.4**            |\n| Video-MME\u003csub\u003ew subtitle\u003c/sub\u003e             | 77.2               | 81.3              | 66.9                | 71.6                | **75.7**            |\n| CG-Bench\u003csub\u003eClue\u003c/sub\u003e                    | 58.6               | 50.9              | -                   | 44.5                | **55.8**            |\n| CG-Bench\u003csub\u003eLong\u003c/sub\u003e                    | 44.9               | 37.8              | -                   | 35.5                | **46.6**            |\n| CG-Bench\u003csub\u003emIoU\u003c/sub\u003e                    | 5.73               | 3.85              | -                   | 2.48                | **13.4**            |\n| HourVideo\u003csub\u003eDev\u003c/sub\u003e                    | -                  | 37.2              | -                   | -                   | **44.5**            |\n| HourVideo\u003csub\u003eTest\u003c/sub\u003e                   | -                  | 37.4              | -                   | -                   | **41.8**            |\n| Charade-STA\u003csub\u003emIoU\u003c/sub\u003e                 | 35.7               | -                 | -                   | 43.6                | **65.9**            |\n| HD-EPIC                                   | -                  | 37.6                 | -                   | -                   | **42.9**            |\n| HRVideoBench                              | -                  | -                 | -                   | -                   | **68.5**            |\n| EgoPlan\u003csub\u003eval\u003c/sub\u003e                      | -                  | -                 | -                   | -                   | **45.3**            |\n\n### Eagle2.5 Embodied Benchmarks\n| Benchmark                                 | GPT-4o             | Gemini-1.5 Pro    | InternVL2.5-8B      | Qwen2.5-VL-8B       | **Eagle2.5-8B**     |\n|--------------------------------------------|--------------------|-------------------|---------------------|---------------------|---------------------|\n| OpenEQA                                    | -                  | -                 | -                   | -                   | **63.5**            |\n| ERQA                                       | 47.0                  | 41.8                 | -                   | -                   | **38.3**            |\n| EgoPlan\u003csub\u003eval\u003c/sub\u003e                      | -                  | -                 | -                   | -                   | **45.3**            |\n\n\n\n### Eagle2.5 Image Benchmarks\n\n| Benchmark                                 | GPT-4o             | Gemini-1.5 Pro    | InternVL2.5-8B      | Qwen2.5-VL-8B       | **Eagle2.5-8B**     |\n|--------------------------------------------|--------------------|-------------------|---------------------|---------------------|---------------------|\n| DocVQA\u003csub\u003etest\u003c/sub\u003e                      | 92.8               | 93.1              | 93.0                | 95.7                | **94.1**            |\n| ChartQA\u003csub\u003etest\u003c/sub\u003e                     | 85.7               | 87.2              | 84.8                | 87.3                | **87.5**            |\n| InfoVQA\u003csub\u003etest\u003c/sub\u003e                     | 79.2               | 81.0              | 77.6                | 82.6                | **80.4**            |\n| TextVQA\u003csub\u003eval\u003c/sub\u003e                      | 77.4               | 78.8              | 79.1                | 84.9                | **83.7**            |\n| OCRBench\u003csub\u003etest\u003c/sub\u003e                    | 736                | 754               | 822                 | 864                 | **869**             |\n| MMstar\u003csub\u003etest\u003c/sub\u003e                      | 64.7               | 59.1              | 62.8                | 63.9                | **66.2**            |\n| RWQA\u003csub\u003etest\u003c/sub\u003e                        | 75.4               | 67.5              | 70.1                | 68.5                | **76.7**            |\n| AI2D\u003csub\u003etest\u003c/sub\u003e                        | 84.6               | 79.1              | 84.5                | 83.9                | **84.5**            |\n| MMMU\u003csub\u003eval\u003c/sub\u003e                         | 69.1               | 62.2              | 56.0                | 58.6                | **55.8**            |\n| MMBench_V11\u003csub\u003etest\u003c/sub\u003e                 | 83.1               | 74.6              | 83.2                | 82.6                | **81.7**            |\n| MMVet\u003csub\u003eGPT-4-Turbo\u003c/sub\u003e                | 69.1               | 64.0              | 62.8                | 67.1                | **62.9**            |\n| HallBench\u003csub\u003eavg\u003c/sub\u003e                    | 55.0               | 45.6              | 50.1                | 52.9                | **54.7**            |\n| MathVista\u003csub\u003etestmini\u003c/sub\u003e               | 63.8               | 63.9              | 64.4                | 68.2                | **67.8**            |\n| Avg Score                                 | 74.9               | 71.7              | 73.1                | 75.6                | **75.6**            |\n\n\n*All numbers are directly extracted from Table 2 and Table 3 of the Eagle 2.5 Tech Report.*\n\n\u003cdetails\u003e\n  \u003csummary\u003eEagle2-1B Results\u003c/summary\u003e\n\n|          Benchmark           | LLaVa-One-Vision-0.5B | InternVL2-1B | InternVL2.5-1B |Qwen2-VL-2B| Eagle2-1B|\n| :--------------------------: | :------------------: | :----------------: | :----------: |:----------: |:----------: |  \n|    DocVQA\u003csub\u003etest\u003c/sub\u003e     |         70.0         |        81.7        |     84.8     |90.1|81.8|\n|    ChartQA\u003csub\u003etest\u003c/sub\u003e    |          61.4         |        72.9        |     75.9     |73.0|77.0|\n|    InfoVQA\u003csub\u003etest\u003c/sub\u003e    |          41.8           |        50.9        |     56.0     |65.5|54.8|\n|    TextVQA\u003csub\u003eval\u003c/sub\u003e     |         -         |        70.0       |     72.0     |79.7|76.6|\n|           OCRBench           |         565          |        754         |     785      |809|767|\n|      MME\u003csub\u003esum\u003c/sub\u003e       |        1438.0     |       1794.4      |    1950.5   |  1872.0| 1790.2|\n|         RealWorldQA          |        55.6     |        50.3       |    57.5     |62.6|55.4|\n|     AI2D\u003csub\u003etest\u003c/sub\u003e      |         57.1         |        64.1        |     69.3    | 74.7 |70.9|\n|      MMMU\u003csub\u003eval\u003c/sub\u003e      |          31.4       |    36.7     | 40.9  |41.1|38.8|\n| MMVet\u003csub\u003eGPT-4-Turbo\u003c/sub\u003e  |         32.2       |        32.7       |    48.8    | 49.5|40.9|             HallBench\u003csub\u003eavg\u003c/sub\u003e    |         27.9      |        34.0       |     39.0     |**41.7**|35.3\n| MathVista\u003csub\u003etestmini\u003c/sub\u003e |         33.8         |        37.7        |     43.2     |43.0|45.3|\n| MMstar |             37.7    |       45.7      |     50.1|48.0|48.5|\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eEagle2-2B Results\u003c/summary\u003e\n\n|          Benchmark           | InternVL2-2B | InternVL2.5-2B | InternVL2-4B |Qwen2-VL-2B| Eagle2-2B|\n| :--------------------------: | :------------------: | :----------------: | :----------: |:----------: |:----------: |  \n|    DocVQA\u003csub\u003etest\u003c/sub\u003e     |         86.9 |        88.7  |     89.2    |90.1|88.0|\n|    ChartQA\u003csub\u003etest\u003c/sub\u003e    |          76.2        |        79.2        |     81.5     |73.0|82.0|\n|    InfoVQA\u003csub\u003etest\u003c/sub\u003e    |         58.9   |        60.9     |    67.0     |65.5|65.8|\n|    TextVQA\u003csub\u003eval\u003c/sub\u003e     |         73.4         |        74.3       |     74.4   |79.7|79.1|\n|           OCRBench           |         784          |        804         |     788      |809|818|\n|      MME\u003csub\u003esum\u003c/sub\u003e       |   1876.8         |    2138.2    |    2059.8   |1872.0  | 2109.8\n|         RealWorldQA          |        57.3     |        60.1      |    60.7     |62.6|63.1|\n|     AI2D\u003csub\u003etest\u003c/sub\u003e      |        74.1   |        74.9     |     74.7    | 78.9 |79.3|\n|      MMMU\u003csub\u003eval\u003c/sub\u003e      |          36.3       |    43.6     | 47.9  |41.1|43.1|\n| MMVet\u003csub\u003eGPT-4-Turbo\u003c/sub\u003e  |         39.5       |       60.8       |     51.0     | 49.5|53.8|\n|   HallBench\u003csub\u003eavg\u003c/sub\u003e    |         37.9      |        42.6      |     41.9     |41.7|45.8\n| MathVista\u003csub\u003etestmini\u003c/sub\u003e |         46.3      |        51.3       |     58.6     |43.0|54.7|\n| MMstar |            50.1    |       53.7     |     54.3|48.0|56.4|\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eEagle2-9B Results\u003c/summary\u003e\n|          Benchmark           | MiniCPM-Llama3-V-2_5 | InternVL-Chat-V1-5 | InternVL2-8B |QwenVL2-7B| Eagle2-9B|\n| :--------------------------: | :------------------: | :----------------: | :----------: |:----------: |:----------: |\n|          Model Size          |         8.5B         |       25.5B        |     8.1B     | 8.3B|8.9B|\n|                              |                      |                    |              | | |\n|    DocVQA\u003csub\u003etest\u003c/sub\u003e     |         84.8         |        90.9        |     91.6     |**94.5**|92.6|\n|    ChartQA\u003csub\u003etest\u003c/sub\u003e    |          -           |        83.8        |     83.3     |83.0|**86.4**|\n|    InfoVQA\u003csub\u003etest\u003c/sub\u003e    |          -           |        72.5        |     74.8     |74.3|**77.2**|\n|    TextVQA\u003csub\u003eval\u003c/sub\u003e     |         76.6         |        80.6        |     77.4     |**84.3**|83.0|\n|           OCRBench           |         725          |        724         |     794      |845|**868**|\n|      MME\u003csub\u003esum\u003c/sub\u003e       |        2024.6        |       2187.8       |    2210.3    |  **2326.8**|2260|\n|         RealWorldQA          |         63.5         |        66.0        |     64.4     |**70.1**|69.3|\n|     AI2D\u003csub\u003etest\u003c/sub\u003e      |         78.4         |        80.7        |     83.8     | - |**83.9**|\n|      MMMU\u003csub\u003eval\u003c/sub\u003e      |         45.8         |    45.2 / 46.8     | 49.3 / 51.8  |54.1|**56.1**|\n|  MMBench_V11\u003csub\u003etest\u003c/sub\u003e   |               |            |     79.5     |79.4|**80.6**|\n| MMVet\u003csub\u003eGPT-4-Turbo\u003c/sub\u003e  |         52.8         |        55.4        |     54.2     | 62.0|**62.2**|\n|          SEED-Image          |         72.3         |        76.0        |     76.2     ||**77.1**|\n|   HallBench\u003csub\u003eavg\u003c/sub\u003e    |         42.4         |        49.3        |     45.2     |**50.6**|49.3\n| MathVista\u003csub\u003etestmini\u003c/sub\u003e |         54.3         |        53.5        |     58.3     |58.2|**63.8**|\n| MMstar |             -    |       -      |      60.9|60.7|**62.6**|\n\n\u003c/details\u003e\n\n## Stremlit Demo\n\nWe provide a [local chat demo](./Eagle2/streamlit_demo/README.md) powered by `Streamlit` to help users get started with `Eagle2` quickly and easily.\nThis demo is built upon [InternVL's template](https://internvl.readthedocs.io/en/latest/get_started/local_chat_demo.html) and extends it with additional video input support for enhanced functionality.\n\n## Inference\n\nWe provide a [inference script](./Eagle2/inference/README.md) to help you quickly start using the model. We support different input types: \n- pure text input\n- single image input\n- multiple image input\n- video input\n\n### 0. Install the dependencies\n\n```bash\npip install transformers==4.37.2\npip install flash-attn\n```\n**Note**: Latest version of transformers is not compatible with the model.\n\n### 1. Prepare the Model worker\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\n```python\n\n\"\"\"\nA model worker executes the model.\nCopied and modified from https://github.com/OpenGVLab/InternVL/blob/main/streamlit_demo/model_worker.py\n\"\"\"\n# Importing torch before transformers can cause `segmentation fault`\nfrom transformers import AutoModel, AutoTokenizer, TextIteratorStreamer, AutoConfig\n\nimport argparse\nimport base64\nimport json\nimport os\nimport decord\nimport threading\nimport time\nfrom io import BytesIO\nfrom threading import Thread\nimport math\nimport requests\nimport torch\nimport torchvision.transforms as T\nfrom PIL import Image\nfrom torchvision.transforms.functional import InterpolationMode\nimport numpy as np\n\n\nIMAGENET_MEAN = (0.485, 0.456, 0.406)\nIMAGENET_STD = (0.229, 0.224, 0.225)\n\nSIGLIP_MEAN = (0.5, 0.5, 0.5)\nSIGLIP_STD = (0.5, 0.5, 0.5)\n\n\ndef get_seq_frames(total_num_frames, desired_num_frames=-1, stride=-1):\n    \"\"\"\n    Calculate the indices of frames to extract from a video.\n\n    Parameters:\n    total_num_frames (int): Total number of frames in the video.\n    desired_num_frames (int): Desired number of frames to extract.\n\n    Returns:\n    list: List of indices of frames to extract.\n    \"\"\"\n    \n    assert desired_num_frames \u003e 0 or stride \u003e 0 and not (desired_num_frames \u003e 0 and stride \u003e 0)\n\n    if stride \u003e 0:\n        return list(range(0, total_num_frames, stride))\n    \n    # Calculate the size of each segment from which a frame will be extracted\n    seg_size = float(total_num_frames - 1) / desired_num_frames\n\n    seq = []\n    for i in range(desired_num_frames):\n        # Calculate the start and end indices of each segment\n        start = int(np.round(seg_size * i))\n        end = int(np.round(seg_size * (i + 1)))\n\n        # Append the middle index of the segment to the list\n        seq.append((start + end) // 2)\n\n    return seq\n\ndef build_video_prompt(meta_list, num_frames, time_position=False):\n    # if time_position is True, the frame_timestamp is used.\n    # 1. pass time_position, 2. use env TIME_POSITION\n    time_position = os.environ.get(\"TIME_POSITION\", time_position)\n    prefix = f\"This is a video:\\n\"\n    for i in range(num_frames):\n        if time_position:\n            frame_txt = f\"Frame {i+1} sampled at {meta_list[i]:.2f} seconds: \u003cimage\u003e\\n\"\n        else:\n            frame_txt = f\"Frame {i+1}: \u003cimage\u003e\\n\"\n        prefix += frame_txt\n    return prefix\n\ndef load_video(video_path, num_frames=64, frame_cache_root=None):\n    if isinstance(video_path, str):\n        video = decord.VideoReader(video_path)\n    elif isinstance(video_path, dict):\n        assert False, 'we not support vidoe: \"video_path\" as input'\n    fps = video.get_avg_fps()\n    sampled_frames = get_seq_frames(len(video), num_frames)\n    samepld_timestamps = [i / fps for i in sampled_frames]\n    frames = video.get_batch(sampled_frames).asnumpy()\n    images = [Image.fromarray(frame) for frame in frames]\n    \n    return images, build_video_prompt(samepld_timestamps, len(images), time_position=True)\n\ndef load_image(image):\n    if isinstance(image, str) and os.path.exists(image):\n        return Image.open(image)\n    elif isinstance(image, dict):\n        if 'disk_path' in image:\n            return Image.open(image['disk_path'])\n        elif 'base64' in image:\n            return Image.open(BytesIO(base64.b64decode(image['base64'])))\n        elif 'url' in image:\n            response = requests.get(image['url'])\n            return Image.open(BytesIO(response.content))\n        elif 'bytes' in image:\n            return Image.open(BytesIO(image['bytes']))\n        else:\n            raise ValueError(f'Invalid image: {image}')\n    else:\n        raise ValueError(f'Invalid image: {image}')\n\ndef build_transform(input_size, norm_type='imagenet'):\n    if norm_type == 'imagenet':\n        MEAN, STD = IMAGENET_MEAN, IMAGENET_STD\n    elif norm_type == 'siglip':\n        MEAN, STD = SIGLIP_MEAN, SIGLIP_STD\n        \n    transform = T.Compose([\n        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),\n        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),\n        T.ToTensor(),\n        T.Normalize(mean=MEAN, std=STD)\n    ])\n    return transform\n\n\ndef find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):\n    \"\"\"\n    previous version mainly foucs on ratio.\n    We also consider area ratio here.\n    \"\"\"\n    best_factor = float('-inf')\n    best_ratio = (1, 1)\n    area = width * height\n    for ratio in target_ratios:\n        target_aspect_ratio = ratio[0] / ratio[1]\n        ratio_diff = abs(aspect_ratio - target_aspect_ratio)\n        area_ratio = (ratio[0]*ratio[1]*image_size*image_size)/ area\n        \"\"\"\n        new area \u003e 60% of original image area is enough.\n        \"\"\"\n        factor_based_on_area_n_ratio = min((ratio[0]*ratio[1]*image_size*image_size)/ area, 0.6)* \\\n                                     min(target_aspect_ratio/aspect_ratio, aspect_ratio/target_aspect_ratio)\n        \n        if factor_based_on_area_n_ratio \u003e best_factor:\n            best_factor = factor_based_on_area_n_ratio\n            best_ratio = ratio\n        \n    return best_ratio\n\n\ndef dynamic_preprocess(image, min_num=1, max_num=6, image_size=448, use_thumbnail=False):\n    orig_width, orig_height = image.size\n    aspect_ratio = orig_width / orig_height\n\n    # calculate the existing image aspect ratio\n    target_ratios = set(\n        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if\n        i * j \u003c= max_num and i * j \u003e= min_num)\n    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])\n\n    # find the closest aspect ratio to the target\n    target_aspect_ratio = find_closest_aspect_ratio(\n        aspect_ratio, target_ratios, orig_width, orig_height, image_size)\n\n    # calculate the target width and height\n    target_width = image_size * target_aspect_ratio[0]\n    target_height = image_size * target_aspect_ratio[1]\n    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]\n\n    # resize the image\n    resized_img = image.resize((target_width, target_height))\n    processed_images = []\n    for i in range(blocks):\n        box = (\n            (i % (target_width // image_size)) * image_size,\n            (i // (target_width // image_size)) * image_size,\n            ((i % (target_width // image_size)) + 1) * image_size,\n            ((i // (target_width // image_size)) + 1) * image_size\n        )\n        # split the image\n        split_img = resized_img.crop(box)\n        processed_images.append(split_img)\n    assert len(processed_images) == blocks\n    if use_thumbnail and len(processed_images) != 1:\n        thumbnail_img = image.resize((image_size, image_size))\n        processed_images.append(thumbnail_img)\n    return processed_images\n\ndef split_model(model_path, device):\n\n    device_map = {}\n    world_size = torch.cuda.device_count()\n    config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)\n    num_layers = config.llm_config.num_hidden_layers\n\n    print('world_size', world_size)\n    num_layers_per_gpu_ = math.floor(num_layers / (world_size - 1))\n    num_layers_per_gpu = [num_layers_per_gpu_] * world_size\n    num_layers_per_gpu[device] = num_layers - num_layers_per_gpu_ * (world_size-1)\n    print(num_layers_per_gpu)\n    layer_cnt = 0\n    for i, num_layer in enumerate(num_layers_per_gpu):\n        for j in range(num_layer):\n            device_map[f'language_model.model.layers.{layer_cnt}'] = i\n            layer_cnt += 1\n    device_map['vision_model'] = device\n    device_map['mlp1'] = device\n    device_map['language_model.model.tok_embeddings'] = device\n    device_map['language_model.model.embed_tokens'] = device\n    device_map['language_model.output'] = device\n    device_map['language_model.model.norm'] = device\n    device_map['language_model.lm_head'] = device\n    device_map['language_model.model.rotary_emb'] = device\n    device_map[f'language_model.model.layers.{num_layers - 1}'] = device\n    return device_map\n\nclass ModelWorker:\n    def __init__(self, model_path, model_name,\n                 load_8bit, device):\n\n        if model_path.endswith('/'):\n            model_path = model_path[:-1]\n        if model_name is None:\n            model_paths = model_path.split('/')\n            if model_paths[-1].startswith('checkpoint-'):\n                self.model_name = model_paths[-2] + '_' + model_paths[-1]\n            else:\n                self.model_name = model_paths[-1]\n        else:\n            self.model_name = model_name\n\n        print(f'Loading the model {self.model_name}')\n\n        tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False)\n        tokens_to_keep = ['\u003cbox\u003e', '\u003c/box\u003e', '\u003cref\u003e', '\u003c/ref\u003e']\n        tokenizer.additional_special_tokens = [item for item in tokenizer.additional_special_tokens if item not in tokens_to_keep]\n        self.tokenizer = tokenizer\n        config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)\n        model_type = config.vision_config.model_type\n        self.device = torch.cuda.current_device()\n        if model_type == 'siglip_vision_model':\n            self.norm_type = 'siglip'\n        elif model_type == 'MOB':\n            self.norm_type = 'siglip'\n        else:\n            self.norm_type = 'imagenet'\n\n        if any(x in model_path.lower() for x in ['34b']):\n            device_map = split_model(model_path, self.device)\n        else:\n            device_map = None\n        \n        if device_map is not None:    \n            self.model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,\n                                               low_cpu_mem_usage=True,\n                                               device_map=device_map, \n                                               trust_remote_code=True,\n                                               load_in_8bit=load_8bit).eval()\n        else:\n            self.model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,\n                                               trust_remote_code=True,\n                                               load_in_8bit=load_8bit).eval()  \n\n        if not load_8bit and device_map is None:\n            self.model = self.model.to(device)\n        self.load_8bit = load_8bit\n        \n        self.model_path = model_path\n        self.image_size = self.model.config.force_image_size\n        self.context_len = tokenizer.model_max_length\n        self.per_tile_len = 256\n\n    def reload_model(self):\n        del self.model\n        torch.cuda.empty_cache()\n        if self.device == 'auto':\n            os.environ['CUDA_LAUNCH_BLOCKING'] = '1'\n            # This can make distributed deployment work properly\n            self.model = AutoModel.from_pretrained(\n                self.model_path,\n                load_in_8bit=self.load_8bit,\n                torch_dtype=torch.bfloat16,\n                device_map=self.device_map,\n                trust_remote_code=True).eval()\n        else:\n            self.model = AutoModel.from_pretrained(\n                self.model_path,\n                load_in_8bit=self.load_8bit,\n                torch_dtype=torch.bfloat16,\n                trust_remote_code=True).eval()\n        if not self.load_8bit and not self.device == 'auto':\n            self.model = self.model.cuda()\n\n    @torch.inference_mode()\n    def generate(self, params):\n        system_message = params['prompt'][0]['content']\n        send_messages = params['prompt'][1:]\n        max_input_tiles = params['max_input_tiles']\n        temperature = params['temperature']\n        top_p = params['top_p']\n        max_new_tokens = params['max_new_tokens']\n        repetition_penalty = params['repetition_penalty']\n        video_frame_num = params.get('video_frame_num', 64)\n        do_sample = True if temperature \u003e 0.0 else False\n\n        global_image_cnt = 0\n        history, pil_images, max_input_tile_list = [], [], []\n        for message in send_messages:\n            if message['role'] == 'user':\n                prefix = ''\n                if 'image' in message:\n                    for image_data in message['image']:\n                        pil_images.append(load_image(image_data))\n                        prefix = prefix + f'\u003cimage {global_image_cnt + 1}\u003e\u003cimage\u003e\\n'\n                        global_image_cnt += 1\n                        max_input_tile_list.append(max_input_tiles)\n                if 'video' in message:\n                    for video_data in message['video']:\n                        video_frames, tmp_prefix = load_video(video_data, num_frames=video_frame_num)\n                        pil_images.extend(video_frames)\n                        prefix = prefix + tmp_prefix\n                        global_image_cnt += len(video_frames)\n                        max_input_tile_list.extend([1] * len(video_frames))\n                content = prefix + message['content']\n                history.append([content, ])\n            else:\n                history[-1].append(message['content'])\n        question, history = history[-1][0], history[:-1]\n\n        if global_image_cnt == 1:\n            question = question.replace('\u003cimage 1\u003e\u003cimage\u003e\\n', '\u003cimage\u003e\\n')\n            history = [[item[0].replace('\u003cimage 1\u003e\u003cimage\u003e\\n', '\u003cimage\u003e\\n'), item[1]] for item in history]\n\n\n        try:\n            assert len(max_input_tile_list) == len(pil_images), 'The number of max_input_tile_list and pil_images should be the same.'\n        except Exception as e:\n            from IPython import embed; embed()\n            exit()\n            print(f'Error: {e}')\n            print(f'max_input_tile_list: {max_input_tile_list}, pil_images: {pil_images}')\n            # raise e\n\n        old_system_message = self.model.system_message\n        self.model.system_message = system_message\n        \n        transform = build_transform(input_size=self.image_size, norm_type=self.norm_type)\n        if len(pil_images) \u003e 0:\n            max_input_tiles_limited_by_contect = params['max_input_tiles']\n            while True:\n                image_tiles = []\n                for current_max_input_tiles, pil_image in zip(max_input_tile_list, pil_images):\n                    if self.model.config.dynamic_image_size:\n                        tiles = dynamic_preprocess(\n                            pil_image, image_size=self.image_size, max_num=min(current_max_input_tiles, max_input_tiles_limited_by_contect),\n                            use_thumbnail=self.model.config.use_thumbnail)\n                    else:\n                        tiles = [pil_image]\n                    image_tiles += tiles\n                if (len(image_tiles) * self.per_tile_len \u003c self.context_len):\n                    break\n                else:\n                    max_input_tiles_limited_by_contect -= 2\n                \n                if max_input_tiles_limited_by_contect \u003c 1:\n                    break\n                    \n            pixel_values = [transform(item) for item in image_tiles]\n            pixel_values = torch.stack(pixel_values).to(self.model.device, dtype=torch.bfloat16)\n            print(f'Split images to {pixel_values.shape}')\n        else:\n            pixel_values = None\n\n        generation_config = dict(\n            num_beams=1,\n            max_new_tokens=max_new_tokens,\n            do_sample=do_sample,\n            temperature=temperature,\n            repetition_penalty=repetition_penalty,\n            max_length=self.context_len,\n            top_p=top_p,\n        )\n\n        response = self.model.chat(\n            tokenizer=self.tokenizer,\n            pixel_values=pixel_values,\n            question=question,\n            history=history,\n            return_history=False,\n            generation_config=generation_config,\n        )\n        self.model.system_message = old_system_message\n        return {'text': response, 'error_code': 0}\n\n\n\n\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser()\n    parser.add_argument('--model-path', type=str, default='nvidia/Eagle2-1B')\n    parser.add_argument('--model-name', type=str, default='Eagle2-1B')\n    parser.add_argument('--device', type=str, default='cuda')\n    parser.add_argument('--load-8bit', action='store_true')\n    args = parser.parse_args()\n    print(f'args: {args}')\n\n    worker = ModelWorker(\n                         args.model_path,\n                         args.model_name,\n                         args.load_8bit,\n                         args.device)\n```\n\u003c/details\u003e\n\n\n### 2. Prepare the Prompt\n\n- Single image input\n```python\nprompt = [\n        {'role': 'system', 'content': 'You are a helpful assistant.'},\n        {'role': 'user', 'content': 'Describe this image in details.', \n            'image':[\n                {'url': 'https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png'}\n            ],\n        }\n    ]\n```\n\n- Multiple image input\n```python\nprompt = [\n        {'role': 'system', 'content': 'You are a helpful assistant.'},\n        {'role': 'user', 'content': 'Describe these two images in details.', \n            'image':[\n                {'url': 'https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png'},\n                {'url': 'https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png'}\n            ],\n        }\n    ]\n```\n\n- Video input\n```python\nprompt = [\n        {'role': 'system', 'content': 'You are a helpful assistant.'},\n        {'role': 'user', 'content': 'Describe this video in details.', \n            'video':[\n                'path/to/your/video.mp4'\n            ],\n        }\n    ]\n```\n\n### 3. Generate the response    \n```python\nparams = {\n    'prompt': prompt,\n    'max_input_tiles': 24,\n    'temperature': 0.7,\n    'top_p': 1.0,\n    'max_new_tokens': 4096,\n    'repetition_penalty': 1.0,\n    }\nworker.generate(params)\n```\n\n## Evaluation\n\nWe evaluate the performance of `Eagle2` based on [VLMEvalKit](https://github.com/open-compass/VLMEvalKit). We temporarily provide a custom [vlmeval](./Eagle2/vlmeval/README.md) implementation that supports `Eagle2` in our repo, and we will support `Eagle2` in the official version as soon as possible.\n\n## TODO\n- [ ] Support vLLM Inference\n- [ ] Provide AWQ Quantization Weights\n- [ ] Provide fine-tuning scripts\n\n\n## Citation\nIf you find this project useful, please cite our work:\n```\n@misc{li2025eagle2buildingposttraining,\n      title={Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models}, \n      author={Zhiqi Li and Guo Chen and Shilong Liu and Shihao Wang and Vibashan VS and Yishen Ji and Shiyi Lan and Hao Zhang and Yilin Zhao and Subhashree Radhakrishnan and Nadine Chang and Karan Sapra and Amala Sanjay Deshmukh and Tuomas Rintamaki and Matthieu Le and Ilia Karmanov and Lukas Voegtle and Philipp Fischer and De-An Huang and Timo Roman and Tong Lu and Jose M. Alvarez and Bryan Catanzaro and Jan Kautz and Andrew Tao and Guilin Liu and Zhiding Yu},\n      year={2025},\n      eprint={2501.14818},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2501.14818}, \n}\n\n@article{shi2024eagle,\n    title = {Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders}, \n    author={Min Shi and Fuxiao Liu and Shihao Wang and Shijia Liao and Subhashree Radhakrishnan and De-An Huang and Hongxu Yin and Karan Sapra and Yaser Yacoob and Humphrey Shi and Bryan Catanzaro and Andrew Tao and Jan Kautz and Zhiding Yu and Guilin Liu},\n    journal={arXiv:2408.15998},\n    year={2024}\n}\n```\n\n\n## License/Terms of Use\n- The code is released under the Apache 2.0 license as found in the [LICENSE](https://huggingface.co/NVEagle/Eagle-X5-13B-Chat/blob/main/LICENSE) file.\n- The pretrained model weights are released under the [Creative Commons Attribution: Non-Commercial 4.0 International](https://spdx.org/licenses/CC-BY-NC-4.0) \u003cbr\u003e\n- The service is a research preview intended for non-commercial use only, and is subject to the following licenses and terms:\n  - Model License of Qwen2.5-7B-Instruct: [Apache-2.0](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/LICENSE)\n  - Model License of LLama: [Llama community license](https://ai.meta.com/llama/license/)\n  - Model License of PaliGemma: [Gemma license](https://ai.google.dev/gemma/terms)\n  - Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.\n\n\n\n\n## Acknowledgement\n\n- [InternVL](https://github.com/OpenGVLab/InternVL): we built the codebase based on InternVL. Thanks for the great open-source project.\n- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): We use vlmeval for evaluation. Many thanks for their wonderful tools.\n\n- Thanks to [Cambrian](https://cambrian-mllm.github.io), [LLaVA-One-Vision](https://llava-vl.github.io/blog/2024-08-05-llava-onevision/) and more great work for their efforts in organizing open-source data.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvlabs%2Feagle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvlabs%2Feagle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvlabs%2Feagle/lists"}