{"id":15116043,"url":"https://github.com/FreedomIntelligence/LongLLaVA","last_synced_at":"2025-09-27T21:31:38.500Z","repository":{"id":255604615,"uuid":"851137885","full_name":"FreedomIntelligence/LongLLaVA","owner":"FreedomIntelligence","description":"LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via Hybrid Architecture","archived":false,"fork":false,"pushed_at":"2025-01-06T06:14:15.000Z","size":4015,"stargazers_count":188,"open_issues_count":1,"forks_count":13,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-01-12T10:04:51.970Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-02T13:49:15.000Z","updated_at":"2025-01-12T08:08:53.000Z","dependencies_parsed_at":"2024-11-24T14:02:43.797Z","dependency_job_id":"d3bd5864-b13d-4858-8f9c-0841b3fe38ab","html_url":"https://github.com/FreedomIntelligence/LongLLaVA","commit_stats":null,"previous_names":["freedomintelligence/longllava"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FLongLLaVA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FLongLLaVA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FLongLLaVA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FLongLLaVA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/LongLLaVA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234460505,"owners_count":18836837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T01:44:08.134Z","updated_at":"2025-09-27T21:31:32.570Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"![header](./assets/header.png) \n\n\u003cp align=\"center\"\u003e\n   📃 \u003ca href=\"https://arxiv.org/abs/2409.02889\" target=\"_blank\"\u003ePaper\u003c/a\u003e • 🌐 \u003ca href=\"\" target=\"_blank\"\u003eDemo\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/LongLLaVA-53B-A13B\" target=\"_blank\"\u003eLongLLaVA-53B-A13B\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/LongLLaVA-9B\" target=\"_blank\"\u003eLongLLaVA-9B\u003c/a\u003e \n\u003c/p\u003e\n\n![efficiency](./assets/singleGPU.png) \n\n## 🌈 Update\n\n* **[2024.09.05]** LongLLaVA repo is published！🎉\n* **[2024.10.12]** [LongLLaVA-53B-A13B](https://huggingface.co/FreedomIntelligence/LongLLaVA-53B-A13B), [LongLLaVA-9b](https://huggingface.co/FreedomIntelligence/LongLLaVA-9B) and [Jamba-9B-Instruct](https://huggingface.co/FreedomIntelligence/Jamba-9B-Instruct) are repleased！🎉 \n\n## Architecture\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick to view the architecture image\u003c/summary\u003e\n\n  ![Architecture Image](./assets/arch.png)\n\n\u003c/details\u003e\n\n\n## Results\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick to view the Results\u003c/summary\u003e\n\n  - Main Results\n      ![Main Results](./assets/result1.png) \n  - Diagnostic Results\n      ![Diagnostic Results](./assets/diaresult.png)\n  - Video-NIAH\n      ![Video-NIAH](./assets/NIAH.png)\n\n\u003c/details\u003e\n\n\n\n## Results reproduction\n\n### 1. Environment Setup\n\n  ```bash\n  pip install -r requirements.txt\n  ```\n\n### 2. Data DownLoad and Construction\n\n\u003cdetails\u003e\n  \u003csummary\u003eDataset Taxonomy\u003c/summary\u003e\n\n  ![Dataset](./assets/dataset.png) \n\n\u003c/details\u003e\n\n- Dataset DownLoading and Construction\n  \u003e Coming Soon.\n\n\n\n\n### 3. Training\n\n- Downloading Language Models\n  \u003cp align=\"left\"\u003e\n   🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Jamba-9B-Instruct\" target=\"_blank\"\u003eJamba-9B-Instruct\u003c/a\u003e \n  \u003c/p\u003e\n\n- Stage I: Single-image Alignment.\n  ```bash\n  bash Align.sh\n  ```\n- Stage II: Single-image Instruction-tuning.\n  ```bash\n  bash SingleImageSFT.sh\n  ```\n- Stage III: Multi-image Instruction-tuning. \n  ```bash\n  bash MultiImageSFT.sh\n  ```\n\n### 4. Evaluation\n\n- Command Line Interface\n\n```bash\npython cli.py --model_dir path-to-longllava\n```\n\n\n- Model Inference\n\n```python\nquery = 'What does the picture show?'\nimage_paths = ['image_path1'] # image or video path\n\nfrom cli import Chatbot\nbot = Chatbot(path-to-longllava)\noutput = bot.chat(query, image_paths)\nprint(output) # Prints the output of the model\n```\n\n- Benchmarks\n```bash\npython Eval.sh\n```\n\n\n### 5. Reproduce other results in Paper\n\n- FLOPs\n```bash\npython /utils/cal_flops.py\n```\n\n- Prefill Time \u0026 Throughput \u0026 GPU Memory Usage\n```bash\npython ./benchmarks/Efficiency/evaluate.py\npython ./benchmarks/Efficiency/evaluatevllm.py\n```\n\n- DownCycling\nTo Transfer Jamba-MoE to Dense \n```bash\npython ./utils/dense_downcycling.py\n```\n\n\n## TO DO\n\n- [ ] Release Data Construction Code\n\n## Acknowledgement\n\n- [LLaVA](https://github.com/haotian-liu/LLaVA): Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.\n\n## Citation\n\n```\n@misc{wang2024longllavascalingmultimodalllms,\n      title={LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via Hybrid Architecture}, \n      author={Xidong Wang and Dingjie Song and Shunian Chen and Chen Zhang and Benyou Wang},\n      year={2024},\n      eprint={2409.02889},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2409.02889}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFreedomIntelligence%2FLongLLaVA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFreedomIntelligence%2FLongLLaVA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFreedomIntelligence%2FLongLLaVA/lists"}