{"id":24251375,"url":"https://github.com/CircleRadon/TokenPacker","last_synced_at":"2025-09-23T16:31:14.868Z","repository":{"id":247444379,"uuid":"823399026","full_name":"CircleRadon/TokenPacker","owner":"CircleRadon","description":"The code for \"TokenPacker: Efficient Visual Projector for Multimodal LLM\".","archived":false,"fork":false,"pushed_at":"2024-12-26T12:05:09.000Z","size":42809,"stargazers_count":235,"open_issues_count":2,"forks_count":9,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-12-26T13:23:51.398Z","etag":null,"topics":["connector","lmm","mllm","token-reduction","tokenpacker","visual-projector"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CircleRadon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-03T01:04:35.000Z","updated_at":"2024-12-26T12:05:13.000Z","dependencies_parsed_at":"2024-07-08T21:15:29.557Z","dependency_job_id":"14074c15-c0b4-476d-ad57-96d706b1eb65","html_url":"https://github.com/CircleRadon/TokenPacker","commit_stats":null,"previous_names":["circleradon/tokenpacker"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CircleRadon%2FTokenPacker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CircleRadon%2FTokenPacker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CircleRadon%2FTokenPacker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CircleRadon%2FTokenPacker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CircleRadon","download_url":"https://codeload.github.com/CircleRadon/TokenPacker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233985939,"owners_count":18761562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["connector","lmm","mllm","token-reduction","tokenpacker","visual-projector"],"created_at":"2025-01-15T02:50:53.024Z","updated_at":"2025-09-23T16:31:14.863Z","avatar_url":"https://github.com/CircleRadon.png","language":"Python","funding_links":[],"categories":["📖 Related Papers"],"sub_categories":["2024.7 ###"],"readme":"\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"assets/title.png\"  width=\"90%\"\u003e\n\u003c/p\u003e\n\n\n\u003cdiv align=center\u003e\n\u003ca href=\"\" target=\"_blank\"\u003e\n    \u003cimg alt=\"TokenPacker-v1\" src=\"https://img.shields.io/badge/TokenPaker-v1-BFE57E\" height=\"25\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/2407.02392\" target=\"_blank\"\u003e\n    \u003cimg alt=\"arXiv\" src=\"https://img.shields.io/badge/arXiv-2407.02392-red?logo=arxiv\" height=\"25\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/collections/sunshine-lwt/tokenpacker-66a234618f0d2327e0cf2cb1\" target=\"_blank\"\u003e\n    \u003cimg alt=\"HF Model\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20_Model-HuggingFace-ffc107?color=ffc107\u0026logoColor=white\" height=\"25\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://zhuanlan.zhihu.com/p/707021763\" target=\"_blank\"\u003e\n    \u003cimg alt=\"ZhiHu\" src=\"https://img.shields.io/badge/Blog-ZhiHu-1E90FF?logo=zhihu\u0026logoColor=02B5FD\" height=\"25\" /\u003e\n\u003c/a\u003e   \n \u003c/div\u003e\n\n\n---\n\n## Comparisons with existing methods 💡\n\u003c!-- \u003cimg src=\"./assets/compare.png\" width=\"80%\"\u003e --\u003e\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/compare.png\"  width=\"60%\"\u003e\n\u003c/p\u003e\n\n## Updates 📌\n- [2025/5/23] TokenPacker is accepted by **IJCV** 🎉🎉🎉. \n- [2024/10/22] We integrated TokenPacker-HD framework with [Osprey](https://github.com/CircleRadon/Osprey) to achieve fine-grained high-resolution pixel-level understanding with large performance gains. Please see the codes in this [branch](https://github.com/CircleRadon/TokenPacker/tree/tokenpacker-hd-osprey) for your reference. \n- [2024/7/25] We released [checkpoints](https://huggingface.co/collections/sunshine-lwt/tokenpacker-66a234618f0d2327e0cf2cb1), please check them.\n- [2024/7/3] We released the [paper](https://arxiv.org/abs/2407.02392) of our TokenPacker on Arxiv.\n- [2024/7/3] We released the training and inference codes. \n\n\n## What is TokenPacker 👀\nTokenPacker is a novel visual projector, which adopts a `coarse-to-fine` scheme\nto inject the enriched characteristics to generate the condensed visual tokens. Using TokenPacker, we can compress the\nvisual tokens by **75%∼89%**, while achieves comparable or even better performance\nacross diverse benchmarks with significantly higher efficiency.\n\u003cimg src=\"./assets/framework.png\" width=\"800px\"\u003e\n\n#### Algorithms\nWe provide the pseudo-codes to showcase the detailed processing flow.\n\u003cimg src=\"./assets/Algorithm.png\" width=\"800px\"\u003e\n\n#### Core codes\nAs a visual projector, TokenPacker is implemented by a `class TokenPacker`, which can be found in [multimodal_projector/builder.py](./llava/model/multimodal_projector/builder.py#L39)\n\n#### Comparisons with various projectors \n\u003cimg src=\"./assets/projector_comparsion.jpg\" width=\"800px\"\u003e\n\n\n## High-Resolution Image Understanding with TokenPacker 🔬\nTo support efficient `high-resolution` image understanding, we further develop an effective image\ncropping method `TokenPacker-HD`.\n\u003cimg src=\"./assets/hd.png\" width=\"800px\"\u003e\n\n\n## Install 🛠️\n1. Clone this repository and navigate to TokenPacker folder\n```\ngit clone https://github.com/CircleRadon/TokenPacker.git\ncd TokenPacker\n```\n2. Install packages\n```\nconda create -n tokenpacker python=3.10 -y\nconda activate tokenpacker\npip install --upgrade pip  # enable PEP 660 support\npip install -e .\n```\n3. Install additional packages for training cases\n```\npip install -e \".[train]\"\npip install flash-attn --no-build-isolation\n```\n\n## Training 🚀\n\n### LLaVA-TokenPacker\n\n#### Dataset\nTo make a fair comparison, we use the same training data as in [LLaVA-1.5](https://github.com/haotian-liu/LLaVA), i.e., [LLaVA-Pretrain-558K](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain/tree/main) for stage 1, and  [Mix665k](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/tree/main) for stage 2.\n\n#### Training \n- Stage1: Image-Text Alignment Pre-training\n```shell\nbash scripts/v1_5/pretrain.sh\n```\n- Stage2: Visual Instruction Tuning\n```shell\nbash scripts/v1_5/finetune.sh\n```\nNote: Using `--scale_factor` to control compression ratio, support [2,3,4]\n\n### LLaVA-TokenPacker-HD\n\n#### Dataset\nTo obtain the competitive high-resolution performance, we use 2.7M data as organized by [Mini-Gemini](https://github.com/dvlab-research/MGM#Dataset), i.e., 1.2M for stage 1 and 1.5M for stage 2.\n\n#### Training \n- Stage1: Image-Text Alignment Pre-training\n```shell\nbash scripts/v1_5/pretrain_hd.sh\n```\n- Stage2: Visual Instruction Tuning\n```shell\nbash scripts/v1_5/finetune_hd.sh\n```\n\nNote: \n- Using `--scale_factor` to control compression ratio, support [2,3,4].\n- Using `--patch_num` to control max patch dividing number, support [9,16,25].\n\n\n## Experiments\n\n\u003cimg src=\"./assets/ex1.png\" width=\"800px\"\u003e\n\n\u003cimg src=\"./assets/high-reso.jpg\" width=\"800px\"\u003e\n\n\n## Model Zoo\n\n| Model              |  Max Res.   |  Compre. Ratio  |  Token Num.  |  Max Patch Num.  |                                           Training Data                                            | Download                                                                              |\n|--------------------|:-----------:|:---------------:|:------------:|:----------------:|:--------------------------------------------------------------------------------------------------:|---------------------------------------------------------------------------------------|\n| TokenPacker-7b     |   336x336   |       1/4       |     144      |        -         |                                             558K+665K                                              | [checkpoints](https://huggingface.co/sunshine-lwt/TokenPacker-7b-144token/tree/main)  |\n| TokenPacker-13b     |   336x336   |       1/4       |     144      |        -         |                                             558K+665K                                              | [checkpoints](https://huggingface.co/sunshine-lwt/TokenPacker-13b-144token/tree/main) |\n| TokenPacker-HD-7b  |  1088x1088  |       1/4       |     ~954     |        9         |                                             1.2M+1.5M                                              | [checkpoints](https://huggingface.co/sunshine-lwt/TokenPacker-HD-7b-9patch-144token/tree/main) |\n| TokenPacker-HD-13b |  1088x1088  |       1/4       |     ~954     |        9         |                                             1.2M+1.5M                                              | [checkpoints](https://huggingface.co/sunshine-lwt/TokenPacker-HD-13b-9patch-144token/tree/main) |\n| TokenPacker-HD-13b |  1344x1344  |       1/4       |    ~1393     |        16        |                                             1.2M+1.5M                                              | [checkpoints](https://huggingface.co/sunshine-lwt/TokenPacker-HD-13b-16patch-144token/tree/main) |\n| TokenPacker-HD-13b |  1344x1344  |       1/9       |     ~619     |        16        |                                             1.2M+1.5M                                              | [checkpoints](https://huggingface.co/sunshine-lwt/TokenPacker-HD-13b-16patch-64token/tree/main)                                                                       |\n| TokenPacker-HD-13b |  1344x1344  |      1/16       |     ~347     |        16        |                                             1.2M+1.5M                                              |  [checkpoints](https://huggingface.co/sunshine-lwt/TokenPacker-HD-13b-16patch-36token/tree/main)                                                                      |\n\nNote: \n- The `token number` of TokenPacker-HD is the `average` statistically across all training and test data.\n- The training data of `558K+665K` follows LLaVA-1.5, the one of `1.2M+1.5M` follows Mini-Gemini.\n- All LLMs use Vicuna-7b/13b  as based LLM.\n\n\n## Visualization\nWe provide some visual examples.\n\n\u003cimg src=\"./assets/vis-1.jpg\" width=\"800px\"\u003e\n\n\nHigh-resolution image understanding.\n\u003cimg src=\"./assets/vis-2.jpg\" width=\"800px\"\u003e\n\n\n## TODO List 📝\n- [x] Release the training and inference codes.\n- [x] Release all checkpoints.\n\n\n## Acknowledgement 💌\n- [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA): the codebase we built upon.\n- [Mini-Gemini](https://github.com/dvlab-research/MGM): the organized data we used for training high-resolution method.\n  \n## More ## \nFor more recent related works, please refer to this repo of  [Awesome-Token-Compress](https://github.com/daixiangzi/Awesome-Token-Compress).\n\n## BibTeX 🖊️\n```\n@misc{TokenPacker,\n  title={TokenPacker: Efficient Visual Projector for Multimodal LLM},\n  author={Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu and Lei Zhang},\n  year={2024},\n  eprint={2407.02392},\n  archivePrefix={arXiv},\n  primaryClass={cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCircleRadon%2FTokenPacker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCircleRadon%2FTokenPacker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCircleRadon%2FTokenPacker/lists"}