{"id":24251390,"url":"https://github.com/FreedomIntelligence/TRIM","last_synced_at":"2025-09-23T16:31:19.125Z","repository":{"id":261940908,"uuid":"861632312","full_name":"FreedomIntelligence/TRIM","owner":"FreedomIntelligence","description":"We introduce new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance.","archived":false,"fork":false,"pushed_at":"2024-12-09T07:04:44.000Z","size":28199,"stargazers_count":11,"open_issues_count":0,"forks_count":0,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-01-10T00:20:29.310Z","etag":null,"topics":["llm","mllm","multimodal","vision-and-language","vision-language-model","vlm"],"latest_commit_sha":null,"homepage":"https://github.com/FreedomIntelligence/TRIM","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-23T08:55:15.000Z","updated_at":"2025-01-07T12:02:06.000Z","dependencies_parsed_at":"2024-11-09T12:29:20.682Z","dependency_job_id":"51e75f7a-40a3-443c-9115-3928ab760e6a","html_url":"https://github.com/FreedomIntelligence/TRIM","commit_stats":null,"previous_names":["freedomintelligence/trim"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FTRIM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FTRIM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FTRIM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FTRIM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/TRIM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233985941,"owners_count":18761563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","mllm","multimodal","vision-and-language","vision-language-model","vlm"],"created_at":"2025-01-15T02:50:53.402Z","updated_at":"2025-09-23T16:31:12.801Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":["📖 Related Papers"],"sub_categories":["2024.9 ###"],"readme":"# ✂️ TRIM\n\n# Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs\n\n[Dingjie Song](https://bbsngg.github.io/), Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, Benyou Wang*\n\n![Python 3.10+](https://img.shields.io/badge/Python-3.10-lightblue) ![Pytorch 2.1.1](https://img.shields.io/badge/PyTorch-2.1-lightblue) ![transformers](https://img.shields.io/badge/transformers-4.37.0.dev0%2B-lightblue) ![accelerate](https://img.shields.io/badge/accelerate-0.28.0-lightblue)\n\u003c/center\u003e\n\n[**🤗 Model**](https://huggingface.co/FreedomIntelligence/llava-v1.5-7b-TRIM) | [**📖 arXiv**](https://arxiv.org/abs/2409.10994) | [**GitHub**](https://github.com/FreedomIntelligence/TRIM)\n\n## 🌈 Update\n\n\n- **[2024.11.30]** 🎉🎉🎉 TRIM is accepted by **COLING25**!🎉🎉🎉\n- **[2024.9.30]** 🎉🎉🎉 TRIM is public!🎉🎉🎉\n\n\n\n## Contents\n\n- [Introduction](#introduction)\n- [Dataset Preparation](#preparation)\n- [Run](#Run)\n- [License](#license)\n- [Contact](#contact)\n- [Citation](#Citation)\n\n\n\n## Introduction\n\nWe introduce new approach, **T**oken **R**eduction using CL**I**P **M**etric (**TRIM**), aimed at improving the efficiency of MLLMs without sacrificing their performance. Inspired by human attention patterns in Visual Question Answering (VQA) tasks, TRIM presents a fresh perspective on the selection and reduction of image tokens. The TRIM method has been extensively tested across 12 datasets, and the results demonstrate a significant reduction in computational overhead while maintaining a consistent level of performance. This research marks a critical stride in efficient MLLM development, promoting greater accessibility and sustainability of high-performing models.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/FreedomIntelligence/TRIM/blob/main/images/TRIM.png\" alt=\"Our approach\" width=\"100%\"\u003e\n\u003c/div\u003e\n\n## Preparation\n\n### 🤖 Environment Setup\nPlease refer to [LLaVA](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#install). 😊\n\n## Run\n\n### Step.0: Set the environment the same as LLaVA-1.5\n\nNote that the core of our proposed module is [here](https://github.com/FreedomIntelligence/TRIM/blob/main/llava/model/multimodal_encoder/clip_encoder.py) in the CLIP image encoder.\n\n### Step.1: Model preparation\n\n#### Train model with TRIM\n\nIf you want to reproduce the result of the model trained with TRIM, configure the [dataset](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#visual-instruction-tuning) path, [vision_tower](https://huggingface.co/openai/clip-vit-large-patch14-336) path, [projecter](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md#projector-weights) path and [LLM](https://huggingface.co/lmsys/vicuna-7b-v1.5) checkpoint path in the training script.\n\nPlease set `reduce_func` as `TRIM`, `reduce_func_param` as `-1` for automatic selection.\n\n```shell\nbash scripts/finetune_8gpu_TRIM.sh\n```\n\n#### or Download checkpoints\n\nIf you want to use TRIM without training the model, please download the checkpoints from [Huggingface liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b) or [Our fine-tuned 7B model with TRIM](https://huggingface.co/FreedomIntelligence/llava-v1.5-7b-TRIM) or [Our fine-tuned 13B model with TRIM](https://huggingface.co/FreedomIntelligence/llava-v1.5-13b-TRIM).\n\n### Step.2 (for inference): Change the methods (TRIM).\n\nIf you wish to implement TRIM in another model, such as liuhaotian/llava-v1.5-7b in Huggingface, **please add the following line** to the `config.json` file in the model's directory.\n\n```json\n    \"mm_vision_token_reduce_func\": \"TRIM:-1\",\n```\n\n### Step.3 (for evaluation): Run the evaluation script.\n\nIf you want to reproduce the result in our paper, for all benchmark，the evaluation script is：\n\n```shell\nbash eval_all_benchmarks.sh\n```\n\nFor example, the evaluation for TextVQA is:\n\n```shell\nbash scripts/v1_5/eval/testvqa.sh\n```\n\nFor other inference scripts, refer to [LLaVA Evaluation](https://github.com/haotian-liu/LLaVA/blob/main/docs/Evaluation.md).\n\n\n\n## License\n\n[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-yellow.svg)](https://github.com/FreedomIntelligence/TRIM/blob/main/LICENSE)\n[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-orange.svg)](https://github.com/FreedomIntelligence/TRIM/blob/main/DATA_LICENSE)\n\nAll software is licensed under the Apache License, Version 2.0 (Apache 2.0).\nAll other materials are licensed under the Creative Commons Attribution 4.0 International License (CC-BY).\n\n\n\n## Contact\n\n- Dingjie Song: dingjiesong.cs@gmail.com\n- Benyou Wang: wangbenyou@cuhk.edu.cn\n\n## Citation\n\nIf you find this repository helpful, please consider citing it:\n\n```\n@misc{song2024moresimpleeffectivetoken,\n      title={Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs},\n      author={Dingjie Song and Wenjun Wang and Shunian Chen and Xidong Wang and Michael Guan and Benyou Wang},\n      year={2024},\n      eprint={2409.10994},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2409.10994},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFreedomIntelligence%2FTRIM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFreedomIntelligence%2FTRIM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFreedomIntelligence%2FTRIM/lists"}