{"id":14497787,"url":"https://github.com/Yuliang-Liu/Monkey","last_synced_at":"2025-08-30T20:31:49.064Z","repository":{"id":206441496,"uuid":"716626889","full_name":"Yuliang-Liu/Monkey","owner":"Yuliang-Liu","description":"【CVPR 2024 Highlight】Monkey (LMM): Image Resolution and Text Label Are Important Things for Large Multi-modal Models","archived":false,"fork":false,"pushed_at":"2024-11-27T11:06:51.000Z","size":85968,"stargazers_count":1855,"open_issues_count":12,"forks_count":132,"subscribers_count":22,"default_branch":"main","last_synced_at":"2024-12-26T13:04:57.627Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Yuliang-Liu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-09T14:31:48.000Z","updated_at":"2024-12-24T01:58:52.000Z","dependencies_parsed_at":"2024-12-12T11:03:23.971Z","dependency_job_id":"2c2d67ad-e277-4d06-ad9b-9bda88b20b77","html_url":"https://github.com/Yuliang-Liu/Monkey","commit_stats":null,"previous_names":["yuliang-liu/monkey"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yuliang-Liu%2FMonkey","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yuliang-Liu%2FMonkey/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yuliang-Liu%2FMonkey/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yuliang-Liu%2FMonkey/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Yuliang-Liu","download_url":"https://codeload.github.com/Yuliang-Liu/Monkey/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231521942,"owners_count":18389472,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-03T12:01:23.345Z","updated_at":"2025-08-30T20:31:49.057Z","avatar_url":"https://github.com/Yuliang-Liu.png","language":"Python","funding_links":[],"categories":["大模型列表","多模态大模型","Vision LLM for Generation","📖 Related Papers","Python"],"sub_categories":["网络服务_其他","2024.3 ###"],"readme":"\u003cp align=\"center\"\u003e\r\n    \u003cimg src=\"https://v1.ax1x.com/2024/08/13/7GXwAh.png\" width=\"500\" style=\"margin-bottom: 0.2;\"/\u003e\r\n\u003cp\u003e\r\n\r\n\u003ch3 align=\"center\"\u003e \u003ca href=\"https://arxiv.org/abs/2311.06607\"\u003eMonkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models\u003c/a\u003e\u003c/h3\u003e\r\n\u003ch2\u003e\u003c/h2\u003e\r\n\r\n\u003ch5 align=\"center\"\u003e Please give us a star ⭐ for the latest update.  \u003c/h5\u003e\r\n\r\n\u003ch5 align=\"center\"\u003e\r\n\r\n \r\n[![arXiv](https://img.shields.io/badge/Arxiv-2311.06607-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2311.06607) \r\n[![License](https://img.shields.io/badge/License-Apache%202.0-yellow)](https://github.com/Yuliang-Liu/Monkey/blob/main/LICENSE) \r\n[![GitHub issues](https://img.shields.io/github/issues/Yuliang-Liu/Monkey?color=critical\u0026label=Issues)](https://github.com/Yuliang-Liu/Monkey/issues?q=is%3Aopen+is%3Aissue)\r\n[![GitHub closed issues](https://img.shields.io/github/issues-closed/Yuliang-Liu/Monkey?color=success\u0026label=Issues)](https://github.com/Yuliang-Liu/Monkey/issues?q=is%3Aissue+is%3Aclosed)  \u003cbr\u003e\r\n\u003c/h5\u003e\r\n\r\n\r\n\r\n\r\n\u003e [**[CVPR 2024] Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models**](https://arxiv.org/abs/2311.06607)\u003cbr\u003e\r\n\u003e Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, Xiang Bai \u003cbr\u003e\r\n[![arXiv](https://img.shields.io/badge/Arxiv-2403.04473-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2311.06607) \r\n[![Source_code](https://img.shields.io/badge/Code-Available-white)](README.md)\r\n[![Detailed Caption](https://img.shields.io/badge/Detailed_Caption-yellow)](http://huggingface.co/datasets/echo840/Detailed_Caption)\r\n[![Model Weight](https://img.shields.io/badge/Model_Weight-gray)](http://huggingface.co/echo840/Monkey)\r\n[![Model Weight in Wisemodel](https://img.shields.io/badge/Model_Weight_in_Wisemodel-gray)](https://www.wisemodel.cn/models/HUST-VLRLab/Monkey/)\r\n\r\n\r\n\r\n\u003e [**TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document**](https://arxiv.org/abs/2403.04473)\u003cbr\u003e\r\n\u003e Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, Xiang Bai \u003cbr\u003e\r\n[![arXiv](https://img.shields.io/badge/Arxiv-2403.04473-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2403.04473) \r\n[![Source_code](https://img.shields.io/badge/Code-Available-white)](monkey_model/text_monkey/README.md)\r\n[![Data](https://img.shields.io/badge/Data-yellow)](https://huggingface.co/datasets/MelosY/TextMonkey_Data/tree/main)\r\n[![Model Weight](https://img.shields.io/badge/Model_Weight-gray)](https://www.modelscope.cn/models/lvskiller/TextMonkey)\r\n\r\n\u003e [**[NeurIPS 2024] MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks**](https://arxiv.org/abs/2406.04801)\u003cbr\u003e\r\n\u003e Xingkui Zhu, Yiran Guan, Dingkang Liang, Yuchao Chen, Yuliang Liu, Xiang Bai \u003cbr\u003e\r\n[![arXiv](https://img.shields.io/badge/Arxiv-2406.04801-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2406.04801)\r\n[![Source_code](https://img.shields.io/badge/Code-Available-white)](https://github.com/Adlith/MoE-Jetpack?tab=readme-ov-file)\r\n\r\n\r\n\u003e [**[ICLR 2025] Mini-Monkey: Multi-Scale Adaptive Cropping for Multimodal Large Language Models**](https://arxiv.org/pdf/2408.02034)\u003cbr\u003e\r\n\u003e Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, Xiang Bai \u003cbr\u003e\r\n[![arXiv](https://img.shields.io/badge/Arxiv-2408.02034-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2408.02034)\r\n[![Source_code](https://img.shields.io/badge/Code-Available-white)](project/mini_monkey)\r\n[![Model Weight in Wisemodel](https://img.shields.io/badge/Model_Weight_in_Wisemodel-gray)](https://www.wisemodel.cn/models/HUST-VLRLab/Mini-Monkey)\r\n[![Model Weight](https://img.shields.io/badge/Model_Weight-gray)](https://huggingface.co/mx262/MiniMokney)\r\n\r\n\u003e [**Liquid: Language Models are Scalable and Unified Multi-modal Generators**](https://arxiv.org/pdf/2408.02034)\u003cbr\u003e\r\n\u003e Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, Xiang Bai\u003cbr\u003e\r\n[![arXiv](https://img.shields.io/badge/Arxiv-2412.04332-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2412.04332)\r\n[![Source_code](https://img.shields.io/badge/Code-Available-white)](https://github.com/FoundationVision/Liquid)\r\n\r\n\u003e [**[ICCV 2025] LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance**](https://arxiv.org/abs/2507.06272)\u003cbr\u003e\r\n\u003e Zhang Li, Biao Yang, Qiang Liu, Shuo Zhang, Zhiyin Ma, Shuo Zhang, Liang Yin, Linger Deng, Yabo Sun, Yuliang Liu, Xiang Bai\u003cbr\u003e\r\n[![arXiv](https://img.shields.io/badge/Arxiv-2507.06272-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2507.06272) \r\n[![Source_code](https://img.shields.io/badge/Code-Available-white)](https://github.com/echo840/LIRA)\r\n\r\n \r\n\u003e [**MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm**](https://arxiv.org/abs/2506.05218)\u003cbr\u003e\r\n\u003e Zhang Li, Yuliang Liu, Qiang Liu, Zhiyin Ma, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiarui Zhang, Xinyu Wang, Xiang Bai\u003cbr\u003e\r\n[![arXiv](https://img.shields.io/badge/Arxiv-2506.05218-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2506.05218) \r\n[![Source_code](https://img.shields.io/badge/Code-Available-white)](https://github.com/Yuliang-Liu/MonkeyOCR)\r\n[![Model Weight](https://img.shields.io/badge/Model_Weight-gray)](https://huggingface.co/echo840/MonkeyOCR)\r\n[![Demo](https://img.shields.io/badge/Demo-blue)](http://vlrlabmonkey.xyz:7685/)\r\n\u003e\r\n\u003e \r\n## News \r\n* ```2025.6.6 ``` 🚀 [MonkeyOCR](https://github.com/Yuliang-Liu/MonkeyOCR): Try our document parsing model — Accurate, Fast, and Easy to Use.\r\n* ```2025.4.17 ``` 🚀 [Liquid](https://arxiv.org/abs/2412.04332): Bridging Text‑to‑Image and Image‑to‑Text in One Framework.\r\n* ```2025.1.23 ``` 🚀 Mini-Monkey is accepted by ICLR 2025. \r\n* ```2024.9.25 ``` 🚀 MoE Jetpack is accepted by NeurIPS 2024.\r\n* ```2024.8.6  ``` 🚀 We release the paper [Mini-Monkey](https://arxiv.org/abs/2408.02034).\r\n* ```2024.4.5  ``` 🚀 Monkey is nominated as CVPR 2024 Highlight paper.\r\n* ```2024.3.8  ``` 🚀 We release the paper [TextMonkey](https://arxiv.org/abs/2403.04473).\r\n* ```2024.2.27 ``` 🚀 Monkey is accepted by CVPR 2024. \r\n* ```2024.1.3  ``` 🚀 Release the basic data generation pipeline. [Data Generation](./data_generation)\r\n* ```2023.11.06``` 🚀 We release the paper [Monkey](https://arxiv.org/abs/2311.06607).\r\n\r\n## 🐳 Model Zoo\r\n\r\nMonkey-Chat\r\n| Model|Language Model|Transformers(HF) |MMBench-Test|CCBench|MME|SeedBench_IMG|MathVista-MiniTest|HallusionBench-Avg|AI2D Test|OCRBench|\r\n|---------------|---------|-----------------------------------------|---|---|---|---|---|---|---|---|\r\n|Monkey-Chat|Qwev-7B|[🤗echo840/Monkey-Chat](https://huggingface.co/echo840/Monkey-Chat)|72.4|48|1887.4|68.9|34.8|39.3|68.5|534|\r\n|Mini-Monkey|internlm2-chat-1_8b|[Mini-Monkey](https://huggingface.co/mx262/MiniMokney)|---|75.5|1881.9|71.3|47.3|38.7|74.7|802|\r\n\r\n\r\n## Environment\r\n\r\n```python\r\nconda create -n monkey python=3.9\r\nconda activate monkey\r\ngit clone https://github.com/Yuliang-Liu/Monkey.git\r\ncd ./Monkey\r\npip install -r requirements.txt\r\n```\r\nYou can download the corresponding version of flash_attention from https://github.com/Dao-AILab/flash-attention/releases/ and use the following code to install:\r\n```python\r\npip install flash_attn-2.3.5+cu117torch2.0cxx11abiFALSE-cp39-cp39-linux_x86_64.whl --no-build-isolation\r\n```\r\n\r\n\r\n## Train\r\n\r\nWe also offer Monkey's model definition and training code, which you can explore above. You can execute the training code through executing `finetune_ds_debug.sh` for Monkey and `finetune_textmonkey.sh` for TextMonkey.\r\n\r\nThe json file used for Monkey training can be downloaded at [Link](https://drive.google.com/file/d/18z_uQTe8Jq61V5rgHtxOt85uKBodbvw1/view?usp=sharing).\r\n\r\n\r\n## Inference\r\nRun the inference code for Monkey and Monkey-Chat:\r\n```\r\npython ./inference.py --model_path MODEL_PATH  --image_path IMAGE_PATH  --question \"YOUR_QUESTION\"\r\n```\r\n\r\n\r\n## Demo\r\n\r\nDemo is fast and easy to use. Simply uploading an image from your desktop or phone, or capture one directly. \r\n[Demo_chat](http://vlrlab-monkey.xyz:7681) is also launched as an upgraded version of the original demo to deliver an enhanced interactive experience.\r\n\r\nWe also provide the source code and the model weight for the original demo, allowing you to customize certain parameters for a more unique experience. The specific operations are as follows:\r\n 1. Make sure you have configured the [environment](#environment).\r\n 2. You can choose to use the demo offline or online:\r\n- **Offline:** \r\n\t- Download the [Model Weight](http://huggingface.co/echo840/Monkey). \r\n\t- Modify `DEFAULT_CKPT_PATH=\"pathto/Monkey\"` in the `demo.py` file to your model weight path. \r\n\t- Run the demo using the following command: \r\n\t```\r\n\tpython demo.py\r\n\t```\r\n- **Online:** \r\n\t- Run the demo and download model weights online with the following command: \r\n\t```\r\n\tpython demo.py -c echo840/Monkey \r\n\t```\r\n\r\nFor TextMonkey you can download the model weight from [Model Weight](https://www.modelscope.cn/models/lvskiller/TextMonkey)  and run the demo code:\r\n``` python\r\npython demo_textmonkey.py -c model_path\r\n```\r\n\r\nBefore 14/11/2023, we have observed that for some random pictures Monkey can achieve more accurate results than GPT4V.  \r\n\u003cbr\u003e\r\n\u003cp align=\"center\"\u003e\r\n    \u003cimg src=\"https://v1.ax1x.com/2024/04/13/7yS2yq.jpg\" width=\"666\"/\u003e\r\n\u003cp\u003e\r\n\u003cbr\u003e\r\n\r\nBefore 31/1/2024, Monkey-chat achieved the fifth rank in the Multimodal Model category on [OpenCompass](https://opencompass.org.cn/home). \r\n\u003cbr\u003e\r\n\u003cp align=\"center\"\u003e\r\n    \u003cimg src=\"https://v1.ax1x.com/2024/04/13/7yShXL.jpg\" width=\"666\"/\u003e\r\n\u003cp\u003e\r\n\u003cbr\u003e\r\n\r\n \r\n## Dataset\r\nYou can download the training and testing data used by monkey from [Monkey_Data](https://huggingface.co/datasets/echo840/Monkey_Data).\r\n\r\nThe json file used for Monkey training can be downloaded at [Link](https://drive.google.com/file/d/18z_uQTe8Jq61V5rgHtxOt85uKBodbvw1/view?usp=sharing).\r\n\r\nThe data from our multi-level description generation method is now open-sourced and available for download at [Link](https://huggingface.co/datasets/echo840/Detailed_Caption). We already upload the images used in multi-level description. Examples:\r\n\r\n\u003cbr\u003e\r\n\u003cp align=\"center\"\u003e\r\n    \u003cimg src=\"https://v1.ax1x.com/2024/04/13/7yS6Ss.jpg\" width=\"666\"/\u003e\r\n\u003cp\u003e\r\n\u003cbr\u003e\r\n\t\r\nYou can download train images of Monkey from [Train](https://pan.baidu.com/s/1svSjXTxWpI-3boALgSeLlw). Extraction code: 4hdh\r\n\r\nYou can download test images and jsonls of Monkey from [Test](https://pan.baidu.com/s/1ABrQKeE9QBeKvtGzXfM8Eg). Extraction code: 5h71\r\n\r\nThe images are from CC3M, COCO Caption, TextCaps, VQAV2, OKVQA, GQA, ScienceQA, VizWiz, TextVQA, OCRVQA, ESTVQA, STVQA, AI2D and DUE_Benchmark. When using the data, it is necessary to comply with the protocols of the original dataset.\r\n\r\n## Evaluate\r\n\r\nWe offer evaluation code for 14 Visual Question Answering (VQA) datasets in the `evaluate_vqa.py` file, facilitating a quick verification of results.  The specific operations are as follows:\r\n\r\n 1. Make sure you have configured the [environment](#environment).\r\n 2. Modify `sys.path.append(\"pathto/Monkey\")`  to the project path.\r\n 3. Prepare the datasets required for evaluation. \r\n 4. Run the evaluation code.\r\n\r\n Take ESTVQA as an example:\r\n - Prepare data according to the following directory structure:\r\n```\r\n├── data\r\n|\t├── estvqa\r\n|\t\t├── test_image\r\n|\t\t\t├── {image_path0}\r\n|\t\t\t├── {image_path1}\r\n|\t\t\t\t  ·\r\n|\t\t\t\t  ·\r\n|\t├── estvqa.jsonl\r\n```\r\n - Example of the format of each line of the annotated `.jsonl` file:\r\n```\r\n{\"image\": \"data/estvqa/test_image/011364.jpg\", \"question\": \"What is this store?\", \"answer\": \"pizzeria\", \"question_id\": 0}\r\n```\r\n - Modify the dictionary `ds_collections`:\r\n```\r\nds_collections = {\r\n\t'estvqa_test': {\r\n\t\t'test': 'data/estvqa/estvqa.jsonl',\r\n\t\t'metric': 'anls',\r\n\t\t'max_new_tokens': 100,\r\n\t},\r\n\t...\r\n}\r\n```\r\n - Run the following command:\r\n```\r\nbash eval/eval.sh 'EVAL_PTH' 'SAVE_NAME'\r\n```\r\n\r\n\r\n## Citing Monkey\r\nIf you wish to refer to the baseline results published here, please use the following BibTeX entries:\r\n\r\n```BibTeX\r\n@inproceedings{li2023monkey,\r\n  title={Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models},\r\n  author={Li, Zhang and Yang, Biao and Liu, Qiang and Ma, Zhiyin and Zhang, Shuo and Yang, Jingxu and Sun, Yabo and Liu, Yuliang and Bai, Xiang},\r\n  booktitle={proceedings of the IEEE/CVF conference on computer vision and pattern recognition},\r\n  year={2024}\r\n}\r\n@article{liu2024textmonkey,\r\n  title={TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document},\r\n  author={Liu, Yuliang and Yang, Biao and Liu, Qiang and Li, Zhang and Ma, Zhiyin and Zhang, Shuo and Bai, Xiang},\r\n  journal={arXiv preprint arXiv:2403.04473},\r\n  year={2024}\r\n}\r\n@article{huang2024mini,\r\n  title={Mini-Monkey: Multi-Scale Adaptive Cropping for Multimodal Large Language Models},\r\n  author={Huang, Mingxin and Liu, Yuliang and Liang, Dingkang and Jin, Lianwen and Bai, Xiang},\r\n  journal={arXiv preprint arXiv:2408.02034},\r\n  year={2024}\r\n}\r\n@article{deng2024r,\r\n  title={R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models},\r\n  author={Deng, Linger and Liu, Yuliang and Li, Bohan and Luo, Dongliang and Wu, Liang and Zhang, Chengquan and Lyu, Pengyuan and Zhang, Ziyang and Zhang, Gang and Ding, Errui and others},\r\n  journal={arXiv preprint arXiv:2410.17885},\r\n  year={2024}\r\n}\r\n```\r\n\r\n## Acknowledgement\r\nThe Monkey series is primarily focused on exploring techniques such as image resolution enhancement and token compression methods to improve the performance of existing multimodal large models. For instance, earlier versions of Monkey and TextMonkey were based on QwenVL, while MiniMonkey is based on InternVL2 and miniCPM, among others. Thanks to\r\n[Qwen-VL](https://github.com/QwenLM/Qwen-VL.git), [LLAMA](https://github.com/meta-llama/llama), [LLaVA](https://github.com/haotian-liu/LLaVA), [OpenCompass](https://github.com/open-compass/opencompass), [InternLM](https://github.com/InternLM/InternLM), and [InternVL](https://github.com/OpenGVLab/InternVL).  \r\n\r\n\r\n## Copyright\r\nMonkey project is intended for non-commercial use only. For commercial inquiries or to explore more advanced versions of the Monkey series LMMs (\u003c1b, 2b, 7b, 72b), please contact Prof. Yuliang Liu at ylliu@hust.edu.cn. \r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYuliang-Liu%2FMonkey","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FYuliang-Liu%2FMonkey","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYuliang-Liu%2FMonkey/lists"}