{"id":15035286,"url":"https://github.com/x-plug/mplug-docowl","last_synced_at":"2025-05-14T08:06:06.683Z","repository":{"id":179370140,"uuid":"661919338","full_name":"X-PLUG/mPLUG-DocOwl","owner":"X-PLUG","description":"mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding","archived":false,"fork":false,"pushed_at":"2024-12-24T07:54:07.000Z","size":109773,"stargazers_count":2169,"open_issues_count":69,"forks_count":127,"subscribers_count":33,"default_branch":"main","last_synced_at":"2025-05-11T07:34:11.847Z","etag":null,"topics":["chart-understanding","document-understanding","mllm","multimodal","multimodal-large-language-models","table-understanding"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/X-PLUG.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-04T01:18:19.000Z","updated_at":"2025-05-08T11:06:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"ae9cef63-b042-4d10-aa39-ed02df9539e2","html_url":"https://github.com/X-PLUG/mPLUG-DocOwl","commit_stats":{"total_commits":132,"total_committers":7,"mean_commits":"18.857142857142858","dds":"0.16666666666666663","last_synced_commit":"fab2fd7214a736794341b1451b0befbad99067ae"},"previous_names":["x-plug/mplug-docowl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/X-PLUG%2FmPLUG-DocOwl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/X-PLUG%2FmPLUG-DocOwl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/X-PLUG%2FmPLUG-DocOwl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/X-PLUG%2FmPLUG-DocOwl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/X-PLUG","download_url":"https://codeload.github.com/X-PLUG/mPLUG-DocOwl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254101615,"owners_count":22014909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chart-understanding","document-understanding","mllm","multimodal","multimodal-large-language-models","table-understanding"],"created_at":"2024-09-24T20:28:04.875Z","updated_at":"2025-05-14T08:06:06.643Z","avatar_url":"https://github.com/X-PLUG.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"assets/mPLUG_new1.png\" width=\"80%\"\u003e\n\u003c/div\u003e\n\n\n\u003cdiv align=\"center\"\u003e\n\u003ch2\u003eThe Powerful Multi-modal LLM Family\n\nfor OCR-free Document Understanding\u003ch2\u003e\n\u003cstrong\u003eAlibaba Group\u003c/strong\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://trendshift.io/repositories/9061\" target=\"_blank\"\u003e\u003cimg src=\"https://trendshift.io/api/badge/repositories/9061\" alt=\"DocOwl | Trendshift\" style=\"width: 250px; height: 55px;\" width=\"250\" height=\"55\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n## 📢 News\n* 🔥🔥🔥 [2024.12.24] We have released the training code of DocOwl2 by [**ms-swift**](https://github.com/modelscope/ms-swift)! Now you can finetune a stronger model with your own data based on DocOwl2!\n* 🔥🔥🔥 [2024.9.28] We have released the training data, inference code and evaluation code of [DocOwl2](./DocOwl2/) on both **HuggingFace** 🤗 and **ModelScope** \u003cimg src=\"./assets/modelscope.png\" width='20'\u003e.\n* 🔥🔥🔥 [2024.9.20] Our paper [DocOwl 1.5](http://arxiv.org/abs/2403.12895) and [TinyChart](https://arxiv.org/abs/2404.16635) is accepted by EMNLP 2024.\n* 🔥🔥 [2024.9.06] We release the arxiv paper of [mPLUG-DocOwl 2](https://arxiv.org/abs/2409.03420), a SOTA 8B Multimodal LLM on OCR-free Multipage Document Understanding, each document image is encoded with just 324 tokens!\n* [2024.7.16] Our paper [PaperOwl](https://arxiv.org/abs/2311.18248) is accepted by ACM MM 2024.\n* [2024.5.08] We have released the training code of [DocOwl1.5](./DocOwl1.5/) supported by DeepSpeed. You can now finetune a stronger model based on DocOwl1.5!\n* [2024.4.26] We release the arxiv paper of [TinyChart](https://arxiv.org/abs/2404.16635), a SOTA 3B Multimodal LLM for Chart Understanding with Program-of-Throught ability (ChartQA: 83.6 \u003e Gemin-Ultra 80.8 \u003e GPT4V 78.5). The demo of TinyChart is available on [HuggingFace](https://huggingface.co/spaces/mPLUG/TinyChart-3B) 🤗. Both codes, models and data are released in [TinyChart](./TinyChart/).\n* [2024.4.3] We build demos of DocOwl1.5 on both [ModelScope](https://modelscope.cn/studios/iic/mPLUG-DocOwl/) \u003cimg src=\"./assets/modelscope.png\" width='20'\u003e and [HuggingFace](https://huggingface.co/spaces/mPLUG/DocOwl) 🤗, supported by the DocOwl1.5-Omni. The source codes of launching a local demo are also released in [DocOwl1.5](./DocOwl1.5/).\n* [2024.3.28] We release the training data (DocStruct4M, DocDownstream-1.0, DocReason25K), codes and models (DocOwl1.5-stage1, DocOwl1.5, DocOwl1.5-Chat, DocOwl1.5-Omni) of [mPLUG-DocOwl 1.5](./DocOwl1.5/) on both **HuggingFace** 🤗 and **ModelScope** \u003cimg src=\"./assets/modelscope.png\" width='20'\u003e.\n* [2024.3.20] We release the arxiv paper of [mPLUG-DocOwl 1.5](http://arxiv.org/abs/2403.12895), a SOTA 8B Multimodal LLM on OCR-free Document Understanding (DocVQA 82.2, InfoVQA 50.7, ChartQA 70.2, TextVQA 68.6).\n* [2024.01.13] Our Scientific Diagram Analysis dataset [M-Paper](https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/PaperOwl) has been available on both **HuggingFace** 🤗 and **ModelScope** \u003cimg src=\"./assets/modelscope.png\" width='20'\u003e, containing 447k high-resolution diagram images and corresponding paragraph analysis.\n* [2023.10.13] Training data, models of [mPLUG-DocOwl](./DocOwl/)/[UReader](./UReader/) has been open-sourced.\n* [2023.10.10] Our paper [UReader](https://arxiv.org/abs/2310.05126) is accepted by EMNLP 2023.\n\u003c!-- * 🔥 [10.10] The source code and instruction data will be released in [UReader](https://github.com/LukeForeverYoung/UReader). --\u003e\n* [2023.07.10] The demo of mPLUG-DocOwl on [ModelScope](https://modelscope.cn/studios/damo/mPLUG-DocOwl/summary) is avaliable.\n* [2023.07.07] We release the technical report and evaluation set of mPLUG-DocOwl.\n\n## 🤖 Models\n- [**mPLUG-DocOwl2**](./DocOwl2/) (Arxiv 2024) - mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding\n\n- [**mPLUG-DocOwl1.5**](./DocOwl1.5/) (EMNLP 2024) - mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding\n\n- [**TinyChart**](./TinyChart/) (EMNLP 2024) - TinyChart: Efficient Chart Understanding with\nVisual Token Merging and Program-of-Thoughts Learning\n\n- [**mPLUG-PaperOwl**](./PaperOwl/) (ACM MM 2024) - mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model\n\n- [**UReader**](./UReader/) (EMNLP 2023) - UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model\n\n- [**mPLUG-DocOwl**](./DocOwl/) (Arxiv 2023) - mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding\n\n## 📺 Online Demo\nNote: The demo of HuggingFace is not as stable as ModelScope because the GPU in ZeroGPU Spaces of HuggingFace is dynamically assigned.\n### 📖 DocOwl 1.5\n- 🤗 [HuggingFace Space](https://huggingface.co/spaces/mPLUG/DocOwl)\n\n- \u003cimg src=\"assets/modelscope.png\" width='20'\u003e [ModelScope Space](https://modelscope.cn/studios/iic/mPLUG-DocOwl/) \n\n### 📈 TinyChart-3B\n- 🤗 [HuggingFace Space](https://huggingface.co/spaces/mPLUG/TinyChart-3B)\n\n\n## 🌰 Cases\n\n![images](assets/docowl2_github_case.jpg)\n\n  \n\n## Related Projects\n\n* [mPLUG](https://github.com/alibaba/AliceMind/tree/main/mPLUG).\n* [mPLUG-2](https://github.com/alibaba/AliceMind).\n* [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fx-plug%2Fmplug-docowl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fx-plug%2Fmplug-docowl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fx-plug%2Fmplug-docowl/lists"}