{"id":13754087,"url":"https://github.com/VITA-MLLM/Woodpecker","last_synced_at":"2025-05-09T22:30:48.213Z","repository":{"id":196606186,"uuid":"696584437","full_name":"BradyFU/Woodpecker","owner":"BradyFU","description":"✨✨Woodpecker: Hallucination Correction for Multimodal Large Language Models. The first work to correct hallucinations in MLLMs.","archived":false,"fork":false,"pushed_at":"2024-06-17T14:34:01.000Z","size":22215,"stargazers_count":611,"open_issues_count":2,"forks_count":29,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-11-16T05:02:08.339Z","etag":null,"topics":["hallucination","hallucinations","large-language-models","llm","mllm","multimodal-large-language-models","multimodality"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BradyFU.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-26T03:29:04.000Z","updated_at":"2024-11-10T07:10:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"b516c9d6-2b19-4e2e-921d-9cd27ab5b77c","html_url":"https://github.com/BradyFU/Woodpecker","commit_stats":null,"previous_names":["bradyfu/hallucination-correction-for-mllms","bradyfu/woodpecker"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BradyFU%2FWoodpecker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BradyFU%2FWoodpecker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BradyFU%2FWoodpecker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BradyFU%2FWoodpecker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BradyFU","download_url":"https://codeload.github.com/BradyFU/Woodpecker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224884613,"owners_count":17386121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hallucination","hallucinations","large-language-models","llm","mllm","multimodal-large-language-models","multimodality"],"created_at":"2024-08-03T09:01:39.325Z","updated_at":"2025-05-09T22:30:48.206Z","avatar_url":"https://github.com/BradyFU.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话","Python"],"sub_categories":["大语言对话模型及数据"],"readme":"# Woodpecker: Hallucination Correction for Multimodal Large Language Models\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./assets/woodpecker.png\" width=\"75%\" height=\"75%\"\u003e\n\u003c/p\u003e\n\n\n\u003cfont size=7\u003e\u003cdiv align='center' \u003e :grapes: \\[[Read our arXiv Paper](https://arxiv.org/pdf/2310.16045.pdf)\\] \u0026nbsp; :apple: \\[[Try our Demo](https://deb6a97bae6fab67ae.gradio.live/)\\] \u003c/div\u003e\u003c/font\u003e\n\n-----------------\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./assets/framework.png\" width=\"96%\" height=\"96%\"\u003e\n\u003c/p\u003e\n\n\u003e Hallucination is a big shadow hanging over the rapidly evolving Multimodal Large Language Models (MLLMs), referring to the phenomenon that the generated text is inconsistent with the image content. In order to mitigate hallucinations, existing studies mainly resort to an instruction-tuning manner that requires retraining the models with specific data. In this paper, we pave a different way, introducing a training-free method named Woodpecker. Like a woodpecker heals trees, it picks out and corrects hallucinations from the generated text. Concretely, Woodpecker consists of five stages: key concept extraction, question formulation, visual knowledge validation, visual claim generation, and hallucination correction. Implemented in a post-remedy manner, Woodpecker can easily serve different MLLMs, while being interpretable by accessing intermediate outputs of the five stages. We evaluate Woodpecker both quantitatively and qualitatively and show the huge potential of this new paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement in accuracy over the baseline MiniGPT-4/mPLUG-Owl.\n\nThis is the first work to correct hallucination in multimodal large language models. If you have any question, please feel free to email bradyfu24@gmail.com or add weChat ID xjtupanda.\n\n\n## 🌋 Evaluation\nWe perform experiments based on four baseline models:\n- [LLaVA](https://github.com/haotian-liu/LLaVA), [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl), [Otter](https://github.com/Luodian/Otter), [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4)\n\nThe experimental results are shown below. For more details, please check out [our paper](https://arxiv.org/pdf/2310.16045.pdf).\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./assets/example.png\" width=\"96%\" height=\"96%\"\u003e\n\u003c/p\u003e\n\n### 📍 POPE Result\n\nThis part focuses on object-level hallucinations.\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./assets/tab1.png\" width=\"70%\"\u003e\n\u003c/p\u003e\n\n### 📍 MME Result\n\nThis part focuses on both object- and attribute-level hallucinations.\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./assets/tab2.png\" width=\"60%\"\u003e\n\u003c/p\u003e\n\n### 📍 LLaVA-QA90 Result\n\nWe also propose to perform open-ended evaluation directly via the recently opened GPT-4V interface. We design two metrics: accuracy and detailedness.\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./assets/tab3.png\" width=\"50%\"\u003e\n\u003c/p\u003e\n\n\n## ▶️ Demo\nPlease feel free to try our [Online Demo](https://deb6a97bae6fab67ae.gradio.live/)!\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./assets/example_demo.png\" width=\"96%\" height=\"96%\"\u003e\n\u003c/p\u003e\n\n## 🛠️ Preliminary\n\n1. Create conda environment\n\n```bash\nconda create -n corrector python=3.10\nconda activate corrector\npip install -r requirements.txt\n```\n\n2. Install required packages and models\n\n- Install `spacy` and relevant model packages, following the instructions in [Link](https://github.com/explosion/spaCy). This is used for some text processing operations.\n\n```bash\npip install -U spacy\npython -m spacy download en_core_web_lg\npython -m spacy download en_core_web_md\npython -m spacy download en_core_web_sm\n```\n- For our **Open-set Detector**. Install GroundingDINO following the instructions in [Link](https://github.com/IDEA-Research/GroundingDINO).\n\n## ⭐ Usage\n\n**1. Inference**\n\nTo make corrections based on an image and a text output from MLLM, run the inference code as follows:\n\n```Shell\npython inference.py \\\n        --image-path {path/to/image} \\\n        --query \"Some query.(e.x. Describe this image.)\" \\\n        --text \"Some text to be corrected.\" \\\n        --detector-config \"path/to/GroundingDINO_SwinT_OGC.py\" \\\n        --detector-model \"path/to/groundingdino_swint_ogc.pth\" \\\n        --api-key \"sk-xxxxxxx\" \\\n\n```\nThe output text will be printed in the terminal, and intermediate results saved by default as ```./intermediate_view.json```.\n\n***\n\n**2. Demo setup**\n\nWe use mPLUG-Owl as our default MLLM in experiments. If you wish to replicate the online demo, please clone the [project](https://github.com/X-PLUG/mPLUG-Owl) and modify the variables in https://github.com/BradyFU/Woodpecker/blob/e3fcac307cc5ff5a3dc079d9a94b924ebcdc2531/gradio_demo.py#L7 and  https://github.com/BradyFU/Woodpecker/blob/e3fcac307cc5ff5a3dc079d9a94b924ebcdc2531/gradio_demo.py#L35-L36\n\nThen simply run:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,1 python gradio_demo.py\n```\nHere we put the corrector components on GPU with id 0 and mPLUG-Owl on GPU with id 1.\n\n\n\n## 🌻 Acknowledgement\nThis repository benefits from [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl), [GroundingDINO](https://github.com/IDEA-Research/GroundingDINO), [BLIP-2](https://huggingface.co/Salesforce/blip2-flan-t5-xxl), and [LLaMA-Adapter](https://github.com/OpenGVLab/LLaMA-Adapter). Thanks for their awesome works.\n\n\n\n## 📑 Citation\nIf you find our project helpful to your research, please consider citing:\n```\n@article{yin2024woodpecker,\n  title={Woodpecker: Hallucination correction for multimodal large language models},\n  author={Yin, Shukang and Fu, Chaoyou and Zhao, Sirui and Xu, Tong and Wang, Hao and Sui, Dianbo and Shen, Yunhang and Li, Ke and Sun, Xing and Chen, Enhong},\n  journal={Science China Information Sciences},\n  volume={67},\n  number={12},\n  pages={220105},\n  year={2024},\n  publisher={Springer}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVITA-MLLM%2FWoodpecker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FVITA-MLLM%2FWoodpecker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVITA-MLLM%2FWoodpecker/lists"}