{"id":20779423,"url":"https://github.com/freedomintelligence/huatuogpt-vision","last_synced_at":"2025-04-12T21:31:40.121Z","repository":{"id":246406574,"uuid":"819886293","full_name":"FreedomIntelligence/HuatuoGPT-Vision","owner":"FreedomIntelligence","description":"Medical Multimodal LLMs","archived":false,"fork":false,"pushed_at":"2025-01-09T11:07:44.000Z","size":63,"stargazers_count":265,"open_issues_count":16,"forks_count":20,"subscribers_count":23,"default_branch":"main","last_synced_at":"2025-04-04T03:05:41.661Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-25T11:24:32.000Z","updated_at":"2025-03-30T02:39:37.000Z","dependencies_parsed_at":"2024-06-27T20:57:55.388Z","dependency_job_id":"949ec019-d3ca-4df5-8085-5150c14c17a2","html_url":"https://github.com/FreedomIntelligence/HuatuoGPT-Vision","commit_stats":null,"previous_names":["freedomintelligence/huatuogpt-vision"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FHuatuoGPT-Vision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FHuatuoGPT-Vision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FHuatuoGPT-Vision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FHuatuoGPT-Vision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/HuatuoGPT-Vision/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248635074,"owners_count":21137169,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:27:52.262Z","updated_at":"2025-04-12T21:31:40.075Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale\n\n\n\u003cdiv align=\"center\"\u003e\n\u003ch5\u003e\n  📃 \u003ca href=\"https://arxiv.org/abs/2406.19280\" target=\"_blank\"\u003ePaper\u003c/a\u003e  • 🖥️ \u003ca href=\"https://vision.huatuogpt.cn/#/\" target=\"_blank\"\u003eDemo\u003c/a\u003e\n\u003c/h5\u003e\n\u003c/div\u003e\n \n\u003cdiv align=\"center\"\u003e\n\u003ch4\u003e\n  📚 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/PubMedVision\" target=\"_blank\"\u003ePubMedVision\u003c/a\u003e \n\u003c/h4\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003ch4\u003e\n  🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/HuatuoGPT-Vision-34B\" target=\"_blank\"\u003eHuatuoGPT-Vision-34B\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/HuatuoGPT-Vision-7B\"\u003eHuatuoGPT-Vision-7B\u003c/a\u003e \n\u003c/h4\u003e\n\u003c/div\u003e\n\n## ✨ Updates\n- [01/09/2025]: We released the evaluation code (including evaluation data). Additionally, we added instructions on how to train your medical multimodal LLM.  \n- [06/28/2024]: We released our medical MLLMs, including [HuatuoGPT-Vision-34B](https://huggingface.co/FreedomIntelligence/HuatuoGPT-Vision-34B) and [HuatuoGPT-Vision-7B](https://huggingface.co/FreedomIntelligence/HuatuoGPT-Vision-7B).\n- [06/26/2024]: We released [PubMedVision](https://huggingface.co/datasets/FreedomIntelligence/PubMedVision), a **1.3M** high-quality medical VQA dataset for injecting medical visual knowledge.\n\n## 🩻 PubMedVision\n- **PubMedVision** is a large-scale, high-quality medical VQA dataset, constructed from the image-text pairs from PubMed and reformatted using GPT-4V.\n\n|                             | # Data     | Download |\n| ------------------ | ---------- | ------------- | \n| **PubMedVision Dataset**   | **1,294,062** | [HF Link](https://huggingface.co/datasets/FreedomIntelligence/PubMedVision) |\n\n- **PubMedVision** could significantly improve the medical multimodal capabilities of MLLMs such as LLaVA-v1.5. \n\n|                                         | **VQA-RAD** | **SLAKE** | **PathVQA** | **PMC-VQA** |\n| --------------------------------------- | ----------- | --------- | ----------- | ----------- |\n| LLaVA-v1.6-34B                          | 58.6        | 67.3      | 59.1        | 44.4        |\n| LLaVA-v1.5-LLaMA3-8B                    | 54.2        | 59.4      | 54.1        | 36.4        |\n| LLaVA-v1.5-LLaMA3-8B + **PubMedVision** | **63.8**    | **74.5**  | **59.9**    | **52.7**    |\n\n|                           | **OmniMedVQA**  | **MMMU Health \u0026 Medicine (Test Set)** |\n| ----------------------------------- | ------------ | -------------------------- |\n| LLaVA-v1.6-34B                      | 61.4         | 48.8                       |\n| LLaVA-v1.5-LLaMA3-8B                | 48.8         | 38.2                       |\n| LLaVA-v1.5-LLaMA3-8B + **PubMedVision** | **75.1**    | **49.1**                   |\n\n## 👨‍⚕️ HuatuoGPT-Vision\nHuatuoGPT-Vision is our medical multimodal LLMs, built on **PubMedVision**.\n\n### Model Access\nOur model is available on Huggingface in two versions:\n|                 | Backbone           | Checkpoint                                                                            |\n|----------------------|--------------------|---------------------------------------------------------------------------------------|\n| **HuatuoGPT-Vision-7B**  | Qwen2-7B           | [HF Link](https://huggingface.co/FreedomIntelligence/HuatuoGPT-Vision-7B)             |\n| **HuatuoGPT-Vision-34B** | Yi-1.5-34B         | [HF Link](https://huggingface.co/FreedomIntelligence/HuatuoGPT-Vision-34B)            |\n\n### Model Usage\n\n- **Command Line Interface**\n\nChat via the command line:\n```bash\npython cli.py --model_dir path-to-huatuogpt-vision-model\n```\n\n- **Model Inference**\n\nInference using our ChatBot:\n```python\nquery = 'What does the picture show?'\nimage_paths = ['image_path1']\n\nfrom cli import HuatuoChatbot\nbot = HuatuoChatbot(path-to-huatuogpt-vision-model)\noutput = bot.inference(query, image_paths)\nprint(output) # Prints the output of the model\n```\n\n### Performance of Medical Multimodal\n|                             | **VQA-RAD** | **SLAKE** | **PathVQA** | **PMC-VQA** |\n|-----------------------------|-------------|-----------|-------------|-------------|\n| LLaVA-Med-7B                   | 51.4        | 48.6      | 56.8        | 24.7        |\n| LLaVA-v1.6-34B              | 58.6        | 67.3      | 59.1        | 44.4        |\n| **HuatuoGPT-Vision-7B**         | 63.7        | 76.2     | 57.9        | 54.3        |\n| **HuatuoGPT-Vision-34B**        | **68.1**    | **76.9**  | **63.5**    | **58.2**    |\n\n|                           | **OmniMedVQA**  | **MMMU Health \u0026 Medicine (Test Set)** |\n|---------------------------|-----------------|---------------------------------------|\n| LLaVA-Med-7B                 | 44.5            | 36.9                                  |\n| LLaVA-v1.6-34B            | 61.4            | 48.8                                  |\n| **HuatuoGPT-Vision-7B**        | 74.0            | 50.6                                  |\n| **HuatuoGPT-Vision-34B**      | **76.9**        | **54.4**                               |\n\n## 📏 Evaluation\n1. For evaluation, you need to download the medical evaluation dataset. We have organized multiple evaluation datasets, which can be downloaded directly via the following link:\n\n| Dataset | Link |\n| --- | --- |\n| Medical_Multimodal_Evaluation_Data | [link](https://huggingface.co/datasets/FreedomIntelligence/Medical_Multimodal_Evaluation_Data) |  \n\nWe have bundled multiple evaluation datasets together. Simply download the data and extract the `images.zip` file.  \n\n2. Then, you can evaluate using the following command:\n```bash\naccelerate launch eval.py --data_path Medical_Multimodal_Evaluation_Data/medical_multimodel_evaluation_data.json  --model_path HuatuoGPT-Vision-7B\n```  \n\n3. Once executed, you will directly obtain the results of the medical multimodal evaluation, including datasets like `VQA-RAD`, `SLAKE`, `PathVQA`, `PMC-VQA`, `OmniMedVQA`, and `MMMU-Medical-Tracks`.\n\n\n## 🏋 Training\nThis project uses LLaVA's code for training, and it is recommended to use LLaVA's code for training. The code is available at [LLaVA](https://github.com/haotian-liu/LLaVA).  \n\u003e To reproduce our results, please train the model using a combination of the PubMedVision dataset and LLaVA's dataset.\n\n## 🩺 HuatuoGPT Series \n\nExplore our HuatuoGPT series:\n- [**HuatuoGPT**](https://github.com/FreedomIntelligence/HuatuoGPT): Taming Language Models to Be a Doctor\n- [**HuatuoGPT-II**](https://github.com/FreedomIntelligence/HuatuoGPT-II): One-stage Training for Medical Adaptation of LLMs\n- [**HuatuoGPT-Vision**](https://github.com/FreedomIntelligence/HuatuoGPT-Vision): Injecting Medical Visual Knowledge into Multimodal LLMs at Scale\n\n\n## Citation\n```\n@misc{chen2024huatuogptvisioninjectingmedicalvisual,\n      title={HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale}, \n      author={Junying Chen and Ruyi Ouyang and Anningzhe Gao and Shunian Chen and Guiming Hardy Chen and Xidong Wang and Ruifei Zhang and Zhenyang Cai and Ke Ji and Guangjun Yu and Xiang Wan and Benyou Wang},\n      year={2024},\n      eprint={2406.19280},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2406.19280}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fhuatuogpt-vision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fhuatuogpt-vision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fhuatuogpt-vision/lists"}