{"id":13560152,"url":"https://github.com/xiaoman-zhang/PMC-VQA","last_synced_at":"2025-04-03T15:31:41.961Z","repository":{"id":169231278,"uuid":"641227214","full_name":"xiaoman-zhang/PMC-VQA","owner":"xiaoman-zhang","description":"PMC-VQA is a large-scale medical visual question-answering dataset, which contains 227k VQA pairs of 149k images that cover various modalities or diseases. ","archived":false,"fork":false,"pushed_at":"2024-12-06T02:24:55.000Z","size":297,"stargazers_count":177,"open_issues_count":13,"forks_count":11,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-06T04:28:04.771Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xiaoman-zhang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-16T03:31:15.000Z","updated_at":"2024-12-05T08:41:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"c18a44dd-f488-4b00-b1fe-b7ae2b52406b","html_url":"https://github.com/xiaoman-zhang/PMC-VQA","commit_stats":null,"previous_names":["xiaoman-zhang/pmc-vqa"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaoman-zhang%2FPMC-VQA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaoman-zhang%2FPMC-VQA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaoman-zhang%2FPMC-VQA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaoman-zhang%2FPMC-VQA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xiaoman-zhang","download_url":"https://codeload.github.com/xiaoman-zhang/PMC-VQA/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247027990,"owners_count":20871632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:00:38.399Z","updated_at":"2025-04-03T15:31:36.947Z","avatar_url":"https://github.com/xiaoman-zhang.png","language":"Python","funding_links":[],"categories":["Data Source ![](https://img.shields.io/badge/Data_Source-yellow)","Medical Imaging \u0026 Radiology"],"sub_categories":["Visual Question Answering Datasets"],"readme":"# PMC-VQA\nThe official codes for [**PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering**](https://arxiv.org/pdf/2305.10415.pdf)  \n\n\t\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pmc-vqa-visual-instruction-tuning-for-medical/medical-visual-question-answering-on-pmc-vqa)](https://paperswithcode.com/sota/medical-visual-question-answering-on-pmc-vqa?p=pmc-vqa-visual-instruction-tuning-for-medical)\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pmc-vqa-visual-instruction-tuning-for-medical/medical-visual-question-answering-on-vqa-rad)](https://paperswithcode.com/sota/medical-visual-question-answering-on-vqa-rad?p=pmc-vqa-visual-instruction-tuning-for-medical)\n\n\nWe propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model, and establish a scalable pipeline to construct a large-scale medical visual question-answering dataset, named PMC-VQA, which contains 227k VQA pairs of 149k images that cover various modalities or diseases.\n\nThe dataset is available at [Huggingface](https://huggingface.co/datasets/xmcmic/PMC-VQA/)\n\nThe model checkpoints are available at [MedVInT-TE](https://huggingface.co/xmcmic/MedVInT-TE/) and [MedVInT-TD](https://huggingface.co/xmcmic/MedVInT-TD/).\n**The previous checkpoint of MedVInT-TD was mistakenly uploaded. \nWe have rectified the issue and updated the model's checkpoint on July 31. \nNow, you can access the correct and improved version of the model.**\n\n\n- [PMC-VQA](#pmc-vqa)\n  - [Usage](#usage)\n    - [1. Create Environment](#1-create-environment)\n    - [2. Prepare Dataset](#2-prepare-dataset)\n    - [3. Model Checkpoints](#3-checkpoints)\n  - [Acknowledgement](#acknowledgement)\n  - [Contribution](#contribution)\n  - [Cite](#cite)\n\n\n## Usage\n\n\u003c!-- Repo Structure\n```bash\nPMC-VQA/: dataset\nLLAMA_Model/: LLaMA pre-trained model path\nsrc/:\n    |--MedVInT_TD/: The code of the model of MedVInT-TD-MLP and MedVInT-TD-Transformer with LLaMA/PMC-LLaMA and CLIP/PMC-CLIP\n    |   |--models/\n    |   |   |--blocks.py \n    |   |   |--QA_model_mlp.py: model MedVInT-TD-MLP\n    |   |   |--QA_model.py:  model MedVInT-TD-Transformer\n    |   |   |--transformer.py \n    |   |--Results/: The checkpoints of the MedVInT_TD with PMC-LLaMA, PMC-CLIP on both blank and choice tasks\n    |   |   |--blank_training/\n    |   |   |--choice_training/\n    |--MedVInT_TE/: The code of MedVInT-TE-Transformer with LLaMA/PMC-LLaMA and CLIP/PMC-CLIP\n    |   |--dataset/\n    |   |   |--dataset.py: Create dataset\n    |   |   |--randaugment.py: data augmentation\n    |   |--ds_config/\n    |   |   |--ds_config_zero2.json: deep speed\n    |   |--models/\n    |   |   |--llama/\n    |   |   |   |--blocks.py \n    |   |   |   |--vqa_model.py \n    |   |   |--pmcclip/: put the checkpoint of pmc_clip here\n    |   |   |--pmc_oa/\n    |   |   |   |--blocks.py \n    |   |   |   |--pmc_clip.py\n    |   |   |   |--timm_model.py\n    |   |   |   |--utils.py\n    |   |--Results/: put the pre-trained checkpoint here\n    |   |--README.md: README for MedVInT_TE\n    |   |--train.py\n    |   |--test.py\n    |   |--train.sh\n``` --\u003e\n\n\n### 1. Create Environment \n\nPlease refer to https://github.com/chaoyi-wu/PMC-LLaMA\n\n### 2. Prepare Dataset \n\nDownload from [Huggingface](https://huggingface.co/datasets/xmcmic/PMC-VQA/) and save into ./PMC-VQA\n\n### 3. Model Checkpoints\n\nDownload the pre-trained [MedVInT-TE](https://huggingface.co/xmcmic/MedVInT-TE/), and save into `./src/MedVInT_TE/Results `directly.  \n\nDownload the pre-trained [MedVInT-TD](https://huggingface.co/xmcmic/MedVInT-TD/), and save into `./src/MedVInT_TD/Results `directly.  \n\nSee [MedVInT_TE](./src/MedVInT_TE/README.md) and [MedVInT_TD](./src/MedVInT_TD/README.md)  for the details of training **MedVInT_TE** and **MedVInT_TD**. \n\n## Acknowledgement\n\nCLIP -- https://github.com/openai/CLIP\n\nPMC-CLIP -- https://github.com/WeixiongLin/PMC-CLIP\n\nPMC-LLaMA -- [https://github.com/zphang/minimal-llama](https://github.com/chaoyi-wu/PMC-LLaMA)\n\nLLaMA: Open and Efficient Foundation Language Models -- https://arxiv.org/abs/2302.13971\n\nWe thank the authors for their open-sourced code and encourage users to cite their works when applicable.\n\n## Contribution\n\nPlease raise an issue if you need help, any contributions are welcomed.\n\n## Citation\n\nIf you use this code or use our pre-trained weights for your research, please cite our [paper](https://arxiv.org/abs/2305.10415)\n\n```\n@article{zhang2023pmcvqa,\n      title={PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering}, \n      author={Xiaoman Zhang and Chaoyi Wu and Ziheng Zhao and Weixiong Lin and Ya Zhang and Yanfeng Wang and Weidi Xie},\n      year={2023},\n      journal={arXiv preprint arXiv:2305.10415},\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiaoman-zhang%2FPMC-VQA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxiaoman-zhang%2FPMC-VQA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiaoman-zhang%2FPMC-VQA/lists"}