{"id":13754167,"url":"https://github.com/openmedlab/XrayPULSE","last_synced_at":"2025-05-09T22:31:21.818Z","repository":{"id":176772558,"uuid":"658232820","full_name":"openmedlab/XrayPULSE","owner":"openmedlab","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-23T07:05:02.000Z","size":1459,"stargazers_count":168,"open_issues_count":3,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-16T06:31:50.737Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openmedlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-06-25T07:04:45.000Z","updated_at":"2024-11-15T08:44:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"a1400d3d-1ebc-4d89-a817-71aa50964344","html_url":"https://github.com/openmedlab/XrayPULSE","commit_stats":null,"previous_names":["openmedlab/xraypulse"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmedlab%2FXrayPULSE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmedlab%2FXrayPULSE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmedlab%2FXrayPULSE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmedlab%2FXrayPULSE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openmedlab","download_url":"https://codeload.github.com/openmedlab/XrayPULSE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253335604,"owners_count":21892703,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:01:46.442Z","updated_at":"2025-05-09T22:31:21.769Z","avatar_url":"https://github.com/openmedlab.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话","Multimodal LLMs in Medicine","医学影像 AI"],"sub_categories":["大语言对话模型及数据"],"readme":"# XrayPULSE\n\n\u003c!--\n**Here are some ideas to get you started:**\n🙋‍♀️ A short introduction - what is your organization all about?\n🌈 Contribution guidelines - how can the community get involved?\n👩‍💻 Useful resources - where can the community find your docs? Is there anything else the community should know?\n🍿 Fun facts - what does your team eat for breakfast?\n🧙 Remember, you can do mighty things with the power of [Markdown](https://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax)\n--\u003e\n\n\u003c!-- Insert the project banner here --\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://\"\u003e\u003cimg width=\"1000px\" height=\"auto\" src=\"./banner.png\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\n---\n\n\u003c!-- Select some of the point info, feel free to delete --\u003e\n\n\n## Key Features\n\nThis repository provides the official implementation of XrayPULSE: \n\nKey feature bulletin points here\n\n- An attempt to extend [PULSE]() to a biomedical multimodal conversational assistant. \n- XrayPULSE is fintuned on Xray-Report paired datasets in Chinese\n\n\n## Details\n\nOur model is based on PULSE. We utilize [MedCLIP](https://github.com/RyanWangZf/MedCLIP)  as our medical visual encoder and Q-former ([BLIP2](https://huggingface.co/docs/transformers/main/model_doc/blip-2)) following a simple linear transformation as the adapter to inject the image to PULSE. For aligning the frozen visual encoder and the LLM by the adapter, we generate Chinese-version Xray-Report paired data from free-text radiology reports of two datasets ([MIMIC-CXR](https://physionet.org/content/mimic-cxr-jpg/2.0.0/) and [OpenI](https://openi.nlm.nih.gov/faq#collection)) with the help of chatGPT.  To facilitate research in biomedical multimodal learning, we will release the data to the public.\n\n\u003c!-- Insert a pipeline of your algorithm here if got one --\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://\"\u003e\u003cimg width=\"1000px\" height=\"auto\" src=\"./framework.png\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\n\n## Get Started\n\n**Installation**\n\n```bash\ngit clone https://github.com/openmedlab/XrayPULSE.git\ncd XrayPULSE\n```\n\n**Environment**\n\n```bash\nconda env create -f env.yml\nconda activate xraypulse\n```\n\n**Prepare the pretrained weights**\n\nYou can find the pretrained model weights.\n\n- [PULSE\\_Model](https://huggingface.co/OpenMEDLab/PULSE-7bv5) \n- [Pretrained_XrayPULSE_Checkpoint](https://drive.google.com/file/d/1VsO61-3DFuK4ysGPvoD4_JZaRFKvAJR_/view?usp=drive_link)\n\nThe weights of PULSE would be in a single folder in a structure similar to the following:\n\n```\npulse_weights\n├── config.json\n├── generation_config.json\n├── tokenizer.json\n├── tokenizer_config.json\n├── special_tokens_map.json \n├── pytorch_model.bin.index.json\n├── pytorch_model-00001-of-00002.bin\n├── pytorch_model-00002-of-00002.bin \n```\n\nThen, set the path of pulse_weights to \"bloom_model\" in the model config file \"xraypulse/configs/models/xraypulse.yaml\"\n\nAnd add the path of the pretrained checkpoint  in \"demo_configs/xraypulse_demo.yaml\".\n\n**Run Demo**\n\n```bash\nbash run_demo.sh\n```\n\n\n\n## 🙏 Acknowledgement\nThis project is built upon the gaint sholders of [XrayGPT](https://github.com/mbzuai-oryx/XrayGPT). Great thanks to it!\n\nWe used medical aware image encoder from [MedCLIP](https://github.com/RyanWangZf/MedCLIP).\n\nThe model architecture of XrayGPT follows [BLIP2](https://huggingface.co/docs/transformers/main/model_doc/blip-2).\n\n\n## 🛡️ License\n\nThis project is under the CC-BY-NC 4.0 license. See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenmedlab%2FXrayPULSE","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenmedlab%2FXrayPULSE","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenmedlab%2FXrayPULSE/lists"}