https://github.com/openmedlab/XrayPULSE
https://github.com/openmedlab/XrayPULSE
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/openmedlab/XrayPULSE
- Owner: openmedlab
- Created: 2023-06-25T07:04:45.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2023-08-23T07:05:02.000Z (almost 3 years ago)
- Last Synced: 2024-11-16T06:31:50.737Z (over 1 year ago)
- Language: Python
- Size: 1.39 MB
- Stars: 168
- Watchers: 3
- Forks: 7
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- StarryDivineSky - openmedlab/XrayPULSE - former（BLIP2）作为适配器，通过简单的线性变换将图像注入PULSE。为了通过适配器对齐冷冻视觉编码器和LLM，我们借助chatGPT从两个数据集（MIMIC-CXR和OpenI）的自由文本放射学报告中生成中文版Xray-Report配对数据。为了促进生物医学多模态学习的研究，我们将向公众发布数据。 (A01_文本生成_文本对话 / 大语言对话模型及数据)
- Awesome-LLM-Healthcare - 2023/06
- awesome-medical-ai-cn - XrayPULSE - square) | OpenMEDLab 胸部 X 光多模态助手，将 PULSE 与 MedCLIP / Q-Former 结合，开源中文化报告数据构建思路、推理 Demo 和 `xraypulse/` 代码。 | (医学影像 AI)
README
# XrayPULSE
---
## Key Features
This repository provides the official implementation of XrayPULSE:
Key feature bulletin points here
- An attempt to extend [PULSE]() to a biomedical multimodal conversational assistant.
- XrayPULSE is fintuned on Xray-Report paired datasets in Chinese
## Details
Our model is based on PULSE. We utilize [MedCLIP](https://github.com/RyanWangZf/MedCLIP) as our medical visual encoder and Q-former ([BLIP2](https://huggingface.co/docs/transformers/main/model_doc/blip-2)) following a simple linear transformation as the adapter to inject the image to PULSE. For aligning the frozen visual encoder and the LLM by the adapter, we generate Chinese-version Xray-Report paired data from free-text radiology reports of two datasets ([MIMIC-CXR](https://physionet.org/content/mimic-cxr-jpg/2.0.0/) and [OpenI](https://openi.nlm.nih.gov/faq#collection)) with the help of chatGPT. To facilitate research in biomedical multimodal learning, we will release the data to the public.
## Get Started
**Installation**
```bash
git clone https://github.com/openmedlab/XrayPULSE.git
cd XrayPULSE
```
**Environment**
```bash
conda env create -f env.yml
conda activate xraypulse
```
**Prepare the pretrained weights**
You can find the pretrained model weights.
- [PULSE\_Model](https://huggingface.co/OpenMEDLab/PULSE-7bv5)
- [Pretrained_XrayPULSE_Checkpoint](https://drive.google.com/file/d/1VsO61-3DFuK4ysGPvoD4_JZaRFKvAJR_/view?usp=drive_link)
The weights of PULSE would be in a single folder in a structure similar to the following:
```
pulse_weights
├── config.json
├── generation_config.json
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── pytorch_model.bin.index.json
├── pytorch_model-00001-of-00002.bin
├── pytorch_model-00002-of-00002.bin
```
Then, set the path of pulse_weights to "bloom_model" in the model config file "xraypulse/configs/models/xraypulse.yaml"
And add the path of the pretrained checkpoint in "demo_configs/xraypulse_demo.yaml".
**Run Demo**
```bash
bash run_demo.sh
```
## 🙏 Acknowledgement
This project is built upon the gaint sholders of [XrayGPT](https://github.com/mbzuai-oryx/XrayGPT). Great thanks to it!
We used medical aware image encoder from [MedCLIP](https://github.com/RyanWangZf/MedCLIP).
The model architecture of XrayGPT follows [BLIP2](https://huggingface.co/docs/transformers/main/model_doc/blip-2).
## 🛡️ License
This project is under the CC-BY-NC 4.0 license. See [LICENSE](LICENSE) for details.