https://github.com/mrgiovanni/synthetictumors

[CVPR 2023] Label-Free Liver Tumor Segmentation
https://github.com/mrgiovanni/synthetictumors

data-synthesis label-free segmentation tumor-segmentation unet

Last synced: about 1 month ago
JSON representation

[CVPR 2023] Label-Free Liver Tumor Segmentation

• Host: GitHub
• URL: https://github.com/mrgiovanni/synthetictumors
• Owner: MrGiovanni
• License: other
• Created: 2022-10-25T03:38:20.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2025-01-11T07:37:09.000Z (5 months ago)
• Last Synced: 2025-04-09T04:02:59.669Z (2 months ago)
• Topics: data-synthesis, label-free, segmentation, tumor-segmentation, unet
• Language: Python
• Homepage: https://www.cs.jhu.edu/~alanlab/Pubs23/hu2023label.pdf
• Size: 75.2 MB
• Stars: 345
• Watchers: 7
• Forks: 27
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
SynTumor




![visitors](https://visitor-badge.laobi.icu/badge?page_id=MrGiovanni/SyntheticTumors)

[![GitHub Repo stars](https://img.shields.io/github/stars/MrGiovanni/SyntheticTumors?style=social)](https://github.com/MrGiovanni/SyntheticTumors/stargazers)



        




**Subscribe us: https://groups.google.com/u/2/g/bodymaps**  



This repository provides extensive examples of synthetic liver tumors generated by our novel strategies. Check to see if you could tell which is real tumor and which is synthetic tumor. More importantly, our synthetic tumors can be used for training AI models, and have proven to achieve a similar (actually, *better*) performance in real tumor segmentation than a model trained on real tumors. 

**Amazing**, right? 





## Paper

Label-Free Liver Tumor Segmentation 


[Qixin Hu](https://scholar.google.com/citations?user=EqD5GP8AAAAJ&hl=en)1, [Yixiong Chen](https://scholar.google.com/citations?hl=en&user=bVHYVXQAAAAJ)2, [Junfei Xiao](https://lambert-x.github.io/)3, Shuwen Sun4, [Jieneng Chen](https://scholar.google.com/citations?hl=en&user=yLYj88sAAAAJ)3, [Alan L. Yuille](https://www.cs.jhu.edu/~ayuille/)3, and [Zongwei Zhou](https://www.zongweiz.com/)3,* 


1 Huazhong University of Science and Technology,  


2 The Chinese University of Hong Kong -- Shenzhen,  


3 Johns Hopkins University,   


4 The First Affiliated Hospital of Nanjing Medical University 


CVPR, 2023 


[paper](https://arxiv.org/pdf/2303.14869.pdf) | [code](https://github.com/MrGiovanni/SyntheticTumors) | [talk](https://www.youtube.com/watch?v=Alqr3vQSDro) (by Alan Yuille) | [talk](https://youtu.be/DhZzAp7gxxw) | [slides](https://github.com/MrGiovanni/SyntheticTumors/blob/main/documents/slides_cvpr23.pdf) | [poster](https://github.com/MrGiovanni/SyntheticTumors/blob/main/documents/poster_cvpr23.pdf)

Synthetic Tumors Make AI Segment Tumors Better 


[Qixin Hu](https://scholar.google.com/citations?user=EqD5GP8AAAAJ&hl=en)1, [Junfei Xiao](https://lambert-x.github.io/)2, [Yixiong Chen](https://scholar.google.com/citations?hl=en&user=bVHYVXQAAAAJ)3, Shuwen Sun4, [Jieneng Chen](https://scholar.google.com/citations?hl=en&user=yLYj88sAAAAJ)2, [Alan L. Yuille](https://www.cs.jhu.edu/~ayuille/)2, and [Zongwei Zhou](https://www.zongweiz.com/)2,* 


1 Huazhong University of Science and Technology,   


2 Johns Hopkins University, 


3 The Chinese University of Hong Kong -- Shenzhen,   


4 The First Affiliated Hospital of Nanjing Medical University 


Medical Imaging Meets NeurIPS, 2022 


[paper](https://arxiv.org/pdf/2210.14845.pdf) | [code](https://github.com/MrGiovanni/SyntheticTumors) | [slides](https://github.com/MrGiovanni/SyntheticTumors/blob/main/documents/slides.pdf) | [poster](https://github.com/MrGiovanni/SyntheticTumors/blob/main/documents/poster.pdf) | demo | [talk](https://www.youtube.com/watch?v=bJpI9tCTsuA)

Early Detection and Localization of Pancreatic Cancer by Label-Free Tumor Synthesis 


[Bowen Li](https://scholar.google.com/citations?user=UfINwO0AAAAJ&hl=en)1, [Yu-Cheng Chou](https://sites.google.com/view/yu-cheng-chou)1, Shuwen Sun2, Hualin Qiao3, [Alan L. Yuille](https://www.cs.jhu.edu/~ayuille/)1, and [Zongwei Zhou](https://www.zongweiz.com/)1,* 


1 Johns Hopkins University, 


2 The First Affiliated Hospital of Nanjing Medical University,   


3 Sepax technologies 


Big Task Small Data, 1001-AI, MICCAI Workshop, 2023 


[paper](https://arxiv.org/pdf/2308.03008.pdf) | [code](https://github.com/MrGiovanni/SyntheticTumors)

**We have documented common questions for the paper in [Frequently Asked Questions (FAQ)](documents/FAQ.md).**

**We have also provided a list of publications related to tumor synthesis in [Awesome Synthetic Tumors](AWESOME.md) [![Awesome](https://awesome.re/badge.svg)](https://awesome.re).**

## Model

| Tumor | Model | Pre-trained? | Download |

| ----  | ----  | ----         | ----     |

| real  | unet  | no | [link](https://www.dropbox.com/s/8jfu22zgz4a8qjk/model.pt) |

| real  | swin_unetrv2_base | yes | [link](https://www.dropbox.com/s/cmh3uvwtfd3jlvp/model.pt) |

| real  | swin_unetrv2_base | no  | [link](https://www.dropbox.com/s/vf3se1yns18t7qm/model.pt) |

| real  | swin_unetrv2_small | no  | [link](https://www.dropbox.com/s/337uz6484zyzjty/model.pt) |

| real  | swin_unetrv2_tiny | no  | [link](https://www.dropbox.com/s/leh04qh5hvfq9i5/model.pt) |

| synt  | unet  | no | [link](https://www.dropbox.com/s/gb01obtxlbktmrp/model.pt) |

| synt  | swin_unetrv2_base | yes | [link](https://www.dropbox.com/s/pxvb6qfmeaha2va/model.pt) |

| synt  | swin_unetrv2_base | no  | [link](https://www.dropbox.com/s/idxwss85bmx3ejo/model.pt) |

| synt  | swin_unetrv2_small | no  | [link](https://www.dropbox.com/s/nkb64yo4jmscoy6/model.pt) |

| synt  | swin_unetrv2_tiny | no  | [link](https://www.dropbox.com/s/ddej5duj03ioh49/model.pt) |

**Use the following command to download everything.**

```bash

wget https://www.dropbox.com/s/jys4tt2ttmr7ig1/runs.tar.gz

tar -xzvf runs.tar.gz

```

   

## 0. Installation

```bash

git clone https://github.com/MrGiovanni/SyntheticTumors.git

cd SyntheticTumors

wget https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/model_swinvit.pt

```

See [installation instructions](https://github.com/MrGiovanni/SyntheticTumors/blob/main/INSTALL.md).

## 1. Train segmentation models using synthetic tumors

```

datapath=/mnt/zzhou82/PublicAbdominalData/

# UNET (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=unet --val_every=200 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12235 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.no_pretrain.unet" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json

# Swin-UNETR-Base (pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=base --val_every=200 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12231 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.pretrain.swin_unetrv2_base" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json --use_pretrained

# Swin-UNETR-Base (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=base --val_every=200 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12231 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.no_pretrain.swin_unetrv2_base" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json

# Swin-UNETR-Small (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=small --val_every=200 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12233 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.no_pretrain.swin_unetrv2_small" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json

# Swin-UNETR-Tiny (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=tiny --val_every=200 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12234 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.no_pretrain.swin_unetrv2_tiny" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json

```

## 2. Train segmentation models using real tumors (for comparison)

```

datapath=/mnt/zzhou82/PublicAbdominalData/

# UNET (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=unet --val_every=200 --val_overlap=0.5 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12235 --cache_num=200 --logdir="runs/real.no_pretrain.unet" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json

# Swin-UNETR-Base (pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=base --val_every=200 --val_overlap=0.5 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12231 --cache_num=200 --logdir="runs/real.pretrain.swin_unetrv2_base" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json --use_pretrained

# Swin-UNETR-Base (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=base --val_every=200 --val_overlap=0.5 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12232 --cache_num=200 --logdir="runs/real.no_pretrain.swin_unetrv2_base" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json

# Swin-UNETR-Small (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=small --val_every=200 --val_overlap=0.5 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12233 --cache_num=200 --logdir="runs/real.no_pretrain.swin_unetrv2_small" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json

# Swin-UNETR-Tiny (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=tiny --val_every=200 --val_overlap=0.5 --max_epochs=4000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12234 --cache_num=200 --logdir="runs/real.no_pretrain.swin_unetrv2_tiny" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json

```

## 3. Evaluation

#### AI model trained by synthetic tumors

```

datapath=/mnt/zzhou82/PublicAbdominalData/

# UNET (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=unet --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.no_pretrain.unet --save_dir out

# Swin-UNETR-Base (pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=base --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.pretrain.swin_unetrv2_base --save_dir out

# Swin-UNETR-Base (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=base --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.no_pretrain.swin_unetrv2_base --save_dir out

# Swin-UNETR-Small (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=small --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.no_pretrain.swin_unetrv2_small --save_dir out

# Swin-UNETR-Tiny (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=tiny --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.no_pretrain.swin_unetrv2_tiny --save_dir out

```

#### AI model trained by real tumors

```

datapath=/mnt/zzhou82/PublicAbdominalData/

# UNET (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=unet --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.no_pretrain.unet --save_dir out

# Swin-UNETR-Base (pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=base --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.pretrain.swin_unetrv2_base --save_dir out

# Swin-UNETR-Base (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=base --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.no_pretrain.swin_unetrv2_base --save_dir out

# Swin-UNETR-Small (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=small --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.no_pretrain.swin_unetrv2_small --save_dir out

# Swin-UNETR-Tiny (no.pretrain)

CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=tiny --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.no_pretrain.swin_unetrv2_tiny --save_dir out

```

## TODO

- [x] Upload the paper to arxiv

- [ ] Make a video about Visual Turing Test (will appear in YouTube)

- [ ] Make an online app for Visual Turing Test

- [x] Apply for a US patent

- [ ] Upload the evaluation code for small tumors

- [ ] Upload the evaluation code for the false-positive study

- [ ] Make a Jupyter Notebook for tumor synthesis

## Citation

```

@inproceedings{hu2023label,

  title={Label-free liver tumor segmentation},

  author={Hu, Qixin and Chen, Yixiong and Xiao, Junfei and Sun, Shuwen and Chen, Jieneng and Yuille, Alan L and Zhou, Zongwei},

  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},

  pages={7422--7432},

  year={2023}

}

@article{hu2022synthetic,

  title={Synthetic Tumors Make AI Segment Tumors Better},

  author={Hu, Qixin and Xiao, Junfei and Chen, Yixiong and Sun, Shuwen and Chen, Jie-Neng and Yuille, Alan and Zhou, Zongwei},

  journal={NeurIPS Workshop on Medical Imaging meets NeurIPS},

  year={2022}

}

@article{li2023early,

  title={Early Detection and Localization of Pancreatic Cancer by Label-Free Tumor Synthesis},

  author={Li, Bowen and Chou, Yu-Cheng and Sun, Shuwen and Qiao, Hualin and Yuille, Alan and Zhou, Zongwei},

  journal={arXiv preprint arXiv:2308.03008},

  year={2023}

}

```

## Acknowledgement

This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the McGovern Foundation. The segmentation backbone is based on [Swin UNETR](https://github.com/Project-MONAI/tutorials/blob/main/3d_segmentation/swin_unetr_btcv_segmentation_3d.ipynb); we appreciate the effort of the [MONAI Team](https://monai.io/) to provide and maintain open-source code to the community. We thank Camille Torrico and Alexa Delaney for improving the writing of this paper. Paper content is covered by patents pending.