https://github.com/mrgiovanni/touchstone
[NeurIPS 2024] Touchstone - Benchmarking AI on 5,172 o.o.d. CT volumes and 9 anatomical structures
https://github.com/mrgiovanni/touchstone
benchmark medical-image-analysis segmentation
Last synced: 6 months ago
JSON representation
[NeurIPS 2024] Touchstone - Benchmarking AI on 5,172 o.o.d. CT volumes and 9 anatomical structures
- Host: GitHub
- URL: https://github.com/mrgiovanni/touchstone
- Owner: MrGiovanni
- License: other
- Created: 2024-06-05T03:35:18.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-17T18:30:27.000Z (8 months ago)
- Last Synced: 2025-03-30T02:07:54.288Z (7 months ago)
- Topics: benchmark, medical-image-analysis, segmentation
- Language: Jupyter Notebook
- Homepage: https://www.cs.jhu.edu/~zongwei/publication/bassi2024touchstone.pdf
- Size: 34.2 MB
- Stars: 113
- Watchers: 6
- Forks: 4
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Touchstone Benchmark
[](https://github.com/MrGiovanni/Touchstone/tree/main?tab=readme-ov-file#touchstone-10-leaderboard)
[](https://github.com/MrGiovanni/Touchstone/tree/main?tab=readme-ov-file#touchstone-10-dataset)
[](https://github.com/MrGiovanni/Touchstone/tree/main?tab=readme-ov-file#touchstone-10-model)
[](https://github.com/MrGiovanni/SuPreM/stargazers)
**Subscribe us: https://groups.google.com/u/2/g/bodymaps**
We present Touchstone, a large-scale medical segmentation benchmark based on annotated **5,195** CT volumes from **76** hospitals for training, and **6,933** CT volumes from **8** additional hospitals for testing. We invite AI inventors to train their models on AbdomenAtlas, and we independently evaluate their algorithms. We have already collaborated with **14** influential research teams, and we remain accepting new submissions.
# Paper
Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?
[Pedro R. A. S. Bassi](https://scholar.google.com.hk/citations?user=NftgL6gAAAAJ)1, [Wenxuan Li](https://scholar.google.com/citations?hl=en&user=tpNZM2YAAAAJ)1, [Yucheng Tang](https://scholar.google.com.hk/citations?hl=en&user=0xheliUAAAAJ)2, [Fabian Isensee](https://scholar.google.com.hk/citations?hl=en&user=PjerEe4AAAAJ)3, ..., [Alan Yuille](https://www.cs.jhu.edu/~ayuille/)1, [Zongwei Zhou](https://www.zongweiz.com/)1
1Johns Hopkins University, 2NVIDIA, 3DKFZ
NeurIPS 2024
[JHU CS News](https://www.cs.jhu.edu/news/a-touchstone-of-medical-artificial-intelligence/)
[](https://youtu.be/tPnTCFUTjjY)
# Touchstone 1.0 Leaderboard
| rank | model | organization | average DSC | paper | github
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | MedNeXt | DKFZ | 89.2 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 🏆 | MedFormer | Rutgers | 89.0 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 3 | STU-Net-B | Shanghai AI Lab | 89.0 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 4 | nnU-Net U-Net | DKFZ | 88.9 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 5 | nnU-Net ResEncL | DKFZ | 88.8 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 6 | UniSeg | NPU | 88.8 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 7 | Diff-UNet | HKUST | 88.5 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 8 | LHU-Net | UR | 88.0 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 9 | NexToU | HIT | 87.8 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 10 | SegVol | BAAI | 87.1 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 11 | U-Net & CLIP | CityU | 87.1 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 12 | Swin UNETR & CLIP | CityU | 86.7 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 13 | UNesT | NVIDIA | 84.9 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 14 | Swin UNETR | NVIDIA | 84.8 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | UNETR | NVIDIA| 83.3 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 16 | UCTransNet | Northeastern University | 81.1 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
| 17 | SAM-Adapter | Duke | 73.4 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
Aorta - NexToU & UCTransNet 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | UCTransNet | Northeastern University | 86.5 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
| 🏆 | NexToU | HIT | 86.4 | [](https://arxiv.org/pdf/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 3 | MedNeXt | DKFZ | 83.1 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 4 | nnU-Net U-Net | DKFZ | 82.8 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 5 | UniSeg | NPU | 82.3 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 6 | MedFormer | Rutgers | 82.1 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 7 | STU-Net-B | Shanghai AI Lab | 82.1 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 8 | nnU-Net ResEncL | DKFZ | 81.4 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 9 | Diff-UNet | HKUST | 81.2 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 10 | SegVol | BAAI | 80.2 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 11 | LHU-Net | UR | 79.5 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 12 | Swin UNETR & CLIP | CityU | 78.1 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 13 | UNesT | NVIDIA | 77.7 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 14| Swin UNETR | NVIDIA | 77.2 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | U-Net & CLIP | CityU | 77.1 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 16 | UNETR | NVIDIA | 76.5 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 17 | SAM-Adapter | Duke | 62.8 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
Gallbladder - STU-Net-B & MedFormer 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | STU-Net-B | Shanghai AI Lab | 85.5 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 🏆 | MedFormer | Rutgers | 85.3 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 3 | MedNeXt | DKFZ | 85.3 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 4 | nnU-Net ResEncL | DKFZ | 84.9 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 5 | nnU-Net U-Net | DKFZ | 84.7 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 6 | UniSeg | NPU | 84.7 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 7 | LHU-Net | UR | 83.9 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 8 | Diff-UNet | HKUST | 83.8 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 9 | NexToU | HIT | 82.3 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 10 | U-Net & CLIP | CityU | 82.1 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 11 | Swin UNETR & CLIP | CityU | 80.2 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 12 | SegVol | BAAI | 79.3 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 13 | UCTransNet | Northeastern University | 77.8 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
| 14 | Swin UNETR | NVIDIA | 76.9 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | UNesT | NVIDIA | 75.1 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 16 | UNETR | NVIDIA | 74.7 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 17 | SAM-Adapter | Duke | 49.4 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
KidneyL - Diff-UNet 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | Diff-UNet | HKUST | 91.9 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 2 | MedFormer | Rutgers | 91.9 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 3 | nnU-Net ResEncL | DKFZ | 91.9 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 4 | STU-Net-B | Shanghai AI Lab | 91.9 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 5 | nnU-Net U-Net | DKFZ | 91.9 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 6 | LHU-Net | UR | 91.8 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 7 | MedNeXt | DKFZ | 91.8 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 8 | SegVol | BAAI | 91.8 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 9 | UniSeg | NPU | 91.5 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 10 | U-Net & CLIP | CityU | 91.1 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 11 | Swin UNETR & CLIP | CityU | 91.0 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 12 | UNesT | NVIDIA | 90.1 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 13 | Swin UNETR | NVIDIA | 89.7 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 14 | NexToU | HIT | 89.6 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 15 | UNETR | NVIDIA | 89.2 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 16 | SAM-Adapter | Duke | 87.3 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
| 17 | UCTransNet | Northeastern University | 86.9 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
KidneyR - Diff-UNet 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | Diff-UNet | HKUST | 92.8 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 2 | MedFormer | Rutgers | 92.8 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 3 | nnU-Net U-Net | DKFZ | 92.7 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 4 | MedNeXt | DKFZ | 92.6 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 5 | nnU-Net ResEncL | DKFZ | 92.6 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 6 | LHU-Net | UR | 92.5 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 7 | STU-Net-B | Shanghai AI Lab | 92.5 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 8 | SegVol | BAAI | 92.5 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 9 | UniSeg | NPU | 92.2 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 10 | U-Net & CLIP | CityU | 91.9 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 11 | Swin UNETR & CLIP | CityU | 91.7 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 12 | UNesT | NVIDIA | 90.9 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 13 | SAM-Adapter | Duke | 90.4 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
| 14 | NexToU | HIT | 90.1 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 15 | UNETR | NVIDIA | 90.1 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 16 | Swin UNETR | NVIDIA | 89.8 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 17 | UCTransNet | Northeastern University | 86.5 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
Liver - MedFormer 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | MedFormer | Rutgers | 96.4 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 2 | MedNeXt | DKFZ | 96.3 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 3 | nnU-Net ResEncL | DKFZ | 96.3 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 4 | LHU-Net | UR | 96.2 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 5 | nnU-Net U-Net | DKFZ | 96.2 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 6 | Diff-UNet | HKUST | 96.2 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 7 | STU-Net-B | Shanghai AI Lab | 96.2 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 8 | UniSeg | NPU | 96.1 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 9 | U-Net & CLIP | CityU | 96.0 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 10 | SegVol | BAAI | 96.0 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 11 | Swin UNETR & CLIP | CityU | 95.8 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 12 | NexToU | HIT | 95.7 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 13 | SAM-Adapter | Duke | 94.1 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
| 14 | UNesT | NVIDIA | 95.3 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | Swin UNETR | NVIDIA | 95.2 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 16 | UNETR | NVIDIA | 95.0 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 17 | UCTransNet | Northeastern University | 93.6 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
Pancreas - MedNeXt 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | MedNeXt | DKFZ | 83.3 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 2 | STU-Net-B | Shanghai AI Lab | 83.2 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 3 | MedFormer | Rutgers | 83.1 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 4 | nnU-Net ResEncL | DKFZ | 82.9 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 5 | UniSeg | NPU | 82.7 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 6 | nnU-Net U-Net | DKFZ | 82.3 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 7 | Diff-UNet | HKUST | 81.9 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 8 | LHU-Net | UR | 81.0 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 9 | U-Net & CLIP | CityU | 80.8 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 10 | Swin UNETR & CLIP | CityU | 80.2 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 11 | NexToU | HIT | 80.2 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 12 | SegVol | BAAI | 79.1 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 13 | UNesT | NVIDIA | 76.2 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 14 | Swin UNETR | NVIDIA | 75.6 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | UNETR | NVIDIA | 72.3 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 16 | UCTransNet | Northeastern University | 59.0 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
| 17 | SAM-Adapter | Duke | 50.2 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
Postcava - STU-Net-B & MedNeXt 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | STU-Net-B | Shanghai AI Lab | 81.3 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 🏆 | MedNeXt | DKFZ | 81.3 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 3 | UniSeg | NPU | 81.2 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 4 | nnU-Net U-Net | DKFZ | 81.0 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 5 | Diff-UNet | HKUST | 80.8 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 6 | MedFormer | Rutgers | 80.7 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 7 | nnU-Net ResEncL | DKFZ | 80.5 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 8 | LHU-Net | UR | 79.4 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 9 | U-Net & CLIP | CityU | 78.5 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 10 | NexToU | HIT | 78.1 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 11 | SegVol | BAAI | 77.8 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 12 | Swin UNETR & CLIP | CityU | 76.8 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 13 | Swin UNETR | NVIDIA | 75.4 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 14 | UNesT | NVIDIA | 74.4 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | UNETR | NVIDIA | 71.5 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | UCTransNet | Northeastern University | 68.1 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
| 17 | SAM-Adapter | Duke | 48.0 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
Spleen - MedFormer 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | MedFormer | Rutgers | 95.5 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 2 | nnU-Net ResEncL | DKFZ | 95.2 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 3 | MedNeXt | DKFZ | 95.2 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 4 | nnU-Net U-Net | DKFZ | 95.1 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 5 | STU-Net-B | Shanghai AI Lab | 95.1 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 6 | Diff-UNet | HKUST | 95.0 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 7 | LHU-Net | UR | 94.9 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 8 | UniSeg | NPU | 94.9 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 9 | SegVol | BAAI | 94.5 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 10 | NexToU | HIT | 94.7 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 11 | U-Net & CLIP | CityU | 94.3 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 12 | Swin UNETR & CLIP | CityU | 94.1 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 13 | UNesT | NVIDIA | 93.2 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 14 | Swin UNETR | NVIDIA | 92.7 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | UNETR | NVIDIA | 91.7 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 16 | SAM-Adapter | Duke | 90.5 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
| 17 | UCTransNet | Northeastern University | 90.2 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
Stomach - STU-Net-B 🏆
| rank | model | organization | DSC | paper | github |
|:---|:---|:---|:---:|:---:|:---:|
| 🏆 | STU-Net-B | Shanghai AI Lab | 93.5 | [](https://arxiv.org/pdf/2304.06716) | [](https://github.com/uni-medical/STU-Net) |
| 2 | MedNeXt | DKFZ | 93.5 | [](https://arxiv.org/pdf/2303.09975) | [](https://github.com/MIC-DKFZ/MedNeXt) |
| 3 | nnU-Net ResEncL | DKFZ | 93.4 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 4 | MedFormer | Rutgers | 93.4 | [](https://arxiv.org/abs/2203.00131) | [](https://github.com/yhygao/CBIM-Medical-Image-Segmentation) |
| 5 | UniSeg | NPU | 93.3 | [](https://arxiv.org/abs/2304.03493) | [](https://github.com/yeerwen/UniSeg) |
| 6 | nnU-Net U-Net | DKFZ | 93.3 | [](https://arxiv.org/abs/1809.10486) | [](https://github.com/MIC-DKFZ/nnUNet) |
| 7 | Diff-UNet | HKUST | 93.1 | [](https://arxiv.org/abs/2303.10326) | [](https://github.com/ge-xing/Diff-UNet) |
| 8 | LHU-Net | UR | 93.0 | [](https://arxiv.org/abs/2404.05102) | [](https://github.com/xmindflow/LHUNet) |
| 9 | NexToU | HIT | 92.7 | [](https://arxiv.org/abs/2305.15911) | [](https://github.com/PengchengShi1220/NexToU) |
| 10 | SegVol | BAAI | 92.5 | [](https://arxiv.org/abs/2311.13385) | [](https://github.com/BAAI-DCAI/SegVol) |
| 11 | U-Net & CLIP | CityU | 92.4 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 12 | Swin UNETR & CLIP | CityU | 92.2 | [](https://arxiv.org/abs/2301.00785) | [](https://github.com/ljwztc/CLIP-Driven-Universal-Model) |
| 13 | UNesT | NVIDIA | 90.9 | [](https://arxiv.org/abs/2303.10745) | [](https://github.com/Project-MONAI/tutorials) |
| 14 | Swin UNETR | NVIDIA | 90.5 | [](https://arxiv.org/abs/2211.11537) | [](https://github.com/Project-MONAI/tutorials) |
| 15 | UNETR | NVIDIA | 88.8 | [](https://arxiv.org/abs/2111.04004) | [](https://github.com/Project-MONAI/tutorials) |
| 16 | SAM-Adapter | Duke | 88.0 | [](https://arxiv.org/abs/2404.09957) | [](https://github.com/mazurowski-lab/finetune-SAM) |
| 17 | UCTransNet | Northeastern University | 81.9 | []([https://arxiv.org/abs/2211.11537](https://arxiv.org/pdf/2109.04335)) | []([https://github.com/McGregorWwww/UCTransNet](https://github.com/McGregorWwww/UCTransNet)) |
# Touchstone 1.0 Dataset
### Training set
- Touchstone 1.0: [AbdomenAtlas1.0Mini](https://huggingface.co/datasets/AbdomenAtlas/AbdomenAtlas1.0Mini) (*N*=5,195)
- Touchstone 2.0: [AbdomenAtlas1.1Mini](https://huggingface.co/datasets/AbdomenAtlas/_AbdomenAtlas1.1Mini) (*N*=9,262)
### Test set
- Proprietary [JHH dataset](https://www.sciencedirect.com/science/article/pii/S2211568419301391) (*N*=5,172)
- Public [TotalSegmentator V2](https://github.com/wasserth/TotalSegmentator) dataset (*N*=1,228)
*Figure 1. Metadata distribution in the test set.*
# Touchstone 1.0 Model
> [!NOTE]
> We are releasing the trained AI models evaluated in Touchstone right here. Stay tuned!
| rank | model | average DSC | parameter | infer. speed | download |
|:---|:---|:---|:---|:---|:---|
| 🏆 | MedNeXt | 89.2 | 61.8M | ★☆☆☆☆ | |
| 🏆 | MedFormer | 89.0 | 38.5M | ★★★☆☆ | |
| 3 | STU-Net-B | 89.0 | 58.3M | ★★☆☆☆ | 
|
| 4 | nnU-Net U-Net | 88.9 | 102.0M | ★★★★☆ | |
| 5 | nnU-Net ResEncL | 88.8 | 102.0M | ★★★★☆ | |
| 6 | UniSeg | 88.8 | 31.0M | ☆☆☆☆☆ | |
| 7 | Diff-UNet | 88.5 | 434.0M | ★★★☆☆ | |
| 8 | LHU-Net | 88.0 | 8.6M | ★★★★★ | |
| 9 | NexToU | 87.8 | 81.9M | ★★★★☆ | |
| 10 | SegVol | 87.1 | 181.0M | ★★★★☆ | |
| 11 | U-Net & CLIP | 87.1 | 19.1M | ★★★☆☆ | |
| 12 | Swin UNETR & CLIP | 86.7 | 62.2M | ★★★☆☆ | |
| 13 | Swin UNETR | 84.8 | 72.8M | ★★★★★ | |
| 14 | UNesT | 84.9 | 87.2M | ★★★★★ | |
| 15 | UNETR | 83.3 | 101.8M | ★★★★★ | |
| 16 | UCTransNet | 81.1 | 68.0M | ★★★★☆ | |
| 17 | SAM-Adapter | 73.4 | 11.6M | ★★★★☆ | |
# Citation
Please cite the following papers if you find our study helpful.
```
@article{bassi2024touchstone,
title={Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?},
author={Bassi, Pedro RAS and Li, Wenxuan and Tang, Yucheng and Isensee, Fabian and Wang, Zifu and Chen, Jieneng and Chou, Yu-Cheng and Kirchhoff, Yannick and Rokuss, Maximilian and Huang, Ziyan and Ye, Jin and He, Junjun and Wald, Tassilo and Ulrich, Constantin and Baumgartner, Michael and Roy, Saikat and Maier-Hein, Klaus H. and Jaeger, Paul and Ye, Yiwen and Xie, Yutong and Zhang, Jianpeng and Chen, Ziyang and Xia, Yong and Xing, Zhaohu and Zhu, Lei and Sadegheih, Yousef and Bozorgpour, Afshin and Kumari, Pratibha and Azad, Reza and Merhof, Dorit and Shi, Pengcheng and Ma, Ting and Du, Yuxin and Bai, Fan and Huang, Tiejun and Zhao, Bo and Wang, Haonan and Li, Xiaomeng and Gu, Hanxue and Dong, Haoyu and Yang, Jichen and Mazurowski, Maciej A. and Gupta, Saumya and Wu, Linshan and Zhuang, Jiaxin and Chen, Hao and Roth, Holger and Xu, Daguang and Blaschko, Matthew B. and Decherchi, Sergio and Cavalli, Andrea and Yuille, Alan L. and Zhou, Zongwei},
journal={Conference on Neural Information Processing Systems},
year={2024},
utl={https://github.com/MrGiovanni/Touchstone}
}
@article{li2024abdomenatlas,
title={AbdomenAtlas: A large-scale, detailed-annotated, \& multi-center dataset for efficient transfer learning and open algorithmic benchmarking},
author={Li, Wenxuan and Qu, Chongyu and Chen, Xiaoxi and Bassi, Pedro RAS and Shi, Yijia and Lai, Yuxiang and Yu, Qian and Xue, Huimin and Chen, Yixiong and Lin, Xiaorui and others},
journal={Medical Image Analysis},
pages={103285},
year={2024},
publisher={Elsevier}
}
@inproceedings{li2024well,
title={How Well Do Supervised Models Transfer to 3D Image Segmentation?},
author={Li, Wenxuan and Yuille, Alan and Zhou, Zongwei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024}
}
@article{qu2023abdomenatlas,
title={Abdomenatlas-8k: Annotating 8,000 CT volumes for multi-organ segmentation in three weeks},
author={Qu, Chongyu and Zhang, Tiezheng and Qiao, Hualin and Tang, Yucheng and Yuille, Alan L and Zhou, Zongwei and others},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2023}
}
```
# Acknowledgement
This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the McGovern Foundation. Paper content is covered by patents pending.
