Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/eigenvivek/diffpose
[CVPR 2024] Intraoperative 2D/3D registration via differentiable X-ray rendering
https://github.com/eigenvivek/diffpose
2d-3d-registration camera-pose-estimation differentiable-rendering medical-imaging
Last synced: 24 days ago
JSON representation
[CVPR 2024] Intraoperative 2D/3D registration via differentiable X-ray rendering
- Host: GitHub
- URL: https://github.com/eigenvivek/diffpose
- Owner: eigenvivek
- License: mit
- Created: 2023-11-21T17:54:43.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-03-12T18:08:15.000Z (8 months ago)
- Last Synced: 2024-04-14T08:52:44.817Z (7 months ago)
- Topics: 2d-3d-registration, camera-pose-estimation, differentiable-rendering, medical-imaging
- Language: Python
- Homepage: http://vivekg.dev/DiffPose/
- Size: 122 MB
- Stars: 66
- Watchers: 2
- Forks: 6
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# DiffPose
> Intraoperative 2D/3D registration via differentiable X-ray rendering
[](https://github.com/eigenvivek/DiffPose/actions/workflows/test.yaml)
[](https://arxiv.org/abs/2312.06358)
[](LICENSE)
[](https://vivekg.dev/DiffPose)
[](https://github.com/psf/black)
## Install
To install `DiffPose` and the requirements in
[`environment.yml`](https://github.com/eigenvivek/DiffPose/blob/main/environment.yml),
run:
``` zsh
pip install diffpose
```
The differentiable X-ray renderer that powers the backend of `DiffPose`
is available at [`DiffDRR`](https://github.com/eigenvivek/DiffDRR).
## Datasets
We evaluate `DiffPose` networks on the following open-source datasets:
| **Dataset** | **Anatomy** | **\# of Subjects** | **\# of 2D Images** | **CTs** | **X-rays** | Fiducials |
|----------------------------------------------------------------------------|--------------------|:------------------:|:-------------------:|:-------:|:----------:|:---------:|
| [`DeepFluoro`](https://github.com/rg2/DeepFluoroLabeling-IPCAI2020) | Pelvis | 6 | 366 | ✅ | ✅ | ❌ |
| [`Ljubljana`](https://lit.fe.uni-lj.si/en/research/resources/3D-2D-GS-CA/) | Cerebrovasculature | 10 | 20 | ✅ | ✅ | ✅ |
- `DeepFluoro` ([**Grupp et al.,
2020**](https://link.springer.com/article/10.1007/s11548-020-02162-7))
provides paired X-ray fluoroscopy images and CT volume of the pelvis.
The data were collected from six cadaveric subjects at John Hopkins
University. Ground truth camera poses were estimated with an offline
registration process. A visualization of one X-ray / CT pair in the
`DeepFluoro` dataset is available
[here](https://vivekg.dev/DiffPose/experiments/render.html).
``` zsh
mkdir -p data/
wget --no-check-certificate -O data/ipcai_2020_full_res_data.zip "http://archive.data.jhu.edu/api/access/datafile/:persistentId/?persistentId=doi:10.7281/T1/IFSXNV/EAN9GH"
unzip -o data/ipcai_2020_full_res_data.zip -d data
rm data/ipcai_2020_full_res_data.zip
```
- `Ljubljana` ([**Mitrovic et al.,
2013**](https://ieeexplore.ieee.org/abstract/document/6507588))
provides paired 2D/3D digital subtraction angiography (DSA) images.
The data were collected from 10 patients undergoing endovascular
image-guided interventions at the University of Ljubljana. Ground
truth camera poses were estimated by registering surface fiducial
markers.
``` zsh
mkdir -p data/
wget --no-check-certificate -O data/ljubljana.zip "https://drive.google.com/uc?export=download&confirm=yes&id=1x585pGLI8QGk21qZ2oGwwQ9LMJ09Tqrx"
unzip -o data/ljubljana.zip -d data
rm data/ljubljana.zip
```
## Experiments
To run the experiments in `DiffPose`, run the following scripts (ensure
you’ve downloaded the data first):
``` zsh
# DeepFluoro dataset
cd experiments/deepfluoro
srun python train.py # Pretrain pose regression CNN on synthetic X-rays
srun python register.py # Run test-time optimization with the best network per subject
```
``` zsh
# Ljubljana dataset
cd experiments/ljubljana
srun python train.py
srun python register.py
```
The training and test-time optimization scripts use SLURM to run on all
subjects in parallel:
- `experiments/deepfluoro/train.py` is configured to run across six
A6000 GPUs
- `experiments/deepfluoro/register.py` is configured to run across six
2080 Ti GPUs
- `experiments/ljubljana/train.py` is configured to run across twenty
2080 Ti GPUs
- `experiments/ljubljana/register.py` is configured to run on twenty
2080 Ti GPUs
The GPU configurations can be changed at the end of each script using
[`submitit`](https://github.com/facebookincubator/submitit).
## Development
`DiffPose` package, docs, and CI are all built using
[`nbdev`](https://nbdev.fast.ai/). To get set up with`nbdev`, install
the following
``` zsh
conda install jupyterlab nbdev -c fastai -c conda-forge
nbdev_install_quarto # To build docs
nbdev_install_hooks # Make notebooks git-friendly
pip install -e ".[dev]" # Install the development verison of DiffPose
```
Running `nbdev_help` will give you the full list of options. The most
important ones are
``` zsh
nbdev_preview # Render docs locally and inspect in browser
nbdev_clean # NECESSARY BEFORE PUSHING
nbdev_test # tests notebooks
nbdev_export # builds package and builds docs
nbdev_readme # Render the readme
```
For more details, follow this [in-depth
tutorial](https://nbdev.fast.ai/tutorials/tutorial.html).
## Citing `DiffPose`
If you find `DiffPose` or
[`DiffDRR`](https://github.com/eigenvivek/DiffDRR) useful in your work,
please cite the appropriate papers:
```
@article{gopalakrishnan2023intraoperative,
title={Intraoperative {2D/3D} Image Registration via Differentiable X-ray Rendering},
author={Gopalakrishnan, Vivek and Dey, Neel and Golland, Polina},
journal={arXiv preprint arXiv:2312.06358},
year={2023}
}
@inproceedings{gopalakrishnan2022fast,
title={Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging},
author={Gopalakrishnan, Vivek and Golland, Polina},
booktitle={Workshop on Clinical Image-Based Procedures},
pages={1--11},
year={2022},
organization={Springer}
}
```