Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ShanghaiTech-IMPACT/TeethDreamer

[MICCAI 2024] TeethDreamer: 3D Teeth Reconstruction from Five Intra-oral Photographs
https://github.com/ShanghaiTech-IMPACT/TeethDreamer

Last synced: about 2 months ago
JSON representation

[MICCAI 2024] TeethDreamer: 3D Teeth Reconstruction from Five Intra-oral Photographs

Awesome Lists containing this project

README 

        
# TeethDreamer: 3D Teeth Reconstruction from Five Intra-oral Photographs

Chenfan Xu*, [Zhentao Liu](https://zhentao-liu.github.io/)*, Yuan Liu, Yulong Dou, Jiamin Wu, Jiepeng Wang, Minjiao Wang, [Dingggang Shen](https://idea.bme.shanghaitech.edu.cn/), and [Zhiming Cui](https://shanghaitech-impact.github.io/)+.



## [[Paper](https://arxiv.org/abs/2407.11419)]|[[Project Page](https://shanghaitech-impact.github.io/TeethDreamer/)]



- [x] Inference code and pretrained models.

- [x] Training code.

- [x] [Rendering dataset](https://pan.baidu.com/s/1uX-JB4GCKdtOBjYbkrFylg?pwd=wuhp) with 16 views for each case.



### Demo

- [x] Five intra-oral photos in `example/oral`.

- [x] Segmented teeth images in `example/teeth`.

- [x] Generated [images](https://shanghaitecheducn-my.sharepoint.com/:f:/g/personal/xuchf2023_shanghaitech_edu_cn/EsjO23PTzMtJmk-XpkMh01oBEfaui-ZZ_5afbU1t9Ijc6g?e=4aG5ZS) and [Reconstruction](https://shanghaitecheducn-my.sharepoint.com/:f:/g/personal/xuchf2023_shanghaitech_edu_cn/EuqQwcmmWH9NsdaeDYnO17kB7z7PDRP_YLzQ_cm-OOQxDQ?e=F2AigU).



### Getting started

1. Install packages in `requirements.txt`. We test our model on a 40G A100 GPU with 11.6 CUDA and 1.12.0 pytorch. 

```angular2html

conda create -n TeethDreamer

conda activate TeethDreamer

pip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu116

pip install -r requirements.txt

```

2. Download pretrained model [checkpoints](https://shanghaitecheducn-my.sharepoint.com/:f:/g/personal/xuchf2023_shanghaitech_edu_cn/Em8U10EGc09Om0SQKA-s6WoBiYX9hkm-0BWiReTFnwIsuQ?e=6vb0Xl)

### Inference

1. Make sure you have the following models.

```bash

SyncDreamer

|-- ckpt

    |-- ViT-L-14.ckpt

    |-- TeethDreamer.ckpt

    |-- zero123-xl.ckpt

    |-- sam_vit_b_01ec64.pth

```

2. Segment upper and lower teeth from your five intral-oral photos. You can use our script `seg_teeth.py` in the command line. Before that, you need renumber your five intral-oral images by 0~4 corresponding to anterior view, left buccal view, right buccal view, maxillary occlusal view, and mandibular occlusal view. And then you can manually segment upper teeth and lower teeth by clicking left mouse button in the target area within the interactive interface created by our script.



Tips: You can refer to image files in `example` folder.

```angular2html

python seg_teeth.py --img directory/of/your/intra-oral/images \

                    --seg directory/to/store/segmented/images \

                    --suffix suffix/of/your/image/files

```

Tips: You need segment upper teeth for the first four intra-oral images and then lower teeth for the last four images. If unexpected regions are segmented, you can click the right mouse button to label the irrelevant area.



3. Make sure you have following file structure.

```bash

|-- your_seg_dir

    |-- XXX_norm_lower

    |-- XXX_norm_upper

```



4. Generate color and normal images of eight viewpoints.

```angular2html

python TeethDreamer.py -b configs/TeethDreamer.yaml \

                       --gpus 0 \

                       --test ckpt/TeethDreamer.ckpt \

                       --output directory/to/store/generated/images \

                       data.params.test_dir=directory/of/segmented/images

```

5. (Optional) Segment foreground of the generated image manually which is necessary for Neus (Because the foreground mask automatically generated by `rembg` package may be wrong sometimes)

```angular2html

python seg_foreground.py --img path/to/your/generated/image \

                         --seg path/to/your/segmented/image \

```

6. Reconstruct tooth model from generation by Neus.

```angular2html

cd instant-nsr-pl

python run.py --img ../example/results/generation/1832_upper_cond_000_000_000_000.png" \

              --cpu 4 \

              --dir ../example/results/reconstruction/ \

              --normal \

              --rembg

```

Explanation: 

- `--img` is the path to your generated image

- `--cpu` is the number of your CPU cores

- `--dir` is the directory to store reconstruction

- `--normal` indicates the generation includes normal images

- `--rembg` indicates background removement (The foreground mask is necessary here)



### Data Preparation for Training

1. Make sure you have normalized tooth models which are segmented from intra-oral scanning models.

2. Render color and normal images by blender scripts. We test following scripts on Windows with Blender 4.0.1

```angular2html

blender --background

        --python normal_render.py

        -- --object_path path/to/your/tooth/model

        --target_dir directory/to/store/rendered/normal/images

        --input_dir directory/to/store/rendered/condition/images

```

```angular2html

blender --background

        --python color_render.py

        -- --object_path path/to/your/tooth/model

        --target_dir directory/to/store/rendered/color/images

        --input_dir directory/to/store/rendered/condition/images

```

Explanation: 

`normal_render.py` only renders target normal images with fixed viewpoints set by `view16` dictionary.

`color_render.py` renders condition images corresponding to segmented intra-oral photos taken by dentists and target color images.

3. Make sure you have a `pkl` file which includes a dictionary with `train`, `val` keys and corresponding lists including cases' ids such as `XXX_norm_lower` and `XXX_norm_upper`.

4. Check if your directory of rendering data has following structures.

```bash

Data

|-- target

|-- normal

|-- input

|-- mv-splits.pkl

```

Explanation: 

The `target` folder is the directory of your rendered color images which is the argument `target_dir` in the script `color_render.py`.



The `normal` folder is the directory of your rendered normal images which is the argument `target_dir` in the script `normal_render.py`.



The `input` folder is the directory of your rendered condition images which is the argument `input_dir` in the script `color_render.py`.



The `mv-splits.pkl` file is the `pkl` file mentioned in the previous step.

5. Finetune pretrained zero123 model on your own data.

```angular2html

python TeethDreamer.py -b configs/TeethDreamer.yaml \

                       --gpus 0 \

                       --finetune_from ckpt/zero123-xl.ckpt \

                       data.target_dir=path/to/your/target/folder \

                       data.input_dir=path/to/your/input/folder \

                       data.uid_set_pkl=path/to/your/pkl/file \

                       data.validation_dir=path/to/your/input/folder

```



## Acknowledgement



We have intensively borrow codes from the following repositories. Many thanks to the authors for sharing their codes.



- [stable diffusion](https://github.com/CompVis/stable-diffusion)

- [Zero123](https://github.com/cvlab-columbia/zero123)

- [SyncDreamer](https://github.com/liuyuan-pal/SyncDreamer)

- [Wonder3D](https://github.com/xxlong0/Wonder3D)

- [Instant-nsr-pl](https://github.com/bennyguo/instant-nsr-pl)

- [Segment-Anything](https://github.com/facebookresearch/segment-anything)



## Citation

If you find this repository useful in your project, please cite the following work. :)

```

@InProceedings{10.1007/978-3-031-72104-5_68,

author="Xu, Chenfan and Liu, Zhentao and Liu, Yuan and Dou, Yulong and Wu, Jiamin and Wang, Jiepeng and Wang, Minjiao and Shen, Dinggang and Cui, Zhiming",

editor="Linguraru, Marius George and Dou, Qi and Feragen, Aasa and Giannarou, Stamatia and Glocker, Ben and Lekadir, Karim and Schnabel, Julia A.",

title="TeethDreamer: 3D Teeth Reconstruction from Five Intra-Oral Photographs",

booktitle="Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024",

year="2024",

publisher="Springer Nature Switzerland",

address="Cham",

pages="712--721",

isbn="978-3-031-72104-5"

}

```