Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://yxlao.github.io/corres-nerf/

Source Code for "CorresNeRF: Image Correspondence Priors for Neural Radiance Fields" (NeurIPS 2023)
https://yxlao.github.io/corres-nerf/

Last synced: 3 months ago
JSON representation

Source Code for "CorresNeRF: Image Correspondence Priors for Neural Radiance Fields" (NeurIPS 2023)

Host: GitHub
URL: https://yxlao.github.io/corres-nerf/
Owner: yxlao
License: mit
Created: 2023-10-27T10:18:07.000Z (about 1 year ago)
Default Branch: main
Last Pushed: 2024-09-04T03:10:03.000Z (4 months ago)
Last Synced: 2024-10-03T12:23:11.035Z (3 months ago)
Homepage: https://yxlao.github.io/corres-nerf
Size: 35.7 MB
Stars: 44
Watchers: 8
Forks: 0
Open Issues: 4
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

awesome-scene-representation - CorresNeRF: Image Correspondence Priors for Neural Radiance Fields - nerf) | [bibtex](./citations/lao2023corresnerf.txt) (Uncategorized / Uncategorized)

README

CorresNeRF: Image Correspondence Priors for Neural Radiance Fields

Yixing Lao
·
Xiaogang Xu
·
Zhipeng Cai
·
Xihui Liu
·
Hengshuang Zhao

(NeurIPS 2023)

For more details, please visit our project page and paper.

## Installation

```bash
# Environment
conda create -n corresnerf python=3.10
conda activate corresnerf

# Install dependencies
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
pip install git+https://github.com/Parskatt/DKM.git
```

## Data

- Download [LLFF dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) for NeRF and extract to `data/nerf_llff`.
- Download [DTU dataset](https://www.dropbox.com/scl/fo/um3wj3ctiuoottbfmqmgb/ABZRltszDvWHJ824UL6DHw0?rlkey=3vjok0aivnoiaf8z5j6w05k92&e=1&dl=0) for NeuS and extract to `data/neus_dtu`.

After extraction, we shall have:

## Pre-trained weights

We release [pre-trained weights](https://drive.google.com/file/d/1C4sNNVd43pWP9qGvKdEu7RdgU-hSJ41Z/view?usp=sharing) for the paper for each scene.
Extract `pretrained_weights.zip` to `logs/` and we shall have:

## Inference with the pre-trained weights

We use the `fern` scene for NeRF on LLFF and the `dtu_scan24` scene for NeuS on DTU as example.

```bash
# Inference with NeRF. This will load the latest checkpoint (050000.tar in this case).
# - The rendered images will be saved in logs/nerf_llff/fern/renderall_050000.
python -m src.tools.run_nerf --config configs/nerf_llff.txt --scene fern --factor 8 --render_all

# Inference with NeuS. This will load the specified checkpoint (ckpt_200000.pth in this case).
# - The rendered images will be saved in logs/neus_dtu/dtu_scan24/renders.
# - The extracted mesh will be saved in logs/neus_dtu/dtu_scan24/meshes.
python -m src.tools.run_neus --conf configs/neus_dtu.conf --case dtu_scan24 --from_checkpoint ckpt_200000.pth render --resolution_level 4
python -m src.tools.run_neus --conf configs/neus_dtu.conf --case dtu_scan24 --from_checkpoint ckpt_200000.pth extract_mesh
```

## Correspondence matching and training

The full pipeline includes:

- Extract correspondences.
- Train NeRF / NeuS models.
- Render with the trained NeRF / NeuS.
- Extract mesh with the trained NeuS.

For NeRF on LLFF, we use the `fern` scene as example.

```bash
# Matcher
python -m src.tools.run_matcher --enable_matcher --enable_filter --matcher_name dkm --config_name default --num_views 3 --dataset llff --scene_dir data/nerf_llff/fern --corres_dir data_corres/nerf_llff/fern

# Train
python -m src.tools.run_nerf --config configs/nerf_llff.txt --scene fern

# Render
python -m src.tools.run_nerf --config configs/nerf_llff.txt --scene fern --factor 8 --render_all
```

For NeuS on DTU, we use the `dtu_scan24` scene as example.

```bash
# Match
python -m src.tools.run_matcher --enable_matcher --enable_filter --matcher_name dkm --config_name default --num_views 3 --dataset dtu --scene_dir data/neus_dtu/dtu_scan24 --corres_dir data_corres/neus_dtu/dtu_scan24

# Train
python -m src.tools.run_neus --conf configs/neus_dtu.conf --case dtu_scan24 --is_continue train

# Render & extract mesh
python -m src.tools.run_neus --conf configs/neus_dtu.conf --case dtu_scan24 --from_checkpoint ckpt_200000.pth render --resolution_level 4
python -m src.tools.run_neus --conf configs/neus_dtu.conf --case dtu_scan24 --from_checkpoint ckpt_200000.pth extract_mesh
```

## Citation

If you find our work useful in your research, please consider citing:

```bib
@inproceedings{lao2023corresnerf,
title = {{CorresNeRF}: Image Correspondence Priors for Neural Radiance Fields},
author = {Lao, Yixing and Xu, Xiaogang and Cai, Zhipeng and Liu, Xihui and Zhao, Hengshuang},
booktitle = {NeurIPS},
year = {2023}
}
```

This project is built with [CamTools](https://github.com/yxlao/camtools).