Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://aibluefisher.github.io/dbarf/
Official Implementation of our CVPR 2023 paper: "DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields"
https://aibluefisher.github.io/dbarf/
Last synced: 12 days ago
JSON representation
Official Implementation of our CVPR 2023 paper: "DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields"
- Host: GitHub
- URL: https://aibluefisher.github.io/dbarf/
- Owner: AIBluefisher
- Created: 2023-03-16T06:25:45.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-11-16T07:18:21.000Z (12 months ago)
- Last Synced: 2024-08-01T16:35:51.301Z (3 months ago)
- Language: Python
- Size: 271 MB
- Stars: 96
- Watchers: 14
- Forks: 5
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-scene-representation - DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields
README
# Deep Bundle Adjusting Generalizable Neural Radiance Fields
Official Implementation of our CVPR 2023 paper: "DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields"
[[Project Page](https://aibluefisher.github.io/dbarf/) | [arXiv](https://arxiv.org/abs/2303.14478)]
## 1. Installation
```sh
conda create -n dbarf python=3.9
conda activate dbarf
# install pytorch
# # CUDA 10.2
# conda install pytorch==1.11.0 torchvision==0.12.0 cudatoolkit=10.2 -c pytorch
# CUDA 11.3
conda install pytorch==1.11.0 torchvision==0.12.0 cudatoolkit=11.3 -c pytorch
# HLoc is used for extracting keypoints and matching features.
git clone --recursive https://github.com/cvg/Hierarchical-Localization/
cd Hierarchical-Localization/
python -m pip install -e .
cd ..
pip install opencv-python matplotlib easydict tqdm networkx einops imageio visdom tensorboardX configargparse lpips
```
## 2. Preprocessing
### 1) Extracting Scene Graph
After installing HLoc, we can extract the scene graph for each scene:
```sh
python3 -m scripts.preprocess_dbarf_dataset --dataset_dir $image_dir --outputs $output_dir --gpu_idx 0 --min_track_length 2 --max_track_length 15 --recon False --disambiguate False --visualize False
```
For debugging, we can also enable incremental SfM (*not necessary for dbarf since our method does not rely on ground-truth camera poses*) by using `--recon True`, removing ambiguous wrong matches by `--disambiguate True`, and visualizing reconstruction results by `--visualize True`.
### 2) Post-processing COLMAP Model
Also, we need to convert colmap's model into the `.npy` format with post-processing:
```sh
python3 -m scripts.colmap_model_to_poses_bounds --input_dir $colmap_model_dir
```
## 3. Dataset Structure
```
IBRNet
├── train
│ ├── real_iconic_noface
│ │ ├── airplants
│ │ │ ├── images/
│ │ │ ├── images_4/
│ │ │ ├── images_8/
│ │ │ ├── database.db
│ │ │ ├── poses_bounds.npy
│ │ │ ├── VG_N_M.g2o
│ │ ├── ...
│ ├── ibrnet_collected_1
│ │ ├── ...
│ ├── ibrnet_collected_2
│ │ ├── ...
│ ├── ...
├── eval
│ ├── nerf_llff_data
│ │ ├── ...
│ ├── ibrnet_collected_more
│ ├── ...
```
## 4. Training & Evaluation
Once your data is prepared, you can train IBRNet and DBARF. At first, you need to edit the corresponding configuration file to train the desired dataset. Note that for IBRNet, we only train a coarse nerf instead of a coarse nerf + a fine nerf. To reproduce our results, it is recommended to use our **[pretrained model](https://drive.google.com/drive/folders/1K7mqz11fwtlCNr5sR8-YKeVWC3Si6DBi?usp=sharing)** for further finetuning.
### Training
```sh
cd scripts/shell
cd scripts/shell
# Train coarse-only IBRNet
./train_coarse_ibrnet.sh pretrain False 0 # For pretraining
./train_coarse_ibrnet.sh finetune False 0 # For finetuning
# Train DBARF
./train_dbarf.sh pretrain False 0 # For pretraining
./train_dbarf.sh finetune False 0 # For finetuning
```
### Evaluation
```sh
cd scripts/shell
ITER_NUMBER=200000
GPU_IDX=0
# For coarse-only IBRNet
./eval_coarse_llff_all.sh $ITER_NUMBER $GPU_IDX
./eval_coarse_scannet.sh $ITER_NUMBER $GPU_IDX
# For DBARF
./eval_dbarf_llff_all.sh $ITER_NUMBER $GPU_IDX
./eval_dbarf_ibr_collected_all.sh $GPU_IDX
./eval_dbarf_scannet.sh $ITER_NUMBER $GPU_IDX
```
### Rendering videos
```sh
cd scripts/shell
ITER_NUMBER=200000
GPU_IDX=0
# For coarse ibrnet
./render_coarse_llff_all.sh $ITER_NUMBER $GPU_IDX
# For dbarf
./render_dbarf_llff_all.sh $ITER_NUMBER $GPU_IDX
```
# Citation
If you find our code is useful for your research, consider cite our paper as following:
```
@InProceedings{Chen_2023_CVPR,
author = {Chen, Yu and Lee, Gim Hee},
title = {DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2023},
pages = {24-34}
}
```