https://github.com/zju3dv/Vox-Fusion
Code for "Dense Tracking and Mapping with Voxel-based Neural Implicit Representation", ISMAR 2022
https://github.com/zju3dv/Vox-Fusion
Last synced: 2 months ago
JSON representation
Code for "Dense Tracking and Mapping with Voxel-based Neural Implicit Representation", ISMAR 2022
- Host: GitHub
- URL: https://github.com/zju3dv/Vox-Fusion
- Owner: zju3dv
- Created: 2022-08-20T15:44:51.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2025-02-24T09:52:43.000Z (3 months ago)
- Last Synced: 2025-03-16T04:13:29.748Z (2 months ago)
- Language: Python
- Homepage:
- Size: 5.56 MB
- Stars: 271
- Watchers: 20
- Forks: 25
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome - zju3dv/Vox-Fusion - Code for "Dense Tracking and Mapping with Voxel-based Neural Implicit Representation", ISMAR 2022 (Python)
README
# Vox-Fusion
### [Project Page](https://zju3dv.github.io/Vox-Fusion/) | [Video](https://youtu.be/Prp28y1b2Qs) | [Paper](https://arxiv.org/abs/2210.15858)
> Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation
>
> [Xingrui Yang*](https://yangxingrui.com/), [Hai Li*](https://garylidd.github.io/), [Hongjia Zhai](https://zhaihongjia.github.io/), Yuhang Ming, Yuqian Liu, [Guofeng Zhang](http://www.cad.zju.edu.cn/home/gfzhang/).
>
> ISMAR 2022
## Correction
We found a bug in the evaluation script which affected the estimated pose accuracy in Tables 1 and 3 in the original paper. We have corrected this problem and re-run the results with updated configurations. The corrected results are comparable (even better for Replica dataset) to the originally reported results in the paper, which do not affect the contribution and conclusion of our work. We have updated the [arxiv version](https://arxiv.org/abs/2210.15858) of our paper and publish all the latest results (including mesh, pose, gt, eval scripts and training configs) on [Google Drive](https://drive.google.com/drive/folders/15bwoX44uh1YCsMT4F3Td9n1h2qiUOuVG?usp=share_link), in case anyone wants to reproduce our results and compare them using different metrics.
Please use the config file replica_all.yaml and scannet_all.yaml in the **Google Drive** to replicate the results from the paper !!!
**If you want to compare our method, please use the data or config file we provide to run.**
## Performance on TUM-RGBD
We provide the results on the TUM-RGBD dataset.
| Metrics | fr1/desk | fr1/desk2 | fr1/room |fr2/xyz | fr3/office |
|:----------------:|:-------------:|:--------------:|:--------------:|:-----------------------:|:---------------------:|
| ATE RMSE [cm] | 2.54 | 3.95 | 12.51 | 1.40 | - |
| ATE Mean [cm] | 2.26 | 3.18 | 11.63 | 1.19 | - |
| ATE Median [cm] | 2.11 | 2.64 | 10.54 | 0.95 | - |
## Installation
It is recommended to install [Pytorch](https://pytorch.org/get-started/locally/) (>=1.10) manually for your hardware platform first. You can then install all dependancies using `pip` or `conda`:
```
pip install -r requirements.txt
```
After you have installed all third party libraries, run the following script to build extra Pytorch modules used in this project.
```bash
sh install.sh
```
Replace the filename in mapping.py with the built library
```python
torch.classes.load_library("third_party/sparse_octree/build/lib.xxx/svo.xxx.so")
```
## Demo
It is simple to run Vox-Fusion on datasets that already have dataloaders. `src/datasets` list all existing dataloaders. You can of course build your own, we will come back to it later. For now, we use the replica dataset as an example.
First you have to modify `configs/replica/room_0.yaml` so the `data_path` section points to the real dataset path. Now you are all set to run the code:
```
python demo/run.py configs/replica/room_0.yaml
```
You should now see a progress bar and some output indicating the system is now running. For now you have to rely on the progress bar to estimate the running time as we are still working on a working GUI.
## Custom Datasets
You can use virtually any RGB-D dataset with Vox-Fusion including self-captured ones. Make sure to adapt the config files and dataloaders and put them in the correct folder. Make sure to implement a `get_init_pose` function for your dataloader, please refer to `src/datasets/tum.py` for an example.
## Acknowledgement
Some of our codes are adapted from [Nerual RGB-D Surface Reconstruction](https://dazinovic.github.io/neural-rgbd-surface-reconstruction/) and [BARF: Bundle Adjusted NeRF](https://github.com/chenhsuanlin/bundle-adjusting-NeRF/blob/main/camera.py).
## Citation
If you find our code or paper useful, please cite
```bibtex
@inproceedings{yang2022vox,
title={Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation},
author={Yang, Xingrui and Li, Hai and Zhai, Hongjia and Ming, Yuhang and Liu, Yuqian and Zhang, Guofeng},
booktitle={2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
pages={499--507},
year={2022},
}
```
## Contact
Contact [Xingrui Yang](mailto:[email protected]) and [Hai Li](mailto:[email protected]) for questions, comments and reporting bugs.