https://github.com/prs-eth/dynamic-lidar-resimulation
[CVPR 2024, highlight] Dynamic LiDAR Re-simulation using Compositional Neural Fields
https://github.com/prs-eth/dynamic-lidar-resimulation
Last synced: over 1 year ago
JSON representation
[CVPR 2024, highlight] Dynamic LiDAR Re-simulation using Compositional Neural Fields
- Host: GitHub
- URL: https://github.com/prs-eth/dynamic-lidar-resimulation
- Owner: prs-eth
- Created: 2023-11-23T14:00:15.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-05-15T13:02:59.000Z (about 2 years ago)
- Last Synced: 2025-03-24T18:11:21.919Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 5.52 MB
- Stars: 84
- Watchers: 4
- Forks: 5
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Dynamic LiDAR Re-simulation using Compositional Neural Fields
This repository represents the official implementation of the paper:
### [Dynamic LiDAR Re-simulation using Compositional Neural Fields](https://arxiv.org/abs/2312.05247)
[Hanfeng Wu1,2](https://www.linkedin.com/in/hanfeng-wu-089602203/), [Xingxing Zuo2](https://xingxingzuo.github.io/), [Stefan Leutenegger2](https://www.professoren.tum.de/leutenegger-stefan), [Or Litany3](https://orlitany.github.io/), [Konrad Schindler1](https://prs.igp.ethz.ch/group/people/person-detail.schindler.html), [Shengyu Huang1](https://shengyuh.github.io) \
1 ETH Zurich | 2 Technical University of Munich | 3 Technion | 4 NVIDIA
## Introduction
This code consists of two parts. The folder `DyNFL` contains our nerfstudio-based DyNFL implementation. The folder `WaymoPreprocessing` contains the preprocessing scripts to generate dataset used for training and evaluation.
## Enrionment setup
This code has been tested on
- Python 3.10.10, PyTorch 1.13.1, CUDA 11.6, Nerfstudio 0.3.4, GeForce RTX 3090/GeForce GTX 1080Ti
To create a conda environment and install the required dependences, please run:
```shell
conda create -n "dynfl" python=3.10.10
conda activate dynfl
```
Then install nerfstudio from [official guidelines](https://docs.nerf.studio/quickstart/installation.html) in your conda environment.
After intalling nerfstudio, install DyNFL and required packages as follows:
```
git clone git@github.com:prs-eth/DyNFL-Dynamic-LiDAR-Re-simulation-using-Compositional-Neural-Fields.git
cd DyNFL
pip install -e .
pip install NFLStudio/ChamferDistancePytorch/chamfer3D/
pip install NFLStudio/raymarching/
```
## Prepare Datasets
Please refer to [WaymoPreprocessing](./WaymoPreprocessing).
## Training
After generating the datasets, please run
```shell
ns-train NFLStudio --pipeline.datamanager.dataparser-config.context_name --pipeline.datamanager.dataparser-config.root_dir --experiment_name
```
You can set the batch-size in [NFLDataManagerConfig](./DyNFL/NFLStudio/datamanager.py) or pass it as arguments along with the `ns-train` command
## Evaluation and visulization
After training, a folder containing weights will be saved in your working folder as `outputs//NFLStudio//nerfstudio_models`
### Quantitative results
In order to get quantitative results, please uncomment in [tester](./DyNFL/NFLStudio/tester.py)
```
# pipeline.get_numbers() # get numbers of the LiDARs
```
and run
```
cd NFLStudio
python tester.py --context_name --model_dir
```
for example
```
python tester.py --context_name "1083056852838271990_4080_000_4100_000" --model_dir "outputs//NFLStudio//nerfstudio_modelss"
```
### Reproduce our results
We provide you the pretrained model weights of 4 Waymo Dynamic scenes to reproduce our results, the weights are avaliable [here](https://mega.nz/file/R7N3nRJT#R_avxijtzFYtxyEM_ExxSHH8AiQdA2gE8-AGxmHaxtg). Unpack the pretrained weights under `DyNFL/NFLStudio/`
and run
```
cd NFLStudio
bash test.sh
```
### Visulization
To visulize the results, uncomment in [tester](./DyNFL/NFLStudio/tester.py)
```
# pipeline.get_pcd(context_name) # get pcd to display
```
and run
```
cd NFLStudio
python tester.py --context_name --model_dir
```
After that, you will have a folder in `NFLStudio` called `pcd_out`.
replace the `dir` and `context_name` with `path/to/pcd_out` and in [NFLStudio/render_full_lidar_inputs.py](./DyNFL/NFLStudio/render_full_lidar_inputs.py)
and run
```
cd NFLStudio
python render_full_lidar_inputs.py
```
It will generate two windows like follows:
## Citation
```bibtex
@inproceedings{Wu2023dynfl,
title={Dynamic LiDAR Re-simulation using Compositional Neural Fields},
author={Wu, Hanfeng and Zuo, Xingxing and Leutenegger, Stefan and Litany, Or and Schindler, Konrad and Huang, Shengyu},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2024},
}
```
## Acknowledgement
We use the nerfstudio framework in this project, we thank the contributors for their open-sourcing and maintenance of the work.