https://github.com/fengres/mixvoxels
ICCV2023 (Oral) MixVoxels https://arxiv.org/abs/2212.00190
https://github.com/fengres/mixvoxels
Last synced: 2 months ago
JSON representation
ICCV2023 (Oral) MixVoxels https://arxiv.org/abs/2212.00190
- Host: GitHub
- URL: https://github.com/fengres/mixvoxels
- Owner: fengres
- License: gpl-3.0
- Created: 2022-11-30T18:57:28.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-08-16T05:41:53.000Z (almost 2 years ago)
- Last Synced: 2024-11-07T06:41:17.795Z (7 months ago)
- Language: Python
- Homepage:
- Size: 113 MB
- Stars: 161
- Watchers: 4
- Forks: 16
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-dynamic-NeRF - Code
README
# [ICCV2023 Oral] MixVoxels: Mixed Neural Voxels for Fast Multi-view Video Synthesis
### [Project Page](https://fengres.github.io/mixvoxels/) | [Paper](https://arxiv.org/pdf/2212.00190.pdf)
Pytorch implementation for the paper: [Mixed Neural Voxels for Fast Multi-view Video Synthesis](https://arxiv.org/pdf/2212.00190.pdf).
https://user-images.githubusercontent.com/43294876/204843741-0a2d10a8-f0b4-4f69-b262-7fc7983a20c5.mp4
More complicated scenes with fast movements and large areas of motions.
https://user-images.githubusercontent.com/43294876/228771137-ed67629f-e109-47cc-b0f1-50904435546d.mp4
We present MixVoxels to better represent the dynamic scenes with fast training speed and competitive rendering qualities.
The proposed MixVoxels represents the 4D dynamic scenes as a mixture of static and dynamic voxels and processes them with different networks.
In this way, the computation of the required modalities for static voxels can be processed by a lightweight model,
which essentially reduces the amount of computation,
especially for many daily dynamic scenes dominated by the static background.
As a result, with 15 minutes of training for dynamic scenes with inputs of 300-frame videos, MixVoxels achieves better PSNR than previous methods.
## Installation
Install environment:
```
conda create -n mixvoxels python=3.8
conda activate mixvoxels
pip install torch torchvision
pip install tqdm scikit-image opencv-python configargparse lpips imageio-ffmpeg kornia lpips tensorboard pyfvvdp
```
## Dataset
1. Download the [Plenoptic Video Dataset](https://github.com/facebookresearch/Neural_3D_Video)
2. Unzip to your directory DATADIR and run the following command:
```
python tools/prepare_video.py ${DATADIR}
```
## Training
To train a dynamic scene, run the following commands, you can train different dynamic scenes by assign DATA to different scene name:
```
DATA=coffee_martini # [coffee_martini|cut_roasted_beef|cook_spinach|flame_salmon|flame_steak|sear_steak]
# MixVoxels-T
python train.py --config configs/schedule5000/${DATA}_5000.txt --render_path 0
# MixVoxels-S
python train.py --config configs/schedule7500/${DATA}_7500.txt --render_path 0
# MixVoxels-M
python train.py --config configs/schedule12500/${DATA}_12500.txt --render_path 0
# MixVoxels-L
python train.py --config configs/schedule25000/${DATA}_25000.txt --render_path 0
```
> Please note that in your first running, the above command will first pre-process the dataset, including resizing the frames by a factor of 2 (to 1K resolution which is a standard), as well as calculating the std of each video and save them into your disk. The pre-processing will cost about 2 hours, but is only required at the first running. After the pre-processing, the command will automatically train your scenes.
We provide the trained model:
scene
PSNR
download
MixVoxels-T (15min)
MixVoxels-M (40min)
MixVoxels-T (15min)
MixVoxels-M (40min)
coffee-martini
28.1339
29.0186
link
link
flame-salmon
28.7982
29.2620
link
link
cook-spinach
31.4499
31.6433
link
link
cut-roasted-beef
32.4078
32.2800
link
link
flame-steak
31.6508
31.3052
link
link
sear-steak
31.8203
31.2136
link
link
## Rendering and Generating Spirals
The following command will generate 120 novel view videos, or you can set the render_path as 1 in the above training command.
```
python train.py --config your_config --render_only 1 --render_path 1 --ckpt log/your_config/your_config.ckpt
```
Generating spirals:
```
python tools/make_spiral.py --videos_path log/your_config/imgs_path_all/ --target log/your_config/spirals --target_video log/your_config/spirals.mp4
```
## Citation
If you find our code or paper helps, please consider citing:
```
@article{wang2022mixed,
title={Mixed Neural Voxels for Fast Multi-view Video Synthesis},
author={Wang, Feng and Tan, Sinan and Li, Xinghang and Tian, Zeyue and Liu, Huaping},
journal={arXiv preprint arXiv:2212.00190},
year={2022}
}
```
## Acknowledge
The codes are based on [TensoRF](https://github.com/apchenstu/TensoRF), many thanks to the authors.