https://github.com/vita-group/comp4d
"Comp4D: Compositional 4D Scene Generation", Dejia Xu*, Hanwen Liang*, Neel P. Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N. Plataniotis, and Zhangyang Wang
https://github.com/vita-group/comp4d
4d gaussian-splatting generative-ai text-to-4d
Last synced: 10 months ago
JSON representation
"Comp4D: Compositional 4D Scene Generation", Dejia Xu*, Hanwen Liang*, Neel P. Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N. Plataniotis, and Zhangyang Wang
- Host: GitHub
- URL: https://github.com/vita-group/comp4d
- Owner: VITA-Group
- Created: 2024-03-25T03:12:02.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-25T05:32:35.000Z (almost 2 years ago)
- Last Synced: 2025-04-22T18:54:26.595Z (about 1 year ago)
- Topics: 4d, gaussian-splatting, generative-ai, text-to-4d
- Language: Python
- Homepage: https://vita-group.github.io/Comp4D/
- Size: 60.1 MB
- Stars: 79
- Watchers: 5
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Comp4D: Compositional 4D Scene Generation
The official implementation of paper "Comp4D: Compositional 4D Scene Generation".
[[Project Page]](https://vita-group.github.io/Comp4D/) | [[Video (narrated)]](https://www.youtube.com/watch?v=9q8SV1Xf_Xw) | [[Video (results)]](https://www.youtube.com/watch?v=gXVoPTGb734) | [[Paper]](https://github.com/VITA-Group/Comp4D/blob/main/assets/Comp4D.pdf) | [[Arxiv]](https://arxiv.org/abs/2403.16993)
## News
- 2024.8.19: Revised to support more objects.
- 2024.4.1: Released code!
- 2024.3.25: Released on arxiv!
## Overview
As shown in the figure above, we introduce **Comp**ositional **4D** Scene Generation. Previous works concentrate on object-centric 4D objects with limited movement. In comparison, our work extends the boundaries to the demanding task of compositional 4D scene generation. We integrate GPT-4 to decompose the scene and design proper trajectories, resulting in larger-scale movements and more realistic object interactions.
## Setup
```bash
conda env create -f environment.yml
conda activate Comp4D
pip install -r requirements.txt
# 3D Gaussian Splatting modules, skip if you already installed them
# a modified gaussian splatting (+ depth, alpha rendering)
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install ./diff-gaussian-rasterization
pip install ./simple-knn
```
## Example Case
#### Prompt Case
"a butterfly flies towards the flower"
#### Compositional Scene training
```
python train_comp.py --configs arguments/comp_butterfly_flower_zs.py --expname butterflyflower_exp --cfg_override 100.0 --image_weight_override 0.02 --nn_weight 1000 --with_reg --loss_dx_weight_override 0.005
```
---
We provide a quick overview of some important arguments:
- `--expname`: Experimental path.
- `--configs`: Configuration of scene training including prompt, object identity, object scales, and trajectory. You can also use [VideoCrafter](https://github.com/AILab-CVC/VideoCrafter) in replace of Zeroscope for video-based diffusion model.
- `--image_weight`: Weight of sds loss from image-based diffusion model.
- `--nn_weight`: Weight of knn based rigidity loss.
- `--loss_dx_weight`: Weight of regularization acceleration loss.
#### Rendering
```
python render_comp_video.py --skip_train --configs arguments/comp_butterfly_flower_zs.py --skip_test --model_path output_demo/date/butterflyflower_exp_date/ --iteration 3000
```
## Static Assets Preparation
We release a set of pre-generated static assets in `data/` directory. During training, we keep the static 3D Gaussians fixed and only optimize the deformation modules. We referred to the first two stages of [4D-fy](https://github.com/sherwinbahmani/4dfy) to generate the static 3D objects. Then we convert them to point clouds (in `data/`) which are used to initialize 3D Gaussians. Thanks to the authors for sharing their awesome work!
#### Example case
```
# cd /path_to_4dfy/
## Stage 1
# python launch.py --config configs/fourdfy_stage_1_low_vram.yaml --train --gpu 0 exp_root_dir=output/ seed=0 system.prompt_processor.prompt="a flower"
## Stage 2
# ckpt=output/fourdfy_stage_1_low_vram/a_flower@timestamp/ckpts/last.ckpt
# python launch.py --config configs/fourdfy_stage_2_low_vram.yaml --train --gpu 0 exp_root_dir=output/ seed=0 system.prompt_processor.prompt="a flower" system.weights=$ckpt
## Post-Process. Convert to mesh file.
# python launch.py --config output/fourdfy_stage_2_low_vram/a_flower@timestamp/configs/parsed.yaml --export --gpu 0 \
# resume=output/fourdfy_stage_2_low_vram/a_flower@timestamp/ckpts/last.ckpt system.exporter_type=mesh-exporter \
# system.exporter.context_type=cuda system.exporter.fmt=obj
## saved to output/fourdfy_stage_2_low_vram/a_flower@timestamp/save/iterations-export/
## Convert to point cloud.
# cd /path_to_Comp4D/
# python mesh2ply_8w.py /path_to_4dfy/output/fourdfy_stage_2_low_vram/a_flower@timestamp/save/iterations-export/model.obj data/a_flower.ply
```
## Acknowledgement
This work is built on many amazing research works and open-source projects. Thanks to all the authors for sharing!
- https://github.com/sherwinbahmani/4dfy
- https://github.com/hustvl/4DGaussians
- https://github.com/dreamgaussian/dreamgaussian
- https://github.com/graphdeco-inria/gaussian-splatting
- https://github.com/graphdeco-inria/diff-gaussian-rasterization
- https://github.com/threestudio-project/threestudio
## Citation
If you find this repository/work helpful in your research, please consider citing the paper and starring the repo ⭐.
```
@article{xu2024comp4d,
title={Comp4D: LLM-Guided Compositional 4D Scene Generation},
author={Xu, Dejia and Liang, Hanwen and Bhatt, Neel P and Hu, Hezhen and Liang, Hanxue and Plataniotis, Konstantinos N and Wang, Zhangyang},
journal={arXiv preprint arXiv:2403.16993},
year={2024}
}
```