{"id":15034110,"url":"https://github.com/hustvl/4dgaussians","last_synced_at":"2025-05-14T04:10:16.557Z","repository":{"id":199928602,"uuid":"699641579","full_name":"hustvl/4DGaussians","owner":"hustvl","description":"[CVPR 2024] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering","archived":false,"fork":false,"pushed_at":"2024-10-27T04:07:37.000Z","size":67801,"stargazers_count":2551,"open_issues_count":113,"forks_count":219,"subscribers_count":35,"default_branch":"master","last_synced_at":"2025-04-10T22:35:17.007Z","etag":null,"topics":["3d","computer-vision","cvpr2024","dynamic-scene","gaussian-splatting","graphics","neural-network","neural-rendering","novel-view-synthesis","radiance-field"],"latest_commit_sha":null,"homepage":"https://guanjunwu.github.io/4dgs/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hustvl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-10-03T03:26:33.000Z","updated_at":"2025-04-10T08:28:44.000Z","dependencies_parsed_at":"2023-12-11T07:42:10.232Z","dependency_job_id":"de000e85-aa3a-41a9-8a28-c9642224400f","html_url":"https://github.com/hustvl/4DGaussians","commit_stats":{"total_commits":51,"total_committers":7,"mean_commits":7.285714285714286,"dds":0.6274509803921569,"last_synced_commit":"9e32cb1cfd8dfef9096b09509db2175c1f1a35f3"},"previous_names":["hustvl/4dgaussians"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2F4DGaussians","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2F4DGaussians/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2F4DGaussians/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2F4DGaussians/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hustvl","download_url":"https://codeload.github.com/hustvl/4DGaussians/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254070112,"owners_count":22009559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","computer-vision","cvpr2024","dynamic-scene","gaussian-splatting","graphics","neural-network","neural-rendering","novel-view-synthesis","radiance-field"],"created_at":"2024-09-24T20:23:58.034Z","updated_at":"2025-05-14T04:10:11.540Z","avatar_url":"https://github.com/hustvl.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering\n\n## CVPR 2024\n\n### [Project Page](https://guanjunwu.github.io/4dgs/index.html)| [arXiv Paper](https://arxiv.org/abs/2310.08528)\n\n[Guanjun Wu](https://guanjunwu.github.io/) \u003csup\u003e1*\u003c/sup\u003e, [Taoran Yi](https://github.com/taoranyi) \u003csup\u003e2*\u003c/sup\u003e,\n[Jiemin Fang](https://jaminfong.cn/) \u003csup\u003e3‡\u003c/sup\u003e, [Lingxi Xie](http://lingxixie.com/) \u003csup\u003e3 \u003c/sup\u003e, \u003c/br\u003e[Xiaopeng Zhang](https://scholar.google.com/citations?user=Ud6aBAcAAAAJ\u0026hl=zh-CN) \u003csup\u003e3 \u003c/sup\u003e, [Wei Wei](https://www.eric-weiwei.com/) \u003csup\u003e1 \u003c/sup\u003e,[Wenyu Liu](http://eic.hust.edu.cn/professor/liuwenyu/) \u003csup\u003e2 \u003c/sup\u003e, [Qi Tian](https://www.qitian1987.com/) \u003csup\u003e3 \u003c/sup\u003e , [Xinggang Wang](https://xwcv.github.io) \u003csup\u003e2‡✉\u003c/sup\u003e\n\n\u003csup\u003e1 \u003c/sup\u003eSchool of CS, HUST \u0026emsp; \u003csup\u003e2 \u003c/sup\u003eSchool of EIC, HUST \u0026emsp; \u003csup\u003e3 \u003c/sup\u003eHuawei Inc. \u0026emsp;\n\n\u003csup\u003e\\*\u003c/sup\u003e Equal Contributions. \u003csup\u003e$\\ddagger$\u003c/sup\u003e Project Lead. \u003csup\u003e✉\u003c/sup\u003e Corresponding Author.\n\n\n\n![block](assets/teaserfig.jpg)\nOur method converges very quickly and achieves real-time rendering speed.\n\nNew Colab demo:[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wz0D5Y9egAlcxXy8YO9UmpQ9oH51R7OW?usp=sharing) (Thanks [Tasmay-Tibrewal\n](https://github.com/Tasmay-Tibrewal))\n\nOld Colab demo:[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/hustvl/4DGaussians/blob/master/4DGaussians.ipynb) (Thanks [camenduru](https://github.com/camenduru/4DGaussians-colab).)\n\nLight Gaussian implementation: [This link](https://github.com/pablodawson/4DGaussians) (Thanks [pablodawson](https://github.com/pablodawson))\n\n\n## News\n\n2024.6.25: we clean the code and add an explanation of the parameters.\n\n2024.3.25: Update guidance for hypernerf and dynerf dataset.\n\n2024.03.04: We change the hyperparameters of the Neu3D dataset, corresponding to our paper.\n\n2024.02.28: Update SIBR viewer guidance.\n\n2024.02.27: Accepted by CVPR 2024. We delete some logging settings for debugging, the corrected training time is only **8 mins** (20 mins before) in D-NeRF datasets and **30 mins** (1 hour before) in HyperNeRF datasets. The rendering quality is not affected.\n\n## Environmental Setups\n\nPlease follow the [3D-GS](https://github.com/graphdeco-inria/gaussian-splatting) to install the relative packages.\n\n```bash\ngit clone https://github.com/hustvl/4DGaussians\ncd 4DGaussians\ngit submodule update --init --recursive\nconda create -n Gaussians4D python=3.7 \nconda activate Gaussians4D\n\npip install -r requirements.txt\npip install -e submodules/depth-diff-gaussian-rasterization\npip install -e submodules/simple-knn\n```\n\nIn our environment, we use pytorch=1.13.1+cu116.\n\n## Data Preparation\n\n**For synthetic scenes:**\nThe dataset provided in [D-NeRF](https://github.com/albertpumarola/D-NeRF) is used. You can download the dataset from [dropbox](https://www.dropbox.com/s/0bf6fl0ye2vz3vr/data.zip?dl=0).\n\n**For real dynamic scenes:**\nThe dataset provided in [HyperNeRF](https://github.com/google/hypernerf) is used. You can download scenes from [Hypernerf Dataset](https://github.com/google/hypernerf/releases/tag/v0.1) and organize them as [Nerfies](https://github.com/google/nerfies#datasets). \n\nMeanwhile, [Plenoptic Dataset](https://github.com/facebookresearch/Neural_3D_Video) could be downloaded from their official websites. To save the memory, you should extract the frames of each video and then organize your dataset as follows.\n\n```\n├── data\n│   | dnerf \n│     ├── mutant\n│     ├── standup \n│     ├── ...\n│   | hypernerf\n│     ├── interp\n│     ├── misc\n│     ├── virg\n│   | dynerf\n│     ├── cook_spinach\n│       ├── cam00\n│           ├── images\n│               ├── 0000.png\n│               ├── 0001.png\n│               ├── 0002.png\n│               ├── ...\n│       ├── cam01\n│           ├── images\n│               ├── 0000.png\n│               ├── 0001.png\n│               ├── ...\n│     ├── cut_roasted_beef\n|     ├── ...\n```\n\n**For multipleviews scenes:**\nIf you want to train your own dataset of multipleviews scenes, you can orginize your dataset as follows:\n\n```\n├── data\n|   | multipleview\n│     | (your dataset name) \n│   \t  | cam01\n|     \t\t  ├── frame_00001.jpg\n│     \t\t  ├── frame_00002.jpg\n│     \t\t  ├── ...\n│   \t  | cam02\n│     \t\t  ├── frame_00001.jpg\n│     \t\t  ├── frame_00002.jpg\n│     \t\t  ├── ...\n│   \t  | ...\n```\nAfter that, you can use the  `multipleviewprogress.sh` we provided to generate related data of poses and pointcloud.You can use it as follows:\n```bash\nbash multipleviewprogress.sh (youe dataset name)\n```\nYou need to ensure that the data folder is organized as follows after running multipleviewprogress.sh:\n```\n├── data\n|   | multipleview\n│     | (your dataset name) \n│   \t  | cam01\n|     \t\t  ├── frame_00001.jpg\n│     \t\t  ├── frame_00002.jpg\n│     \t\t  ├── ...\n│   \t  | cam02\n│     \t\t  ├── frame_00001.jpg\n│     \t\t  ├── frame_00002.jpg\n│     \t\t  ├── ...\n│   \t  | ...\n│   \t  | sparse_\n│     \t\t  ├── cameras.bin\n│     \t\t  ├── images.bin\n│     \t\t  ├── ...\n│   \t  | points3D_multipleview.ply\n│   \t  | poses_bounds_multipleview.npy\n```\n\n\n## Training\n\nFor training synthetic scenes such as `bouncingballs`, run\n\n```\npython train.py -s data/dnerf/bouncingballs --port 6017 --expname \"dnerf/bouncingballs\" --configs arguments/dnerf/bouncingballs.py \n```\n\nFor training dynerf scenes such as `cut_roasted_beef`, run\n```python\n# First, extract the frames of each video.\npython scripts/preprocess_dynerf.py --datadir data/dynerf/cut_roasted_beef\n# Second, generate point clouds from input data.\nbash colmap.sh data/dynerf/cut_roasted_beef llff\n# Third, downsample the point clouds generated in the second step.\npython scripts/downsample_point.py data/dynerf/cut_roasted_beef/colmap/dense/workspace/fused.ply data/dynerf/cut_roasted_beef/points3D_downsample2.ply\n# Finally, train.\npython train.py -s data/dynerf/cut_roasted_beef --port 6017 --expname \"dynerf/cut_roasted_beef\" --configs arguments/dynerf/cut_roasted_beef.py \n```\nFor training hypernerf scenes such as `virg/broom`: Pregenerated point clouds by COLMAP are provided [here](https://drive.google.com/file/d/1fUHiSgimVjVQZ2OOzTFtz02E9EqCoWr5/view). Just download them and put them in to correspond folder, and you can skip the former two steps. Also, you can run the commands directly.\n\n```python\n# First, computing dense point clouds by COLMAP\nbash colmap.sh data/hypernerf/virg/broom2 hypernerf\n# Second, downsample the point clouds generated in the first step. \npython scripts/downsample_point.py data/hypernerf/virg/broom2/colmap/dense/workspace/fused.ply data/hypernerf/virg/broom2/points3D_downsample2.ply\n# Finally, train.\npython train.py -s  data/hypernerf/virg/broom2/ --port 6017 --expname \"hypernerf/broom2\" --configs arguments/hypernerf/broom2.py \n```\n\nFor training multipleviews scenes,you are supposed to build a configuration file named (you dataset name).py under \"./arguments/mutipleview\",after that,run\n```python\npython train.py -s  data/multipleview/(your dataset name) --port 6017 --expname \"multipleview/(your dataset name)\" --configs arguments/multipleview/(you dataset name).py \n```\n\n\nFor your custom datasets, install nerfstudio and follow their [COLMAP](https://colmap.github.io/) pipeline. You should install COLMAP at first, then:\n\n```python\npip install nerfstudio\n# computing camera poses by colmap pipeline\nns-process-data images --data data/your-data --output-dir data/your-ns-data\ncp -r data/your-ns-data/images data/your-ns-data/colmap/images\npython train.py -s data/your-ns-data/colmap --port 6017 --expname \"custom\" --configs arguments/hypernerf/default.py \n```\nYou can customize your training config through the config files.\n\n## Checkpoint\n\nAlso, you can train your model with checkpoint.\n\n```python\npython train.py -s data/dnerf/bouncingballs --port 6017 --expname \"dnerf/bouncingballs\" --configs arguments/dnerf/bouncingballs.py --checkpoint_iterations 200 # change it.\n```\n\nThen load checkpoint with:\n\n```python\npython train.py -s data/dnerf/bouncingballs --port 6017 --expname \"dnerf/bouncingballs\" --configs arguments/dnerf/bouncingballs.py --start_checkpoint \"output/dnerf/bouncingballs/chkpnt_coarse_200.pth\"\n# finestage: --start_checkpoint \"output/dnerf/bouncingballs/chkpnt_fine_200.pth\"\n```\n\n## Rendering\n\nRun the following script to render the images.\n\n```\npython render.py --model_path \"output/dnerf/bouncingballs/\"  --skip_train --configs arguments/dnerf/bouncingballs.py \n```\n\n## Evaluation\n\nYou can just run the following script to evaluate the model.\n\n```\npython metrics.py --model_path \"output/dnerf/bouncingballs/\" \n```\n\n\n## Viewer\n[Watch me](./docs/viewer_usage.md)\n## Scripts\n\nThere are some helpful scripts, please feel free to use them.\n\n`export_perframe_3DGS.py`:\nget all 3D Gaussians point clouds at each timestamps.\n\nusage:\n\n```python\npython export_perframe_3DGS.py --iteration 14000 --configs arguments/dnerf/lego.py --model_path output/dnerf/lego \n```\n\nYou will a set of 3D Gaussians are saved in `output/dnerf/lego/gaussian_pertimestamp`.\n\n`weight_visualization.ipynb`:\n\nvisualize the weight of Multi-resolution HexPlane module.\n\n`merge_many_4dgs.py`:\nmerge your trained 4dgs.\nusage:\n\n```python\nexport exp_name=\"dynerf\"\npython merge_many_4dgs.py --model_path output/$exp_name/sear_steak\n```\n\n`colmap.sh`:\ngenerate point clouds from input data\n\n```bash\nbash colmap.sh data/hypernerf/virg/vrig-chicken hypernerf \nbash colmap.sh data/dynerf/sear_steak llff\n```\n\n**Blender** format seems doesn't work. Welcome to raise a pull request to fix it.\n\n`downsample_point.py` :downsample generated point clouds by sfm.\n\n```python\npython scripts/downsample_point.py data/dynerf/sear_steak/colmap/dense/workspace/fused.ply data/dynerf/sear_steak/points3D_downsample2.ply\n```\n\nIn my paper, I always use `colmap.sh` to generate dense point clouds and downsample it to less than 40000 points.\n\nHere are some codes maybe useful but never adopted in my paper, you can also try it.\n\n## Awesome Concurrent/Related Works\n\nWelcome to also check out these awesome concurrent/related works, including but not limited to\n\n[Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://ingra14m.github.io/Deformable-Gaussians/)\n\n[SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes](https://yihua7.github.io/SC-GS-web/)\n\n[MD-Splatting: Learning Metric Deformation from 4D Gaussians in Highly Deformable Scenes](https://md-splatting.github.io/)\n\n[4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency](https://vita-group.github.io/4DGen/)\n\n[Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models](https://github.com/VITA-Group/Diffusion4D)\n\n[DreamGaussian4D: Generative 4D Gaussian Splatting](https://github.com/jiawei-ren/dreamgaussian4d)\n\n[EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction](https://github.com/yifliu3/EndoGaussian)\n\n[EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting](https://github.com/HKU-MedAI/EndoGS)\n\n[Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting](https://arxiv.org/abs/2401.16416)\n\n\n\n## Contributions\n\n**This project is still under development. Please feel free to raise issues or submit pull requests to contribute to our codebase.**\n\n\nSome source code of ours is borrowed from [3DGS](https://github.com/graphdeco-inria/gaussian-splatting), [K-planes](https://github.com/Giodiro/kplanes_nerfstudio), [HexPlane](https://github.com/Caoang327/HexPlane), [TiNeuVox](https://github.com/hustvl/TiNeuVox), [Depth-Rasterization](https://github.com/ingra14m/depth-diff-gaussian-rasterization). We sincerely appreciate the excellent works of these authors.\n\n## Acknowledgement\n\nWe would like to express our sincere gratitude to [@zhouzhenghong-gt](https://github.com/zhouzhenghong-gt/) for his revisions to our code and discussions on the content of our paper.\n\n## Citation\n\nSome insights about neural voxel grids and dynamic scenes reconstruction originate from [TiNeuVox](https://github.com/hustvl/TiNeuVox). If you find this repository/work helpful in your research, welcome to cite these papers and give a ⭐.\n\n```\n@InProceedings{Wu_2024_CVPR,\n    author    = {Wu, Guanjun and Yi, Taoran and Fang, Jiemin and Xie, Lingxi and Zhang, Xiaopeng and Wei, Wei and Liu, Wenyu and Tian, Qi and Wang, Xinggang},\n    title     = {4D Gaussian Splatting for Real-Time Dynamic Scene Rendering},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2024},\n    pages     = {20310-20320}\n}\n\n@inproceedings{TiNeuVox,\n  author = {Fang, Jiemin and Yi, Taoran and Wang, Xinggang and Xie, Lingxi and Zhang, Xiaopeng and Liu, Wenyu and Nie\\ss{}ner, Matthias and Tian, Qi},\n  title = {Fast Dynamic Radiance Fields with Time-Aware Neural Voxels},\n  year = {2022},\n  booktitle = {SIGGRAPH Asia 2022 Conference Papers}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhustvl%2F4dgaussians","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhustvl%2F4dgaussians","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhustvl%2F4dgaussians/lists"}