{"id":14260816,"url":"https://github.com/oppo-us-research/SpacetimeGaussians","last_synced_at":"2025-08-13T09:32:29.877Z","repository":{"id":214452285,"uuid":"736495761","full_name":"oppo-us-research/SpacetimeGaussians","owner":"oppo-us-research","description":"[CVPR 2024] Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis","archived":false,"fork":false,"pushed_at":"2024-08-23T20:06:14.000Z","size":44426,"stargazers_count":546,"open_issues_count":24,"forks_count":40,"subscribers_count":21,"default_branch":"main","last_synced_at":"2024-08-24T13:15:10.168Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://oppo-us-research.github.io/SpacetimeGaussians-website/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oppo-us-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-28T04:16:32.000Z","updated_at":"2024-08-24T13:07:52.000Z","dependencies_parsed_at":"2024-08-22T13:12:54.866Z","dependency_job_id":null,"html_url":"https://github.com/oppo-us-research/SpacetimeGaussians","commit_stats":null,"previous_names":["oppo-us-research/spacetimegaussians"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oppo-us-research%2FSpacetimeGaussians","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oppo-us-research%2FSpacetimeGaussians/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oppo-us-research%2FSpacetimeGaussians/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oppo-us-research%2FSpacetimeGaussians/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oppo-us-research","download_url":"https://codeload.github.com/oppo-us-research/SpacetimeGaussians/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229754467,"owners_count":18119124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-22T13:00:34.055Z","updated_at":"2024-12-14T20:30:37.371Z","avatar_url":"https://github.com/oppo-us-research.png","language":"Python","funding_links":[],"categories":["2024"],"sub_categories":["Real"],"readme":"# Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis\n\n## CVPR 2024\n\n[Project Page](https://oppo-us-research.github.io/SpacetimeGaussians-website/) | [Paper](https://arxiv.org/abs/2312.16812) | [Video](https://youtu.be/YsPPmf-E6Lg) | [Viewer \u0026 Pre-Trained Models](https://huggingface.co/stack93/spacetimegaussians/tree/main)\n\n\nThis is an official implementation of the paper \"Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis\".\u003c/br\u003e\n[Zhan Li](https://lizhan17.github.io/web/)\u003csup\u003e1,2\u003c/sup\u003e, \n[Zhang Chen](https://zhangchen8.github.io/)\u003csup\u003e1,\u0026dagger;\u003c/sup\u003e, \n[Zhong Li](https://sites.google.com/site/lizhong19900216)\u003csup\u003e1,\u0026dagger;\u003c/sup\u003e, \n[Yi Xu](https://www.linkedin.com/in/yi-xu-42654823/)\u003csup\u003e1\u003c/sup\u003e \u003c/br\u003e\n\u003csup\u003e1\u003c/sup\u003e OPPO US Research Center, \u003csup\u003e2\u003c/sup\u003e Portland State University \u003c/br\u003e\n\u003csup\u003e\u0026dagger;\u003c/sup\u003e Corresponding authors \u003c/br\u003e\n\n\u003cimg src=\"assets/output.gif\" width=\"100%\"/\u003e\u003c/br\u003e\n\n## Updates and News\n- `Jun 16, 2024`: Added fully fused mlp for testing ours-full models on Technicolor and Neural 3D dataset (40 FPS improvement compared to paper).\n- `Jun 13, 2024`: Fixed minors for reproducity on the scenes ```coffee_martini``` and ```flame_salmon_1``` (~ 0.1 PSNR).\n- `Jun 9, 2024` : Supported lazy loading and ground truth image as int8 in GPU.\n- `Dec 28, 2023`: Paper and Code are released.\n\n\n\n## Table of Contents\n1. [Installation](#installation)\n1. [Preprocess Datasets](#processing-datasets)\n1. [Training](#training)\n1. [Testing](#testing)\n1. [Real-Time Viewer](#real-time-viewer)\n1. [Creating Your Gaussians](#create-your-new-representations-and-rendering-pipeline)\n1. [License Infomration](#license-information)\n1. [Acknowledgement](#acknowledgement)\n1. [Citations](#citations)\n\n\n## Installation\n### Windows users with WSL2 :\nPlease first refer to [here](./script/wsl.md) to install the WSL2 system (Windows Subsystem for Linux 2) and install dependencies inside WSL2. Then you can set up our repo inside the Linux sub-system same as other Linux users. \n### Linux users :\nClone the source code of this repo.\n```\ngit clone https://github.com/oppo-us-research/SpacetimeGaussians.git --recursive\ncd SpacetimeGaussians\n```\n\nThen run the following command to install the environments with conda.\nNote we will create two environments, one for preprocessing with colmap (```colmapenv```) and one for training and testing (```feature_splatting```). Training, testing and preprocessing have been tested on Ubuntu 20.04. \u003c/br\u003e\n```\nbash script/setup.sh\n```\nNote that you may need to manually install the following packages if you encounter errors during the installation of the above command. \u003c/br\u003e\n\n```\nconda activate feature_splatting\npip install thirdparty/gaussian_splatting/submodules/gaussian_rasterization_ch9\npip install thirdparty/gaussian_splatting/submodules/gaussian_rasterization_ch3\npip install thirdparty/gaussian_splatting/submodules/forward_lite\npip install thirdparty/gaussian_splatting/submodules/forward_full\n```\n\n## Processing Datasets\nNote, our paper uses the sparse points that follow 3DGS. Our per frame SfM points only use ```point_triangulator``` in Colmap instead of dense points.  \n### Neural 3D Dataset\nDownload the dataset from [here](https://github.com/facebookresearch/Neural_3D_Video.git).\nAfter downloading the dataset, you can run the following command to preprocess the dataset. \u003c/br\u003e\n```\nconda activate colmapenv\npython script/pre_n3d.py --videopath \u003clocation\u003e/\u003cscene\u003e\n```\n```\u003clocation\u003e``` is the path to the dataset root folder, and ```\u003cscene\u003e``` is the name of a scene in the dataset. \u003c/br\u003e\n\n- For example if you put the dataset at ```/home/Neural3D```, and want to preprocess the ```cook_spinach``` scene, you can run the following command\n```\nconda activate colmapenv\npython script/pre_n3d.py --videopath /home/Neural3D/cook_spinach/\n```\n\nOur codebase expects the following directory structure for the Neural 3D Dataset after preprocessing:\n```\n\n\u003clocation\u003e\n|---cook_spinach\n|   |---colmap_\u003c0\u003e\n|   |---colmap_\u003c...\u003e\n|   |---colmap_\u003c299\u003e\n|---flame_salmon1\n\n```\n### Technicolor Dataset\nPlease reach out to the authors of the paper \"Dataset and Pipeline for Multi-View Light-Field Video\" for access to the Technicolor dataset. \u003c/br\u003e\nOur codebase expects the following directory structure for this dataset before preprocessing:\n```\n\n\u003clocation\u003e\n|---Fabien\n|   |---Fabien_undist_\u003c00257\u003e_\u003c08\u003e.png\n|   |---Fabien_undist_\u003c.....\u003e_\u003c..\u003e.png\n|---Birthday\n\n```\nThen run the following command to preprocess the dataset. \u003c/br\u003e\n```\nconda activate colmapenv\npython script/pre_technicolor.py --videopath \u003clocation\u003e/\u003cscene\u003e\n```\n### Google Immersive Dataset \nDownload the dataset from [here](https://github.com/augmentedperception/deepview_video_dataset).\nAfter downloading and unzip the dataset, you can run the following command to preprocess the dataset. \u003c/br\u003e\n```\nconda activate colmapenv\npython script/pre_immersive_distorted.py --videopath \u003clocation\u003e/\u003cscene\u003e\npython script/pre_immersive_undistorted.py --videopath \u003clocation\u003e/\u003cscene\u003e\n```\n```\u003clocation\u003e``` is the path to the dataset root folder, and ```\u003cscene\u003e``` is the name of a scene in the dataset. Please rename the orginal file to the name list ```Immersiveseven```in [here](./script/pre_immersive_distorted.py) \n\n- For example if you put the dataset at ```/home/immersive```, and want to preprocess the ```02_Flames``` scene, you can run the following command\n```\nconda activate colmapenv\npython script/pre_immersive_distorted.py --videopath /home/immersive/02_Flames/\n```\n\n\n\n1. Our codebase expects the following directory structure for immersive dataset before preprocessing\n```\n\u003clocation\u003e\n|---02_Flames\n|   |---camera_0001.mp4\n|   |---camera_0002.mp4\n|---09_Alexa\n```\n\n2. Our codebase expects the following directory structure for immersive dataset (raw video, decoded images, distorted and undistorted) after preprocessing:\n\n```\n\u003clocation\u003e\n|---02_Flames\n|   |---camera_0001\n|   |---camera_0001.mp4\n|   |---camera_\u003c...\u003e\n|---02_Flames_dist\n|   |---colmap_\u003c0\u003e\n|   |---colmap_\u003c...\u003e\n|   |---colmap_\u003c299\u003e\n|---02_Flames_undist\n|   |---colmap_\u003c0\u003e\n|   |---colmap_\u003c...\u003e\n|   |---colmap_\u003c299\u003e\n|---09_Alexa\n|---09_Alexa_dist\n|---09_Alexa_undist\n```\n\n3. Copy the picked views files to the scene dir. The views is generated by inferencing our model initialized with ```duration=1``` points without training. We provide generated views in pkl for reproducity and simplicity. \n- For example, for the scene ```09_Alexa``` with distortion model.\ncopy ```configs/im_view/09_Alexa/pickview.pkl``` to ```\u003clocation\u003e/09_Alexa_dist/pickview.pkl```\n\n\n\n## Training\nYou can train our model by running the following command: \u003c/br\u003e\n\n```\nconda activate feature_splatting\npython train.py --quiet --eval --config configs/\u003cdataset\u003e_\u003clite|full\u003e/\u003cscene\u003e.json --model_path \u003cpath to save model\u003e --source_path \u003clocation\u003e/\u003cscene\u003e/colmap_0\n```\nIn the argument to ```--config```, ```\u003cdataset\u003e``` can be ```n3d``` (for Neural 3D Dataset) or ```techni``` (for Technicolor Dataset), and you can choose between ```full``` model or ```lite``` model. \u003c/br\u003e\nYou need 24GB GPU memory to train on the Neural 3D Dataset. \u003c/br\u003e\nYou need 48GB GPU memory to train on the Technicolor Dataset. \u003c/br\u003e\nThe large memory requirement is because training images are loaded into GPU memory. \u003c/br\u003e\n- For example, if you want to train the **lite** model on the first 50 frames of the ```cook_spinach``` scene in the Neural 3D Dataset, you can run the following command \u003c/br\u003e\n```\npython train.py --quiet --eval --config configs/n3d_lite/cook_spinach.json --model_path log/cook_spinach_lite --source_path \u003clocation\u003e/cook_spinach/colmap_0 \n```\n\n- If you want to train the **full** model, you can run the following command \u003c/br\u003e\n\n```\npython train.py --quiet --eval --config configs/n3d_full/cook_spinach.json --model_path log/cook_spinach/colmap_0 --source_path \u003clocation\u003e/cook_spinach/colmap_0 \n```\nPlease refer to the .json config files for more options.\n\n\n- If you want to train the **full** model with **distorted** immersive dataset, you can run the following command \u003c/br\u003e\n\n```\nPYTHONDONTWRITEBYTECODE=1 python train_imdist.py --quiet --eval --config configs/im_distort_full/02_Flames.json --model_path log/02_Flames/colmap_0 --source_path \u003clocation\u003e/02_Flames_dist/colmap_0 \n```\n\nNote, sometimes pycache file somehow affects the results. Please remove every pycache file and retrain the model without generating BYTECODE by ```PYTHONDONTWRITEBYTECODE=1```.\n\n\n\n- If you want to train the **lite** model with **undistorted** immersive dataset.   \nNote, we remove the ```--eval``` to reuse the loader of technicolor and also to train with all cameras.  ```gtmask 1``` is specially for training with undistorted fisheye images that have black pixels.\n\n```\npython train.py --quiet --gtmask 1 --config configs/im_undistort_lite/02_Flames.json --model_path log/02_Flames/colmap_0 --source_path \u003clocation\u003e/02_Flames_undist/colmap_0 \n```\n\nPlease refer to the .json config files for more options.\n\n\n## Testing\n\n- Test model on Neural 3D Dataset\n\n```\npython test.py --quiet --eval --skip_train --valloader colmapvalid --configpath configs/n3d_\u003clite|full\u003e/\u003cscene\u003e.json --model_path \u003cpath to model\u003e --source_path \u003clocation\u003e/\u003cscene\u003e/colmap_0\n```\n\n- Test model on Technicolor Dataset\n```\npython test.py --quiet --eval --skip_train --valloader technicolorvalid --configpath configs/techni_\u003clite|full\u003e/\u003cscene\u003e.json --model_path \u003cpath to model\u003e --source_path \u003clocation\u003e/\u003cscenename\u003e/colmap_0\n```\n- Test on Google Immersive Dataset with distortion camera model \n\nFist Install fused mlp layer.\n```\npip install thirdparty/gaussian_splatting/submodules/forward_full\n```\n\n```\nPYTHONDONTWRITEBYTECODE=1 CUDA_VISIBLE_DEVICES=0 python test.py --quiet --eval --skip_train --valloader immersivevalidss --configpath configs/im_distort_\u003clite|full\u003e/\u003cscene\u003e.json --model_path \u003cpath to model\u003e --source_path \u003clocation\u003e/\u003cscenename\u003e/colmap_0\n```\n\n\n## Real-Time Viewer \nThe viewer is based on [SIBR](https://sibr.gitlabpages.inria.fr/) and [Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting). \n### Pre-built Windows Binary\nDownload the viewer binary from [this link](https://huggingface.co/stack93/spacetimegaussians/tree/main) and unzip it. The binary works for Windows with CUDA \u003e= 11.0.\nWe also provide pre-trained models in the link. For example, [n3d_sear_steak_lite_allcam.zip](https://huggingface.co/stack93/spacetimegaussians/blob/main/n3d_sear_steak_lite_allcam.zip) contains the lite model that uses all views during training for the sear_steak scene in the Neural 3D Dataset.\n### Installation from Source \nplease see bottom commented text [this link](./script/setup.sh)\n### Running the Real-Time Viewer\nAfter downloading the pre-built binary or installing from source, you can use the following command to run the real-time viewer. Adjust ```--iteration``` to match the training iterations of model. \u003c/br\u003e\n```\n./\u003cSIBR install dir\u003e/bin/SIBR_gaussianViewer_app_rwdi.exe --iteration 25000 -m \u003cpath to trained model\u003e \n``` \nThe above command has beed tested on Nvidia RTX 3050 Laptop GPU + Windows 10.\n- For 8K rendering, you can use the following command. \u003c/br\u003e\n```\n./\u003cSIBR install dir\u003e/bin/SIBR_gaussianViewer_app_rwdi.exe --iteration 25000 --rendering-size 8000 4000 --force-aspect-ratio 1 -m \u003cpath to trained model\u003e \n``` \n8K rendering has been tested on Nvidia RTX 4090 + Windows 11. \n\n### Third Party Implemented Web Viewer \nWe thank Kevin Kwok (Antimatter15) for the amazing web viewer of our method: splaTV . The web viewer is released at [github](https://github.com/antimatter15/splaTV).\nYou can view one of our scene from the [web viewer](http://antimatter15.com/splaTV/).\n## Create Your New Representations and Rendering Pipeline\nIf you want to customize our codebase for your own models, you can refer to the following steps \u003c/br\u003e\n- Step 1: Create a new Gaussian representation in this [folder](./thirdparty/gaussian_splatting/scene/). You can use ```oursfull.py``` or ```ourslite.py``` as a template. \u003c/br\u003e\n- Step 2: Create a new rendering pipeline in this [file](./thirdparty/gaussian_splatting/renderer/__init__.py). You can use the ```train_ours_full``` function as a template. \u003c/br\u003e\n- Step 3 (For new dataset, optional): Create a new dataloader in this [file](./thirdparty/gaussian_splatting/scene/__init__.py) and this [file](./thirdparty/gaussian_splatting/scene/dataset_readers.py). \u003c/br\u003e\n- Step 4: Update the intermidiate API in ```getmodel``` (for Step 1) and ```getrenderpip``` (for Step 2) functions in ```helper_train.py```.\u003c/br\u003e\n\n\n## License Information\nThe code in this repository (except the thirdparty folder) is licensed under MIT licence, see [LICENSE](LICENSE). thirdparty/gaussian_splatting is licensed under Gaussian-Splatting license, see [thirdparty/gaussian_splatting/LICENSE.md](thirdparty/gaussian_splatting/LICENSE.md). thirdparty/colmap is licensed under new BSD license, see [thirdparty/colmap/LICENSE.txt](thirdparty/colmap/LICENSE.txt).\n\n\n## Acknowledgement\nWe sincerely thank the owners of the following source code repos, which are used by our released codes:\n[Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting),\n[Colmap](https://github.com/colmap/colmap).\nSome parts of our code referenced the following repos:\n[Gaussian Splatting with Depth](https://github.com/JonathonLuiten/diff-gaussian-rasterization-w-depth),\n[SIBR](https://gitlab.inria.fr/sibr/sibr_core.git), \n[fisheye-distortion](https://github.com/Synthesis-AI-Dev/fisheye-distortion).\n\nWe sincerely thank the anonymous reviewers of CVPR2024 for their helpful feedbacks. \n\n\nwe also thank Michael Rubloff for his post on [radiancefields](https://radiancefields.com/splatv-dynamic-gaussian-splatting-viewer/). \nWe also want to thank MrNeRF for [posts](https://x.com/janusch_patas/status/1740621964480217113?s=20) about our paper and other Guassian Splatting based papers. \n\n\n## Citations\nPlease cite our paper if you find it useful for your research.\n```\n@InProceedings{Li_STG_2024_CVPR,\n    author    = {Li, Zhan and Chen, Zhang and Li, Zhong and Xu, Yi},\n    title     = {Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2024},\n    pages     = {8508-8520}\n}\n```\n\nPlease also cite the following paper if you use Gaussian Splatting.\n```\n@Article{kerbl3Dgaussians,\n      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\\\"u}hler, Thomas and Drettakis, George},\n      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},\n      journal      = {ACM Transactions on Graphics},\n      number       = {4},\n      volume       = {42},\n      month        = {July},\n      year         = {2023},\n      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foppo-us-research%2FSpacetimeGaussians","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foppo-us-research%2FSpacetimeGaussians","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foppo-us-research%2FSpacetimeGaussians/lists"}