{"id":17485408,"url":"https://github.com/city-super/Scaffold-GS","last_synced_at":"2025-03-03T22:30:57.988Z","repository":{"id":210092350,"uuid":"723598714","full_name":"city-super/Scaffold-GS","owner":"city-super","description":"[CVPR 2024 Highlight] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering","archived":false,"fork":false,"pushed_at":"2024-09-26T03:17:34.000Z","size":18735,"stargazers_count":787,"open_issues_count":31,"forks_count":70,"subscribers_count":19,"default_branch":"main","last_synced_at":"2024-11-20T03:22:19.739Z","etag":null,"topics":["3d","cvpr2024","gaussian-splatting","reconstruction","rendering"],"latest_commit_sha":null,"homepage":"https://city-super.github.io/scaffold-gs","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/city-super.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-26T07:05:29.000Z","updated_at":"2024-11-19T13:48:16.000Z","dependencies_parsed_at":"2024-07-10T07:23:14.974Z","dependency_job_id":null,"html_url":"https://github.com/city-super/Scaffold-GS","commit_stats":null,"previous_names":["city-super/scaffold-gs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/city-super%2FScaffold-GS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/city-super%2FScaffold-GS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/city-super%2FScaffold-GS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/city-super%2FScaffold-GS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/city-super","download_url":"https://codeload.github.com/city-super/Scaffold-GS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241752080,"owners_count":20014224,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","cvpr2024","gaussian-splatting","reconstruction","rendering"],"created_at":"2024-10-19T02:00:54.916Z","updated_at":"2025-03-03T22:30:52.973Z","avatar_url":"https://github.com/city-super.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"# Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering\n\n[Tao Lu](https://github.com/inspirelt), [Mulin Yu](https://scholar.google.com/citations?user=w0Od3hQAAAAJ), [Linning Xu](https://eveneveno.github.io/lnxu), [Yuanbo Xiangli](https://kam1107.github.io/), [Limin Wang](https://wanglimin.github.io/), [Dahua Lin](http://dahua.site/), [Bo Dai](https://daibo.info/) \u003cbr /\u003e\n\n\n[[`Project Page`](https://city-super.github.io/scaffold-gs/)][[`arxiv`](https://arxiv.org/abs/2312.00109)][[`Viewer`](https://drive.google.com/file/d/17nPVnRRxO4zMQJ_fKHm13HzvD0wUZ8XF/view?usp=sharing)]\n\n## News\n\n**[2024.09.25]** 🎈We propose **Octree-AnyGS**, a general anchor-based framework that supports explicit Gaussians (2D-GS, 3D-GS) and neural Gaussians (Scaffold-GS). Additionally, **Octree-GS** has been adapted to the aforementioned Gaussian primitives, enabling Level-of-Detail representation for large-scale scenes. This framework holds potential for application to other Gaussian-based methods, with relevant SIBR visualizations forthcoming.(https://github.com/city-super/Octree-AnyGS)\n\n**[2024.05.28]**  We update the [viewer](https://github.com/city-super/Scaffold-GS/tree/main/SIBR_viewers) to conform to the file structure at training.\n\n**[2024.04.05]**  Scaffold-GS is selected as a 🎈**highlight** in CVPR2024.\n\n**[2024.03.27]**  🎈We release [Octree-GS](https://city-super.github.io/octree-gs), supporting an explicit *LOD* representation, rendering faster in large-scale scene with high quality.\n\n**[2024.03.26]**  🎈We release [GSDF](https://city-super.github.io/GSDF/), which improves rendering and reconstruction quality simultaneously.\n\n**[2024.02.27]**  Accepted to [CVPR 2024](https://cvpr.thecvf.com/).\n\n**[2024.01.22]**  We add the appearance embedding to Scaffold-GS to handle wild scenes.\n\n**[2024.01.22]** 🎈👀 The [viewer](https://github.com/city-super/Scaffold-GS/tree/main/SIBR_viewers) for Scaffold-GS is available now. \n\n**[2023.12.10]** We release the code.\n\n## TODO List\n- [ ] Explore on removing the MLP module\n- [ ] Improve the training configuration system\n\n## Overview\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/pipeline.png\" width=100% height=100% \nclass=\"center\"\u003e\n\u003c/p\u003e\n\n\nWe introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians, and predicts their attributes on-the-fly based on viewing direction and distance within the view frustum.\n\nOur method performs superior on scenes with challenging observing views. e.g. transparency, specularity, reflection, texture-less regions and fine-scale details.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/teaser_big.png\" width=100% height=100% \nclass=\"center\"\u003e\n\u003c/p\u003e\n\n\n\n\n\n## Installation\n\nWe tested on a server configured with Ubuntu 18.04, cuda 11.6 and gcc 9.4.0. Other similar configurations should also work, but we have not verified each one individually.\n\n1. Clone this repo:\n\n```\ngit clone https://github.com/city-super/Scaffold-GS.git --recursive\ncd Scaffold-GS\n```\n\n2. Install dependencies\n\n```\nSET DISTUTILS_USE_SDK=1 # Windows only\nconda env create --file environment.yml\nconda activate scaffold_gs\n```\n\n## Data\n\nFirst, create a ```data/``` folder inside the project path by \n\n```\nmkdir data\n```\n\nThe data structure will be organised as follows:\n\n```\ndata/\n├── dataset_name\n│   ├── scene1/\n│   │   ├── images\n│   │   │   ├── IMG_0.jpg\n│   │   │   ├── IMG_1.jpg\n│   │   │   ├── ...\n│   │   ├── sparse/\n│   │       └──0/\n│   ├── scene2/\n│   │   ├── images\n│   │   │   ├── IMG_0.jpg\n│   │   │   ├── IMG_1.jpg\n│   │   │   ├── ...\n│   │   ├── sparse/\n│   │       └──0/\n...\n```\n\n\n### Public Data\n\nThe BungeeNeRF dataset is available in [Google Drive](https://drive.google.com/file/d/1nBLcf9Jrr6sdxKa1Hbd47IArQQ_X8lww/view?usp=sharing)/[百度网盘[提取码:4whv]](https://pan.baidu.com/s/1AUYUJojhhICSKO2JrmOnCA). The MipNeRF360 scenes are provided by the paper author [here](https://jonbarron.info/mipnerf360/). And we test on scenes ```bicycle, bonsai, counter, garden, kitchen, room, stump```. The SfM data sets for Tanks\u0026Temples and Deep Blending are hosted by 3D-Gaussian-Splatting [here](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/input/tandt_db.zip). Download and uncompress them into the ```data/``` folder.\n\n### Custom Data\n\nFor custom data, you should process the image sequences with [Colmap](https://colmap.github.io/) to obtain the SfM points and camera poses. Then, place the results into ```data/``` folder.\n\n\n## Training\n\n### Training multiple scenes\n\nTo train multiple scenes in parallel, we provide batch training scripts: \n\n - Tanks\u0026Temples: ```train_tnt.sh```\n - MipNeRF360: ```train_mip360.sh```\n - BungeeNeRF: ```train_bungee.sh```\n - Deep Blending: ```train_db.sh```\n - Nerf Synthetic: base -\u003e```train_nerfsynthetic.sh```; with warmup-\u003e```train_nerfsynthetic_withwarmup.sh```\n\n run them with \n\n ```\nbash train_xxx.sh\n ```\n\n \u003e Notice 1: Make sure you have enough GPU cards and memories to run these scenes at the same time.\n\n \u003e Notice 2: Each process occupies many cpu cores, which may slow down the training process. Set ```torch.set_num_threads(32)``` accordingly in the ```train.py``` to alleviate it.\n\n### Training a single scene\n\nFor training a single scene, modify the path and configurations in ```single_train.sh``` accordingly and run it:\n\n```\nbash ./single_train.sh\n```\n\n- scene: scene name with a format of ```dataset_name/scene_name/``` or ```scene_name/```;\n- exp_name: user-defined experiment name;\n- gpu: specify the GPU id to run the code. '-1' denotes using the most idle GPU. \n- voxel_size: size for voxelizing the SfM points, smaller value denotes finer structure and higher overhead, '0' means using the median of each point's 1-NN distance as the voxel size.\n- update_init_factor: initial resolution for growing new anchors. A larger one will start placing new anchor in a coarser resolution.\n\n\u003e For these public datasets, the configurations of 'voxel_size' and 'update_init_factor' can refer to the above batch training script. \n\n\nThis script will store the log (with running-time code) into ```outputs/dataset_name/scene_name/exp_name/cur_time``` automatically.\n\n\n\n\n\n## Evaluation\n\nWe've integrated the rendering and metrics calculation process into the training code. So, when completing training, the ```rendering results```, ```fps``` and ```quality metrics``` will be printed automatically. And the rendering results will be save in the log dir. Mind that the ```fps``` is roughly estimated by \n\n```\ntorch.cuda.synchronize();t_start=time.time()\nrendering...\ntorch.cuda.synchronize();t_end=time.time()\n```\n\nwhich may differ somewhat from the original 3D-GS, but it does not affect the analysis.\n\nMeanwhile, we keep the manual rendering function with a similar usage of the counterpart in [3D-GS](https://github.com/graphdeco-inria/gaussian-splatting), one can run it by \n\n```\npython render.py -m \u003cpath to trained model\u003e # Generate renderings\npython metrics.py -m \u003cpath to trained model\u003e # Compute error metrics on renderings\n```\n\n## Viewer\n\nThe [viewer](https://github.com/city-super/Scaffold-GS/tree/main/SIBR_viewers) for Scaffold-GS is available now. \n\nRecommended dataset structure in the source path location:\n\n```\n\u003clocation\u003e\n|---sparse\n    |---0\n        |---cameras.bin\n        |---images.bin\n        |---points3D.bin\n```\n\nor\n\n```\n\u003clocation\u003e\n|---points3D.ply\n|---transforms.json\n```\n\nRecommended checkpoint  structure in the model path location:\n\n```\n\u003clocation\u003e\n|---point_cloud\n|   |---point_cloud.ply\n|   |---color_mlp.pt\n|   |---cov_mlp.pt\n|   |---opacity_mlp.pt\n(|   |---embedding_appearance.pt)\n|---cfg_args\n|---cameras.json\n(|---input.ply)\n```\n\n\n## Contact\n\n- Tao Lu: taolu@smail.nju.edu.cn\n- Mulin Yu: yumulin@pjlab.org.cn\n\n## Citation\n\nIf you find our work helpful, please consider citing:\n\n```bibtex\n@inproceedings{scaffoldgs,\n  title={Scaffold-gs: Structured 3d gaussians for view-adaptive rendering},\n  author={Lu, Tao and Yu, Mulin and Xu, Linning and Xiangli, Yuanbo and Wang, Limin and Lin, Dahua and Dai, Bo},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={20654--20664},\n  year={2024}\n}\n```\n\n## LICENSE\n\nPlease follow the LICENSE of [3D-GS](https://github.com/graphdeco-inria/gaussian-splatting).\n\n## Acknowledgement\n\nWe thank all authors from [3D-GS](https://github.com/graphdeco-inria/gaussian-splatting) for presenting such an excellent work.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcity-super%2FScaffold-GS","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcity-super%2FScaffold-GS","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcity-super%2FScaffold-GS/lists"}