{"id":13441209,"url":"https://github.com/spla-tam/SplaTAM","last_synced_at":"2025-03-20T11:37:45.143Z","repository":{"id":210803292,"uuid":"725762086","full_name":"spla-tam/SplaTAM","owner":"spla-tam","description":"SplaTAM: Splat, Track \u0026 Map 3D Gaussians for Dense RGB-D SLAM (CVPR 2024)","archived":false,"fork":false,"pushed_at":"2024-06-19T09:46:15.000Z","size":14649,"stargazers_count":1524,"open_issues_count":38,"forks_count":173,"subscribers_count":37,"default_branch":"main","last_synced_at":"2024-10-22T05:51:49.356Z","etag":null,"topics":["computer-vision","cvpr2024","gaussian-splatting","robotics","slam"],"latest_commit_sha":null,"homepage":"https://spla-tam.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spla-tam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-30T20:26:47.000Z","updated_at":"2024-10-22T04:01:53.000Z","dependencies_parsed_at":"2024-06-19T18:50:22.319Z","dependency_job_id":null,"html_url":"https://github.com/spla-tam/SplaTAM","commit_stats":null,"previous_names":["spla-tam/splatam"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spla-tam%2FSplaTAM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spla-tam%2FSplaTAM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spla-tam%2FSplaTAM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spla-tam%2FSplaTAM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spla-tam","download_url":"https://codeload.github.com/spla-tam/SplaTAM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221759943,"owners_count":16876322,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","cvpr2024","gaussian-splatting","robotics","slam"],"created_at":"2024-07-31T03:01:31.167Z","updated_at":"2024-10-28T01:30:21.217Z","avatar_url":"https://github.com/spla-tam.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003c!-- PROJECT LOGO --\u003e\n\n\u003cp align=\"center\"\u003e\n\n  \u003ch1 align=\"center\"\u003eSplaTAM: Splat, Track \u0026 Map 3D Gaussians for Dense RGB-D SLAM\u003c/h1\u003e\n  \u003ch3 align=\"center\"\u003eCVPR 2024\u003c/h3\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://nik-v9.github.io/\"\u003e\u003cstrong\u003eNikhil Keetha\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://jaykarhade.github.io/\"\u003e\u003cstrong\u003eJay Karhade\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://krrish94.github.io/\"\u003e\u003cstrong\u003eKrishna Murthy Jatavallabhula\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://gengshan-y.github.io/\"\u003e\u003cstrong\u003eGengshan Yang\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://theairlab.org/team/sebastian/\"\u003e\u003cstrong\u003eSebastian Scherer\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://www.cs.cmu.edu/~deva/\"\u003e\u003cstrong\u003eDeva Ramanan\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.vision.rwth-aachen.de/person/216/\"\u003e\u003cstrong\u003eJonathon Luiten\u003c/strong\u003e\u003c/a\u003e\n  \u003c/p\u003e\n  \u003ch3 align=\"center\"\u003e\u003ca href=\"https://arxiv.org/pdf/2312.02126.pdf\"\u003ePaper\u003c/a\u003e | \u003ca href=\"https://youtu.be/jWLI-OFp3qU\"\u003eVideo\u003c/a\u003e | \u003ca href=\"https://spla-tam.github.io/\"\u003eProject Page\u003c/a\u003e\u003c/h3\u003e\n  \u003cdiv align=\"center\"\u003e\u003c/div\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"\"\u003e\n    \u003cimg src=\"./assets/1.gif\" alt=\"Logo\" width=\"100%\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cbr\u003e\n\n## Stay Tuned for a Faster and Better Variant of SplaTAM! \n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails open=\"open\" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#demo\"\u003eOnline Demo\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#downloads\"\u003eDownloads\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#benchmarking\"\u003eBenchmarking\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#acknowledgement\"\u003eAcknowledgement\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#developers\"\u003eDevelopers\u003c/a\u003e\n    \u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## Installation\n\n##### (Recommended)\nSplaTAM has been benchmarked with Python 3.10, Torch 1.12.1 \u0026 CUDA=11.6. However, Torch 1.12 is not a hard requirement and the code has also been tested with other versions of Torch and CUDA such as Torch 2.3.0 \u0026 CUDA 12.1.\n\nThe simplest way to install all dependences is to use [anaconda](https://www.anaconda.com/) and [pip](https://pypi.org/project/pip/) in the following steps: \n\n```bash\nconda create -n splatam python=3.10\nconda activate splatam\nconda install -c \"nvidia/label/cuda-11.6.0\" cuda-toolkit\nconda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge\npip install -r requirements.txt\n```\n\n\u003c!-- Alternatively, we also provide a conda environment.yml file :\n```bash\nconda env create -f environment.yml\nconda activate splatam\n``` --\u003e\n\n#### Windows\n\nFor installation on Windows using Git bash, please refer to the [instructions shared in Issue#9](https://github.com/spla-tam/SplaTAM/issues/9#issuecomment-1848348403).\n\n#### Docker and Singularity Setup\n\nWe also provide a docker image. We recommend using a venv to run the code inside a docker image:\n\n\n```bash\ndocker pull nkeetha/splatam:v1\nbash bash_scripts/start_docker.bash\ncd /SplaTAM/\npip install virtualenv --user\nmkdir venv\ncd venv\nvirtualenv --system-site-packages splatam\nsource ./splatam/bin/activate\npip install -r venv_requirements.txt\n```\n\nSetting up a singularity container is similar:\n```bash\ncd \u003c/path/to/singularity/folder/\u003e\nsingularity pull splatam.sif docker://nkeetha/splatam:v1\nsingularity instance start --nv splatam.sif splatam\nsingularity run --nv instance://splatam\ncd \u003cpath/to/SplaTAM/\u003e\npip install virtualenv --user\nmkdir venv\ncd venv\nvirtualenv --system-site-packages splatam\nsource ./splatam/bin/activate\npip install -r venv_requirements.txt\n```\n\n## Demo\n\n### Online\n\nYou can SplaTAM your own environment with an iPhone or LiDAR-equipped Apple device by downloading and using the \u003ca href=\"https://apps.apple.com/au/app/nerfcapture/id6446518379\"\u003eNeRFCapture\u003c/a\u003e app.\n\nMake sure that your iPhone and PC are connected to the same WiFi network, and then run the following command:\n\n ```bash\nbash bash_scripts/online_demo.bash configs/iphone/online_demo.py\n```\n\nOn the app, keep clicking send for successive frames. Once the capturing of frames is done, the app will disconnect from the PC and check out SplaTAM's interactive rendering of the reconstruction on your PC! Here are some cool example results:\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"\"\u003e\n    \u003cimg src=\"./assets/collage.gif\" alt=\"Logo\" width=\"75%\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n### Offline\n\nYou can also first capture the dataset and then run SplaTAM offline on the dataset with the following command:\n\n```bash\nbash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.py\n```\n\n### Dataset Collection\n\nIf you would like to only capture your own iPhone dataset using the NeRFCapture app, please use the following command:\n\n```bash\nbash bash_scripts/nerfcapture2dataset.bash configs/iphone/dataset.py\n```\n\n## Usage\n\nWe will use the iPhone dataset as an example to show how to use SplaTAM. The following steps are similar for other datasets.\n\nTo run SplaTAM, please use the following command:\n\n```bash\npython scripts/splatam.py configs/iphone/splatam.py\n```\n\nTo visualize the final interactive SplaTAM reconstruction, please use the following command:\n\n```bash\npython viz_scripts/final_recon.py configs/iphone/splatam.py\n```\n\nTo visualize the SplaTAM reconstruction in an online fashion, please use the following command:\n\n```bash\npython viz_scripts/online_recon.py configs/iphone/splatam.py\n```\n\nTo export the splats to a .ply file, please use the following command:\n\n```bash\npython scripts/export_ply.py configs/iphone/splatam.py\n```\n\n`PLY` format Splats can be visualized in viewers such as [SuperSplat](https://playcanvas.com/supersplat/editor) \u0026 [PolyCam](https://poly.cam/tools/gaussian-splatting).\n\nTo run 3D Gaussian Splatting on the SplaTAM reconstruction, please use the following command:\n\n```bash\npython scripts/post_splatam_opt.py configs/iphone/post_splatam_opt.py\n```\n\nTo run 3D Gaussian Splatting on a dataset using ground truth poses, please use the following command:\n\n```bash\npython scripts/gaussian_splatting.py configs/iphone/gaussian_splatting.py\n```\n\n## Downloads\n\nDATAROOT is `./data` by default. Please change the `input_folder` path in the scene-specific config files if datasets are stored somewhere else on your machine.\n\n### Replica\n\nDownload the data as below, and the data is saved into the `./data/Replica` folder. Note that the Replica data is generated by the authors of iMAP (but hosted by the authors of NICE-SLAM). Please cite iMAP if you use the data.\n\n```bash\nbash bash_scripts/download_replica.sh\n```\n\n### TUM-RGBD\n\n```bash\nbash bash_scripts/download_tum.sh\n```\n\n### ScanNet\n\nPlease follow the data downloading procedure on the [ScanNet](http://www.scan-net.org/) website, and extract color/depth frames from the `.sens` file using this [code](https://github.com/ScanNet/ScanNet/blob/master/SensReader/python/reader.py).\n\n\u003cdetails\u003e\n  \u003csummary\u003e[Directory structure of ScanNet (click to expand)]\u003c/summary\u003e\n\n```\n  DATAROOT\n  └── scannet\n        └── scene0000_00\n            └── frames\n                ├── color\n                │   ├── 0.jpg\n                │   ├── 1.jpg\n                │   ├── ...\n                │   └── ...\n                ├── depth\n                │   ├── 0.png\n                │   ├── 1.png\n                │   ├── ...\n                │   └── ...\n                ├── intrinsic\n                └── pose\n                    ├── 0.txt\n                    ├── 1.txt\n                    ├── ...\n                    └── ...\n```\n\u003c/details\u003e\n\n\nWe use the following sequences: \n```\nscene0000_00\nscene0059_00\nscene0106_00\nscene0181_00\nscene0207_00\n```\n\n### ScanNet++\n\nPlease follow the data downloading and image undistortion procedure on the \u003ca href=\"https://kaldir.vc.in.tum.de/scannetpp/\"\u003eScanNet++\u003c/a\u003e website. \nAdditionally for undistorting the DSLR depth images, we use our \u003ca href=\"https://github.com/Nik-V9/scannetpp\"\u003eown variant of the official ScanNet++ processing code\u003c/a\u003e. We will open a pull request to the official ScanNet++ repository soon.\n\nWe use the following sequences: \n\n```\n8b5caf3398\nb20a261fdf\n```\n\nFor b20a261fdf, we use the first 360 frames, due to an abrupt jump/teleportation in the trajectory post frame 360. Please note that ScanNet++ was primarily intended as a NeRF Training \u0026 Novel View Synthesis dataset.\n\n### Replica-V2\n\nWe use the Replica-V2 dataset from vMAP to evaluate novel view synthesis. Please download the pre-generated replica sequences from \u003ca href=\"https://github.com/kxhit/vMAP\"\u003evMAP\u003c/a\u003e.\n\n## Benchmarking\n\nFor running SplaTAM, we recommend using [weights and biases](https://wandb.ai/) for the logging. This can be turned on by setting the `wandb` flag to True in the configs file. Also make sure to specify the path `wandb_folder`. If you don't have a wandb account, first create one. Please make sure to change the `entity` config to your wandb account. Each scene has a config folder, where the `input_folder` and `output` paths need to be specified. \n\nBelow, we show some example run commands for one scene from each dataset. After SLAM, the trajectory error will be evaluated along with the rendering metrics. The results will be saved to `./experiments` by default.\n\n### Replica\n\nTo run SplaTAM on the `room0` scene, run the following command:\n\n```bash\npython scripts/splatam.py configs/replica/splatam.py\n```\n\nTo run SplaTAM-S on the `room0` scene, run the following command:\n\n```bash\npython scripts/splatam.py configs/replica/splatam_s.py\n```\n\nFor other scenes, please modify the `configs/replica/splatam.py` file or use `configs/replica/replica.bash`.\n\n### TUM-RGBD\n\nTo run SplaTAM on the `freiburg1_desk` scene, run the following command:\n\n```bash\npython scripts/splatam.py configs/tum/splatam.py\n```\n\nFor other scenes, please modify the `configs/tum/splatam.py` file or use `configs/tum/tum.bash`.\n\n### ScanNet\n\nTo run SplaTAM on the `scene0000_00` scene, run the following command:\n\n```bash\npython scripts/splatam.py configs/scannet/splatam.py\n```\n\nFor other scenes, please modify the `configs/scannet/splatam.py` file or use `configs/scannet/scannet.bash`.\n\n### ScanNet++\n\nTo run SplaTAM on the `8b5caf3398` scene, run the following command:\n\n```bash\npython scripts/splatam.py configs/scannetpp/splatam.py\n```\n\nTo run Novel View Synthesis on the `8b5caf3398` scene, run the following command:\n\n```bash\npython scripts/eval_novel_view.py configs/scannetpp/eval_novel_view.py\n```\n\nFor other scenes, please modify the `configs/scannetpp/splatam.py` file or use `configs/scannetpp/scannetpp.bash`.\n\n### ReplicaV2\n\nTo run SplaTAM on the `room0` scene, run the following command:\n\n```bash\npython scripts/splatam.py configs/replica_v2/splatam.py\n```\n\nTo run Novel View Synthesis on the `room0` scene post SplaTAM, run the following command:\n\n```bash\npython scripts/eval_novel_view.py configs/replica_v2/eval_novel_view.py\n```\n\nFor other scenes, please modify the config files.\n\n## Acknowledgement\n\nWe thank the authors of the following repositories for their open-source code:\n\n- 3D Gaussians\n  - [Dynamic 3D Gaussians](https://github.com/JonathonLuiten/Dynamic3DGaussians)\n  - [3D Gaussian Splating](https://github.com/graphdeco-inria/gaussian-splatting)\n- Dataloaders\n  - [GradSLAM \u0026 ConceptFusion](https://github.com/gradslam/gradslam/tree/conceptfusion)\n- Baselines\n  - [Nice-SLAM](https://github.com/cvg/nice-slam)\n  - [Point-SLAM](https://github.com/eriksandstroem/Point-SLAM)\n\n## Citation\n\nIf you find our paper and code useful, please cite us:\n\n```bib\n@inproceedings{keetha2024splatam,\n        title={SplaTAM: Splat, Track \u0026 Map 3D Gaussians for Dense RGB-D SLAM},\n        author={Keetha, Nikhil and Karhade, Jay and Jatavallabhula, Krishna Murthy and Yang, Gengshan and Scherer, Sebastian and Ramanan, Deva and Luiten, Jonathon},\n        booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n        year={2024}\n      }\n```\n\n## Developers\n- [Nik-V9](https://github.com/Nik-V9) ([Nikhil Keetha](https://nik-v9.github.io/))\n- [JayKarhade](https://github.com/JayKarhade) ([Jay Karhade](https://jaykarhade.github.io/))\n- [JonathonLuiten](https://github.com/JonathonLuiten) ([Jonathan Luiten](https://www.vision.rwth-aachen.de/person/216/))\n- [krrish94](https://github.com/krrish94) ([Krishna Murthy Jatavallabhula](https://krrish94.github.io/))\n- [gengshan-y](https://github.com/gengshan-y) ([Gengshan Yang](https://gengshan-y.github.io/))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspla-tam%2FSplaTAM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspla-tam%2FSplaTAM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspla-tam%2FSplaTAM/lists"}