{"id":26333860,"url":"https://github.com/YOUNG-bit/open_semantic_slam","last_synced_at":"2025-03-16T00:02:35.978Z","repository":{"id":282004685,"uuid":"945232538","full_name":"YOUNG-bit/open_semantic_slam","owner":"YOUNG-bit","description":"ICRA2025: OpenGS-SLAM: Open-Set Dense Semantic SLAM with 3D Gaussian Splatting for Object-Level Scene Understanding","archived":false,"fork":false,"pushed_at":"2025-03-12T08:58:13.000Z","size":39286,"stargazers_count":107,"open_issues_count":1,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-12T09:39:24.056Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://young-bit.github.io/opengs-github.github.io/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YOUNG-bit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-09T00:30:41.000Z","updated_at":"2025-03-12T09:03:32.000Z","dependencies_parsed_at":"2025-03-12T09:49:31.197Z","dependency_job_id":null,"html_url":"https://github.com/YOUNG-bit/open_semantic_slam","commit_stats":null,"previous_names":["young-bit/open_semantic_slam"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YOUNG-bit%2Fopen_semantic_slam","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YOUNG-bit%2Fopen_semantic_slam/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YOUNG-bit%2Fopen_semantic_slam/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YOUNG-bit%2Fopen_semantic_slam/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YOUNG-bit","download_url":"https://codeload.github.com/YOUNG-bit/open_semantic_slam/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243806046,"owners_count":20350775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-16T00:01:35.695Z","updated_at":"2025-03-16T00:02:35.944Z","avatar_url":"https://github.com/YOUNG-bit.png","language":null,"funding_links":[],"categories":["3DGSNeRF"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e OpenGS-SLAM: Open-Set Dense Semantic SLAM with 3D Gaussian Splatting for Object-Level Scene Understanding \u003c/h1\u003e\n\n\u003ch3 align=\"center\"\u003e Dianyi Yang, Yu Gao, Xihan Wang, Yufeng Yue, Yi Yang∗, Mengyin Fu \u003c/h3\u003e\n\n\u003c!-- \u003ch3 align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2408.12677\"\u003ePaper\u003c/a\u003e | \u003ca href=\"https://youtu.be/rW8o_cRPZBg\"\u003eVideo\u003c/a\u003e | \u003ca href=\"https://gs-fusion.github.io/\"\u003eProject Page\u003c/a\u003e\n\u003c/h3\u003e --\u003e\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"https://www.youtube.com/watch?v=uNJ4vTpfGU0\"\u003eVideo\u003c/a\u003e | \u003ca href=\"https://young-bit.github.io/opengs-github.github.io/\"\u003eProject Page\u003c/a\u003e\n\u003c/h3\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"\"\u003e\n    \u003cimg src=\"./media/github.gif\" alt=\"teaser\" width=\"100%\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e All the reported results are obtained from a single Nvidia RTX 4090 GPU. \u003c/p\u003e\n\nAbstract: *Recent advancements in 3D Gaussian Splatting have significantly improved the efficiency and quality of dense semantic SLAM. However, previous methods are generally constrained by limited-category pre-trained classifiers and implicit semantic representation, which hinder their performance in open-set scenarios and restrict 3D object-level scene understanding. To address these issues, we propose OpenGSSLAM, an innovative framework that utilizes 3D Gaussian representation to perform dense semantic SLAM in open-set environments. Our system integrates explicit semantic labels derived from 2D foundational models into the 3D Gaussian framework, facilitating robust 3D object-level scene understanding. We introduce Gaussian Voting Splatting to enable fast 2D label map rendering and scene updating. Additionally, we propose a Confidence-based 2D Label Consensus method to ensure consistent labeling across multiple views. Furthermore, we employ a Segmentation Counter Pruning strategy to improve the accuracy of semantic scene representation. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of our method in scene understanding, tracking, and mapping, achieving 10× faster semantic rendering and 2× lower storage costs compared to existing methods.*\n\n\n\n## Environments\nInstall requirements\n```bash\nconda create -n opengsslam python==3.9\nconda activate opengsslam\nconda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia\npip install -r requirements.txt\n```\nInstall submodules\n\n```bash\nconda activate opengsslam\npip install submodules/diff-gaussian-rasterization\npip install submodules/simple-knn\n```\n\n## Scene Interaction Demo\n### 1. Download our pre-constructed Semantic 3D Gaussian scenes for the Replica dataset from the following link: [Driver](https://drive.google.com/drive/folders/1-bGoaZQRRKLHXFQGq3_6gu1KXhoePbQv?usp=drive_link) \n\n### 2. Scene Interaction\n```\npython ./final_vis.py --scene_npz [download_path]/room1.npz\n```\nHere, users can click on any object in the scene to interact with it and use our Gaussian Voting method for real-time semantic rendering. Note that we use the **pynput** library to capture mouse clicks, which retrieves the click position on **the entire screen**. To map this position to the display window, we subtract an offset `(x_off, y_off)`, representing the window’s top-left corner on the screen. All tests were conducted on an Ubuntu system with a 2K resolution.\n\n### *Key Press Description*\n\n- **T**: Toggle between color and label display modes.  \n- **J**: Toggle between showing all objects or a single object.  \n- **K**: Capture the current view.  \n- **A**: Translate the object along the x-axis by +0.01.  \n- **S**: Translate the object along the y-axis by +0.01.  \n- **D**: Translate the object along the z-axis by +0.01.  \n- **Z**: Translate the object along the x-axis by -0.01.  \n- **X**: Translate the object along the y-axis by -0.01.  \n- **C**: Translate the object along the z-axis by -0.01.  \n- **F**: Rotate the object around the x-axis by +1 degree.  \n- **G**: Rotate the object around the y-axis by +1 degree.  \n- **H**: Rotate the object around the z-axis by +1 degree.  \n- **V**: Rotate the object around the x-axis by -1 degree.  \n- **B**: Rotate the object around the y-axis by -1 degree.  \n- **N**: Rotate the object around the z-axis by -1 degree.  \n- **O**: Output the current camera view matrix.  \n- **M**: Switch to the next mapping camera view.  \n- **L**: Increase the scale of all Gaussians.  \n- **P**: Downsample Gaussians using a voxel grid.  \n\n\n## SLAM Source Code\n\nComing soon!\n\n\u003c!-- ## Note\n\nThis repository contains the code used in the paper \"OpenGS-SLAM: Open-Set Dense Semantic SLAM with 3D Gaussian Splatting for Object-Level Scene Understanding\". The full code will be released upon acceptance of the paper. --\u003e\n\n## Acknowledgement\nWe sincerely thank the developers and contributors of the many open-source projects that our code is built upon.\n\n* [GS_ICP_SLAM](https://github.com/Lab-of-AI-and-Robotics/GS_ICP_SLAM)\n* [SplaTAM](https://github.com/spla-tam/SplaTAM/tree/main)\n\n## Citation\n\nIf you find our paper and code useful, please cite us:\n```bibtex\n@article{yang2025opengs,\n  title={OpenGS-SLAM: Open-Set Dense Semantic SLAM with 3D Gaussian Splatting for Object-Level Scene Understanding},\n  author={Yang, Dianyi and Gao, Yu and Wang, Xihan and Yue, Yufeng and Yang, Yi and Fu, Mengyin},\n  journal={arXiv preprint arXiv:2503.01646},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYOUNG-bit%2Fopen_semantic_slam","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FYOUNG-bit%2Fopen_semantic_slam","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYOUNG-bit%2Fopen_semantic_slam/lists"}