{"id":13441482,"url":"https://github.com/GAP-LAB-CUHK-SZ/SAMPro3D","last_synced_at":"2025-03-20T12:31:04.892Z","repository":{"id":209484506,"uuid":"724188919","full_name":"GAP-LAB-CUHK-SZ/SAMPro3D","owner":"GAP-LAB-CUHK-SZ","description":"SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation","archived":false,"fork":false,"pushed_at":"2024-01-12T15:26:11.000Z","size":774,"stargazers_count":87,"open_issues_count":0,"forks_count":7,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-08-01T03:34:43.482Z","etag":null,"topics":["3d-scene-understanding","segment-anything","zero-shot-segmentation"],"latest_commit_sha":null,"homepage":"https://mutianxu.github.io/sampro3d/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GAP-LAB-CUHK-SZ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-11-27T15:18:07.000Z","updated_at":"2024-07-18T10:19:21.000Z","dependencies_parsed_at":"2024-01-16T02:45:42.512Z","dependency_job_id":null,"html_url":"https://github.com/GAP-LAB-CUHK-SZ/SAMPro3D","commit_stats":null,"previous_names":["gap-lab-cuhk-sz/sampro3d"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GAP-LAB-CUHK-SZ%2FSAMPro3D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GAP-LAB-CUHK-SZ%2FSAMPro3D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GAP-LAB-CUHK-SZ%2FSAMPro3D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GAP-LAB-CUHK-SZ%2FSAMPro3D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GAP-LAB-CUHK-SZ","download_url":"https://codeload.github.com/GAP-LAB-CUHK-SZ/SAMPro3D/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221760010,"owners_count":16876334,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-scene-understanding","segment-anything","zero-shot-segmentation"],"created_at":"2024-07-31T03:01:34.544Z","updated_at":"2025-03-20T12:31:04.885Z","avatar_url":"https://github.com/GAP-LAB-CUHK-SZ.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003c!-- PROJECT LOGO --\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://mutianxu.github.io/sampro3d/static/images/icon_final.jpg\" alt=\"\" width=\"150\" height=\"50\"/\u003e\n  \u003ch1 align=\"center\"\u003eSAMPro3D: Locating SAM Prompts in 3D \n  \n  for Zero-Shot Scene Segmentation\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://mutianxu.github.io\"\u003e\u003cstrong\u003eMutian Xu\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003cstrong\u003eXingyilang Yin\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://lingtengqiu.github.io/\"\u003e\u003cstrong\u003eLingteng Qiu\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://xueyuhanlang.github.io/\"\u003e\u003cstrong\u003eYang Liu\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.microsoft.com/en-us/research/people/xtong/\"\u003e\u003cstrong\u003eXin Tong\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://gaplab.cuhk.edu.cn/\"\u003e\u003cstrong\u003eXiaoguang Han\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    SSE, CUHKSZ\n    ·\n    FNii, CUHKSZ\n    ·\n    Microsoft Research Asia\n\n  \u003c/p\u003e\n  \u003c!-- \u003ch2 align=\"center\"\u003eCVPR 2023\u003c/h2\u003e --\u003e\n  \u003ch3 align=\"center\"\u003e\u003ca href=\"https://arxiv.org/abs/2311.17707\"\u003ePaper\u003c/a\u003e | \u003ca href=\"https://mutianxu.github.io/sampro3d/\"\u003eProject Page\u003c/a\u003e\u003c/h3\u003e\n  \u003cdiv align=\"center\"\u003e\u003c/div\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"\"\u003e\n    \u003cimg src=\"https://mutianxu.github.io/sampro3d/static/images/teaser.jpg\" alt=\"Logo\" width=\"80%\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n*SAMPro3D* can segment **ANY** 😯😯😯 3D indoor scenes \u003cb\u003eWITHOUT\u003c/b\u003e training ❗️❗️❗️. It achieves higher quality and more diverse segmentation than previous zero-shot or fully supervised approaches, and in many cases even surpasses human-level annotations.\n\u003cbr\u003e\n\nIf you find our code or work helpful, please cite:\n```bibtex\n@article{xu2023sampro3d,\n        title={SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation}, \n        author={Mutian Xu and Xingyilang Yin and Lingteng Qiu and Yang Liu and Xin Tong and Xiaoguang Han},\n        year={2023},\n        journal = {arXiv preprint arXiv:2311.17707}\n  }\n```\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails open=\"open\" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#-news\"\u003eNews\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#requirements-and-installation\"\u003eRequirements and Installation\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#data-preparation\"\u003eData Preparation\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#run-sampro3d\"\u003eRun SAMPro3D\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#animated-qualitative-comparison\"\u003eAnimated Qualitative Comparison\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#segment-your-own-3d-scene\"\u003eSegment Your Own 3D Scene\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#todo\"\u003eTODO\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#contact\"\u003eContact\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#acknowledgement\"\u003eAcknowledgement\u003c/a\u003e\n    \u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## 📢 News\n- The initial code is released. :fire::fire::fire: (Dec.31, 2023 UTC)\n- The first major revision of code is out. Try the latest code! 💪💪💪 (Jan.2, 2024 UTC)\n\n\n## Requirements and Installation\n\n### Hardware requirements\nAt least 1 GPU to hold around 8000MB. Moreover, it is highly recommended to utilize both a CPU with ample processing power and a disk with fast I/O capabilities. Additionally, the disk needs to be large enough (about 50 MB for a 2D frame of resolution 240*320, totally around 160 GB for 2500 frames of a large-scale scene).\n\n### Software installation\nFollow the [installation instruction](installation.md) to install all required packages.\n\n## Data Preparation\n\nFollow the [data pre-processing instruction](dataset_preprocess/README.md) to download and preprocess data.\n\n## Run SAMPro3D\n\n### 3D Prompt Proposal\nThe initial stage of SAMPro3D involves generating a 3D prompt and executing SAM segmentation, followed by saving the SAM outputs for subsequent stages. To initiate this process, simply run:\n```\npython 3d_prompt_proposal.py --data_path /PATH_TO/ScanNet_data --scene_name sceneXXXX_XX --prompt_path /PATH_TO/initial_prompt --sam_output_path /PATH_TO/SAM_outputs --device cuda:0\n```\nThis stage will be the only step to perform SAM inference, accounting for the majority of computational time and memory usage within our entire pipeline.\n\n**Note** on time efficiency: This stage will save SAM outputs into `.npy` files for later use. Due to different hardware conditions (CPU and disk), the I/O speed of SAM output files may vary a lot and impact the running time of our pipeline. Please refer to the hardware recommendations mentioned before to prepare your hardware for the best efficiency.\n\n(Optional: *Partial-Area* Segmentation): At this stage, you can also perform 3D segmentation on partial point clouds captured by limited 2D frames, by simply changing the `frame_id_init` and `frame_id_end` at [here](https://github.com/GAP-LAB-CUHK-SZ/SAMPro3D/blob/main/3d_prompt_proposal.py#L169), then running the script. Sometimes this works better than segmenting the whole point clouds (thanks to less complicated scenes and better frame-consistency).\n\n### Finish Segmentation and Result Visualization\nNext, we will proceed with filtering and consolidating the initial prompts, leveraging the saved SAM outputs generated during the 3D Prompt Proposal phase, to obtain the final 3D segmentations. This can be realized by executing the following command:\n```\npython main.py --data_path /PATH_TO/ScanNet_data --scene_name sceneXXXX_XX --prompt_path /PATH_TO/initial_prompt --sam_output_path /PATH_TO/SAM_outputs --pred_path /PATH_TO/sampro3d_predictions --output_vis_path /PATH_TO/result_visualization --device cuda:0\n```\nAfter finishing this, the visualization result of the final 3D segmentation will be *automatically* 😊 saved as `sceneXXXX_XX.ply` file in the path specified by `--output_vis_path`.\n\n**Note** on post-processing of the floor: Using our framework, you can usually get a decent segmentation of the floor. However, for a large-scale floor, we use post-processing for perfect segmentation of the floor. For small-scale scenes (e.g., scene0050_00 in ScanNet), you can skip this step by simply adding `--args.post_floor False` to the previous command.\n\n### Time estimation ⚡️\nIf everything goes well, the entire pipeline will just take 15 min for a large-scale 3D scene captured by 2000 2D frames. (**WE DO NOT NEED TRAIN❗️**)\n\n## Animated Qualitative Comparison\n\nhttps://github.com/GAP-LAB-CUHK-SZ/SAMPro3D/assets/48080726/3a459c80-ac17-4750-a763-d477d33640bd\n\nhttps://github.com/GAP-LAB-CUHK-SZ/SAMPro3D/assets/48080726/ac7aaf2c-9223-4d0e-94c4-cf38413eba74\n\n## 🌟 Segment Your Own 3D Scene \nWith our advanced framework, you can generate high-quality segmentations on your own 3D scene without the need for training! Here are the steps you can follow:\n\n- Data preparation: Follow the tips mentioned in [data pre-processing instruction](dataset_preprocess/README.md) to prepare your data.\n- Familiarize yourself with the instructions: Read and understand the guidelines, and pay attention to any specific recommendations.\n- Run SAMPro3D: Execute the segmentation framework on your prepared data. You may also need to adjust some parameters such as *eps* at [here](https://github.com/GAP-LAB-CUHK-SZ/SAMPro3D/blob/main/utils/main_utils.py#L139) according to the code comments. \n- Monitor the segmentation process: Keep an eye on the segmentation process to ensure it is running smoothly. Depending on the size and complexity of your scene, the segmentation may take some time to complete.\n- Evaluate the segmentation output: Once the segmentation process is finished, assess the quality of the segmentation results. Check if the segments align with the desired objects or regions in your 3D scene. You may also compare the output to ground truth data or visually inspect the results for accuracy.\n- Refine if necessary: If the segmentation output requires improvement or refinement, consider adjusting the parameters or settings of SAMPro3D or applying post-processing techniques to enhance the segmentation quality.\n- Analyze and utilize the segmentation: Utilize the segmented output for your intended purposes, such as further analysis, visualization, or integration with other applications or systems.\n\n## 🚩 TODO\n- [ ]  Add the visualization code for showing the result of SAM3D, Mask3D and ScanNet200's annotations.\n- [ ]  Add the evaluation code for calculating segmentation mIoU.\n- [ ]  Add the code for incorporating [HQ-SAM](https://github.com/SysCV/sam-hq) and [Mobile-SAM](https://github.com/ChaoningZhang/MobileSAM) in our pipeline.\n- [ ]  Support the jupyter notebook for step-by-step running.\n- [ ]  Support in-website qualitative visualization.\n- [ ]  Support more datasets.\n\n## Contact\nYou are welcome to submit issues, send pull requests, or share some ideas with us. If you have any other questions, please contact Mutian Xu (mutianxu@link.cuhk.edu.cn).\n\n## Acknowledgement\nOur code base is partially borrowed or adapted from [SAM](https://github.com/facebookresearch/segment-anything), [OpenScene](https://github.com/pengsongyou/openscene) and [Pointcept](https://github.com/Pointcept).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGAP-LAB-CUHK-SZ%2FSAMPro3D","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FGAP-LAB-CUHK-SZ%2FSAMPro3D","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGAP-LAB-CUHK-SZ%2FSAMPro3D/lists"}