{"id":13441969,"url":"https://github.com/Pointcept/SegmentAnything3D","last_synced_at":"2025-03-20T13:31:39.247Z","repository":{"id":156258069,"uuid":"628918235","full_name":"Pointcept/SegmentAnything3D","owner":"Pointcept","description":"[ICCV'23 Workshop] SAM3D: Segment Anything in 3D Scenes","archived":false,"fork":false,"pushed_at":"2024-04-21T14:57:50.000Z","size":12785,"stargazers_count":1087,"open_issues_count":22,"forks_count":71,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-03-18T15:18:24.073Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2306.03908","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Pointcept.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-17T08:49:23.000Z","updated_at":"2025-03-17T14:12:28.000Z","dependencies_parsed_at":"2024-10-28T03:23:39.268Z","dependency_job_id":"3931b505-f407-440f-b194-f7ed60b42e9d","html_url":"https://github.com/Pointcept/SegmentAnything3D","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pointcept%2FSegmentAnything3D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pointcept%2FSegmentAnything3D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pointcept%2FSegmentAnything3D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pointcept%2FSegmentAnything3D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Pointcept","download_url":"https://codeload.github.com/Pointcept/SegmentAnything3D/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244619148,"owners_count":20482369,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:01:40.190Z","updated_at":"2025-03-20T13:31:39.239Z","avatar_url":"https://github.com/Pointcept.png","language":"Python","funding_links":[],"categories":["Python","Paper List","Uncategorized"],"sub_categories":["Follow-up Papers","Uncategorized"],"readme":"# Segment Anything 3D\n- Authors: [Yunhan Yang](https://yhyang-myron.github.io), [Xiaoyang Wu](https://xywu.me), [Tong He](https://tonghe90.github.io), [Hengshuang Zhao](https://hszhao.github.io), [Xihui Liu](https://xh-liu.github.io)\n- Institutes: Shanghai Artificial Intelligence Lab, The University of Hong Kong\n- Technical Report: [\\[arxiv\\]](https://arxiv.org/abs/2306.03908)\n\nWe extend [Segment Anything](https://github.com/facebookresearch/segment-anything) to 3D perception by transferring the segmentation information of 2D images to 3D space. We expect that the segment information can be helpful to 3D traditional perception and the open world perception. This project is still in progress, and it will be embedded into our perception codebase [Pointcept](https://github.com/Pointcept/Pointcept). We very much welcome any issue or pull request.\n\n## Result\n![](./docs/0.png)\nExample mesh is available [here](./example_mesh/).\n\n## Installation\n```\nconda create -n sam3d python=3.8 -y\nconda activate sam3d\n# Choose version you want here: https://pytorch.org/get-started/previous-versions/\nconda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch\nconda install plyfile -c conda-forge -y\npip install scikit-image opencv-python open3d imageio\npip install git+https://github.com/facebookresearch/segment-anything.git \n\ncd libs/pointops\n# usual\npython setup.py install\n# docker \u0026 multi GPU arch\nTORCH_CUDA_ARCH_LIST=\"ARCH LIST\" python  setup.py install\n# e.g. 7.5: RTX 3000; 8.0: a100 More available in: https://developer.nvidia.com/cuda-gpus\nTORCH_CUDA_ARCH_LIST=\"7.5 8.0\" python  setup.py install\ncd ../..\n```\n\n## Data Preparation\n### ScanNet v2\nDownload the [ScanNet](http://www.scan-net.org/) v2 dataset.\\\nRun preprocessing code for raw ScanNet as follows:\n- Prepare PointCloud data (follow [Pointcept](https://github.com/Pointcept/Pointcept))\n```\n# RAW_SCANNET_DIR: the directory of downloaded ScanNet v2 raw dataset.\n# PROCESSED_SCANNET_DIR: the directory of processed ScanNet dataset (output dir).\npython scannet-preprocess/preprocess_scannet.py --dataset_root ${RAW_SCANNET_DIR} --output_root ${PROCESSED_SCANNET_DIR}\n```\n- Prepare RGBD data (follow [BPNet](https://github.com/wbhu/BPNet))\n```\npython scannet-preprocess/prepare_2d_data/prepare_2d_data.py --scannet_path data/scannetv2 --output_path data/scannetv2_images --export_label_images\n```\n\n## Getting Started\nPlease try it via [sam3d.py](./sam3d.py)\n```\n# RGB_PATH: the path of rgb data\n# DATA_PATH: the path of pointcload data\n# SAVE_PATH: Where to save the pcd results\n# SAVE_2DMASK_PATH: Where to save 2D segmentation result from SAM\n# SAM_CHECKPOINT_PATH: the path of checkpoint for SAM\n\npython sam3d.py --rgb_path $RGB_PATH --data_path $DATA_PATH --save_path $SAVE_PATH --save_2dmask_path $SAVE_2DMASK_PATH --sam_checkpoint_path $SAM_CHECKPOINT_PATH \n```\n\n## Pipeline\nOur SAM3D pipeline looks as follows:\n\n1. **SAM Generate Masks**\\\nUse SAM to get the segmentation masks on 2D frames and then map them into the 3D space via depth information.\n\u003cdiv align=center\u003e\u003cimg src=\"./docs/1.png\" alt=\"Image\" width=\"80%\"\u003e\u003c/div\u003e\n\n2. **Merge Two Adjacent Pointclouds**\\\nUse \"Bidirectional-group-overlap-algorithm\" (modified from [\nContrastiveSceneContexts](https://github.com/facebookresearch/ContrastiveSceneContexts)) to merge two adjacent pointclouds.\n\u003cdiv align=center\u003e\u003cimg src=\"./docs/2.png\" alt=\"Image\" width=\"70%\"\u003e\u003c/div\u003e\n\n3. **Region Merging Method**\\\nMerge the entire pointcloud by region merging method.\n\u003cdiv align=center\u003e\u003cimg src=\"./docs/3.png\" alt=\"Image\" width=\"80%\"\u003e\u003c/div\u003e\n\n4. **Merge 2 Segmentation Results**\\\nWe apply Felzenswalb and Huttenlocher's Graph Based Image Segmentation algorithm to the scenes using the default parameters. Please refer to the [original repository](https://github.com/ScanNet/ScanNet/tree/master/Segmentator) for details. Then merge the 2 segmentation results to get the final result (merging code is in sam3d.py/pcd_ensemble).\n\u003cdiv align=center\u003e\u003cimg src=\"./docs/4.png\" alt=\"Image\" width=\"70%\"\u003e\u003c/div\u003e\n\n## Citation\nIf you find _SAM3D_ useful to your research, please cite our work:\n```\n@misc{yang2023sam3d,\n      title={SAM3D: Segment Anything in 3D Scenes}, \n      author={Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao and Xihui Liu},\n      year={2023},\n      eprint={2306.03908},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n## Acknowledgements\nSAM3D is inspirited by the following repos: [Segment Anything](https://github.com/facebookresearch/segment-anything), [Pointcept](https://github.com/Pointcept/Pointcept), [BPNet](https://github.com/wbhu/BPNet), [ContrastiveSceneContexts](https://github.com/facebookresearch/ContrastiveSceneContexts).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPointcept%2FSegmentAnything3D","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPointcept%2FSegmentAnything3D","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPointcept%2FSegmentAnything3D/lists"}