{"id":13443521,"url":"https://github.com/thangvubk/SoftGroup","last_synced_at":"2025-03-20T16:31:42.841Z","repository":{"id":38363650,"uuid":"465564660","full_name":"thangvubk/SoftGroup","owner":"thangvubk","description":"[CVPR 2022 Oral] SoftGroup for Instance Segmentation on 3D Point Clouds","archived":false,"fork":false,"pushed_at":"2024-01-22T03:29:57.000Z","size":63597,"stargazers_count":341,"open_issues_count":92,"forks_count":80,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-08-01T03:43:53.001Z","etag":null,"topics":["3d-instance-segmentation","3d-object-detection","panoptic-segmentation","point-cloud","s3dis","scannet","semantic-kitti"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thangvubk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-03-03T04:15:39.000Z","updated_at":"2024-07-26T03:38:19.000Z","dependencies_parsed_at":"2024-01-18T14:44:14.130Z","dependency_job_id":"f34e6e30-b651-450b-942d-fcb2752417cb","html_url":"https://github.com/thangvubk/SoftGroup","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thangvubk%2FSoftGroup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thangvubk%2FSoftGroup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thangvubk%2FSoftGroup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thangvubk%2FSoftGroup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thangvubk","download_url":"https://codeload.github.com/thangvubk/SoftGroup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221780032,"owners_count":16879040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-instance-segmentation","3d-object-detection","panoptic-segmentation","point-cloud","s3dis","scannet","semantic-kitti"],"created_at":"2024-07-31T03:02:02.812Z","updated_at":"2024-10-28T04:31:13.992Z","avatar_url":"https://github.com/thangvubk.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# SoftGroup\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/softgroup-for-3d-instance-segmentation-on/3d-instance-segmentation-on-scannetv2)](https://paperswithcode.com/sota/3d-instance-segmentation-on-scannetv2?p=softgroup-for-3d-instance-segmentation-on) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/softgroup-for-3d-instance-segmentation-on/3d-instance-segmentation-on-s3dis)](https://paperswithcode.com/sota/3d-instance-segmentation-on-s3dis?p=softgroup-for-3d-instance-segmentation-on) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/softgroup-for-3d-instance-segmentation-on/3d-object-detection-on-scannetv2)](https://paperswithcode.com/sota/3d-object-detection-on-scannetv2?p=softgroup-for-3d-instance-segmentation-on)\n![Architecture](./docs/architecture.png)\n\nWe provide code for reproducing results of two papers \n\n[**SoftGroup for 3D Instance Segmentation on Point Clouds**](https://arxiv.org/abs/2203.01509)\\\nThang Vu, Kookhoi Kim, Tung M. Luu, Thanh Nguyen, and Chang D. Yoo.\\\n**CVPR 2022 (Oral)**.\n\n[**Scalable SoftGroup for 3D Instance Segmentation on Point Clouds**](https://arxiv.org/abs/2209.08263)\\\nThang Vu, Kookhoi Kim, Tung M. Luu, Thanh Nguyen, Junyeong Kim, and Chang D. Yoo.\\\n**TPAMI 2023 (accepted)**.\n\n## Update\n- 25/Nov/2022: Support [SoftGroup++](https://arxiv.org/abs/2209.08263).\n- 12/Sep/2022: Support panoptic segmentation on SemanticKITTI dataset.\n- 28/Jun/2022: Support STPLS3D dataset. Add custom dataset guideline.\n- 16/Apr/2022: The code base is refactored. Coding is more extendable, readable, and consistent. The following features are supported:\n  - Support up-to-date pytorch 1.11 and spconv 2.1.\n  - Support distributed and mix precision training. Training time on ScanNet v2 (on 4GPUs) reduces from 4 day to 10 hours.\n  - Faster inference speed, which requires only 288 ms per ScanNet scan on single Titan X.\n\n## Introduction\n\nExisting state-of-the-art 3D instance segmentation methods perform semantic segmentation followed by grouping. The hard predictions are made when performing semantic segmentation such that each point is associated with a single class. However, the errors stemming from hard decision propagate into grouping that results in (1) low overlaps between the predicted instance with the ground truth and (2) substantial false positives. To address the aforementioned problems, this paper proposes a 3D instance segmentation method referred to as SoftGroup by performing bottom-up soft grouping followed by top-down refinement. SoftGroup allows each point to be associated with multiple classes to mitigate the problems stemming from semantic prediction errors and suppresses false positive instances by learning to categorize them as background. Experimental results on different datasets and multiple evaluation metrics demonstrate the efficacy of SoftGroup. Its performance surpasses the strongest prior method by a significant margin of +6.2% on the ScanNet v2 hidden test set and +6.8% on S3DIS Area 5 of AP_50.\n\n![Learderboard](./docs/leaderboard.png)\n\n## Feature\n* State of the art performance on the [ScanNet benchmark](http://kaldir.vc.in.tum.de/scannet_benchmark/semantic_instance_3d) and S3DIS dataset (3/Mar/2022).\n* High speed of 345 ms per scan on ScanNet dataset, which is comparable with the existing fastest methods ([HAIS](https://github.com/hustvl/HAIS)). Our refactored implementation (this code) further reduce the inference time to 288 ms per scan.\n* Support multiple datasets: ScanNet, S3DIS, STPLS3D, SemanticKITTI.\n\n## Installation\nPlease refer to [installation guide](docs/installation.md).\n\n## Data Preparation\nPlease refer to [data preparation](dataset/README.md).\n\n## Pretrained models\n\n### Instance segmentation\n\n|   Dataset  |   Model     |   AP  | AP_50 | AP_25 |                                           Download                                         |\n|:----------:|:-----------:|:----:|:-----:|:-----:|:-------------------------------------------------------------------------------------------:|\n|    S3DIS   | SoftGroup   | 51.4 |  66.5 |  75.4 | [model](https://drive.google.com/file/d/1-f7I6-eIma4OilBON928N6mVcYbhiUFP/view?usp=sharing) |\n|    S3DIS   | SoftGroup++ | 50.9 |  67.8 |  76.0 | [model](https://drive.google.com/file/d/1OLbC8lmWkAQbqYAjiFj84egLQmJr-PmQ/view?usp=sharing) |\n| ScanNet v2 | SoftGroup   | 45.8 |  67.4 |  79.1 | [model](https://drive.google.com/file/d/1XUNRfred9QAEUY__VdmSgZxGQ7peG5ms/view?usp=sharing) |\n| ScanNet v2 | SoftGroup++ | 45.9 |  67.9 |  79.4 | above |\n|  STPLS3D   | SoftGroup   | 47.3 |  63.1 |  71.4 | [model](https://drive.google.com/file/d/1xCkKLTCYtQmSjXYH_sSg21M_6dcAskd8/view?usp=sharing) |\n|  STPLS3D   | SoftGroup++ | 46.5 |  62.9 |  71.8 | above |\n\n\u003e **_NOTE:_**  SoftGroup and SoftGroup++ use can use same trained model for inference on ScanNet v2 and STPLS3D.\n\n### Panoptic segmentation\n\n|    Dataset    |  PQ  | Config | Model |\n|:-------------:|:----:|:------:|:-----:|\n| SemanticKITTI | 60.2 | [config](https://github.com/thangvubk/SoftGroup/blob/main/configs/softgroup_kitti.yaml) | [model](https://drive.google.com/file/d/10Ln-xLfl8Z3DX3G3lnO_RruJtYUYDfI7/view?usp=sharing)     |\n\n## Training\nWe use the checkpoint of [HAIS](https://github.com/hustvl/HAIS) as pretrained backbone. **We have already converted the checkpoint to work on ``spconv2.x``**. Download the pretrained HAIS-spconv2 model and put it in ``SoftGroup/`` directory.\n\nConverted hais checkpoint: [model](https://drive.google.com/file/d/1FABsCUnxfO_VlItAzDYAwurdfcdK-scs/view?usp=sharing)\n\nNoted that for fair comparison with implementation in STPLS3D paper, we train SoftGroup on this dataset from scratch without pretrained backbone.\n### Training S3DIS dataset\nThe default configs suppose training on 4 GPU. If you use smaller number of GPUs, you should reduce the learning rate linearly. \n\nFirst, finetune the pretrained HAIS point-wise prediction network (backbone) on S3DIS.\n```\n./tools/dist_train.sh configs/softgroup_s3dis_backbone_fold5.yaml 4\n```\nThen, train model from frozen backbone.\n```\n./tools/dist_train.sh configs/softgroup_s3dis_fold5.yaml 4\n```\n\n### Training ScanNet V2 dataset\nTraining on ScanNet doesnot require finetuning the backbone. Just freeze pretrained backbone and train the model.\n```\n./tools/dist_train.sh configs/softgroup_scannet.yaml 4\n```\n\n### Training STPLS3D dataset\n```\n./tools/dist_train.sh configs/softgroup_stpls3d_backbone.yaml 4\n./tools/dist_train.sh configs/softgroup_stpls3d.yaml 4\n```\n\n## Inference\n```\n./tools/dist_test.sh $CONFIG_FILE $CHECKPOINT $NUM_GPU\n```\n\n### Inference without label\nFor example, on scannet test split, just change [``prefix``](https://github.com/thangvubk/SoftGroup/blob/cf88d9be41ae83a70f9100856a3ca15ee4ddcee9/configs/softgroup_scannet.yaml#L49) to ``test`` and [``with_label``](https://github.com/thangvubk/SoftGroup/blob/cf88d9be41ae83a70f9100856a3ca15ee4ddcee9/configs/softgroup_scannet.yaml#L52) to ``False`` before running inference. \n\n### Bounding box evaluation of ScanNet V2 dataset.\nWe provide script to evaluate detection performance on axis-aligned boxes from predicted/ground-truth instance.\n- Step 1: Change ``save_instance`` to ``True`` in [config file](https://github.com/thangvubk/SoftGroup/blob/99ffb9756e553e0edfb2c43e2ab6a6f646892bb5/config/softgroup_default_scannet.yaml#L72).\n- Step 2: Run evaluation code.\n```\nCUDA_VISIBLE_DEVICES=0 python test.py --config config/softgroup_default_scannet.yaml --pretrain $PATH_TO_PRETRAIN_MODEL$\n```\n- Step 3: Evaluate detection performance.\n```\npython eval_det.py\n```\n\n## Visualization\nPlease refer to [visualization guide](docs/visualization.md) for visualizing ScanNet and S3DIS results.\n\n## Custom dataset\nPlease refer to [custom dataset guide](docs/custom_dataset.md).\n\n## Citation\nIf you find our work helpful for your research. Please consider citing our paper.\n\n```\n@inproceedings{vu2022softgroup,\n  title={SoftGroup for 3D Instance Segmentation on 3D Point Clouds},\n  author={Vu, Thang and Kim, Kookhoi and Luu, Tung M. and Nguyen, Xuan Thanh and Yoo, Chang D.},\n  booktitle={CVPR},\n  year={2022}\n}\n```\n## Acknowledgements\nCode is built based on [HAIS](https://github.com/hustvl/HAIS), [PointGroup](https://github.com/dvlab-research/PointGroup), and [spconv](https://github.com/traveller59/spconv)\n\nThis work was partly supported by Institute for Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (2021-0-01381, Development of Causal AI through Video Understanding, and partly supported by Institute of Information \\\u0026 Communications Technology Planning \\\u0026 Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01371, Development of brain-inspired AI with human-like intelligence).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthangvubk%2FSoftGroup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthangvubk%2FSoftGroup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthangvubk%2FSoftGroup/lists"}