{"id":13443086,"url":"https://github.com/cardwing/Codes-for-PVKD","last_synced_at":"2025-03-20T15:33:30.059Z","repository":{"id":37624824,"uuid":"497782651","full_name":"cardwing/Codes-for-PVKD","owner":"cardwing","description":"Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation (CVPR 2022)","archived":false,"fork":false,"pushed_at":"2024-05-31T08:22:02.000Z","size":145,"stargazers_count":479,"open_issues_count":20,"forks_count":114,"subscribers_count":28,"default_branch":"main","last_synced_at":"2025-03-14T00:01:48.327Z","etag":null,"topics":["cvpr2022","lidar-segmentation","nuscenes","point-cloud-segmentation","pytorch","range-image","semantickitti","waymo-dataset"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cardwing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-30T04:02:44.000Z","updated_at":"2025-03-07T08:39:01.000Z","dependencies_parsed_at":"2024-10-28T04:53:16.575Z","dependency_job_id":"5ad45d98-2897-4990-8a94-afa98483a27d","html_url":"https://github.com/cardwing/Codes-for-PVKD","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cardwing%2FCodes-for-PVKD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cardwing%2FCodes-for-PVKD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cardwing%2FCodes-for-PVKD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cardwing%2FCodes-for-PVKD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cardwing","download_url":"https://codeload.github.com/cardwing/Codes-for-PVKD/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244640434,"owners_count":20486040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2022","lidar-segmentation","nuscenes","point-cloud-segmentation","pytorch","range-image","semantickitti","waymo-dataset"],"created_at":"2024-07-31T03:01:55.853Z","updated_at":"2025-03-20T15:33:29.684Z","avatar_url":"https://github.com/cardwing.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation (CVPR 2022)\n\nOur model achieves state-of-the-art performance on three challenges, i.e., ranks **1st** in [Waymo 3D Semantic Segmentation Challenge](https://waymo.com/open/challenges/2022/3d-semantic-segmentation/) (the \"Cylinder3D\" and \"Offboard_SemSeg\" entries, May 2022), ranks **1st** in [SemanticKITTI LiDAR Semantic Segmentation Challenge](https://competitions.codalab.org/competitions/20331#results) (single-scan, the \"Point-Voxel-KD\" entry, Jun 2022), ranks **2nd** in [SemanticKITTI LiDAR Semantic Segmentation Challenge](https://competitions.codalab.org/competitions/20331#results) (multi-scan, the \"PVKD\" entry, Dec 2021). Do not hesitate to use our trained models!\n\n## News\n\n- **2022-11** [NEW:fire:] Some useful training tips have been provided.\n\n- **2022-11** The distillation codes and some training tips will be released after CVPR DDL.\n\n- **2022-7** We provide a trained model of [CENet](https://github.com/huixiancheng/CENet), a range-image-based LiDAR segmentation method. The reproduced performance is much higher than the reported value! \n\n- **2022-6** Our method ranks **1st** in [SemanticKITTI LiDAR Semantic Segmentation Challenge](https://competitions.codalab.org/competitions/20331#results) (single-scan, the \"Point-Voxel-KD\" entity)\n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"./img/semantickitti_single_scan.PNG\" width=\"30%\"\u003e \n\u003c/p\u003e\n\n- **2022-5** Our method ranks **1st** in [Waymo 3D Semantic Segmentation Challenge](https://waymo.com/open/challenges/2022/3d-semantic-segmentation/) (the \"Cylinder3D\" and \"Offboard_SemSeg\" entities)\n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"./img/waymo.PNG\" width=\"30%\"\u003e \n\u003c/p\u003e\n\n## Installation\n\n### Requirements\n- PyTorch \u003e= 1.2 \n- yaml\n- tqdm\n- numba\n- Cython\n- [torch-scatter](https://github.com/rusty1s/pytorch_scatter)\n- [nuScenes-devkit](https://github.com/nutonomy/nuscenes-devkit) (optional for nuScenes)\n- [spconv](https://github.com/traveller59/spconv) (tested with spconv==1.2.1 and cuda==10.2)\n\n## Data Preparation\n\n### SemanticKITTI\n```\n./\n├── \n├── ...\n└── path_to_data_shown_in_config/\n    ├──sequences\n        ├── 00/           \n        │   ├── velodyne/\t\n        |   |\t├── 000000.bin\n        |   |\t├── 000001.bin\n        |   |\t└── ...\n        │   └── labels/ \n        |       ├── 000000.label\n        |       ├── 000001.label\n        |       └── ...\n        ├── 08/ # for validation\n        ├── 11/ # 11-21 for testing\n        └── 21/\n\t    └── ...\n```\n\n### nuScenes\n```\n./\n├── \n├── ...\n└── path_to_data_shown_in_config/\n\t\t├──v1.0-trainval\n\t\t├──v1.0-test\n\t\t├──samples\n\t\t├──sweeps\n\t\t├──maps\n\n```\n\n### Waymo\n```\n./\n├── \n├── ...\n└── path_to_data_shown_in_config/\n\t\t├──first_return\n\t\t├──second_return\n\n```\n\n## Test\nWe take evaluation on the SemanticKITTI test set (single-scan) as example.\n\n1. Download the [pre-trained models](https://drive.google.com/drive/folders/1LyWhVCqMzSVDe44c8ARDp8b94w1ct-tR?usp=sharing) and put them in `./model_load_dir`.\n\n2. Generate predictions on the SemanticKITTI test set.\n\n```\nCUDA_VISIBLE_DEVICES=0 python -u test_cyl_sem_tta.py\n```\n\nWe perform test-time augmentation to boost the performance. The model predictions will be saved in `./out_cyl/test` by default.\n\n\n3. Convert label number back to the original dataset format before submitting:\n```\npython remap_semantic_labels.py -p out_cyl/test -s test --inverse\ncd out_cyl/test\nzip -r out_cyl.zip sequences/\n```\n\n4. Upload out_cyl.zip to the [SemanticKITTI online server](https://competitions.codalab.org/competitions/20331#participate).\n\n## Train\n\n```\nCUDA_VISIBLE_DEVICES=0 python -u train_cyl_sem.py\n```\n\nRemember to change the `imageset` of `val_data_loader` to `val`, `return_test` of `dataset_params` to `False` in `semantickitti.yaml`. Currently, we only support vanilla training.\n\n\n## Useful Training Tips\n1. Finetuning.\n\nYou can finetune the model using both train and val sets as well as a smaller learning rate (1/3 or 1/4 of the original learning rate).\n\n2. Model ensemble. \n\nYou can use models of different epochs as an ensemble. Different models can also be taken as an ensemble, e.g., SPVCNN and Cylinder3D.\n\n3. Semi-supervised learning. \n\nYou can follow [GuidedContrast](https://arxiv.org/abs/2110.08188) to use pseudo labels of the test set to complement the original training set. (**DO NOT** use it in the supervised training. It can only be used in the semi-supervised setting to prove the value of the proposed semi-supervised algorithm.)\n\n4. More data augmentations. \n\nYou can use [LaserMix](https://arxiv.org/abs/2207.00026), [Instance Augmentation](https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/dataloader/instance_augmentation.py) and [PolarMix](https://arxiv.org/abs/2208.00223) to increase the diversity of training samples.\n\n5. Knowledge distillation (KD).\n\nYou can refer to [CRD](https://github.com/HobbitLong/RepDistiller) to apply KD to boost the performance of LiDAR segmentation models. We will release a more efficient and effective version of the PVKD algorithm soon.\n\n6. Using more inputs.\n\nIn addition to the (x, y, z), you can also use the intensity, range, azimuth, inclination and elongation as additional inputs. Remember to normalize these input signals if necessary. Tanh function is a good normalizer in some cases.\n\n7. Increasing the model size.\n\nYou can either increase the width (more channels) or the depth (more layers) of the model to boost the performance.\n\n8. Test time augmentation (TTA).\n\nYou can use more augmentations (flipping, rotation, scaling, translation) in TTA to boost the performance. A proper combination of them is vital to the final performance.\n\n## Performance\n\nAbbreviation:\n\ncyl: Cylinder3D, sem: SemanticKITTI, nusc: nuScenes, ms: multi-scan task, tta: test-time augmentation,\n\n1.5x: channel expansion ratio, 72_4: performance (mIoU), 64x512: resolution of the range image\n\n1. SemanticKITTI test set (single-scan):\n\n|Model|Reported|Reproduced|Gain|Weight|\n|:---:|:---:|:---:|:---:|:---:|\n|SPVNAS|66.4%|71.4%|**5.0%**|--|\n|Cylinder3D_1.5x|--|**72.4%**|--|[cyl_sem_1.5x_72_4.pt](https://drive.google.com/drive/folders/1LyWhVCqMzSVDe44c8ARDp8b94w1ct-tR?usp=sharing)|\n|Cylinder3D|68.9%|71.8%|**2.9%**|[cyl_sem_1.0x_71_8.pt](https://drive.google.com/drive/folders/1LyWhVCqMzSVDe44c8ARDp8b94w1ct-tR?usp=sharing)|\n|Cylinder3D_0.5x|71.2%|71.4%|0.2%|[cyl_sem_0.5x_71_4.pt](https://drive.google.com/drive/folders/1LyWhVCqMzSVDe44c8ARDp8b94w1ct-tR?usp=sharing)|\n|CENet_1.0x|64.7%|67.6%|2.9%|[CENet_64x512_67_6](https://drive.google.com/drive/folders/1LyWhVCqMzSVDe44c8ARDp8b94w1ct-tR?usp=sharing)|\n\n2. SemanticKITTI test set (multi-scan):\n\n|Model|Reported|Reproduced|Gain|Weight|\n|:---:|:---:|:---:|:---:|:---:|\n|Cylinder3D|52.5%|--|--|--|\n|Cylinder3D_0.5x|58.2%|58.4%|0.2%|[cyl_sem_ms_0.5x_58_4.pt](https://drive.google.com/drive/folders/1LyWhVCqMzSVDe44c8ARDp8b94w1ct-tR?usp=sharing)|\n\n3. Waymo test set:\n\n|Model|Reported|Reproduced|Gain|Weight|\n|:---:|:---:|:---:|:---:|:---:|\n|Cylinder3D|71.18%|71.18%|--|--|\n|Cylinder3D_0.5x|--|--|--|--|\n\n4. nuScenes val set:\n\n|Model|Reported|Reproduced|Gain|Weight|\n|:---:|:---:|:---:|:---:|:---:|\n|Cylinder3D|76.1%|--|--|--|\n|Cylinder3D_0.5x|76.0%|76.15%|0.15%|[cyl_nusc_0.5x_76_15.pt](https://drive.google.com/drive/folders/1LyWhVCqMzSVDe44c8ARDp8b94w1ct-tR?usp=sharing)|\n\n## Citation\nIf you use the codes, please consider citing the following publications:\n```\n@inproceedings{pvkd,\n    title     = {Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation},\n    author    = {Hou, Yuenan and Zhu, Xinge and Ma, Yuexin and Loy, Chen Change and Li, Yikang},\n    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},\n    pages     = {8479-8488}\n    year      = {2022},\n}\n\n@inproceedings{cylinder3d,\n    title={Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation},\n    author={Zhu, Xinge and Zhou, Hui and Wang, Tai and Hong, Fangzhou and Ma, Yuexin and Li, Wei and Li, Hongsheng and Lin, Dahua},\n    booktitle={IEEE Conference on Computer Vision and Pattern Recognition},\n    pages={9939--9948},\n    year={2021}\n}\n\n@article{cylinder3d-tpami,\n    title={Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perception},\n    author={Zhu, Xinge and Zhou, Hui and Wang, Tai and Hong, Fangzhou and Li, Wei and Ma, Yuexin and Li, Hongsheng and Yang, Ruigang and Lin, Dahua},\n    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},\n    year={2021},\n    publisher={IEEE}\n}\n```\n\n## Acknowledgements\nThis repo is built upon the awesome [Cylinder3D](https://github.com/xinge008/Cylinder3D).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcardwing%2FCodes-for-PVKD","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcardwing%2FCodes-for-PVKD","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcardwing%2FCodes-for-PVKD/lists"}