{"id":21269132,"url":"https://github.com/yanx27/2dpass","last_synced_at":"2025-04-05T18:11:53.824Z","repository":{"id":44347876,"uuid":"512084655","full_name":"yanx27/2DPASS","owner":"yanx27","description":"2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds (ECCV 2022) :fire:","archived":false,"fork":false,"pushed_at":"2023-04-22T11:36:59.000Z","size":11550,"stargazers_count":422,"open_issues_count":34,"forks_count":55,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-03-29T17:11:22.207Z","etag":null,"topics":["deep-learning","knowledge-distillation","lidar-point-cloud","nuscenes","pytorch","semantic-segmentation","semantickitti"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yanx27.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-07-09T03:35:08.000Z","updated_at":"2025-03-25T14:08:15.000Z","dependencies_parsed_at":"2024-01-18T14:52:22.889Z","dependency_job_id":null,"html_url":"https://github.com/yanx27/2DPASS","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yanx27%2F2DPASS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yanx27%2F2DPASS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yanx27%2F2DPASS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yanx27%2F2DPASS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yanx27","download_url":"https://codeload.github.com/yanx27/2DPASS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247378152,"owners_count":20929297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","knowledge-distillation","lidar-point-cloud","nuscenes","pytorch","semantic-segmentation","semantickitti"],"created_at":"2024-11-21T08:07:27.169Z","updated_at":"2025-04-05T18:11:53.794Z","avatar_url":"https://github.com/yanx27.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/2dpass-2d-priors-assisted-semantic/3d-semantic-segmentation-on-semantickitti)](https://paperswithcode.com/sota/3d-semantic-segmentation-on-semantickitti?p=2dpass-2d-priors-assisted-semantic)[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/2dpass-2d-priors-assisted-semantic/lidar-semantic-segmentation-on-nuscenes)](https://paperswithcode.com/sota/lidar-semantic-segmentation-on-nuscenes?p=2dpass-2d-priors-assisted-semantic)\n\n# 2DPASS\n\n[![arXiv](https://img.shields.io/badge/arXiv-2203.09065-b31b1b.svg)](https://arxiv.org/pdf/2207.04397.pdf)\n[![GitHub Stars](https://img.shields.io/github/stars/yanx27/2DPASS?style=social)](https://github.com/yanx27/2DPASS)\n![visitors](https://visitor-badge.glitch.me/badge?page_id=https://github.com/yanx27/2DPASS)\n\n\n\nThis repository is for **2DPASS** introduced in the following paper\n\n[Xu Yan*](https://yanx27.github.io/), [Jiantao Gao*](https://github.com/Gao-JT), [Chaoda Zheng*](https://github.com/Ghostish), Chao Zheng, Ruimao Zhang, Shuguang Cui, [Zhen Li*](https://mypage.cuhk.edu.cn/academics/lizhen/), \"*2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds*\", ECCV 2022 [[arxiv]](https://arxiv.org/pdf/2207.04397.pdf).\n ![image](figures/2DPASS.gif)\n\nIf you find our work useful in your research, please consider citing:\n```latex\n@inproceedings{yan20222dpass,\n  title={2dpass: 2d priors assisted semantic segmentation on lidar point clouds},\n  author={Yan, Xu and Gao, Jiantao and Zheng, Chaoda and Zheng, Chao and Zhang, Ruimao and Cui, Shuguang and Li, Zhen},\n  booktitle={European Conference on Computer Vision},\n  pages={677--695},\n  year={2022},\n  organization={Springer}\n}\n\n@InProceedings{yan2022let,\n      title={Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis}, \n      author={Xu Yan and Heshen Zhan and Chaoda Zheng and Jiantao Gao and Ruimao Zhang and Shuguang Cui and Zhen Li},\n      year={2022},\n      booktitle={NeurIPS}\n}\n\n@article{yan2023benchmarking,\n  title={Benchmarking the Robustness of LiDAR Semantic Segmentation Models},\n  author={Yan, Xu and Zheng, Chaoda and Li, Zhen and Cui, Shuguang and Dai, Dengxin},\n  journal={arXiv preprint arXiv:2301.00970},\n  year={2023}\n}\n```\n## News\n* **2023-04-01** We merge MinkowskiNet and official SPVCNN models from [SPVNAS](https://github.com/mit-han-lab/spvnas) in our codebase. You can check these models in `config/`. We rename our baseline model from `spvcnn.py` to `baseline.py`.\n* **2023-03-31** We provide codes for the robustness evaluation on SemanticKITTI-C.\n* **2023-03-27** We release a model with higher performance on SemanticKITTI and codes for naive instance augmentation.\n* **2023-02-25** We release a new robustness benchmark for LiDAR semantic segmentation at [SemanticKITTI-C](https://yanx27.github.io/RobustLidarSeg/). Welcome to test your models!\n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"figures/semantickittic.png\" width=\"80%\"\u003e \n\u003c/p\u003e\n\n\n* **2022-10-11** Our new work for cross-modal knowledge distillation is accepted at NeurIPS 2022:smiley: [paper](https://arxiv.org/pdf/2210.04208.pdf) / [code](https://github.com/ZhanHeshen/PointCMT).\n* **2022-09-20** We release codes for SemanticKITTI single-scan and NuScenes :rocket:!\n* **2022-07-03** 2DPASS is accepted at **ECCV 2022** :fire:!\n* **2022-03-08** We achieve **1st** place in both single and multi-scans of [SemanticKITTI](http://semantic-kitti.org/index.html) and **3rd** place on [NuScenes-lidarseg](https://www.nuscenes.org/) :fire:! \n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"figures/singlescan.jpg\" width=\"80%\"\u003e \n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"figures/multiscan.jpg\" width=\"80%\"\u003e \n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"figures/nuscene.png\" width=\"80%\"\u003e \n\u003c/p\u003e\n\n## Installation\n\n### Requirements\n- pytorch \u003e= 1.8 \n- yaml\n- easydict\n- pyquaternion\n- [lightning](https://github.com/Lightning-AI/lightning) (tested with pytorch_lightning==1.3.8 and torchmetrics==0.5)\n- [torch-scatter](https://github.com/rusty1s/pytorch_scatter) (pip install torch-scatter -f https://data.pyg.org/whl/torch-1.9.0+${CUDA}.html)\n- [nuScenes-devkit](https://github.com/nutonomy/nuscenes-devkit) (optional for nuScenes)\n- [spconv](https://github.com/traveller59/spconv) (tested with spconv==2.1.16 and cuda==11.1, pip install spconv-cu111==2.1.16)\n- [torchsparse](https://github.com/mit-han-lab/torchsparse) (optional for MinkowskiNet and SPVCNN. sudo apt-get install libsparsehash-dev, pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0)\n\n## Data Preparation\n\n### SemanticKITTI\nPlease download the files from the [SemanticKITTI website](http://semantic-kitti.org/dataset.html) and additionally the [color data](http://www.cvlibs.net/download.php?file=data_odometry_color.zip) from the [Kitti Odometry website](http://www.cvlibs.net/datasets/kitti/eval_odometry.php). Extract everything into the same folder.\n```\n./dataset/\n├── \n├── ...\n└── SemanticKitti/\n    ├──sequences\n        ├── 00/           \n        │   ├── velodyne/\t\n        |   |\t├── 000000.bin\n        |   |\t├── 000001.bin\n        |   |\t└── ...\n        │   └── labels/ \n        |   |   ├── 000000.label\n        |   |   ├── 000001.label\n        |   |   └── ...\n        |   └── image_2/ \n        |   |   ├── 000000.png\n        |   |   ├── 000001.png\n        |   |   └── ...\n        |   calib.txt\n        ├── 08/ # for validation\n        ├── 11/ # 11-21 for testing\n        └── 21/\n\t    └── ...\n```\n\n### NuScenes\nPlease download the Full dataset (v1.0) from the [NuScenes website](https://www.nuscenes.org/) with lidarseg and extract it.\n```\n./dataset/\n├── \n├── ...\n└── nuscenes/\n    ├──v1.0-trainval\n    ├──v1.0-test\n    ├──samples\n    ├──sweeps\n    ├──maps\n    ├──lidarseg\n```\n\n## Training\n### SemanticKITTI\nYou can run the training with\n```shell script\ncd \u003croot dir of this repo\u003e\npython main.py --log_dir 2DPASS_semkitti --config config/2DPASS-semantickitti.yaml --gpu 0\n```\nThe output will be written to `logs/SemanticKITTI/2DPASS_semkitti` by default. \n### NuScenes\n```shell script\ncd \u003croot dir of this repo\u003e\npython main.py --log_dir 2DPASS_nusc --config config/2DPASS-nuscenese.yaml --gpu 0 1 2 3\n```\n\n### Vanilla Training without 2DPASS\nWe take SemanticKITTI as an example.\n```shell script\ncd \u003croot dir of this repo\u003e\npython main.py --log_dir baseline_semkitti --config config/2DPASS-semantickitti.yaml --gpu 0 --baseline_only\n```\n\n## Testing\nYou can run the testing with\n```shell script\ncd \u003croot dir of this repo\u003e\npython main.py --config config/2DPASS-semantickitti.yaml --gpu 0 --test --num_vote 12 --checkpoint \u003cdir for the pytorch checkpoint\u003e\n```\nHere, `num_vote` is the number of views for the test-time-augmentation (TTA). We set this value to 12 as default (on a Tesla-V100 GPU), and if you use other GPUs with smaller memory, you can choose a smaller value. `num_vote=1` denotes there is no TTA used, and will cause about ~2\\% performance drop.\n\n## Robustness Evaluation\nPlease download all subsets of [SemanticKITTI-C](https://arxiv.org/pdf/2301.00970.pdf) from [this link](https://cuhko365-my.sharepoint.com/personal/218012048_link_cuhk_edu_cn/_layouts/15/onedrive.aspx?id=%2Fpersonal%2F218012048%5Flink%5Fcuhk%5Fedu%5Fcn%2FDocuments%2FSemanticKITTIC\u0026ga=1) and extract them.\n```\n./dataset/\n├── \n├── ...\n└── SemanticKitti/\n    ├──sequences\n    ├──SemanticKITTI-C\n        ├── clean_data/           \n        ├── dense_16beam/           \n        │   ├── velodyne/\t\n        |   |\t├── 000000.bin\n        |   |\t├── 000001.bin\n        |   |\t└── ...\n        │   └── labels/ \n        |   |   ├── 000000.label\n        |   |   ├── 000001.label\n        |   |   └── ...\n\t    ...\n```\nYou can run the robustness evaluation with\n```shell script\ncd \u003croot dir of this repo\u003e\npython robust_test.py --config config/2DPASS-semantickitti.yaml --gpu 0  --num_vote 12 --checkpoint \u003cdir for the pytorch checkpoint\u003e\n```\n\n## Model Zoo\nYou can download the models with the scores below from [this Google drive folder](https://drive.google.com/drive/folders/1Xy6p_h827lv8J-2iZU8T6SLFkxfoXPBE?usp=sharing).\n### SemanticKITTI\n|Model (validation)|mIoU (vanilla)|mIoU (TTA)|Parameters|\n|:---:|:---:|:---:|:---:|\n|MinkowskiNet|65.1%|67.1%|21.7M|\n|SPVCNN|65.9%|67.8%|21.8M|\n|2DPASS (4scale-64dimension)|68.7%|70.0%|1.9M|\n|2DPASS (6scale-256dimension)|70.7%|72.0%|45.6M|\n\nHere, we fine-tune 2DPASS models on SemanticKITTI with more epochs and thus gain the higher mIoU. If you train with 64 epochs, it should be gained about 66%/69% for vanilla and 69%/71% after TTA.\n\n### NuScenes\n|Model (validation)|mIoU (vanilla)|mIoU (TTA)|Parameters|\n|:---:|:---:|:---:|:---:|\n|MinkowskiNet|74.3%|76.0%|21.7M|\n|SPVCNN|74.9%|76.9%|21.8M|\n|2DPASS (6scale-128dimension)|76.7%|79.6%|11.5M|\n|2DPASS (6scale-256dimension)|78.0%|80.5%|45.6M|\n\n**Note that the results on benchmarks are gained by training with additional validation set and using instance-level augmentation.**\n\n## Acknowledgements\nCode is built based on [SPVNAS](https://github.com/mit-han-lab/spvnas), [Cylinder3D](https://github.com/xinge008/Cylinder3D), [xMUDA](https://github.com/valeoai/xmuda) and [SPCONV](https://github.com/traveller59/spconv).\n\n## License\nThis repository is released under MIT License (see LICENSE file for details).\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyanx27%2F2dpass","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyanx27%2F2dpass","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyanx27%2F2dpass/lists"}