https://github.com/ldkong1205/LaserMix

[CVPR 2023 Highlight] LaserMix for Semi-Supervised LiDAR Semantic Segmentation
https://github.com/ldkong1205/LaserMix

autonomous-driving lidar segmentation semi-supervised-learning

Last synced: 2 months ago
JSON representation

[CVPR 2023 Highlight] LaserMix for Semi-Supervised LiDAR Semantic Segmentation

• Host: GitHub
• URL: https://github.com/ldkong1205/LaserMix
• Owner: ldkong1205
• License: apache-2.0
• Created: 2022-06-21T08:59:57.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2024-05-14T03:39:28.000Z (about 1 year ago)
• Last Synced: 2024-08-01T03:42:41.408Z (10 months ago)
• Topics: autonomous-driving, lidar, segmentation, semi-supervised-learning
• Language: Python
• Homepage: https://ldkong.com/LaserMix
• Size: 8.61 MB
• Stars: 279
• Watchers: 16
• Forks: 17
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Citation: CITATION.cff

Awesome Lists containing this project

• Awesome-Mixup - [Code

README

        





  

  

  
LaserMix for Semi-Supervised LiDAR Semantic Segmentation

  

  


      Lingdong Kong, 

      Jiawei Ren, 

      Liang Pan, 

      Ziwei Liu

    


  S-Lab, Nanyang Technological University

  




  

    

  

  

  

    

  

  

  

    

  

  

  

    

  

  

  

    

  

  

  

    

  

  



## About

LaserMix is a semi-supervised learning (SSL) framework designed for LiDAR semantic segmentation. It leverages the strong spatial prior of driving scenes to construct low-variation areas via laser beam mixing, and encourages segmentation models to make confident and consistent predictions before and after mixing.






  

  


  Fig. Illustration for laser beam partition based on inclination φ.






Visit our project page to explore more details. :red_car:

## Updates

- \[2024.05\] - Our improved framework, [LaserMix++](https://arxiv.org/abs/2405.05258) :rocket:, is avaliable on arXiv.

- \[2024.01\] - The [toolkit](https://github.com/robodrive-24/toolkit) tailored for [The RoboDrive Challenge](https://robodrive-24.github.io/) has been released. :hammer_and_wrench:

- \[2023.12\] - We are hosting [The RoboDrive Challenge](https://robodrive-24.github.io/) at [ICRA 2024](https://2024.ieee-icra.org/). :blue_car:

- \[2023.12\] - Introducing [FRNet](https://github.com/Xiangxu-0103/FRNet), an efficient and effective **real-time** LiDAR segmentation model that achieves promising semi-supervised learning results on `SemanticKITTI` and `nuScenes`. Code and checkpoints are available for downloading.

- \[2023.03\] - Intend to test the **robustness** of your LiDAR semantic segmentation models? Check our recent work, :robot: [Robo3D](https://github.com/ldkong1205/Robo3D), a comprehensive suite that enables OoD robustness evaluation of 3D segmentors on our newly established datasets: `SemanticKITTI-C`, `nuScenes-C`, and `WOD-C`.

- \[2023.03\] - LaserMix was selected as a :sparkles: highlight :sparkles: at [CVPR 2023](https://cvpr.thecvf.com/) (top 10% of accepted papers).

- \[2023.02\] - LaserMix was accepted to [CVPR 2023](https://cvpr.thecvf.com/)! :tada:

- \[2023.02\] - LaserMix has been integrated into the [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) codebase! Check [this](https://github.com/open-mmlab/mmdetection3d/pull/2302) PR in the `dev-1.x` branch to know more details. :beers:

- \[2023.01\] - As suggested, we will establish a *sequential track* taking into account the LiDAR data collection nature in our semi-supervised LiDAR semantic segmentation benchmark. The results will be gradually updated in [RESULT.md](docs/RESULT.md).

- \[2022.12\] - We support a wider range of LiDAR segmentation backbones, including [RangeNet++](https://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/milioto2019iros.pdf), [SalsaNext](https://arxiv.org/abs/2003.03653), [FIDNet](https://arxiv.org/abs/2109.03787), [CENet](https://arxiv.org/abs/2207.12691), [MinkowskiUNet](https://github.com/NVIDIA/MinkowskiEngine), [Cylinder3D](https://openaccess.thecvf.com/content/CVPR2021/papers/Zhu_Cylindrical_and_Asymmetrical_3D_Convolution_Networks_for_LiDAR_Segmentation_CVPR_2021_paper.pdf), and [SPVCNN](https://arxiv.org/pdf/2007.16100), under both fully- and semi-supervised settings. The checkpoints will be available soon!

- \[2022.12\] - The derivation of spatial-prior-based SSL is available [here](https://ldkong.com/LaserMix/derivation.pdf). Take a look! :memo:

- \[2022.08\] - LaserMix achieves 1st place among the semi-supervised semantic segmentation leaderboards of [nuScenes](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-25), [SemanticKITTI](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-24), and [ScribbleKITTI](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-23), based on [Paper-with-Code](https://paperswithcode.com/paper/lasermix-for-semi-supervised-lidar-semantic). :bar_chart:

- \[2022.08\] - We provide a [video demo](https://youtu.be/Xkwa5-dT0g4) for visual comparisons on the SemanticKITTI val set. Take a look!

- \[2022.07\] - Our paper is available on arXiv, click here to check it out. Code will be available soon!

## Outline

- [Installation](#installation)

- [Data Preparation](#data-preparation)

- [Getting Started](#getting-started)

- [Video Demo](#video-demo)

- [Main Results](#main-results)

- [TODO List](#todo-list)

- [License](#license)

- [Acknowledgement](#acknowledgement)

- [Citation](#citation)

## Installation

Please refer to [INSTALL.md](docs/INSTALL.md) for the installation details.

## Data Preparation

Please refer to [DATA_PREPARE.md](docs/DATA_PREPARE.md) for the details to prepare the 1[nuScenes](https://www.nuscenes.org), 2[SemanticKITTI](http://www.semantic-kitti.org/), and 3[ScribbleKITTI](https://github.com/ouenal/scribblekitti) datasets.

## Getting Started

Please refer to [GET_STARTED.md](docs/GET_STARTED.md) to learn more usage about this codebase.

## Video Demo

| Demo 1 | Demo 2| Demo 3 | 

| :-: | :-: | :-: |

|  |   |   | 

| [Link](https://youtu.be/Xkwa5-dT0g4) :arrow_heading_up: | [Link](https://youtu.be/OlKNDt8_um4) :arrow_heading_up: | [Link](https://youtu.be/f8UKgxi5mow) :arrow_heading_up: |

## Main Result

### Framework Overview



  



### Range View

   

      Method

      nuScenes

      SemanticKITTI

      ScribbleKITTI

   

   

      1% 10% 20% 50%

      1% 10% 20% 50%

      1% 10% 20% 50%

   

   

      Sup.-only

      38.3 57.5 62.7 67.6

      36.2 52.2 55.9 57.2

      33.1 47.7 49.9 52.5

   

   

      LaserMix

      49.568.270.673.0

      43.458.859.461.4

      38.354.455.658.7

   

  

      improv. ↑

      +11.2 +10.7 +7.9 +5.4

      +7.2  +6.6  +3.5 +4.2

      +5.2  +6.7  +5.7 +6.2

   

   

      LaserMix++

             

             

             

   

   

      improv. ↑

             

             

             

   

### Voxel

   

      Method

      nuScenes

      SemanticKITTI

      ScribbleKITTI

   

   

      1% 10% 20% 50%

      1% 10% 20% 50%

      1% 10% 20% 50%

   

   

      Sup.-only

      50.9 65.9 66.6 71.2

      45.4 56.1 57.8 58.7

      39.2 48.0 52.1 53.8

   

   

      LaserMix

      55.3 69.9 71.8 73.2

      50.6 60.0 61.9 62.3

      44.2 53.7 55.1 56.8

   

   

      improv. ↑

      +4.4 +4.0 +5.2 +2.0

      +5.2 +3.9 +4.1 +3.6

      +5.0 +5.7 +3.0 +3.0

   

   

      LaserMix++

             

             

             

   

   

      improv. ↑

             

             

             

   

### Ablation Studies



  



### Qualitative Examples

![qualitative](docs/figs/qualitative.png)

### Checkpoints & More Results

For more experimental results and pretrained weights, please refer to [RESULT.md](docs/RESULT.md).

## TODO List

- [x] Initial release. :rocket:

- [x] Add license. See [here](#license) for more details.

- [x] Add video demos :movie_camera:

- [x] Add installation details.

- [x] Add data preparation details.

- [ ] Add evaluation details.

- [ ] Add training details.

## Citation

If you find this work helpful, please kindly consider citing our paper:

```bibtex

@inproceedings{kong2023lasermix,

  title = {LaserMix for Semi-Supervised LiDAR Semantic Segmentation},

  author = {Kong, Lingdong and Ren, Jiawei and Pan, Liang and Liu, Ziwei},

  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},

  pages = {21705--21715},

  year = {2023},

}

```

## License






This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

## Acknowledgement

This work is developed based on the [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) codebase.

>


> MMDetection3D is an open-source toolbox based on PyTorch, towards the next-generation platform for general 3D perception. It is a part of the OpenMMLab project developed by MMLab.

We acknowledge the use of the following public resources during the course of this work: 1[nuScenes](https://www.nuscenes.org), 2[nuScenes-devkit](https://github.com/nutonomy/nuscenes-devkit), 3[SemanticKITTI](http://www.semantic-kitti.org/), 4[SemanticKITTI-API](https://github.com/PRBonn/semantic-kitti-api), 5[ScribbleKITTI](https://github.com/ouenal/scribblekitti), 6[FIDNet](https://github.com/placeforyiming/IROS21-FIDNet-SemanticKITTI), 7[CENet](https://github.com/huixiancheng/CENet), 8[SPVNAS](https://github.com/mit-han-lab/spvnas), 9[Cylinder3D](https://github.com/xinge008/Cylinder3D), 10[TorchSemiSeg](https://github.com/charlesCXK/TorchSemiSeg), 11[MixUp](https://github.com/facebookresearch/mixup-cifar10), 12[CutMix](https://github.com/clovaai/CutMix-PyTorch), 13[CutMix-Seg](https://github.com/Britefury/cutmix-semisup-seg), 14[CBST](https://github.com/yzou2/CBST), 15[MeanTeacher](https://github.com/CuriousAI/mean-teacher), and 16[Cityscapes](https://www.cityscapes-dataset.com).

We would like to thank Fangzhou Hong for the insightful discussions and feedback. ❤️