Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/AronCao49/MoPA
[ICRA24] MoPA: Multi-Modal Prior Aided Domain Adaptation for 3D Semantic Segmentation
https://github.com/AronCao49/MoPA
Last synced: about 1 month ago
JSON representation
[ICRA24] MoPA: Multi-Modal Prior Aided Domain Adaptation for 3D Semantic Segmentation
- Host: GitHub
- URL: https://github.com/AronCao49/MoPA
- Owner: AronCao49
- License: apache-2.0
- Created: 2023-09-12T08:25:16.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-08-15T13:02:56.000Z (5 months ago)
- Last Synced: 2024-08-15T14:57:14.832Z (5 months ago)
- Language: Python
- Homepage:
- Size: 13.6 MB
- Stars: 23
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- Awesome-Segment-Anything - [code
README
MoPA: Multi-Modal Prior Aided Domain Adaptation for
3D Semantic Segmentation
Haozhi Cao1,
Yuecong Xu2,
Jianfei Yang3*,
Pengyu Yin1,
Shenghai Yuan1,
Lihua Xie1
1: Centre for Advanced Robotics Technology Innovation (CARTIN), Nanyang Technological University
2: Department of Electrical and Computer Engineering, National University of Singapor
3: School of EEE, Nanyang Technological University
📄 [Arxiv] |
🎬 [Video] |
📖 [IEEEXplore]
## :scroll: About MoPA (ICRA'24)
MoPA is an MM-UDA method that aims to alleviate the imbalanced class-wise performance on Rare Objects (ROs) and the lack of 2D dense supervision signals through Valid Ground-based Insertion (VGI) and Segment Anything Mask consistency (SAM consistency). The overall structure is as follows.
Specifically, VGI inserts more ROs from the wild with ground truth to guide the recognition of ROs during UDA process without introducing artificial artifacts, while SAM consistency leverages image masks from [Segment Anything Model](https://github.com/facebookresearch/segment-anything) to encourage mask-wise prediction consistency.
## Installation and Prerequisite
### 1. Installation
To ease the effort during environment setup, we recommend you leverage [Docker](https://www.docker.com/) and [NVIDIA Container Toolkit](https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/docker.html). With Docker installed, you can either locally build the docker image for MoPA using [this Dockerfile](docker/Dockerfile) by running ```docker build -t mopa docker/ ```, or pull our pre-built image from Dockerhub by ```docker pull aroncao49/mopa:latest```.
You can then run a container using the docker image. Before running our code in the container, some prerequisites are needed to be installed. To do so, go to this repo folder and run ```bash install.sh```.
Remarks: you may ignore the ERROR warning saying werkzeug version is not compatible with open3d.
### 2. Patchwork++
To install [Patchwork++](https://github.com/url-kaist/patchwork-plusplus) for ground identification, follow the below command:
```bash
# Make sure you are in this repo folder
$ mkdir mopa/third_party && cd mopa/third_party
$ git clone https://github.com/url-kaist/patchwork-plusplus
$ cd patchwork-plusplus && make pyinstall
```
## Dataset Preparation
Please refer to [DATA_PREPARE.md](mopa/data/DATA_PREPARE.md) for the data preparation and pre-processing details.
## Training and Testing
Here we provide our pre-trained checkpoints for testing:
Method
USA→Singapore
Day→Night
A2D2→KITTI
2D 3D xM ckpt
2D 3D xM ckpt
2D 3D xM ckpt
xMUDA
58.5 51.2 61.0 link
47.7 42.1 52.3 link
42.6 44.9 47.2 link
MoPA+PL
61.8 57.8 64.5 link
51.9 46.9 54.8 link
49.1 56.2 54.1 link
MoPA+PLx2
62.1 56.8 64.0 link
51.7 44.6 55.3 link
50.0 56.8 54.7 link
**Note**: During our refactoration, we find the same reproduction issue as in the vanilla xMUDA (as in this [issue](https://github.com/valeoai/xmuda/issues/18)), where the performance fluctuates among different runs. This happens much more frequently on NuScenes benchmarks (Day→Night especially), so we suggest you to use our provided checkpoints for performance validation.
Before conducting training and testing, you are suggested to create or an output directory to capture the logs and checkpoints, and link that folder to ```mopa/exp```.
### 1. Testing
To conduct testing on, for example, A2D2→KITTI, simply download and extract the checkpoints or prepare your own trained networks, and use the following command:
```bash
$ CUDA_VISIBLE_DEVICES=0 python mopa/test/test.py \\
--cfg=configs/a2d2_semantic_kitti/xmuda_pl_pcmm_ema.yaml \\
--model_prefix=/path/to/checkpoint/dir \\
--ckpt2d=/2d_checkpoint/name \\
--ckpt3d=/3d_checkpoint/name \\
```
The class-wise results will be stored as a *.xls file in your checkpoint folder.
To generate pseudo-labels for training, include the extra arguments in the command:
```bash
$ CUDA_VISIBLE_DEVICES=0 python mopa/test/test.py \\
--cfg=configs/a2d2_semantic_kitti/xmuda_pl_pcmm_ema.yaml \\
--pselab_dir=DIR_NAME \\
VAL.BATCH_SIZE 1 DATASET_TARGET.TEST "('train',)"
```
The pseudo-labels will be stored in the folder ```ps_label/DIR_NAME``` under the dataset root dir.
### 2. Training
To conduct training with MoPA on, for example, A2D2→KITTI, simply use the following command:
```bash
$ CUDA_VISIBLE_DEVICES=0 python mopa/train/train_mopa.py \\
--cfg=configs/a2d2_semantic_kitti/xmuda_pl_pcmm_ema.yaml \\
DATASET_TARGET.SemanticKITTISCN.ps_label_dir DIR_NAME
```
You can also change those arguments in the config files directly.
## :eyes: Updates
* [2024.08] Our new MM-TTA paper for 3D segmentation has been accepted by ECCV 2024! Code will also be released soon. Check our [project site](https://sites.google.com/view/eccv24-latte) for more details!
* [2024.08] Release training/testing details and all checkpoints. We may further release the ROs we extracted if permitted.
* [2024.05] Release installation, prerequisite details, and data preparation procedures.
* [2024.03] We are now refactoring our code and evaluating its feasibility. Code will be available shortly.
* [2024.01] Our paper is accepted by ICRA 2024! Check our paper on arxiv [here](https://arxiv.org/abs/2309.11839).
## :envelope: Contact
For any further questions, please contact [Haozhi Cao](https://sites.google.com/view/haozhicao) ([email protected])
## :clap: Acknowledgement
We greatly appreciate the contributions of the following public repos:
- [torchsparse](https://github.com/mit-han-lab/torchsparse)
- [SPVNAS](https://github.com/mit-han-lab/spvnas)
- [SalsaNext](https://github.com/TiagoCortinhal/SalsaNext)
- [Patchwork++](https://github.com/url-kaist/patchwork-plusplus)
- [xMUDA](https://github.com/valeoai/xmuda)
## :pencil: Citation
```
@inproceedings{cao2024mopa,
title={Mopa: Multi-modal prior aided domain adaptation for 3d semantic segmentation},
author={Cao, Haozhi and Xu, Yuecong and Yang, Jianfei and Yin, Pengyu and Yuan, Shenghai and Xie, Lihua},
booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
pages={9463--9470},
year={2024},
organization={IEEE}
}
```