https://github.com/jiangxb98/mwsis-plugin
https://github.com/jiangxb98/mwsis-plugin
Last synced: 7 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/jiangxb98/mwsis-plugin
- Owner: jiangxb98
- Created: 2023-12-03T09:12:34.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-16T13:29:21.000Z (about 1 year ago)
- Last Synced: 2024-06-16T14:43:32.346Z (about 1 year ago)
- Language: Python
- Size: 1.43 MB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- Awesome-Segment-Anything - [code
README
# MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving
⭐ Our MWSIS is acceped by the AAAI24.
⭐ [Arxiv](https://arxiv.org/abs/2312.06988) and [Website](https://jiangxb98.github.io/mwsis.github.io/)
## ToDo
- [x] Release the MWSIS-pts code.
- [ ] Improve data processing
- [ ] Reorganize the current code (pts).
- [ ] Replace the CCL.
- [x] Release the MWSIS-img code which is same as the MWSIS-pts.
## News
- [24-01-10] The code for MWSIS-img is released here.
- [23-12-16] The code for MWSIS-pts is released here.
## Instructions for MWSIS.
```bash
# python version is 3.7
# install torch
pip3 install torch==1.8.2 torchvision==0.9.2 torchaudio==0.8.2 --extra-index-url https://download.pytorch.org/whl/lts/1.8/cu111
# install mmdet3d==1.0.0rc5 dependency
pip install mmcv-full==1.6.0 mmsegmentation==0.27.0 mmdet==2.25.1 spconv-cu111 open3d minio==7.1.11 pymongo==3.12.3 waymo-open-dataset-tf-2-6-0==1.4.9
# install mmdet3d
git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v1.0.0rc5
pip install -e .
# clone the mwsis repo
git clone https://github.com/jiangxb98/mwsis-plugin.git
cd mwsis-plugin
# install pairwise_loss
cd boxinst-plugin-main
pip install -e .
mv boxinst_plugin/ops/ /mwsis_plugin/ops
# install torchex https://github.com/tusen-ai/SST
cd torchex
pip install -e .
# install torch scatter
pip install torch-scatter==2.1.0
# install easydict
pip install easydict
# download the resnet50 checkpoint
wget https://download.openmmlab.com/pretrain/third_party/resnet50_msra-5891d200.pth
mv .pth .cache/torch/hub/checkpoints/
```
## Data convert.
**Train data**
Please follow the tutorial given by [mmdet3d](https://github.com/open-mmlab/mmdetection3d) to process the data, and then use our [waymo conversion code](./tools/data_converter/waymo_converter_local.py) to process the data. (Note: You should `mv ./mwsis_plugin/tools/data_converter/waymo_converter_local.py ./tools/data_converter`)
**Val data generation**
Please use `mwsis_plugin/tools/data_converter/parallel_mask3d_eval_local.py`
## How to train MWSIS.
**Note:**
+ **About CCL**
Please use funcition `connected_components()` to replace function `voxel_spccl3d()`. Later, we will change the CCL operator to the operator in the torchex library. Currently, sicpy library is used instead.
+ **Warning**
Through experiment, we found that the torchex.connected_components unable to get the same results as the scipy.sparse.csgraph.connected_components.
```bash
# signle gpu
python tools/train.py mwsis/config/mwsis_pts_model_config.py --work-dir
# multi gpu
bash tools/dist_train.sh ...
```
## Acknowledgments
This project is based on the following codebases.
* [MMDetection3D](https://github.com/open-mmlab/mmdetection3d)
* [SST](https://github.com/tusen-ai/SST)