https://github.com/chrockey/benchmark_sparse
Benchmarking various sparse convolution libraries: MinkowskiEngine, SpConv, TorchSparse, and Open3D.
https://github.com/chrockey/benchmark_sparse
Last synced: about 1 year ago
JSON representation
Benchmarking various sparse convolution libraries: MinkowskiEngine, SpConv, TorchSparse, and Open3D.
- Host: GitHub
- URL: https://github.com/chrockey/benchmark_sparse
- Owner: chrockey
- Created: 2023-01-29T13:49:21.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-04-10T02:57:35.000Z (about 3 years ago)
- Last Synced: 2025-03-24T23:51:26.269Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 5.23 MB
- Stars: 12
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Benchmark of sparse convolution libraries
Benchmarking various sparse convolution libraries: [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine), [SpConv](https://github.com/traveller59/spconv), [TorchSparse](https://github.com/mit-han-lab/torchsparse), and [Open3D](https://github.com/isl-org/Open3D).
### Environments
- A6000 GPU
- Ubuntu 22.04
- CUDA 11.7
- PyTorch 2.0.0
- SpConv v2.3.5
- MinkowskiEngine v0.5.4
- TorchSparse v1.4.0
- libsparsehash-dev # apt-get install libsparsehash-dev
### Installation
```bash
~/benchmark_sparse$ conda create -n bench python=3.8 -y
~/benchmark_sparse$ conda activate bench
(bench) ~/benchmark_sparse$ pip install torch ninja open3d
(bench) ~/benchmark_sparse$ pip install spconv-cu117
(bench) ~/benchmark_sparse$ pip install -U git+https://github.com/NVIDIA/MinkowskiEngine -v --no-deps \
--install-option="--blas_include_dirs=${CONDA_PREFIX}/include" \
--install-option="--blas=openblas"
(bench) ~/benchmark_sparse$ pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0
```
### Comparsion
| Library | Voxelization | Mixed Precision Training |
|:----------------|:------------:|:------------------------:|
| MinkowskiEngine | :white_check_mark: | |
| SpConv | :white_check_mark: | :white_check_mark: |
| TorchSparse | :white_check_mark: | :white_check_mark: |
### Benchmark results
Each forward/backward time is the minimum time of 10 trials.
#### 1. Sample data (1,296,440 points) & Single convolution layer
| Library | Forward Time (ms) | Backward Time (ms) |
|:----------------|------------------:|-------------------:|
| MinkowskiEngine | 14.879 | 10.764 |
| SpConv | 8.764 | 0.492 |
| TorchSparse | 29.397 | 7.705 |
#### 2. Sample data (1,296,440 points) & Cylindrical network (no down/up sampling)
The cylindrical network is a stack of eight conv-bn-relu blocks.
| Library | Forward Time (ms) | Backward Time (ms) |
|:----------------|------------------:|-------------------:|
| MinkowskiEngine | 49.448 | 67.557 |
| SpConv | 24.142 | 3.119 |
| TorchSparse | 146.081 | 132.985 |
#### 3. ScanNetV2 & [MinkowskiNet42](https://openaccess.thecvf.com/content_CVPR_2019/papers/Choy_4D_Spatio-Temporal_ConvNets_Minkowski_Convolutional_Neural_Networks_CVPR_2019_paper.pdf)
I measured the average time of each training iteration (forward + backward) of MinkowskiNet42 on ScanNetV2 (3D semantic segmentation) with voxel size of 2cm and batch size of 8.
Note that each time includes data loading time with common data augmentations (e.g., random rotation).
| Library | Average time (sec/iter) |
|:----------------|------------------------:|
| MinkowskiEngine | 0.943 |
| SpConv | 0.452 |
### Conclusion
For now, the benchmark results show that SpConv is the fastest sparse conovlution library among MinkowskiEngine, SpConv, and TorchSparse.
However, it is worth noting that MinkowskiEngine with [TensorField](https://github.com/NVIDIA/MinkowskiEngine/blob/master/MinkowskiEngine/MinkowskiTensorField.py) support is more flexible than SpConv and TorchSparse, which is a big advantage for researchers.
Therefore, I recommend using MinkowskiEngine for research purposes and SpConv for production purposes where latency matters.
Any feedback is welcome!
### Todos
- [x] Benchmark results with a more complex network (e.g., UNet).
- [x] Benchmark the actual training time of the network on 3D semantic segmentation task.
- [ ] Add Open3D's sparse convolution.
### Acknowlegement
This repo heavily borrowed codes from [Chris Choy's benchmark code](https://gist.github.com/chrischoy/d8e971daf8308aa1dcba1524bf1fd91a).