https://github.com/grimoire/mmdetection-to-tensorrt
convert mmdetection model to tensorrt, support fp16, int8, batch input, dynamic shape etc.
https://github.com/grimoire/mmdetection-to-tensorrt
cascade-rcnn faster-rcnn inference mmdetection object-detection retinanet ssd tensorrt yolov3
Last synced: about 2 months ago
JSON representation
convert mmdetection model to tensorrt, support fp16, int8, batch input, dynamic shape etc.
- Host: GitHub
- URL: https://github.com/grimoire/mmdetection-to-tensorrt
- Owner: grimoire
- License: apache-2.0
- Created: 2020-07-12T10:50:30.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2024-02-18T10:06:42.000Z (over 1 year ago)
- Last Synced: 2024-10-11T18:09:58.382Z (7 months ago)
- Topics: cascade-rcnn, faster-rcnn, inference, mmdetection, object-detection, retinanet, ssd, tensorrt, yolov3
- Language: Python
- Homepage:
- Size: 416 KB
- Stars: 586
- Watchers: 14
- Forks: 85
- Open Issues: 65
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# MMDet to TensorRT
> [!NOTE]
>
> The main branch is used to support model conversion of MMDetection>=3.0.
> If you want to convert model on older MMDetection, Please switch to branch:
> - [mmdet2trt=v0.5.0](https://github.com/grimoire/mmdetection-to-tensorrt/tree/v0.5.0)
> - [torch2trt_dynamic=v0.5.0](https://github.com/grimoire/torch2trt_dynamic/tree/v0.5.0)
> - [amirstan_plugin=v0.5.0](https://github.com/grimoire/amirstan_plugin/tree/v0.5.0).
## News
- 2024.02: Support MMDetection>=3.0
## Introduction
This project aims to support End2End deployment of models in MMDetection with TensorRT.
Mask support is **experiment**.
Features:
- fp16
- int8(experiment)
- batched input
- dynamic input shape
- combination of different modules
- DeepStream
## Requirement
- install MMDetection:
```bash
pip install openmim
mim install mmdet==3.3.0
```
- install [torch2trt_dynamic](https://github.com/grimoire/torch2trt_dynamic):
```bash
git clone https://github.com/grimoire/torch2trt_dynamic.git torch2trt_dynamic
cd torch2trt_dynamic
pip install -e .
```
- install [amirstan_plugin](https://github.com/grimoire/amirstan_plugin):
- Install tensorrt: [TensorRT](https://developer.nvidia.com/tensorrt)
- clone repo and build plugin
```bash
git clone --depth=1 https://github.com/grimoire/amirstan_plugin.git
cd amirstan_plugin
git submodule update --init --progress --depth=1
mkdir build
cd build
cmake -DTENSORRT_DIR=${TENSORRT_DIR} ..
make -j10
```
> [!NOTE]
>
> **DON'T FORGET** setting the environment variable(in `~/.bashrc`):
>
> ```bash
> export AMIRSTAN_LIBRARY_PATH=${amirstan_plugin_root}/build/lib
> ```
## Installation
### Host
```bash
git clone https://github.com/grimoire/mmdetection-to-tensorrt.git
cd mmdetection-to-tensorrt
pip install -e .
```
### Docker
Build docker image
```bash
sudo docker build -t mmdet2trt_docker:v1.0 docker/
```
Run (will show the help for the CLI entrypoint)
```bash
sudo docker run --gpus all -it --rm -v ${your_data_path}:${bind_path} mmdet2trt_docker:v1.0
```
Or if you want to open a terminal inside de container:
```bash
sudo docker run --gpus all -it --rm -v ${your_data_path}:${bind_path} --entrypoint bash mmdet2trt_docker:v1.0
```
Example conversion:
```bash
sudo docker run --gpus all -it --rm -v ${your_data_path}:${bind_path} mmdet2trt_docker:v1.0 ${bind_path}/config.py ${bind_path}/checkpoint.pth ${bind_path}/output.trt
```
## Usage
Create a TensorRT model from mmdet model.
detail can be found in [getting_started.md](./docs/getting_started.md)
### CLI
```bash
# conversion might take few minutes.
mmdet2trt ${CONFIG_PATH} ${CHECKPOINT_PATH} ${OUTPUT_PATH}
```
Run mmdet2trt -h for help on optional arguments.
### Python
```python
shape_ranges=dict(
x=dict(
min=[1,3,320,320],
opt=[1,3,800,1344],
max=[1,3,1344,1344],
)
)
trt_model = mmdet2trt(cfg_path,
weight_path,
shape_ranges=shape_ranges,
fp16_mode=True)
# save converted model
torch.save(trt_model.state_dict(), save_model_path)
# save engine if you want to use it in c++ api
with open(save_engine_path, mode='wb') as f:
f.write(trt_model.state_dict()['engine'])
```
> [!NOTE]
>
> The input of the engine is the tensor **after preprocess**.
> The output of the engine is `num_dets, bboxes, scores, class_ids`. if you enable the `enable_mask` flag, there will be another output `mask`.
> The bboxes output of the engine did not divided by `scale_factor`.
how to perform inference with the converted model.
```python
from mmdet.apis import inference_detector
from mmdet2trt.apis import create_wrap_detector
# create wrap detector
trt_detector = create_wrap_detector(trt_model, cfg_path, device_id)
# result share same format as mmdetection
result = inference_detector(trt_detector, image_path)
```
Try demo in `demo/inference.py`, or `demo/cpp` if you want to do inference with c++ api.
Read [getting_started.md](./docs/getting_started.md) for more details.
## How does it works?
Most other project use pytorch=>ONNX=>tensorRT route, This repo convert pytorch=>tensorRT directly, avoid unnecessary ONNX IR.
Read [how-does-it-work](https://github.com/NVIDIA-AI-IOT/torch2trt#how-does-it-work) for detail.
## Support Model/Module
> [!NOTE]
>
> Some models have only been tested on MMDet<3.0. If you found any failed model,
> Please report in the issue.
- [x] Faster R-CNN
- [x] Cascade R-CNN
- [x] Double-Head R-CNN
- [x] Group Normalization
- [x] Weight Standardization
- [x] DCN
- [x] SSD
- [x] RetinaNet
- [x] Libra R-CNN
- [x] FCOS
- [x] Fovea
- [x] CARAFE
- [x] FreeAnchor
- [x] RepPoints
- [x] NAS-FPN
- [x] ATSS
- [x] PAFPN
- [x] FSAF
- [x] GCNet
- [x] Guided Anchoring
- [x] Generalized Attention
- [x] Dynamic R-CNN
- [x] Hybrid Task Cascade
- [x] DetectoRS
- [x] Side-Aware Boundary Localization
- [x] YOLOv3
- [x] PAA
- [ ] CornerNet(WIP)
- [x] Generalized Focal Loss
- [x] Grid RCNN
- [x] VFNet
- [x] GROIE
- [x] Mask R-CNN(experiment)
- [x] Cascade Mask R-CNN(experiment)
- [x] Cascade RPN
- [x] DETR
- [x] YOLOX
Tested on:
- torch=2.2.0
- tensorrt=8.6.1
- mmdetection=3.3.0
- cuda=11.7
## FAQ
read [this page](./docs/FAQ.md) if you meet any problem.
## License
This project is released under the [Apache 2.0 license](LICENSE).