https://github.com/kentaroy47/benchmark-fp32-fp16-int8-with-tensorrt
Benchmark inference speed of CNNs with various quantization methods in Pytorch+TensorRT with Jetson Nano/Xavier
https://github.com/kentaroy47/benchmark-fp32-fp16-int8-with-tensorrt
hardware image-inference jetson-nano jetson-xavier tensorrt
Last synced: 8 months ago
JSON representation
Benchmark inference speed of CNNs with various quantization methods in Pytorch+TensorRT with Jetson Nano/Xavier
- Host: GitHub
- URL: https://github.com/kentaroy47/benchmark-fp32-fp16-int8-with-tensorrt
- Owner: kentaroy47
- License: mit
- Created: 2020-04-08T05:41:51.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2023-05-31T20:13:44.000Z (over 2 years ago)
- Last Synced: 2025-01-20T23:52:37.412Z (9 months ago)
- Topics: hardware, image-inference, jetson-nano, jetson-xavier, tensorrt
- Language: Jupyter Notebook
- Homepage:
- Size: 5.29 MB
- Stars: 54
- Watchers: 2
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Benchmark-FP32-FP16-INT8-with-TensorRT
Benchmark inference speed of CNNs with various quantization methods with TensorRT!
:star: if it helps you.
# Image classification
Run:
`inference_tensorrt.py`
## Hardware:Jetson Nano.
TRT notes TensorRT compiled models in the noted precision.
Latency of image inference (1,3,256,256) [ms]
| | TRT FP32 | TRT FP16 | TRT INT8 |
|:--------:|------|:----:|------|
| resnet18 | 26 | 18 | |
| resnet34 | 48 | 30 | |
| resnet50 | 79 | 42 | |
Jetson Nano does not support INT8..
## Hardware:Jetson Xavier.
TRT notes TensorRT compiled models in the noted precision.
Latency of image inference (1,3,256,256) [ms]
| | resnet18 | resnet34 | resnet50 |
|------|----------|----------|----------|
| PytorchRaw | 11 | 12 | 16 |
| TRT FP32 | 3.8 | 5.6 | 9.9 |
| TRT FP16 | 2.1 | 3.3 | 4.4 |
| TRT INT8 | 1.7 | 2.7 | 3.0 |
# Image segmentation
## Hardware:Jetson Xavier.
TRT notes TensorRT compiled models in the noted precision.
Latency of image inference (1,3,512,512) [ms]
| | fcn_resnet50 | fcn_resnet101 | deeplabv3_resnet50 | deeplabv3_resnet101 |
|------|--------------|---------------|--------------------|---------------------|
| PytorchRaw | 200 | 344 | 281 | 426 |
| TRT FP32 | 173 | 290 | 252 | 366 |
| TRT FP16 | 36 | 57 | 130 | 151 |
| TRT INT8 | 21 | 32 | 97 | 108 |
## Hardware:Jetson Nano.
Latency of image inference (1,3,256,256) [ms]
| | fcn_resnet50 |
|------|--------------|
| PytorchRaw | 6800 |
| TRT FP32 | 767 |
| TRT FP16 | 40 |
| TRT INT8 | NA |
# Hardware setup
The hardware setup seems tricky.
* Install pytorch
https://forums.developer.nvidia.com/t/pytorch-for-jetson-nano-version-1-4-0-now-available/72048
**The stable version for Jetson nano seems to be torch==1.1**
**For Xavier, torch==1.3 worked fine for me.**
* Install torchvision
I followed this instruction and installed torchvision==0.3.0
https://medium.com/hackers-terminal/installing-pytorch-torchvision-on-nvidias-jetson-tx2-81591d03ce32
```bash
sudo apt-get install libjpeg-dev zlib1g-dev
git clone -b v0.3.0 https://github.com/pytorch/vision torchvision
cd torchvision
sudo python3 setup.py install
```
* Install torch2trt
Followed readme.
https://github.com/NVIDIA-AI-IOT/torch2trt
```bash
sudo apt-get install libprotobuf* protobuf-compiler ninja-build
git clone https://github.com/NVIDIA-AI-IOT/torch2trt
cd torch2trt
sudo python3 setup.py install --plugins
```