Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/talkuhulk/image-animation-turbo-boost
Aim to accelerate the image-animation-model inference through the inference frameworks such as onnx、tensorrt and openvino.
https://github.com/talkuhulk/image-animation-turbo-boost
cpp fomm image-animation onnx openvino tensorrt tpsmm
Last synced: 3 months ago
JSON representation
Aim to accelerate the image-animation-model inference through the inference frameworks such as onnx、tensorrt and openvino.
- Host: GitHub
- URL: https://github.com/talkuhulk/image-animation-turbo-boost
- Owner: TalkUHulk
- Created: 2022-06-03T12:19:38.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-02-23T14:03:33.000Z (12 months ago)
- Last Synced: 2024-02-23T15:25:36.279Z (12 months ago)
- Topics: cpp, fomm, image-animation, onnx, openvino, tensorrt, tpsmm
- Language: C++
- Homepage:
- Size: 34.2 MB
- Stars: 63
- Watchers: 2
- Forks: 7
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Image Animation Turbo Boost
Aim to accelerate the image-animation-model inference through the inference frameworks such as onnx、tensorrt and openvino.
****
https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/assets/18474163/a6cbb56c-10b3-4328-b9ce-ccece2e92aad
https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/assets/18474163/49ddfd0c-1f4d-4837-914e-4f1efaed1e92
## FOMM
The model using from [FOMM](https://github.com/AliaksandrSiarohin/first-order-model)
### Convert
* Convert to onnx:
```
python export_onnx.py --output-name-kp kp_detector.onnx --output-name-fomm fomm.onnx --config config/vox-adv-256.yaml --ckpt ./checkpoints/vox-adv-cpk.pth.tar
```
* Convert to trt:
dev environment: `docker pull chaoyiyuan/tensorrt8:latest`
Run:
```
onnx2trt fomm.onnx -o fomm.trt
```
### Demo
* [Python Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/FOMM/demo/TRT)
****
## TPSMM
The model using from [TPSMM](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model)
### Convert
* Convert to onnx:
```
python export_onnx.py --output-name-kp kp_detector.onnx --output-name-tpsmm tpsmm.onnx --config config/vox-256.yaml --ckpt ./checkpoints/vox.pth.tar
```
* Convert to openvino:
dev environment: `docker pull openvino/ubuntu18_dev:2021.4.2_src`
```
python3 mo.py --input_model ./tpsmm.onnx --output_dir ./openvino --data_type FP32
```
### Demo
#### ONNXRuntime
* [Python Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/TPSMM/demo/ONNXRuntime/python)
* [C++ Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/TPSMM/demo/ONNXRuntime/cpp)
To test python demo run:
```
python demo/ONNXRuntime/python/demo.py --source ../assets/source.png --driving ../assets/driving.mp4 --onnx-file-tpsmm tpsmm.onnx --onnx-file-kp kp_detector.onnx
```
To test c++ demo run:
* build
```
mkdir build && cd build
cmake ..
make -j8
./onnx_demo xxx/tpsmm.onnx xxx/kp_detector.onnx xxx/source.png xxx/driving.mp4 ./generated_onnx.mp4
```
#### OpenVINO
* [Python Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/TPSMM/demo/OpenVINO/python)
* [C++ Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/TPSMM/demo/OpenVINO/cpp)
To test python demo run:
```
python demo/OpenVINO/python/demo.py --source ../assets/source.png --driving ../assets/driving.mp4 --xml-kp xxxx/kp_detector_sim.xml --xml-tpsmm xxx/tpsmm_sim.xml --bin-kp xxx/kp_detector_sim.bin --bin-tpsmm xxx/tpsmm_sim.bin
```
To test c++ demo run:
* build
```
mkdir build && cd build
cmake ..
make -j8
./openvino_demo xxx/tpsmm.xml xxx/tpsmm.bin xxx/kp_detector.xml xxx/kp_detector.bin xxx/source.png xxx/driving.mp4 ./generated_onnx.mp4
```
#### Result
| FrameWork | Elapsed(s) | Language |
| ---- | ---- | ---- |
| pytorch(cpu) | 6 | python |
| ONNXRuntime | ~1.2 | python |
| ONNXRuntime | ~1.6 | c++ |
| OpenVINO | ~0.6 | python |
| OpenVINO | ~0.6 | c++ |
> ONNXRuntime C++ is slower compared with python, maybe related to libraries which compiled by myself.
generated by python onnx.
generated by python openvino.
generated by cpp onnx.
generated by cpp openvino.
#### To Do
Failed to convert to tensorrt, maybe scatter ops is not supported. This will be fixed in 8.4GA, according to [issues](https://github.com/NVIDIA/TensorRT/issues/1541)
### Pretrained Models
Please download the pre-trained models from the following links.
| Path | Description
| :--- | :----------
|[FOMM](https://github.com/AliaksandrSiarohin/first-order-model) | Original Pretrained Pytorch Model.
|[TPSMM](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model) | Original Pretrained Pytorch Model.
|[FOMM Onnx](https://drive.google.com/drive/folders/1CFY8XW9g2EuEbdlwjQOauDlFOefqjwJd?usp=sharing) | onnx model of fomm.
|[FOMM TensorRT](https://drive.google.com/drive/folders/1LiDw7EbylouwXAdDrSjCMMYgERCl4Qps?usp=sharing) | trt model of fomm.
|[TPSMM Onnx](https://drive.google.com/drive/folders/16aF7LUWt1QIzmmlqn0ZNi3ytnw28rKwP?usp=sharing) | onnx model of tpsmm.
|[TPSMM OpenVINO](https://drive.google.com/drive/folders/1VvTzsAg4hsXVMu7b5ijFzbw23mhj6oJK?usp=sharing) | openvino model of tpsmm.
# Acknowledgments
FOMM is [AliaksandrSiarohin](https://github.com/AliaksandrSiarohin/first-order-model)'s work.
TPSMM is [yoyo-nb](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model)'s work.
Thanks for the excellent works!
My work is to modify part of the network，and enable the model can be converted to onnx、openvino or tensorrt.