{"id":19635952,"url":"https://github.com/talkuhulk/image-animation-turbo-boost","last_synced_at":"2025-10-29T12:22:41.142Z","repository":{"id":37647253,"uuid":"499500422","full_name":"TalkUHulk/Image-Animation-Turbo-Boost","owner":"TalkUHulk","description":"Aim to accelerate the image-animation-model inference through the inference frameworks such as onnx、tensorrt and openvino.","archived":false,"fork":false,"pushed_at":"2024-02-23T14:03:33.000Z","size":35812,"stargazers_count":63,"open_issues_count":0,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-02-23T15:25:36.279Z","etag":null,"topics":["cpp","fomm","image-animation","onnx","openvino","tensorrt","tpsmm"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TalkUHulk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-03T12:19:38.000Z","updated_at":"2024-01-19T08:40:46.000Z","dependencies_parsed_at":"2022-09-10T08:50:37.234Z","dependency_job_id":null,"html_url":"https://github.com/TalkUHulk/Image-Animation-Turbo-Boost","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TalkUHulk%2FImage-Animation-Turbo-Boost","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TalkUHulk%2FImage-Animation-Turbo-Boost/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TalkUHulk%2FImage-Animation-Turbo-Boost/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TalkUHulk%2FImage-Animation-Turbo-Boost/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TalkUHulk","download_url":"https://codeload.github.com/TalkUHulk/Image-Animation-Turbo-Boost/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224102398,"owners_count":17256158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","fomm","image-animation","onnx","openvino","tensorrt","tpsmm"],"created_at":"2024-11-11T12:27:46.425Z","updated_at":"2025-10-29T12:22:36.085Z","avatar_url":"https://github.com/TalkUHulk.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Image Animation Turbo Boost\n\n\u003cimg src=\"https://img.shields.io/badge/python-3-lightgrey\"\u003e\u003c/a\u003e\n\u003cimg src=\"https://img.shields.io/badge/c%2B%2B-11-blue\"\u003e\u003c/a\u003e\n\u003cimg src=\"https://img.shields.io/badge/onnxruntime-1.9-orange\"\u003e\u003c/a\u003e\n\u003cimg src=\"https://img.shields.io/badge/openvino-2021.4-green\"\u003e\u003c/a\u003e\n\u003cimg src=\"https://img.shields.io/badge/tensorrt-8-yellowgreen\"\u003e\u003c/a\u003e\n\n\nAim to accelerate the image-animation-model inference through the inference frameworks such as onnx、tensorrt and openvino.\n\n****\n\n\n\nhttps://github.com/TalkUHulk/Image-Animation-Turbo-Boost/assets/18474163/a6cbb56c-10b3-4328-b9ce-ccece2e92aad\n\n\n\nhttps://github.com/TalkUHulk/Image-Animation-Turbo-Boost/assets/18474163/49ddfd0c-1f4d-4837-914e-4f1efaed1e92\n\n\n\n## FOMM\n\nThe model using from [FOMM](https://github.com/AliaksandrSiarohin/first-order-model)\n\n### Convert\n\n* Convert to onnx:\n\n```\npython export_onnx.py --output-name-kp kp_detector.onnx --output-name-fomm fomm.onnx --config config/vox-adv-256.yaml --ckpt ./checkpoints/vox-adv-cpk.pth.tar\n```\n\n* Convert to trt:\n\ndev environment: `docker pull chaoyiyuan/tensorrt8:latest`\n\nRun:\n```\nonnx2trt fomm.onnx -o fomm.trt\n```\n\n### Demo\n\n* [Python Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/FOMM/demo/TRT)\n\n****\n\n## TPSMM\n\nThe model using from [TPSMM](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model)\n\n### Convert\n\n* Convert to onnx:\n\n```\npython export_onnx.py --output-name-kp kp_detector.onnx --output-name-tpsmm tpsmm.onnx --config config/vox-256.yaml --ckpt ./checkpoints/vox.pth.tar\n```\n\n* Convert to openvino:\n\ndev environment: `docker pull openvino/ubuntu18_dev:2021.4.2_src`\n\n```\npython3 mo.py --input_model ./tpsmm.onnx  --output_dir ./openvino --data_type FP32\n```\n\n### Demo\n\n#### ONNXRuntime\n\n* [Python Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/TPSMM/demo/ONNXRuntime/python)\n* [C++ Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/TPSMM/demo/ONNXRuntime/cpp)\n\n\nTo test python demo run:\n```\npython demo/ONNXRuntime/python/demo.py --source ../assets/source.png --driving ../assets/driving.mp4 --onnx-file-tpsmm tpsmm.onnx --onnx-file-kp kp_detector.onnx\n```\n\nTo test c++ demo run:\n\n* build\n```\nmkdir build \u0026\u0026 cd build\ncmake ..\nmake -j8\n./onnx_demo xxx/tpsmm.onnx xxx/kp_detector.onnx xxx/source.png xxx/driving.mp4 ./generated_onnx.mp4\n```\n\n#### OpenVINO\n\n* [Python Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/TPSMM/demo/OpenVINO/python)\n* [C++ Demo](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost/tree/main/TPSMM/demo/OpenVINO/cpp)\n\nTo test python demo run:\n```\npython demo/OpenVINO/python/demo.py --source ../assets/source.png --driving ../assets/driving.mp4 --xml-kp xxxx/kp_detector_sim.xml --xml-tpsmm xxx/tpsmm_sim.xml --bin-kp xxx/kp_detector_sim.bin --bin-tpsmm xxx/tpsmm_sim.bin\n```\n\nTo test c++ demo run:\n\n* build\n```\nmkdir build \u0026\u0026 cd build\ncmake ..\nmake -j8\n./openvino_demo xxx/tpsmm.xml xxx/tpsmm.bin xxx/kp_detector.xml xxx/kp_detector.bin xxx/source.png xxx/driving.mp4 ./generated_onnx.mp4\n```\n\n#### Result\n\n|  FrameWork   | Elapsed(s)  | Language |\n|  ----  | ----  | ----  |\n| pytorch(cpu)  | 6 | python |\n| ONNXRuntime  | ~1.2 | python |\n| ONNXRuntime  | ~1.6 | c++ |\n| OpenVINO  | ~0.6 | python |\n| OpenVINO  | ~0.6 | c++ |\n\n\u003e  ONNXRuntime C++ is slower compared with python, maybe related to libraries which compiled by myself.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/generated_py_onnx.gif\" width=\"640px\"/\u003e\n\u003cbr\u003e\ngenerated by python onnx.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/generated_py_opv.gif\" width=\"640px\"/\u003e\n\u003cbr\u003e\ngenerated by python openvino.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/generated_cpp_onnx.gif\" width=\"640px\"/\u003e\n\u003cbr\u003e\ngenerated by cpp onnx.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/generated_cpp_opv.gif\" width=\"640px\"/\u003e\n\u003cbr\u003e\ngenerated by cpp openvino.\u003c/p\u003e\n\n\n#### To Do\n\nFailed to convert to tensorrt, maybe scatter ops is not supported. This will be fixed in 8.4GA, according to [issues](https://github.com/NVIDIA/TensorRT/issues/1541)\n\n\n### Pretrained Models\n\nPlease download the pre-trained models from the following links.\n| Path | Description\n| :--- | :----------\n|[FOMM](https://github.com/AliaksandrSiarohin/first-order-model)  | Original Pretrained Pytorch Model.\n|[TPSMM](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model)  | Original Pretrained Pytorch Model.\n|[FOMM Onnx](https://drive.google.com/drive/folders/1CFY8XW9g2EuEbdlwjQOauDlFOefqjwJd?usp=sharing)  | onnx model of fomm.\n|[FOMM TensorRT](https://drive.google.com/drive/folders/1LiDw7EbylouwXAdDrSjCMMYgERCl4Qps?usp=sharing)  | trt model of fomm.\n|[TPSMM Onnx](https://drive.google.com/drive/folders/16aF7LUWt1QIzmmlqn0ZNi3ytnw28rKwP?usp=sharing)  | onnx model of tpsmm.\n|[TPSMM OpenVINO](https://drive.google.com/drive/folders/1VvTzsAg4hsXVMu7b5ijFzbw23mhj6oJK?usp=sharing)  | openvino model of tpsmm.\n\n\n# Acknowledgments\n\nFOMM is [AliaksandrSiarohin](https://github.com/AliaksandrSiarohin/first-order-model)'s work.\n\nTPSMM is [yoyo-nb](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model)'s work.\n\nThanks for the excellent works!\n\nMy work is to modify part of the network，and enable the model can be converted to onnx、openvino or tensorrt.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftalkuhulk%2Fimage-animation-turbo-boost","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftalkuhulk%2Fimage-animation-turbo-boost","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftalkuhulk%2Fimage-animation-turbo-boost/lists"}