{"id":13437735,"url":"https://github.com/yas-sim/openvino-ep-enabled-onnxruntime","last_synced_at":"2025-04-22T13:40:03.177Z","repository":{"id":185235189,"uuid":"271786346","full_name":"yas-sim/openvino-ep-enabled-onnxruntime","owner":"yas-sim","description":"Describing How to Enable OpenVINO Execution Provider for ONNX Runtime","archived":false,"fork":false,"pushed_at":"2020-06-29T23:18:19.000Z","size":19502,"stargazers_count":19,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-29T15:34:43.361Z","etag":null,"topics":["deep-learning","inference","inference-api","inference-engine","inference-library","intel","onnx","onnx-backend","onnx-format","onnxruntime","openvino","openvino-toolkit"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yas-sim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-06-12T11:53:48.000Z","updated_at":"2024-08-22T19:19:30.000Z","dependencies_parsed_at":"2023-08-01T07:09:27.497Z","dependency_job_id":null,"html_url":"https://github.com/yas-sim/openvino-ep-enabled-onnxruntime","commit_stats":null,"previous_names":["yas-sim/openvino-ep-enabled-onnxruntime"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-ep-enabled-onnxruntime","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-ep-enabled-onnxruntime/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-ep-enabled-onnxruntime/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-ep-enabled-onnxruntime/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yas-sim","download_url":"https://codeload.github.com/yas-sim/openvino-ep-enabled-onnxruntime/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250250219,"owners_count":21399589,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","inference","inference-api","inference-engine","inference-library","intel","onnx","onnx-backend","onnx-format","onnxruntime","openvino","openvino-toolkit"],"created_at":"2024-07-31T03:00:59.822Z","updated_at":"2025-04-22T13:40:03.109Z","avatar_url":"https://github.com/yas-sim.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"# OpenVINO Execution Provider Enabled `onnxruntime`\r\n\r\n## 1. Description\r\nONNX runtime is a deep learning inferencing library developed and maintained by Microsoft.  \r\n\u003ca href=https://github.com/microsoft/onnxruntime\u003e'microsoft/onnxruntime' on GitHub\u003c/a\u003e  \r\n\r\nONNX runtime can load the ONNX format DL models and run it on a wide variety of systems. It supports multiple processors, OSes, and programming languages.  \r\nAlso, ONNX runtime supports multiple `execution providers(EP)` which is the backend inferencing engine library. ONNX runtime supports **Intel OpenVINO toolkit**, Intel DNNL, Intel nGraph, nVIDIA TensorRT, DirectML, ARM compute library, Android neural networks API, and many more EPs.  \r\n\r\nHowever, Intel OpenVINO EP is not enabled in the prebuilt binary distribution of ONNX runtime (v1.3.0).  \r\n\r\nIn this project, I built the ONNX runtime from the source code and enabled the OpenVINO execution provider.  \r\nThe project includes the steps to build and install ONNX runtime and a simple sample code to try ONNX runtime.  \r\n\r\n**You can load the ONNX DL model directly in your program and run it as fast and efficient as OpenVINO with this OpenVINO EP.**  \r\nAlso, OpenVINO EP **supports various Intel processors and accelerators** such as CPU_FP32, GPU_FP32, GPU_FP16, MYRIAD_FP16, VAD-M_FP16, and VAD-F_FP16 (VAD = Intel Vision Accelerator Design = HDDL).  \r\nThe performance is OpenVINO \u003e\u003e ONNX runtime (C++) \u003e ONNX runtime (Python) in general. The performance difference between OpenVINO and ONNX runtime is around 2ms to 10ms based on my test result (HW: Core i7-6770HQ).  \r\n\r\nThe sample code are simple CNN image classification programs and the DL model is ResNet-50.  \r\n\r\n----\r\n\r\nONNX runtimeはマイクロソフトが開発、保守しているディープラーニング推論ライブラリです。  \r\n\u003ca href=https://github.com/microsoft/onnxruntime\u003e'microsoft/onnxruntime' on GitHub\u003c/a\u003e  \r\n\r\nONNX runtimeはサポートする数多くのシステム上でONNXフォーマットのDLモデルをロードし実行することが可能です。多くのプロセッサ、OS、プログラミング言語をサポートします。  \r\nまたONNX runtimeは複数の`Execution provider(EP)`と呼ばれるバックエンド推論ライブラリをサポートします。**Intel OpenVINO toolkit**, Intel DNNL, Intel nGraph, nVIDIA TensorRT, DirectML, ARM compute library, Android neural networks APIなど数多くのEPをサポートしています。  \r\n\r\nしかしながら提供されているビルド済みバイナリパッケージではOpenVINO EPがイネーブルされていません (v1.3.0)。\r\n\r\nそこで、このプロジェクトではONNX runtimeをソースコードからビルドし、OpenVINO EPをイネーブルしてみました。  \r\n\r\n**このOpenVINO EPにより、(MOでIRに変換することなく)プログラム内で直接ONNX DLモデルを読み込み、OpenVINOのように高速に効率よく推論を行うことが可能になります。**  \r\nまた、OpenVINO EPを使用することにより**各種インテルプロセッサやアクセラレータも利用可能**となります。(CPU_FP32, GPU_FP32, GPU_FP16, MYRIAD_FP16, VAD-M_FP16, and VAD-F_FP16 (VAD = Intel Vision Accelerator Design = HDDL))  \r\nパフォーマンスはおおむねOpenVINO \u003e\u003e ONNX runtime (C++) \u003e ONNX runtime (Python)の順になるようです。私のテスト結果によると、パフォーマンスの差は2ms~10ms程度でした。(HW:Core i7-6770HQ)  \r\n\r\nサンプルプログラムは簡単な画像分類(CNN)のプログラムでResnet-50を使用しています。\r\n\r\n\r\n## 2. Prerequisites\r\n- **OpenVINO 2020.2**\r\n  - ONNX runtime v1.3.0 is compatible with OpenVINO 2020.2.  \r\n  - If you haven't installed it, go to the OpenVINO web page and follow the [*Get Started*](https://software.intel.com/en-us/openvino-toolkit/documentation/get-started) guide to do it.  \r\n\r\n\r\n## 3. Build and Install ONNX runtime\r\n\r\n**Note:** You should have either one of `onnxruntime` or `onnxruntime-openvino`. If you have installed `onnxruntime` beforehand, please uninstall it before you install `onnxruntime-openvino`.  \r\n\r\nLinux\r\n```sh\r\n# Build onnxruntime\r\ngit clone https://github.com/microsoft/onnxruntime\r\ngit checkout v1.3.0\r\ncd onnxruntime\r\n./build.sh --config Release --build_shared_lib --build_wheel --enable_pybind --use_openvino CPU_FP32 --skip_tests\r\n\r\n#Install Python module (optional)\r\npip3 install ./build/Linux/Release/dist/onnxruntime_openvino-1.3.0-cp36-cp36m-linux_x86_64.whl\r\n```\r\n\r\nWindows\r\n```sh\r\n# Build onnxruntime\r\ngit clone https://github.com/microsoft/onnxruntime\r\ngit checkout v1.3.0\r\ncd onnxruntime\r\nbuild.amd64.1411.bat --config Release --cmake_generator \"Visual Studio 16 2019\" --build_shared_lib --build_wheel --enable_pybind --use_openvino CPU_FP32 --skip_tests\r\n\r\n#Install Python module (optional)\r\npip install .\\build\\Windows\\Release\\Release\\dist\\onnxruntime_openvino-1.3.0-cp36-cp36m-win_amd64.whl\r\n```\r\n\r\n## 4. Build and Run the sample applications (C++ and Python)\r\n\r\n### Linux (C++)\r\n```sh\r\n# Download Resnet-18 model and class label text\r\nwget https://github.com/onnx/models/raw/master/vision/classification/resnet/model/resnet18-v2-7.onnx\r\nwget https://raw.githubusercontent.com/HoldenCaulfieldRye/caffe/master/data/ilsvrc12/synset_words.txt\r\n\r\n#Build C++ sample app\r\nmkdir -p build\r\ncd build\r\ncmake -DCMAKE_BUILD_TYPE=Release ..\r\nmake\r\ncd ..\r\n\r\n#Run\r\ncp $INTEL_OPENVINO_DIR/deployment_tools/demo/car.png .\r\ncp build/onnxtest .\r\ncp ../onnxruntime/build/Linux/Release/libonnxruntime.so .\r\ncp ../onnxruntime/build/Linux/Release/libcustom_op_library.so .\r\n./onnxtest\r\n```\r\n\r\n### Windows (C++)\r\n```sh\r\n# Download Resnet-18 model and class label text\r\nbitsadmin /transfer download https://github.com/onnx/models/raw/master/vision/classification/resnet/model/resnet18-v2-7.onnx %CD%\\resnet18-v2-7.onnx\r\nbitsadmin /transfer download https://raw.githubusercontent.com/HoldenCaulfieldRye/caffe/master/data/ilsvrc12/synset_words.txt %CD%\\synset_words.txt\r\n\r\n#Build C++ sample app\r\nmkdir build\r\ncd build\r\ncmake -G \"Visual Studio 16 2019\" -DCMAKE_BUILD_TYPE=Release ..\r\nmsbuild onnxtest.sln /p:Configuration=Release\r\ncd ..\r\n\r\n#Run\r\ncopy \"%INTEL_OPENVINO_DIR%\\deployment_tools\\demo\\car.png\" .\r\ncopy build\\Release\\onnxtest.exe .\r\ncopy ..\\onnxruntime\\build\\Windows\\Release\\Release\\onnxruntime.dll .\r\ncopy ..\\onnxruntime\\build\\Windows\\Release\\Release\\custom_op_library.dll .\r\nonnxtest.exe\r\n```\r\n\r\n### Linux (Python)\r\n**Command options:**  \r\n`-d`: OpenVINO device name. Must be one of CPU_FP32, GPU_FP16, GPU_FP32, MYRIAD_FP16, VAD-M_FP16, VAD-F_FP16. (default: CPU_FP32)  \r\n`-m`: ONNX model file name (default: resnet18-v2-7.onnx)  \r\n```sh\r\ncp $INTEL_OPENVINO_DIR/deployment_tools/demo/car.png .\r\npython3 onnxruntime-test.py [-d device_name] [-m model_name]\r\n```\r\n\r\n### Windows (Python)\r\n**Command options:**  \r\n`-d`: OpenVINO device name. Must be one of CPU_FP32, GPU_FP16, GPU_FP32, MYRIAD_FP16, VAD-M_FP16, VAD-F_FP16. (default: CPU_FP32)  \r\n`-m`: ONNX model file name (default: resnet18-v2-7.onnx)  \r\n```sh\r\ncp $INTEL_OPENVINO_DIR/deployment_tools/demo/car.png .\r\npython3 onnxruntime-test.py [-d device_name] [-m model_name]\r\n```\r\n\r\n## 5. Test Environment\r\n- Ubuntu 18.04 / Windows 10 1909  \r\n- OpenVINO 2020.2  \r\n- ONNX runtime 1.3.0  \r\n\r\n## See Also  \r\n* [Using Open Model Zoo demos](../../README.md)  \r\n* [Model Optimizer](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html)  \r\n* [Model Downloader](../../../tools/downloader/README.md)  \r\n* [ONNX runtime](https://github.com/microsoft/onnxruntime)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino-ep-enabled-onnxruntime","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyas-sim%2Fopenvino-ep-enabled-onnxruntime","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino-ep-enabled-onnxruntime/lists"}