{"id":26795917,"url":"https://github.com/yas-sim/openvino_asynchronous_api_performance_demo","last_synced_at":"2026-04-29T17:39:22.460Z","repository":{"id":285034100,"uuid":"956863972","full_name":"yas-sim/OpenVINO_Asynchronous_API_Performance_Demo","owner":"yas-sim","description":"This project demonstrates the high performance of OpenVINO asynchronous inference API","archived":false,"fork":false,"pushed_at":"2025-03-29T02:47:24.000Z","size":29663,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T03:27:05.884Z","etag":null,"topics":["accelerator","asynchronous","deep-learning","edge-ai","embedded","face-detection","high-performance","inference","inference-api","openvino","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yas-sim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-29T02:14:32.000Z","updated_at":"2025-03-29T02:47:27.000Z","dependencies_parsed_at":"2025-03-29T03:37:10.737Z","dependency_job_id":null,"html_url":"https://github.com/yas-sim/OpenVINO_Asynchronous_API_Performance_Demo","commit_stats":null,"previous_names":["yas-sim/openvino_asynchronous_api_performance_demo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2FOpenVINO_Asynchronous_API_Performance_Demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2FOpenVINO_Asynchronous_API_Performance_Demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2FOpenVINO_Asynchronous_API_Performance_Demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2FOpenVINO_Asynchronous_API_Performance_Demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yas-sim","download_url":"https://codeload.github.com/yas-sim/OpenVINO_Asynchronous_API_Performance_Demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246223325,"owners_count":20743168,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accelerator","asynchronous","deep-learning","edge-ai","embedded","face-detection","high-performance","inference","inference-api","openvino","python"],"created_at":"2025-03-29T18:16:27.561Z","updated_at":"2026-04-29T17:39:22.417Z","avatar_url":"https://github.com/yas-sim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenVINO asynchronous inference performance demo\r\n\r\n## Description\r\nOpenVINO Toolkit supports both synchronous and asynchronous inference APIs.  \r\nThe synchronous (blocking) inference API is easy to use, but it can leverage only a small fraction of the processor's performance.  \r\nYou can utilize full processor performance with the asynchronous inference API.  \r\n\r\nAsynchronous inference is also essential when you use a DL accelerator such as an integrated GPU or NPU. If you use the synchronous inference API for accelerators, **the CPU just waits until the inference on the accelerator is completed. It's just a waste of CPU time.**\r\n\r\nThis project demonstrates the difference in inference performance between synchronous API and asynchronous API.\r\nThis demo program uses a callback function and multiple threads to utilize the highest possible advantage of asynchronous inference.\r\n```python\r\n# callback function to receive the asynchronous inference result\r\ndef callback(self, request, userdata):      # userdata == input image for inferencing\r\n    res = list(request.results.values())[0]\r\n    if time.perf_counter() - self.time_last_callback \u003c 1/30:        # submit the result to the rendering thread every 1/30 sec only\r\n        return\r\n    self.time_last_callback = time.perf_counter()\r\n    self.queue_inference_result.put((res, userdata))\r\n```\r\n\r\nAlso, setting appropriate performance parameters and hints is critical for the best performance.\r\n```python\r\n# OpenVINO performance optimize parameters and hints\r\nconfig={'CACHE_DIR':'./cache'}\r\nconfig.update({hints.performance_mode: hints.PerformanceMode.THROUGHPUT})\r\nconfig.update({hints.num_requests:\"32\"})            # number of request queue\r\nconfig.update({props.inference_num_threads: \"16\"})  # number of thread used by OpenVINO runtime\r\nconfig.update({props.num_streams: \"8\"})             # number of simultaneous inference request execution\r\n```\r\n## Screen capture of the demo\r\n\r\nconfig = Core i7-10700K, Mem 64GB, Win11, OpenVINO 2025.0\r\n\r\n### Synchronous API (device=CPU) - 212fps\r\nCPU utilization is 45%  \r\n**Note:** The initial version had 60fps performance. The fps performance improved after I changed the program to update the screen every 1/30sec only (reducing the screen update burden).\r\n\r\n![Synchronous](./resources/sync.jpg)\r\n\r\n### Acynshronous API (device=CPU) - 1,024fps (x4.8 to sync)\r\nCPU utilization is 100%.\r\n![Asynchronous](./resources/async.jpg)\r\n\r\n## Reference\r\n\r\n- [Image Classification Async Sample](https://docs.openvino.ai/2025/get-started/learn-openvino/openvino-samples/image-classification-async.html)\r\n\r\n- [OpenVINO™ Runtime Python API Advanced Inference](https://docs.openvino.ai/2025/openvino-workflow/running-inference/integrate-openvino-with-your-application/python-api-advanced-inference.html)\r\n\r\n- [API: openvino.AsyncInferQueue](https://docs.openvino.ai/2025/api/ie_python_api/_autosummary/openvino.AsyncInferQueue.html)\r\n\r\n- [High-level Performance Hints](https://docs.openvino.ai/2025/openvino-workflow/running-inference/optimize-inference/high-level-performance-hints.html)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino_asynchronous_api_performance_demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyas-sim%2Fopenvino_asynchronous_api_performance_demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino_asynchronous_api_performance_demo/lists"}