{"id":20685449,"url":"https://github.com/yas-sim/openvino_auto_benchmark","last_synced_at":"2025-04-22T13:40:00.556Z","repository":{"id":185235202,"uuid":"476543214","full_name":"yas-sim/openvino_auto_benchmark","owner":"yas-sim","description":"Semi-automated OpenVINO benchmark_app with variable parameters. User can specify multiple options for any parameters in the benchmark_app and the progam runs the benchmark with all combinations of given options.","archived":false,"fork":false,"pushed_at":"2022-04-19T14:17:05.000Z","size":10,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T15:34:43.361Z","etag":null,"topics":["automation","automation-test","benchmark","deep-learning","deep-neural-networks","deeplearning","inference","intel","openvino","performance-ben","performance-test","performance-testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yas-sim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-04-01T02:10:45.000Z","updated_at":"2023-07-29T20:09:54.000Z","dependencies_parsed_at":"2023-08-01T07:09:27.711Z","dependency_job_id":null,"html_url":"https://github.com/yas-sim/openvino_auto_benchmark","commit_stats":null,"previous_names":["yas-sim/openvino_auto_benchmark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino_auto_benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino_auto_benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino_auto_benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino_auto_benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yas-sim","download_url":"https://codeload.github.com/yas-sim/openvino_auto_benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250250175,"owners_count":21399586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","automation-test","benchmark","deep-learning","deep-neural-networks","deeplearning","inference","intel","openvino","performance-ben","performance-test","performance-testing"],"created_at":"2024-11-16T22:27:25.987Z","updated_at":"2025-04-22T13:40:00.456Z","avatar_url":"https://github.com/yas-sim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Semi-automated OpenVINO benchmark_app with variable parameters  \n\n## Description  \nThis program allows the users to specify variable parameters in the OpenVINO benchmark_app and run the benchmark with all combinations of the given parameters automatically.  \nThe program will generate the report file in the CSV format with coded date and time file name ('`result_DDmm-HHMMSS.csv`'). You can analyze or visualize the benchmark result with MS Excel or a spreadsheet application.  \n\n**The program is just a front-end for the OpenVINO official benchmark_app.**  \nThis program utilizes the benchmark_app as the benchmark core logic. So the performance result measured by this program must be consistent with the one measured by the benchmark_app.  \nAlso, the command line parameters and their meaning are compatible with the benchmark_app.  \n\n### Requirements  \n- OpenVINO 2022.1 or higher  \nThis program is not compatible with OpenVINO 2021.\n\n### How to run  \n1. Install required Python modules.  \n```sh\npython -m pip install --upgrade pip setuptools\npython -m pip install -r requirements.txt\n```\n\n2. Run the auto benchmark (command line example)  \n```sh\npython auto_benchmark_app.py -m resnet.xml -niter 100 -nthreads %1,2,4,8 -nstreams %1,2 -d %CPU,GPU -cdir cache\n```\nWith this command line, `-nthreads` has 4 options (1,2,4,8), `-nstreams` has 2 options (1,2), and `-d` option has 2 options (CPU,GPU). As the result, 16 (4x2x2) benchmarks will be performed in total.  \n\n### Parameter options  \nYou can specify variable parameters by adding following prefix to the parameters.  \n|Prefix|Type|Description/Example|\n|---|---|---|\n|$|range|`$1,8,2` == `range(1,8,2)` =\u003e `[1,3,5,7]`\u003cbr\u003eAll `range()` compatible expressions are possible. e.g. `$1,5` or `$5,1,-1`|\n|%|list|`%CPU,GPU` =\u003e `['CPU', 'GPU']`, `%1,2,4,8` =\u003e `[1,2,4,8]`|\n|@|ir-models|`@models` == IR models in the \\'`./models`' dir =\u003e `['resnet.xml', 'googlenet.xml', ...]`\u003cbr\u003eThis option will recursively search the '.xml' files in the specified directory.|\n\n### Examples of command line  \n`python auto_benchmark_app.py -cdir cache -m resnet.xml -nthreads $1,5 -nstreams %1,2,4,8 -d %CPU,GPU`  \n- Run benchmark with `-nthreads`=`range(1,5)`=[1,2,3,4], `-nstreams`=[1,2,4,8], `-d`=['CPU','GPU']. Total 32 combinations.  \n\n`python auto_benchmark_app.py -m @models -niter 100 -nthreads %1,2,4,8 -nstreams %1,2 -d CPU -cdir cache`\n- Run benchmark with `-m`=[all .xml files in `models` directory], `-nthreads` = [1,2,4,8], `-nstreams`=[1,2].  \n\n### Example of a result file  \nThe last 4 items in each line are the performance data in the order of 'count', 'duration (ms)', 'latency AVG (ms)', and 'throughput (fps)'.  \n```\n#CPU: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz\n#MEM: 33947893760\n#OS: Windows-10-10.0.22000-SP0\n#OpenVINO: 2022.1.0-7019-cdb9bec7210-releases/2022/1\n#Last 4 items in the lines : test count, duration (ms), latency AVG (ms), and throughput (fps)\nbenchmark_app.py,-m,models\\FP16\\googlenet-v1.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,100,772.55,30.20,129.44\nbenchmark_app.py,-m,models\\FP16\\resnet-50-tf.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,100,1917.62,75.06,52.15\nbenchmark_app.py,-m,models\\FP16\\squeezenet1.1.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,100,195.28,7.80,512.10\nbenchmark_app.py,-m,models\\FP16-INT8\\googlenet-v1.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,104,337.09,24.75,308.53\nbenchmark_app.py,-m,models\\FP16-INT8\\resnet-50-tf.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,100,1000.39,38.85,99.96\nbenchmark_app.py,-m,models\\FP16-INT8\\squeezenet1.1.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,104,64.22,4.69,1619.38\nbenchmark_app.py,-m,models\\FP32\\googlenet-v1.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,100,778.90,30.64,128.39\nbenchmark_app.py,-m,models\\FP32\\resnet-50-tf.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,100,1949.73,76.91,51.29\nbenchmark_app.py,-m,models\\FP32\\squeezenet1.1.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,100,182.59,7.58,547.69\nbenchmark_app.py,-m,models\\FP32-INT8\\googlenet-v1.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,104,331.73,24.90,313.51\nbenchmark_app.py,-m,models\\FP32-INT8\\resnet-50-tf.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,100,968.38,38.45,103.27\nbenchmark_app.py,-m,models\\FP32-INT8\\squeezenet1.1.xml,-niter,100,-nthreads,1,-nstreams,1,-d,CPU,-cdir,cache,104,67.70,5.04,1536.23\nbenchmark_app.py,-m,models\\FP16\\googlenet-v1.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,1536.14,15.30,65.10\nbenchmark_app.py,-m,models\\FP16\\resnet-50-tf.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,3655.59,36.50,27.36\nbenchmark_app.py,-m,models\\FP16\\squeezenet1.1.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,366.73,3.68,272.68\nbenchmark_app.py,-m,models\\FP16-INT8\\googlenet-v1.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,872.87,8.66,114.56\nbenchmark_app.py,-m,models\\FP16-INT8\\resnet-50-tf.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,1963.67,19.54,50.93\nbenchmark_app.py,-m,models\\FP16-INT8\\squeezenet1.1.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,242.28,2.34,412.74\nbenchmark_app.py,-m,models\\FP32\\googlenet-v1.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,1506.14,14.96,66.39\nbenchmark_app.py,-m,models\\FP32\\resnet-50-tf.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,3593.88,35.88,27.83\nbenchmark_app.py,-m,models\\FP32\\squeezenet1.1.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,366.28,3.56,273.01\nbenchmark_app.py,-m,models\\FP32-INT8\\googlenet-v1.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,876.52,8.69,114.09\nbenchmark_app.py,-m,models\\FP32-INT8\\resnet-50-tf.xml,-niter,100,-nthreads,2,-nstreams,1,-d,CPU,-cdir,cache,100,1934.72,19.25,51.69\n```\n\nEND\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino_auto_benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyas-sim%2Fopenvino_auto_benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino_auto_benchmark/lists"}